The Fallacies of Distributed Systems: What Will Go Wrong
Your code works flawlessly on your laptop, then falls apart the moment it talks to another machine over a network. That's not bad luck — it's a set of assumptions every developer makes and the network quietly breaks. Here are the eight fallacies and how to design around them.
The Bug That Only Happens in Production
On your laptop, the service call returns in a millisecond and never fails. In staging, same thing. Then you ship, traffic ramps, and suddenly requests hang, retries pile up, and a single slow downstream service drags your whole app down with it.
Nothing in your code changed. What changed is that your code is now making the same silent assumptions everyone makes about networks — assumptions that are fine on localhost and false across a real network. In 1994, Peter Deutsch and colleagues at Sun catalogued these as the Fallacies of Distributed Computing. Thirty years later, developers rediscover them the hard way in every new stack.
The value isn't memorizing the list. It's recognizing, when you write await otherService.call(), exactly which comforting lies you're about to believe.
The Eight Fallacies
Each one is a statement that feels true and is not.
| The Fallacy (what you assume) | The Reality | What it costs you if ignored |
|---|---|---|
| The network is reliable | Packets drop, connections reset, services vanish | Lost requests, silent data loss |
| Latency is zero | Every hop adds real, variable delay | Chatty designs that crawl in prod |
| Bandwidth is infinite | Pipes are finite and shared | Congestion, timeouts under load |
| The network is secure | It's hostile by default | Interception, tampering, breaches |
| Topology doesn't change | Nodes move, scale, and fail constantly | Hardcoded hosts, stale routing |
| There is one administrator | Many teams, clouds, and vendors | No single place to fix or trace |
| Transport cost is zero | Serialization + bandwidth cost money and CPU | Surprise bills, wasted cycles |
| The network is homogeneous | Mixed protocols, versions, hardware | Incompatibility, subtle corruption |
The first two cause the most day-to-day pain, so they're worth pulling apart.
Fallacy #1: The Network Is Reliable
Here's the assumption in code form:
// This will fail. Not "might" — will.
const result = await paymentService.charge(order);
saveOrder(order, result); // assumes the line above always returnsThat call travels across switches, load balancers, and NICs, any of which can drop it. The service on the other end can crash mid-request. And the worst case isn't a clean failure — it's the ambiguous one: the request succeeded but the response got lost. Did the charge happen? You genuinely don't know.
You can't make the network reliable. You can only make your code resilient to it:
Timeouts on everything
A call with no timeout can hang forever, holding a thread or connection. Every network call needs a deadline — and a plan for what happens when it's hit.
Retries with backoff
Retry transient failures, but with exponential backoff and jitter so you don't hammer a struggling service into the ground (the "retry storm").
Idempotency for safe retries
Since you can't tell "failed" from "succeeded-but-lost-the-response," retries must be safe to run twice. That means idempotency keys on anything that mutates state.
Circuit breakers
When a downstream is clearly down, stop calling it for a while. Fail fast locally instead of piling up timeouts that exhaust your own resources.
⚠️ The retry storm
Naive retries turn a small blip into an outage. A downstream hiccups, every caller retries at once, the extra load keeps it down, callers retry again. Always combine retries with backoff, jitter, and a circuit breaker — retries without limits are a self-inflicted DDoS.
Fallacy #2: Latency Is Zero
On localhost, a function call and a service call feel identical. Across a data center they're not even close, and across regions they're worlds apart.
The number itself matters less than what it does to design. If one service call costs 2ms and your endpoint makes 50 of them sequentially, that's 100ms of pure waiting — before any real work. This is why chatty architectures die in production: they were designed as if latency were zero.
✅ Design against latency, not around it
Batch requests instead of looping. Fetch in parallel instead of in sequence. Move data closer to compute. Cache aggressively. And measure the p99 latency of every dependency — the average lies, and it's the slow tail that takes you down.
The Rest, Briefly
- Bandwidth isn't infinite. Shipping large payloads (fat JSON, N+1 over the wire) saturates links and inflates latency. Send less; compress; paginate.
- The network isn't secure. Assume every hop is hostile. Encrypt in transit (TLS), authenticate service-to-service (mTLS), and never trust input just because it came from "inside."
- Topology changes. Instances autoscale, get rescheduled, and fail. Never hardcode a host — use service discovery and DNS, and expect any address to disappear.
- There isn't one admin. Your request crosses teams, clouds, and vendors. This is why distributed tracing exists: without a trace ID following the request, "why is this slow" has no single place to look.
- Transport isn't free. Serialization burns CPU; bandwidth costs money (especially cross-region egress). Format choice — JSON vs Protobuf — is a real performance and cost decision.
- The network isn't homogeneous. Mixed protocol versions and encodings cause subtle corruption. Version your APIs and standardize your wire formats.
Turning the Fallacies Into a Checklist
The fallacies are most useful as a design-review lens. Whenever you add a network call, walk the list:
Does this call have a timeout AND a defined failure behavior?
Key Takeaways
- The fallacies are the default assumptions your code makes that
localhostupholds and real networks violate. Distributed bugs are these assumptions failing. - The network is not reliable and latency is not zero cause most production pain. Answer them with timeouts, backoff+jitter retries, idempotency, and circuit breakers — and with batched, parallel, cache-friendly designs.
- Treat every network call as hostile, slow, and capable of vanishing. Assume the worst case (succeeded-but-lost-response) and make it safe.
- Use the fallacies as a design-review checklist, not trivia. Every new call is a chance to believe one of them by accident.
References
- Fallacies of Distributed Computing Explained (Rotem-Gal-Oz) — the classic essay expanding each fallacy with mitigations
- Timeouts, retries, and backoff with jitter — AWS Builders' Library — production patterns for the unreliable-network problem
- Latency Numbers Every Programmer Should Know — the orders-of-magnitude that make "latency is zero" so dangerous