Consistency vs Availability: The CAP Theorem in Practice
CAP gets taught as "pick two of three," which is both wrong and useless. The real theorem says something much narrower and much more actionable: when the network splits, you either refuse the request or you serve possibly-wrong data. Here's what that choice looks like in real systems.
You Already Made This Choice
Two nodes of your database sit in different availability zones. The link between them drops — not a crash, just silence. Each node is up, healthy, and now completely unable to tell whether the other one is dead or merely unreachable.
A write arrives at one of them.
There are exactly two things that node can do. It can accept the write, knowing the other node has no idea it happened and may be accepting conflicting writes of its own. Or it can refuse the write, staying correct at the cost of telling a perfectly healthy user "try again later."
There is no third option. Whichever your system does, it does because someone configured it that way — usually a default, usually not deliberately. That's the CAP theorem. Not a menu of three things to pick two from. A single forced choice, and only during a partition.
What the Theorem Actually Says
For a distributed system, Consistency (every read sees the most recent write — technically, linearizability), Availability (every request to a non-failing node gets a non-error response), and Partition tolerance (the system keeps working when the network drops messages between nodes) cannot all hold simultaneously. Proven by Gilbert and Lynch in 2002, from a conjecture by Eric Brewer in 2000.
The framing that ruins it is "pick two." Partition tolerance isn't a design choice — it's a description of the world. Networks partition. Cables get cut, switches reboot, cloud AZs lose connectivity, a bad BGP update makes half your fleet unreachable. If your system runs on more than one machine, you don't get to opt out of partitions any more than you get to opt out of gravity.
So P isn't on the menu. The theorem reduces to: during a partition, choose C or A. That's it. When the network is healthy — which is almost always — you can have both, and CAP has nothing to say about your system at all.
Both clients are talking to a healthy node. Neither node can reach the other. x is about to have two different values, or one of those users is about to get an error. Pick.
CP and AP, Concretely
| CP — choose consistency | AP — choose availability | |
|---|---|---|
| During a partition | Minority side refuses reads/writes | All sides keep serving |
| Failure mode users see | Errors, timeouts, 'service unavailable' | Stale or conflicting data |
| What you must build | Failover, quorum, leader election | Conflict detection and resolution |
| Recovery | Minority rejoins, catches up cleanly | Sides merge; you decide who wins |
| Fits | Balances, inventory, uniqueness, config, locks | Carts, likes, feeds, sessions, metrics, DNS |
| Examples | etcd, ZooKeeper, Consul, Spanner, single-primary Postgres | Cassandra, DynamoDB (eventual mode), Riak, CRDT stores |
The CP systems are the ones running consensus protocols — Raft, Paxos, ZAB. Consensus is precisely the machinery for making a majority agree on an ordering while a minority is cut off. The minority doesn't get to serve, and that's the point: two nodes on the wrong side of a partition can never accidentally elect themselves.
The AP systems flip it. Every node serves. Divergence is expected, so the design work moves to reconciliation: version vectors to detect concurrent writes, last-write-wins if you're willing to lose data, or CRDTs — data types whose merges are mathematically guaranteed to converge regardless of order.
⚠️ Last-write-wins is silent data loss
LWW is the default conflict resolution in a lot of AP stores because it's simple and needs no application logic. It's also a policy that says "when two users write concurrently, throw one of them away, based on clock timestamps from machines whose clocks disagree." For a cache, fine. For a shopping cart or a document, you've just built an unreproducible bug that only appears during network events. Prefer a merge that's actually correct for your data — union the cart, append to a list, use a counter CRDT.
The Part That Matters Every Day: PACELC
Partitions are rare. Yet you feel the consistency-versus-something trade-off constantly, which is a hint that CAP is describing only a corner of the problem.
An extension by Daniel Abadi: if there's a Partition, trade Availability against Consistency (that's CAP) — Else, in normal operation, trade Latency against Consistency. Strong consistency requires coordination between nodes, and coordination costs a network round-trip on every operation. Even with a perfectly healthy network.
This is the trade-off you actually pay for. A linearizable read has to confirm with a quorum that it isn't serving stale data — that's milliseconds within a region, and tens to hundreds of milliseconds across regions. Every strongly consistent read in a multi-region system pays the speed of light.
That's why a single-region read replica serving stale data at 2ms is often the right engineering answer, and why "just make everything strongly consistent" is a performance decision disguised as a correctness decision.
Consistency Isn't a Boolean
"Consistent or eventually consistent" is a false binary. There's a ladder between them, and most useful systems sit in the middle.
| Model | Guarantee | Costs |
|---|---|---|
| Linearizable | Reads see the latest completed write, globally ordered | Quorum round-trip per operation; unavailable on the minority side |
| Sequential | All nodes see operations in the same order, not necessarily real-time | Cheaper than linearizable; reads can lag |
| Causal | If A happened-before B, everyone sees A before B; concurrent writes unordered | Available during partitions; needs causal metadata |
| Read-your-writes | You always see your own changes; others' may lag | Cheap — session pinning or a sticky window |
| Eventual | Given no new writes, replicas converge — eventually | Cheapest and fastest; anything goes in the meantime |
Causal consistency is the underrated one. It's the strongest model that remains available during a partition, and it eliminates the class of bugs users actually notice — a reply appearing before the comment it replies to, a "deleted" item resurfacing above the delete event.
✅ Choose per operation, not per system
The mistake is treating this as a database-wide setting. One application has both kinds of operations. "Add to cart" should never fail — take the write, merge later. "Charge this card" and "claim this username" must be linearizable — a duplicate is worse than an error. Most databases let you pick consistency per query or per transaction. Use that.
Where the Choice Bites in Real Design
Three examples where the abstract choice becomes a concrete product decision:
Username registration — CP, no argument
Two users claim @harry from opposite sides of a partition. Under AP, both succeed and you're left reconciling two accounts that each believe they own the handle. Uniqueness is a global invariant; global invariants need coordination. Refuse the write.
Shopping cart — AP, no argument
An unreachable node means a user can't add an item. That's lost revenue for a guarantee nobody asked for. Accept every write on every side, and merge carts by union on recovery. Amazon's Dynamo paper made this exact argument — the worst case of a merged cart is a resurrected item, which the user removes; the worst case of a rejected write is a lost sale.
Inventory — the genuinely hard one
Stock counts look like they need CP, and at zero they do. But blocking all sales during a partition to prevent overselling a handful of items is usually the wrong trade. The common real answer is neither pure C nor pure A: allow sales optimistically against a reserved buffer, detect oversells on reconciliation, and handle them as a business process — refund, backorder, apologise. The system stays available; the rare conflict becomes a support cost you priced in.
That third one is the lesson underneath the theorem. CAP forces a choice at the storage layer, but you get to decide where the compensation lives. A lot of "we need strong consistency" turns out to mean "we haven't designed what happens when we're wrong."
Picking a Side
If two nodes accept conflicting versions of this write, can you merge them correctly later?
🔴 'Highly available' is not the same as CAP-Available
CAP's A is an absolute: every request to every non-failing node succeeds. That's a stricter bar than a service with 99.99% uptime. A CP system with fast automatic failover is unavailable for a few seconds during a leader election and is, in practice, extremely highly available. Don't reject CP designs because a theorem calls them "not available" — read that word as the narrow technical term it is.
Key Takeaways
- Partition tolerance isn't optional. Networks fail. The real theorem is a binary choice — C or A — and it applies only during a partition.
- CP means the minority side stops serving. You need failover, quorums, and leader election; users see errors, not wrong data.
- AP means every side keeps serving. You need conflict detection and a merge strategy; users see stale or divergent data, not errors.
- Last-write-wins is a data-loss policy, not a conflict resolution strategy. Use a merge that's correct for your data type.
- PACELC is the trade-off you pay daily: even with a healthy network, consistency costs coordination, and coordination costs latency.
- Consistency is a ladder, not a switch. Causal and read-your-writes cover most user-visible complaints without the cost of linearizability.
- Decide per operation. Uniqueness and money need CP. Carts, feeds, and metrics want AP. And for the hard middle, ask where the compensation lives rather than which letter to pick.
References
- Brewer's CAP Theorem: 12 Years Later — Eric Brewer — the author walking back the "pick two" framing
- Please stop calling databases CP or AP — Martin Kleppmann — why the labels are too coarse to describe real systems
- Consistency Tradeoffs in Modern Distributed Database Design — Daniel Abadi — the PACELC formulation
- Dynamo: Amazon's Highly Available Key-value Store — the shopping-cart argument for AP, from the source