Read Replicas: The Lag You Need to Design Around
Adding a read replica is the easiest scaling win there is — until a user saves their profile, gets redirected, and sees the old values staring back. Replicas are always a little bit behind, and that gap isn't a bug you can fix. It's a property you design around.
The Bug That Only Happens Right After a Write
A user edits their display name and hits Save. Your API writes the row, returns 200, and the client redirects to the profile page. The profile page loads — showing the old name.
They refresh. Still old. They refresh again three seconds later and it's correct.
Nothing is broken. The write went to the primary. The read went to a replica that hadn't applied that write yet. You added a read replica last month to take pressure off the primary, it worked beautifully, and it also quietly introduced a window of time where your database disagrees with itself.
That window is replication lag, and every replica has it. You can shrink it, you can measure it, you cannot make it zero. What you can do is decide, per query, whether you can live with it.
What a Replica Actually Is
A read-only copy of your database that continuously receives and applies the primary's changes. The primary streams its write-ahead log (WAL in PostgreSQL, binlog in MySQL) to each replica, and the replica replays those records to reproduce the same state. Clients can SELECT from a replica, but every INSERT, UPDATE, and DELETE still goes to the single primary.
The critical detail is in the word continuously. Replication is a pipeline, not a broadcast. A write becomes visible on a replica only after it has been written to the primary's log, shipped over the network, received by the replica, written to the replica's log, and finally replayed into its data files. Each of those steps takes time.
This buys you exactly one thing: read capacity. Ten replicas mean ten machines answering SELECTs. It buys you nothing on the write side — every write still funnels through one primary. If your bottleneck is write throughput, replicas won't move it, and you're looking at sharding instead.
Where the Lag Actually Comes From
"The network is slow" is rarely the real answer. Lag has a few distinct sources, and they fail in different ways.
| Source | What's happening | Typical scale |
|---|---|---|
| Network transfer | WAL records travelling primary → replica | Sub-millisecond same-AZ; 50-150ms cross-region |
| Replay throughput | Replica applying records, often far less parallel than the primary's writers | Grows without bound under write bursts |
| Long-running query on the replica | A big report holds a snapshot; replay conflicts with it and stalls or cancels the query | Seconds to minutes |
| Replica I/O saturation | Replay competes with read traffic for the same disk | Grows with read load |
| Write bursts | A bulk import or migration produces WAL faster than replay can drain it | Minutes to hours |
The one that surprises people is replay throughput. The primary applies your writes with many concurrent backends; a replica replaying that log has far less concurrency to work with. A bulk UPDATE that the primary chews through in 30 seconds can leave a replica minutes behind, and the lag keeps growing the whole time the write burst continues.
application_name | state | write_lag | flush_lag | replay_lag ------------------+-----------+-----------+-----------+------------ replica-a | streaming | 00:00:00.001 | 00:00:00.002 | 00:00:00.004 replica-b | streaming | 00:00:00.002 | 00:00:00.003 | 00:00:12.884 -- replica-b is nearly 13 seconds behind on replay. -- Network is fine (write_lag is 2ms). The bottleneck is apply.
Notice the three separate columns. write_lag tells you about the network, replay_lag tells you what readers actually see. They diverge, and only the last one matters to your users.
⚠️ Lag looks perfect right up until it doesn't
On an idle system, replication lag is microseconds, so a dashboard showing "lag: 0ms" tells you almost nothing. Lag is a function of write volume. Measure it during your peak write window and during a migration — that's where the number you have to design against lives. Alert on lag and on lag's rate of change; a steadily climbing replay lag means replay has fallen behind and will not catch up on its own.
Synchronous Replication: Trading Latency for Freshness
You can make the primary wait for replicas before it confirms a commit. This closes the freshness gap and opens a different one.
| Mode | Primary waits for | Read-after-write | Cost |
|---|---|---|---|
| Asynchronous | Nothing — commits locally, ships later | Stale reads possible | Fastest writes; data loss window on failover |
| Synchronous (flush) | Replica to durably write the WAL record | Still stale — written ≠ replayed | Every commit pays a network round-trip |
| Synchronous (apply) | Replica to actually replay the record | Fresh on that replica | Slowest commits; a slow replica slows all writes |
| Quorum / semi-sync | N of M replicas to acknowledge | Fresh on the quorum only | Balanced; still couples write latency to replica health |
The subtlety worth internalizing: synchronous replication that waits for a durable write is not the same as waiting for the data to be visible. The record is safely on disk at the replica while still sitting unapplied in its log. A SELECT there still returns the old row. If you configure synchronous replication expecting read-your-writes and don't specifically wait on apply, you paid the latency and didn't buy the guarantee.
🔴 Synchronous replication couples your uptime to your replicas
With synchronous_commit waiting on a replica, that replica is now on the critical path of every write. If it gets slow, your writes get slow. If it dies and you haven't configured a fallback set of candidates, your writes can block entirely. You've turned a spare read machine into a dependency. Usually the right call is asynchronous replication plus explicit handling of the stale reads it produces.
Four Ways to Design Around Lag
You don't need every read to be fresh. You need the right reads to be fresh. These are the patterns, roughly in order of how often they're the correct answer.
Route by query, not by connection
Classify reads at the call site. Analytics, dashboards, search, feeds, recommendations, most list views — all fine on a replica. Anything that feeds a decision the user is about to act on, or that gets written back, goes to the primary. This single distinction solves most of the problem and costs you nothing at runtime.
Read-your-writes via a sticky window
After a user performs a write, route that user's reads to the primary for a short window — a few seconds, tracked in their session or a cookie. They see their own change instantly; everyone else's reads still go to replicas. Cheap, and it fixes the profile-page bug directly.
Wait for a specific position
The precise version: capture the primary's log position (LSN) at commit time, pass it along with the request, and have the replica wait until it has replayed at least that far before serving the query. You get correctness without sending traffic to the primary — at the cost of a request that sometimes waits. Needs a timeout and a fallback to the primary.
Bounded staleness — measure and reject
Let the router check each replica's current lag and skip any replica beyond a threshold, falling back to the primary or to a fresher replica. This turns "unknown staleness" into "at most N seconds stale," which is something you can actually put in a design doc and reason about.
✅ Monotonic reads matter as much as fresh reads
Two replicas at different lag positions will happily serve a user alternating requests that go backwards in time — item present, then absent, then present again. That's more confusing than plain staleness. Pin a given user's reads to a single replica for the duration of their session so their view only ever moves forward.
Failover: The Other Reason Lag Matters
Replicas aren't only for scaling reads. They're your standby for when the primary dies. And that's where asynchronous lag becomes a data question rather than a freshness question.
If the primary fails with a replica 12 seconds behind, and you promote that replica, those 12 seconds of committed writes are gone. The primary told those clients "committed." Your users got confirmation emails for orders that no longer exist.
This is the real argument for synchronous replication, and it's a much better argument than read freshness. If losing the last few seconds of writes is unacceptable — payments, ledgers, anything where "we lost your transaction" is a legal problem — you want at least one synchronous replica and you accept the write latency. If losing a few seconds means re-sending some analytics events, async is fine.
⚠️ Two primaries is worse than no primary
A failover that promotes a replica while the old primary is still alive and accepting writes gives you split-brain: two databases diverging, with no automatic way to merge them. Whatever promotes your replica must first be certain the old primary is fenced off — killed, network-isolated, or demoted. Automatic failover without fencing turns a brief outage into a manual data-reconciliation project.
Deciding Where a Query Goes
Does this query write, or read data it's about to write back?
Key Takeaways
- Replicas scale reads, not writes. Every write still lands on one primary. If writes are your ceiling, replicas won't raise it.
- Replication lag is a property, not a defect. It's near-zero when idle and grows under exactly the conditions you care about: write bursts, migrations, and heavy read load on the replica.
replay_lagis the number that matters to readers. Network lag being fine tells you nothing about whether your data is visible.- Synchronous replication buys durability more than freshness. Waiting for a durable write at the replica doesn't make the data readable there, and it puts replica health on your write path.
- Classify reads at the call site. Most reads tolerate staleness. A small set doesn't. The whole design problem is knowing which is which — then adding sticky windows or lag thresholds for the ones that don't.
- Pin a user's session to one replica so their view of the world never moves backwards.
- Failover with async replication loses the lag window. Decide in advance whether those seconds are recoverable or catastrophic, and fence the old primary before promoting anything.
References
- PostgreSQL — Hot Standby and monitoring replication — replay conflicts,
pg_stat_replication, and standby behaviour - PostgreSQL — Synchronous replication modes — the difference between
remote_write,on, andremote_apply - Designing Data-Intensive Applications, Ch. 5 — Replication — replication lag, read-your-writes, and monotonic reads from first principles
- Jepsen — analyses of replicated database behaviour under failure — what actually happens to these systems during partitions and failovers