Concrete scenario
A primary acknowledges an order and fails while one replica is caught up, another lags, and a read-only region is partitioned.
Mechanics
Single-leader replication orders writes at a leader; multi-leader accepts writes in several domains and needs conflict handling; leaderless systems contact configurable replica sets. Synchronous modes wait for a defined set of durable acknowledgments—not necessarily every replica. Read and write quorums overlap only under the actual routing and failure assumptions.
Failure mode
Acknowledging one replica does not guarantee zero loss under correlated failure, nondurable acknowledgment, or promotion of a stale node. W + R > N alone is insufficient with sloppy quorums, concurrent writes, failed writes, or weak version resolution.
Trade-off
More acknowledgments improve durability and freshness but add latency and may reject writes during faults. Asynchronous replicas improve latency and read scale while exposing lag and recovery-point risk.
Worked example: what failover actually loses
A primary handles 5,000 writes per second, replicating asynchronously to two replicas. Replica A lags by 50 ms and replica B by 2 seconds, because it is in another region. The primary's disk dies.
- Promote A: about 5,000 x 0.05 = 250 acknowledged writes are lost.
- Promote B: about 10,000 acknowledged writes are lost.
- With semi-synchronous replication to A (the primary waits for A to acknowledge receipt before acknowledging the client), promoting A loses nothing that was acknowledged, but every commit pays the round trip to A, roughly 0.5 to 1 ms in the same region.
Now add a network blip. If A is slow to acknowledge, MySQL semi-sync by default falls back to async after rpl_semi_sync_master_timeout (10 seconds). Your zero-loss guarantee quietly degrades exactly when you need it. That is why Raft-based systems require a majority acknowledgement with no fallback.
Topologies compared
| Topology | Write availability | Conflicts | Read freshness | Examples |
|---|---|---|---|---|
| Single leader, async | Leader only | None | Replicas lag | MySQL and Postgres defaults |
| Single leader, consensus | Needs a majority | None | Linearizable through the leader | etcd, CockroachDB ranges, Kafka KRaft metadata |
| Multi-leader | Every region | Must be resolved | Regional | BDR for Postgres, CouchDB, collaborative editors |
| Leaderless quorum | Any W nodes | Must be resolved (vector clocks, LWW, CRDTs) | Tunable through R and W | Cassandra, Riak, DynamoDB internals |
Quorum arithmetic and its limits
With N = 3, W = 2 and R = 2, every read quorum overlaps every write quorum in at least one node, so a read can see the latest acknowledged write. It still breaks when:
- Sloppy quorums and hinted handoff accept writes on nodes outside the key's home set, so the overlap is lost.
- A failed write that reached one node is not rolled back. Later reads may or may not see it.
- Concurrent writes resolved by timestamp can drop one write silently.
Failure modes
- Split brain: the old primary comes back and accepts writes. Fence it (STONITH, epoch numbers) before promoting a new one.
- Replication lag spikes from a long transaction or a schema migration. Alert on lag in seconds, not just bytes.