Concrete scenario
A scheduler leader pauses for 20 seconds after its lease expires while a replacement begins running jobs.
Mechanics
Raft elects a leader by term and majority vote, with an up-to-date-log restriction that preserves committed entries. Election timeout ranges are deployment parameters, not universal constants. Leadership authorizes log coordination; external side effects still need idempotency or fencing accepted by the target.
Failure mode
Randomized timeouts reduce split votes but do not prevent every split-brain effect. A paused or partitioned old leader can continue local work, and clock-based leases can expire unnoticed by that process.
Trade-off
A leader simplifies ordering and ownership but creates failover pauses and a hotspot. Leaderless designs spread coordination differently but move conflict and quorum logic to each operation.
Worked example: a Raft election timeline
Five nodes, heartbeat 100 ms, election timeout randomised between 1000 and 2000 ms. The leader L (term 4) crashes at t = 0.
- t = 1,230 ms: follower B's timer fires first. B increments its term to 5, votes for itself, and sends RequestVote(term 5, lastLogIndex 812, lastLogTerm 4).
- C, D and E have last index 812 or lower and have not voted in term 5, so they grant their votes. B has 4 of 5 votes, a majority, and becomes leader by about t = 1,235 ms.
- B sends heartbeats immediately, which resets the others' timers before they fire. Total unavailability is about 1.2 seconds.
Now suppose L was not dead but paused for 3 seconds. It wakes up still thinking it leads term 4 and sends AppendEntries. Every follower replies with "term 5", so L steps down. Any client write L accepted after waking cannot commit, because it can never reach a majority in term 4.
Election approaches
| Approach | Safety basis | Failover time | Used by |
|---|---|---|---|
| Raft or Zab internal election | Majority vote per term | About the election timeout | etcd, Consul, ZooKeeper |
| Lease in a consensus store | Store's linearizability plus lease TTL | Lease TTL, often 10 to 15 s | Kubernetes controllers (Lease objects), Patroni |
| ZooKeeper sequential ephemeral nodes | Lowest sequence number wins. Session expiry releases it | Session timeout | HBase, older Kafka controllers |
| Bully / ring algorithms | Highest ID wins. Assumes reliable failure detection | Varies | Textbooks. Unsafe under partitions |
| Database row lock or advisory lock | The database's locking | Connection timeout | Simple cron leaders |
Failure modes
- Flapping leadership when timeouts sit near p99 network or disk latency. Raft's PreVote extension stops a partitioned node from disrupting the cluster with inflated terms when it rejoins.
- Leader in the wrong place: if the leader sits in a far region, every write pays the long round trip. Some systems support leader preferences or transfers.
- Clock-based leases on the leader (for fast local reads) are only safe with bounded clock drift. etcd and CockroachDB document this assumption.