Concrete scenario
A worker pauses beyond its lease, then resumes after another worker has acquired the same lock.
Mechanics
A lease limits how long the coordinator recognizes ownership, but the old client cannot infer validity from its local clock. For external writes, use a monotonically increasing fencing token that the target rejects when stale, or an equivalent conditional write/version check enforced by the resource.
Failure mode
A lock service can expire ownership while paused code continues. If the target accepts both writers without fencing, mutual exclusion at the coordinator does not protect the data.
Trade-off
Fencing is powerful when the target can validate it, but it is not the only correctness design: transactions, compare-and-swap, idempotency, or single-owner logs may fit. Locks add availability and recovery dependencies.
Worked example: the paused lock holder
Without the token check at storage, both writes succeed and W1 overwrites W2's newer data. Note that W1 did nothing wrong by its own reckoning: it cannot observe its own pause.
Lease maths
With a lease TTL T, a maximum clock drift rate d, and a holder that renews every T/3:
- The holder should stop working at about T x (1 - d) minus a safety margin, measured with a monotonic clock from when it sent the acquire request, not when it received the reply.
- For T = 10 s the holder might stop at 8 s. But a 15-second pause makes any client-side check meaningless, which is why the check belongs at the resource.
Ways to get mutual exclusion
| Mechanism | Safe against pauses? | Needs resource support | Notes |
|---|---|---|---|
| Lock plus fencing token | Yes | Yes: stores the maximum token and rejects lower ones | ZooKeeper zxid or etcd revision make good tokens |
| Compare-and-set or version column | Yes | Yes, but most databases already have it | Often removes the need for a lock |
| Database row lock (SELECT FOR UPDATE) | Yes, within a transaction | Same database only | Holds a connection |
| Single partition owner (Kafka consumer group) | Mostly, but zombies need fencing | Producer epochs fence zombies | Ownership via rebalance |
| Redis SET NX PX without a token check | No | No | Fine for efficiency and deduplication of work |
How real systems do it
- Chubby issues sequencers that servers can check. It also delays releasing a crashed holder's lock (the lock-delay, up to a minute) to reduce stale writes from services that ignore sequencers.
- Kubernetes leader election uses Lease objects with a resourceVersion compare-and-swap, a 15-second lease duration, and a 10-second renew deadline by default.
- HDFS NameNode HA uses fencing (SSH kill or shared-storage fencing) plus QJM epoch numbers so that an old active node cannot write to the edit log.