The Role of Distributed Coordination Systems
Modern microservices and distributed storage systems (Kubernetes, HBase, Hadoop, and Kafka before KRaft) require a centralized, strongly consistent brain to manage cluster topology, service IP discovery, active leader registration, and feature flags. This is the domain of Distributed Consensus Key-Value Stores: Apache ZooKeeper, CoreOS Etcd, and HashiCorp Consul.
Hierarchical Node Trees & Ephemeral Sessions
Under the Hood: Key Architectural Primitives
1. Ephemeral Nodes & Heartbeat Leases
When a payment service instance boots, it opens a session with ZooKeeper/Etcd and registers an Ephemeral Node at /services/payment-service/node_01 with its IP and port. Ephemeral nodes exist only as long as the client's heartbeat session remains active. If the instance crashes, the session times out, and the cluster manager automatically deletes the ephemeral node.
2. Event Watchers (Push Notifications)
Clients avoid polling the metadata service. Instead, they register Watchers on parent nodes (e.g., watch('/services/payment-service')). Etcd uses HTTP/2 multiplexed streams, while ZooKeeper uses persistent TCP connections. When a child node is added or deleted, the server pushes an event to all subscribed gateways, which then update their routing tables. Classic ZooKeeper watches are one-shot triggers: the client must re-read the node and re-register the watch, so changes in between are seen only through that re-read. Persistent and recursive watches arrived in ZooKeeper 3.6. etcd watches are streams that can resume from a revision.
3. Distributed Leader Election via Sequential Ephemeral Nodes
Candidates create a sequential ephemeral node: /election/leader_000000001, /election/leader_000000002. Whichever candidate holds the lowest numerical suffix is the elected leader. Non-leaders watch the node immediately preceding them in sequence, eliminating the 'herd effect' where all nodes wake up on leader crash.
Worked example: service registration with an etcd lease
- An instance of
paymentsstarts up and grants a lease with a 10-second TTL (lease ID 0x6b). - It writes
/services/payments/10.0.1.5:8080attached to lease 0x6b. - It sends a KeepAlive every 3 seconds.
- The gateway watches the prefix
/services/payments/starting from revision 5,120, and receives a PUT event. - The instance is killed by the OOM killer. KeepAlives stop, and within about 10 seconds etcd revokes the lease and deletes the key.
- The gateway receives the DELETE event at revision 5,187 and removes the instance from its pool.
The worst-case detection time is the TTL. A shorter TTL detects failures faster but risks false expiry during GC pauses or network blips. Most deployments land between 5 and 30 seconds and also use active health checks at the load balancer.
Discovery approaches
| Approach | Consistency | Failure detection | Examples |
|---|---|---|---|
| Consensus KV with sessions or leases | Linearizable writes | Lease or session expiry | ZooKeeper, etcd, Consul catalogue |
| Gossip membership | Eventual | SWIM or phi-accrual | Cassandra, Serf, memberlist |
| DNS-based | Eventual, and cached by TTL | External health checks | Kubernetes Services, AWS Cloud Map |
| Self-preservation registry | AP by design | Heartbeats, but eviction pauses during mass loss | Netflix Eureka |
| Control plane pushes to proxies | Eventual (xDS) | Health checks plus outlier detection | Envoy with Istio, Consul Connect |
How real systems do it
- Kubernetes stores all cluster state in etcd. Controllers use list-then-watch from a resourceVersion, and the API server caches state so clients do not hammer etcd.
- Kafka kept broker registrations and controller election in ZooKeeper for a decade, then replaced it with KRaft (a Raft metadata quorum inside Kafka). ZooKeeper mode was removed in Kafka 4.0.
- HBase finds the meta table and the active master through ZooKeeper, and RegionServers register ephemeral znodes.
Failure modes
- Herd effect when thousands of clients watch one key. Use the sequential "watch your predecessor" recipe, or a caching layer.
- Session expiry storms after a coordinator pause: many ephemeral nodes vanish at once and trigger mass rebalancing.
- An etcd database that fills its quota (2 GB by default) raises a NOSPACE alarm and the cluster becomes read-only until you compact and defragment it.