The Reality of Distributed Physics: Why Networks Fail
A distributed system is a collection of autonomous computing nodes that communicate over a network, coordinating actions by passing messages. In single-machine systems, memory reads and function calls are deterministic, microsecond operations. In distributed systems, computing across physical space introduces the 8 Fallacies of Distributed Computing (the first seven listed by Peter Deutsch at Sun in 1994; the eighth, "the network is homogeneous", added by James Gosling around 1997):
- The network is reliable.
- Latency is zero.
- Bandwidth is infinite.
- The network is secure.
- Topology doesn't change.
- There is one administrator.
- Transport cost is zero.
- The network is homogeneous.
Failure Models in Distributed Infrastructure
The Three Distributed Failure Modes
1. Crash-Stop (Fail-Stop)
A node halts operations abruptly (e.g., kernel panic, power loss, OOM kill). Once stopped, it never transmits another packet. This is the simplest failure model to design for, as dead nodes remain silent.
2. Crash-Recovery
A node crashes, restarts after several seconds or minutes, reads its persistent write-ahead log (WAL) from disk, and rejoins the cluster with state that lags behind the current state. Algorithms must handle stale reads and catch-up log streams.
3. Byzantine Faults (Arbitrary / Malicious)
Nodes transmit corrupted messages, lie about state, or act maliciously. Solved via Byzantine Fault Tolerant (BFT) consensus (e.g., PBFT, Tendermint, HotStuff; Proof-of-Stake chains pair stake-based Sybil resistance with protocols like these). In private datacenter environments, engineers assume non-Byzantine failure models (Crash-Recovery) to avoid BFT's O(N2) message complexity.
Worked example: why a timeout cannot tell you what happened
Service A calls B to charge a card, with a 1-second timeout. Measured p99 latency is 300 ms, but during GC pauses B sometimes takes 4 seconds. A times out and retries.
- Case 1: the request was lost. Retrying is correct.
- Case 2: B charged the card and the response was lost. Retrying double-charges.
- Case 3: B is mid-GC and will process both the original and the retry.
The only safe design is an idempotency key. A sends charge_id=abc, and B stores it with the result in the same transaction, so the retry returns the stored result.
Timing models
| Model | Assumption | What is possible | Reality check |
|---|---|---|---|
| Synchronous | Known upper bounds on message delay and processing time | Perfect failure detection by timeout | Not true of real networks or GC runtimes |
| Partially synchronous | Bounds hold eventually, after some unknown time | Consensus (Raft, Paxos) with safety always and liveness once stable | The practical model for datacentres |
| Asynchronous | No timing bounds at all | No deterministic consensus (FLP, 1985) | A useful worst case for proving safety |
Node failure models
| Model | Behaviour | Typical defence |
|---|---|---|
| Crash-stop | Halts and never returns | Replication plus failover |
| Crash-recovery | Halts, restarts, may lose in-memory state | Durable logs (fsync), rejoin and catch up |
| Omission | Drops some messages | Retries with idempotency |
| Byzantine | Arbitrary or malicious behaviour | BFT protocols with 3f+1 nodes to tolerate f faults, plus signatures |
Real-world evidence
- Bailis and Kingsbury's survey "The Network is Reliable" documents real partitions at Google, Microsoft, Amazon and others, including partial and asymmetric ones.
- Jepsen tests have repeatedly found data loss in databases whose designs assumed clocks or networks behave well.
- Google's Chubby and Spanner papers both treat long process pauses and clock uncertainty as normal operating conditions.
Design checklist
- Every remote call has a timeout, and every retried operation is idempotent.
- Every piece of state has exactly one owner, or an explicit conflict-resolution rule.
- Failure detection is treated as a suspicion, never a certainty. Anything that acts on a suspicion, such as failover, must fence the old owner.