Cascading Failures: How a Slow Downstream Kills the Fleet
In a microservices mesh, a single degraded downstream dependency (e.g., a recommendation engine with latency spiking from 20ms to 5,000ms) is far more dangerous than a completely dead dependency. When a service stalls, upstream callers keep connections and worker threads open while waiting for response timeouts.
Within seconds, thread pools exhaust across the API gateway and web tiers, causing total application blackout: a cascading collapse.
Circuit Breaker State Machine Under the Hood
Detailed Circuit Breaker Mechanics
1. Closed State
All traffic flows through to the downstream service. The circuit breaker monitors responses in a sliding ring-buffer window (e.g., the last 100 requests). If the percentage of slow calls (latency > threshold) or HTTP 5xx errors exceeds the error rate threshold (typically 50%), the breaker trips to Open.
2. Open State (Fail-Fast)
The breaker immediately short-circuits all incoming calls without sending network packets downstream. Requests instantly execute fallback logic (serving cached data or returning HTTP 503). This grants the struggling downstream service time to recover its database connection pools.
3. Half-Open State
After a configured sleep window (e.g., 15 seconds), the breaker transitions to Half-Open. It permits a small number of trial requests (e.g., 5 probe requests). How the trial results are judged depends on the implementation: some breakers reopen on the first failed probe, while Resilience4j applies the failure-rate (and slow-call-rate) threshold to the permitted half-open calls, closing if the rate is below the threshold and reopening for another sleep interval otherwise.
Exponential Backoff with Full Jitter Formula
When clients retry failed requests simultaneously, they synchronize into devastating periodic retry waves (resonance). Clients must apply Exponential Backoff with Full Jitter:
// Exponential Backoff with Full Jitter (AWS Recommended)
function getBackoffTime(attempt, base = 100, cap = 10000) {
const temp = Math.min(cap, base * (2 ** attempt));
return Math.random() * temp; // Uniform random between 0 and temp
}
Worked example: why slow is worse than dead
An API tier has 200 worker threads per instance and handles 1,000 rps per instance. Each request calls a recommendations service, normally in 20 ms.
- Recommendations dead (connection refused in 1 ms): threads are freed instantly. The API returns errors or fallbacks, but stays responsive.
- Recommendations slow (5 s responses, 10 s timeout): by Little's law, concurrent in-flight requests are 1,000 x 5 = 5,000, but only 200 threads exist. The pool is exhausted in 0.2 seconds, and every endpoint on the instance stalls, including ones that never call recommendations.
- With a 150 ms timeout and a breaker: at most 1,000 x 0.15 = 150 threads are tied up. Once the failure rate crosses 50 percent over a 10-second window, the breaker opens and calls return the fallback in microseconds. Thread use drops to near zero.
Choosing timeouts
- Set the timeout from the dependency's measured p99.9 plus a margin, not a round number. Revisit it when latency changes.
- Use separate connect and read timeouts. Connect should be short, around 100 to 500 ms in a datacentre.
- Propagate a deadline, for example gRPC deadlines, so downstream services stop work when the caller has given up.
Resilience patterns compared
| Pattern | Protects | Failure it handles | Risk if misused |
|---|---|---|---|
| Timeout | The caller's threads | Slow dependency | Too short a timeout causes false failures |
| Retry with backoff and jitter | Success rate | Transient errors | Retry storms, and duplicates if not idempotent |
| Retry budget (for example, at most 10 percent extra) | The dependency | Retry amplification | Slightly lower success during blips |
| Circuit breaker | Both sides | Sustained failure | Flapping if thresholds are noisy |
| Bulkhead (separate pools) | Other features | One dependency hogging threads | Under-used capacity |
| Load shedding | The server itself | Overload | Must shed low-priority work first |
How real systems do it
- Netflix Hystrix (now in maintenance) popularised breakers plus bulkheads. Resilience4j is the common JVM successor, with count- or time-based sliding windows and a configurable number of half-open trial calls.
- Envoy and Istio implement breaking as connection and pending-request limits plus outlier detection, which ejects failing hosts, rather than the classic three-state machine.
- AWS SDKs use exponential backoff with jitter and a client-side retry token bucket (retry quota).
- Google SRE recommends client-side adaptive throttling: reject locally with probability based on the recent ratio of accepted requests to total requests.