Concrete scenario
A cache normally absorbs 99 percent of 100,000 reads per second, then becomes unreachable.
Mechanics
Protect the database with a concurrency limit or rate limiter based on measured capacity. Fail fast on cache timeouts, serve explicitly bounded stale local data where allowed, and reject excess work. A soft TTL starts refresh; a hard TTL is the product limit after which stale data must not be served.
Failure mode
Unrestricted fallback can overload the database, but a high hit ratio alone does not prove how the database was provisioned. Failover may expose cold replicas or stale data, and retrying cache calls can exhaust workers.
Trade-off
Stale responses preserve availability only for data whose correctness permits them. Redundant cache clusters reduce outage probability but add cost, failover complexity, replication lag, and recovery miss storms.
Worked example: sizing the fallback
Traffic is 100,000 reads per second with a 99 percent hit ratio, so the database normally serves 1,000 qps. Load tests show it saturates at 4,000 qps, with p99 latency degrading above 3,000. If the cache disappears:
- Unprotected, the database receives 100,000 qps, 25 times its capacity. Connection pools fill, latency explodes, and requests for uncached data fail too.
- With a fallback limiter of 2,500 qps spread across app servers (for example 25 qps each on 100 instances), the database stays healthy. The remaining 97,500 rps must be served from an in-process stale copy, served as a degraded response, or rejected quickly with 503 and Retry-After.
- If a local L1 with a 60-second soft TTL covers 80 percent of keys, only about 20,000 rps need the database, and the limiter decides which 2,500 of those get through.
Options when the cache is down
| Strategy | Availability | Correctness | Cost |
|---|---|---|---|
| Unlimited DB fallback | Collapses under load | Fresh while it lasts | None, until the outage |
| Rate-limited fallback plus fast reject | Partial | Fresh for admitted requests | Limiter and tuning |
| Serve stale from L1 within a hard TTL | High | Bounded staleness | Memory on every instance |
| Replicated cache across zones | High | Replication lag | Two to three times the cache cost |
| Gutter pool (Facebook) | High | Short TTLs limit staleness | A small spare cache pool |
| Static degraded response | Highest | Feature partially off | Product design effort |
How real systems do it
- Netflix EVCache writes to every availability zone and reads from the local zone. On a miss or timeout it reads from another zone before going to the source of truth.
- Facebook routes requests for a failed memcache host to a small Gutter pool with short TTLs instead of rehashing onto healthy hosts, which avoids a cascade.
- Circuit breaker around the cache client: after repeated timeouts, skip the cache immediately and go straight to the rate-limited fallback, rather than waiting on every request.
Recovery
When the cache returns empty, the hit ratio climbs from 0 percent. Keep the fallback limiter on during warm-up. Optionally pre-warm the top N keys from access logs, and shift traffic back in steps (10, 25, 50, 100 percent).