Under the Hood: Graceful Degradation Under Overload
When demand exceeds maximum hardware capacity, a system cannot simply try to serve everyone; attempting to process 150,000 QPS on a cluster sized for 100,000 QPS pushes CPU and queueing delays past client timeouts. The system drowns in work that completes too late to be useful, and goodput collapses: nearly all 150,000 QPS time out, not just the excess 50,000. Load Shedding intentionally rejects lower-priority traffic to ensure core operations complete successfully.
Priority Ingress Tier with Load Shedding
Under the Hood: Adaptive Concurrency Limits
Static rate limits (e.g., max 10,000 requests/sec) fail because server capacity varies dynamically with garbage collection, cache hit ratios, and network latency. Netflix's open-source concurrency-limits library implements Adaptive Concurrency Limits based on TCP congestion-control ideas such as Vegas and on Little's Law (L = λ W). (Stripe's published approach instead combines rate limiters with priority-based load shedders.)
- The system continuously measures minimum round-trip time (minRTT) and current in-flight requests (L).
- If observed latency increases while concurrency is climbing, queueing has begun.
- The concurrency limit dynamically contracts, rejecting excess requests at the gateway boundary with
HTTP 503(ideally withRetry-After; per-client quota violations use429) before server thread pools exhaust.
Worked example: goodput under overload
A service has 100 worker slots, each request needs 50 ms of work, and clients time out at 1 s. Capacity is therefore 100 / 0.05 = 2,000 requests per second.
- Offered 3,000 per second, no shedding, unbounded queue: the queue grows by 1,000 per second. After a few seconds, queueing delay passes 1 s, so requests are served only after their client has already given up. The server stays 100% busy but goodput (successful, in-time responses) can fall toward zero. This is congestion collapse.
- Offered 3,000 per second with a queue-time limit: reject any request that has waited more than, say, 200 ms, or cap in-flight plus queued work at about 400. Now roughly 2,000 per second succeed within the deadline and 1,000 per second get a fast 503. Goodput stays at capacity.
- With priorities: if 30% of traffic is batch or prefetch requests, shedding those first means user-facing requests (2,100 per second) are almost all served.
Little's law ties this together: in-flight requests = throughput x latency. At 2,000 per second and a 50 ms target, only about 100 requests should be in flight. A concurrency limit near that number is a direct way to protect latency.
Adaptive concurrency limits
Netflix's open-source concurrency-limits library adjusts the allowed in-flight count using ideas from TCP congestion control (for example Vegas and Gradient algorithms): it tracks the minimum observed latency as the no-queue baseline, and when measured latency rises above it, queueing has started and the limit is reduced; when latency is at the baseline, the limit probes upward. Requests over the limit are rejected immediately.
Strategy table
| Technique | What it protects | Trade-off |
|---|---|---|
| Per-client rate limiting (token bucket) | Fairness between clients | Does not reflect the server's actual capacity |
| Static concurrency limit | Worker pool and memory | Needs retuning as the service changes |
| Adaptive concurrency limit | Latency near its no-queue baseline | Can oscillate; needs good latency signals |
| Queue time deadline (drop requests that waited too long) | Wasted work on requests clients abandoned | Requires request arrival timestamps |
| LIFO or CoDel-style queueing under overload | Fresh requests still succeed | Old requests starve |
| Criticality-based shedding | Business-critical traffic | Requires tagging every request |
| Degraded responses (cached or partial) | User experience | Stale or incomplete data |
Failure modes
- Expensive rejections: if rejecting still requires TLS handshake, logging and auth, a flood can overload the server anyway; shed as early as possible, even at the load balancer.
- Health checks shed too: if health checks are rejected, the load balancer marks nodes unhealthy and sends even more traffic to the survivors. Exempt them.
- Metastable states: retries and cold caches can keep the system overloaded after the trigger ends; combine shedding with retry budgets.
How real systems do it
Google tags RPCs with criticality levels (for example CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS, SHEDDABLE), as described in the SRE book's chapter on handling overload. Amazon services reject excess work early and measure goodput separately from throughput. Facebook uses adaptive LIFO and CoDel-style queue management in its RPC servers so that under overload recent requests still succeed.