Retries, timeouts, and idempotency

Under the Hood: The Retry Storm Catastrophe

When a downstream service experiences high load and begins timing out, naive client retry loops with little or no backoff multiply the offered load, a Retry Storm. In the worst case, where every attempt fails, 10,000 clients each retrying three times turn 10,000 requests into 40,000; with retries at several layers the multipliers compound (4 × 4 × 4 = 64 for three layers). The extra load raises the failure rate, which triggers more retries: a feedback loop that can keep the service overloaded even after the original trigger is gone.

The Production Retry Pattern: Exponential Backoff with Full Jitter

To break up synchronized retry waves, Amazon AWS architecture popularized Full Jitter Exponential Backoff:

// Production-Grade Exponential Backoff with Full Jitter
function calculateBackoff(attempt, baseDelayMs = 100, maxDelayMs = 5000) {
  const exponentialLimit = Math.min(maxDelayMs, baseDelayMs * Math.pow(2, attempt));
  // Uniform random between 0 and exponentialLimit
  return Math.floor(Math.random() * exponentialLimit);
}
      

Resilience Guidelines

Under the Hood: Retry Budgets & Timeout Tiers

• Retry Budgets: A client service should dedicate at most 10% of its total outbound traffic to retries. Once retries would exceed that ratio, further retries are dropped while original requests still go through. A circuit breaker is a separate mechanism that stops sending any requests while the error rate stays high.
• Timeout Splitting: Always set distinct connection timeouts (e.g., 200ms to establish TCP/TLS handshake) and socket read timeouts (e.g., 800ms for backend processing) to avoid holding sockets indefinitely.

Worked example: how retries amplify an outage

A service normally receives 10,000 requests per second and can handle 12,000. A dependency slows down and 50% of requests start timing out.

• Naive clients, 3 immediate retries: each failing request can be sent up to 4 times. Offered load rises toward 10,000 x (1 + 0.5 + 0.25 + 0.125), about 18,750 requests per second, well over capacity, which pushes the failure rate higher still, which adds more retries. This feedback loop can keep the system overloaded even after the original trigger is gone, a metastable failure.
• Retry budget of 10%: the client allows retries only while retries are under 10% of its successful request rate (Finagle and gRPC-style budgets also allow a small minimum per second). Offered load is capped around 11,000 requests per second, within capacity, so the system can recover.

Backoff: with base 100 ms, cap 5 s and full jitter, attempt 1 waits uniformly 0 to 200 ms, attempt 2 waits 0 to 400 ms, attempt 3 waits 0 to 800 ms. Without jitter, thousands of clients that failed at the same moment retry at the same moment, creating spikes; with full jitter their retries spread across the window.

Timeouts: measure the dependency's p99.9 latency (say 180 ms) and set the timeout a little above it (say 250 ms), not an arbitrary 30 s. Also keep the sum of timeout and retries within the caller's deadline: 250 ms times 3 attempts plus backoff must fit in the budget the caller gave you.

Which failures to retry

Making writes idempotent

• Idempotency keys: the client generates a unique key per logical operation and reuses it on retries; the server stores key, request hash and response atomically with the side effect.
• Natural idempotency: PUT of a full resource, "set status to shipped" instead of "advance status", or conditional writes with version numbers.
• Deduplication at consumers: store processed message IDs in the same transaction as the effect.

How real systems do it

AWS SDKs use exponential backoff with jitter and a client-side retry token bucket by default. gRPC supports retry policies with a retry throttling ratio configured per service. Google's SRE book recommends per-request retry limits and per-client retry budgets of around 10% to prevent retry amplification.