Under the Hood: Flow Control Across Asynchronous Boundaries
When an upstream producer generates work faster than a downstream consumer can process it, memory must buffer the delta. If the buffer is unbounded and the overload is sustained, the consumer will eventually crash with an Out-of-Memory (OOM) error, and latency grows unacceptably long before that. Backpressure is the communication feedback loop that signals the producer to slow down or pause.
Backpressure Feedback Loop: Pull-Based Credit Flow
Under the Hood: The Three Backpressure Defenses
- Push vs. Pull (Reactive Streams Specification): Push-based streams are inherently vulnerable to overwhelming consumers. Pull- or demand-based streams invert control: in Reactive Streams consumers signal
request(n)for a specific number of items, and HTTP/2 and gRPC flow control grant byte-level window credits. Kafka consumers are simply pull-based: they fetch from the broker at their own pace, with no credit protocol. - TCP Window Size (Kernel Flow Control): When an application thread stops reading from a TCP socket, the OS kernel receive buffer fills. The TCP stack automatically advertises a
Window Size = 0back to the sender, forcing the sender's kernel to pause transmitting packets. - Dropping Strategies: When queues reach hard capacity, systems must execute an explicit drop policy:
- Tail Drop: Drop newly arriving messages (protects existing in-flight transactions).
- Head Drop: Drop the oldest messages in the queue (ideal for time-sensitive data like live GPS telemetry).
Worked example: how fast a queue fills
A consumer processes 1,000 messages per second. A traffic spike sends 1,500 messages per second for 10 minutes. The backlog grows by 500 per second, so after 10 minutes there are 300,000 queued messages. At 2 KB each that is about 600 MB of memory for an in-memory queue.
- Once the spike ends and arrivals drop back to, say, 800 per second, the consumer drains only 200 per second of backlog, so clearing 300,000 messages takes 25 minutes. A message arriving at the end of the spike waits about 300,000 / 1,000 = 5 minutes before processing.
- If the spike instead lasts all day, the queue grows by about 43 million messages; no buffer is big enough. Only more consumers, less work per message, dropping, or slowing the producer can fix it.
Little's law makes the link explicit: items in the system = arrival rate x time in system. With a bound of 10,000 queued items and a drain rate of 1,000 per second, the worst queueing delay is about 10 seconds; the bound on the queue is effectively a bound on latency.
Strategies at a limit
| Strategy | How it works | Good for | Cost |
|---|---|---|---|
| Block the producer | Send or put call waits when the buffer is full | In-process pipelines, TCP streams | Producer threads stall; risk of deadlock across cycles |
| Credit or demand signalling | Consumer grants N credits; producer sends at most N | Reactive Streams, HTTP/2 and gRPC flow control, AMQP prefetch | Protocol support required |
| Pull-based consumption | Consumer fetches when ready | Kafka consumers, SQS polling | Backlog accumulates in the broker instead; monitor lag |
| Reject with 429 or 503 | Tell the client to retry later | Service edges, public APIs | Clients must back off properly |
| Drop newest (tail drop) | Refuse new items when full | Work where existing items matter more | New work is lost |
| Drop oldest (head drop) | Discard stale items to make room | Telemetry, live locations, metrics | Old items lost; fine when only the latest matters |
| Sample or aggregate | Keep 1 in N, or merge updates | Metrics, logs, UI updates | Precision |
Backpressure in real stacks
- TCP: the receiver advertises its free buffer as the receive window. If the application stops reading, the window drops to zero and the sender's kernel stops transmitting.
- HTTP/2 and gRPC: per-stream and per-connection flow-control windows are updated with WINDOW_UPDATE frames.
- Node.js streams:
write()returns false when the buffer exceeds its high-water mark; producers should wait for thedrainevent. - Kafka: consumers pull at their own pace; the log is the buffer, bounded by retention. Consumer lag is the metric to alert on.
Failure modes
- Retry storms that defeat backpressure: rejected clients that retry immediately keep load high.
- Head-of-line stalls: blocking on one slow consumer can halt a shared pipeline; isolate per destination.