Under the Hood: The Cost of Synchronous Blocking RPCs
In a thread-per-request server, a synchronous downstream call can hold a worker while the dependency computes. Event-loop and asynchronous runtimes can overlap that wait without one operating-system thread per request, but the request still occupies connection, memory, deadline, and downstream capacity. This model suffers from fatal scaling bottlenecks:
- Thread Memory & Context Switching: Each OS thread allocates 1MB to 8MB of virtual memory for its stack. Ten thousand concurrent blocked connections reserve gigabytes of virtual address space (resident memory is usually much smaller, because stack pages are committed only when touched) and add significant kernel context-switching overhead.
- Cascading Outages: If a downstream billing or notification service experiences a latency spike from 50ms to 5,000ms, upstream connection pools fill to capacity within seconds, exhausting web server worker threads and crashing unrelated endpoints.
- Temporal Coupling: Synchronous systems require the sender, the network, and the receiver to be online and available at the exact same millisecond.
Synchronous Blocking vs Asynchronous Delegation
Under the Hood: Kernel Event Multiplexing
High-throughput asynchronous systems decouple I/O from thread allocation using kernel multiplexing primitives: Linux epoll, BSD/macOS kqueue, and modern Linux io_uring. A single worker thread registers hundreds of thousands of non-blocking sockets with the kernel. The kernel notifies the application event loop only when bytes arrive on the socket, allowing a small number of event-loop threads to coordinate many connections. Capacity still depends on kernel socket buffers, application state, TLS, message rates, and per-connection limits; connection count alone is not a throughput result.
Worked example: sign-up flow
A sign-up request writes the user row (5 ms), then synchronously sends a welcome email through a provider (p50 300 ms, p99 4 s), resizes the avatar (800 ms) and notifies the CRM (200 ms, sometimes down). The user waits about 1.3 seconds normally and more than 5 seconds at p99, and a CRM outage breaks sign-up entirely.
Delegated version: in one transaction, insert the user plus three outbox rows (send_welcome, resize_avatar, sync_crm) and return 201 in about 10 ms. A relay publishes the outbox rows to a queue, and workers process each job with retries and backoff. A CRM outage now just grows one queue. Sign-up keeps working, and the jobs drain when the CRM recovers.
Sync or async?
| Question | Keep synchronous | Delegate |
|---|---|---|
| Does the user need the result to continue? | Yes: payment authorisation, login | No: emails, thumbnails, analytics |
| Latency of the work | Milliseconds | Seconds to hours |
| Downstream reliability | Highly available | Flaky or rate-limited third party |
| Traffic shape | Smooth | Bursty, where the queue absorbs peaks |
| Failure handling | The user can retry | The system retries and dead-letters |
What you take on when you go async
- Idempotency: every job carries a stable ID, and workers record processed IDs, or make the effect naturally idempotent (upsert, set state).
- Visibility: job status (queued, running, succeeded, failed), oldest-message age, and DLQ size.
- Ordering: generally not guaranteed across workers. If "update then delete" order matters, partition by entity key.
- Poison messages: cap retries (for example, 5 attempts with exponential backoff), then send to the DLQ with the error attached.
How real systems do it
- Shopify runs huge volumes of background jobs (webhooks, emails, imports) on Redis-backed job queues with per-shop throttling.
- GitHub moved many web-request side effects into background jobs (Resque, and later Aqueduct), so request latency does not depend on third parties.
- AWS SQS uses visibility timeouts: a received message is hidden for N seconds and reappears if it is not deleted, which gives at-least-once delivery with automatic redelivery.