Under the Hood: Amortizing System Call and Network Overhead
Sending 10,000 independent network calls each containing a 100-byte payload incurs astronomical overhead: 10,000 TCP packet headers (40 bytes each), 10,000 kernel context switches (sys_sendto), and, if the calls are sequential, 10,000 round-trip latency delays. Batching coalesces multiple operations into a single network packet or disk write, trading minor latency for orders of magnitude higher throughput.
Batching Trade-Off: Size vs. Latency Timer
Under the Hood: Production Batching Invariants
1. Kafka Producer Batching (`linger.ms` and `batch.size`)
Apache Kafka achieves millions of messages per second on commodity hardware through batching combined with sequential disk I/O, the OS page cache and zero-copy transfers. The producer library holds records in a memory buffer. It flushes when either batch.size (e.g., 16KB) is reached OR linger.ms (e.g., 10ms) expires. Setting linger.ms = 5 adds up to 5ms of delay but can raise throughput substantially under load, because requests carry fuller and better-compressed batches; the gain depends on record size and rate. Kafka 4.0 (KIP-1030) changed the default from 0 to 5 ms.
2. Partial Batch Failure Semantics
When sending a batch of 500 records to an upstream service, individual records may fail (e.g., a validation error on record #42). Production batch APIs should return granular per-item statuses rather than failing the entire batch, preventing infinite redelivery loops. Note that a single multi-row SQL INSERT ... VALUES (...), (...) is atomic: a duplicate key on one row fails the whole statement unless you use ON CONFLICT or per-row handling.
Worked example: inserting click events
The shortener must store 1,200 click events per second on average. Suppose each database round trip costs 1 ms of network and commit overhead and each row adds 0.02 ms of work.
- One insert per event, one connection: 1 ms + 0.02 ms per event, so at most about 980 events per second per connection. You need several connections just to keep up, and every one of them pays a commit (and an fsync on the primary).
- Batches of 100 rows per insert: 1 ms + 100 x 0.02 ms = 3 ms per batch, or about 33,000 events per second per connection, a 34x improvement. The cost: an event waits until 100 have accumulated or a timer (say 20 ms) fires, so worst-case added latency is about 20 ms.
- Batches of 10,000: 201 ms per batch, only slightly more throughput per connection than 1,000-row batches, but a single failure now affects 10,000 events and locks are held longer. Returns diminish once the fixed cost is a small fraction of the batch.
The general formula: throughput per worker = batch size / (fixed cost + batch size x per-item cost). Once the per-item term dominates, bigger batches stop helping.
Trade-off table
| Setting | Throughput | Latency | Failure blast radius |
|---|---|---|---|
| No batching | Lowest, bounded by per-call overhead | Lowest at low load; high at saturation | One item |
| Small batches with short linger (for example 5 ms) | High | Adds at most the linger time | Small |
| Large batches with long linger | Highest, with diminishing returns | Adds up to the linger time to every item | Large; retries are expensive |
| Smart batching (take whatever is queued, no fixed wait) | High under load | No added wait when idle | Varies with load |
Where batching appears
- Kafka producer:
batch.size(default 16 KB per partition) andlinger.ms(default 0 before Kafka 4.0, 5 ms from 4.0) control how long records wait to share a request; compression applies to the whole batch, so larger batches also compress better. - Database group commit: PostgreSQL and MySQL can flush the write-ahead log for several concurrent transactions with one fsync.
- Multi-row SQL:
INSERT ... VALUES (...), (...)orCOPY. Note a single multi-row INSERT is atomic: one duplicate key fails the whole statement unless you useON CONFLICT DO NOTHINGor equivalent. - API batching and DataLoader: GraphQL DataLoader collects individual lookups within one tick and issues a single batched query, removing N+1 patterns.
- GPU inference: model servers batch requests dynamically because a GPU processes 32 inputs nearly as fast as one.
Failure modes
- Losing a buffered batch on crash: in-memory batches are not durable until flushed; acknowledge callers only after the flush if they need durability.
- Partial failure handling: the API must report which items failed so the caller retries only those, idempotently.