The Fundamental Abstraction: The Partitioned Append-Only Log
A distributed log (like Apache Kafka or Apache Pulsar) is an ordered, append-only sequence of immutable records. It serves as the durable event backbone of modern event-driven architectures, decoupling producers from consumers with multi-gigabyte/sec throughput.
Partitioning & Consumer Group Architecture
flowchart LR
P["Producers"] --> T["Topic: Orders"]
T --> P0["Partition 0: Append-Only Log"]
T --> P1["Partition 1: Append-Only Log"]
T --> P2["Partition 2: Append-Only Log"]
P0 --> C1["Consumer 1: Offset Tracker"]
P1 --> C2["Consumer 2: Offset Tracker"]
P2 --> C3["Consumer 3: Offset Tracker"]
Why Append-Only Logs Achieve Immense Throughput
- Sequential Disk I/O: Random I/O is slow on HDDs, and even on SSDs sequential writes are cheaper (less write amplification, better prefetching). Appending to the end of a log segment operates purely on sequential I/O, approaching raw physical disk bandwidth.
- Zero-Copy Network Transfers: Traditional servers read data into kernel memory, copy it to application user-space memory, and copy it back to the socket buffer. Kafka leverages the Linux
sendfile()system call (Zero-Copy), streaming data directly from OS PageCache into the network socket without CPU copying. This applies to plaintext listeners: with TLS enabled on the broker, data must be encrypted in user space and zero-copy is lost. - Batching: Producers and consumers batch messages together, amortizing network packet overhead and compression costs across thousands of records.
Offset-Based Consumer State
Traditional message queues (RabbitMQ classic and quorum queues, SQS) delete messages as soon as they are acknowledged by a consumer, incurring state overhead per message. In a distributed log, messages are retained for a configurable time window (e.g., 7 days). Consumers simply advance a lightweight 64-bit integer offset, allowing multiple independent consumer groups to read the same data at their own pace or replay historical events.
Worked example: sizing a cluster
- Ingest: 500,000 messages per second x 1 KB = about 500 MB/s. With replication factor 3, the cluster writes 1.5 GB/s to disk and moves about 1 GB/s of replication traffic between brokers.
- Retention: 7 days x 500 MB/s is about 302 TB of logical data, or about 907 TB raw. With 12 brokers that is about 76 TB each. Tiered storage (KIP-405, generally available in Kafka 3.9) can move older segments to object storage instead.
- Partitions: if one consumer instance can process about 5 MB/s, you need at least 100 partitions to keep up. Choose 128 or more to leave headroom, because adding partitions later changes which partition each key maps to.
- Consumer lag is the number to alert on: the latest offset minus the committed offset, per partition. Growing lag means consumers cannot keep up.
Log vs queue
| Property | Kafka / Kinesis / Pulsar (log) | RabbitMQ / SQS (queue) |
|---|---|---|
| Consumption | Offset per consumer group; messages stay after being read | Message removed after acknowledgement (classic queues) |
| Replay | Yes, within retention | No, except RabbitMQ Streams |
| Ordering | Per partition | Per queue (FIFO variants), weaker with competing consumers |
| Per-message routing and retries | Limited; you build retry topics yourself | Rich: routing keys, dead-letter queues, delays |
| Throughput | Very high with batching | High, lower per node |
| Best for | Event streams, CDC, analytics fan-out | Task queues and work distribution |
Delivery semantics
flowchart LR
P["Producer, idempotent, acks all"] --> L["Leader replica"]
L --> F1["Follower 1, in ISR"]
L --> F2["Follower 2, in ISR"]
L -->|"high watermark"| C["Consumer reads committed offsets"]
C --> O["Commit offset after processing"]
- At-least-once (the default pattern): process the message, then commit the offset. A crash between the two causes reprocessing, so make handlers idempotent.
- At-most-once: commit before processing. A crash loses the message.
- Exactly-once within Kafka: idempotent producers plus transactions that atomically write the output and commit the input offsets (Kafka Streams
exactly_once_v2). Side effects outside Kafka, such as sending email or charging a card, still need idempotency keys. - Metadata: Kafka 4.0 removed ZooKeeper entirely. Clusters now use the built-in KRaft Raft quorum for controller metadata.