The Architectural Divide: Message Queues vs Event Streams
Engineers frequently confuse message task queues (RabbitMQ, Amazon SQS, Celery) with distributed event streams (Apache Kafka, Redpanda, AWS Kinesis). While both buffer asynchronous data, they represent fundamentally different architectural models.
Comparison: Task Queue vs Partitioned Event Stream
Detailed Tradeoff Matrix
| Dimension | Task Queue (RabbitMQ / SQS) | Event Stream (Kafka / Redpanda) |
|---|---|---|
| Message Lifecycle | Usually tracks each job through available, leased/in-flight, acknowledged, and redelivery states; deletion or retention behavior is broker-specific. | Records are retained by time or size policy independently of one consumer group's progress. |
| Replayability | Often limited after acknowledgement, though dead-lettering, mirroring, or explicit retention can preserve history. | Possible while records remain retained; replay must still handle changed code, schemas, and duplicate effects. |
| Consumer Tracking | Broker commonly tracks availability, lease/visibility, acknowledgement, and redelivery per job. | A group commits progress per partition, plus application state needed to make effects and retries safe. |
| Ordering Guarantees | Depends on queue type, worker concurrency, leases, and retries. | Offsets order records within a partition; concurrent handlers and retries can still complete side effects out of order. |
| Throughput Profile | Measure with the required payload, durability, replication, partition count, acknowledgement policy, and consumer work; product labels do not determine a universal rate. | |
Worked example: the same workload on each
An image-processing workload: 2,000 jobs per second, each taking 50 ms to 30 s (heavily skewed).
- SQS or RabbitMQ: 500 workers each pull the next job. A 30-second job occupies one worker while the others continue. Scale by adding workers, and there is no partition limit.
- Kafka with 50 partitions: at most 50 consumers in a group. A 30-second job at offset 1,000 of partition 7 holds up every later message in partition 7 unless the consumer hands work to an internal thread pool, which then complicates offset commits.
Now an order-events workload: 5,000 OrderPlaced events per second, needed by billing, search, the email service and a data lake, with the ability to replay the last 7 days after a bug fix. On Kafka this is one topic with 7-day retention and four consumer groups, each with its own offsets, and a replay is an offset reset. On a queue it would take four queues fed by a fan-out, and no replay once messages are acknowledged.
Semantics side by side
| Aspect | Task queue (SQS, RabbitMQ classic) | Log stream (Kafka, Kinesis, Pulsar) |
|---|---|---|
| Unit of tracking | Per message: in flight, acked, redelivered | Per partition: a committed offset |
| After consumption | Deleted | Retained until time or size limits |
| Multiple consumers of one message | Needs a fan-out (SNS to SQS, exchanges) | Native: one consumer group per use |
| Parallelism | Any number of workers | At most the partition count per group |
| Per-message retry or delay | Native (visibility timeout, DLQ, delay) | Build it with retry topics |
| Ordering | Best effort (SQS FIFO per message group) | Strict per partition |
| Replay | No, unless archived | Yes, within retention |
Convergence
The line is blurring. RabbitMQ Streams add a retained log, Kafka share groups add queue-like per-record acknowledgement, and Pulsar supports both subscription models on one topic. In interviews, reason about the semantics you need (per-message ack versus retained log, fan-out, replay, ordering) rather than brand names.