SD System Design In Depth
0%
From-Scratch Build

Partitioned Commit Log & Consumer Group Broker

Append-only partitioned commit log broker with deterministic key-hash routing, monotonic per-partition offsets, consumer group cooperative partition rebalancing, and independent offset commits.

~35 min Intermediate Lessons: Design: a Kafka-style log, Task queue vs event stream, Event bus for product events, Backpressure, Queue lag
Interactive Build Lab 0/4 Steps Completed

Code & Verify in the Browser Workbench

Complete 4 progressive engineering steps in JavaScript, TypeScript, or C++20 (WebAssembly) against an in-browser Node.js v22, Bun v1.1 & Clang/WASI runtime (virtual filesystem, loopback TCP/HTTP, and automated unit tests) — zero local setup required.

1. Architectural Overview: Log-Based Brokers vs Traditional Queues

Unlike destructive AMQP queues that delete a message once acknowledged, a Log-Based Broker (Apache Kafka, Redpanda, AWS Kinesis) models a topic as a set of append-only, totally ordered partitions.

flowchart LR P["Producer"] -->|"hash(key) mod N"| Part0["Partition 0: [0, 1, 2, 3]"] P -->|"hash(key) mod N"| Part1["Partition 1: [0, 1, 2]"] Part0 --> CG1["Consumer Group A (Worker 1)"] Part1 --> CG2["Consumer Group A (Worker 2)"] Part0 --> Audit["Consumer Group B (Audit Replay @ Offset 0)"] Part1 --> Audit

Key Architectural Invariants

  1. Per-Key Ordering via Deterministic Hashing: Global ordering across a distributed cluster requires coordination bottlenecks, whereas per-key ordering only requires routing fnv1aHash(key) % numPartitions. All events for account-99 land in the exact same partition in strict sequential offset order.
  2. Exclusive Partition Ownership Within a Consumer Group: Within a single ConsumerGroup, each partition is assigned to at most one consumer instance (generationId++ on join/leave rebalance). This guarantees order-preserving parallel consumption without distributed locks.
  3. Non-Destructive Reads & Offset Replay: Because consumers track progress via committedOffsets[partition] = offset + 1, multiple independent services (e.g., billing-service and fraud-detector) consume the same stream concurrently, and any consumer can seek(partition, 0) to replay historical events.

2. Verification & Running the Lab

bash
node demo.js
node test.js