Under the Hood: The Low-Level Storage Stream Engine
Modern cloud object storage engines (such as Azure Storage and Apache BookKeeper / Pulsar) do not write directly to arbitrary filesystem files. Instead, they operate on top of an internal Append-Only Stream Layer. A stream is an unbounded, durable sequence of immutable bytes composed of ordered extents (Azure's term; this unit also calls them chunks).
The Chunk Append Pipeline
Handling Network Partitions: Lease Fencing & Sealed Chunks
The hardest challenge in distributed append-only storage is network split-brain: what happens if a primary storage node experiences a network blip and another node is promoted to take its place?
- Sealing Chunks: To prevent two writers from appending conflicting data to the same chunk, chunks are strictly sealed. When a master promotes a new stream writer, it contacts the storage node quorum and updates the chunk status to
SEALEDat the last universally agreed byte offset. - Fencing (BookKeeper style): In Apache BookKeeper, ledger recovery fences the ledger on the storage nodes (bookies), so appends from the old, zombie writer are rejected; Kafka achieves the same with leader epochs. Azure's stream layer instead relies on the stream manager sealing the extent and allocating a new one for the new writer.
Worked example: sealing after a failure
An extent is replicated on three extent nodes, and the primary acknowledges an append only after all three replicas have written it.
- Appends up to byte 800 have been acknowledged to clients.
- The primary sends the next append (bytes 800-1000) to the replicas and crashes. Replicas A and B have written it; replica C has not.
- The stream manager seals the extent. It asks reachable replicas for their lengths: A = 1000, B = 1000, C = 800. It seals at the smallest length among the replicas it can reach, 800.
- Is anything lost? No: bytes 800-1000 were never acknowledged, so the client will retry. A and B trim to 800 when the seal is applied.
- The stream manager creates a new extent on a fresh set of nodes and the writer continues there.
If replica C were unreachable at seal time, the manager would seal at 1000; when C returns, it syncs to the sealed length. Either way all replicas of a sealed extent become byte-identical.
Comparing append-only log layers
| System | Unit | Replication and acknowledgement | Split-brain protection |
|---|---|---|---|
| Azure Storage stream layer | Extent, sealed around 1 GB | Chain to 3 replicas; ack after all | Stream manager (Paxos) seals and reassigns |
| Apache BookKeeper | Ledger, split into entries | Ensemble with write quorum and ack quorum | Recovery sets a fence on bookies so the old writer's adds are rejected |
| Kafka | Partition log split into segments | Leader plus in-sync replicas; acks=all | Leader epochs; controller fences stale leaders |
| HDFS | Block in a file | Pipeline to 3 datanodes | NameNode lease recovery and generation stamps |
Why build object storage on streams
- Sequential appends are the cheapest disk pattern and make replication simple to reason about.
- Sealed extents are immutable, so they can be erasure coded, scrubbed and moved without coordination with writers.
- The partition layer above (Azure's tables, blobs and queues) keeps its own logs and checkpoints in streams, so every higher layer inherits the same durability model.