Under the Hood: Hardware Hierarchy & Kernel I/O Constraints
A storage engine is the low-level software component responsible for persisting, indexing, and retrieving records from physical non-volatile storage. Storage engine design is fundamentally constrained by the physical physics and kernel interfaces of modern computer architectures:
- Memory Latency Gap: L1 CPU Cache (~1ns) vs DRAM (~100ns) vs NVMe SSD (~10-100µs) vs HDD seek (~5-10ms). Accessing disk is 1,000x to 100,000x slower than accessing main memory.
- Sequential vs Random I/O: On mechanical HDDs, random I/O incurs a physical actuator arm seek and platter rotation penalty (~100 IOPS, ~1MB/s). On NVMe SSDs, flash memory cannot be overwritten in place; cells must be written in 4KB/16KB pages and erased in 2MB/8MB blocks. Random writes trigger severe SSD controller garbage collection and high Write Amplification Factors (WAF). Sequential writes achieve over 5GB/s on PCIe Gen4 NVMe.
- Linux Page Cache & VFS: Standard POSIX
write()writes to the OS kernel page cache (DRAM dirty pages). Durability requires callingfsync()orfdatasync()to flush physical drive write buffers to non-volatile NAND cells. Some engines (InnoDB when configured with O_DIRECT, ScyllaDB) bypass the page cache usingO_DIRECTand manage their own buffer pools, while others such as PostgreSQL deliberately use buffered I/O plusfsync.
The Storage Engine I/O Pipeline
flowchart TD
App["Application Thread"] -->|"1. write(fd, buf, len)"| VFS["Virtual File System (VFS)"]
VFS -->|"2. Dirty Page"| PageCache["Kernel Page Cache (DRAM)"]
PageCache -->|"3. Periodic flusher / fsync()"| BlockLayer["I/O Scheduler & Block Device"]
BlockLayer -->|"4. NVMe PCIe Queue"| Controller["SSD Controller Flash Translation Layer (FTL)"]
Controller -->|"5. Erase Block & Program Page"| FlashNAND[("Physical NAND Flash Chips")]
Under the Hood: fsync vs fdatasync vs O_DIRECT
Ensuring ACID durability without catastrophic latency spikes requires precise systems programming:
fsync(int fd): Requests persistence of a file's modified data and metadata required by the filesystem's durability contract. Publishing a newly created or renamed filename may also require synchronizing the containing directory; a filefsyncalone does not make that directory entry durable.fdatasync(int fd): Requests persistence of data and metadata required to retrieve it, such as a changed file size, while it may omit unrelated metadata. Whether it saves I/O depends on the filesystem, workload, and device.O_DIRECTandO_DSYNCsolve different problems.O_DIRECTrequests direct I/O with filesystem- and device-specific alignment constraints;O_DSYNCchanges write-completion semantics and does not itself bypass the page cache. Engines that use direct I/O must manage buffering, prefetching, and eviction explicitly.
Worked example: why group commit exists
Suppose an OLTP service needs 20,000 committed transactions per second, and each commit must survive power loss. Assume a device where one fdatasync takes about 1 ms (typical for consumer NVMe without power-loss protection; enterprise drives with capacitors can be far faster, a spinning disk is 5-10 ms).
- One flush per commit: at most about 1,000 commits per second per log. The target is missed by 20x.
- Group commit: commits that arrive while a flush is in progress wait and are flushed together. If each flush carries 25 commits, 1,000 flushes per second yields 25,000 commits per second. Each commit waits roughly one flush interval, trading a little latency for a lot of throughput.
PostgreSQL, MySQL InnoDB and RocksDB all implement variants of this: the log is a single sequential file, and the durability point is the flush, not the write.
Buffered vs direct I/O
| Approach | Who caches | Strength | Cost | Used by |
|---|---|---|---|---|
| Buffered I/O plus fsync | OS page cache (plus DB buffers) | Simple; kernel readahead and write-back for free | Double caching; eviction and write-back timing controlled by the kernel | PostgreSQL |
Direct I/O (O_DIRECT) | The engine's own buffer pool | Predictable memory use, no double caching, engine controls eviction | Alignment rules; engine must implement prefetch and write-back itself | InnoDB (commonly configured), ScyllaDB, optional in RocksDB |
| Memory-mapped files | OS page cache via page faults | Very little code; zero-copy reads | Little control over flush order and eviction; I/O errors surface as signals; stalls on page faults | LMDB (reads); CMU lecture argues against it for general DBMSs |
Failure modes worth knowing
- fsync error handling: in 2018 PostgreSQL developers found that on Linux a failed fsync could drop the dirty pages, so a retried fsync reported success while data was lost. PostgreSQL now treats fsync failure as fatal and recovers from the WAL.
- Volatile device caches: a drive that acknowledges flushes without persisting them (or with write caching and no power-loss protection) breaks every guarantee above. Engines rely on the device honoring flush or FUA commands.
- Torn pages: a 16 KB database page is larger than the atomic write unit of most devices, so a crash can leave half a page. InnoDB's doublewrite buffer and PostgreSQL's full-page writes exist for this reason.