Under the Hood: Silent Data Corruption & Bit Rot
Modern storage systems do not fail merely by crashing or throwing I/O errors. Physical hardware experiences Silent Data Corruption (Bit Rot): firmware bugs, misdirected or lost writes, media errors, cable noise and memory bit flips (for example from cosmic rays hitting DRAM) corrupt data without returning an operating system error code. High-durability systems implement End-to-End Checksums: CRCs to catch accidental corruption, and cryptographic hashes where tampering matters.
The End-to-End Verification Pipeline
flowchart TD
Client["Client SDK"] -->|"1. Compute CRC32C / MD5: '0x8A4B291F'"| Gateway["API Gateway"]
Gateway -->|"2. Verify Checksum during ingest"| ChunkServer["Storage Node"]
ChunkServer -->|"3. Write Block + Checksum Footer to Disk"| Disk[("NVMe / HDD Media")]
subgraph Scrubber ["Background Scrubbing Daemon"]
Cron["Weekly Scrubbing Worker"] -->|"4. Read Block & Recalculate CRC32C"| Verifier{"Checksum Matches Footer?"}
Verifier -->|"Yes"| OK["Data Valid"]
Verifier -->|"No (Bit Rot Detected!)"| Heal["Trigger Auto-Repair: Rebuild Chunk via Erasure Coding"]
end
Under the Hood: Checksum Algorithms & Hardware Acceleration
- CRC32C (Castagnoli): The gold standard for storage engines. Modern Intel and AMD x86 CPUs feature dedicated hardware instructions (
SSE 4.2 CRC32) capable of calculating CRC32C at several GB/s per core for a single stream, and 20+ GB/s per core with interleaved multi-stream implementations. - Block-Level Checksum Footers: Instead of computing one checksum per 100GB file, storage engines compute a 32-bit checksum for every 4KB block. This enables immediate corruption detection during random reads without scanning the entire file.
- Continuous Background Scrubbing: A low-priority background daemon continuously reads cold disks sequentially, re-verifies block checksums, and invokes erasure coding repair before a second disk failure causes catastrophic data loss.
Worked example: an upload verified end to end
- The client computes CRC32C of the 64 MB file while reading it from local disk and sends it in
x-amz-checksum-crc32c. - The service computes the checksum of the bytes it received and rejects the PUT if it does not match; the client retries.
- Each stored shard carries its own block-level checksums, verified when written and on every read.
- On download the service returns the stored checksum, and the client verifies the bytes it received.
- Background scrubbers re-read data at rest, and a bad block is rebuilt from replicas or parity before it is needed.
Overhead of block checksums
1 PB stored in 4 KB blocks is about 244 billion blocks. At 4 bytes of CRC32C each, that is about 1 TB of checksums, roughly 0.1% overhead. With hardware CRC instructions, a single core can checksum many gigabytes per second, so the CPU cost is small compared to the I/O.
Choosing an algorithm
| Algorithm | Size | Speed | Detects | Typical use |
|---|---|---|---|---|
| CRC32C | 32 bits | Very fast with SSE4.2 or ARMv8 CRC | All burst errors up to 32 bits, most random errors | RocksDB blocks, iSCSI, ext4 metadata, S3 option |
| xxHash64 or XXH3 | 64 bits | Very fast in software | Accidental corruption, lower collision rate | RocksDB option, many caches |
| MD5 | 128 bits | Moderate | Accidental corruption; broken against attackers | Legacy ETags and Content-MD5 |
| SHA-256 | 256 bits | Slower without hardware support | Accidental and malicious changes | Content addressing, ZFS dedup, S3 option |
Where silent corruption comes from
- Memory bit flips: the reason servers use ECC RAM; checksums computed after a flip will faithfully protect the wrong data, so compute early.
- Firmware bugs: misdirected writes (right data, wrong place) and phantom writes (acknowledged but never written).
- Defective CPUs: Google and Meta have both published on "silent data corruption" from faulty cores that compute wrong results.
- Media decay: latent sector errors on disks that are only discovered when read, which is why scrubbing matters.