Under the Hood: 11 9s Durability & Reed-Solomon Erasure Coding
Achieving 99.999999999% (11 9s) annual durability means that if you store 10,000,000 objects, you can expect to lose a single object once every 10,000 years. Physical hard drives have an Annualized Failure Rate (AFR) of 1% to 4%. To survive constant disk, server, and datacenter failures at scale, storage architectures transition from simple replication to Reed-Solomon Erasure Coding.
Replication (3x) vs Erasure Coding (8+4)
Under the Hood: Erasure Coding Math
An RS(k, m) scheme splits an object into k data fragments and generates m parity fragments using matrix multiplication over Galois finite fields GF(2w):
- Storage Overhead: (k + m)/k. For RS(8, 4), the overhead is 12/8 = 1.5× (50% storage overhead), compared to 3.0× (200% overhead) for 3-way replication.
- Fault Tolerance: The system can reconstruct the entire original object from any k surviving fragments out of the total k + m fragments. In an RS(8, 4) cluster, the code can tolerate any four unavailable fragments. Real durability also depends on placing fragments in independent failure domains, detecting corruption, repairing before additional losses, and keeping metadata recoverable.
Worked example: how often do disks fail?
A cluster of 10,000 drives with a 2% annualized failure rate loses about 200 drives per year, one every 1.8 days. If each drive holds 16 TB and repair runs at 200 MB/s per drive rebuilt, rebuilding one drive's data serially takes about 22 hours. Real systems rebuild in parallel from many peers, so a drive's contents are re-protected in minutes to hours. The shorter that window, the less likely additional failures overlap it.
Replication and erasure codes compared
| Scheme | Storage overhead | Losses tolerated | Repair reads per lost fragment | Used by |
|---|---|---|---|---|
| 3-way replication | 3.0x | 2 | 1 copy | HDFS default, many databases |
| RS(6,3) | 1.5x | 3 | 6 fragments | HDFS erasure coding policy |
| RS(10,4) | 1.4x | 4 | 10 fragments | Facebook HDFS warehouse |
| RS(17,3) | about 1.18x | 3 | 17 fragments | Backblaze Vaults |
| LRC(12,2,2) | about 1.33x | Any 3, plus many 4-failure patterns | 6 fragments for a single loss | Azure Storage |
The repair column is the hidden cost: rebuilding one lost fragment in RS(10,4) reads ten fragments across the network. Local Reconstruction Codes add local parities so the common single-failure case reads fewer fragments.
What 11 nines really means
AWS phrases it as: store 10 million objects and you can expect, on average, to lose one object every 10,000 years. That design target assumes independent failure domains (S3 Standard spreads data across at least three Availability Zones), continuous integrity checking, and fast repair. It does not protect against a user deleting their own data; versioning, Object Lock and replication to another region address that.
Failure modes
- Correlated failures: a firmware bug or bad batch of drives hitting many disks at once.
- Repair storms: a rack loss triggers massive rebuild traffic that competes with user traffic; systems throttle and prioritize the most at-risk objects.
- Silent corruption: without checksums and scrubbing, a fragment can be bad long before you need it.