Object Storage vs. Block & File Systems
Object storage (like Amazon S3, Google Cloud Storage, or MinIO) is designed for large binary objects such as images, videos, and backups. It generally does not support in-place byte mutation, but many services do allow a key to be replaced or deleted; immutable names, retention controls, or versioning are separate policies. Unlike POSIX file systems, object APIs commonly expose key-oriented operations such as GET, PUT, DELETE, and HEAD.
Storage Tiering Architecture
Separation of Metadata and Payload Data
- Metadata Tier: Stores object key, bucket name, size, ETag/checksum, ownership, and mapping to physical disk blocks. Stored in high-speed distributed KV stores (like Spanner, CockroachDB, or Cassandra).
- Data Storage Tier: Large object payloads are split into fixed-size chunks (e.g., 64MB or 128MB) and written append-only to storage daemon drives.
Durability via Erasure Coding vs. 3x Replication
Standard 3x replication carries a 200% storage overhead (300GB disk used for 100GB data). Modern object stores implement Reed-Solomon Erasure Coding (e.g., 8 + 4 scheme):
- Data is split into 8 data chunks, and 4 parity chunks are generated (total 12 chunks).
- The object can survive the loss of any 4 of its 12 chunks simultaneously (so any 4 drives, nodes or racks only if the 12 chunks are placed in 12 independent failure domains) while imposing only a 50% storage overhead, saving millions of dollars in hardware costs.
Worked example: 3x replication vs RS(8,4)
| Scheme | Raw storage for 10 PB logical | Failures tolerated | Read to repair one lost 20 TB drive | Read latency |
|---|---|---|---|---|
| 3x replication | 30 PB | 2 copies lost | About 20 TB (copy from one replica) | Read from 1 node |
| Reed-Solomon 8+4 | 15 PB | Any 4 of 12 chunks lost | About 160 TB (8 chunks per lost chunk) | Read 8 chunks in parallel; slow nodes add tail latency |
| Hybrid (replicate recent objects, erasure-code them after N days) | Between the two | Both | Both | Fast for new objects |
Saving 15 PB of raw disk is worth millions per year, which is why large stores erasure-code warm and cold data. The extra repair traffic is why codes that reduce repair reads, such as Local Reconstruction Codes used in Azure Storage, exist.
Write path
The object only becomes visible when the metadata commit happens, after the data is durable. That ordering is how S3 provides strong read-after-write consistency, which it has done for all operations since December 2020. A crash before the commit leaves orphaned chunks, which a background garbage collector cleans up.
Numbers worth knowing
- S3 is designed for 99.999999999% (eleven nines) annual durability, and advertises at least 3,500 writes and 5,500 reads per second per prefix, scaling out by adding prefixes.
- Multipart upload: parts of 5 MiB to 5 GiB, at most 10,000 parts, objects up to 5 TiB. Parts upload in parallel and can be retried individually.
- Metadata scale: 100 billion objects at about 300 B of metadata each is about 30 TB. It must be sharded by bucket and key range, which is why listing a prefix is fast but "find all objects larger than X" is not supported natively.