Under the Hood: High-Throughput Inode Engines
In an object storage system, object metadata (bucket, key, size, ETag, owner, ACLs, creation timestamp, and chunk pointers) is tiny (~500 bytes per object), but queries against it are astronomical. The metadata database must support millions of transactions per second, atomic key updates, and instantaneous lexicographical delimiter listings (e.g. GET /bucket?prefix=logs/2026/&delimiter=/).
Metadata Schema and Index Architecture
Under the Hood: Lexicographical Prefix Traversal
Unlike traditional POSIX filesystems that represent directories as physical inode link trees (which require walking directory paths sequentially), object storage engines treat paths as flat string keys. By storing keys in a sorted B-tree or LSM-tree ordered by (bucket_id, key_name), listing the contents of a "virtual directory" reduces to an efficient range scan that starts at photos/2026/march/ and continues while keys still begin with that prefix.
Worked example: a delimiter listing
Index rows for bucket photos, in sorted order:
2026/march/a.jpg2026/march/b.jpg2026/march/trip/c.jpg2026/march/trip/d.jpg2026/may/e.jpg
Request GET /photos?prefix=2026/march/&delimiter=/:
- Seek to the first key at or after
2026/march/. - Return
a.jpgandb.jpgas objects. 2026/march/trip/c.jpgcontains another/after the prefix, so emit the common prefix2026/march/trip/once and skip ahead past every key starting with it.- Stop at
2026/may/e.jpg, which no longer matches the prefix.
S3 returns up to 1,000 entries per page and an opaque continuation token that encodes where the scan stopped, so the next call resumes the range scan exactly there instead of starting over.
Sizing
100 billion objects at about 500 bytes of metadata each is 50 TB, or 150 TB with 3 replicas. At 1 million metadata operations per second and 50,000 operations per second per shard, you need at least 20 shards before headroom, and realistically hundreds to allow for hot spots.
Partitioning the metadata
| Approach | Listing | Hotspots | Example |
|---|---|---|---|
| Range partition on (bucket, key) | Efficient range scans | Sequential names concentrate load; needs automatic splits | S3 index partitions by key prefix |
| Hash partition by directory id | Efficient within one directory | Very hot directories | Tectonic name layer on ZippyDB |
| Hash partition by object or file id | Needs a separate name index | Well spread | Tectonic file and block layers |
| Metadata stored in a general-purpose distributed DB | Depends on schema | Depends on schema | Colossus metadata in Bigtable |
Failure modes
- Commit ordering: write data first, then the metadata entry; otherwise readers see entries pointing at nothing.
- Orphaned data: data written but metadata commit failed; needs garbage collection by scanning for unreferenced chunks.
- Listing large prefixes: expensive scans; systems often offer inventory reports as an alternative.