Under the Hood: The Economics and Hardware of Data Lifecycles
In large-scale platforms, data often exhibits steep temporal access decay: in many workloads the large majority of reads (for example 90%) target data written in the last few days. Measure your own access pattern before choosing tier boundaries. Keeping multi-year historical logs or audit records on high-performance NVMe SSDs in primary transactional clusters costs hundreds of thousands of dollars per month in idle compute and storage provisioned IOPS.
Multi-Tier Storage Architecture
Hardware, Latency, and Cost Breakdown
| Storage Tier | Underlying Hardware | P99 Access Latency | Cost per TB / Month |
|---|---|---|---|
| Hot (Transactional) | Local NVMe SSD / Provisioned IOPS EBS (io2) | Sub-millisecond to 5ms | 00.00 - 80.00 |
| Warm (Infrequent) | Multi-tenant Cloud Object Store (S3 Standard / Infrequent) | 50ms - 150ms | 2.50 - $23.00 |
| Cold (Glacier) | Not disclosed by AWS | Flexible Retrieval: 1-5 min (Expedited), 3-5 h (Standard), 5-12 h (Bulk). Glacier Instant Retrieval: milliseconds | $3.60 |
| Deep Archive | Not disclosed by AWS | 12 - 48 hours retrieval | $0.99 (99% cost reduction) |
Tombstones and Lifecycle Compaction
When archiving data from distributed databases like Cassandra, deleted rows, expired TTLs and null writes leave tombstone markers. If cold archival jobs blindly export tables without compacting tombstones, query engines reading the archived Parquet files encounter performance degradation. Modern pipelines run compaction jobs before compressing data into columnar Parquet/ORC chunks for cold storage.