The Anatomy of a Hotspot: When Hashing Fails
Hash-based sharding guarantees uniform key distribution across shards—assuming keys are accessed uniformly. In real-world systems, access distributions follow Zipf's Law or Pareto distributions (80/20 rule). When a single entity (such as a celebrity with 100 million followers, a Black Friday flash sale item, or a major breaking news thread) receives millions of concurrent reads and writes, the single shard hosting that partition key collapses under CPU saturation, lock contention, or connection pool exhaustion, causing cascading cluster latency spikes.
Architectural Mitigation: Compound Key Salting & Scatter-Gather
Production Mitigation Patterns
1. Write Salting (Compound Partition Keys)
To distribute write throughput for a hot counter (e.g., likes on a viral video), append a random salt suffix to the partition key: partition_key = video_id + "_" + random(0, K-1) where K is the number of sub-partitions. Writes spread evenly across K physical shards. When reading the total count, the application queries all K sub-keys in parallel and sums them.
2. Two-Tier In-Memory Caching (L1 Local + L2 Distributed)
For read hotspots (e.g., Taylor Swift's profile or high-volume API configuration), routing all requests to Redis can overwhelm the single Redis node hosting that key. High-performance systems deploy an in-process L1 cache (e.g., Guava/Caffeine in JVM or sync.Map in Go) with a short TTL (1-5 seconds) directly inside the API instances. Most reads for the hot key (often 99%+) are served from this in-process memory cache, never touching the network or database.
3. Read Replica Fan-out & Read Quorums
In leaderless systems such as Cassandra, hot read keys can be served by increasing the replication factor (N=5) and allowing reads to hit any replica with read consistency level ONE. DynamoDB does not let users set the replication factor, so there the answer is DAX or an application cache.
Worked example: a viral like counter
A DynamoDB partition supports about 1,000 WCU/s (1,000 writes/s of items up to 1 KB) and 3,000 RCU/s. A viral post receives 20,000 likes/s.
- Single key: 20k writes/s against a 1k/s partition limit means 95% of writes are throttled. Adaptive capacity and split-for-heat help, but cannot split a single item.
- Salted with K = 32: keys
post#123#0topost#123#31receive about 625 writes/s each, which is under the limit. - Read cost: the total is 32 GetItems (or one BatchGetItem) per read. Cache the summed total for 1-2 s so reads do not multiply the problem.
- Alternative: buffer likes in a stream (Kinesis or Kafka) and apply one aggregated increment per post per second. That is 1 write/s instead of 20k.
Mitigations compared
| Technique | Fixes | Cost | Example |
|---|---|---|---|
| Key salting / write sharding | Write-hot keys | Reads fan out K ways | DynamoDB write sharding, counter shards |
| Local in-process cache (1-5 s TTL) | Read-hot keys | Staleness; memory per instance | Caffeine/Guava near-cache in front of Redis |
| Request coalescing (single-flight) | Thundering herd on a miss | Slight latency for waiters | Discord data services, Go singleflight |
| Write buffering / batching | High-frequency increments | Delay before visible | Stream aggregation, Redis INCR then periodic flush |
| Split or move partition | Hot ranges or tenants | Data movement | Bigtable/Spanner load-based splitting, Shopify pod moves |
| More replicas for reads | Read-hot keys | Replication cost; consistency level ONE | Cassandra RF, Redis read replicas |
Detecting hot keys
- DynamoDB CloudWatch Contributor Insights lists the most-accessed keys.
- Redis:
redis-cli --hotkeys(with an LFU eviction policy) or client-side sampling. - Cassandra:
nodetool toppartitions. - Alert on per-partition p99 latency or throttles, not on cluster averages, which hide the hotspot.
Failure modes of the fixes themselves
- Salting a key that needs strong reads: summing K sub-keys is not atomic, so a reader may see a total that never existed at any instant. That is fine for likes, wrong for inventory that must never oversell. Use a reservation or queue for those.
- Static K: choosing K = 32 for every key wastes 32x reads on cold keys. Promote keys to salted mode dynamically once they cross a rate threshold, and record the K in a small metadata item.
- Near-cache stampede: if every API instance expires its local copy at the same moment, the backend gets N simultaneous misses. Add TTL jitter and single-flight.