High-Scale Architecture: 100M Daily Writes, 1B Daily Reads
Scaling a URL shortener to internet scale requires addressing three distinct bottlenecks: distributed ID generation without coordination, caching hot links to avoid database overload, and partitioning URL records horizontally.
Scale Architecture
Distributed ID Generation: Range-Based & Snowflake
- Range Allocation: A coordination service (e.g., Apache ZooKeeper) allocates distinct numeric ranges (e.g., Node 1 gets IDs 1–1,000,000; Node 2 gets 1,000,001–2,000,000). Nodes generate IDs entirely in local memory with zero inter-server network calls.
- Twitter Snowflake: Generates 64-bit IDs containing timestamp (41 bits), datacenter ID (5 bits), worker ID (5 bits), and local sequence number (12 bits). A 64-bit ID encodes to about 11 Base62 characters, so on its own it does not meet a 7-character code goal.
Database Sharding by Short Code Hash
With billions of mappings, storage must be sharded. Partitioning by a hash of the short key distributes read and write traffic evenly across database nodes. Avoid a plain hash(short_key) % N_shards, which remaps most keys when N changes; use consistent hashing or a fixed set of logical shards mapped to nodes.
Caching & The 80/20 Pareto Principle
Read traffic heavily follows the Pareto distribution: 20% of short links drive 80% of click traffic. Caching top links in Redis with an LRU (Least Recently Used) eviction policy allows 80%+ of reads to be served in under 2 milliseconds from memory.
Worked example: 100M writes and 1B reads per day
- Write rate: 100 million / 86,400 is about 1,160 per second on average, perhaps 3,000 at peak.
- Read rate: 1 billion / 86,400 is about 11,600 per second on average, perhaps 40,000 at peak.
- Storage over 5 years: 100 million x 365 x 5 = about 183 billion links. At about 500 B each that is about 90 TB before replication, so the data must be sharded. For example, 64 shards of about 1.4 TB each.
- Code space: 183 billion is well under 62^7 (about 3.5 trillion), so 7 characters are enough for many years.
- Cache: if the hottest 50 million links serve 80% of reads, at about 600 B per entry including overhead that is 30 GB in Redis. Cache-hit reads cost under a millisecond, and the database sees about 20% of 40,000, or 8,000 reads per second across 64 shards, which is trivial per shard.
ID generation at scale
| Scheme | Code length | Coordination per ID | Failure behaviour |
|---|---|---|---|
| Single database sequence | 7 | Every insert | Single bottleneck and single point of failure |
| Range allocation (a key service hands out blocks of 1 million) | 7 | Once per block | A crashed node wastes the rest of its block, which is harmless |
| Snowflake (41-bit time + 10-bit worker + 12-bit sequence) | About 11 | None | Depends on clocks; must refuse to issue IDs if the clock moves backwards |
| Random + conditional insert | 7 | A unique-index check on the target shard | Rare retries |
Read path with analytics off the critical path
Caching redirects at the CDN makes clicks cheap but hides them from the origin, so the CDN's logs become part of the analytics pipeline. Bitly-style products usually keep the TTL short, or count clicks at the edge. Deleting or disabling a link must purge it from the CDN and from Redis. Otherwise a link taken down for malware keeps redirecting until the cached entries expire.