The Fanout Dilemma: Social Feeds & Notifications
When a user publishes content (a tweet, an Instagram photo, or a LinkedIn post), how is that content distributed to all of their followers? Social network engineering relies on two foundational strategies: Fanout-on-Write (Push) and Fanout-on-Read (Pull).
Architectural Comparison: Push vs Pull
flowchart TD
subgraph Push ["1. Fanout-on-Write (Push Model: Default)"]
Author1["Alice Posts Tweet"] --> WriteWorker["Background Fanout Worker"]
WriteWorker -->|"Push Tweet ID to Inboxes"| Box1["Bob's Redis Feed"]
WriteWorker -->|"Push Tweet ID"| Box2["Charlie's Redis Feed"]
WriteWorker -->|"Push Tweet ID"| Box3["Dave's Redis Feed"]
Note1["Feed Read: one fast range read from Redis, e.g. LRANGE on a list or ZRANGE at O(log N + M) on a sorted set"]
end
subgraph Pull ["2. Fanout-on-Read (Pull Model: Celebrities)"]
Celebrity["Taylor Swift (100M Followers) Posts"] --> PostDB[("Taylor's Outbox Table")]
Follower["Follower Opens App"] --> ReadAgg["Feed Aggregation Service"]
ReadAgg -->|"Fetch Posts"| PostDB
Note2["Zero write amplification. Read aggregates followed accounts on the fly."]
end
Under the Hood: The Hybrid Fanout Architecture
Neither pure push nor pure pull works at scale:
- The Push Catastrophe (Write Amplification): If an account with 100,000,000 followers tweets, fanout-on-write requires generating and writing 100,000,000 Redis entries. At 50,000 writes/sec, the fanout pipeline takes 33 minutes to complete.
- The Pull Catastrophe (Read Latency): If every user load requires fetching and merging posts from 1,000 followees, p99 feed load latency skyrockets to seconds.
- The Twitter / Instagram Solution (Hybrid Model): Standard users (below some follower threshold; no exact public figure exists) use Fanout-on-Write. Celebrities above the threshold do not fan out on write. When a follower opens their feed, the system reads their pre-materialized Redis timeline and merges the recent posts of any followed celebrities on the fly.
Worked example: capacity maths
Assume 200 million daily active users, 500 million posts per day (about 5,800 per second on average, 3 times that at peak), an average of 200 followers, and 20 billion feed reads per day (about 230,000 per second).
- Pure push: 5,800 x 200 = 1.16 million inbox inserts per second on average, about 3.5 million at peak. Feasible on a sharded Redis or Cassandra tier. Reads are a single range query.
- A celebrity with 100 million followers posting once needs 100 million inserts. At a fanout budget of 1 million per second that is 100 seconds of the whole pipeline for one post, and some celebrities post many times a day.
- Pure pull: each feed read merges recent posts from about 200 followees, so 230,000 x 200 = 46 million post-list reads per second. Too expensive.
- Hybrid: push for authors below a threshold (commonly discussed in the range of about 10,000 to 1 million followers), and pull for the rest. A feed read fetches the precomputed inbox and merges the recent posts of the few followed celebrities, cached hot, so about 5 to 10 extra reads per feed load.
Strategy comparison
| Strategy | Write cost | Read cost | Freshness | Storage | Best for |
|---|---|---|---|---|---|
| Fanout on write (push) | O(followers) per post | One range read | Seconds of fanout delay | An inbox per user | Typical accounts, read-heavy feeds |
| Fanout on read (pull) | O(1) | O(followees) merge | Instant | Minimal | Celebrities, rarely read feeds |
| Hybrid | Bounded | Inbox plus a few merges | Mixed | Moderate | Large social networks |
| Message fanout (SNS to SQS, Kafka consumer groups) | One publish | One queue per subscriber | Near real time | Per-subscriber copies or offsets | Service-to-service events |
How real systems do it
- Twitter (per Raffi Krikorian's "Timelines at Scale" talk) kept home timelines as lists of tweet IDs in Redis, capped at about 800 entries, fanned out on write. Very high-follower accounts were merged at read time.
- Instagram and Facebook rank feeds at read time from candidate sets. Fanout builds candidates and ranking picks the order.
- Inbox caps (for example the last 800 IDs) bound storage. Older history falls back to pull.
Failure modes
- Fanout backlog during viral events. Prioritise active followers first.
- Deletes and blocks must be filtered at read time, because inboxes may still hold the IDs.