Designing a News Feed
Generating a timeline of content from people a user follows is a classic system design challenge. The core decision is how content moves from a creator to a consumer's feed.
Push Model (Fan-out on Write)
When a user posts, the system immediately "pushes" the post ID into the pre-computed feed caches of all their followers. When a follower opens the app, their feed is already built and stored in memory (e.g., a Redis List).
Pros: Reads are fast and bounded: a range read of one precomputed page plus hydration and ranking, rather than a merge across every followed account.
Cons: The "Justin Bieber Problem." If a celebrity with 100 million followers makes a post, the system must write to 100 million Redis lists. This massive fan-out takes time, consumes enormous CPU/Network, and delays feed updates.
Pull Model (Fan-out on Read)
When a user posts, it is simply saved to their personal outbox. No feed updates are made. When a follower opens the app, the system queries the outboxes of all people they follow, merges the posts, sorts them by timestamp, and returns the result.
Pros: Writes are instant. No celebrity fan-out issue.
Cons: Reads are slow and computationally expensive, especially for users who follow thousands of people.
The Hybrid Approach
Modern systems use a hybrid model. For normal users, the system uses Push (fan-out on write) to keep feeds fast. For celebrities or highly active accounts, the system uses Pull. When you load your feed, the system grabs your pre-computed Push feed, then dynamically Pulls recent posts from the celebrities you follow, and merges them together at read-time.
Worked example: the numbers behind the hybrid
Assume 300 million daily users, an average of 200 followers per author, and 50 million posts per day.
- Pure push: 50M posts x 200 followers = 10 billion timeline inserts per day, about 116,000 per second on average and several times that at peak. Storing 800 post IDs of 8 bytes for 300M users is about 1.9 TB of timeline data (before replication), which fits in a large Redis or Cassandra cluster.
- A celebrity post: one account with 100 million followers needs 100 million inserts. Even at 1 million inserts per second across the cluster that is 100 seconds of lag and a burst that starves everyone else's fan-out.
- Pure pull: a user following 500 accounts triggers 500 outbox reads (or scatter-gather over many shards) plus a merge on every feed open; at 175,000 feed opens per second that is tens of millions of reads per second.
- Hybrid: push for authors under a threshold (say 100,000 followers), pull for the rest. A typical user follows perhaps 10 to 20 such large accounts, so a read is one precomputed list plus 10 to 20 small, heavily cached outbox reads, merged by time or score.
Trade-off table
| Dimension | Push (fan-out on write) | Pull (fan-out on read) | Hybrid |
|---|---|---|---|
| Write cost per post | Proportional to follower count | One write | Proportional to followers for normal authors only |
| Read latency | Lowest | Highest; merge of many sources | Low; small merge |
| Storage | Timeline per user | Outboxes only | Both |
| Freshness | Lags behind large fan-outs | Always fresh | Fresh for celebrities, near-fresh for others |
| Wasted work | Fan-out to users who never open the app | None | Reduced by skipping inactive users |
How real systems do it
- Twitter (as described by Raffi Krikorian in 2012 to 2013) fanned out tweet IDs into Redis home timelines capped at about 800 entries, and very large accounts could take minutes to fan out, which is what motivated merging celebrity tweets at read time.
- Facebook News Feed is largely pull-and-rank: candidates are gathered from friends and pages at read time and scored by ranking models, because ranking needs fresh features anyway.
- Instagram used push-based feeds stored in Cassandra, then layered ranking over the candidate set.