Designing a View Counter (Like YouTube)
YouTube's view counter is one of the most famous examples of scaled counting. It must handle billions of views while preventing fraud and abuse.
The Architecture
When a user watches a video, the client sends a "heartbeat" or log event to an API gateway. This event is immediately pushed to a Kafka stream. The architecture splits into two paths:
- Real-time Path (Approximate): Stream processors (like Flink) consume the stream, aggregate counts in memory, and update a fast NoSQL database (like Redis or Cassandra) to provide immediate feedback to creators.
- Batch Path (Accurate & Audited): The raw logs are dumped into a data lake (HDFS/S3). MapReduce or Spark jobs run periodically to analyze the logs, remove bot traffic, deduplicate views based on strict business logic, and update the authoritative "true" view count in the main database.
Why do views sometimes freeze?
Famously, YouTube view counts used to stall at "301+". The commonly cited explanation is that the fast counter was shown until roughly 300 views, after which public updates paused while slower verification checked that the traffic was not from bots; YouTube never fully documented the internal mechanics. This is historical: in August 2015 YouTube retired the 301+ freeze, and it now counts views it is confident come from real people as they are recorded while continuing to review the rest.
Worked example: sizing
| Quantity | Assumption | Result |
|---|---|---|
| Views per day | 1 billion | about 11,600 per second average |
| Peak | 5 times average | about 58,000 per second |
| Heartbeat events | One every 10 s of watch time, 3 minute average watch | about 18 events per view, about 1 million events per second at peak |
| Raw log volume | 18 billion events x 200 bytes | about 3.6 TB per day |
| Dedup state | Keys of user and video for a 24 hour window | up to 1 billion keys; use keyed state in Flink with TTL, or approximate with Bloom filters |
Heartbeats dominate volume, so the first stream stage collapses them into one "qualified view" event per user and video once the watch threshold is reached.
Speed layer vs batch layer
| Aspect | Real-time path | Batch path |
|---|---|---|
| Latency | Seconds | Hours |
| Dedup | Short window, best effort | Full history, exact |
| Fraud filtering | Simple rules, rate limits | ML models, cross-signal analysis |
| Used for | Displayed count, creator dashboards | Monetisation, official counts, corrections |
Handling corrections
When the batch job decides 30,000 views were bots, the displayed count can drop. Products handle this by publishing the batch result as a new version of the count and letting the real-time path add only events newer than the batch watermark: displayed = batch count as of time T plus stream count since T. This is the classic Lambda architecture pattern; a Kappa-style alternative replays the full log through one improved streaming job.
Failure modes
- Hot video: a premiere sends a large share of all heartbeats for one video ID to one partition. Pre-aggregate at the edge or salt the key, then merge.
- Replay double counting: if the stream job restarts from an old checkpoint and its sink is not idempotent, the displayed count jumps. Write per-window results keyed by video and window, overwriting rather than incrementing.
- Bot bursts: view farms produce many short, identical sessions. Rate-limit per device and IP in the fast path and let the batch path apply heavier models before counts become official or monetised.
Interview framing
State the definition of a view first, then the two paths, then how the displayed number reconciles. Interviewers mainly probe dedup, hot keys and what happens when the paths disagree.