The Two-Stage Recommendation Funnel
Large-scale recommendation platforms (YouTube, Netflix, TikTok, Instagram) cannot run complex deep learning models over their entire catalog of millions of items in real time. Instead, industry systems use a multi-stage funnel:
Candidate Generation (Fast Retrieval)
Reduces the candidate pool from millions down to hundreds using lightweight retrieval models:
- Vector Embeddings & Approximate Nearest Neighbors (ANN): User history and items are mapped into dense vector spaces (using two-tower neural networks). High-speed vector indexes (Faiss, HNSW, Milvus) retrieve the top K nearest items in low single-digit milliseconds.
- Collaborative Filtering & Graph Walk: Recommends items interacted with by users with similar historical engagement patterns.
Deep Ranking & Scoring
The candidate set passes to a heavier ranking model that evaluates hundreds of real-time features: user demographics, current device, time of day, context, and explicit interaction history. The model predicts probabilities of engagement: P(Click), P(WatchTime > 30s), and P(Like).
Exploration vs. Exploitation
A pure exploitation model traps users in filter bubbles. Production architectures introduce multi-armed bandit algorithms (e.g., Thompson Sampling or Upper Confidence Bound) to blend new or unrated content into the feed to gather engagement data.
Worked example: a 100 ms latency budget
| Stage | Input to output | Budget | Typical technique |
|---|---|---|---|
| Candidate retrieval | 10M to about 1,500 (several sources merged) | 15 ms | Two-tower embeddings with ANN (HNSW, ScaNN, Faiss IVF-PQ), co-visitation lists, followed creators |
| Filtering | 1,500 to 1,000 | 5 ms | Already seen, blocked, region or age restrictions, via Bloom filters or lookups |
| Feature fetch | 1,000 items x about 200 features | 20 ms | Online feature store (Redis, Cassandra) with batched reads |
| Ranking | 1,000 to top 100 | 40 ms | Deep model (for example DLRM or a multi-task network) scored in batches on GPU or CPU |
| Re-ranking | 100 to 20 shown | 5 ms | Diversity (for example maximal marginal relevance), freshness boost, ads insertion, exploration slots |
Sizing the retrieval index
- 10 million items x 128-dimensional float32 embeddings is 10M x 512 B, about 5 GB. It fits in RAM on one node, and replicas are added for query throughput.
- HNSW typically reaches 90 to 98% recall@100 in a few milliseconds. Its graph links add memory on top of the raw vectors, roughly 25 to 50% more with common settings. Product quantisation shrinks vectors 8 to 32 times with some recall loss, which lets billion-item indexes fit in memory.
- The item tower is recomputed offline, often daily, and the index rebuilt. The user tower runs online, using the user's latest actions, so recommendations react within the same session.
Architecture
Log impressions (what was shown and in which position), not just clicks. Without them you cannot train on negatives, and you cannot correct for position bias: items at the top get clicked more simply because they are at the top. YouTube's 2016 paper "Deep Neural Networks for YouTube Recommendations" describes this two-stage design. TikTok's Monolith paper (2022) describes real-time training with collisionless embedding tables.