Under the Hood: Implicit Feedback & Learning to Rank (LTR)
While BM25 scores textual overlap, real-world user intent requires factoring in non-textual signals: item popularity, geographical distance, price competitiveness, seller ratings, and real-time user behavior. Learning to Rank (LTR) uses machine learning models to combine hundreds of disparate features into a single ranking score.
Two-Stage Search Architecture: Retrieval & Reranking
Under the Hood: Feature Engineering & Position Bias
1. Signal Categories
- Query-Document Features: BM25 score, phrase match proximity, vector cosine similarity.
- Document Static Features: Historical conversion rate, return rate, average review rating, page load speed.
- User Context Features: User location, browsing history, past brand affinity, device type.
2. The Position Bias Trap
Users click the #1 result significantly more often than the #5 result, regardless of true relevance. Training an ML model directly on raw click logs creates a feedback loop that permanently cements existing top results. High-performance search engines apply Inverse Propensity Scoring (IPS) to debias click data before model training.
Worked example: inverse propensity weighting
Suppose eye-tracking or randomisation experiments estimate the probability that a user examines each position:
| Position | Examination propensity | Clicks observed | Weighted clicks, clicks divided by propensity |
|---|---|---|---|
| 1 | 1.00 | 500 | 500 |
| 3 | 0.50 | 200 | 400 |
| 10 | 0.10 | 60 | 600 |
Raw clicks say the position-1 document is best by far. After correcting for how rarely position 10 is even seen, the position-10 document shows the strongest relevance signal. In practice propensities are clipped, for example to at least 0.05, because a click at a rarely seen position gets a huge weight and makes training noisy.
Where propensities come from
- Result randomisation: swap positions for a small slice of traffic and measure how CTR changes with position alone.
- Intervention harvesting: reuse natural ranking changes between model versions as free experiments.
- Click models: cascade and position-based models fit examination and attractiveness jointly from logs.
Signal trade-offs
| Signal | Volume | Noise | Notes |
|---|---|---|---|
| Click | High | High | Position and presentation biased; attractive thumbnails inflate it |
| Dwell time over about 30 s | Medium | Medium | Better satisfaction proxy; depends on content type |
| Add to cart or purchase | Low | Low | Strongest for e-commerce; sparse on the long tail |
| Human judgments | Very low | Low | Expensive; used for evaluation sets and calibration |
Production pattern: log every impression with its position, the ranker version and features at serving time; join with downstream events; build labels offline; train a LambdaMART or neural ranker; ship behind an interleaving or A/B test. Airbnb, Etsy and many e-commerce teams have described this loop publicly. Logging features at serving time avoids train and serve skew.
Failure modes
- Rich-get-richer loops: without exploration, new items never collect clicks. Reserve a small share of impressions for exploration or boost fresh items explicitly.
- Bot and fraud clicks: filter automated traffic before labels are built, or competitors can manipulate rankings.
- Presentation bias: larger images, badges and prices shown in bold change CTR independently of relevance; include presentation features in the model or control for them.