Under the Hood: Quantifying Search Quality
You cannot improve search relevance without rigorous mathematical evaluation metrics. In information retrieval, search quality is evaluated using both offline benchmark datasets (judged by human evaluators) and online user telemetry.
The Core IR Evaluation Metrics
| Metric | Formula & Focus | When to Use |
|---|---|---|
| Precision@K | (Relevant in top K) / K. Focuses on purity of the first page. | E-commerce product search where top 10 items must be accurate. |
| Recall@K | (Relevant in top K) / (Total relevant in corpus). Focuses on completeness. | Legal discovery, patent search, and medical research. |
| MAP (Mean Average Precision) | Average of precision scores at each relevant document rank across queries. | Binary relevance evaluations across diverse query sets. |
| NDCG@K | DCG_K / IDCG_K where DCG = Σ (2^relᵢ - 1) / log₂(i + 1). | The Gold Standard: Evaluates graded relevance (0-4 stars) with position discounting. |
Under the Hood: Interleaving A/B Testing
Rank 2: Model B #1
Rank 3: Model A #2
Rank 4: Model B #2"] Combined --> ClickTelemetry["Detect which algorithm's items won the user's click!"] end
Interleaving blends the top results of Model A and Model B into a single merged result list presented to every user. Because every user sees both rankers, between-user variance largely cancels out, and published results (Netflix, Bing, Airbnb) report detecting ranker preferences with roughly 10x to over 100x less traffic, depending on the setting.
Worked example: scoring one ranking five ways
A query returns five results with graded relevance (0 to 3) of [3, 2, 3, 0, 1]. Assume these four relevant documents are all the relevant ones that exist.
| Metric | Computation | Value |
|---|---|---|
| Precision@5, relevant = grade 1 or more | 4 relevant out of 5 | 0.80 |
| Recall@5 | 4 of 4 relevant found | 1.00 |
| Reciprocal rank | First relevant at position 1 | 1.00 |
| Average precision | Mean of P@1, P@2, P@3, P@5 = 1, 1, 1, 0.8 | 0.95 |
| NDCG@5 | DCG 12.78 divided by ideal DCG 13.35 | 0.96 |
DCG uses gain 2 to the power rel minus 1, divided by log2 of position plus 1: 7/1 + 3/1.585 + 7/2 + 0 + 1/2.585 = 12.78. The ideal order [3, 3, 2, 1, 0] gives 7 + 4.42 + 1.5 + 0.43 = 13.35. The only loss is that a grade-3 document sits at position 3 instead of 2, so NDCG is high but not perfect.
Online vs offline evaluation
| Method | Needs | Strength | Weakness |
|---|---|---|---|
| Offline judged set | Human labels for a few thousand queries | Fast iteration, repeatable | Labels age; misses user behaviour |
| A/B test | Traffic split, business metric | Measures real outcomes | Needs large samples, takes days or weeks |
| Interleaving | Merged result lists | Detects ranking preference with far less traffic | Tells you which ranker users prefer, not by how much revenue |
Netflix and Microsoft Bing have both published results showing interleaving detects ranker differences with one to two orders of magnitude less traffic than a conventional A/B test, which is why teams use it as a fast pre-screen before a full A/B launch.
Choosing a metric
Use NDCG@10 when you have graded labels and care about the first page; MRR for navigational queries where one right answer exists, such as "gmail login"; Recall@1000 to evaluate the first retrieval stage of a two-stage system, since the re-ranker cannot fix documents that were never retrieved; and business metrics such as conversion or successful sessions to decide launches. Report confidence intervals: a 0.5 point NDCG gain on 200 queries is often noise.