Search evals
Search that understands how listeners type, measured by query type
Exact titles, misspellings, Devanagari and described asks like “bhoot wali kahani”. Every search system is scored on the same graded queries, with what the model step costs and how long it takes.
Internal benchmark on synthetic data · run run-202610230802 · 2026-10-23 · 7 model runs, same instruction per task
An invented catalog of 46 titles. Relevance grades are derived from catalog attributes and what each query was written to ask, so they are an answer key for the demo, not listener data. BM25, spelling and script folding, and every metric run in your browser.
Try a query
“bhoot wali kahani gaon mein” · nDCG@10 for this query: 0.92
- 1Gaon Ki Chudail · गाँव की चुड़ैलHindi · audio · horror · village, witch, ghostbest
- 2Shamshaan Ki Raat · श्मशान की रातHindi · audio · horror · ghost, night, cremation groundgood
- 3Bhutachi WadiMarathi · audio · horror · village, ghost, haunted housebest
- 4Hostel Ke Bhoot · हॉस्टल के भूतHindi · audio · comedy · ghost, college, hostelnot relevant
- 5Contract Wali Shaadi · कॉन्ट्रैक्ट वाली शादीHindi · micro-drama · romance · contract marriage, ceo, billionairenot relevant
- 6Bhooter BariBengali · audio · horror · haunted house, ghostgood
- 7Deyyam IlluTelugu · audio · horror · haunted house, ghostgood
- 8Pei VeeduTamil · audio · horror · haunted house, ghost, family secretgood
- 9Whispers in Hollow LaneEnglish · audio · horror · haunted house, small town, ghostgood
- 10Haveli Ka Raaz · हवेली का राज़Hindi · micro-drama · horror · haunted house, family secret, ghostgood
What the listener meant (answer key)
horror · village · ghost
8 good answers in the catalog
The model's reading
{
"title_guess": null,
"genre": "horror",
"tags": [
"ghost",
"village",
"night"
],
"language": null,
"format": null
}Every system on the same 54 test queries
nDCG@10 on graded relevance (best 3, good 2, related 1). The gate ships a change only if no query type drops by more than 0.02 against the keyword baseline and p95 latency stays under 400 ms.
| System | nDCG@10 | Exact title | Misspelled title | Title in Devanagari | Described, not named | Empty pages | p95 model step | Per 1,000 | A month at 3,00,00,000 | Gate |
|---|---|---|---|---|---|---|---|---|---|---|
| Keyword | 0.60 | 0.84 | 0.62 | 0.00 | 0.74 | 30% | none | ₹0 | ₹0 | baseline |
| + spelling and script | 0.81 | 0.84 | 0.88 | 0.86 | 0.74 | 6% | none | ₹0 | ₹0 | Ship |
| + query reading: Dvar · GLM 5.3 | 0.98 | 0.97 | 0.98 | 0.97 | 0.98 | 0% | 2.2 s | ₹56.84 | ₹17.1 lakh | Block: too slow |
| + query reading: Dvar · gpt-oss-120b | 0.97 | 0.96 | 0.97 | 0.96 | 0.97 | 0% | 3.1 s | ₹14.47 | ₹4.34 lakh | Block: too slow |
| + query reading: claude-sonnet-5.5 | 0.97 | 0.97 | 0.97 | 0.97 | 0.97 | 0% | 6.1 s | ₹215 | ₹64.4 lakh | Block: too slow |
The same harness scores recommendation changes: replace the queries with listening sessions and the graded answers with what was played to the end. Relevance labels can be drafted by a judge model and checked by your team on a sample.
Run live
Pick up to 5 test items. Dvar · GLM 5.3 and Claude Sonnet 5.5 run them now with the same instruction, scored against the answer key by the same code as the table above.