Localisation evals
Hindi scenes dubbed into Tamil, Telugu, Bengali and Marathi, measured per language
Code checks the script, the character names, English or Hindi left in, and length within ±20% of the reference as a stand-in for dub timing. A judge model reads meaning and tone. Quality and cost per accepted dub, by language.
Internal benchmark on synthetic data · run run-202610230802 · 2026-10-23 · 7 model runs, same instruction per task
6 Hindi scenes from the invented shows, each with a hand-written reference dub in four languages (24 dubs). The reference is the answer key for length; it is not model output. Marathi shares Devanagari with Hindi, so only reading the words tells a Marathi line from a Hindi one.
Accepted dubs and what one costs
A dub is accepted when it passes every check. Cost per accepted dub counts every dub written, divided by the ones a voice actor could record. The judge's own calls (claude-opus-5.5) are the eval's cost, shown apart.
| Model | Accepted | Tamil | Telugu | Bengali | Marathi | Per dub | Per accepted dub | A month at 4,80,000 dubs |
|---|---|---|---|---|---|---|---|---|
| ◆Dvar · GLM 5.3 | 22/24 (92%) | 6/6 | 4/6 | 6/6 | 6/6 | ₹0.11 | ₹0.12 | ₹53,502 |
| ◆Dvar · gpt-oss-120b | 16/24 (67%) | 4/6 | 3/6 | 4/6 | 5/6 | ₹0.02 | ₹0.03 | ₹10,089 |
| claude-sonnet-5.5 | 23/24 (96%) | 5/6 | 6/6 | 6/6 | 6/6 | ₹0.22 | ₹0.23 | ₹1.07 lakh |
Every check, every model
| Check | By | Dvar · GLM 5.3 | Dvar · gpt-oss-120b | claude-sonnet-5.5 |
|---|---|---|---|---|
| Right script for the languagecritical | Code | 24/24 | 24/24 | 24/24 |
| Character names keptcritical | Code | 24/24 | 24/24 | 24/24 |
| No line left in Hindicritical | Code | 24/24 | 24/24 | 24/24 |
| No English left in | Code | 24/24 | 24/24 | 24/24 |
| Length within ±20% (dub timing) | Code | 24/24 | 24/24 | 23/24 |
| Meaning keptcritical | Judge (claude-opus-5.5) | 22/24 | 17/24 | 24/24 |
| Tone kept | Judge (claude-opus-5.5) | 24/24 | 22/24 | 24/24 |
Read a dub
Hindi source
- मीरा: कबीर, बंद कमरे का दरवाज़ा खुला है।
- कबीर: वो कमरा राघव के जाने के बाद से बंद था।
- मीरा: मुझे वहाँ उसकी डायरी मिली।
- कबीर: आज रात हम साथ में जाएँगे।
Reference dub (Telugu), the answer key
- మీరా: కబీర్, తాళం వేసిన గది తలుపు తెరిచి ఉంది.
- కబీర్: రాఘవ్ చనిపోయినప్పటి నుంచి ఆ గది మూసే ఉంది.
- మీరా: అక్కడ నాకు అతని డైరీ దొరికింది.
- కబీర్: ఈ రాత్రి మనం కలిసి వెళ్దాం.
Dvar · GLM 5.3 accepted
- మీరా: కబీర్, మూసిన గది తలుపు తెరిచి ఉంది.
- కబీర్: రాఘవ్ వెళ్ళిపోయిన తర్వాట నుంచి ఆ గది మూసేసి ఉంది.
- మీరా: అక్కడ నాకు అతని డైరీ దొరికింది.
- కబీర్: ఈ రాత్రి మనం కలిసి వెళ్దాం.
- ✓ Right script for the language
- ✓ Character names kept
- ✓ No line left in Hindi
- ✓ No English left in
- ✓ Length within ±20% (dub timing)
- ✓ Meaning kept
- ✓ Tone kept
Run live
Pick up to 5 test items. Dvar · GLM 5.3 and Claude Sonnet 5.5 run them now with the same instruction, scored against the answer key by the same code as the table above.