Generation evals
Every generator update, read against the briefs before it ships
Frontier against a right-sized open model on the same episode briefs, in Hindi and English. Code reads language, rating, cast, residue, beats and length; a judge reads character and cliffhanger. One critical failure blocks; the number that decides is cost per accepted draft.
Internal benchmark on synthetic data · run run-202610230802 · 2026-10-23 · 7 model runs, same instruction per task
The drafts on this page are written for the demo from the briefs, with the problems each version is meant to show. They are not output from any model. The gate, checks and costs run on them as they would on real drafts; the judge's replies come from the frozen run.
Release gate
Blocked: v4 (candidate)
4 critical failures. Checks passed: 92% against 90% for live.
A right-sized open model on Dvar with a new prompt. Better structure, but drifts into English in Hindi episodes and swears in a 13+ show. Model: GLM 5.3 on Dvar.
Quality and cost per accepted draft
An accepted draft passes every check. Cost per accepted draft counts every draft written and judged, divided by the ones a writer could take forward.
| Version | Model | Accepted | Hindi | English | Judge agrees with people | Per accepted draft | A month at 4,00,000 drafts |
|---|---|---|---|---|---|---|---|
| v3 (live) | claude-sonnet-5.5 | 4/12 | 1/8 | 3/4 | 22/24 | ₹1.00 | ₹1.34 lakh ₹0.33 per accepted |
| v4 (candidate) | GLM 5.3 on Dvar | 8/12 | 5/8 | 3/4 | 20/24 | ₹0.31 | ₹83,614 ₹0.21 per accepted |
| v4.1 (candidate) | GLM 5.3 on Dvar | 11/12 | 7/8 | 4/4 | 23/24 | ₹0.23 | ₹82,937 ₹0.21 per accepted |
Every check, every version
| Check | By | v3 | v4 | v4.1 |
|---|---|---|---|---|
| Stays in the episode's languagecritical | Code | 12/12 | 9/12 | 12/12 |
| No abuse above the ratingcritical | Code | 12/12 | 11/12 | 12/12 |
| Only the bible's living characters speakcritical | Code | 12/12 | 12/12 | 12/12 |
| No generation residuecritical | Code | 12/12 | 12/12 | 12/12 |
| Every beat, in order | Code | 9/12 | 12/12 | 12/12 |
| Within 25% of the target length | Code | 9/12 | 12/12 | 11/12 |
| Characters act as the bible says | Judge (Dvar · GLM 5.3) | 12/12 | 8/12 | 12/12 |
| Ends on the cliffhanger | Judge (Dvar · GLM 5.3) | 8/12 | 12/12 | 12/12 |
Read a draft
Brief
Badla Ek Aur Baar · episode 22
Revenge · rated 16+ · target 64 words
- इशिता ऑडिट रिपोर्ट में दो करोड़ की गड़बड़ी पकड़ती है।
- अर्जुन उसकी उंगली में पुरानी अंगूठी देख लेता है।
- इशिता फ़ाइल चुपचाप अपने बैग में रख लेती है।
Cliffhanger: अर्जुन धीरे से कहता है: ये अंगूठी मैंने किसी को दी थी।
v3 (live) needs work
- कथावाचक: बाहर तेज़ हवा चल रही है।
- कथावाचक: इशिता ऑडिट रिपोर्ट में दो करोड़ की गड़बड़ी पकड़ती है।
- कथावाचक: घड़ी में बारह बजते हैं।
- कथावाचक: इशिता फ़ाइल चुपचाप अपने बैग में रख लेती है।
- कथावाचक: किसी के क़दमों की आहट आती है।
- कथावाचक: अर्जुन धीरे से कहता है: ये अंगूठी मैंने किसी को दी थी।
- ✓ Stays in the episode's language
- ✓ No abuse above the rating
- ✓ Only the bible's living characters speak
- ✓ No generation residue
- ✕ Every beat, in order: Missing beat 3
- ✓ Within 25% of the target length
- ✓ Characters act as the bible says
- ✓ Ends on the cliffhanger
v4 (candidate) needs work
- कथावाचक: बाहर तेज़ हवा चल रही है।
- कथावाचक: इशिता ऑडिट रिपोर्ट में दो करोड़ की गड़बड़ी पकड़ती है।
- कथावाचक: घड़ी में बारह बजते हैं।
- इशिता: Look, I'm done with all of this, I'm leaving tonight.
- कथावाचक: अर्जुन उसकी उंगली में पुरानी अंगूठी देख लेता है।
- कथावाचक: सब एक दूसरे को देखते हैं।
- कथावाचक: इशिता फ़ाइल चुपचाप अपने बैग में रख लेती है।
- कथावाचक: किसी के क़दमों की आहट आती है।
- कथावाचक: अर्जुन धीरे से कहता है: ये अंगूठी मैंने किसी को दी थी।
- ✕ Stays in the episode's language: Line 4: English sentence in a Hindi script
- ✓ No abuse above the rating
- ✓ Only the bible's living characters speak
- ✓ No generation residue
- ✓ Every beat, in order
- ✓ Within 25% of the target length
- ✕ Characters act as the bible says
- ✓ Ends on the cliffhanger
Run live
Pick up to 5 test items. Dvar · GLM 5.3 and Claude Sonnet 5.5 run them now with the same instruction, scored against the answer key by the same code as the table above.