| Model | Quality | p99 latency | Cost per request | vs today | Result |
|---|---|---|---|---|---|
| Claude Sonnet 5.5 Today's modelopenrouter/anthropic/claude-sonnet-5.5 | 93.3% replayed on 135 gold rows | 2,785 ms benchmark replies on the same messages | ₹0.1111 benchmark replies on the same messages | — | Today |
| openrouter/openai/gpt-6-luna | 77.0% replayed on 135 gold rows | 1,290 ms replayed on 135 gold rows | ₹0.0056 replayed on 135 gold rows | −95% | Fails 3 checks |
openai/gpt-6-luna is at worst −24.6 pt against today's model, beyond the −1 point margin; fails a critical check on 8 gold rows that today's model passes; falls below the minimum on Crime: 0.71 vs 0.99 (14 gold rows); Fantasy: 0.60 vs 0.99 (10 gold rows); Revenge: 0.71 vs 0.99 (14 gold rows); Romance: 0.74 vs 0.85 (42 gold rows); Self-help: 0.55 vs 0.99 (11 gold rows). Every other gated topic is at or above its minimum. Weakest topic: Self-help 0.55 | |||||
| openai_compatible/dvar:glm-5.3-flash | 80.0% replayed on 135 gold rows | 892 ms replayed on 135 gold rows | ₹0.0064 replayed on 135 gold rows | −94% | Fails 3 checks |
dvar:glm-5.3-flash is at worst −21.3 pt against today's model, beyond the −1 point margin; fails a critical check on 10 gold rows that today's model passes; falls below the minimum on Crime: 0.93 vs 0.99 (14 gold rows); Family: 0.80 vs 0.86 (15 gold rows); Fantasy: 0.70 vs 0.99 (10 gold rows); Horror: 0.79 vs 0.96 (29 gold rows); Revenge: 0.93 vs 0.99 (14 gold rows); Romance: 0.74 vs 0.85 (42 gold rows); Self-help: 0.82 vs 0.99 (11 gold rows). Weakest topic: Fantasy 0.70 | |||||
| openrouter/anthropic/claude-opus-5.5 | 96.3% replayed on 135 gold rows | 4,191 ms replayed on 135 gold rows | ₹0.2221 replayed on 135 gold rows | +100% | Fails 3 checks |
anthropic/claude-opus-5.5 is at worst −2.1 pt against today's model, beyond the −1 point margin; fails a critical check on 1 gold row that today's model passes; falls below the minimum on Crime: 0.86 vs 0.99 (14 gold rows); Self-help: 0.91 vs 0.99 (11 gold rows). Every other gated topic is at or above its minimum. Weakest topic: Crime 0.86 | |||||
| Fine-tuned open modelnext step | Measured after tuning on your data. No numbers yet. | Talk to us | |||
Fine-tuned on your masked gold set. Masked rows only. Needs your data-sharing opt-in. | |||||
No candidate passes every check. Open a model's row for its reasons.
Prices are per request, at each provider's list price.
Judge: 0 of 5 checks trusted.
| Topic | Rows | Today · Claude Sonnet 5.5 | Candidate · GPT-6 Luna | Candidate · dvar:glm-5.3-flash | Candidate · Claude Opus 5.5 | |||
|---|---|---|---|---|---|---|---|---|
| Romance | 42 | 0.86 | 0.74 | −0.12 | 0.74 | −0.12 | 1.00 | +0.14 |
| Horror | 29 | 0.97 | 0.97 | 0.00 | 0.79 | −0.18 | 0.97 | 0.00 |
| Family | 15 | 0.87 | 0.87 | 0.00 | 0.80 | −0.07 | 0.93 | +0.06 |
| Crime | 14 | 1.00 | 0.71 | −0.29 | 0.93 | −0.07 | 0.86 | −0.14 |
| Revenge | 14 | 1.00 | 0.71 | −0.29 | 0.93 | −0.07 | 1.00 | 0.00 |
| Self-help | 11 | 1.00 | 0.55 | −0.45 | 0.82 | −0.18 | 0.91 | −0.09 |
| Fantasy | 10 | 1.00 | 0.60 | −0.40 | 0.70 | −0.30 | 1.00 | 0.00 |
| All topics, weighted by traffic | 135 | 0.93 | 0.77 | −0.16 | 0.80 | −0.13 | 0.96 | +0.03 |