Does routing hold on tasks the models haven't seen?
HumanEval and GSM8K are old, and the models have almost certainly read the answers. So we built a fresh split: the same routing policies, scored on problems published after the models could have trained on them. The short version is that contamination was real, routing earns more on fresh work, and we caught our own harness quietly strangling the models on the way. Every number here re-grades offline.
Contaminated versus fresh
Same four policies, scored separately on the old benchmarks (HumanEval + GSM8K) and on a fresh set of LiveCodeBench problems dated 2025 or later. oracle is ideal routing, the cheapest model that actually solved each task.
| Policy | Likely-contaminated (20) | Fresh, 2025+ (15) |
|---|---|---|
| cheapest model only (gpt-5-nano) | 90.0% | 60.0% |
| oracle (ideal routing) | 100.0% | 93.3% |
| always the frontier model (claude-opus-5) | 100.0% | 86.7% |
| how often ideal routing reached for the frontier | 0% | 20% |
Three things fall out of that. The frontier model's fine on fresh problems (86.7%). The cheapest model drops from 90% to 60%, so the old benchmarks were flattering it, which is the contamination the commenters warned about. And routing earns far more on the fresh set: on the old tasks it buys ten points over the cheap model, on the fresh tasks it buys thirty-three, while escalating to the expensive model a fifth of the time instead of never. Routing pays off hardest exactly where the models are tested on work they can't have memorised.
We caught the harness truncating the models
The first fresh run looked catastrophic, and it was wrong. Our output budget was 4,096 tokens, which is plenty for "write this short function" and nowhere near enough for a hard competitive-programming problem. The reasoning models thought at length, hit the ceiling mid-thought, and emitted no code at all. Every one of the frontier model's fresh failures was a blank page. We only spotted it because we publish every response and could read them. We made the budget configurable, gave the hard suite room, and re-ran. Truncation dropped from every failure to a single task. A closed benchmark makes that same mistake and ships the number, and nobody's the wiser.
Caveats, up front
Small samples
20 contaminated and 15 fresh tasks. It's a first signal, not a verdict, and the suite grows from here.
Difficulty is tangled with freshness
The fresh set is competitive-programming, which is both newer and harder than grade-school maths. Some of the drop is difficulty, not just contamination. We can't fully separate the two yet.
One residual truncation
Even at 16k tokens, one fresh problem still ran the model out of room. That one's genuine difficulty now, not a harness fault.
oracle is a ceiling
It's chosen with hindsight. A shippable router can't be this good; the gap to it is the real scorecard.
Re-grade the run offline
The whole run, responses included, is in the repo. verify re-runs the graders against the published answers and re-derives this table with no API calls.
# re-grade the corrected fresh run, no keys needed $ pip install -e . $ python -m omnisbench.cli verify --run runs/fresh-16k-2026-08-20 VERIFY OK
See the run and the code on GitHub →
Back to the full leaderboard →