Does routing hold on tasks the models haven't seen?

Contaminated versus fresh

Same four policies, scored separately on the old benchmarks (HumanEval + GSM8K) and on a fresh set of LiveCodeBench problems dated 2025 or later. oracle is ideal routing, the cheapest model that did solve each task.

Three things fall out of that. The frontier model's fine on fresh problems (86.7%). The cheapest model drops from 90% to 60%, so the old benchmarks were flattering it, which is the contamination the commenters warned about. And routing earns far more on the fresh set: on the old tasks it buys ten points over the cheap model, on the fresh tasks it buys thirty-three, while escalating to the expensive model a fifth of the time instead of never. Routing pays off hardest exactly where the models are tested on work they can't have memorised.

The dates behind the tag

Calling a split "fresh" is a judgement, so here's the evidence. The 15 fresh tasks are LiveCodeBench problems published between 2025-01-04 and 2025-04-06, median 2025-02-22, filtered to anything dated 2025-01-01 or later. The model providers don't publish exact training cutoffs, so we hand you the release dates and let you judge how far past the line this run sits, rather than a bare clean-or-contaminated call. Those dates ship in results.json and re-derive under omnisbench verify like everything else. The measured version is the perturbation-gap probe, described step by step below.

How the perturbation-gap test works

1. Take a fresh problem and its exact test cases, the input-to-output pairs from LiveCodeBench.
2. A separate rewriting model, never the one being graded on that problem, rewrites the statement. It changes the prose, the variable names and the surface story, and holds the input format, the output format and every constraint word for word. The test cases never change, so the correct answer is identical.
3. Score every model twice, on the original and on the reworded copy. Grading runs the model's code against the same unchanged tests, so a pass means the same thing both times.
4. Drop any reworded problem no model can solve, a sign the rewrite broke it, and any problem no model solved at all.
5. The gap is the drop, original accuracy minus reworded accuracy. A model that leaned on the exact wording falls. A model that understood the problem holds.

The pool spans two providers, and a rewriter can favour or punish its own family's style, so we reword the whole pool twice, once with claude-sonnet-5 and once with gpt-5, and read each model's gap under each. That shows how much the rewriter itself moves the number, not the model.

One number survives changing the rewriter: gpt-5 gives up about 20 points whoever rewords it, a real sign its score on these recent problems leaned on the phrasing. Every other gap swings with the rewriter. The Anthropic models barely move under gpt-5 and fall hard under sonnet, gpt-5-nano the other way, so at 15 to 17 problems the rewriter and the surviving tasks move the number more than the model does. What holds for all four: every model loses at least 6 points when a fresh problem is reworded, so a post-cutoff date is necessary and not the whole story. Every reworded prompt and response is published, and omnisbench verify re-derives the gaps offline. Both runs and the method: docs/perturbation-probe.md.

Caveats, up front

Re-grade the run offline

The whole run, responses included, is in the repo. verify re-runs the graders against the published answers and re-derives this table with no API calls.

# re-grade the corrected fresh run, no keys needed
$ pip install -e .
$ python -m omnisbench.cli verify --run runs/fresh-16k-2026-08-20
VERIFY OK

See the run and the code on GitHub →
Back to the full leaderboard →