Ideal routing hits 99.7% quality at a tenth of the cost.
OmnisBench measures how close an LLM routing policy gets to the ideal quality-per-dollar frontier — and publishes every response so anyone can re-grade the numbers offline, with zero API calls. This is the first run.
Every policy, plotted against the frontier
Up and to the left is better: more task success, less money. The two policies on the dashed Pareto frontier aren't beaten on both axes by anything else. Ideal routing (oracle) sits top-left; brute-forcing every request through the frontier model buys no extra quality for ~10× the spend.
Leaderboard
Four policies over the same 364 tasks and the same pinned prices. oracle is the theoretical ceiling — the cheapest model that actually solved each item, chosen with hindsight. It's the target a real router aims at, not a shippable router.
| Policy | What it does | Task success | $ / 1k | vs frontier model | Frontier |
|---|
The honest read: on this suite the cheapest model alone (gpt-5-nano) already scores 94.5%. Routing's real prize here is recovering the last ~5 points of quality while staying ~10× cheaper than the frontier — not a magic headline number.
Where ideal routing actually sends the work
For each task, oracle picks the cheapest model that solves it. 342 of 364 tasks are solved most cheaply by the tiniest model — routing only escalates for the hard tail. That distribution is the opportunity a live router chases.
Task success by dataset
HumanEval (164 code tasks, unit-tested) · GSM8K (200 grade-school math, exact-match).
| Policy | HumanEval | GSM8K |
|---|
Don't take our word — re-grade it
Most routing savings claims are unverifiable marketing. OmnisBench publishes the exact model response for every task inside results.json. omnisbench verify re-runs the graders against those responses and re-derives this entire leaderboard — quality and cost — with no API calls and no keys. Change one stored answer and it fails.
# reproduce the whole run, or just audit the published one $ pip install -e . $ python scripts/prepare_datasets.py # HumanEval + GSM8K → data/ $ python -m omnisbench.cli run --config configs/v0.yaml --run runs/mine $ python -m omnisbench.cli report --run runs/mine $ python -m omnisbench.cli verify --run runs/2026-08-19 # zero-API re-grade VERIFY OK
Frequently asked
Straight answers to the questions people actually search for about LLM routing and OmnisBench.
What is an LLM router?
How much can LLM routing actually save?
Is OmnisBench an open-source alternative to Weave Router or RouterBench?
How do I verify the results myself?
Which models and datasets does it cover?
Caveats, stated up front
oracle is a ceiling, not a product
It's chosen with hindsight (cheapest model that did solve each task). No live router can be this good; the gap between a real router and this line is the real scorecard.
The cheap model is already strong
On this task mix gpt-5-nano alone hits 94.5%. The suite will grow to harder, more agentic tasks where the quality gap — and routing's value — widens.
Two datasets, four models
v0 covers HumanEval + GSM8K across opus-5 / gpt-5 / haiku-4.5 / gpt-5-nano. MBPP, LiveCodeBench, MMLU-Pro and more models are config-only additions.
Pinned, dated prices
Costs come from a committed pricing snapshot (2026-08-18), not live lookups — so the numbers are reproducible even as list prices move.