The open, reproducible LLM routing benchmark.
OmnisBench measures how close a routing policy gets to the ideal quality-per-dollar frontier — and publishes every response so anyone can re-grade the numbers offline, with zero API calls. It's a benchmark, not a router: it grades routing policies, it doesn't route your traffic.
TL;DR — Routers make savings claims that are hard to check. OmnisBench is the open scoreboard that checks them: it plots every routing policy against an ideal quality-per-dollar frontier over a fixed task suite, publishes the underlying model responses, and lets anyone re-grade the entire leaderboard offline with omnisbench verify. First run: ideal routing hit 99.7% task success at roughly 90% lower cost than always using the frontier model.
Routing efficiency, not routing product
OmnisBench doesn't ship a router and doesn't touch your traffic. It runs a fixed suite of tasks against a fixed model pool under several routing policies — including oracle, the theoretical ceiling that always picks the cheapest model that actually solved a given task — and plots each policy's task success against its cost. The policies on the resulting Pareto frontier aren't beaten on both axes by anything else in the pool. That frontier is the yardstick any real router, including a future one from us, gets graded against.
99.7% task success at ~90% lower cost
In OmnisBench's first run (2026-08-19), ideal routing reached 99.7% task success at $0.62 per 1,000 requests — about 90% cheaper than always using the frontier model, Claude Opus 5, which scored 99.2% at $6.25/1k. The honest caveat: the cheapest model alone, GPT-5-nano, already scored 94.5% on this suite, so routing's real prize here is recovering the last few points of quality cheaply, not a too-good-to-be-true headline number.
| Policy | What it does | Task success | $ / 1k | Frontier |
|---|---|---|---|---|
| oracle | cheapest model that solved each task | 99.7% | $0.62 | frontier |
| always_cheap | always gpt-5-nano | 94.5% | $0.43 | frontier |
| always_big | always claude-opus-5 (frontier model) | 99.2% | $6.25 | dominated |
Anyone can check the number, not just trust it
Most LLM routing savings claims are marketing you cannot verify — a percentage on a landing page with no published methodology, no model, no dataset. OmnisBench takes the opposite approach: the code is Apache-2.0, the task suite is named and fixed, the model pool is disclosed, and every individual model response is published inside results.json. Running omnisbench verify re-runs the graders against those stored responses and re-derives the entire leaderboard — quality and cost — with zero API calls and no keys. Change one stored answer and it fails.
Run it yourself
The full run and the offline verification live in the public GitHub repository. No account, no API keys required to re-grade the published results.
# reproduce the whole run, or just audit the published one $ pip install -e . $ python scripts/prepare_datasets.py # HumanEval + GSM8K → data/ $ python -m omnisbench.cli run --config configs/v0.yaml --run runs/mine $ python -m omnisbench.cli verify --run runs/2026-08-19 # zero-API re-grade VERIFY OK
See how it stacks up
OmnisBench vs RouterBench →
How a live, one-command-reproducible benchmark compares to the respected academic RouterBench baseline.
Weave Router alternative →
Weave Router is a closed-routing product with unverified savings claims — here's how to check them.
Want the full first-run results, charts, and leaderboard? See the homepage →