A citable academic baseline, and a live one you can re-run today.
RouterBench (Martian, arXiv:2403.12031) is a respected, peer-reviewed academic benchmark for LLM routing — a frozen dataset and paper. OmnisBench takes the same core idea and ships it as a live, one-command-reproducible tool: current models, pinned current prices, and results anyone can independently re-grade offline. Apache-2.0.
How they compare
| Dimension | RouterBench | OmnisBench |
|---|---|---|
| Type | Academic paper + frozen dataset | Live, open-source CLI tool |
| Models covered | Fixed set, pinned at publication (2024) | Config-driven pool — currently Claude Opus 5, GPT-5, Claude Haiku 4.5, GPT-5-nano |
| Freshness | Frozen at publication date | Pinned pricing snapshot, re-runnable against current models anytime |
| Reproducible today | Dataset available; not designed to re-run against today's models | Yes — omnisbench run against any configured pool |
| Independently verifiable | Peer-reviewed methodology, published paper | Yes — omnisbench verify re-grades published responses offline, zero API calls |
| License | Academic dataset release | Apache-2.0 |
| Cost / pricing basis | Prices as of 2024 publication | Pinned, dated pricing snapshot (2026-08-18), reproducible as list prices move |
The model landscape moves faster than a paper can
RouterBench did valuable work pinning down a methodology for scoring routing policies against a quality-cost frontier. But its model list and prices are frozen at publication — new frontier and budget models simply aren't in it. OmnisBench's model pool and pricing snapshot are config, not code, so a new run against today's models (or tomorrow's) is a config change and a re-run, not a new paper.
Two different kinds of trust
RouterBench earns trust through peer review and a citable, fixed methodology — the right bar for an academic result. OmnisBench earns trust differently: every model response for every task is published in results.json, and omnisbench verify re-runs the graders against those responses and re-derives the whole leaderboard offline, with zero API calls. Change one stored answer and it fails. That's a complementary, tool-shaped kind of verifiability — not a replacement for peer review, but a way to check a number yourself in minutes.
OmnisBench builds on ideas RouterBench helped establish
We say this plainly: RouterBench helped establish the idea of scoring LLM routing policies against a quality-per-dollar frontier, and that framing is core to how OmnisBench presents its own leaderboard. OmnisBench isn't claiming to out-rank RouterBench's rigor — it's a different artifact (a live tool, not a paper) built for a different job: getting a fresh, checkable number against whatever models you actually care about today.
Pick based on the job
RouterBench
You want a citable, peer-reviewed methodology and baseline for academic work, a paper to reference, or a fixed historical comparison point.
OmnisBench
You want to know, right now, how much routing efficiency is actually available across today's models and prices — and you want the number to be independently checkable, not just asserted.
OmnisBench's first run
Ideal routing hit 99.7% task success at ~90% lower cost than always using the frontier model — and every figure is re-gradable offline.
See the full leaderboard on the homepage →
Reproduce it on GitHub →
Read the open-source LLM routing benchmark overview →