A citable academic baseline, and a live one you can re-run today.
RouterBench (Martian, arXiv:2403.12031) is a respected, peer-reviewed academic benchmark for LLM routing, a frozen dataset and paper. OmnisBench takes the same core idea and ships it as a live, one-command-reproducible tool: current models, pinned current prices, and results anyone can independently re-grade offline. Apache-2.0.
How they compare
| Dimension | RouterBench | OmnisBench |
|---|---|---|
| Type | Academic paper + frozen dataset | Live, open-source CLI tool |
| Models covered | Fixed set, pinned at publication (2024) | Config-driven pool: currently Claude Opus 5, GPT-5, Claude Haiku 4.5, GPT-5-nano |
| Freshness | Frozen at publication date | Pinned pricing snapshot, re-runnable against current models anytime |
| Reproducible today | Dataset available; not designed to re-run against today's models | Yes, via omnisbench run against any configured pool |
| Independently verifiable | Peer-reviewed methodology, published paper | Yes, omnisbench verify re-grades published responses offline, zero API calls |
| License | Academic dataset release | Apache-2.0 |
| Cost / pricing basis | Prices as of 2024 publication | Pinned, dated pricing snapshot (2026-08-18), reproducible as list prices move |
Models move faster than a paper can
RouterBench did valuable work pinning down a methodology for scoring routing policies against a quality-cost frontier. But its model list and prices are frozen at publication, so new frontier and budget models simply aren't in it yet. OmnisBench's model pool and pricing snapshot are config, not code, so a new run against today's models (or tomorrow's) is a config change and a re-run, not a new paper.
Two different kinds of trust
RouterBench earns trust through peer review and a citable, fixed methodology, which is the right bar for an academic result. OmnisBench earns trust differently: every model response for every task is published in results.json, and omnisbench verify re-runs the graders against those responses and re-derives the whole leaderboard offline, with zero API calls. Change one stored answer and it fails. That's a complementary, tool-shaped kind of verifiability. It does not replace peer review. It lets you check a number yourself in minutes.
OmnisBench builds on ideas RouterBench helped establish
We say this plainly: RouterBench helped establish the idea of scoring LLM routing policies against a quality-per-dollar frontier, and that framing is core to how OmnisBench presents its own leaderboard. OmnisBench isn't claiming to out-rank RouterBench's rigor. It's a different artifact (a live tool, not a paper) built for a different job: getting a fresh, checkable number against whatever models you actually care about today.
Pick based on the job
RouterBench
You want a citable, peer-reviewed methodology and baseline for academic work, a paper to reference, or a fixed historical comparison point.
OmnisBench
You want to know, right now, how much routing efficiency is actually available across today's models and prices, and you want the number to be independently checkable, not merely asserted.
OmnisBench's fresh split
On tasks the models can't have memorised, ideal routing hit 93.3% task success, above the frontier model itself, at about 60% lower cost, and every figure re-grades offline.
See the full leaderboard on the homepage →
Reproduce it on GitHub →
Read the open-source LLM routing benchmark overview →