ModelRig / what ModelRig does
Backups you declared. Swaps a bake-off earned.
Everyone is benchmaxxing, and everyone has the duct tape — retry loops, fallback chains, a second API key for when the first one melts. The only benchmark that can't be gamed is your own logged traffic, so that's the one we run: replay last week's real requests against any challenger configuration, and let the winner serve behind declarations — which models may serve each task, what happens per failure class, how much a run may spend. Model changes become pull requests instead of incidents.
Shadow replay
The bake-off runs on replayed requests. Zero production calls, zero risk to what your customers see.
The whole configuration, not just the model
Vary prompt, model, output rung, and reasoning scaffold together. The model name is one axis of many.
CI-gated verdicts
A challenger has to beat the incumbent's conformance confidence floor and its effective cost before it can win. No swap on vibes.
Candidate sets
The allowed models for a task are declared, and the invariant is tested, not asserted: nothing outside the set can serve the route.
Typed failures, budgeted retries, spend stops
Retries are budgeted per failure class — content-invalid, network, capacity-shed — with cross-provider fallback down the candidate set. A run declares its envelope: a runaway loop can't spend what you didn't approve, and when the envelope is empty, requests stop with a typed error.
Hot-swaps as pull requests
Route names are stable; the model behind one changes in a commit you review. Revert is one commit. Never silent.
The test that re-runs itself
A price or model change triggers a suggested re-test, so last quarter's decision doesn't quietly rot.
The odometer
Verified savings, measured only — and auditable against the same per-request telemetry the meter bills from. Side-by-side sample diffs, with your own human grade recorded next to the measured ones.
Compliance as predicates
Require grounded, trace_visible, schema_conformant, or a zero-retention route, and only candidates that satisfy the predicate can serve it. ZDR is a routing constraint, not a meeting.
The JSON ladder
When native strict JSON isn't available, the ladder is explicit and measured per model: native strict, then coached JSON, then a repair rung — and you see which rung served.
BYOK, zero margin on the hop
Bring your own provider keys and the fee doesn't ride the routing decision — no routing tax.
We don't benchmark models in general. We bake them off on your traffic.
The leaderboard ranks models in general; the bake-off ranks them on your traffic. See the general standing, then prove it on your own requests from get started. Validity isn't accuracy: perfect JSON, wrong numbers. We gate on both.
See a route bundle
A route is YAML in your repo: candidate models, an output schema, and retry/budget policy. The complete example — and the one-command bake-off that proves a swap before you merge it — is on get started.
How a call actually behaves
Before a swap ships, it is worth knowing how a single call runs: the candidate ladder tries your declared models in order, validates each output against the schema, repairs and retries within typed budgets, and falls through only on structure and infrastructure — never on a quality judgment. Routing & reliability documents the whole runtime model — and the boundary of what it will not do.