ModelRig

ModelRig is the open-source operating layer for AI workstreams: your model calls become routes — YAML in your repo — every run lands in an inspectable artifact store, and each route serves the cheapest candidate model that passes your output schema, with probed evidence behind the choice.

Point your coding agent at it.

The fastest migration is the one your agent performs — that path comes first. Prefer your own hands? The steps, spelled out. Either way, Step 0 is the same.

Step 0 — create your org (two minutes, human-only)

Hosted sync — the console, synced telemetry, grades, the artifact store — needs one human prerequisite before either path below. It stays human by design: your coding agent never creates accounts, signs in, or holds credentials.

Show me how (stay on this page) →
  1. Open the console signup — it opens in a new tab, so this page stays put.
  2. Enter your email and a name for your organization. That's the whole form.
  3. In the console, issue an API key (the rig_sk_… value shown once — copy it).
  4. Back here, export it beside your provider keys: MODELRIG_API_KEY=rig_sk_…
Create your org ↗

No account needed for the open lane: with only your provider keys set, routes, probes, and telemetry run entirely on your machine — the hosted console and synced telemetry are what the key unlocks. You can add it later; nothing below requires it.

step 1 — paste one prompt

Paste this into Claude Code, Codex, or Cursor from inside your project. Your agent reads the docs, finds your LLM call sites, and turns them into routes:

prompt — paste into your coding agent
Migrate this project's LLM calls to ModelRig (https://modelrig.dev).
Work the details from the docs yourself; STOP and present to your human at each ⏸ checkpoint.

OPERATING RULES (read first; they override anything you remember):
R1. Ground in live surfaces, never memory: which models are probed, which knobs a
    provider honors, what a claim means — ask https://modelrig.dev/llms.txt, the
    raw-lane support matrix (modelrig.dev/route-bundles.html), rawKnobSupport(),
    or the MCP oracle. What the routing engine does and does not do at run time
    (fall-through triggers, the repair rung, what is stored, what is NOT built) is
    https://modelrig.dev/routing-reliability.html — read its "Not yet implemented"
    section before proposing any behaviour that depends on a route-declared judge or
    raw-lane structure. Declared capability and probed behaviour differ — that gap
    is ModelRig's entire thesis.
R2. Key custody: never create accounts, sign in, or read/transmit/store provider
    key VALUES. Existence checks by NAME only; humans place keys.
R3. Behaviour-identical, or a NAMED change presented at a checkpoint. Never a
    silent change — not sampling, not caching, not grounding, not enforcement.
R4. A ModelRig knob that is absent is byte-identical to before; a declared knob a
    provider cannot honor FAILS CLOSED with a classified error. If you hit one,
    that is the product working — consult the matrix, don't work around it.
R5. If the human's answers conflict with each other or with the code you read,
    STOP and surface the conflict. If they ask for something ModelRig does not
    support, say so directly and propose the closest supported alternative —
    never improvise.
R6. The rig has a lifecycle: create it ONCE (module singleton for a server;
    per-invocation for a one-shot job) and await rig.close() before exit, or
    telemetry is lost and timers hang the process.
R7. Run the project's own tests before proposing any merge; show the diff.
R8. Every pipeline execution is a run. Wrap each execution in run.start /
    run.scope, name one step() per model-call family, and artifact.save the work
    products; tag subject and feature (inside a run context the SDK stamps run_id
    AND step for you — a value you pass yourself always wins). subject is your
    end-customer's opaque id — it powers the per-customer tenant statements on
    /tenants; a tag named client/tenant is only a cost dimension. Acceptance: the run
    appears on /runs with its steps and artifacts — a migration is not done until
    one real execution shows a run with steps there.

0. PERSISTENT GROUNDING (do this before anything else):
   a. Add to this repo's agent instructions (CLAUDE.md, AGENTS.md, .cursorrules —
      whichever exists; create CLAUDE.md if none):
        "Before writing ModelRig code, fetch https://modelrig.dev/llms.txt —
         probed facts change; do not rely on memorized models or parameters."
   b. If your tool supports MCP servers, register the ModelRig oracle and ASK it
      rather than recalling (query_registry, get_leaderboard, explain_pricing,
      get_call_notes):
        { "mcpServers": { "modelrig": { "command": "npx", "args": ["modelrig-oracle"] } } }
   c. Keys: if MODELRIG_API_KEY is not set and the human wants hosted
      telemetry/console sync, STOP and ask them to create an org and issue a
      rig_sk_ key at https://app.modelrig.ai (about two minutes, human-only).
      Without it, proceed in local-only mode; that is fully supported.

1. DISCOVERY-LITE (ask, don't infer, these three; the code answers the rest):
   - What is this project, and is ModelRig replacing existing LLM calls or
     starting fresh?
   - Which providers does it (or should it) use, and which provider keys exist?
   - Hosted telemetry (app.modelrig.ai) or local-only?
   Then read https://modelrig.dev/routing-reliability.html (how a call actually
   behaves and what is not built), https://modelrig.dev/migration-playbook.html
   (this prompt follows its T0-T2 autonomy ladder and checkpoints) and
   https://modelrig.dev/quickstart.html.

2. START AT THE BOTTOM OF THE LADDER, not at routes. T0: point an existing
   OpenAI-compatible client at the ModelRig raw lane by swapping its base URL —
   transport only, nothing about the request or model changes, telemetry lights
   up. T1: run `modelrig observe` to see runs without migrating anything. T2
   (routes, the steps below) is a per-call-site graduation you earn AFTER T0/T1
   prove the plumbing — not the opening move.

3. ⏸ CHECKPOINT — fit and lane. Inventory the LLM call sites (generation calls
   incl. JSON-extractor followups; embeddings and web-search tool calls are out
   of scope). If the codebase already has a router layer, runtime-assembled
   prompts, or env-resolved model choice, do NOT externalize it into routes —
   migrate ONE naive call site as a reference route and integrate the router at
   the rig.runRaw seam instead. Present this as a MIGRATION RECOMMENDATION and
   get explicit approval before writing any code:
     Use case: <one sentence>            Lane: <T0 / T1 / T2 routes / runRaw seam>
     Call sites: <count, by provider>    Keys present: <names only>
     Knobs to carry (each with why): <sampling / cache / grounding / tier / reasoning …>
     Named behaviour changes (if any): <e.g. provider-strict JSON -> own extractor>
     Open questions / assumptions: <what you inferred that the human should confirm>
     Ready to proceed?

4. ⏸ CHECKPOINT — caching, grounding AND SAMPLING inventory, BEFORE writing any
   route or seam call: search for provider caching (cachedContent, cache_control,
   prompt_cache_key, cached-token usage fields), provider-native search/grounding,
   and explicit sampling params (temperature, top_p, max_tokens/max_output_tokens).
   Caching must be carried through (the cache handle on rig.run / rig.runRaw —
   modelrig.dev/caching-lifecycle.html); grounding gaps must be REPORTED as named
   behaviour changes; sampling must be PRESERVED by declaring policy.sampling
   { temperature, top_p, max_output_tokens } on the route (or the same fields on
   runRaw) — a classifier at temperature 0.1 is a deliberate choice, so carry it
   over exactly (absent, adapters use their own defaults; Gemini runs at 1.0).
   Never migrate past any of the three silently.

5. Write routes (T2; install with the project's own package manager — pnpm/npm/
   yarn; Node >= 20): modelrig/routes/<task>.yaml with the prompt in a FILE
   (prompt.system: ./prompts/<task>.md, never inline), schema, candidates,
   policy. Replace call sites with rig.run("<route>", { input, tags }) — tags is
   REQUIRED and every tag key must be declared as a dimension in rig.yaml.
   result.output arrives parsed AND schema-validated — delete your manual
   JSON.parse / fence-stripping. Routes resolve from ./modelrig/routes relative
   to the process CWD: in a monorepo put modelrig/ at the app root and run from
   there, or set MODELRIG_ROUTES_DIR. A runRaw-only seam needs no routes at all —
   construct with routesDir: null. Keep prompt and schema semantics unchanged; if
   the original was JSON-mode + app-side validation, adding schema enforcement is
   an UPGRADE, not identity — present that choice at the prove-it checkpoint, or
   set policy.json: json_mode for closest identity. To load-check the bundle
   WITHOUT any provider key, run `modelrig validate` — it structurally loads
   rig.yaml + every route and reports per-route OK / config error.

6. ⏸ CHECKPOINT — candidates. Any model an adapter can reach may be a candidate —
   probes gate CLAIMS, not serving. Pin what production resolves to today as
   candidate #1, always. Probed models (llms.txt or the public registry) carry
   evidence; an UNPROBED candidate is flagged as such and costs exactly this: no
   json: native claim (the emulation ladder serves structure), no leaderboard/
   bake-off priors, and possibly conservative envelope pricing until the next
   sweep. Never silently substitute a probed sibling for the model production
   runs; offer a probe request or a probed sibling as candidate #2. Present the
   list for the human to ratify.

7. ⏸ CHECKPOINT — prove it: run the project's tests, show the diff and every
   changed file, and after first traffic compare the route's cache-hit rate and
   cost against the pre-migration baseline in the console (this metric needs a
   REAL run — a single probe cannot produce it; say so if it must wait). A zero
   hit rate on a route that cached before is a regression; say so.

8. ⏸ CHECKPOINT — see the run (R8). Open https://app.modelrig.ai/runs: your
   execution is listed, expands to its steps in order, and each step carries its
   inference and saved artifacts; /projects groups the same run by its tags. An
   empty /runs after a real execution means it was not wrapped in run.start /
   run.scope — fix that before calling the migration done. `modelrig status`
   prints "runs recorded: N (last: …)" as the local mirror of the same fact.

9. ATTACH A GRADE, THEN PROVE A CHEAPER MODEL. Capture is not the whole
   loop — a grade is what makes /coverage readiness and /insights findings
   meaningful. A grade is your judgment — from your code, a human, or an AI — a
   0–1 score with an optional comment. If the pipeline already has a quality
   signal (a validator, a downstream accept/reject, a human QA gate), emit it as
   a grade — score 1 on a clean pass, 0 on a rejection:
     rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "human" }); // one call
     rig.grade({ kind: "run", id: run_id }, { score: 0, kind: "human", comment: "…" });          // whole run
   The write is synchronous and local, mirrored by the exporter, and never blocks
   or throws into your path. A route with ten attached grades reaches Rung 1 on
   /coverage; a route with no schema to conform to (a prose output) gains proposal
   confidence only this way. Then prove a cheaper model on YOUR own traffic, on the
   lane that fits: a DECLARED route uses `modelrig bakeoff --route <route>
   --replay-last N`; a RAW/run-based pipeline (a rig.runRaw seam, no routes)
   promotes a real output to a golden (rig.artifacts.artifact.promoteToEval(handle,
   { task }), or the console's one-click "make golden") and then runs `modelrig
   bakeoff --from-eval-cases <task> --variants default,<challenger>`. A bake-off
   measures; it never switches what serves a route — that is still a human-approved
   PR.

10. CONTENT CUSTODY IS ON BY DEFAULT — and Optimization on is the recommended
   default. Set `capture: true` on your routes so ModelRig keeps that route's
   telemetry rows and the bounded review samples your bake-offs replay — row-scoped
   in your organization, visible and deletable in the console — repaid with community
   priors and evidence on your own routes at the same flat 2%. Your organization's
   content custody posture is managed by default, so a route that captures content
   lands its prompts, model outputs and evidence in your managed store, scrubbed for
   the PII/PHI you classify, yours to see and export in the console. Opting out is one
   setting, not a default you have to escape: from your account settings (the console or
   the `set_org_settings` MCP tool) set posture `off` to capture nothing, or posture
   `metadata` to keep hashes only, or require `zero_retention` on a route to keep
   nothing there.

the agent will

  • read the quickstart and route-bundle docs
  • write route YAML into your repo and swap each direct LLM call site to rig.run()
  • run your existing tests and report every file it changed — you review the diff

it won't

  • create accounts or sign in anywhere — Step 0 is yours, and takes about two minutes
  • touch your API keys — those stay in environment variables you set yourself
  • change prompt or schema semantics — call sites are swapped, not rewritten

Your routes are YAML in your repo. Leaving is a git rm. That's why you'll stay.

One flat 2%, your keys or ours. On your own keys, your first million requests each month are free. Nothing is chargeable until prepaid balances ship. The pricing approach.

BYOK is free during launch — no platform fee on usage routed on your own provider keys while launch terms are in effect. Launch terms end when we announce pricing. You get 60 days’ notice, and organizations created during launch keep launch terms for 6 months after the announcement.

How it all fits together

One map: routes live in your git, your app calls the task, and the probed registry is what routing decisions read. What the console holds is the lane you choose, per route — two lanes, and you pick per route:

lanewhat's keptwhere it liveswhat you get back
Pure router(capture off)Nothing retained. Real, supported, selectable, and the honest answer for a ZDR-strict or data-residency-bound workload.
Optimization on(recommended)Per-inference telemetry rows, and bounded review samples for the bake-offs you create.Your organization, row-scoped by the database, not by our application code.Community priors on your very first bake-off and RigIndex-grade evidence on your own routes. Same flat 2% either way — the corpus earns its place by being useful, not by being a discount.

One flat 2%, your keys or ours. On your own keys, your first million requests each month are free. See it all in the console, delete it, export it, take your routes and go — they were always YAML in your git.

what "RigIndex" means

Where things stand

Current state, stated plainly. What works today: the gateway installs from npm (modelrig 0.8.1, Apache-2.0), accounts and keys are live at app.modelrig.ai, runs, steps and artifacts flow into the console's inspectable store (content custody is early access; metadata and lineage are live), and the probe suite, registry, leaderboard and contribution kits are public and reproducible. What is not out yet: prepaid balances, which are what puts the fee on rails — so nothing is chargeable and we would rather say that than have you find it out. Where this goes: the hosted path is the short one — sign up, issue a rig_sk_ key, point the exporter at us, and your routes stay YAML in your git. Self-host/BYOK stays the open lane: real, supported, open source, and the honest answer when a workload can't have a vendor holding payloads. On either one you pick a lane per route — Pure router keeps capture off and retains nothing; Optimization on keeps that route's telemetry rows and the review samples your bake-offs need, and repays them with community priors and evidence on your own routes — at the same flat 2% either way. How we charge, and what we keep.

Hosted setup docs land with billing.

The probe suite's verify command is fully standalone today — every published stat can be independently recomputed from its raw samples. See Reproduce our numbers in the public repo.

Prove it before you commit

We don't benchmark models in general. We bake them off on your traffic. Once your routes capture traffic, the bake-off replays your own logged requests against cheaper candidates:

shell
modelrig bakeoff --route example.support_summarize --variants gemini-3.1-flash-lite,gpt-5.4-nano --replay-last 50

No declared route — a rig.runRaw seam, or a run-based pipeline? There is nothing to --replay-last. Promote a real /runs output to a golden (the console's one-click make golden, or promoteToEval) and bake off the eval cases instead:

shell
modelrig bakeoff --from-eval-cases report.section --variants default,gemini-3.1-flash-lite

A bake-off scores conformance; a grade records whether the answer was actually good — your judgment, from your code, a human, or an AI, as a 0–1 score with an optional comment. Attach one with rig.grade from a signal you already have — a validator, a downstream accept/reject, a human QA gate:

app code
rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "human" }); // one call
rig.grade({ kind: "run", id: run_id }, { score: 0, kind: "human", comment: "…" });          // whole run

The write is synchronous and local, mirrored by the exporter, and never blocks your pipeline. Ten grades on a route reach Rung 1 on /coverage, and a route with no schema to conform to earns proposal confidence only this way. See the grade protocol.

Validity isn't accuracy: perfect JSON, wrong numbers. We gate on both.

Retention is a per-route choice between the two lanes above — same flat 2% on either. The two lanes.

More than the five steps

Once your runs are flowing, ModelRig also gives you per-customer tenant statements, shareable published receipts, a metadata-only analytics query API, provenance-first assembly & trust, and pairwise, regret-bounded bake-offs. The whole surface, one line each: how it fits →