ModelRig

ModelRig / reports / declared-vs-probed

Declared vs. probed: what 37 models' capability flags get wrong

The machine-readable capability flags the ecosystem routes on, probed against measured behavior. 29 of 37 models carry at least one discrepancy — 39 in total, in both directions. Every number on this page is computed from the live registry at render; raw per-sample records are published, CC-BY.

What “declared” means here

“Declared” here is not a provider's marketing page. It is the pinned machine-readable capability snapshot plus adapter flags — supports_response_schema, supports_web_search, supports_prompt_caching — the metadata layer that routers and gateways actually route on. That is why the gaps are expensive: a wrong flag doesn't mislead a reader, it misroutes traffic.

Better than their flags admit · 13 models

No declared schema support, yet probed conformance is high. The capability exists; the flag layer doesn't know. A router trusting the flags skips these models — and they are often the cheap ones, which is how capability-blind routing overpays.

Models that pass every schema fixture range from $0.082 to $52.827 per 1,000 conformant outputs on the current leaderboard — a 642× spread. Which end is your stack on?

The json_object-vs-schema gap · 6 models

Declared schema support — but every probed sample was served via json_mode coaching, with no native strict enforcement on this serving path. Valid JSON is not your schema; the rung matters, and the flag doesn't distinguish them.

  • deepseek/deepseek-chat — declared supports_response_schema, but every probed sample was served via json_mode coaching — no native strict enforcement exists on this serving path (the json_object-vs-schema gap)
  • deepseek/deepseek-v4-flash — declared supports_response_schema, but every probed sample was served via json_mode coaching — no native strict enforcement exists on this serving path (the json_object-vs-schema gap)
  • deepseek/deepseek-reasoner — declared supports_response_schema, but every probed sample was served via json_mode coaching — no native strict enforcement exists on this serving path (the json_object-vs-schema gap)
  • fireworks/gpt-oss-120b — declared supports_response_schema, but every probed sample was served via json_mode coaching — no native strict enforcement exists on this serving path (the json_object-vs-schema gap)
  • deepseek/deepseek-v4-pro — declared supports_response_schema, but every probed sample was served via json_mode coaching — no native strict enforcement exists on this serving path (the json_object-vs-schema gap)
  • fireworks/glm-5p2 — declared supports_response_schema, but every probed sample was served via json_mode coaching — no native strict enforcement exists on this serving path (the json_object-vs-schema gap)

Search flags that don't ground · 6 models

Declared web-search support, but probed answers carried citations well below the grounding bar. The capability may exist and simply not engage on this path — which is exactly what a router cannot know from the flag alone.

  • grok/grok-4-1-fast — declared supports_web_search, but only 27% of probed answers carried citations (bar: 50%)
  • grok/grok-4.3 — declared supports_web_search, but only 20% of probed answers carried citations (bar: 50%)
  • grok/grok-4.5 — declared supports_web_search, but only 27% of probed answers carried citations (bar: 50%)
  • openai/gpt-5.2 — declared supports_web_search, but only 7% of probed answers carried citations (bar: 50%)
  • openai/gpt-5.5 — declared supports_web_search, but only 0% of probed answers carried citations (bar: 50%)
  • openai/gpt-5.5-pro — declared supports_web_search, but only 0% of probed answers carried citations (bar: 50%)

Citations with no declared search · 14 models

A high share of probed answers cited URLs with no declared search flag. This does not mean the model secretly searches: either the flag layer is incomplete, or the citations are fabricated-shaped. Both readings demand verification — that is the finding.

  • deepinfra/meta-llama/Llama-4-Scout-17B-16E-Instruct — no declared web-search support, but 100% of probed answers cited URLs — verify whether these are real or fabricated sources
  • deepseek/deepseek-chat — no declared web-search support, but 100% of probed answers cited URLs — verify whether these are real or fabricated sources
  • deepseek/deepseek-v4-flash — no declared web-search support, but 80% of probed answers cited URLs — verify whether these are real or fabricated sources
  • deepinfra/meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 — no declared web-search support, but 100% of probed answers cited URLs — verify whether these are real or fabricated sources
  • deepseek/deepseek-reasoner — no declared web-search support, but 93% of probed answers cited URLs — verify whether these are real or fabricated sources
  • deepinfra/Qwen/Qwen3-30B-A3B — no declared web-search support, but 87% of probed answers cited URLs — verify whether these are real or fabricated sources
  • deepinfra/Qwen/Qwen3-Coder-480B-A35B-Instruct-Turbo — no declared web-search support, but 73% of probed answers cited URLs — verify whether these are real or fabricated sources
  • deepinfra/MiniMaxAI/MiniMax-M3 — no declared web-search support, but 87% of probed answers cited URLs — verify whether these are real or fabricated sources
  • anthropic/claude-haiku-4-5 — no declared web-search support, but 87% of probed answers cited URLs — verify whether these are real or fabricated sources
  • deepinfra/zai-org/GLM-5.2 — no declared web-search support, but 60% of probed answers cited URLs — verify whether these are real or fabricated sources
  • anthropic/claude-sonnet-5 — no declared web-search support, but 93% of probed answers cited URLs — verify whether these are real or fabricated sources
  • fireworks/kimi-k3 — no declared web-search support, but 60% of probed answers cited URLs — verify whether these are real or fabricated sources
  • anthropic/claude-opus-5 — no declared web-search support, but 93% of probed answers cited URLs — verify whether these are real or fabricated sources
  • anthropic/claude-fable-5 — no declared web-search support, but 93% of probed answers cited URLs — verify whether these are real or fabricated sources

Caching flags vs. the documented opt-in path — not counted above · 6 models

Declared prompt-caching support, but a repeat-call probe on the documented opt-in path still reported zero cached tokens (the marker was sent where the provider requires one; the prefix exceeded each provider's minimum). For implicit-cache providers a zero can be within spec — implicit caching is not guaranteed — so we deliberately do not count these as discrepancies until a prime-side cache-write confirmation is in the probe. What survives either way: the flag does not mean caching is automatic.

  • gemini/gemini-3.1-flash-lite — declared supports_prompt_caching, but no eligible probed repeat call on the documented caching path reported cached tokens
  • anthropic/claude-haiku-4-5 — declared supports_prompt_caching, but no eligible probed repeat call on the documented caching path reported cached tokens
  • gemini/gemini-3-flash-preview — declared supports_prompt_caching, but no eligible probed repeat call on the documented caching path reported cached tokens
  • anthropic/claude-sonnet-5 — declared supports_prompt_caching, but no eligible probed repeat call on the documented caching path reported cached tokens
  • gemini/gemini-3.1-pro-preview — declared supports_prompt_caching, but no eligible probed repeat call on the documented caching path reported cached tokens
  • openai/gpt-5.5-pro — declared supports_prompt_caching, but no eligible probed repeat call on the documented caching path reported cached tokens

Method, and its limits

Method: sampled probes with 95% confidence intervals, one serving path per model per cycle, declared layer pinned with as_of stamps. The corpus is v0 and finance-weighted — read rates as corpus-wide, not domain-general. The probe suite is reproducible; rerun it before you cite it.

Rerun it yourself: every published stat ships with its raw samples. Rerun it yourself · Contribute fixtures

data as of 2026-08-19 · vintage 2026-08-19T14:18:42.501Z · live @ 41d1634

What we do about it: the gateway routes on probed capability, not declared flags — and when a cheaper model passes on your own replayed traffic, the swap ships as a pull request you review. How A/B-tested routing works →