feat(ai): validate catalog generation invariants and fail violating regens

[snimu]

Purpose

Make catalog regeneration self-checking: a regen that violates known invariants fails loudly and refuses to write, instead of shipping bad rows that surface as user-facing 400s, early compaction, or dead /effort levels. Fixes the current violations by regen-time derivation, not hand-edits. Squashes #1683/#1680, #1406, #1597, the live case of #1725, and guards the #1679/#1724 staleness class at the generator.

Mechanism (new scripts/validate-model-catalog.ts, wired into generate-models)

Four invariants, checked on the deduped catalog before it is written; violations print by name and the generator exits nonzero without writing:

1. maxTokens <= contextWindow on every row ([Bug] Generated model catalog contains impossible maxTokens > contextWindow values #1683/[Bug] Generated catalog contains models where maxTokens exceeds contextWindow (Inkling-Small, openrouter gpt-3.5-turbo-0613) #1680).
2. Every github-copilot id must match an explicitly classified family, and the row's api must equal that classification. Unclassified families fail by name instead of silently defaulting to chat completions — the exact mechanism that shipped the Grok misroute (GitHub Copilot grok-4.5 is routed to Chat Completions during model generation #1406).
3. openai-codex rows may not diverge more than 2x from the same model's openai contextWindow ([Bug] openai-codex applies one flat 272k context window to every model, including 1M-context models #1597/[Bug] Cross-provider catalog limit contradictions cause silent truncation and premature compaction #1724 class).
4. Same-model rows across providers must agree on selectable thinking levels ([Bug] thinkingLevelMap inconsistently populated across providers for the same model - /effort diverges by provider #1725). The off level is excluded from comparison: whether thinking can be disabled legitimately varies per transport (reasoning effort none exists only on native OpenAI Responses; the generator encodes that deliberately per provider).

generateModels().catch now sets a nonzero exit code (previously catch(console.error) exited 0 even on total failure).

Data fixes (all by derivation; catalog regenerated, 22 rows changed, zero unrelated drift)

fix	derivation	rows
Copilot Grok + MAI-Code 400 on /chat/completions (#1406)	family classification routes grok-/mai- (and gpt-5/oswe) to openai-responses; matches upstream pi-mono #906, which validated Grok as Responses-only	4
Codex GPT-5.6 family stuck at the flat 272k (#1597)	codex contextWindow derives from the same model's openai row when the API side serves 1M+; the 272k constant stays for the generation it was measured on (5.1-5.5 divergences are <2x measured limits)	3 → 1,050,000
thinkingmachines/Inkling-Small maxTokens 1,048,576 > contextWindow 524,288 (#1683)	models.dev ingestion clamps maxTokens to contextWindow (upstream data carries the 1M sibling's output limit; same clamp precedent as createPrimeInferenceModel)	1
Kimi K3 /effort divergence (#1725's one live conflict)	KIMI_K3_THINKING_LEVEL_MAP aligned with the low/high/max levels the model demonstrably supports on OpenRouter and Prime Inference (both routes pass efforts through to Moonshot serving)	12 gain low/high

Verified: the regenerated catalog is byte-stable across a second regen, and the CI-mirroring npm run build (which still regenerates on main) reproduces it exactly.

Tests

• New test/model-catalog-validation.test.ts (6 tests): one pin that the committed catalog passes with zero violations (this is the standing regression guard for all four invariants), one fixture-violation test per invariant proving it fails loudly, and one pin that off-level variance alone is accepted.
• supports-xhigh.test.ts: existing 272000 pin updated to 1050000 (fail-unfixed for the codex fix).
• E2E fail path exercised in a sandbox: reintroducing the kimi drift makes npm run generate-models exit 1 with the named violation and leaves the committed catalog untouched.

Premise corrections vs the RES-1273 plan

• validate-model-catalog.ts did not exist on main; feat: refresh models from a hosted catalog #1928 introduces a 9-line schema wrapper at the same path on its branch. This PR creates the invariant validator there with composable exports (validateModelCatalog(catalogLike)), so feat: refresh models from a hosted catalog #1928 can call the same invariants on its hosted aggregate after rebase. Heads-up left for coordination: same path, trivial to reconcile, no second validator.
• defaults-exist is deliberately NOT a generator invariant: defaultModelPerProvider lives in packages/coding-agent, and importing it from the ai generation script requires a built pi-ai dist (generation must work pre-build) — a fragile reverse dependency. Open fix(ai): omit the default service tier, reprice cache writes from message_delta, repoint the zai default #2032 already adds the catalog-existence pin at the right layer (coding-agent tests), which fails in the same PR as any violating regen. Duplicating it in the generator would be a wrong-layer second checkpoint bought with a brittle import.
• The plan counted 2 current violations; the requested invariants actually surface 3 in-scope (Inkling-Small, codex 5.6 contexts, kimi-k3 thinking levels — all fixed here) plus the pre-existing zai default gap owned by fix(ai): omit the default service tier, reprice cache writes from message_delta, repoint the zai default #2032.
• A full cross-provider thinkingLevelMap merge pass (propagating maps to the ~70 families where blind sources emit none) is deliberately out of scope: it is a large behavioral change requiring per-transport verification, and only one family conflicts today. The invariant now makes any future conflict loud.

Size

Net src/scripts LOC: +127 mechanism (validator 108, generator wiring +19 net), 2 test files +72/-1, catalog data ±58 regenerated lines. Mechanism-vs-deletion: the invariants are additive tooling; the fixes delete wrong premises (prefix heuristic → explicit classification, flat constant → derived per-model context, stale hand map → source-aligned map).

Linear: RES-1273

---

Note

Medium Risk
Touches model routing and catalog metadata used by all providers; mistakes could break completions or effort UX, but validation fails closed and tests pin the committed catalog.

Overview
Adds validateModelCatalog and runs it at the end of generate-models so a regen that breaks known rules logs named violations, sets a nonzero exit code, and does not write models.generated.ts. Invariants cover maxTokens vs context window, explicit GitHub Copilot API family routing (including grok- / mai- → openai-responses), openai-codex context staying within 2× of the matching openai row, and consistent selectable thinking levels across providers (with a generator pass that nulls levels transports cannot send).

The regen fixes the violations those checks caught: Copilot Grok/MAI-Code misrouted to chat completions, GPT-5.6 openai-codex context raised to ~1M when the OpenAI twin serves it, Kimi K3 low/high effort on providers that support them, Moonshot/Copilot Kimi rows stripped of bogus effort UI, and Inkling-Small maxTokens clamped. getCompat is exported so validation reuses the same transport rules as runtime.

In coding-agent, switching to a reasoning model with zero selectable effort levels no longer overwrites the persisted default to off; cycle effort is a no-op on those models.

^{Reviewed by Cursor Bugbot for commit 36e9b01. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Validate catalog generation invariants and fail violating regens

• Adds a catalog validator in validate-model-catalog.ts that checks maxTokens vs contextWindow, Copilot family/API agreement, Codex/OpenAI context divergence, cross-provider thinking-level agreement, and unsendable reasoning levels; the generator now exits non-zero on any violation.
• Regenerates the model catalog in models.generated.ts: Grok and MAI-Code Copilot rows route to the Responses API, Codex GPT-5.6 rows use a 1,050,000 context window, Inkling-Small maxTokens is capped at 524,288, and compatible Kimi K3 rows expose low/high reasoning levels.
• Exports getCompat from openai-completions.ts so validation reuses the same transport compatibility logic as the runtime.
• Fixes thinking-level persistence in agent-session.ts: switching to a fixed-effort model no longer overwrites the saved thinking default, cycling is a no-op when no levels exist, and switching away restores the persisted default.
• Behavioral Change: catalog generation now fails on invariant violations instead of silently writing bad data; Codex GPT-5.6 context window changed from 272,000 to 1,050,000 in tests in supports-xhigh.test.ts.

^{Macroscope summarized 36e9b01.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1b8199e. Configure here.}