Summary
Have the model catalog refresh refuse to propose a snapshot that drops a large share of a provider's models, instead of relying on a downstream type error to notice.
What happened
#92 proposed a Cloudflare AI Gateway snapshot that removed 26 models:
main : anthropic-messages 10, openai-completions 13, openai-responses 34
#92 : anthropic-messages 10, openai-responses 21
The whole openai-completions listing vanished and openai-responses fell by 13, while anthropic-messages was untouched. That shape reads as a partial fetch rather than a vendor retirement, and the same listing had gained eight models in #86 days earlier.
It was caught, but by accident. With no openai-completions models left, the generated type union lost that key and stopped matching the streams cloudflare-ai-gateway.ts:19 declares, so build:offline failed with TS2353. That check exists to keep provider code and catalog data in sync, not to detect data loss.
Nothing would have caught it if the disappearing provider had no such declaration. A snapshot that silently deletes models reaches users as "model not found" against a configuration that worked yesterday.
Required behavior
Before opening a refresh pull request, compare the regenerated snapshot against the checked-in one and refuse to propose it when a provider loses more than an agreed share of its models — or loses an entire API grouping — unless explicitly overridden.
The report should name what disappeared, so a human can tell a retirement from a fetch failure at a glance. Growth needs no gate; this is about deletions.
An override is needed for genuine mass retirements, which do happen: #86 legitimately retired nine Cloudflare Anthropic IDs. A workflow_dispatch input, or a label on the PR, is enough.
Acceptance criteria
Related
Summary
Have the model catalog refresh refuse to propose a snapshot that drops a large share of a provider's models, instead of relying on a downstream type error to notice.
What happened
#92 proposed a Cloudflare AI Gateway snapshot that removed 26 models:
The whole
openai-completionslisting vanished andopenai-responsesfell by 13, whileanthropic-messageswas untouched. That shape reads as a partial fetch rather than a vendor retirement, and the same listing had gained eight models in #86 days earlier.It was caught, but by accident. With no
openai-completionsmodels left, the generated type union lost that key and stopped matching the streamscloudflare-ai-gateway.ts:19declares, sobuild:offlinefailed withTS2353. That check exists to keep provider code and catalog data in sync, not to detect data loss.Nothing would have caught it if the disappearing provider had no such declaration. A snapshot that silently deletes models reaches users as "model not found" against a configuration that worked yesterday.
Required behavior
Before opening a refresh pull request, compare the regenerated snapshot against the checked-in one and refuse to propose it when a provider loses more than an agreed share of its models — or loses an entire API grouping — unless explicitly overridden.
The report should name what disappeared, so a human can tell a retirement from a fetch failure at a glance. Growth needs no gate; this is about deletions.
An override is needed for genuine mass retirements, which do happen: #86 legitimately retired nine Cloudflare Anthropic IDs. A
workflow_dispatchinput, or a label on the PR, is enough.Acceptance criteria
Related