The router could learn real context limits instead of trusting declared ones (#13922, #14318, #14135, #14931)

[behrnt-slatgng]

Disclosure up front: I build Grunz, a hosted chat + coding agent on open-weight models. I'm not selling anything here; I read the recent context-limit issues and think OmniRoute is in an unusually good position to fix the root cause.

Every context-limit bug here is a declared number that was wrong

Going through the last few weeks:

• [BUG] deepseek-flash falls back to 128000 context limit (actual window is 1M) #13922: deepseek-flash fell back to 128,000 while the real window is 1M.
• bug(discovery): max_tokens is read as the context window and becomes inputTokenLimit (#14159) #14318: max_tokens was read as the context window.
• fix(api): /v1/models catalog emits a degraded duplicate entry keyed by the node UUID (vision dropped, context_length 128000) and input_modalities=null on the aliased entry #14135: a duplicate catalog entry emitted context_length: 128000.
• fix(backend): Context guard rejects Codex CLI at ~50k real tokens: chars/4 estimator over-counts ~1MB of tool schemas 5-6x, checked against a generic 128k fallback limit (model catalog miss → combo-min) #14931: a chars/4 estimate over-counted about 1 MB of Codex tool schemas 5 to 6 times (257,448 estimated vs. 49,822 reported by the provider), then checked that against a 128K fallback that combo-min spread across the whole combo.

Each fix corrects a declared number: the catalog, the discovery parser, the fallback. That's the right short-term fix, but the declared number can also be wrong in the other direction. On most hosted OpenAI-compatible endpoints I've checked, the context actually served is about 32K no matter what the model card or /v1/models says. When you go over it, some providers reject the request and some just truncate.

OmniRoute already has the data to learn the real limit

A router sees two things no catalog does:

1. Provider usage on success. The prompt_tokens the provider actually counted. fix(backend): Context guard rejects Codex CLI at ~50k real tokens: chars/4 estimator over-counts ~1MB of tool schemas 5-6x, checked against a generic 128k fallback limit (model catalog miss → combo-min) #14931's 49,822 came from there. Each success is a lower bound on the real ceiling for that provider and model.
2. The rejection text on failure. Providers usually state the real maximum in the 400 body ("maximum context length is N"). That's an upper bound.

Keeping observed_min_ok and observed_max_rejected per provider/model/account, and preferring them over catalog values once they exist, would make the guard self-correcting. It would also let combo-min take the minimum of observed limits instead of the minimum of guesses. And it gives you a way to catch a real window lower than declared, which no catalog fix can.

For the estimator, the same data gives you a correction factor for each request shape (the provider's count divided by your estimate), so a tool-heavy client like Codex stops being over-counted without needing a tokenizer per model.

Questions

1. Is the provider-reported prompt_tokens stored anywhere per request today, or only surfaced in usage stats?
2. Would you accept a PR that records observed bounds and exposes them with a source field, so /v1/models can say context_length: 262144 (observed) vs. (catalog) vs. (fallback)?

[diegosouzapw]

Thanks for the write-up, @behrnt-slatgng, and for the disclosure. The diagnosis matches what we've been fixing one number at a time. #14931 is still open, and it's the clearest case: a chars/4 estimate checked against a fallback, both of them wrong.

On your questions:

1. Is provider-reported prompt_tokens stored per request? Yes. Every request lands in the call_logs table with provider, model, connection_id, status and the provider-reported tokens_in / tokens_out, plus cache read/creation and reasoning tokens (schema in src/lib/db/migrations/025_call_logs_summary_storage.sql). So the lower bound you describe (largest successful tokens_in per provider/model/account) can be computed from data we already keep. The upper bound isn't stored in structured form today. The 400 text survives only as the error message on the log row, so parsing "maximum context length is N" would be new work.

2. Would we take a PR? Yes, with a few conditions, since this changes routing:

• Opt-in and advisory first. Record the observed bounds, expose them with a source field (observed / catalog / fallback) as you suggest, and let the guard use them only behind a default-off flag until there's evidence they beat the catalog.
• Scoped to provider + model + connection. Some providers do cap context per account or plan, so a bound learned on one account shouldn't leak to another.
• Monotonic and cheap. The lower bound only ever increases. The rejected-at value is kept only when the 400 is actually classified as a context overflow (isContextOverflow400), so an unrelated 400 doesn't poison it. And none of this adds a DB write on the hot path beyond what call_logs already does.
• Tests on the failure side, where a wrong limit hurts.

The estimator correction factor is a good second step. #14931 is the concrete case to validate it against.

On "about 32K served regardless of the card": we haven't measured that across providers ourselves, so I'd keep it as a hypothesis the observed data can confirm or refute, not as a design premise.