Skip to content

feat: a role can name several models, and falls back when one is down - #141

Merged
thedancingdeveloper merged 1 commit into
mainfrom
feat/model-fallback-chains
Aug 4, 2026
Merged

feat: a role can name several models, and falls back when one is down#141
thedancingdeveloper merged 1 commit into
mainfrom
feat/model-fallback-chains

Conversation

@thedancingdeveloper

Copy link
Copy Markdown
Contributor

Why

Probing every model the gateway advertises: 8 of 42 answered. 19 returned
model service unavailable, 8 server maintenance, 5 returned nothing at
all. That included two of the three models chosen for implementation.

A role that names one model is a fleet that stops when that name is down, and
until now there was no way to say "or this one".

What

A role holds an ordered chain. First that answers does the work.

--implementer deepseek-v4-flash,glm-5.2,gpt-5.4 --reviewer gpt-5.6

Stored as models: [...], alongside model as the preferred one — so a map
written before chains existed still reads, and every reader that knows only
model (including the RoleRoute wire schema) still sees a route.

The whole chain is tried before any backoff

This is the point. A provider that is down answers in milliseconds; backing
off against it for minutes before looking at the second choice would waste the
fallback entirely. The attempt ladder engages only after every route has
failed, and only if something was transient — if every route refused or is
out of budget, another cycle cannot help, so RequestRefused / CapExhausted
are raised immediately, in the terms the executor already acts on.

Falling back is not the same as parking

Fallback happens on any failure, deliberately including a refusal: one
vendor's refusal is routinely another's answer. Parking does not — a refusal
says something about the request, not the model's health, and idling a working
endpoint over one bad prompt would be a self-inflicted outage.

Two bounds, both tested

  • A chain protects against a model being unavailable, not against running
    out of budget: a spend cap belongs to the account, so it parks every
    model behind that endpoint.
  • /api/roles, readiness and the independence warning report the preferred
    route. A fallback that has not been needed is not what you configured.

Live proof

With deepseek-v4-flash and glm-5.2 genuinely down at the time:

implementer error — The Claw Bay is temporarily unavailable due to server maintenance.
implementer error — This model is temporarily unavailable on The Claw Bay.
implementer ok — fell back to gpt-5.4 (preferred deepseek-v4-flash)

Three calls, ~8 seconds, no backoff, item completed end to end. Across the run
every planner and implementer call fell back and said so.

15 new tests; the suite, ruff and mypy . green.

Measured on the endpoint this runs against: 34 of 42 advertised models were
unavailable at the same moment, including two of the three an operator had
just chosen for implementation. A role that names one model is a fleet that
stops when that name is down, and the harness had no way to say "or this one".

A role now holds an ordered chain. The first route that answers does the work;
the rest exist for the moment it will not. `--implementer a,b,c` on the CLI,
`models: [...]` in the stored map -- which also keeps `model` as the preferred
one, so a map written before chains existed still reads and every reader that
knows only `model` still sees a route.

**The whole chain is tried before any backoff.** A provider that is down
answers in milliseconds; sleeping on it for minutes before looking at the
second choice would waste the fallback entirely. Only when every route has
failed does the attempt ladder engage, and only if something was transient --
if every route refused or is out of budget, another cycle cannot help, so the
refusal or the cap is raised immediately in the terms the executor already
acts on.

Falling back happens on any failure, deliberately including a refusal: one
vendor's refusal is routinely another's answer. Parking does not: a refusal
says something about the request, not about the model's health, and idling a
working endpoint over one bad prompt would be a self-inflicted outage.

Which model answered is recorded, because a fleet quietly running on its third
choice for a week is a fleet whose cost and results nobody can explain.

Live, with the two preferred models genuinely down:

    implementer error — ...temporarily unavailable due to server maintenance.
    implementer error — This model is temporarily unavailable on The Claw Bay.
    implementer ok — fell back to gpt-5.4 (preferred deepseek-v4-flash)

Three calls, eight seconds, no backoff, item completed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant