feat: a role can name several models, and falls back when one is down - #141
Merged
Conversation
Measured on the endpoint this runs against: 34 of 42 advertised models were
unavailable at the same moment, including two of the three an operator had
just chosen for implementation. A role that names one model is a fleet that
stops when that name is down, and the harness had no way to say "or this one".
A role now holds an ordered chain. The first route that answers does the work;
the rest exist for the moment it will not. `--implementer a,b,c` on the CLI,
`models: [...]` in the stored map -- which also keeps `model` as the preferred
one, so a map written before chains existed still reads and every reader that
knows only `model` still sees a route.
**The whole chain is tried before any backoff.** A provider that is down
answers in milliseconds; sleeping on it for minutes before looking at the
second choice would waste the fallback entirely. Only when every route has
failed does the attempt ladder engage, and only if something was transient --
if every route refused or is out of budget, another cycle cannot help, so the
refusal or the cap is raised immediately in the terms the executor already
acts on.
Falling back happens on any failure, deliberately including a refusal: one
vendor's refusal is routinely another's answer. Parking does not: a refusal
says something about the request, not about the model's health, and idling a
working endpoint over one bad prompt would be a self-inflicted outage.
Which model answered is recorded, because a fleet quietly running on its third
choice for a week is a fleet whose cost and results nobody can explain.
Live, with the two preferred models genuinely down:
implementer error — ...temporarily unavailable due to server maintenance.
implementer error — This model is temporarily unavailable on The Claw Bay.
implementer ok — fell back to gpt-5.4 (preferred deepseek-v4-flash)
Three calls, eight seconds, no backoff, item completed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Probing every model the gateway advertises: 8 of 42 answered. 19 returned
model service unavailable, 8server maintenance, 5 returned nothing atall. That included two of the three models chosen for implementation.
A role that names one model is a fleet that stops when that name is down, and
until now there was no way to say "or this one".
What
A role holds an ordered chain. First that answers does the work.
Stored as
models: [...], alongsidemodelas the preferred one — so a mapwritten before chains existed still reads, and every reader that knows only
model(including theRoleRoutewire schema) still sees a route.The whole chain is tried before any backoff
This is the point. A provider that is down answers in milliseconds; backing
off against it for minutes before looking at the second choice would waste the
fallback entirely. The attempt ladder engages only after every route has
failed, and only if something was transient — if every route refused or is
out of budget, another cycle cannot help, so
RequestRefused/CapExhaustedare raised immediately, in the terms the executor already acts on.
Falling back is not the same as parking
Fallback happens on any failure, deliberately including a refusal: one
vendor's refusal is routinely another's answer. Parking does not — a refusal
says something about the request, not the model's health, and idling a working
endpoint over one bad prompt would be a self-inflicted outage.
Two bounds, both tested
out of budget: a spend cap belongs to the account, so it parks every
model behind that endpoint.
/api/roles, readiness and the independence warning report the preferredroute. A fallback that has not been needed is not what you configured.
Live proof
With
deepseek-v4-flashandglm-5.2genuinely down at the time:Three calls, ~8 seconds, no backoff, item completed end to end. Across the run
every planner and implementer call fell back and said so.
15 new tests; the suite, ruff and
mypy .green.