Skip to content

fix: move a Claude model you chose to the default before a message fails once Claude stops offering it - #26459

Closed
brennanb2025 wants to merge 18 commits into
mainfrom
brennanb2025/ssh-chat-model-discovery-retry
Closed

brennanb2025 wants to merge 18 commits into
mainfrom
brennanb2025/ssh-chat-model-discovery-retry

Conversation

@brennanb2025

@brennanb2025 brennanb2025 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 34 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​2426 $\color{#cf222e}{\Huge{\mathbf{−}}}$​43 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​2383
Prod 54 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​898 $\color{#cf222e}{\Huge{\mathbf{−}}}$​207 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​691

The problem

Orca saves each account's list of Claude models. A chat's selected model can be missing from that saved list, either because the list is out of date or because Claude really stopped offering the model. On main, Orca launched the missing model anyway. The first message failed with Claude's "model not found" reply, and only then did the chat fall back to the default model.

Earlier rounds of this PR fixed that, and review turned up four more problems:

  • A model you configured outside Orca could be replaced. When a chat started with nothing picked in Orca, Claude ran whatever its own settings chose (~/.claude/settings.json, a project's .claude/settings.json, ANTHROPIC_MODEL, a Bedrock id). Orca saved that model as if you had picked it, so a later list that didn't name it could move the chat to another model.
  • A model a caller named outright could be replaced. orca orchestration worker-start --model claude-sonnet-4-5, or orca agent launch --model, saves that model on the chat just like a picker choice. Claude's model list is the picker's set, not every model Claude can run, so an older or dated id that Claude still runs could be swapped for the account's default. Nobody watches a background worker's picker, so nobody would notice.
  • One project's narrower list could call a model gone for every chat. A running chat lists the models its own project allows (a project can restrict models, or route Claude to another provider). For the next minute that narrower list stood in for the whole account's list, so a chat in another project could start on a different model.
  • A stopped chat's picker showed the old model until you sent. Reopening a stopped chat starts a fresh listing, and the host answers with the saved model while that listing runs. Nothing read the chat's options again once the listing landed, so the picker kept showing the gone model until the send replaced it.

What you see now

  • Only a model your own selection chose is ever replaced. That is either a model picked for this chat, or the remembered new-chat selection a new chat started from:
    • Picked for this chat: a pick in the chat's picker on desktop or phone, running or stopped. Technically, any model sent through agentSession.setOption, or in the options of agentSession.create (the picks a new chat's picker held before the chat existed), from any client. Today only the desktop and phone pickers call these, but they are public calls, so any client that uses them counts as a picker.
    • The remembered new-chat selection: the model you last chose for new chats, which a new chat starts with when nothing else names one. This includes orca orchestration worker-start and orca agent launch runs without --model. So a background worker started without --model can have its model replaced when Claude stops offering it, even though nobody is watching that worker's picker.
  • A model a caller named always runs as given. That means orca orchestration worker-start --model and orca agent launch --model (calls through agent launch with an explicit model). If Claude can't run it, Claude's own error shows, as on main.
  • Chats created before this change keep main's behaviour until you pick a model in them. Their model is never replaced. Once you pick a model in one, that pick counts as yours and can be replaced like any other.
  • Your model is never dropped just because a saved list is old. If the saved list is more than a minute old and doesn't have your picked model, Orca asks Claude for a fresh list once before deciding anything.
  • A model Claude really removed is replaced before a message fails. When the fresh list doesn't have it, the picker shows the account's default model, marked as the default, and the next start launches that model by name. Your effort level moves to the nearest level that model offers (for example, max becomes xhigh when the default stops at xhigh).
  • Stopped and restored chats show the replacement before you send, on desktop and on the phone. When the fresh list lands, the picker reads the chat's options again. It also reads them again, once, if the host's catalog answer arrives after the chat's options answer and names a replacement those options still lack. That can happen when the catalog request is slow, for example on a paired server.
  • A model your Claude config chose is left alone. If Orca sent no model and no row of Claude's list names the one it ran, Orca no longer saves it as your pick, so the next start lets Claude's config choose again. A model saved under the id an alias runs (for example claude-sonnet-5 for Sonnet) counts as listed.
  • Nothing waits long. A chat start spends at most 3 seconds on the model check, then starts as saved. Reading a stopped chat's options never waits.
  • While Orca can't check (the list is still loading, listing failed, or the server is older), your model is kept and Orca offers no effort levels it can't confirm. A new chat's picker shows the plain "Model" placeholder for a saved pick the list doesn't name, as main already does.
  • A floating chat's picker shows the model the start will run: the account's listed default, even where the workspace hides which model is the default.

How it works

  • Each chat records who chose its model. A new optional field on the chat's saved record, modelChosenBy, has one of three values:

    • picker: a model picked for this chat (agentSession.setOption with a model, running or stopped, or a model in agentSession.create's options).
    • new-chat-default: a new chat started from the remembered new-chat selection, including orchestration and agent launches without --model.
    • caller: orchestration or an agent launch passed its own --model.

    Keeping an explicit pick apart from the remembered selection means a later question, such as "was this the user's own choice for this chat?", can be answered without guessing. Only picker and new-chat-default can be replaced.

    • The field is kept while the chat keeps that model, including when Claude reports the same model under another id, or when the start replaced a gone model. It is dropped when the model goes away. A model saved from Claude's own report for a chat that had none gets no value.
    • When the field is absent, Orca treats the model as a caller's. That covers every record written before this change.
    • Read leniently. A value this build doesn't know, for example one a newer build adds, reads as absent (a caller's), and the rest of the chat loads normally. An unknown value never sets the whole chat aside.
  • Only the user's own selection is checked. The catalog service looks at a chat's saved model only when modelChosenBy is picker or new-chat-default. A start reads the catalog only for such a chat, and only for agents that replace a gone model (Claude). The whole start read, including the workspace-config checks it may wait on, is capped at 3 seconds in one place (the start's own cap; the read has no second, inner wait).

  • Only the account-level listing can call a model gone. The "account-level listing" is the list Orca asks Claude for outside any chat, with no project settings applied. Orca names a replacement only while that listing is newer than any running chat's list and less than a minute old. A running chat's list, which its project's settings may narrow, never decides that a model is gone. Each save of a running chat's list is dated by when that chat first saved it, so an old chat's list never counts as newer than a fresh one.

  • One fresh listing per decision. Orca lists once if all of these hold: the account-level listing is at least a minute old, the list doesn't name the picked model by id or by the id an alias runs, and no listing failed in the last 30 seconds. A start waits up to 3 seconds for it. A picker's follow-up read waits up to the existing 30-second picker limit. Every other read answers "listing in progress" right away.

  • The host names the replacement. A catalog answer carries a new optional unlistedModelReplacement field, sent only for a picked model while the list is current. The host's start step, the host's answer for a stopped chat, and the desktop picker all use that one value with one shared rule (agent-session-model-fallback.ts). The same file now holds the one "is this model listed" check, which Claude's option code uses too, so the copies can't drift apart.

  • Stopped chats re-read their options. On desktop and phone, the picker's waiting catalog read already landed when the listing finished, but its answer was ignored once the chat's own options had arrived. Now the picker also reads the chat's options again at that point, and they come back decided against the fresh list. It does the same once for a catalog answer that didn't wait, if that answer names a replacement and the chat's options still show a model the answer's list lacks. A re-read never triggers another one: the catalog read doesn't run again because the options changed. There is no wire change.

  • Resolved ids are kept. Claude's listing names the model id each alias runs. That id now stays on the host's saved rows: through the listing, the saved file, and the optional wire field resolvedModel. Matching uses it.

  • What a start saves. When a Claude chat started with no model from Orca, its reported model is saved only if a listed row names it. Otherwise the record stays unpicked.

  • An unknown model is left out, not sent as ''. When nothing identifies a running chat's model, its options leave current.model out, following main's optional-model rule.

Merged with main

main's new shared model catalog landed while this PR was open: one saved catalog for every agent, account-level listings kept apart from running-chat listings, a learned configured default, and saved-only reads. This PR builds on it and drops its own copies of what main already does:

  • Stopped chats follow main. At rest, a Claude chat names the model and effort Claude's config resolves to, once main has learned them.
  • Running chats follow main. A running Claude chat doesn't read the host's saved list. With no list of its own, it shows the built-in list, as on main.

Why

The aim is what you would expect: a model you picked is never silently dropped because a list is old, a model Claude really removed is replaced before a message fails, a model you or a script named on purpose keeps running, and checking never holds up a start.

Alternatives considered:

  • Replace any saved model the current list lacks. Rejected: Claude's list is the picker's set, not every model Claude runs, so a model named on purpose would be swapped silently.
  • Refuse an explicit unlisted --model at start. Rejected for now: main runs it as asked. Claude's own error already says when it can't run.
  • Work out who chose the model from existing fields instead of storing it. Not possible: nothing on the record says whether a model came from the picker, the remembered selection or a caller. Orchestration, agent launch and the picker all write the same options.
  • One value for both a pick and the remembered selection. Rejected: the remembered selection can hold a model you never picked for this chat (see the known gap below), so the two are kept apart.
  • Wait for Claude's "model not found" reply. Rejected: you see a failed message before the fallback. It remains the backstop when the 3-second wait runs out.
  • Let the start wait for the listing. Rejected: bookkeeping must never delay what you asked for, and paired servers time out after 15 seconds.

How this differs from the common pattern:

  • Intended (matches only in part): Orca stores who chose the model next to the model.
    • What matches: the common pattern also keeps a source beside the model, separates an explicit choice from a remembered client preference, and reads an absent source as the conservative value.
    • What differs: the common pattern's source is per request (carried with each queued message), not stored on the chat. Its values decide whether a request updates the thread's and the project's saved model, not whether a model is replaced; there, replacement is the picker's own recovery of an unavailable model. Orca's value is persisted per chat, and it gates replacement, because a stopped chat's start is decided on the host, possibly long after the pick. Its absent value means "caller" (never replaced), because every record from before this change must keep main's behaviour.
  • Intended: an explicit model change that the list doesn't name is not refused. Claude's own error appears instead, matching what main does for --model today. The common pattern refuses such a change with a plain message.
  • Intended: the host also decides at start, not only the picker. A new chat's agent starts as soon as its tab opens, so a "use this model on the next send" marker would arrive after the old model had already launched. Both places use the same value and rule.
  • Intended: the start waits at most 3 seconds, then starts as saved. The common pattern checks only when a model changes, never at session start. Claude's own "model not found" reply covers the rest.
  • Intended: effort moves to the nearest level the replacement offers (the higher one on a tie), which matches the common pattern.
  • Intended: running chats never decide that a model is gone, and their (possibly project-narrowed) list never names a replacement. The common pattern doesn't let running sessions write the list at all. main lets them add which models exist.
  • Intended: while the list can't be checked, a saved pick it doesn't name shows no effort levels. Orca launches the selected model with its effort, so another model's levels could be rejected.
  • Intended: only Claude replaces a gone model. The other agents keep main's catalog rules.
  • Temporary: the common pattern stores only the model the app itself sent. Orca also saves a model Claude's config chose when a listed row names it (saved under that row's id). That half is inherited from main. Such a model has no modelChosenBy, so it is never replaced. It does, however, stop the chat from following a later settings.json change. Follow-up: stop saving a model Orca never sent.
  • Temporary: starting chats on a managed SSH server, and Retry there, still wait on the separate managed-server SDK packaging fix and its SSH QA.

One change for every agent, disclosed: a running chat whose agent reports no model now shows the quiet "Model" placeholder instead of naming its list's default as what it runs. The exception is an agent whose built-in list proves its default is what a launch with no model runs (Grok). main named the listed default for every agent. A child that reports no model has said nothing about what it runs, and the catalog's own rule is to name a default only where it is provable. In practice this is the moment before a chat's first report, or an agent that never reports one.

Known gap (from main, not changed here): at rest and in a running chat, a no-pick Claude chat names the configured default main learned. If you then change only its effort, the picker saves that effort together with that model, so new chats launch it by name. Not addressed here; worth a follow-up decision.

Compatibility (compared with main)

Clients and servers update independently.

  • New optional answer fields: unlistedModelReplacement on the catalog answer and resolvedModel on model rows. Older clients ignore both. Neither side has a strict schema for them.
  • New optional saved-record field: modelChosenBy. It never crosses the wire. A record without it behaves as on main (never replaced). An unknown value reads as absent instead of setting the chat aside.
  • No new request fields: the host works out the selected model and its source itself.
  • Absent current model:
    • On main, a running Claude chat that reported no model was given the list's default.
    • Now its options leave the model out, so an older client without the optional-model capability gets main's existing "client capability missing" refusal (fix(acp): start chats for agents that report no model #26587) for that read.
    • That window is only before Claude's first report, or with an older CLI. Claude 2.1.280 reports the model.
  • Changes in what the host publishes:
    • A stopped chat whose picked model is confirmed gone is reported on the default.
    • Starts launch the replacement for a picked model.
    • A no-pick Claude chat's record no longer names a config-chosen model that no listed row names.
  • New client with an older server: there is no replacement field, so the selection is kept and Claude's "model not found" reply still moves the chat after its first message.

Linked Issue

Internal SSH structured-chat remediation plan, model-discovery failure (U28). No new issue opened.

Visual Proof

A live run on the desktop app with a local folder workspace. It used a stand-in Claude CLI whose model list can drop "Fake Opus" on demand; the real claude never ran. The run was on the final head 59adf2864e3, after the last merge with main.

1. Pick "Fake Opus (listed)" at Extra high and send a first message. The start launched --model claude-opus-4-6 --effort xhigh.

01-picker-opus.png

2. Claude stops offering Fake Opus. The stand-in's fresh listing drops claude-opus-4-6. Reopening the stopped chat, with nothing sent, shows the account's default "Fake Sonnet (listed)", and the effort moves from Extra high to High. The picker no longer lists Fake Opus.

04-composer-sonnet-replacement.png

05-picker-sonnet-replacement.png

3. The next message runs on the replacement with no failed turn. The start launched --model claude-sonnet-4-6 --effort high --resume=….

06-second-message-sonnet.png

4. When Claude offers Fake Opus again, a new chat's picker lists it again, with all its effort levels.

07-new-chat-opus-offered.png

What the run also showed:

  • The first reopen kept Fake Opus. This is by design. At that point the saved account list was still fresh and still named Fake Opus, so there was nothing to re-check. The swap happened once the saved list was refreshed and no longer named the model.
  • Nothing on screen says the model was changed. The picker shows the new model and the effort level, and that is all.
  • The sidebar's chat row keeps showing the old model id until the next run.

Testing

  • I manually tested these changes locally
  • Automated tests added/updated

Run on macOS with env -u ORCA_USER_DATA_PATH, at most two workers:

  • Tests this PR adds or changes: 32 files, 287 passing, 1 skipped (on the head merged with current main; with the rest of the catalog folder, 42 files and 427 passing).

  • Phone: the 6 option test files under mobile/src/session/, including both stopped-chat tests, 50 passing.

  • Proof the new tests catch the bugs: removing each piece one at a time makes its own test fail:

    • marking a model picked on a stopped chat as picker
    • the lenient read (a strict check, or skipping the read at load)
    • marking a seeded create as new-chat-default
    • treating new-chat-default as replaceable (catalog answer and start)
    • the one shared "is it listed" rule in the picker view
    • the late re-read, on desktop and on the phone

    Earlier rounds' ablations (the account-level-only check, the caller check in the catalog and at start, the 3-second cap, the waiting re-reads, the start's save rule) still apply.

  • Real Claude CLI: claude-structured-real-cli.test.ts and claude-structured-real-cli-fold.test.ts pass (13 tests, with the user's own login).

  • Typecheck: tsconfig.node.json, tsconfig.tc.web.json and mobile (with its tests-typecheck ratchet) are all clean.

  • Repo checks: orca-ci-checks --no-typecheck passes 22/23. The one red is 5 no-cycle warnings in mobile transport and terminal files this PR doesn't touch, a known local-only false red from generated mobile files. After the merge with main, oxlint, oxfmt --check and check:code-quality:changed pass on every changed file.

@brennanb2025 brennanb2025 changed the title fix: preserve model choices when discovery is unavailable fix: keep model choices usable when discovery fails Oct 8, 2026
…atalog row

When model discovery is empty or fails and the current model is missing from the
saved catalog, it was added with no effort choices, so the picker refused efforts
the provider accepts. Give it the same unknown-capability choices as saved rows;
a real catalog still constrains.
… chats

Idle and restored chats treated a saved model catalog as current truth, so an
effort-only change could save an unchosen model and valid effort choices could
disappear. Live and resting reads now use the same projection: saved rows give
labels only, unknown capabilities stay permissive, and a real child listing
still restricts choices.
Reuse the registered Claude saved-choice policy when serving modelCatalog,
so remembered capability claims cannot reject effort picks before startup.
Keep explicit launch choices and live-child authority, with local/paired
picker and runtime composition regressions.
…-model-discovery-retry

# Conflicts:
#	src/main/native-chat/agent-model-catalog/agent-model-catalog-service.ts
A model selected before the host knew it (launch seed, held pick) could lose
its effort choices when a remembered catalog without it arrived. The shared
option snapshot now gives any selected Claude model missing from a non-live
list the permissive unknown-model effort choices, so every picker input gets
the same rule; real live listings still restrict. Updates two tests whose
setup predated main's initialize-time startup report.
@brennanb2025

Copy link
Copy Markdown
Contributor Author

Review summary

Problem: when Claude couldn't list its models on an SSH host (the list failed or came back empty), that failure stuck: Retry and new chats kept failing until the server restarted. Fixing that surfaced a family of related picker bugs. A saved, older model list was being treated as the truth about what the current model supports. So valid effort choices (like Extra high) could disappear or be refused, and changing only the effort could quietly save a model you never picked.

User-facing change:

  • A failed or empty model list is retried instead of sticking.
  • Saved model names stay available as choices, but they no longer decide which effort levels you can pick, which model is the default, or whether Fast is offered. Only a model list from the running agent restricts the choices.
  • Any model you've selected keeps its effort choices even if a saved list doesn't include it.
  • Changing only the effort never saves a model you didn't pick.

How: one rule for saved model lists, applied at the single place that builds every picker's choices: a running chat, a stopped or restored chat, a new chat before its server-side record exists, and the phone. The SSH server also applies it to the model list it sends, so older desktops and phones get the corrected list too.

Review rounds: each of the earlier rounds found the same bug at a different picker input. Round 5 moved the rule to the shared place that builds the picker's choices, and checked every input against it.

  • Round 6 (independent, head 51b2cbd): no findings. 430 tests pass. The two failures in claude-stream-json-connection.test.ts also fail on main and aren't touched by this PR.
  • After syncing with main, two tests were updated because main now reports the agent ready earlier, before it reads its settings. The review confirmed nothing regressed.

Not verified: a live SSH chat hitting a failed model list and pressing Retry, the rendered desktop and phone pickers, and Linux, Windows or WSL. Live SSH QA across all the SSH chat fixes is running now, and screenshots will go in the PR body before this is marked ready. Claude chats over SSH also need #26450 (the server build fix) to start at all.

Merge-order note: if #26116 (the chat permission picker) merges first, this PR's model-choice code for a stopped chat has to be combined with #26116's permission handling there. The combined test build found and fixed that interaction, and the same fix goes into whichever PR merges second.

A saved model list that lacks the chat's selected model no longer gets every
effort level offered for it. An aged list is re-listed once before it may say
the model is gone; a current list that lacks it moves the chat to the listed
default, at rest, at the next start, and in the picker before the chat starts.
While the list is loading or failing, the selection stays and no effort levels
are invented for it.
… a Claude start

- A model Claude's own settings or env chose (one Orca never sent, which no
  listed row names) is no longer saved as the chat's pick, so a later list can't
  call it gone; the next start lets the CLI choose again.
- Host model rows keep the id each alias runs, and a pick saved as that id
  counts as listed.
- A running child's list keeps the age of its start and never replaces a newer
  listing, so it can't make a newer pick look gone.
- A start waits at most 3 s for the one re-listing, then launches as saved;
  at-rest reads never wait.
- The host names the replacement (the account's listed default) so a floating
  chat's picker shows what the start runs; effort moves to the nearest level
  the replacement offers.
- Remove the unused effortDefaultsToModel and fallbackModels rules.
…-model-discovery-retry

# Conflicts:
#	src/main/claude/claude-structured-effort-default-at-rest.test.ts
#	src/main/claude/claude-structured-session-options.ts
#	src/main/claude/claude-structured-session-startup.ts
#	src/main/claude/claude-structured-session-state.ts
#	src/main/native-chat/agent-model-catalog/agent-model-catalog-service.ts
#	src/main/native-chat/agent-model-catalog/agent-model-catalog-store.ts
#	src/main/native-chat/agent-session-wire/structured-agent-session-options-read.ts
#	src/main/runtime/rpc/methods/structured-agent-session-options-read.ts
#	src/main/runtime/structured-agent-model-catalog-wiring.ts
#	src/renderer/src/components/native-chat/use-structured-agent-session-option-state.ts
#	src/shared/agent-session-wire.ts
#	src/shared/structured-agent-session-options.ts
…o listed row names

Two runtime tests pinned the scripted CLI's unlisted init model as the saved pick.
…n list

- Record who chose a chat's model (`modelSource`: picker or caller). Only a
  picker's selection is checked against the catalog or replaced; a model a
  caller named (orchestration or agent-launch `--model`) runs as given, and
  records from before this field read as a caller's.
- Name a replacement only while the merged list is the current account-level
  listing, never a running chat's (possibly project-narrowed) list.
- Desktop and phone re-read a chat's options when the listing the host said was
  running lands, so a stopped chat shows the replacement before a send.
- A start reads the catalog only for a picker's model on an agent that replaces
  one, and the whole read is capped at the start's 3 s wait.
- One "is this model listed" rule; a live child reporting no model names the
  listed default only where the agent's seed proves it.
- The scripted CLI lists real `resolvedModel` rows; the unanswered-control test
  waits for the start's own write.
…-model-discovery-retry

# Conflicts:
#	src/main/claude/claude-structured-session-startup.ts
#	src/main/native-chat/agent-session-wire/structured-agent-session-provider-started.test.ts
#	src/main/runtime/rpc/methods/structured-agent-session-create.ts
- Rename the persisted modelSource to modelChosenBy: 'caller' | 'picker' |
  'new-chat-default'. Picks and the remembered new-chat selection are
  replaceable; a caller's or an absent value never is. Unknown stored values
  read as absent instead of setting the chat aside.
- Re-read a stopped chat's options once when a catalog answer that did not
  wait names a replacement those options still lack (desktop and phone).
- The picker view uses the shared listing rule for a seed.
- Move the child listing clock to its own file; a start's read relies on its
  caller's 3 s cap instead of a second inner wait.
- Revert the unrelated @pnpm/exe lockfile churn.
…-model-discovery-retry

# Conflicts:
#	src/main/codex/codex-background-terminals.ts
#	src/main/native-chat/agent-model-catalog/agent-model-catalog-persistence.ts
#	src/main/native-chat/agent-model-catalog/agent-model-catalog-service.ts
#	src/shared/agent-session-wire.ts
#	src/shared/structured-agent-session-options.ts
@brennanb2025 brennanb2025 changed the title fix: keep model choices usable when discovery fails fix: move a Claude model you chose to the default before a message fails once Claude stops offering it Oct 10, 2026
A new-chat seed matched only by a row's resolved model id now commits that
row's id, so the picker selects it instead of adding a raw-id duplicate.
Reword the setOption comment (any client of the public RPC counts as a
picker) and pin the predicate's live-source guard with a unit test.
@brennanb2025

Copy link
Copy Markdown
Contributor Author

Review summary (final head 9e2b81d8c2f)

What this PR fixes. When Claude stopped offering a model a chat had picked, the first message failed with "model not found". Only after that did the chat fall back to the default. Now a picked model Claude really removed is replaced before anything is sent. A model that is only missing from an old saved list is re-checked first.

Decisions made during review

  • Which models can be replaced. The default here follows the common pattern: only a selection made in the picker is replaced when it disappears.
    • That covers a pick in this chat, or the remembered new-chat pick a chat started from.
    • A model named outright (worker-start --model, agent launch --model) always runs as given, the same as on main and the same as the desktop does today. If Claude can't run it, Claude's own error shows.
    • Each chat records modelChosenBy (picker, new-chat-default or caller). It is an optional field, and old records read as caller, so they are never replaced.
  • When a replacement is named. Only from the account-level listing taken in the last 60 seconds. Never from one project's narrower list.

Review rounds 7–11. Each round was a full review of the current head, followed by a fix with a regression test that failed first.

  • r7–r8: made sure a model chosen in your Claude config, or named outright by a caller, is never replaced.

    • A stopped chat's picker now updates when the fresh listing lands, on desktop and on the phone.
    • A replacement is named only from the account-level listing.
  • r9: removed lockfile churn that had slipped in.

    • Reverted unrelated formatting.
    • Added tests for a pick made on a stopped chat and for a catalog answer that arrives late.
    • Merged the "is this model listed" checks into one rule.
  • r10: a fresh review of the head merged with main (72f1a89d6d1) found every r9 finding fixed at its cause, with three P3 polish items left.

  • r11 (9e2b81d8c2f): fixed those three:

    • A seed saved under the id an alias runs now selects that alias's listed row, instead of adding a duplicate choice.
    • A code comment was reworded.
    • A guard test was added.

    I reviewed the r11 diff myself. It has 4 files and +71/−9 lines, and adds no new behaviour beyond the row match.

Live proof (on 72f1a89d6d1; screenshots in the body). This used a desktop rig with a stand-in Claude; the real CLI never ran.

  • A chat picked "Fake Opus" at Extra high.
  • After Claude's list dropped Fake Opus, the reopened chat showed "Fake Sonnet" at High before any send.
  • The next start launched --model claude-sonnet-4-6 --effort high with no failed turn.
  • Once Fake Opus was offered again, a new chat listed it again.

Checks on 9e2b81d8c2f

  • CI: all checks passed and the rest were skipped. There were no failures, including the cross-version wire compatibility check.
  • Merge state: MERGEABLE.
  • Local tests: the 32 test files this PR adds or changes have 287 passing and 1 skipped. The phone option tests pass.
  • Typecheck: node and web are clean.
  • Real Claude CLI suite: 13/13 on the r10 heads. r11 doesn't touch the Claude CLI code.

Known gaps, not fixed here

  • No notice when the model changes. The picker simply shows the new model and effort. This matches the common pattern of swapping a stale picker selection without a message. Intended.
  • The sidebar's chat row keeps the old model id until the next run. Minor. Temporary; no follow-up filed yet.

@brennanb2025
brennanb2025 marked this pull request as ready for review October 10, 2026 09:25
@coderabbitai

coderabbitai Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 60371190-10b3-4d30-aaa2-d9ec0f4623c9


📥 Commits

Reviewing files that changed from the base of the PR and between 9e2b81d and 59adf28.



📒 Files selected for processing (11)
  • mobile/src/session/use-mobile-structured-agent-options.ts
  • src/main/native-chat/agent-model-catalog/agent-model-catalog-service.ts
  • src/main/native-chat/agent-model-catalog/agent-model-catalog-workspace-default.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-options-read.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-options-resting-catalog.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-registered-definition.test.ts
  • src/main/native-chat/subagent-tracker/subagent-tracker.test.ts
  • src/main/runtime/rpc/methods/structured-agent-session-options-read.ts
  • src/renderer/src/components/native-chat/use-structured-agent-session-option-state.ts
  • src/shared/agent-session-wire-model-catalog.ts
  • src/shared/agent-session-wire.ts


🚧 Files skipped from review as they are similar to previous changes (1)
  • src/shared/agent-session-wire.ts


Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.




📝 Walkthrough
📝 Walkthrough

Walkthrough

The change adds resolved model IDs, catalog freshness and replacement handling, and model-selection provenance. Catalog-backed replacements can affect at-rest option views and launch options for eligible selections. Claude discovery and persistence distinguish reported models from listed models. Renderer and mobile hooks reread options after catalog answers. Tests cover alias matching, listing retries, replacement behavior, startup options, and delayed catalog responses.



Priority: ➖ Normal

Merge Risk: 🔵 Low · up to 59adf

The change looks mergeable with minor awareness. Two earlier low-severity concerns remain open: a test that does not set up the aged-listing scenario it describes, and a model replacement that may overwrite the effort a user saved for the target model.

Pre-merge checks | Passed 4 | Failed 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 44.07% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 59 functions across 55 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly summarizes the primary change: replacing a Claude model selected by the user before a message fails when Claude no longer offers it.
Description check Passed The description is detailed and covers the problem, user-visible behavior, implementation, rationale, compatibility, testing, and visual proof. It does not reproduce every template heading, and the li…

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR



  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f0d9f240-8538-42ca-b9de-aa80934a1083
📥 Commits

Reviewing files that changed from the base of the PR and between f307391 and 9e2b81d.

📒 Files selected for processing (86)
  • mobile/src/session/use-mobile-host-model-catalog-upgrade.ts
  • mobile/src/session/use-mobile-structured-agent-options.stopped-replacement.test.ts
  • mobile/src/session/use-mobile-structured-agent-options.ts
  • src/main/claude/claude-model-catalog-probe.ts
  • src/main/claude/claude-sdk-model-discovery-snapshot.test.ts
  • src/main/claude/claude-stream-json-connection.test.ts
  • src/main/claude/claude-structured-agent-definition.ts
  • src/main/claude/claude-structured-applied-options.test.ts
  • src/main/claude/claude-structured-cached-model-capabilities.test.ts
  • src/main/claude/claude-structured-catalog-listing.ts
  • src/main/claude/claude-structured-effort-default-at-rest.test.ts
  • src/main/claude/claude-structured-effort-reporting.test.ts
  • src/main/claude/claude-structured-live-catalog-age.test.ts
  • src/main/claude/claude-structured-model-catalog.ts
  • src/main/claude/claude-structured-model-discovery-retry.test.ts
  • src/main/claude/claude-structured-session-adapter-turns.test.ts
  • src/main/claude/claude-structured-session-options.ts
  • src/main/claude/claude-structured-session-startup.test.ts
  • src/main/claude/claude-structured-session-startup.ts
  • src/main/claude/claude-structured-unknown-model-persistence.test.ts
  • src/main/native-chat/agent-model-catalog/agent-model-catalog-child-listing-clock.ts
  • src/main/native-chat/agent-model-catalog/agent-model-catalog-entry.ts
  • src/main/native-chat/agent-model-catalog/agent-model-catalog-persistence.ts
  • src/main/native-chat/agent-model-catalog/agent-model-catalog-required-model.test.ts
  • src/main/native-chat/agent-model-catalog/agent-model-catalog-retry.test.ts
  • src/main/native-chat/agent-model-catalog/agent-model-catalog-selected-model.ts
  • src/main/native-chat/agent-model-catalog/agent-model-catalog-selection.ts
  • src/main/native-chat/agent-model-catalog/agent-model-catalog-service.ts
  • src/main/native-chat/agent-model-catalog/agent-model-catalog-store.ts
  • src/main/native-chat/agent-model-catalog/agent-model-catalog-workspace-default.ts
  • src/main/native-chat/agent-session-wire/structured-agent-definition.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-acquisition.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-adapter-router-registry.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-attach-flow.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-attach-orchestration.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-attach.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-host.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-mutation-turn-context.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-option-models.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-options-read.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-options-resting-catalog.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-provider-started.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-registered-definition.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-start-model.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-startup-attempt.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-turns-options.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-turns.ts
  • src/main/runtime/agent-session-record-founding.ts
  • src/main/runtime/agent-session-record-options.test.ts
  • src/main/runtime/agent-session-record-options.ts
  • src/main/runtime/agent-session-record-store.ts
  • src/main/runtime/agent-session-reservation-admission.ts
  • src/main/runtime/claude-structured-startup-unanswered-control-request.test.ts
  • src/main/runtime/orca-runtime-get-structured-agent-session-create-support.ts
  • src/main/runtime/orca-runtime-structured-agent-session-create-intent.test.ts
  • src/main/runtime/rpc/methods/orchestration/worker/structured-worker-launch-seed-options.test.ts
  • src/main/runtime/rpc/methods/structured-agent-session-create-handler.ts
  • src/main/runtime/rpc/methods/structured-agent-session-create-message.test.ts
  • src/main/runtime/rpc/methods/structured-agent-session-create.ts
  • src/main/runtime/rpc/methods/structured-agent-session-options-read.test.ts
  • src/main/runtime/rpc/methods/structured-agent-session-options-read.ts
  • src/main/runtime/structured-agent-model-catalog-wiring.test.ts
  • src/main/runtime/structured-agent-model-catalog-wiring.ts
  • src/main/runtime/structured-agent-session-runtime-provider-started.test.ts
  • src/main/runtime/structured-claude-scripted-runtime-test-support.ts
  • src/renderer/src/components/native-chat/use-host-model-catalog-upgrade.ts
  • src/renderer/src/components/native-chat/use-structured-agent-session-option-state.ts
  • src/renderer/src/components/native-chat/use-structured-agent-session-options.stopped-replacement.test.tsx
  • src/renderer/src/components/native-chat/use-structured-agent-session-options.ts
  • src/shared/agent-session-legacy-handoff-lease.ts
  • src/shared/agent-session-model-fallback.test.ts
  • src/shared/agent-session-model-fallback.ts
  • src/shared/agent-session-option-catalog-types.ts
  • src/shared/agent-session-options-replacement.ts
  • src/shared/agent-session-record-stored-form.ts
  • src/shared/agent-session-record.ts
  • src/shared/agent-session-wire.ts
  • src/shared/claude-model-list-probe.test.ts
  • src/shared/claude-model-list-probe.ts
  • src/shared/commit-message-agent-spec.ts
  • src/shared/commit-message-model-parsers.ts
  • src/shared/structured-agent-session-discovered-model.ts
  • src/shared/structured-agent-session-option-view.ts
  • src/shared/structured-agent-session-options.test.ts
  • src/shared/structured-agent-session-options.ts
  • tests/e2e/structured-claude-starting-model-catalog.unit.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 6 remain after this review.

Comment on lines +91 to +106
const effort = next.valuesByModel[selected]?.effort
const effortOption = catalog.models
.find((model) => model.id === target)
?.options.find((option) => option.id === 'effort')
const carried = nearestAgentEffort(
typeof effort?.value === 'string' ? effort.value : undefined,
effortOption?.kind.type === 'select'
? effortOption.kind.choices.map((choice) => choice.value)
: []
)
if (effort && carried) {
next.valuesByModel[target] = {
...next.valuesByModel[target],
effort: { ...effort, value: carried }
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

set -eu
printf '%s\n' '--- target file ---'
nl -ba src/shared/agent-session-model-fallback.ts | sed -n '1,180p'
printf '%s\n' '--- references ---'
rg -n -F --glob '*.{ts,tsx}' -- 'verifiedListReplacementRecord' src tests 2>/dev/null || true
rg -n -F --glob '*.{ts,tsx}' -- 'nearestAgentEffort' src tests 2>/dev/null || true
rg -n -F --glob '*.{ts,tsx}' -- 'valuesByModel[target]' src tests 2>/dev/null || true
printf '%s\n' '--- replacement-related symbols ---'
rg -n -i --glob '*.{ts,tsx}' -- 'replacement|modelChosenBy|effort' src | grep -E 'agent|claude|session|model' | head -240 || true

Repository: stablyai/orca

Length of output: 42011


🏁 Script executed:

set -eu
printf '%s\n' '--- host catalog selection ---'
nl -ba src/main/native-chat/agent-model-catalog/agent-model-catalog-selection.ts | sed -n '1,180p'
printf '%s\n' '--- structured option view ---'
nl -ba src/shared/structured-agent-session-option-view.ts | sed -n '1,160p'
printf '%s\n' '--- replacement tests and catalog wiring ---'
nl -ba src/main/runtime/structured-agent-model-catalog-wiring.test.ts | sed -n '1,180p'
nl -ba src/shared/agent-session-model-fallback.test.ts | sed -n '1,220p'
printf '%s\n' '--- option state types and clone behavior ---'
nl -ba src/shared/native-chat-session-option-state.ts | sed -n '1,180p'

Repository: stablyai/orca

Length of output: 23657


Preserve an existing effort for the replacement model.

valuesByModel retains effort values per model. When the replacement model already has an effort, do not overwrite it with the source model's carried effort.

Suggested fix
-  if (effort && carried) {
+  if (effort && carried && next.valuesByModel[target]?.effort === undefined) {

The host start path already applies the same nearest-effort carry rule for its flat saved options.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const effort = next.valuesByModel[selected]?.effort
const effortOption = catalog.models
.find((model) => model.id === target)
?.options.find((option) => option.id === 'effort')
const carried = nearestAgentEffort(
typeof effort?.value === 'string' ? effort.value : undefined,
effortOption?.kind.type === 'select'
? effortOption.kind.choices.map((choice) => choice.value)
: []
)
if (effort && carried) {
next.valuesByModel[target] = {
...next.valuesByModel[target],
effort: { ...effort, value: carried }
}
}
const effort = next.valuesByModel[selected]?.effort
const effortOption = catalog.models
.find((model) => model.id === target)
?.options.find((option) => option.id === 'effort')
const carried = nearestAgentEffort(
typeof effort?.value === 'string' ? effort.value : undefined,
effortOption?.kind.type === 'select'
? effortOption.kind.choices.map((choice) => choice.value)
: []
)
if (effort && carried && next.valuesByModel[target]?.effort === undefined) {
next.valuesByModel[target] = {
...next.valuesByModel[target],
effort: { ...effort, value: carried }
}
}

Comment on lines +58 to +62
store.recordSuccess(fingerprint, 'claude', {
models: [SONNET],
origin: 'probe',
fastModeTierByModel: new Map()
})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Pass the 'discovery' source to recordSuccess. The test does not set up the list the way it says.

AgentModelCatalogStore.recordSuccess takes four arguments: (fingerprint, agent, success, source). This call passes only three, so source is undefined. agentModelCatalogEntryWithSuccess then stores the Sonnet list as the live listing, and entry.discovered stays null. As a result, isCurrent is always false, and the store skips the discovery bookkeeping (dueEntries, failures). The test says it sets up "a saved Sonnet-only list a minute old". What it actually sets up is a running-child listing with no discovery, so the age advance at Line 63 has no effect. The fastModeTierByModel field is also not part of AgentModelCatalogSuccess, both here and at Line 69. Both mismatches fail type checking if this file is part of a typechecked project.

Proposed fix
--- "a/tests/e2e/structured-claude-starting-model-catalog.unit.test.ts"
+++ "b/tests/e2e/structured-claude-starting-model-catalog.unit.test.ts"
@@ -55,11 +55,7 @@
   })
   let now = 1_000
   const store = new AgentModelCatalogStore({ now: () => now })
-  store.recordSuccess(fingerprint, 'claude', {
-    models: [SONNET],
-    origin: 'probe',
-    fastModeTierByModel: new Map()
-  })
+  store.recordSuccess(fingerprint, 'claude', { models: [SONNET], origin: 'probe' }, 'discovery')
   now += AGENT_MODEL_CATALOG_CURRENT_MS
   const relists = input.relists
   const probe = vi.fn(async () => {

At Line 69, return { models: relists, origin: 'probe' as const }.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
store.recordSuccess(fingerprint, 'claude', {
models: [SONNET],
origin: 'probe',
fastModeTierByModel: new Map()
})
store.recordSuccess(fingerprint, 'claude', { models: [SONNET], origin: 'probe' }, 'discovery')

@brennanb2025

Copy link
Copy Markdown
Contributor Author

Synced with main again (head 59adf2864e3). main moved after this PR was marked ready, and it had started to conflict.

Conflicts

  • agent-model-catalog-service.ts: main (fix(native-chat): a new chat's model and effort pickers match their first frame before the agent starts #27031) changed recordConfiguredDefault where this PR had moved it out (to agent-model-catalog-workspace-default.ts). The move is kept, with main's change applied inside the moved file. That change looks up a session's folder only for a local session.
  • structured-agent-session-options-read.ts: main now reads the chat folder's own catalog first, then the account catalog for clients that need a model. main's two reads are kept, and the stopped-chat replacement is applied to whichever answer is used.
    • I checked that this can't name a replacement from a project-narrowed list. The replacement still comes only from the account's current probe listing (origin === 'probe', current, not listing, no failure) in agent-model-catalog-selected-model.ts.

Fixed without conflict markers

  • Test signature: a test needed main's new reader argument.
  • Line limit: with both sides, src/shared/agent-session-wire.ts went one line over its 300-line limit. The model-catalog wire types moved unchanged to agent-session-wire-model-catalog.ts, which the old file re-exports, so nothing on the wire changed. The new file's name keeps it under the cross-version CI trigger.

One unrelated formatting change: the commit hook reformatted one long line from main in subagent-tracker.test.ts. It's formatting only.

Checks

  • Typecheck: node, web, cli and mobile clean.
  • Tests: 475 targeted tests, plus 57 mobile option tests.
  • CI: 18 passed, 13 skipped, 0 failed. MERGEABLE.

Live proof rerun on this head (the body's screenshots are replaced). A stopped chat whose picked Fake Opus / Extra high was dropped from a fresh listing showed Fake Sonnet / High before any send. Its next start launched --model claude-sonnet-4-6 --effort high on the same session with no failed turn. Once Fake Opus was listed again, a new chat offered it again.

@brennanb2025

Copy link
Copy Markdown
Contributor Author

Replaced by #27826, a much smaller version of this fix that decides the model at send time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant