Skip to content

fix(native-chat): settle a dead agent's leftover work in one write that always fits - #26664

Merged
brennanb2025 merged 5 commits into
mainfrom
brennanb2025/chat-settle-at-owner-change
Oct 9, 2026
Merged

brennanb2025 merged 5 commits into
mainfrom
brennanb2025/chat-settle-at-owner-change

Conversation

@brennanb2025

@brennanb2025 brennanb2025 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 18 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​1942 $\color{#cf222e}{\Huge{\mathbf{−}}}$​100 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​1842
Prod 22 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​542 $\color{#cf222e}{\Huge{\mathbf{−}}}$​342 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​200

ELI5

When a chat's agent died mid-response (Orca quit or crashed, or the agent exited), Orca has to tidy up what the agent left behind: mark the response as interrupted, stop its running tool calls, cancel its open approval, and decide what happens to messages that were waiting to be sent. On main that tidy-up was split into several separately saved pieces, each in a database row capped at 1.5 MB. A single leftover item bigger than the cap (for example a tool call whose ~1.6 MB file content had been saved) made the tidy-up fail every time it ran. The chat then stayed "Working" forever, an old approval could take over the composer, and every new message failed with "Your message was not sent." / "OMP couldn't restart." Restarting Orca didn't help.

Now the whole tidy-up is saved as one all-or-nothing database transaction, and an oversized item is saved as its own full-size row instead of being forced into a capped one. After a crash, reopening the chat shows the old response as interrupted, the old approval as Cancelled, and the next message is delivered normally.

Before / after (real app, hidden test window, fake model, chat seeded with a 1.6 MB running tool call)

Before (main): the old response is still "Working" five minutes later and the new message is refused.

before: stuck Working, message not sent

After, on reopening: the old response shows as ended, the tool call as interrupted, the approval as Cancelled, and the composer is free.

after: reopened chat settled

After, sending: the message is delivered and answered.

after: message delivered

After quitting and relaunching Orca: still settled, with no duplicate stop rows, and a third message is answered.

after: relaunch

What Changed

  • One transaction per tidy-up. The journal's existing writer plans the whole settlement (ended turns, ended tool calls, cancelled prompts, finished reasoning, and the outcomes of sends that were waiting) and saves it in one SQLite transaction. Bounded rows (1.5 MB / 200 changes) carry the ordinary items, and any item too large for one is written as its own full-body row in the same transaction. Readers are notified once, after commit. Nobody can ever see half a tidy-up, and running it again after success writes nothing, including after a reopen.
  • Codex exits. Codex's own exit cleanup now hands the whole settlement to that one transaction instead of saving it in chunks. When the settlement is too large for the Codex write queue's 16 MiB budget, the queue refuses it and the general exit handling settles everything in one commit right after. A test drives that path through the real exit flow.
  • Repeated items. When one settlement mentions the same item twice (for example two Codex questions sharing an id written by the model), the changes are applied in order across all of the settlement's rows. Main did the same within a single row.
  • Every row it builds is one the table accepts. A rewritten turn keeps every field it had, including fields a newer Orca build saved (also inside context usage). It only drops known fields whose saved values are invalid. An end time is never invented: a proven exit with no valid end time stays interrupted with no completedAt, which the schema already allows. Items are addressed by their exact saved key, so old or unusual keys are rewritten in place rather than duplicated.
  • No new UI, copy, persisted shape, or wire field.

Why

On main, the separately saved pieces could leave a partial result, and the size cap made some chats fail every retry with no way out. Truncating the big item would lose content, and catching the error would leave the chat stuck anyway. Building only rows the table accepts and saving them all together removes that whole class of failure for every caller, using the writer the journal already has. That's the common pattern: unfinished work is turned into terminal records in one transaction.

Differences from the common pattern:

  • TEMPORARY: this PR does not change when the tidy-up runs (today: on agent exit, and when a chat is opened or an agent is attached). Follow-ups will move it to the moment a new owner takes over, with opening a chat becoming read-only (feat(native-chat): keep the proof that an agent is gone after the next owner starts #26667 records which agent generations are proven gone; a later PR moves the settlement there).
  • INTENDED: the tidy-up runs only when the agent is proven to have exited. If execution can't be verified, the work stays unverifiable, because agents can outlive the app and run over SSH, and losing contact is not proof of death.

Linked Issue

Supersedes #26576: this PR includes its size calculation and full-body test cases, but not its loop that commits chunks separately. #26576 will be closed once this lands.

Testing

  • I manually tested these changes locally (hidden isolated app rig on macOS with a fake model, screenshots above; no real agent login)

  • Automated tests added/updated, or explained why not below

  • Real-SQLite tests: the whole settlement commits once or not at all (rollback on a later row leaves disk, memory, cursor and subscribers unchanged), full-size rows for oversized items, idempotent replay and reopen, repeated items within and across rows, all-oversized settlements, unusual saved keys, untimed exits, preserved unknown turn fields, the Codex exit path with a refused queue, and repeated Codex question ids.

  • 44 targeted test files, 552 tests, pass on the merged head. node / web / cli typechecks pass. orca-ci-checks reports 22/22.

  • QA (laptop, hidden window, before = main at the PR's base, after = this branch). Before: the chat stayed "Working" and both sends failed. After: settled on reopen, three messages delivered across a quit and relaunch, and the journal shows the 1.6 MB tool call rewritten in full with no partial or duplicate rows. Not exercised: Windows, Linux, a real SSH host, real Codex/Claude CLIs.

AI Disclosure

Built and reviewed with AI agents.

Review

Two review rounds plus a narrow check of the review fixes. Deferred, all pre-existing on main: a Codex turn-end settlement over 16 MiB is retried while the chat stays Working until the agent exits; the live end-of-turn cleanup rebuilds very old item keys from scratch; an oversized item body is serialized a few times while it's being sized. Merge order for the stack: this PR, #26555 and #26577 in any order, then #26667 and the move-the-settlement PR on top of it.

Agent skill upstream boundary

  • Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

The execution host still owns the journal and its tidy-up; nothing here probes a remote host locally. Persisted rows and wire messages keep their shapes, so older clients read the same rows they read today. Folder workspaces are covered by the fixtures.

Author: Brennan Benson (@BrennanKB5).

Checklist

  • This PR is small and focused
  • I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
  • Before/after screenshots or videos attached for UI changes, or N/A with reason
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
  • pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

…at always fits

A dead generation's cleanup was several separately committed writes, and a
single leftover item larger than one batch row was rejected on every try, so
the chat stayed stuck. Plan every row first and commit them in one
transaction: bounded batch rows, an oversize item as an ordinary row with its
full body, and the settlement's own send verdicts. Readers see it only after
the commit. Settlement rows are rebuilt only from schema-known fields, address
saved items by their stored key, never invent an end time, and new writes
require safe revisions, so cleanup never builds a row it cannot save.
…s a newer build saved

- Codex's own exit cleanup hands the whole settlement to the journal's one-transaction planner instead of committing chunk by chunk.
- One shared planner rejects an item named twice, on both the direct and resolved paths.
- Rewriting a turn keeps fields this build does not know, as the journal schema promises; the legacy carrier's kind still cannot override it.
- Drop the write-only revision admission and the planner skip that worked around it: insert admission is the reader's again, and a saved revision that cannot advance is skipped by the reducer while the rest settles, as before.
- Remove two dead-generation journal members settlement no longer calls.
…nd keep nested turn fields

Repeated Codex question ids in one write fold in order again instead of failing the write,
and a saved turn's context usage keeps fields a newer build stored inside it.
…-fix

# Conflicts:
#	src/main/native-chat/agent-session-journal/journal-row-writer.ts
@brennanb2025

Copy link
Copy Markdown
Contributor Author

Review summary

Problem. When a chat's agent died mid-response, Orca's tidy-up of what it left behind (the running response, tool calls, an open approval, messages waiting to be sent) was saved in several separate pieces, each in a row capped at 1.5 MB. If one leftover item was bigger than the cap, such as a tool call holding a ~1.6 MB file, the tidy-up failed on every attempt. The chat stayed "Working" forever, an old approval could take over the composer, and new messages failed with "Your message was not sent." Restarting didn't help.

What changes for the user. After a crash, reopening the chat shows the old response as interrupted and the old approval as Cancelled, and the next message is delivered. No new UI or wording.

Fixed during review

  • Repeating an item within one tidy-up dropped the second change silently. The first fix for that then made repeated Codex question ids fail the write. Repeated items now apply in order across all rows of a tidy-up, as main did within one row.
  • Codex's exit cleanup saved its tidy-up in chunks. It is now one transaction. When the Codex write queue refuses an oversized settlement, the general exit handling settles everything in one commit right after (covered by a real exit-flow test).
  • Rewriting a turn removed fields a newer Orca build had saved, including fields inside context usage. They are now kept.
  • Removed a new revision check and its workaround that only rejected corrupted saved data. Main's existing behaviour (skip that one item, apply the rest) is kept.
  • Main merged in once (one small conflict in the journal row writer, both sides kept). The three new SQLite tests were registered in the Node test project, which CI requires.

Deferred (pre-existing on main, not introduced here)

  • A Codex turn-end settlement over the 16 MiB queue budget is retried while the chat shows Working, until the agent exits.
  • The live end-of-turn cleanup rebuilds very old item keys from scratch instead of using the saved key.
  • An oversized item body is serialized a few times while it is being sized; this happens once per tidy-up.

Verified

  • Real-SQLite tests: all-or-nothing commit and rollback, full-size rows for oversized items, idempotent replay and reopen, repeated items, unusual saved keys, untimed exits, preserved unknown fields, the Codex refused-queue exit path. 44 targeted files / 552 tests pass on the merged head; node/web/cli typechecks pass; static checks 22/22.
  • Hidden-window app QA on macOS with a fake model (screenshots in the PR body). On main the chat stayed Working and both sends failed. On this branch the chat was settled on reopen, three messages were delivered across a quit and relaunch, the journal holds the 1.6 MB tool call in full, and there are no partial or duplicate rows. No internal terms appear in the chat.

Not verified: Windows, Linux, a real SSH host, real Codex/Claude/OpenCode CLIs, mobile.

Seen during QA, not caused by this PR (it doesn't touch these surfaces): Agent Session History listed the same chat twice; the chat's status read "Failed" after the crash was tidied up, then "Done" after the next reply.

Not changed here (TEMPORARY): when the tidy-up runs. It still happens on exit and when a chat is opened or an agent is attached. Moving it to the moment a new owner takes over is the follow-up stack, starting with #26667.

CI at 4a307ac: every check finished: 15 passed (all five unit test shards, relay integration, static analysis and typecheck, cross-version wire compatibility, package on macOS and Windows), and 17 were skipped by their path filters. No failures.

@brennanb2025
brennanb2025 marked this pull request as ready for review October 9, 2026 06:34
@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 3a9b0ba5-ebcf-43cc-80af-fe51acd49171
📥 Commits

Reviewing files that changed from the base of the PR and between dec18a8 and 4a307ac.

📒 Files selected for processing (41)
  • config/scripts/vitest-sqlite-runtime-files.mjs
  • src/main/acp/acp-structured-host-crash.test.ts
  • src/main/codex/codex-journal-command-turn.ts
  • src/main/codex/codex-structured-journal-exit-convergence.test.ts
  • src/main/codex/codex-structured-journal-repeated-questions.test.ts
  • src/main/codex/codex-structured-journal-settlement-rollback.test.ts
  • src/main/codex/codex-structured-journal-settlement.ts
  • src/main/codex/codex-structured-journal-sink.ts
  • src/main/codex/codex-structured-journal-translation-settlement.test.ts
  • src/main/native-chat/agent-session-journal/journal-atomic-terminal-settlement.test.ts
  • src/main/native-chat/agent-session-journal/journal-dispatch-settlement.test.ts
  • src/main/native-chat/agent-session-journal/journal-item-revision.ts
  • src/main/native-chat/agent-session-journal/journal-lifecycle-batch-appender.test.ts
  • src/main/native-chat/agent-session-journal/journal-lifecycle-batch-appender.ts
  • src/main/native-chat/agent-session-journal/journal-lifecycle-batch-partition.test.ts
  • src/main/native-chat/agent-session-journal/journal-lifecycle-batch-partition.ts
  • src/main/native-chat/agent-session-journal/journal-lifecycle-mutation.ts
  • src/main/native-chat/agent-session-journal/journal-pending-submission-recovery.ts
  • src/main/native-chat/agent-session-journal/journal-row-builders.ts
  • src/main/native-chat/agent-session-journal/journal-row-schema.test.ts
  • src/main/native-chat/agent-session-journal/journal-row-schema.ts
  • src/main/native-chat/agent-session-journal/journal-row-writer.test.ts
  • src/main/native-chat/agent-session-journal/journal-row-writer.ts
  • src/main/native-chat/agent-session-journal/journal-store-collaborators.ts
  • src/main/native-chat/agent-session-journal/journal-store-contracts.ts
  • src/main/native-chat/agent-session-journal/journal-store.ts
  • src/main/native-chat/agent-session-journal/journal-subagent-liveness.test.ts
  • src/main/native-chat/agent-session-journal/journal-terminal-settlement-validation.test.ts
  • src/main/native-chat/agent-session-timeline/provider-timeline-settlement.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-dead-generation-settlement.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-dead-generation-settlement.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-event-sink.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-exit-turn-scope.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-stale-turn-verdict.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-stale-turn-verdict.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-startup-failure-exit.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-unexpected-exit.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-unfinished-work.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-unopened-send-withdrawal.ts
  • src/shared/agent-session-turn-record.test.ts
  • src/shared/agent-session-turn-record.ts
💤 Files with no reviewable changes (1)
  • src/main/native-chat/agent-session-wire/structured-agent-session-unfinished-work.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.


📝 Walkthrough

Walkthrough

The change adds shared lifecycle mutation and item-revision helpers, updates journal batch planning and commits, and combines dispatch resolutions with lifecycle settlement. It also changes interrupted-turn handling to omit invalid completion times and filters invalid optional turn fields. New tests cover transaction rollback, retries, settlement idempotence, Codex exit convergence, prompt revisions, and persistence after reopening.

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 4a307

Dead-agent cleanup now writes its settlement in one transaction, and oversized leftover items no longer block the chat. No concrete outstanding defects were identified, so the change appears ready to merge after normal checks.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 24.62% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 65 functions across 40 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check Passed The title clearly describes the main change: settling leftover work from a dead agent in one write that fits size limits.
Description check Passed The description follows the required template and provides a clear user impact, implementation details, rationale, issue reference, visual proof, testing results, limitations, and completed checklist.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@brennanb2025
brennanb2025 merged commit ded0144 into main Oct 9, 2026
56 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant