Repository navigation
fix(orchestration): own a worker terminal from creation, not after the boot wait - #19608
Merged
Merged
Conversation
…e boot wait A worker pane is visible on desktop and phone the moment it is created, but the worker_terminal_resources row saying orchestration owns it was written only after the agent TUI went idle (up to 60s). A keystroke into the booting pane found no owned row, markWorkerTerminalUserOwned returned 0, and the takeover was dropped - so a later worker-release closed the pane under the user. Record custody on the branches that create a terminal, right after creation and before the tui-idle wait. The Dispatch capability still waits for the agent to come up. An explicit --terminal reuse is untouched: it transfers at authority. With the row present from creation, the failed-start adoption is dead. What a failed start still needs is the Dispatch-context pane identity release re-proves through, which is now copied from the custody row.
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 10 included reviews per hour; 7 remain after this review. 📝 WalkthroughWalkthroughWorker startup now records custody for newly created agent terminals before the boot wait completes. The authority validates that the dispatch is pending and the worker is starting before creating an owned terminal resource. Explicitly reused and structured terminals skip this record. Failed starts no longer resolve or adopt residual terminals. They record matching terminal identity in the failed dispatch context, and recovery derives cleanup hints from persisted ownership. Tests cover custody, takeover, release, failure, and inactive-dispatch behavior. Priority: ⬇️ Low Merge Risk: ⚪ Minimal · up to The change removes several runtime-required minimum-contrast translation entries. No concrete current-head defect or merge-blocking production impact is established. 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
Full details: Description checkExplanation The description clearly explains the behavior change, rationale, scope, and verification results. However, it omits required template information, including a Resolution Add the missing required sections and information. Include a valid Full details: Docstring CoverageExplanation Docstring coverage is 38.46% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 13 functions across 8 files. (1 skipped: 1 unsupported.) Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes
- Custody-at-creation — new
recordCreatedWorkerTerminalCustody(DB method inworker-dispatch-authority.ts+ RPC wrapper increated-worker-terminal-custody.ts) writes theownedrow right after terminal creation, inside the samepending && startingguard asprepareStartingWorkerAuthority. The capability itself still waits for the boot. - Failed-start identity backfill —
adoptFailedStartTerminalandresolveResidualAgentTerminalare deleted (net −161 lines); the survivingrecordFailedStartDispatchIdentitycopies pane identity into the failed dispatch'sdispatch_contexts(gated oncapability_hash IS NULL) soworker-releasecan re-prove identity. - Failure receipt —
failWorkerStartWithReceiptnow derivesreleasablefromownership_state === 'owned' && !isStructuredWorkerHandle(handle)instead of an explicit adoption identity;tearDownFailedWorkerStartreturnsvoid. - Tests — one new suite (
worker-terminal-custody-at-creation.test.ts) holds the boot wait open viavi.spyOn(waitForTerminal)and asserts mid-start DB state, keystroke-takeover during boot, release retention, and refusal on a stopped dispatch.
The change is well-constructed and the two invariants the author claims are both verified against surrounding code: prepareStartingWorkerAuthority's existing !getWorkerTerminalResourceByOwner(...) guard (worker-dispatch-authority.ts:109) now finds the creation-time row and skips, so no double-insert; and the isStructuredWorkerHandle gate in worker-start-receipt.ts:41 correctly suppresses recovery for structured sessions (whose owned row arrives at authority, not creation) as the deleted structuredSession ? undefined : ... branch did before. The recordFailedStartDispatchIdentity terminal-handle match relies on agent_terminal_handle being set by persistWorkerReadinessStage/persistGatedSetupSpawnFailure before any failed-start path, which holds for every created-terminal path. The reuse and unknown-outcome boundaries match the documented scope. Clean.
DeepSeek Pro (free via Pullfrog for OSS) ｜ 𝕏
Jinwoo-H
added a commit
that referenced
this pull request
Sep 8, 2026
…s possible A handle-addressed report during the worker's tui-idle wait now finds the custody row written at terminal creation, so it flips the pane to user_owned and worker-release retains it instead of closing it under the user.
Jinwoo-H
added a commit
that referenced
this pull request
Sep 8, 2026
* fix(orchestration): fence worker release on mobile keystrokes A settled worker's terminal stayed ownership_state='owned' unless a takeover was recorded, and the only recorder was orchestration.workerTerminalUserInput, which only the desktop/web xterm input signal and the native-chat composer call. Mobile input arrives as terminal.send / stream input frames instead of a report, so a phone user typing in a settled worker's pane never fenced anything: worker-list kept recommending release and worker-release closed the PTY under them. Give the host one definition of "a human typed into this terminal" and route every lane through it. The mobile input floor claim is that definition and already exists on both byte lanes: it is taken only for deliberate phone input, never for the emulator's own query replies, and never for an agent's `orca terminal send`, which names itself a desktop client and so is indistinguishable from a keystroke at this layer. Settling that claim after an accepted write now records the takeover through the same code the RPC reporter uses, throttled to one write per pane per 30s so a keystroke does not pay for an immediate transaction. The record lands on the runtime that owns both the terminal and the orchestration database, so SSH-hosted and remote workers behave exactly like local ones. No mobile change: mobile already sends client.type (mobile/src/terminal/terminal-send-request.ts:24). * fix(orchestration): ask the database, do not remember, whether a pane is fenced The keystroke throttle armed on the attempt rather than on the outcome, so a zero-row or thrown record poisoned the pane for 30s. A phone keystroke during the worker-start readiness wait lands before prepareStartingWorkerAuthority creates the owned resource; a real keystroke seconds later was then suppressed, the worker settled, and workerRelease closed the terminal under the phone user. A SQLITE_BUSY on the first write did the same, with no retry. The cache was the defect, not its arming condition. Its precondition is the set of owned resources on the pane, which changes underneath it, and any cache keyed on ownership identity would have to read the database to learn that identity -- which is the whole question. So the input lane now asks: a read using the same predicate the write uses answers "is anything still fenceable here?" without taking BEGIN IMMEDIATE, and only then is the write attempted. Ordinary typing costs a lookup instead of a write lock, a failed write is retried by the next keystroke, and a takeover writes once per ownership epoch rather than once per window, because the flip to user_owned removes the pane from the candidate set. Sharing the predicate keeps the probe from drifting from the writer. Adds the two escape cases as permanent regressions, drives the mocked send through the real RuntimeTerminalWriter, and asserts a mobile takeover lifts the settled-worker resume fence, which no test covered. * refactor(orchestration): let the database dedupe the takeover, drop the read probe The probe was meant to keep keystrokes off BEGIN IMMEDIATE, so it had to earn that with a number. Measured against a real WAL database it costs more than the write it avoids: at 25 live workers the probe is 0.19ms and the no-op write is 0.10ms, because the probe runs the same candidate selection with each statement taking its own read snapshot instead of sharing the transaction's. It is a compensating mechanism with negative value, so it goes, along with the database method and the predicate extraction it needed. owned -> user_owned is one-way and scoped to a resource, so the database is already the dedupe: every deliberate human write attempts the transition, the second attempt matches no row, and the fence sweep runs only on changed > 0. Nothing is remembered between keystrokes, so no state can outlive the ownership it described -- a keystroke before the worker's authority attaches, a write the database refuses, and a re-dispatch onto the same pane all resolve against the rows as they are at that instant. An attempt costs about 0.1ms at typical fleet size and 0.34ms at 100 live workers, on mobile writes only. Replaces the write-count test, which asserted the old mechanism, with the invariant: many keystrokes settle into one takeover and one fence sweep. Adds the re-dispatch case, where a pane's next worker is fenced on its own merits. * refactor(terminal): name the provenance rule the takeover fence hangs off The fence rode the mobile input floor claim, with only a comment tying the two together. The floor is arbitration -- who may write next -- while the fence needs provenance -- who produced the bytes. They agree today, so anyone reweighing the floor would have moved the fence without noticing. isDeliberateHumanInput states the provenance rule on its own terms, and both byte lanes decide with it when they open a write: the claim carries the verdict beside the handle, and settlement records the takeover only when a human produced the bytes. No behavior change -- afterWrite is wired only where the predicate already answers true -- and the rule is now pinned by its own cases, so a future arbitration change has to answer this question again rather than inherit it. * test(orchestration): prove the unary lane classifies a metadata-less phone A phone build older than client.type is recognised only by its pane's mobile driver, which the unary lane passes as the provenance evidence. Nothing proved it did: replacing that argument with false left all 17 tests green while a shipped phone silently stopped fencing worker release. The new case drives a clientless send on a mobile-driven pane and fails under that mutation. The stream lane now passes false outright. Its isMobile is read off the same client object it carries, so the metadata-less phone cannot reach it, and passing the flag suggested a legacy path that does not exist there. Also states what the per-keystroke cost scales with. A pane owning no resource misses the pane_key index and falls through to a scan of owned resources, so the figure is tens of microseconds at realistic worker counts rather than a flat 0.1ms, and it grows with rows that are never released. * fix(terminal): let provenance alone decide the takeover, on every accepted write A phone older than client.type sends no client metadata, and both stream initializers derive isMobile from that metadata alone, so such a subscription reported false and took the stream lane's uninstrumented branch: provenance was computed and then never consumed. Bytes from a real person landed through both frame adapters and the resource stayed owned, so workerRelease closed the PTY under them. The unary lane already fenced that population off the pane's mobile driver, which is the host's standing reading of clientless input, so the two byte lanes disagreed at the destructive boundary. The predicate was still subordinate to floor plumbing: it could only be consulted where a floor client id existed. Now the accepted-write callback attaches on both lanes regardless of whether a floor was reserved, and humanInput alone decides recording; a write holding no claim commits nothing. Arbitration keeps its own condition around reserveWrite, where it belongs, and the unary lane's duplicate outer provenance filter is gone. The stream lane reads clientless provenance from the pane's driver, the same policy the unary lane uses. The claim holder is now TerminalInputWrite, carrying the verdict beside an optional floorClaim, so the structure says what the doc said: a write may fence without holding the floor. Regressions drive both real frame adapters, clientless direct delivery, and the paired-web desktop negative. Metadata-only provenance fails 3 on the stream lane and 1 on the unary lane; gating the callback on a reservation fails the same 3. * fix(runtime): resolve retained handles before mobile input provenance A renderer reload clears transient handles while retaining runtime-owned PTY identities. Legacy mobile provenance saw no leaf, then sendTerminal restored the same handle and delivered an unfenced key. Normalize through getLivePtyForHandle at the shared live-leaf resolver entry so classification and writes agree, preserving existing leaf generation/incarnation checks. Caller audit: - terminal-send-method: driver, query-reply authority, lock and floor checks now resolve the retained PTY before sending. - terminal-input-delivery: legacy mobile classification and exact-PTY binding now see the same target as the writer; equality checks remain. - terminal-multiplex-subscribe-resolution: retained PTYs resolve directly without a spurious missing-terminal wait. - terminal-lifecycle-methods resize and terminal-viewport-methods display mode, restore-fit and updateViewport retain their original PTY target. - inspectTerminalProcess: avoids false terminal_gone after reload while preserving provider inspection and incarnation fences. - getLivePaneKeyForTerminalHandle and getOrchestrationDispatchAuthority: unaffected because both already call getLivePtyForHandle first. No wire/schema changes, host fallback, process-death inference, or Git workspace assumptions; SSH providers keep ownership of execution evidence. Validation: - Unmodified round-3 reviewer probe: reproduced 2/2 failures, then 2/2 pass. - Unmodified round-2 reviewer probes: 13/13 pass. - Checked-in takeover suites: 24/24 pass. Removing only the resolver call fails both new reload cases; source restored afterward. - RPC orchestration + terminal, aggregate runtime handle registry, handle incarnation, mobile tab mount, stale geometry, and reload probe: 2027 passed, 1 skipped (89 files). - tc:node and check:code-quality:changed pass; background launch enabled. * test(rpc): require unconditional terminal afterWrite callbacks Update exact sendTerminal expectations for the round-2 accepted-write contract. Preserve beforeWrite expectations, absence of reserveWrite, byte payloads and call-count checks; require afterWrite to be a function. Reproduced the requested two-file run: 5 failed, 31 passed. The full RPC suite exposed the same stale shape in ACK budget/overflow, desktop resize (including its later retry), and agent-prompt fallback assertions. Update those too, for 11 assertions across six test files. No production changes. Validation: ORCA_BACKGROUND_LAUNCH=1 full src/main/runtime/rpc suite: 264 files passed; 2292 tests passed, 1 skipped. Changed-code quality and staged oxlint/React Doctor/oxfmt checks passed. Ran lint-staged --no-stash manually to honor checkout safety rather than its default backup hook. * fix(mobile): report worker takeover outside terminal byte delivery New phones announce accepted real user input through the existing worker report RPC, addressed by terminal handle. Share a per-client/per-handle 30-second gate with one bounded retry; report through the same RPC client as the input. Cover live commits and dictation via their shared sender, accessory keys, gestures, buffered submit, paste and accepted native chat. Query replies, attachment heals, triage and diff-review sends do not report. Phones predating this build do not fence release. Remove byte provenance and takeover callbacks from host delivery. Restore both lanes' pre-PR floor-claim plumbing and the original options assertions. Keep the host recorder uncached with its conditional resume-fence sweep. No DB schema or stream change; terminal is an optional report address. Retain the shared resolver recovery independently of takeover: the new SSH inspection test fails without it during renderer reload. Other callers still benefit for subscription, resize, viewport and exact-PTY binding; unary driver/lock checks see the retained PTY. Pane routing and dispatch authority already recover through getLivePtyForHandle and are unaffected. Existing leaf generation checks and first-PTY adoption remain unchanged. No other input-plumbing hunk is retained relative to the PR base. Replace byte-takeover tests with handle-addressed local/SSH report and unknown-handle tests, plus real unary/stream writes asserting zero SQL prepare/exec calls. Mobile send-site integration covers reports, exclusions, rejected writes and gate counts. Desktop report tests are unchanged. Register replacement coverage in the settled-worker release manifest. Validation (all background): host/RPC/runtime 3541 passed, 2 skipped; mobile session/terminal 2045 passed; node and mobile typechecks, changed quality, mobile oxlint, reliability manifest and max-lines ratchet passed. All five requested mutations fail assertions; resolver revert also fails independent inspection. Staged checks run manually with --no-stash. Final src diff against PR base: 5 files, +165/-13 (previously +839/-85). * fix(runtime): allow the takeover report from mobile-scoped tokens The mobile RPC allow-list gates every phone request before dispatch and the reporter swallows a refusal, so without this entry every phone shipped unfenced. Pin it beside the report tests, and pin the once-per-takeover fence sweep the replaced byte-lane suite used to assert. * fix(mobile): a no-op takeover report does not arm the gate; Stop reports too A key during worker startup reports before the resource is owned; caching that zero-change reply for 30 s suppressed the report that would have fenced the worker once it attached. Native-chat Stop is deliberate input and now reports on an accepted Escape. * fix(mobile): takeover gate ignores the host answer, like desktop Reopening the gate on a zero-change reply made every accepted key on an ordinary terminal an RPC plus a host write transaction (round 6: 100 for 100). The startup window it closed is unreachable: the agent has no prompt to accept input until after its resource row exists. Plain terminals now pay one report per 30 s window; the native-chat Stop report stays. Send-site fixture answers the report RPC with a changed count; the draft test filters to terminal.send calls. * docs(runtime): say why resolveLiveLeafForHandle re-links before lookup * chore(i18n): regenerate the runtime-required catalog for the contrast floor strings * test(orchestration): give the stopping-worker guard fixtures a Run * test(orchestration): drop fence-sweep assertions retired by the settled-worker policy * test(orchestration): pin the mid-boot phone takeover that #19608 makes possible A handle-addressed report during the worker's tui-idle wait now finds the custody row written at terminal creation, so it flips the pane to user_owned and worker-release retains it instead of closing it under the user.
brennanb2025
pushed a commit
that referenced
this pull request
Sep 8, 2026
…absent The merge conflict was one call, and it was a design question rather than a text clash: main's #19608 records worker-terminal custody at CREATION, so tearDownFailedWorkerStart no longer resolves a residual terminal and takes only the dispatch id and the structured session. This branch had changed where that session comes from, because a structured worker's worktree is now created with no startup agent. The resolution keeps both: main's custody-era signature, this branch's `placed`-sourced session. The catch block reads `placed?.` on purpose — a failure before placement has no session to discard. Separately, a review found that awaitStructuredWorkerSetupGate flattened two different outcomes into the same silent null. A wait that cannot be TAKEN is an absent precondition and needs no receipt. A wait that was taken and then THREW is a lost observation, and it was being discarded with no trace — the same class of dishonest receipt this branch exists to remove. It now records a `wait_unevaluated` effect carrying the error, so the start's residual resources say the gate went unevaluated and why. The outcome is deliberately unchanged: it still returns no verdict and the start still proceeds, because a worker start must not fail on missing evidence. Only the silence is fixed. The same review flagged that a created worktree can survive a later start failure. That is pre-existing and already reported: worktree creation pushes a `kind: 'worktree'` effect and the failure receipt carries residualResources, so the worktree is surfaced for recovery rather than leaked silently. Teardown has never removed worktrees, before or after this branch, so it is not a regression here and is left alone.
7 of 9 tasks
This was referenced Sep 12, 2026
8 of 9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What the user notices
Before: a worker terminal is visible on desktop and phone the moment it is created, but the row saying "orchestration owns this pane" is only written after the agent's TUI goes idle (up to 60 s). A keystroke into the booting pane is accepted by the PTY, the host's takeover write finds no
ownedrow and returns 0, and the client's 30 s report gate is armed anyway. A real takeover keystroke inside the rest of that window is suppressed, so a laterworker-releasecloses the pane under the user. Desktop and phone reporters share this shape.After: the custody row exists from creation, so a keystroke during boot flips it to
user_ownedlike any other keystroke. No client, wire, or typing-path change.The change
Only the custody insert moves.
prepareStartingWorkerAuthoritystill runs after the boot wait and still writes the capability, assignee, and mailbox fence; a capability must not be handed to a process that has not come up. Its existing "row already exists" guard now finds the row and skips.The new
recordCreatedWorkerTerminalCustodyruns right after the terminal is created, inside the samedispatch pending && worker startingguard authority uses, so a dispatch stopped mid-creation gets no row.What this deletes
adoptFailedStartTerminalandresolveResidualAgentTerminalexisted only to backfill this row when a start died before authority. With the row written at creation they are gone (net −161 lines). The one half that is still needed, copying the pane identity into the dispatch context so release can re-prove it on a failed start, survives asrecordFailedStartDispatchIdentity.Not changed, and why
--terminalreuse: the caller's own pane; custody transfers at authority as today.Verification
tc:node,tc:cliclean; changed-code quality and react-doctor gates 0 findings; max-lines ratchet OK.src/main/runtimesuite: 716 files, 7360 tests pass. Orchestration + rpc + runtime-tests + cli + e2e-unit: 358 files, 2997 tests pass.--terminalreuse (5).Follow-up to #19337, which is held until this lands.