fix(coding-agent): failed workers recover on touch; roster gaps answer a structured recovering error - #2047
Closed
snimu wants to merge 7 commits into
Closed
fix(coding-agent): failed workers recover on touch; roster gaps answer a structured recovering error#2047snimu wants to merge 7 commits into
snimu wants to merge 7 commits into
Conversation
…r roster gaps with a structured recovering error The recovery ladder parked a live-but-silent worker lifecycle='failed' after its deferred probe rounds ran out, and nothing but the manual retry_worker command ever exited that state: create reuse hard-rejected failed workers, and only client-owned workers auto-recovered on attach. A worker frozen longer than the probe budget (dark wake, paused VM, heavy swap-in) was written off permanently even though its process identity was verified current. Failed is no longer terminal for an identity-verified live worker: attach, create reuse, and retry_worker all share one retryWorkerRecovery path, and the client-owned attach special case becomes the general descriptor-matched pre-pass. Workers whose identity is gone or unverifiable keep the old failure answer. Session addressability gets the same single-truth treatment: findWorker resolved selectors only through the in-memory roster, which is hydrated lazily from live workers after adoption, so every supervisor replacement re-opened a window where known sessions answered "Unknown active session". On a roster miss the supervisor now consults the persisted worker descriptors it already holds and answers with a structured, retryable session_recovering error (new DaemonErrorInfo variant, schema revision 27). Failed workers deliberately keep the unknown answer so first-party clients fall back to the create path, which reclaims or retries them; main.ts and the agents view treat the recovering error like that fallback.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 7f94df0. Configure here.
…ng to explicit attach Round-1 review fixes on the failed-worker recovery PR: - canRetryFailedWorker now owns the stop-marker exclusion: a user-stopped worker (intentionalStop or persisted stopRequestedAt) is never revived by attach or create reuse; only the explicit retry_worker command clears a stop. The in-memory stop-count guard alone did not survive a supervisor restart, where the markers arrive from disk. - The generalized attach pre-pass carries the owner-only payload (telemetry gate, launchEnv adoption, recovery context) inside the owned branch again; shared workers get only the descriptor lookup and the failed-retry gate. - An explicit --attach-agent on a recovering session now surfaces the typed retryable error instead of exiting with "No active agent found": the get_state lookup failure classification moves into an exported resolveActiveSessionLookupFailure, throwing for session_recovering and keeping undefined (saved-session fallback) for unknown.
…r per touch; active id on the recovering wire error Fresh-eyes review fixes on the failed-worker recovery PR: - A parked worker keeps its roster row (parking marks rows failed), so findWorker matched it and forwardToWorker threw the untyped lifecycle error before any attach could retry recovery. The --attach-agent preflight's get_state died there, making the on-touch recovery unreachable for the primary first-party path. forwardToWorker now runs the same canRetryFailedWorker/retryWorkerRecovery owner as attach and create, completing the promised attach/prompt/create touch surface. - One touch runs at most one recovery ladder: the post-match attach retry now skips the descriptor worker the pre-pass already retried, instead of resetting counters and running the ladder twice when recovery exhausted and re-parked the worker. - The session_recovering wire error now always carries the worker's rootActiveSessionId; a rootSessionId selector no longer mislabels a stable session id as an active-session id.
…ecovering The descriptor fallback in findWorker matched full ids only, while roster matching also resolves unambiguous hex session-id suffixes; a suffix-addressed root in a roster gap therefore answered Unknown instead of the retryable session_recovering. The fallback now applies the same matchesSessionIdSuffix rule to both descriptor ids, exact matches first; ambiguous suffixes stay unknown.
…rker recovery When one forwarded command started retryWorkerRecovery, the lifecycle left 'failed' and a concurrent forwardToWorker on the same worker skipped the retry gate straight into requireAvailableWorkerClient, throwing 'Session worker is recovering' mid-ladder for a recovery that was about to succeed. Join the existing worker.recovery promise instead — the same rule create reuse already follows — preserving one ladder per worker. The attach path deliberately keeps its fast typed failure: the client reconnect loop owns those retries.
Shared retryableWorkerFixture/retrySupervisor builders replace repeated scaffolding across the six recovery pins, and multi-line comment blocks collapse to one-line invariant guards. No behavior or coverage change.
Contributor
sethkarten
added a commit
that referenced
this pull request
Sep 7, 2026
) * refactor(coding-agent): move the semantic-edge ledger onto the event-log substrate The recorder's private append/replay/repair IO is deleted; EventLog owns it, the same move #1987 made for the RLM spawn ledger. One durability rule is unified in the substrate rather than dropped: an unterminated final line is an uncommitted append, skipped on read and truncated before the next append — never newline-completed and never surfaced to a consumer whose next append destroys it. * fix(coding-agent): make the explicit ledger reader's ENOENT contract atomic readSemanticEdgeLedger probed with statSync before reading through EventLog, which swallows ENOENT; a ledger deleted between the two returned [] instead of throwing. The missing-file decision now lives at the single open (replaySync missingFileThrows), so no check-then-read window exists. * docs(coding-agent): state the event-log tail rule once The unterminated-tail contract was restated four times (module doc, replaySync doc, two test comments). It now lives once in the module doc; the method doc keeps only its own parse/missing-file semantics and the test comments reference the contract. * fix(coding-agent): write event-log appends fully and gate appends on tail repair writeSync may write short (ENOSPC after a prefix); appendSync now loops until the payload is fully on disk so write-before-action callers never act on a torn record reported as success. A tail-repair failure (e.g. append-only ACL permitting O_APPEND but not r+) now propagates instead of being swallowed: writing through an unrepaired torn tail would weld it to the new record as permanent interior corruption. ENOENT and the concurrent-writer instability path keep their existing semantics. * fix(coding-agent): reclaim short event-log writes instead of completing them The rlm spawn ledger is multi-writer by documented design (supervisor plus each worker over one file), so completing a short O_APPEND write with a second write could interleave with a rival append and weld two records. A short write now truncates its own torn prefix back off (only while this writer still owns the tail) and fails the append; a torn tail is read-tolerated, a weld is permanent corruption. The append fd opens a+ so the ownership check can read the tail. * fix(coding-agent): leave the torn tail on a short write instead of reclaiming it The tail-match reclaim could truncate a rival's committed record whose final bytes coincide with our torn prefix - committed-data loss, strictly worse than the torn tail it prevented. A short write now just fails the append: the torn tail is the one tolerated shape, skipped on read and truncated by any writer's next repair (verified for both topologies: a resumed single-writer recorder repairs on its first append; every rlm-ledger writer repairs before each append). * refactor(coding-agent): compress event-log comments * fix(ai): omit the default service tier, reprice cache writes from message_delta, repoint the zai default Incorporates #2032 at f82c7fa. * fix(tui,coding-agent): survive lone surrogates in table cells and terminate the WebP EXIF scan Incorporates #2033 at a3d1139. * fix(coding-agent): restart dead kernels on ensure() and read mcp>=2 tool schemas Incorporates #2034 at 749e216. * fix: one crash-safe owner for durable state writes Incorporates #2035 at f0f02d2. * fix(coding-agent): one zombie-aware process-liveness probe Incorporates #2041 at 92a0eac. * fix(coding-agent): snapshot transfer ids from the materialized cursor; mismatches settle the transfer, not the worker channel Incorporates #2044 at 5af3bbe. * fix(coding-agent): failed workers recover on touch; roster gaps answer a structured recovering error Incorporates #2047 at 77b747a. * fix(coding-agent): seven session and IO correctness defects Incorporates #2037 at 41b5d72. * fix(coding-agent): coalesce child-usage attribution and gate agent-status persistence on real changes Incorporates #2050 at 6b0af5d. * fix(coding-agent): incremental single-flight session metadata scans Incorporates #2043 at df032c1. * fix(coding-agent): memoize the passive RLM topology derivation Incorporates #2051 at 0ee114c. * fix(coding-agent): preserve accounting and metadata across deferred updates Keep durable child-usage aggregates separate from pending sibling usage. Retry optional topology metadata after transient reads. Completes #2050 and #2051 integration. * fix: preserve session accounting and read-only persistence boundaries --------- Co-authored-by: Seth <seth@primeintellect.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Part of the worker-state single-truth program (Linear RES-1270); squashes the confirmed residuals of discussions #1742 and #1650 and the mechanism behind #1870/#1641 (and the #1904d reconnect gap).
Purpose
Two owners for two worker-state facts:
MAX_DEFERRED_RECOVERY_ROUNDSthe recovery ladder parks a live-but-silent workerlifecycle='failed'terminally: no timer re-probes it,reuseWorkerForCreatehard-rejects it, and only client-owned workers auto-recover on attach. A worker frozen longer than the ~2-3 min probe budget (dark wake, paused VM, swap-in) stays written off even thoughisProcessAlive+processStartIdprove it is ours and alive ([Bug] Sleeping the machine permanently orphans daemon sessions: a live worker is marked failed and never re-checked #1742 residual). The create path threw exactly in the gap where neitherretry_workernor a launchEnv-bearing fresh create applied ([Bug] Prime Agent 0.8.0: saved session becomes permanently unopenable after failed worker recovery #1650 residual).findWorkerresolves selectors only through the in-memory roster, hydrated lazily from live workers after adoption; the persisted worker descriptors that carryrootActiveSessionId/rootSessionIdwere consulted only in the client-owned attach special case. Every supervisor replacement re-opens a window where attach/prompt/kill on a provably-known session answerUnknown active session, and clients cannot distinguish gone from recovering (v0.8.1: supervisor restart loses live worker active-session mapping during RLM fan-out #1870, [Bug] v0.8.0 supervisor restarts make every TUI fail to reattach with “Unknown active session” #1641).Change
canRetryFailedWorker(failed + no stop in flight + process identity verified current) andretryWorkerRecovery(the state resetretry_workeralready performed) become the single retry semantics.retry_worker, attach, and create reuse all share them; the parked state is now exit-able on any touch. Identity gone/replaced/unverifiable keeps the old hard answers.findWorkeron a roster miss consults descriptors: a root session on a recovery-candidate worker answers with a structured, retryablesession_recoveringerror (DaemonSessionRecoveringError, newDaemonErrorInfovariant,DAEMON_SCHEMA_REVISION27). Failed workers deliberately keepUnknown active sessionso first-party clients take the saved-session/create fallback, which reclaims or retries them (that fallback is what self-heals identity-gone workers vialaunchEnv).main.ts(get_state lookup) and the agents view treatsession_recoveringlike the unknown fallback, converging on the create path.Wire classification: backward-compatible. The new variant rides the existing optional
errorInfo; old clients see a readable message, old daemons never send it, nothing is capability-gated because it degrades to a plain error.Net src LOC: +102/-38, of which the supervisor is +84/-38 (about half is the owned-attach special case becoming the general pre-pass, plus the two shared helpers replacing three inline copies of the reset).
Tests
session_recovering(typed, withactiveSessionId); a failed worker and a truly unknown selector keepUnknown active session(fail-unfixed verified: removing the fallback yields Unknown).daemon-errors.test.ts: serialize/deserialize round-trip for new clients, readable message for old clients, and plain old-daemon failures do not misclassify.Ran locally: daemon-supervisor-{monitor,process,eviction,admission,input-pause}, daemon-agent-roster, daemon-supervisor-lazy-subagents, daemon-errors, regressions 4656 + 4603, agent-connection-daemon, daemon-ps, agents-view suite, main-interactive-routing — 540 tests pass;
npm run checkgreen.Note
Medium Risk
Touches core daemon supervisor routing, attach/create, and worker lifecycle; behavior changes for failed-but-live workers and session lookup errors, though guarded by process identity and extensive new tests.
Overview
Fixes daemon sessions getting stuck when a worker is parked
failedbut its process is still alive, and when the supervisor knows a session id but cannot route to it yet.Failed worker recovery: Shared
canRetryFailedWorker/retryWorkerRecoveryreplace duplicated reset logic.retry_worker, session create reuse, attach, and any command forwarded to the worker now automatically retry recovery when failure is not terminal (current process identity, no stop in flight). User-stopped workers stay stopped until an explicitretry_worker.Addressability: When roster lookup misses but a persisted descriptor matches a recovering (non-failed) worker,
findWorkerreturnsDaemonSessionRecoveringErrorwith wireerrorInfo.code: session_recovering(daemon schema 27). Failed workers still answerUnknown active sessionso clients can fall through to create/reclaim.resolveActiveSessionLookupFailureand the agents view treat recovering like unknown for the saved-session reopen path, while explicit attach can surface the typed retryable error.Reviewed by Cursor Bugbot for commit 77b747a. Bugbot is set up for automated code reviews on this repo. Configure here.
LOC
Total src: +148/−52 (net +96); tests: +260/−1 (net +259).
Note
Retry failed daemon workers on touch and add
DaemonSessionRecoveringErrorto protocol revision 27retryWorkerRecoveryhelper that clears stop/archive markers and waits for recoveryDaemonSessionRecoveringErrorwith an active session identifier, serialized over the wire as a structured error variant; clients deserialize it distinctly from unknown-session failures so they no longer treat a known recovering session as absentfindWorkerresolves descriptor-known recovering root sessions to the typed error instead of unknown-session, preserving the failed-session create fallbackresolveActiveSessionLookupFailurein main.ts classifies structured recovering failures as thrown errors while keeping the saved-session fallback for unknown sessions;agents-view.openAgentsViewSessionfalls back to the saved session on recovering attach failuresdeserializeDaemonErrorsupport see only the plain message and cannot distinguish recovering sessions. Stop-marked or identity-invalid failed workers are excluded from retry bycanRetryFailedWorker— reviewers should confirm no lifecycle path clears stop markers unintentionallyMacroscope summarized 77b747a.