Skip to content

fix(coding-agent): failed workers recover on touch; roster gaps answer a structured recovering error - #2047

Closed
snimu wants to merge 7 commits into
mainfrom
sebastian/worker-state-recovery
Closed

snimu wants to merge 7 commits into
mainfrom
sebastian/worker-state-recovery

Conversation

@snimu

@snimu snimu commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Part of the worker-state single-truth program (Linear RES-1270); squashes the confirmed residuals of discussions #1742 and #1650 and the mechanism behind #1870/#1641 (and the #1904d reconnect gap).

Purpose

Two owners for two worker-state facts:

  1. Liveness/failure. After MAX_DEFERRED_RECOVERY_ROUNDS the recovery ladder parks a live-but-silent worker lifecycle='failed' terminally: no timer re-probes it, reuseWorkerForCreate hard-rejects it, and only client-owned workers auto-recover on attach. A worker frozen longer than the ~2-3 min probe budget (dark wake, paused VM, swap-in) stays written off even though isProcessAlive + processStartId prove it is ours and alive ([Bug] Sleeping the machine permanently orphans daemon sessions: a live worker is marked failed and never re-checked #1742 residual). The create path threw exactly in the gap where neither retry_worker nor a launchEnv-bearing fresh create applied ([Bug] Prime Agent 0.8.0: saved session becomes permanently unopenable after failed worker recovery #1650 residual).
  2. Addressability. findWorker resolves selectors only through the in-memory roster, hydrated lazily from live workers after adoption; the persisted worker descriptors that carry rootActiveSessionId/rootSessionId were consulted only in the client-owned attach special case. Every supervisor replacement re-opens a window where attach/prompt/kill on a provably-known session answer Unknown active session, and clients cannot distinguish gone from recovering (v0.8.1: supervisor restart loses live worker active-session mapping during RLM fan-out #1870, [Bug] v0.8.0 supervisor restarts make every TUI fail to reattach with “Unknown active session” #1641).

Change

  • canRetryFailedWorker (failed + no stop in flight + process identity verified current) and retryWorkerRecovery (the state reset retry_worker already performed) become the single retry semantics. retry_worker, attach, and create reuse all share them; the parked state is now exit-able on any touch. Identity gone/replaced/unverifiable keeps the old hard answers.
  • The client-owned attach pre-pass generalizes to any descriptor-matched worker (owned keeps its launchEnv/recoveryConfig relaunch semantics; non-owned recovers only when identity-current-failed, and never blocks on an in-flight recovery). A child-session attach onto a failed worker retries after roster resolution.
  • findWorker on a roster miss consults descriptors: a root session on a recovery-candidate worker answers with a structured, retryable session_recovering error (DaemonSessionRecoveringError, new DaemonErrorInfo variant, DAEMON_SCHEMA_REVISION 27). Failed workers deliberately keep Unknown active session so first-party clients take the saved-session/create fallback, which reclaims or retries them (that fallback is what self-heals identity-gone workers via launchEnv).
  • main.ts (get_state lookup) and the agents view treat session_recovering like the unknown fallback, converging on the create path.

Wire classification: backward-compatible. The new variant rides the existing optional errorInfo; old clients see a readable message, old daemons never send it, nothing is capability-gated because it degrades to a plain error.

Net src LOC: +102/-38, of which the supervisor is +84/-38 (about half is the owned-attach special case becoming the general pre-pass, plus the two shared helpers replacing three inline copies of the reset).

Tests

  • New pin: create reuse on a failed worker with a current process identity retries recovery instead of throwing (fail-unfixed verified: forcing the old terminal semantics rejects with "could not be safely reclaimed").
  • New pin: a descriptor-known unaddressable root session answers session_recovering (typed, with activeSessionId); a failed worker and a truly unknown selector keep Unknown active session (fail-unfixed verified: removing the fallback yields Unknown).
  • Both-direction wire pins in daemon-errors.test.ts: serialize/deserialize round-trip for new clients, readable message for old clients, and plain old-daemon failures do not misclassify.
  • Existing pins kept green: retry-while-stopping rejection, intentional-stop tombstone reset, failed-unreclaimable create rejection (identity gone), owned-session recovery paths.

Ran locally: daemon-supervisor-{monitor,process,eviction,admission,input-pause}, daemon-agent-roster, daemon-supervisor-lazy-subagents, daemon-errors, regressions 4656 + 4603, agent-connection-daemon, daemon-ps, agents-view suite, main-interactive-routing — 540 tests pass; npm run check green.


Note

Medium Risk
Touches core daemon supervisor routing, attach/create, and worker lifecycle; behavior changes for failed-but-live workers and session lookup errors, though guarded by process identity and extensive new tests.

Overview
Fixes daemon sessions getting stuck when a worker is parked failed but its process is still alive, and when the supervisor knows a session id but cannot route to it yet.

Failed worker recovery: Shared canRetryFailedWorker / retryWorkerRecovery replace duplicated reset logic. retry_worker, session create reuse, attach, and any command forwarded to the worker now automatically retry recovery when failure is not terminal (current process identity, no stop in flight). User-stopped workers stay stopped until an explicit retry_worker.

Addressability: When roster lookup misses but a persisted descriptor matches a recovering (non-failed) worker, findWorker returns DaemonSessionRecoveringError with wire errorInfo.code: session_recovering (daemon schema 27). Failed workers still answer Unknown active session so clients can fall through to create/reclaim. resolveActiveSessionLookupFailure and the agents view treat recovering like unknown for the saved-session reopen path, while explicit attach can surface the typed retryable error.

Reviewed by Cursor Bugbot for commit 77b747a. Bugbot is set up for automated code reviews on this repo. Configure here.

LOC

Total src: +148/−52 (net +96); tests: +260/−1 (net +259).

Note

Retry failed daemon workers on touch and add DaemonSessionRecoveringError to protocol revision 27

  • Failed workers whose recorded process identity is still current are now retried on attach, create, retry, and command forwarding via a shared retryWorkerRecovery helper that clears stop/archive markers and waits for recovery
  • Adds a typed DaemonSessionRecoveringError with an active session identifier, serialized over the wire as a structured error variant; clients deserialize it distinctly from unknown-session failures so they no longer treat a known recovering session as absent
  • findWorker resolves descriptor-known recovering root sessions to the typed error instead of unknown-session, preserving the failed-session create fallback
  • resolveActiveSessionLookupFailure in main.ts classifies structured recovering failures as thrown errors while keeping the saved-session fallback for unknown sessions; agents-view.openAgentsViewSession falls back to the saved session on recovering attach failures
  • Risk: daemon protocol schema advances to revision 27; older clients without deserializeDaemonError support see only the plain message and cannot distinguish recovering sessions. Stop-marked or identity-invalid failed workers are excluded from retry by canRetryFailedWorker — reviewers should confirm no lifecycle path clears stop markers unintentionally

Macroscope summarized 77b747a.

…r roster gaps with a structured recovering error

The recovery ladder parked a live-but-silent worker lifecycle='failed'
after its deferred probe rounds ran out, and nothing but the manual
retry_worker command ever exited that state: create reuse hard-rejected
failed workers, and only client-owned workers auto-recovered on attach.
A worker frozen longer than the probe budget (dark wake, paused VM,
heavy swap-in) was written off permanently even though its process
identity was verified current.

Failed is no longer terminal for an identity-verified live worker:
attach, create reuse, and retry_worker all share one retryWorkerRecovery
path, and the client-owned attach special case becomes the general
descriptor-matched pre-pass. Workers whose identity is gone or
unverifiable keep the old failure answer.

Session addressability gets the same single-truth treatment: findWorker
resolved selectors only through the in-memory roster, which is hydrated
lazily from live workers after adoption, so every supervisor replacement
re-opened a window where known sessions answered "Unknown active
session". On a roster miss the supervisor now consults the persisted
worker descriptors it already holds and answers with a structured,
retryable session_recovering error (new DaemonErrorInfo variant, schema
revision 27). Failed workers deliberately keep the unknown answer so
first-party clients fall back to the create path, which reclaims or
retries them; main.ts and the agents view treat the recovering error
like that fallback.
Comment thread packages/coding-agent/src/main.ts Outdated
Comment thread packages/coding-agent/src/modes/daemon/daemon-supervisor.ts Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 7f94df0. Configure here.

Comment thread packages/coding-agent/src/modes/daemon/daemon-supervisor.ts Outdated
…ng to explicit attach

Round-1 review fixes on the failed-worker recovery PR:

- canRetryFailedWorker now owns the stop-marker exclusion: a user-stopped
  worker (intentionalStop or persisted stopRequestedAt) is never revived
  by attach or create reuse; only the explicit retry_worker command
  clears a stop. The in-memory stop-count guard alone did not survive a
  supervisor restart, where the markers arrive from disk.
- The generalized attach pre-pass carries the owner-only payload
  (telemetry gate, launchEnv adoption, recovery context) inside the
  owned branch again; shared workers get only the descriptor lookup and
  the failed-retry gate.
- An explicit --attach-agent on a recovering session now surfaces the
  typed retryable error instead of exiting with "No active agent found":
  the get_state lookup failure classification moves into an exported
  resolveActiveSessionLookupFailure, throwing for session_recovering and
  keeping undefined (saved-session fallback) for unknown.
…r per touch; active id on the recovering wire error

Fresh-eyes review fixes on the failed-worker recovery PR:

- A parked worker keeps its roster row (parking marks rows failed), so
  findWorker matched it and forwardToWorker threw the untyped lifecycle
  error before any attach could retry recovery. The --attach-agent
  preflight's get_state died there, making the on-touch recovery
  unreachable for the primary first-party path. forwardToWorker now runs
  the same canRetryFailedWorker/retryWorkerRecovery owner as attach and
  create, completing the promised attach/prompt/create touch surface.
- One touch runs at most one recovery ladder: the post-match attach
  retry now skips the descriptor worker the pre-pass already retried,
  instead of resetting counters and running the ladder twice when
  recovery exhausted and re-parked the worker.
- The session_recovering wire error now always carries the worker's
  rootActiveSessionId; a rootSessionId selector no longer mislabels a
  stable session id as an active-session id.
Comment thread packages/coding-agent/src/modes/daemon/daemon-supervisor.ts Outdated
…ecovering

The descriptor fallback in findWorker matched full ids only, while roster
matching also resolves unambiguous hex session-id suffixes; a
suffix-addressed root in a roster gap therefore answered Unknown instead
of the retryable session_recovering. The fallback now applies the same
matchesSessionIdSuffix rule to both descriptor ids, exact matches first;
ambiguous suffixes stay unknown.
Comment thread packages/coding-agent/src/modes/daemon/daemon-supervisor.ts
…rker recovery

When one forwarded command started retryWorkerRecovery, the lifecycle
left 'failed' and a concurrent forwardToWorker on the same worker
skipped the retry gate straight into requireAvailableWorkerClient,
throwing 'Session worker is recovering' mid-ladder for a recovery that
was about to succeed. Join the existing worker.recovery promise instead
— the same rule create reuse already follows — preserving one ladder per
worker. The attach path deliberately keeps its fast typed failure: the
client reconnect loop owns those retries.
Shared retryableWorkerFixture/retrySupervisor builders replace repeated
scaffolding across the six recovery pins, and multi-line comment blocks
collapse to one-line invariant guards. No behavior or coverage change.
sethkarten added a commit that referenced this pull request Sep 7, 2026
@sethkarten

Copy link
Copy Markdown
Contributor

Included in #2028: #2028

@sethkarten sethkarten closed this Sep 7, 2026
sethkarten added a commit that referenced this pull request Sep 7, 2026
)

* refactor(coding-agent): move the semantic-edge ledger onto the event-log substrate

The recorder's private append/replay/repair IO is deleted; EventLog owns it, the same move #1987 made for the RLM spawn ledger. One durability rule is unified in the substrate rather than dropped: an unterminated final line is an uncommitted append, skipped on read and truncated before the next append — never newline-completed and never surfaced to a consumer whose next append destroys it.

* fix(coding-agent): make the explicit ledger reader's ENOENT contract atomic

readSemanticEdgeLedger probed with statSync before reading through EventLog, which swallows ENOENT; a ledger deleted between the two returned [] instead of throwing. The missing-file decision now lives at the single open (replaySync missingFileThrows), so no check-then-read window exists.

* docs(coding-agent): state the event-log tail rule once

The unterminated-tail contract was restated four times (module doc, replaySync doc, two test comments). It now lives once in the module doc; the method doc keeps only its own parse/missing-file semantics and the test comments reference the contract.

* fix(coding-agent): write event-log appends fully and gate appends on tail repair

writeSync may write short (ENOSPC after a prefix); appendSync now loops until the payload is fully on disk so write-before-action callers never act on a torn record reported as success. A tail-repair failure (e.g. append-only ACL permitting O_APPEND but not r+) now propagates instead of being swallowed: writing through an unrepaired torn tail would weld it to the new record as permanent interior corruption. ENOENT and the concurrent-writer instability path keep their existing semantics.

* fix(coding-agent): reclaim short event-log writes instead of completing them

The rlm spawn ledger is multi-writer by documented design (supervisor plus each worker over one file), so completing a short O_APPEND write with a second write could interleave with a rival append and weld two records. A short write now truncates its own torn prefix back off (only while this writer still owns the tail) and fails the append; a torn tail is read-tolerated, a weld is permanent corruption. The append fd opens a+ so the ownership check can read the tail.

* fix(coding-agent): leave the torn tail on a short write instead of reclaiming it

The tail-match reclaim could truncate a rival's committed record whose final bytes coincide with our torn prefix - committed-data loss, strictly worse than the torn tail it prevented. A short write now just fails the append: the torn tail is the one tolerated shape, skipped on read and truncated by any writer's next repair (verified for both topologies: a resumed single-writer recorder repairs on its first append; every rlm-ledger writer repairs before each append).

* refactor(coding-agent): compress event-log comments

* fix(ai): omit the default service tier, reprice cache writes from message_delta, repoint the zai default

Incorporates #2032 at f82c7fa.

* fix(tui,coding-agent): survive lone surrogates in table cells and terminate the WebP EXIF scan

Incorporates #2033 at a3d1139.

* fix(coding-agent): restart dead kernels on ensure() and read mcp>=2 tool schemas

Incorporates #2034 at 749e216.

* fix: one crash-safe owner for durable state writes

Incorporates #2035 at f0f02d2.

* fix(coding-agent): one zombie-aware process-liveness probe

Incorporates #2041 at 92a0eac.

* fix(coding-agent): snapshot transfer ids from the materialized cursor; mismatches settle the transfer, not the worker channel

Incorporates #2044 at 5af3bbe.

* fix(coding-agent): failed workers recover on touch; roster gaps answer a structured recovering error

Incorporates #2047 at 77b747a.

* fix(coding-agent): seven session and IO correctness defects

Incorporates #2037 at 41b5d72.

* fix(coding-agent): coalesce child-usage attribution and gate agent-status persistence on real changes

Incorporates #2050 at 6b0af5d.

* fix(coding-agent): incremental single-flight session metadata scans

Incorporates #2043 at df032c1.

* fix(coding-agent): memoize the passive RLM topology derivation

Incorporates #2051 at 0ee114c.

* fix(coding-agent): preserve accounting and metadata across deferred updates

Keep durable child-usage aggregates separate from pending sibling usage. Retry optional topology metadata after transient reads. Completes #2050 and #2051 integration.

* fix: preserve session accounting and read-only persistence boundaries

---------

Co-authored-by: Seth <seth@primeintellect.ai>
@kevinjosethomas
kevinjosethomas deleted the sebastian/worker-state-recovery branch September 8, 2026 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants