Skip to content

fix(host): own agent processes and preserve session setup - #90

Open
fergusean wants to merge 1 commit into
leepokai:devfrom
fergusean:fergusean/agent-resource-lifecycle
Open

fergusean wants to merge 1 commit into
leepokai:devfrom
fergusean:fergusean/agent-resource-lifecycle

Conversation

@fergusean

@fergusean fergusean commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Agent cleanup now retains ownership through leader exit and actor drop. Unix keeps the leader unreaped with waitid(WNOWAIT) until its group is terminated; Windows assigns the suspended shell to a KILL_ON_JOB_CLOSE job before resuming it. Intentional shutdown preserves dev's ten-second stdin-close grace period so CLI login refreshes can finish.

Abandoned ACP answers remove their pending requests. Interrupted allocations remain marked uncertain until a session identity is confirmed, and session snapshots retain first-prompt profiles and thread introductions across setup failure or restart. Fast mode and idle closes have a separate request path. Confirmed new/fork identities are recorded before fallible model setup so retry uses the same conversation.

Reconstructed on current dev (acd8998a), retaining its thirty-minute idle policy, background-descendant guard and fresh-session fallback. The previous per-process cap, recycling policy, cap-specific recovery code and 125-second timer test are removed. All clients keep their existing API and notice handling; there are no client UI or version changes.

Review follow-up

  • Missing main or thread transcripts recover with a fresh session and divider, including Claude's actual Resource not found response.
  • Agents without loadSession can keep multiple threads and start a new main session without a session cap. Fork-only support remains.
  • Session selection and idle release come from dev; there is no HashMap-based capacity eviction.
  • A returned load failure uses dev's fresh-session fallback without first retiring the agent.
  • Forking does not reload or replace main as part of a capacity reservation. Parent context retains dev's existing fork path.
  • The obsolete default-suite 125-second timer test is dropped. Fast mode changes never mark an allocation uncertain.

Validation

  • macOS: cargo fmt --check; all-target Clippy with -D warnings; 294 binary tests passed, two existing tests ignored; ten focused actor end-to-end checks and five upstream turn/Fast mode checks passed.
  • Ownership checks cover unexpected leader exit with stdout held by a descendant, actor drop/panic, pending-answer cancellation and graceful child completion. Actor checks cover resumable/nonresumable interrupted allocation, Fast mode, setup retries, pending Prelude after another lane/restart, fork-only support, multiple nonresumable threads and Claude-style missing-session recovery.
  • Exact rebased source d3f5a50f: real local host with Claude Code / claude-agent-acp 0.85.0: exact replies in main and thread; graceful host restart and same-ID continuation in both; deliberately nonexistent saved IDs recover main and thread with divider notices. Existing user bots, credentials and saved conversations were untouched.
  • Exact rebased source d3f5a50f: real local host with released codex-acp 2.2.2 passes main and independent thread prompts, graceful host restart with same-ID continuation in both, and deliberately missing saved IDs recovering fresh with divider notices in both. All six replies match exactly. The first fork reply takes 4.754 seconds; the first prompt follows fork directly without an intervening load.
  • The earlier Codex timeout is resolved as an obsolete probe dependency: codex-acp 2.1.1 unsubscribed newly forked native threads, so Codex completed internally while ACP emitted no reply chunks or prompt completion. This reproduces on the same host binary; released 2.2.2 retains the subscription through upstream's merged fix. Updating only the probe adapter makes all six checks pass. No host workaround, timeout extension, assertion change or product source change was needed.
  • Nonvisual Markdown link/fence checks and Git whitespace checks passed. Host CI is green on Linux, macOS and Windows, including native-memory checks (run 38899877091). Installed-app acceptance is left to integration validation.

What's New

zh-Hant

en-US

@leepokai leepokai left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the deep work here! The process-ownership half is excellent and I'd like to land it, but the lifecycle policy needs changes first.

Great, ready to merge on their own

  • Process tree ownership: waitid(WNOWAIT) keeping the leader unreaped until the group is killed, and the suspended launch + KILL_ON_JOB_CLOSE job on Windows. Much more reliable than taskkill /T.
  • Answer's Drop removing abandoned pending requests.
  • allocation_uncertain and persisting the first-prompt Prelude in the snapshot.

Needs changes

  1. Idle retirement kills user-started background processes. Retiring after IDLE_GRACE (2 min) SIGKILLs the whole process group, including anything the agent launched in the background, e.g. a dev server Claude started with a background Bash. retire_if_idle only looks at host-side state (turn, queue, approvals, requests), not at work the agent still has running. Users will see their servers vanish two minutes after the reply.
  2. Two minutes means a cold start on most phone messages. Gaps over 2 min are the norm in chat, so nearly every message would pay launch + session/load replay (your own Cursor probe measured ~11–12 s). Please make it much longer (e.g. 15–30 min) or pressure-based (only under memory pressure / over the live-session bound).
  3. A failed session/load now leaves the conversation stuck. Previously a failed load started a fresh session with a divider notice; now it returns "retry to resume it". When the saved session can't be loaded (bot cwd changed, which keeps the main session; session file deleted; adapter changed), retrying never succeeds. Main can recover through New session, but threads have no reset path, so a thread errors forever. Same for agents without loadSession after a host restart or crash: they used to start fresh automatically and now error on every message. Please keep the fresh-session fallback (with the notice) for unrecoverable load failures.

Minor

  • Forking now also requires sessions.load, so fork-only adapters lose the parent context for threads.
  • fork_thread loads the main session twice; the first load can be released immediately by make_session_room.

Suggestion: split into (a) process ownership + pending-answer cleanup, which I'm happy to merge as-is, and (b) the lifecycle policy, revised per the points above. The CI failure was the MIN_HOST floor test on dev; that's fixed now (dev includes 2.12.0), so a rebase should go green.

@fergusean
fergusean force-pushed the fergusean/agent-resource-lifecycle branch from 86020e0 to a998e1b Compare October 12, 2026 20:12
@fergusean fergusean changed the title fix(host): bound idle agents and live session resources fix(host): own agent processes and bound live sessions Oct 12, 2026
@fergusean
fergusean force-pushed the fergusean/agent-resource-lifecycle branch 2 times, most recently from 4edd205 to 9326c09 Compare October 13, 2026 09:29
@fergusean
fergusean marked this pull request as ready for review October 13, 2026 10:33
@fergusean
fergusean requested a review from leepokai October 13, 2026 10:33
@fergusean
fergusean force-pushed the fergusean/agent-resource-lifecycle branch 3 times, most recently from 1230571 to 280dc5d Compare October 15, 2026 19:31

@leepokai leepokai left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the continued work on this, @fergusean. The process-ownership half is still excellent, and I'd like to land it. The lifecycle half has changed shape since my last review, and dev has moved on underneath it, so here's where things stand.

dev now releases idle agents (f5d6350, host/src/agent/bot/lifecycle.rs). A session unused for 30 minutes is closed through session/close when the agent advertises it. Once all of a bot's sessions are idle, its agent stops with its process group, and the next message starts it again and loads the session. Nothing is released mid-turn or with a queue, approval or request waiting. Agents that can't loadSession keep everything. While the agent has a descendant in a process group of its own, such as a dev server from Claude Code's run_in_background, the agent stays. That replaces the per-process live-session cap here, so please drop lifecycle.rs's cap/recycling from this PR.

What I'd merge now, as its own PR: the process ownership. That's waitid(WNOWAIT) keeping the leader unreaped, the suspended launch plus KILL_ON_JOB_CLOSE job on Windows, Answer's Drop, allocation_uncertain and the persisted Prelude. Splitting it out would let it land quickly.

Problems in the current head's lifecycle and recovery changes, if you keep any of them:

  1. Claude's missing session is never recovered. When the transcript is gone, claude-agent-acp answers session/load with -32002 "Resource not found: ". unavailable_saved_session matches neither the code nor the message (every phrase contains "session"), so main and threads stay stuck on "retry to resume it" forever. dev currently starts a fresh session with a divider on any failed load.
  2. The cap breaks agents without loadSession/session/close. With main plus one thread live, New session is refused and a second thread (or any routine run) fails with "reached its live session limit" until the host restarts (lifecycle.rs:66-70, :138).
  3. The session to release is effectively random. It's the first key of a HashMap (lifecycle.rs:75), so a routine run can close main's session and, without session/close, kill the whole process group.
  4. A failed session/load kills the whole agent first (session_setup.rs:124), taking the live main session and its tools with it, even when closing the one session would do.
  5. Forking from a parent that can't be reloaded silently replaces main's session. The "starting a fresh one" divider then lands in the thread, not in main.
  6. agent_lifecycle_e2e.rs:319 sleeps 125 s in the default suite to prove a timer that no longer exists. Please drop or #[ignore] it. The pre-commit hook now stops host tests after 5 minutes.

Rebase notes: besides the textual conflicts (Cargo.toml, bot/{mod,session,turn}.rs, tests/common/mod.rs), dev's Fast mode switches the live session through session_request. With this PR's allocation_uncertain, that would make every Fast mode switch recycle the agent, so it needs its own request path.

For the next round, please include a real-host run with Claude Code (and Codex if you can) under Validation, covering the paths you change. The fixtures don't reproduce the real adapters' errors, which is where 1–3 come from.

@fergusean
fergusean force-pushed the fergusean/agent-resource-lifecycle branch from 280dc5d to 93fe2fd Compare October 16, 2026 12:14
@fergusean fergusean changed the title fix(host): own agent processes and bound live sessions fix(host): own agent processes and preserve session setup Oct 16, 2026
Keep the upstream thirty-minute idle policy and fresh-session recovery.
Own process trees through leader exit, actor drop and graceful shutdown,
remove abandoned pending answers, and retain initial instructions across
interrupted session setup.

Co-authored-by: Codex <codex@openai.com>
@fergusean
fergusean force-pushed the fergusean/agent-resource-lifecycle branch from 93fe2fd to d3f5a50 Compare October 16, 2026 12:20
@fergusean

fergusean commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review, @leepokai. I've narrowed the PR at d3f5a50f to process ownership, abandoned Answer cleanup, interrupted-allocation tracking and persisted first-prompt instructions, rebased onto current dev (acd8998a).

  • Dropped the session cap, recycling/eviction policy and obsolete 125-second timer test. The PR retains dev's thirty-minute idle policy and background-descendant guard, plus the ten-second shutdown grace for CLI login refreshes.
  • Retained dev's fresh-session fallback and divider for failed loads, including Claude's actual Resource not found response in main and threads. A returned load error no longer triggers the removed agent-retirement path.
  • Kept fork-only support and removed the capacity-reservation path that reloaded/replaced main. Nonresumable agents can retain multiple threads and start a new main session without the removed cap.
  • Fast mode and idle-close requests use the non-allocation request path, so they do not mark allocations uncertain.

Validation: CI is green on Linux, macOS and Windows, including native-memory checks: run 38899877091. On this exact head, real Claude main/thread prompts, graceful restart, same-ID continuation and missing-transcript recovery all pass. Real Codex main, restart, same-ID continuation and missing-main recovery also pass.

Full Codex thread acceptance remains incomplete. An earlier-build probe reached its 180-second harness deadline waiting for a thread reply in Codync. Checking its retained native Codex transcript shows the requested reply and turn completion within six seconds, while Codync's store has no thread reply. That establishes a delivery/completion gap, but not its precise adapter/host cause; the complete thread scenario has not been rerun on this head. The PR description now makes that scope explicit.

Marking as draft until that's resolved.

@fergusean
fergusean marked this pull request as draft October 16, 2026 14:16
@fergusean

Copy link
Copy Markdown
Contributor Author

The Codex thread blocker is resolved, @leepokai.

The same d3f5a50f host reproduces the failure with the probe's pinned codex-acp 2.1.1: the adapter unsubscribes the newly forked native thread, so Codex completes internally but ACP delivers neither reply chunks nor prompt completion. Released 2.2.2 includes upstream's subscription fix.

Updating only the probe adapter to 2.2.2 makes all six real-host checks pass on this exact PR head:

  • Main and independent thread prompts return the exact expected replies.
  • Both resume with the same session IDs after graceful host restart.
  • Missing main and thread session IDs both recover fresh with divider notices and exact replies.

The initial fork reply completes in 4.754 seconds, with no intervening session/load. The host binary and existing 180-second cutoff are unchanged; no Codync workaround or product source change was needed. These are isolated real-host/adapter checks; the additional installed-app walkthrough remains pending.

The PR description records the results, and Linux, macOS and Windows CI remain green. Thanks again for the review. Marking this ready for another look.

@fergusean
fergusean marked this pull request as ready for review October 16, 2026 15:00
@leepokai leepokai added P1: high Important bug or blocked contributor work. Fix in the next release. bug Something isn't working labels Oct 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working P1: high Important bug or blocked contributor work. Fix in the next release.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants