fix(invocation): session interrupt self-heal + QueueProcessor terminal write backstop by mindfn · Pull Request #1146 · zts212653/clowder-ai

mindfn · 2026-07-13T11:39:34Z

Summary

Invocation reliability fix addressing repeated false stall-kills across multiple threads and cats.

Root cause

ProcessLivenessProbe classifies "CLI waiting for LLM API response" as idle-silent (no CPU growth, no NDJSON output) — identical to "CLI truly stuck". The 7-minute stall auto-kill threshold caused false kills during normal inference.

Evidence

thread_mrj033qnuamv9ghe (sol): 4 consecutive stall kills during active tool-call → API-wait cycles
thread_mriwjlo1955spve4 (codex): review invocation killed after 16s of work + 7m API wait
thread_mrk2p6e198arm2pj (codex): zero output — killed before first response arrived

Changes (4 files, 6 commits)

invoke-single-cat.ts: New buildStallAutoKillConfig(cliTimeoutMs) — stall threshold dynamically tracks resolved CLI_TIMEOUT_MS (default 30 min). CLI_TIMEOUT_MS=0 → stallAutoKill disabled. Custom values respected.
invoke-helpers.ts: classifyResumeFailure() routes session interrupted / already interrupted / session terminated to missing_session → session self-heal drops stale sessionId + retries fresh.
QueueProcessor.ts: F194 Z3 ensureTerminalStatus backstop in finally block (parity with messages.ts). chainTracker.release() unconditional (prevents Map leak).
invoke-single-cat.test.js + queue-processor.test.js: 11 new assertions — interrupt classification, negative regex (keyboard interrupt etc.), buildStallAutoKillConfig (default/custom/disabled), QueueProcessor backstop regression.

Test plan

118/118 invoke-single-cat tests pass
103/103 queue-processor tests pass (1 new backstop test)
TypeScript clean, Biome clean
Cross-family review: sol (P1/P2/P3 addressed) + codex (P2 addressed)

🐾 Generated with Cat Café

…eueProcessor terminal write backstop (zts212653#1145) Two reliability fixes for invocation availability: 1. classifyResumeFailure() now matches "session interrupted" / "already interrupted" / "session terminated" as 'missing_session', triggering session self-heal (drop stale sessionId + retry fresh). Previously, our stall auto-kill (SIGTERM after 7 min idle-silent) caused the CLI daemon to mark sessions "interrupted", but the classifier returned null → no self-heal → cascading resume failures on every subsequent invocation until manual intervention or session TTL expiry. 2. QueueProcessor.executeEntry() finally block now calls ensureTerminalStatus() with RouteChainCompletionTracker, matching messages.ts F194 Z3 parity. If the catch block fails to update the record to 'failed' (e.g. Redis blip), the CAS-guarded backstop ensures the invocation reaches a terminal status instead of staying 'running' and becoming a zombie after 600s grace. The previously silent `catch { /* ignore */ }` now logs a warning. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mindfn · 2026-07-13T13:08:34Z

CI failure is unrelated to this PR — flaky eval harness test:

test/harness-eval/eval-capability-wakeup-trace.test.js:330
"treats matching live preview evidence inside the window as a real browser-preview opportunity"
Expected: 'miss', Actual: 'false_positive'

This file has zero diff from upstream/main (git diff upstream/main -- test/harness-eval/eval-capability-wakeup-trace.test.js is empty)
Our changes only touch invoke-helpers.ts, QueueProcessor.ts, and invoke-single-cat.test.js — completely different modules
Main branch last 3 CI runs: all green
Root cause: test uses Date.now() for previewAvailability.observedAt (line 344), likely a time-window boundary flake

Re-running CI.

7 minutes was too aggressive for LLM inference — when CLI waits for an API response with large context, it appears idle-silent (no CPU growth, no NDJSON output) even though a legitimate network request is in flight. This caused false stall kills, which interrupted active invocations. 15 min gives ample room for long inference while still catching truly stuck CLIs well before CLI_TIMEOUT_MS (30 min). F194 zombie detection won't false-fire during this window because InvocationTracker slot is held by the live generator. Root cause of: codex sessions getting "interrupted" mid-work. [宪宪/claude-opus-4-6🐾] Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…653#1145) Existing test hardcoded 7*60_000 (420000ms). Updated to match the new 15*60_000 (900000ms) threshold from the prior commit. [宪宪/claude-opus-4-6🐾] Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…s212653#1145) The ProcessLivenessProbe cannot distinguish "CLI waiting for LLM API response" from "CLI truly stuck" — both present as idle-silent (no CPU growth, no NDJSON output). Any threshold shorter than CLI_TIMEOUT_MS causes false kills during normal inference. Evidence: thread_mrj033qnuamv9ghe (sol) and thread_mriwjlo1955spve4 (codex) both show repeated "CLI 响应超时 (420s)" kills during active tool-call → API-wait cycles. Codex review invocation was killed after only 16s of work + 7m of waiting for terra API response. Align to 30 min so CLI_TIMEOUT_MS is the single binding idle constraint. [宪宪/claude-opus-4-6🐾] Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…tracker leak, negative tests (zts212653#1145) P1: Replace hardcoded 30-min constant with buildStallAutoKillConfig() that tracks resolved CLI_TIMEOUT_MS. When CLI_TIMEOUT_MS=0 (disabled), stallAutoKill is turned off entirely. Custom values (e.g. 60 min) are respected instead of being capped at 30 min. P2: Move chainTracker.release() out of the .get() guard in QueueProcessor.ts so it always runs — prevents Map leak when InvocationRecordStoreLike lacks .get(). P3: Add negative regex tests (keyboard interrupt, process interrupted, interrupted system call) to verify the interrupt classifier doesn't match non-session interrupt messages. Tests: 118/118 pass (3 new assertions for buildStallAutoKillConfig + 3 negative regex assertions). [宪宪/claude-opus-4-6🐾] Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…st (zts212653#1145) Addresses codex review P2: the QueueProcessor terminal backstop had no test coverage. New test verifies that when the catch-block update(failed) throws (e.g. Redis blip), ensureTerminalStatus fires a CAS write with expectedStatus=running to ensure the record reaches terminal status. Also exercises the .get() path and confirms the backstop reads the record before attempting the CAS update. [宪宪/claude-opus-4-6🐾] Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

… call count The zts212653#1145 backstop test was a false green: updateCallCount===1 fired on the userMessageId backfill (call #1), not the catch-block's failed write (call #3). The throw never reached the intended code path, and get() was hardcoded to return 'running' regardless of actual state. Fix: trigger throw on `status==='failed' && !expectedStatus` (catch-block signature) instead of call count. Track record state so get() returns the realistic status after each successful update. Verify the full chain: catch-block throw → record stays running → CAS backstop → terminal failed. Addresses codex review P2 on 37ad884. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mindfn · 2026-07-14T06:49:27Z

@codex review

Fixed the P2 false-green backstop test per your guidance:

Trigger condition changed from updateCallCount === 1 && patch?.status === 'failed' to patch?.status === 'failed' && !patch?.expectedStatus — now correctly fires on the catch-block write (call fix: restore original README overwritten by sync #3), not the userMessageId backfill (call Welcome Beta Testers! Start Here #1)
Mock maintains actual record state so get() returns realistic status (stays 'running' after catch-block throw)
Added assertion: recordStatus === 'failed' verifying the full chain: catch-block throw → record stays running → CAS backstop → terminal failed

SHA: 4303a43

chatgpt-codex-connector · 2026-07-14T06:56:52Z

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 4303a43a16

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

zts212653

Maintainer review PASS on 8839569f688de5cc1b92026b640651e6a654347b.

Linked issue #1145 is accepted (triaged + bug).
Current-head CI is green.
git range-diff proves the two merge-main refreshes are patch-equivalent to the cloud-reviewed 4303a43a16.
No P1/P2 findings; Feature Anchor Guard passed.
Intake disposition: intentional no-intake/public-only hotfix, recorded after merge; the next full source sync must prove supersession before overwriting this public delta.

[小太阳·砚砚/GPT-5.6 Sol🐾]

…merged)

mindfn requested a review from zts212653 as a code owner July 13, 2026 11:39

mindfn and others added 6 commits July 14, 2026 09:26

mindfn added 2 commits July 14, 2026 14:57

Merge branch 'main' into fix/invocation-stall-and-session-recovery

3396ff0

Merge branch 'main' into fix/invocation-stall-and-session-recovery

8839569

zts212653 added the hotfix Hotfix — requires 2-week upgrade review label Jul 14, 2026

zts212653 approved these changes Jul 14, 2026

View reviewed changes

zts212653 merged commit fecbffe into zts212653:main Jul 14, 2026
5 checks passed

zts212653 mentioned this pull request Jul 14, 2026

Invocation reliability: interrupted session cascading failure + QueueProcessor missing terminal write backstop #1145

Closed

mindfn deleted the fix/invocation-stall-and-session-recovery branch July 14, 2026 14:25

zts212653 mentioned this pull request Jul 14, 2026

feat(F113): Windows drive picker + VS Code-style "D:" breadcrumb #1103

Merged

12 tasks

mindfn added a commit to mindfn/clowder-ai that referenced this pull request Jul 14, 2026

Merge upstream/main into develop_base — sync fecbffe (zts212653#1146 …

97678a9

…merged)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(invocation): session interrupt self-heal + QueueProcessor terminal write backstop#1146

fix(invocation): session interrupt self-heal + QueueProcessor terminal write backstop#1146
zts212653 merged 9 commits into
zts212653:mainfrom
mindfn:fix/invocation-stall-and-session-recovery

mindfn commented Jul 13, 2026 •

edited

Loading

Uh oh!

mindfn commented Jul 13, 2026

Uh oh!

mindfn commented Jul 14, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 14, 2026

Uh oh!

zts212653 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mindfn commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root cause

Evidence

Changes (4 files, 6 commits)

Test plan

Uh oh!

mindfn commented Jul 13, 2026

Uh oh!

mindfn commented Jul 14, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 14, 2026

Uh oh!

zts212653 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mindfn commented Jul 13, 2026 •

edited

Loading