Skip to content

fix(kernelforge): run opportunity analysis on resumable Codex without global hooks - #1617

Merged
xiaofei-zheng merged 9 commits into
mainfrom
feature/yunkai/codex-opportunity-hooks
Sep 23, 2026
Merged

xiaofei-zheng merged 9 commits into
mainfrom
feature/yunkai/codex-opportunity-hooks

Conversation

@BaoYunkai

@BaoYunkai BaoYunkai commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

Problem

Forge Rewrite opportunity analysis (OpportunityAnalysisAgent) requires the implementer backend to enforce staging contracts while the agent investigates handoff evidence and writes operator drafts.

  • Claude declares stop_hooks and runs AgentRunSpec.hooks in-session (_AnalysisToolGuard: PreToolUse write confinement, Stop blocking while rejection.json stands).
  • Codex is resumable but does not execute Forge hooks. Codex-only controller runs used to fail in OpportunityAnalysisAgent.__init__ with “requires a provider with tool hooks”.

Fix

  • Gate: accept backends with stop_hooks or resumable.
  • Claude / hook-capable providers: attach _AnalysisToolGuard hooks on AgentRunSpec (unchanged).
  • Codex / hookless resumable providers: after the main session returns, _resume_for_refused_staging_drafts drives the same Stop refusal loop outside the provider:
    • Call publish_complete_staged_tasks(..., quiescent_sec=0.0) before each refusal check so a corrected draft clears stale rejection.json.
    • While _on_stop blocks, backend.resume with the refusal text, capped by _MAX_STOP_DENIALS and the same analysis deadline as the primary session (remaining budget passed into each resume; stop if the deadline is exhausted mid-recovery).
    • Record AgentProviderError (and other resume failures) as ANALYSIS_STATUS_FAILED instead of silently completing.
  • Not in this PR: global Codex stop_hooks, CODEX_HOME/config.toml command hooks, or a shared hook bridge — those would affect every Codex caller (implementer, lanes, fusion, applyback).

Write confinement on Codex continues to rely on existing sandbox / workspace_guard / protected_paths.

Test plan

  • pytest src/kernelforge/tests/kernel_rewrite_controller/test_opportunity_agent.py (hookless resume republication, deadline cap, resume failure reporting, hooks-or-resumable gate)
  • pytest src/kernelforge/tests/test_lane_session_guarantees.py (Codex lane advisory for missing stop_hooks)
  • ruff check + ruff format --check on touched files

…ooks

Codex sessions need stop_hooks and pre-tool guards for Rewrite opportunity
analysis. Write CODEX_HOME/config.toml command hooks that delegate to
codex_hook_runner, enable stop_hooks on the Codex provider, and align tests
with the Claude hook contract.
@BaoYunkai
BaoYunkai requested a review from a team as a code owner September 23, 2026 03:56
@BaoYunkai

Copy link
Copy Markdown
Collaborator Author

What it does: Enables Forge Rewrite opportunity analysis on Codex by writing CODEX_HOME managed PreToolUse/Stop command hooks, materializing guard state before each Codex session, and declaring stop_hooks on the Codex provider.

Blocking issues: 1

  1. [C2:codex-read-outside-staging] PreToolUse denies reads outside staging [verified]
    Problem: handle_pre_tool_use treats any tool with a file_path/path outside staging_root as a deny. Claude's _AnalysisToolGuard only applies that check to Edit|Write|MultiEdit|NotebookEdit; Read|Grep use _cap_investigation_result and may read handoff evidence via additional_directories. Codex opportunity runs must read hot-kernel source trees outside staging.
    path: src/kernelforge/agent_backends/codex_hook_runner.py:59-87
    Impact: Legitimate Read/Grep (or Codex equivalents with a path) against repo or handoff paths fail with "may only write task.json and driver.py under the supplied staging directory", blocking opportunity analysis even though stop_hooks now pass the gate check.
    Action: Mirror Claude matchers — cap Read|Grep via updatedInput limits; restrict staging writes to write-like tools only; do not deny investigation reads outside staging.

Checked: codex_hook_runner.py, codex_managed_hooks.py, codex.py, opportunity_agent.py (_AnalysisToolGuard), registry.py, test_codex_managed_hooks.py | Ran: pytest test_codex_managed_hooks.py and related registry/lane tests (passed locally) | SKIPPED: full CI matrix

Apply staging write checks only to write-like tools, cap read/grep like
Claude's _AnalysisToolGuard, and allow investigation reads outside staging.
@BaoYunkai

Copy link
Copy Markdown
Collaborator Author

Addressed the blocking review note (codex-read-outside-staging).

  • handle_pre_tool_use now mirrors _AnalysisToolGuard: shell/task/agent tools denied when hooks are active; read/grep capped via the same _MAX_READ_LINES / _MAX_GREP_MATCHES helpers; staging path checks apply only to write-like tools (apply_patch, edit, write, …).
  • Added tests for read/grep outside staging (allowed + capped) while keeping the write-outside-staging deny case.

Pushed on feature/yunkai/codex-opportunity-hooks.

@ZhengGong-amd

Copy link
Copy Markdown
Collaborator

PR #1617 -- please re-scope

What it does: flips Codex to stop_hooks=True and writes CODEX_HOME/config.toml command hooks that run a hand-copied _AnalysisToolGuard, so opportunity analysis stops rejecting Codex.

Blocking issues: 4

  1. [free:hooks-never-execute] The hooks never run [verified]
    Problem: shlex.quote(_hook_runner_argv()) quotes python -m module as one word, so the command exits 127 (not found). codex-cli 0.144.4 hooks/list also reports both hooks as source=user, isManaged=false, trustStatus=untrusted. codex_managed_hooks.py:45
    Impact: Codex claims stop_hooks but no guard runs. A failing hook allows the tool call instead of denying it.
  2. [C1/D1] stop_hooks=True gives every hooked caller the opportunity-analysis guard [verified]
    Problem: the runner ignores the spec.hooks callbacks. agent.py:649 turns off the outer gate loop for Codex, so gate._on_stop never runs. If the hooks did run, deny_shell_tools would deny Bash to the implementer, analysis, lanes, fusion and applyback sessions. cli.py:580 drops the lane warning. codex.py:372, registry.py:427, codex_managed_hooks.py:39
    Impact: Codex implementer sessions lose their canonical Stop gate, and the operator sees no warning.
  3. [R5] config.toml is overwritten and outlives the session [verified]
    Problem: the file is overwritten with no merge when FORGE_AGENT_OPTIONS_JSON sets home. CODEX_HOME belongs to the backend instance, so the hooks=None summarizer resume (agent.py:846-851) runs under the previous session's hooks. codex_managed_hooks.py:69
    Impact: the operator's Codex config is destroyed, and later sessions get hooks they never asked for.
  4. [X1/X2/T1] The description and tests do not match the behaviour [verified]
    Problem: the body calls these hooks "managed" and does not mention the lost outer gate, the dropped lane warning or the config overwrite. The tests never execute the generated command and never go through CodexBackend._execute. All 80 pass with the 127 command.

Action: re-scope rather than patch. Codex genuinely has no in-session hooks. agent.py:600-649 already handles that case with an outer loop.

  • Revert stop_hooks=True in codex.py and registry.py, and delete codex_managed_hooks.py and codex_hook_runner.py.
  • In OpportunityAnalysisAgent, accept stop_hooks or resumable. On the resume path, after each session call pending_rejections(staging) and backend.resume with the same refusal text, up to the existing _MAX_STOP_DENIALS. This mirrors uses_outer_gate.
  • Confine writes with what Codex already declares (sandbox, workspace_guard, protected_paths), not a hook. Keep _AnalysisToolGuard for providers that do run hooks.
  • A general Codex hook bridge (callbacks run through host IPC, trust records, per-session CODEX_HOME, tool-name mapping) is a separate PR if other callers ever need it.

Checked: codex.py, codex_managed_hooks.py, codex_hook_runner.py, registry.py, opportunity_agent.py, orchestrator/agent.py, cli.py, insession_gate.py, analysis.py, fusion/author.py, applyback.py, config.py | Ran: generated command via sh -c; codex-cli 0.144.4 app-server hooks/list; pytest on the 3 changed test files (80 passed) | Base: 09d1ff5 | Head: 947c408 | SKIPPED: independent second reader

@jiaqiang-dot-liu

Copy link
Copy Markdown
Collaborator

PR #1617 -- feat(kernelforge): materialize Forge hooks as Codex managed command hooks

What it does: Forge Rewrite opportunity analysis refuses any provider that does not declare stop_hooks, which left Codex-only sessions failing before the first turn. The PR declares stop_hooks=True for Codex and, before each session, writes a CODEX_HOME/config.toml with PreToolUse and Stop command hooks plus a forge_guard_state.json, both delegating to a new out-of-process CLI (codex_hook_runner) that re-implements the opportunity-analysis guard.

Blocking issues: 4

  1. [free:hooks-never-execute] The generated hook command cannot be executed [verified]
    Problem: runner = shlex.quote(_hook_runner_argv()) quotes "<sys.executable> -m kernelforge.agent_backends.codex_hook_runner" as a single token, so the command written into config.toml has that whole string as argv[0]. src/kernelforge/agent_backends/codex_managed_hooks.py:45
    Impact: no hook ever runs. Codex now declares stop_hooks while running no guard at all, and a hook that fails to spawn is fail-open, so the tool call proceeds. Every assertion the capability flag underwrites is false at runtime.
    Action: quote the interpreter and the module separately (shlex.quote(sys.executable) + -m <module>), and add a test that actually spawns the generated command with a JSON payload on stdin and asserts the decision it prints.

  2. [C1] stop_hooks=True rewires every Codex caller, not just opportunity analysis [verified]
    Problem: five call sites build AgentRunSpec.hooks (opportunity_agent.py:503, orchestrator/agent.py:607, orchestrator/analysis.py:1541, fusion/author.py:1189, rewrite_by_flydsl/applyback.py:360). materialize_codex_managed_hooks ignores their callbacks and always installs the opportunity guard, keyed only on bool(spec.hooks.pre_tool_use) and on spec.cwd as the staging root. src/kernelforge/agent_backends/codex_managed_hooks.py:34,39; src/kernelforge/agent_backends/registry.py:427
    Impact: orchestrator/agent.py:649 now evaluates uses_outer_gate to False for Codex, so the implementer loses its canonical Stop gate and gets nothing in its place; cli.py:582 stops warning the operator that a Codex lane runs no hooks. If the command in finding 1 were fixed, implementer, fusion and applyback sessions would additionally have shell tools denied and writes confined to spec.cwd.
    Action: either honour spec.hooks (map each group's matchers and callbacks) or keep stop_hooks=False for Codex and let OpportunityAnalysisAgent accept stop_hooks or resumable, driving pending_rejections through backend.resume the way uses_outer_gate already does.

  3. [R5] config.toml is overwritten and outlives the session that asked for it [verified]
    Problem: the file is written wholesale with no merge. _child_environment honours runtime.options["home"], so an operator-supplied CODEX_HOME has its config.toml replaced; and because self._codex_home is per backend instance, a later hooks=None session on the same backend -- the summarizer resume at orchestrator/agent.py:851 -- reuses the directory while _materialize_managed_hooks returns early without removing the previous config. src/kernelforge/agent_backends/codex_managed_hooks.py:69; src/kernelforge/agent_backends/codex.py:726-728
    Impact: an operator's Codex configuration (provider, auth, approvals) is destroyed by a Forge run, and sessions that explicitly asked for no hooks inherit the previous session's guard.
    Action: write the hooks into a session-scoped file the backend owns and removes, or merge into the existing config.toml and restore it afterwards; clear the hook stanza on the spec.hooks is None path instead of returning early.

  4. [X1] The description and the tests do not cover what the diff changes [verified]
    Problem: the body describes the hook materialization only. It does not mention that Codex implementer sessions lose the outer Stop gate, that the lanes warning disappears, or that CODEX_HOME/config.toml is overwritten. src/kernelforge/tests/test_codex_managed_hooks.py:29-41 asserts substrings in the generated config and calls handle_pre_tool_use / handle_stop directly; nothing drives CodexBackend._execute or executes the command.
    Impact: the suite is green while the feature does not run, which is how finding 1 reached the current head.
    Action: state the three operator-visible effects in the body, and add a test at CodexBackend._execute level plus one that spawns the generated command.

Note on CI: test (shard 1/6) fails on both 3.10 and 3.11 with test_optimize_loop_walkthrough.py::test_both_arms_dry_walks_the_rest_of_the_chain - assert 'KERNEL_AGENT' == 'CLOSE'. The same test fails identically on main (run 35828923745), so it is baseline, not a regression from this PR.

SKIPPED: Step 7 independent reader -- no second reader was available; the findings were re-derived against the head tree instead.

Checked: codex.py (_child_environment, _sdk_config, _materialize_managed_hooks), codex_managed_hooks.py, codex_hook_runner.py, registry.py, orchestrator/agent.py:595-660 and :840-860, cli.py:_require_lane_provider_capabilities, opportunity_agent.py:_AnalysisToolGuard, base.py:AgentHooks, the three changed test files | Ran: shlex.quote/shlex.split on the generated command; gh run view on the failing shards and on the same workflow on main | Base: 09d1ff5 | Head: 7c99f62

Drop the CODEX_HOME managed-hook bridge. Opportunity analysis now accepts
providers with stop_hooks or resumable sessions; hookless resumable backends
get the same refused-draft resume loop the implementer outer gate uses.
@BaoYunkai

Copy link
Copy Markdown
Collaborator Author

Re-scoped per your review (removed the CODEX_HOME managed-hook bridge).

  • Deleted codex_managed_hooks.py, codex_hook_runner.py, and their tests; reverted Codex stop_hooks=True in codex.py / registry.
  • OpportunityAnalysisAgent now requires stop_hooks or resumable. Hooks attach only when stop_hooks; for hookless resumable backends (Codex), _resume_for_refused_staging_drafts drives the same refused-draft Stop loop via backend.resume after the session returns, mirroring agent.py uses_outer_gate.
  • Write confinement stays on Codex’s existing sandbox / workspace guard / protected_paths; _AnalysisToolGuard pre-tool hooks remain Claude-only.

Pushed on feature/yunkai/codex-opportunity-hooks. Happy to squash the earlier hook commits on this branch if you prefer a single commit for merge.

@xiaofei-zheng

Copy link
Copy Markdown
Collaborator

PR #1617 -- feat(kernelforge): materialize Forge hooks as Codex managed command hooks

What it does: opportunity analysis refused any backend that does not run AgentRunSpec.hooks, so a Codex-only controller run died in OpportunityAnalysisAgent.__init__ before the first turn. The PR relaxes that gate to stop_hooks or resumable, attaches the _AnalysisToolGuard hooks only for providers that actually run them, and for a hookless resumable provider drives the Stop refusal from outside the session: after backend_task returns, _resume_for_refused_staging_drafts calls guard._on_stop and, while it blocks, resumes the session with the refusal as the prompt.

Blocking issues: 4

  1. [free:stale-refusal-loop] The outer Stop loop cannot observe a draft being corrected, and its escape hatch can [verified]
    Problem: the loop's predicate is pending_rejections(staging_root), which keys off the presence of rejection.json. That file is written and cleared only by publish_complete_staged_tasks / publish_staged_task, and the poll loop that calls it (while not backend_task.done(), opportunity_agent.py:558-567) has already exited by the time the resume loop runs; the next call is in the finally block at :603, after the loop is over. So a draft the agent corrects during a resumed turn still reads as refused. What the loop can observe is withdrawal, because pending_rejections honours _is_withdrawn. src/kernelforge/kernel_rewrite_controller/opportunity_agent.py:226-229, src/kernelforge/kernel_rewrite_controller/task_publisher.py:354-364
    Impact: every refused draft costs the full _MAX_STOP_DENIALS resumed Codex sessions even when the first one fixed it, and the agent is handed the same refusal each time while the refusal text tells it "the host will revalidate it within a few seconds" (opportunity_agent.py:169-171), which nothing does on this path. The refusal also offers withdrawal as the way out, and an agent that takes it loses a valid operator for good. Reproduced with a fake hookless resumable backend against a real staging layout: correcting on resume 1 still yields 3 resumes; correcting on resume 1 and withdrawing on resume 2 ends status=completed, published=0 with an empty tasks_root.
    Action: refresh the loop's input between turns the way orchestrator/agent.py:649-715 refreshes gate.edit_count -- call publish_complete_staged_tasks(layout, quiescent_sec=0.0, refused=refused) after each resumed turn (or move the loop inside the poll loop) so a corrected draft clears its own refusal, and fix the refusal text if revalidation is not in fact periodic on this path.

  2. [P6] The resumed sessions are charged to no budget [verified]
    Problem: _resume_for_refused_staging_drafts runs after the deadline watchdog at opportunity_agent.py:558-564 has exited, takes no deadline argument, and awaits backend.resume directly, so the finally block's backend_task.cancel() cannot reach it. Each resumed turn is capped at spec.timeout_sec (codex.py:764, 785), which is the whole analysis budget rather than what is left of it. src/kernelforge/kernel_rewrite_controller/opportunity_agent.py:571-577
    Impact: analysis can run up to 4x ANALYSIS_BUDGET_SEC while still reporting completed. Measured 9.52s against timeout_sec=2 with resumes starting at t+0.46/3.47/6.48, all after the deadline. dispatch_prepared_tasks is handed the same absolute controller_deadline_unix (controller.py:318) and skips every task once remaining < MIN_TASK_START_REMAINING_SEC (scheduler.py:164), so the overrun is taken straight out of the rewrite phase.
    Action: pass the remaining budget into the loop and stop resuming when it is exhausted, mirroring the deadline_sec=self.timeout_sec already given to run_session_with_api_resume at :547.

  3. [X2] Title and description describe an implementation that is not in the diff [verified]
    Problem: the body's Fix section claims CODEX_HOME/config.toml command hooks, a kernelforge.agent_backends.codex_hook_runner module, hook materialization in CodexBackend and stop_hooks=True on the Codex provider in the registry. None of that exists at head 4e21180: the diff touches opportunity_agent.py and two test files, codex_hook_runner.py and codex_managed_hooks.py are absent, and registry.py:418-440 does not set stop_hooks. All three test-plan boxes are ticked against src/kernelforge/tests/test_codex_managed_hooks.py and test_provider_registry.py::test_only_a_hook_running_provider_declares_stop_hooks, neither of which exists in the head tree. The title says the same.
    Impact: the description is the record the release cut aggregates, and a reader of this PR gets the design that was deleted in fix(kernelforge): run opportunity Stop refusals on resumable Codex rather than the one that landed -- including the new operator-visible effects (Codex analysis now runs; up to three extra resumed sessions after the deadline).
    Action: rewrite the title and the body against the current diff, and replace the test plan with the tests that exist.

  4. [S1] A failed resume leaves no trace [verified]
    Problem: except Exception: break swallows every failure of backend.resume -- CodexExecutionError included -- with no log, no status and no reason. The twin at orchestrator/agent.py:696 records gate.end_reason = "resume_error" and a finding; the pre-existing handler at opportunity_agent.py:593 that would have set ANALYSIS_STATUS_FAILED is bypassed by this inner catch. AGENTS.md:101 bans new broad except Exception. src/kernelforge/kernel_rewrite_controller/opportunity_agent.py:236-239
    Impact: a Codex resume that times out or dies is reported as status=completed with an empty reason; the only artifact records the draft rejection that would have been there anyway, so the operator cannot tell a provider failure from an agent that chose not to fix its draft.
    Action: catch the backend's declared error type and record it (status or reason) before breaking, as the outer-gate loop does.

Checked: opportunity_agent.py (_AnalysisToolGuard._on_stop, _resume_for_refused_staging_drafts, OpportunityAnalysisAgent.run, run_opportunity_analysis), task_publisher.py (pending_rejections, publish_complete_staged_tasks, _write_rejection, _is_withdrawn), agent_backends/base.py (AgentCapabilities, ResumableAgentBackend), codex.py (_execute, resume), registry.py codex provider, orchestrator/agent.py:649-715, scheduler.py, controller.py:305-320, the two changed test files, AGENTS.md | Ran: pytest src/kernelforge/tests/kernel_rewrite_controller/test_opportunity_agent.py -k "requires_hooks or resumable_hookless" (2 passed); three repros driving OpportunityAnalysisAgent.run with a hookless resumable backend (correct-on-first-resume; correct-then-withdraw; budget overrun) | Base: 146c113 | Head: 4e21180

Republish staged drafts before each outer Stop check, honor the analysis
deadline across resume turns, surface resume failures, and count publications
from the recovery loop.
@BaoYunkai BaoYunkai changed the title feat(kernelforge): materialize Forge hooks as Codex managed command hooks fix(kernelforge): run opportunity analysis on resumable Codex without global hooks Sep 23, 2026
_repair_staged_task(staging / "draft", repo)

backend = _HooklessResumableBackend(
lambda staging: _invalid_staged_draft(staging),
return await super().resume(spec, session_id, feedback, usage=usage)

backend = _SlowHookless(
lambda staging: _refused_draft(staging),
layout = ControllerLayout(tmp_path / "output")
monkeypatch.setenv("FORGE_AGENT_API_RETRY_BASE_SEC", "0")
backend = _HooklessResumableBackend(
lambda staging: _refused_draft(staging),
_repair_staged_task(staging / "draft", repo)

backend = _HooklessResumableBackend(
lambda staging: _invalid_staged_draft(staging),
return await super().resume(spec, session_id, feedback, usage=usage)

backend = _SlowHookless(
lambda staging: _refused_draft(staging),
layout = ControllerLayout(tmp_path / "output")
monkeypatch.setenv("FORGE_AGENT_API_RETRY_BASE_SEC", "0")
backend = _HooklessResumableBackend(
lambda staging: _refused_draft(staging),
@BaoYunkai

Copy link
Copy Markdown
Collaborator Author

Addressed the re-review on head eec77046d (re-scoped implementation; no CODEX_HOME hooks).

  1. stale-refusal-loop_resume_for_refused_staging_drafts now calls publish_complete_staged_tasks(..., quiescent_sec=0.0) before each outer _on_stop check so a corrected draft clears rejection.json; resume publications are merged into the analysis ledger. Stop prompt no longer promises revalidation “within a few seconds.” Test: test_hookless_resume_republication_clears_a_corrected_refusal.

  2. analysis budget (P6) — the resume loop shares the primary session deadline_monotonic, caps each backend.resume with remaining timeout_sec, and stops after the deadline (including post-resume). Test: test_hookless_resume_stops_when_the_analysis_deadline_is_exhausted.

  3. title/body (X2) — PR title and description updated to match the current diff (resumable Codex + outer Stop loop; hooks only when stop_hooks).

  4. resume failures (S1)AgentProviderError is logged and surfaced as ANALYSIS_STATUS_FAILED with reason; no silent except Exception: break. Test: test_hookless_resume_failure_is_reported.

CI: Lint (ruff E402 + format on opportunity_agent.py) fixed on the same branch.

Please re-review when convenient.

@xiaofei-zheng xiaofei-zheng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #1617 -- fix(kernelforge): run opportunity analysis on resumable Codex without global hooks

What it does: opportunity analysis refused any backend that does not run AgentRunSpec.hooks, so a Codex-only controller run died in OpportunityAnalysisAgent.__init__ before the first turn. The gate is relaxed to stop_hooks or resumable, _AnalysisToolGuard hooks are attached only for providers that execute them, and for a hookless resumable provider _resume_for_refused_staging_drafts drives the Stop refusal from outside the session: republish (quiescent_sec=0.0) so a corrected draft clears its own rejection.json, then, while _on_stop still blocks, resume the session with the refusal text -- bounded by _MAX_STOP_DENIALS and by what is left of the analysis deadline, with resume failures recorded instead of swallowed.

Blocking issues: none.

All four blocking findings from the previous review are closed at this head, re-verified here rather than taken from the diff:

  1. Stale refusal loop -- publish_complete_staged_tasks(layout, quiescent_sec=0.0, refused=refused) now runs at the top of every iteration, so a draft corrected during a resumed turn clears its refusal. Re-ran my own repro that never unlinks rejection.json (the agent has no tool that can, per the refusal text): resume calls = 1, status = completed, published = 1, was 3 resumes before. The correct-then-withdraw trap is gone too: same repro ends published = 1 instead of an empty tasks_root. The refusal text at :172-180 was corrected to match what the host actually does.
  2. Unbudgeted resumes -- the loop takes deadline_monotonic, refuses to start a resume with no budget left, caps each turn via replace(spec, timeout_sec=max(1, int(remaining))), and re-checks the deadline after each resume, setting end_reason="timeout" -> ANALYSIS_STATUS_TIMED_OUT. CodexBackend.resume reaches _execute(timeout=spec.timeout_sec) (codex.py:764), so the shrunken cap is enforced on the real provider.
  3. Title/description -- both now describe the diff that is here (outer resume loop, no CODEX_HOME hooks), and the test plan names tests that exist in the head tree.
  4. Silent resume failure -- AgentProviderError is caught explicitly, logged, and turned into ANALYSIS_STATUS_FAILED with the provider message in reason; asyncio.CancelledError is re-raised; the remaining broad catch logs and records the same way its twin at orchestrator/agent.py:696 does.

Checked: _resume_for_refused_staging_drafts and the status block in OpportunityAnalysisAgent.run (:587-640), _AnalysisToolGuard._on_stop denial cap and refusal text, task_publisher.pending_rejections / publish_complete_staged_tasks / _write_rejection / _is_withdrawn, codex.py resume/_execute timeout path, registry.py codex capabilities, the three changed test files | Ran: pytest src/kernelforge/tests/kernel_rewrite_controller/test_opportunity_agent.py (28 passed; the one failure, test_agent_staging_always_gets_a_private_git_baseline, reproduces identically at the merge base on this Windows checkout and is unrelated), pytest src/kernelforge/tests/test_lane_session_guarantees.py -k provider (7 passed), plus three repros driving OpportunityAnalysisAgent.run against a fake hookless resumable backend and a real staging layout (correct-on-first-resume, correct-then-withdraw, deadline exhaustion) | Base: 146c113 | Head: eec7704

@xiaofei-zheng
xiaofei-zheng merged commit fac5626 into main Sep 23, 2026
33 checks passed
@xiaofei-zheng
xiaofei-zheng deleted the feature/yunkai/codex-opportunity-hooks branch September 23, 2026 11:07
zoroyihan7 added a commit that referenced this pull request Sep 23, 2026
#1620 added RUF100 to the ruff selection at 19:06; #1617 added a
`# noqa: BLE001` to opportunity_agent at 19:07. Each was green on its own
branch -- RUF100 did not exist when #1617's checks ran -- and main has been red
since they landed 97 seconds apart. BLE001 does not fire on this handler anyway:
ruff exempts one that logs through `log.exception`.

Only the directive goes; the author's reason stays as a plain comment. The
change is comment-only, verified by comparing the token stream with comments
stripped.

This does not belong to this PR's subject and is here only because the merge
brought main's red lint onto the branch, where it blocks review.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lishuoshuo-amd added a commit that referenced this pull request Sep 23, 2026
The two landed 97 seconds apart. #1620 stopped BLE001 from firing on a handler
that calls log.exception, and #1617 was written before it, so the directive it
carries is unused and RUF100 reds every branch that merges main.

Carried here rather than in its own PR to unblock this one; it reverts no
behaviour and the comment it removes explains a suppression that no longer
suppresses anything.

Co-authored-by: Cursor <cursoragent@cursor.com>
lishuoshuo-amd added a commit that referenced this pull request Sep 23, 2026
#1609 landed the same way #1617 did: written before #1620 stopped BLE001 from
firing on a handler that re-raises, merged after it, so the directive is unused
and RUF100 reds main and every branch that merges it.

Same reason as the previous one for carrying it here rather than in its own PR.

Co-authored-by: Cursor <cursoragent@cursor.com>
ZhengGong-amd pushed a commit that referenced this pull request Sep 23, 2026
…it (#1595)

* Bound the wait on a Ray round so an unschedulable task cannot park a thread

_await_or_cancel polled ray.wait inside a `while True` with no wall-clock
deadline, relying entirely on someone else cancelling the scope. When Ray
cannot schedule the task at all, nobody ever does.

Observed 2026-09-21: Ray reported 8.0/8.0 GPU in use while the physical GPUs
sat at 0% with no serving processes -- ghost reservations left by a specialist
that died without releasing. Two integrate_patch warmups requesting
{'CPU': 1.0, 'GPU': 4.0, 'serving_slot': 1.0} stayed in PENDING_NODE_ASSIGNMENT
forever, and py-spy found two worker threads parked in this loop for over an
hour. Each held its pool slot, so its task stayed 'running', its lane leases
lapsed unreclaimed, and the coordinator went silent for 64 minutes.

The deadline is the round's own cap plus slack for what actually happens
outside it, configurable via INFERENCE_OPTIMIZER_RAY_ROUND_WAIT_SEC. Bailing
out goes through the cooperative cancel sequence the cancel path already uses
-- ask the actor first, escalate only after the grace period -- because
ray.kill runs neither __ray_terminate__ nor atexit and would strand the served
process tree of a round that is genuinely still running.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Reclaim lane leases from holders that already ended

Lane occupancy was decided by `SELECT lane, holder_id FROM leases` with no
regard for whether anybody was still using the row. Rows carry an expires_at,
but release happened only through an explicit DELETE, so a holder that ended
outside its release path held its lanes for the life of the session.

Observed 2026-09-21: a specialist raised ExecutionCleanupUnconfirmed and the
dispatcher deliberately retained capacity rather than releasing. Its six lanes
were still held two hours later with 19 integrate_patch tasks starved behind
them; the session made no progress until the rows were deleted by hand, twice.

Reclamation keys on the holder's task being terminal, not on the TTL. A TTL is
a static per-action budget that nothing enforces -- `explore` budgets 7200s and
holds server_lifecycle plus benchmark_lane across a benchmark that can exceed
it -- so a lapse says only that a long run outlived its estimate, while the
terminal state is the holder's own account of having stopped. Reading no
timestamp also keeps clock skew out of the predicate entirely.

Two guards come straight from reap_dead_holders: owner_scope must match this
boot/PID namespace, and a task with a live gpu_leases row is left alone,
because a lane records its coordinator rather than the specialist's GPU worker.
The sweep shares the caller's cursor so it and the capacity read that follows
sit in one BEGIN IMMEDIATE, and it runs as its own pass too -- the dispatcher's
lane gate reads lane_holders before any acquire, so a leaked row starves the
queue without an acquire ever running to reclaim it.

Running ahead of the round rules, this also settles rounds that a leaked lane
row used to wedge open indefinitely, either by hand-off or by expiry on the
terminal-holder cap. Both outcomes are pinned, as is the bound: a holder whose
cleanup never confirmed gets its lanes back without its round being advanced.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Break the import cycle instead of deferring around it

CodeQL flagged the deferred `from ..state.task_registry import TERMINAL_STATES`
inside resource_lock as the start of an import cycle, and it is right: the
cycle is real (task_registry imports resource_lock's SqliteLeaseBackend), the
deferral only hid it from the runtime.

The states are a vocabulary, not behaviour, so they move to a storage-free
`state/task_states` that anything may import. task_registry re-exports them, so
every existing import site is untouched, and resource_lock now imports at module
level with no cycle to step around.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Reclaim a lane only once its holder's process group is provably empty

The previous commit treated a terminal task as proof that nobody was using
its lane. An external review showed that inverts the contract: when cleanup
is unconfirmed, sub_agent_runner deliberately SKIPS the release and writes
terminal anyway, because the process tree may still be alive and the lane is
what stops conflicting work from starting. Reclaiming on terminal alone
deletes exactly the ownership that was retained on purpose -- and the
gpu_leases exemption does not cover it, since most of the nine
ExecutionCleanupUnconfirmed raise sites have no GPU lease at all.

Reclamation now needs one of two proofs: cleanup confirmed, or a recorded
process group with nothing left under it. Both launch sites spawn with
start_new_session, so the id is the group's and stays meaningful after the
root exits -- which is why it catches the common survivor, a child left in
the dead root's group. The evidence key and the helpers say "process group"
rather than "pid" because that is what is actually checked; a targeted build
records the pgid its handle already carries. Reclamation never signals a
process; it only asks.

What it cannot see is documented rather than papered over: a descendant that
calls setsid leaves both the group and the tree, and nothing in /proc ties it
back. A per-spawn cgroup would name it and is a separate project. A test
pins the gap so it reads as known rather than missed.

Anything unprovable keeps its lane, with no TTL or age heuristic anywhere in
the predicate -- silence is not proof. Since lane leases live in a
per-session database, a fresh run can never inherit such a row; only
resuming an old session meets one. So the operator story is the diagnostic:
one warning per row, de-duplicated, naming the lane, the holder, why it
cannot be verified, and a paste-ready sqlite statement against this
session's real database path, prefaced by the caution to confirm no process
of that task is still running. A count rides along in the maintenance
summary.

That report deliberately does not reuse the reclamation query. Its INNER
JOIN on tasks would drop a holder pruned out from under its lease, and its
gpu_leases exemption would silence the GPU path -- the one release_resources
fails on, and the shape of the 2026-09-21 incident. Reclamation must respect
both filters; reporting must not.

The rows leaked on 2026-09-21 are not healed by this: the code that wrote
them recorded no group, so they take the unprovable path and keep their
lanes. What this ends is the recurrence.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Stop trying to prove a lane is free; hold it and tell the operator

Three proofs were tried and each was shown by probe to be a proxy a real
process slips out of. Terminal state says the task stopped, not its children.
An empty spawn process group says nothing: a served process is setsid'd by
design, as _server_lifecycle notes where it reads a pidfile, so it leaves that
group. A pidfile naming no live server says nothing either -- it is written
only after the server answers, leaving the whole model-load window unnamed,
teardown unlinks it unconditionally, and matching a cmdline is the same kind
of guess one level down.

The asymmetry decides it. A lane held too long stalls a queue until an
operator spends ninety seconds; a lane released too early puts two rounds on
the same cards, which corrupts quietly and may never be noticed. Closing the
gap for real needs an identity a descendant cannot escape -- a per-execution
cgroup -- and that is its own project.

So reclamation by inspection is gone. What remains is liveness, which settles
itself (reap_dead_holders is untouched), and a diagnostic for everything else:
one warning per (lane, holder), naming the lane, the holder, why it cannot be
verified, the spawn process group as a lead explicitly not offered as proof,
and a paste-ready sqlite statement against this session's real database path,
prefaced by the caution to confirm no process of that task is still running. A
count rides in the maintenance summary each tick.

Every shape is reported, including ones earlier revisions kept quiet: a holder
still winding down (no longer distinguishable from an abandoned one, and the
noise is the price), a holder pruned out from under its lease, a holder still
on its GPU cards, a row from another boot, and a holder that confirmed cleanup
yet left its row behind -- which should not happen and so is worth hearing.

2026-09-21 would still wedge under this code. It would announce itself in the
log within a tick, with the command to clear it, instead of taking an hour of
py-spy and sqlite to find.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Keep a timed-out round's GPUs reserved instead of handing them back

The wall-clock ceiling killed the actor once the grace passed, which returns
its GPUs to the Ray scheduler. But ray.kill runs neither __ray_terminate__ nor
atexit, so a round that really was still running keeps its server subprocesses
-- and the next round is then placed on cards a live server still maps. That
trades a bounded wait for concurrent GPU use, which corrupts quietly and may
never be noticed.

The timeout path now abandons the round and leaves the actor alive. Its
devices stay reserved, so nothing else can be placed on them, and the caller
still gets the subprocess.TimeoutExpired its handlers already expect. The cost
is that those GPUs stay out of circulation until the session ends or an
operator intervenes; the log says exactly that, and what to check before
killing the actor by hand.

This is the same asymmetry the lane leases settle on, applied to the same kind
of resource: a resource stuck is recoverable in ninety seconds, a resource
shared is not recoverable at all. The cancel path keeps the behaviour it had
on main -- an operator or a shutdown asked for that teardown, and the caller
is waiting on it rather than on a round.

Also finishes removing the rejected inspection design: _ended_holder_rows was
left behind as unused reclamation-only code, and the comments and tests around
the recorded process group still described it as something a later reaper
would probe. Nothing probes it. It is recorded, and now documented, as a lead
for the operator who has to clear a retained lane by hand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Make the timed-out lease quarantine survive its caller's teardown

Keeping the actor alive inside _await_or_cancel was not a quarantine. Every
real caller — baseline, explore, integrate_patch — closes the lease in a
finally:, and close() kills the actor once its stop request fails, handing
those GPUs back to the scheduler while the served tree may still map them.
run_grid then moves to the next variant and is placed on them. The state was
declared in one method and undone in the next.

ServingLease now carries the quarantine explicitly, and all three entry points
honour it. close() leaves the actor alive and says why. ensure() raises
ServingLeaseQuarantined rather than minting a replacement actor behind its
back. run_session_kill refuses the round and reports it the way an ensure
failure is reported — deliberately not raising, because run_grid moves to the
next variant on a non-zero rc and each of those attempts has to be refused
too, whereas raising would escape the variant loop and widen the blast radius
past what this change is entitled to.

The cancel path is untouched: an operator or a shutdown asked for that
teardown and the caller waits on it, rather than on a round nobody answered.

Also finishes the documentation cleanup. The evidence-key block still called
the recorded group the only durable trace a later reaper has, and warned that
key drift would stop lanes being freed; two tests were named and documented
the same way. No reaper probes the group and these lanes are never freed
automatically. Both are now described as what they are: the lead an operator
gets for a lane that is retained on purpose.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Hold the quarantined actor at session scope, not on the lease object

The quarantine guarded all three entry points but only for as long as the
ServingLease object lived. Every real owner keeps that lease in an
action-local variable, closes it in a finally: and drops it. A Ray actor lives
as long as a handle to it does, so losing the last reference collects the
actor and returns its GPUs to the scheduler exactly as ray.kill would have --
the quarantine was undone by ordinary garbage collection rather than by any
code path.

The handle is now parked in a module-level list when the quarantine is
declared, which for these purposes is session scope: the process holding it is
the session. Nothing removes entries, deliberately. A test drops the lease and
forces a collection, and fails without that parking.

Also finishes the documentation pass. Three places still said a recurrence
carrying a process-group id could be settled by a probe -- the reconcile
docstring, its report attribute, and the maintenance summary -- which the
final design contradicts: nothing probes that id, and no coordinator-owned
lane is reclaimed automatically. What changed on 2026-09-21's shape is not
that it heals, but that it announces itself within a tick with the statement
that clears it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Measure how often teardown goes unconfirmed, before anyone builds for it

leases_unverifiable says how many lanes are held right now. It cannot say
whether that is a rare accident or the ordinary outcome, and that is the
question which decides what to do next.

Three portable ways to release such a lane automatically were designed and
measured against the environments this actually runs in, and all three are
undeployable on the production path: a per-task cgroup needs a delegated
writable subtree that claw mode does not have; a PID namespace needs a fresh
procfs, and mount() returns EPERM there; one Kubernetes pod per specialist does
not exist in local mode. The only candidate left -- an inherited descriptor
sealed behind an unprivileged seccomp filter, because a bare descriptor can be
closed by the workload -- is safety-critical to get right, since a filter that
misses one fd-destruction route would release lanes on a guarantee that does
not hold.

So measure first. The ratio rides the maintenance summary next to the count it
explains, reads what _write_terminal already records, costs one query and
writes nothing. A task whose history carries no cleanup account at all counts
as unconfirmed: that is exactly the shape which strands a lane.

If stranding turns out to be rare, the diagnostic is sufficient and nobody
should take the risk. If it is routine, this is the number that argues for it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Address review: document the ceiling knob, fix a garbled docstring, drop a swallow

Three findings from review, all verified against the tree before acting.

The wall-clock ceiling's override was invisible. INFERENCE_OPTIMIZER_RAY_ROUND_WAIT_SEC
appeared only in source, so an operator who needed to widen it -- or to
reproduce the stall deliberately -- had nowhere to look. Documented in both
places, with the semantics taken from the code rather than the PR text: the
unset default is the round's own timeout plus the slack, <= 0 disables the
ceiling and is also what an uncapped round gets, and a non-finite value is
rejected rather than tolerated, because nan > 0 is false and one typo would
otherwise switch off the very ceiling that prevents the stall.

_unverifiable_holders' docstring had two sentence halves merged into one line,
left by an earlier edit. It is the function the PR puts forward as the
operator's only route to a remedy, so a garbled sentence in it is worse there
than elsewhere.

maintenance.py wrapped the new cleanup_unconfirmed read in a broad except that
demoted any failure to debug. AGENTS.md forbids that, and it would have hidden
exactly the wrong thing: the summary would still look complete while missing
the one ratio the retention decision rests on -- including if _locks were ever
renamed, since the call reached into the reconciler's private attribute. The
swallow is gone and the call now goes through a Reconciler method, so the
summary loses its tick loudly instead. The maintenance test's reconciler double
gains the method, which is what surfaced the change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Drop a noqa directive that became unused between two merges

#1620 added RUF100 to the ruff selection at 19:06; #1617 added a
`# noqa: BLE001` to opportunity_agent at 19:07. Each was green on its own
branch -- RUF100 did not exist when #1617's checks ran -- and main has been red
since they landed 97 seconds apart. BLE001 does not fire on this handler anyway:
ruff exempts one that logs through `log.exception`.

Only the directive goes; the author's reason stays as a plain comment. The
change is comment-only, verified by comparing the token stream with comments
stripped.

This does not belong to this PR's subject and is here only because the merge
brought main's red lint onto the branch, where it blocks review.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Revert "Drop a noqa directive that became unused between two merges"

This reverts commit 77cb461.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants