Skip to content

Preserve hook execution and scope evidence to each drill - #3

Merged
mmelqadi merged 1 commit into
mainfrom
codex/review-hookdrill-20261011
Oct 11, 2026
Merged

mmelqadi merged 1 commit into
mainfrom
codex/review-hookdrill-20261011

Conversation

@mmelqadi

@mmelqadi mmelqadi commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

What this changes

Healthy lifecycle hooks could be reported as NOT FIRED when a project also had CLAUDE.md: the only planned prompt was a canary check, which deliberately removes user hooks. Directly invoked executable scripts also lost their executable bits when copied into the sandbox. These behaviors made the tool itself produce false regression reports.

  • Plan a separate session check when lifecycle hooks have no tool probe, and exclude canary controls from negative hook evidence. The dry plan shows the additional prompt.
  • Preserve POSIX executable bits on copied scripts while keeping sandbox copies writable.
  • Pair tool-use ids with the prompt index so an invocation in one fresh session cannot hide an unreached call in another.
  • Preserve raw stdin and the wrapped hook's exit status for JSON arrays, null, strings, numbers and booleans; unavailable metadata is recorded as stdin_json: false.
  • Reject nonfinite timeouts before CLI execution and reject malformed comparison fields before rendering a diff.
  • Fully mock process-group termination in the existing mocked-timeout test.
  • Document the planning, evidence and input-validation behavior.

Reproduced before the fix

  • A temporary project with CLAUDE.md and three healthy lifecycle handlers produced one canary prompt, three NOT FIRED verdicts and exit 1. It now runs a session check plus a canary check, reports the three hooks FIRED and exits 0. A failed hook drill remains INCONCLUSIVE even if canaries succeed.
  • A directly invoked executable .claude/hooks/verify.sh was reported ERRORED with exit 126 after copying. It now runs successfully in the sandbox.
  • Reusing a tool-use id in two prompt fixtures hid the second prompt's unhooked call. The report now identifies that call.
  • Non-object JSON input crashed the shim before running the hook; malformed saved-run fields crashed diff; nan and inf timeouts reached the CLI version probe.

How it was tested

  • Baseline: 65 tests run, 63 passed and 2 expected platform skips on Linux / Python 3.12.14. The existing timeout test's placeholder process-group kill was guarded during this baseline run.
  • Added seven regression methods and ran them against the original implementation; they reproduced the failures above.
  • python -m unittest discover -s tests -v: 72 tests run, 70 passed and 2 expected Windows-only skips after the fixes.
  • python -m pip wheel --no-deps --no-build-isolation --no-index --wheel-dir <temporary-output> .: offline wheel build passed.
  • Installed the wheel into a fresh virtualenv outside the source checkout. CLI/module versions, dry planning, fake baseline and upgrade runs, and saved-run comparison passed; the documented simulated upgrade showed 4 changed surfaces.
  • All agent runs used bundled fake CLIs or mocks. No real vendor/model calls or paid prompts were used.
  • PR CI passed all six jobs: Windows/macOS/Linux on Python 3.9 and 3.13, including wheel builds and fresh-venv CLI smoke checks.

Reviewed against 58ba9b3ac8f66d4ad6ef584f90476dbbc2695081. Main was unchanged and no open PR overlapped immediately before publication. This is a draft; no release was published.

Checklist

  • Tests added or updated
  • Test suite passes locally
  • Docs updated for changed behavior
  • No secrets, personal data or machine-specific paths

@mmelqadi
mmelqadi marked this pull request as ready for review October 11, 2026 19:14
@mmelqadi
mmelqadi merged commit ddfa4a1 into main Oct 11, 2026
6 checks passed
@mmelqadi
mmelqadi deleted the codex/review-hookdrill-20261011 branch October 11, 2026 19:14
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-11T19:15:39.962617Z 9d7258c Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant