hookdrill

Prove your agent hooks still fire and your rule files still load after every CLI upgrade.

Works with Anthropic's Claude Code CLI (project command hooks in .claude settings and CLAUDE.md) and the OpenAI Codex CLI (AGENTS.md) on Windows, macOS and Linux (Python 3.9+, standard library only, no LLM of its own, no network access of its own). Use it when an agent hook silently stops firing after a CLI upgrade, when your AI coding agent seems to ignore CLAUDE.md or AGENTS.md because the rule file never reached the model, or when you need a saved before and after comparison that shows which hooks and rule-file sections changed between two CLI versions.

Demo

This demo uses the bundled fake CLIs to simulate an upgrade. It does not contact an agent service or spend prompts.

$ hookdrill run --project examples/project --claude examples/fake-cli/claude_upgraded.py --codex examples/fake-cli/codex_upgraded.py --yes --out examples/runs/after.json
hookdrill 0.1.0  project: examples/project
claude: 1.1.0-fake (Fake Claude), 3 prompt(s)
codex: codex-cli 1.1.0-fake, 1 prompt(s)

hooks
  SessionStart banner: FIRED (2 calls)
  PreToolUse guard: FIRED (2 calls)
  PostToolUse format: FIRED (1 call)
  Stop verify: NOT FIRED
  Notification notify: SKIPPED (event not drilled in v0.1)
rule files
  CLAUDE.md: start SEEN, middle SEEN, end SEEN
  AGENTS.md: start SEEN, middle NOT SEEN, end NOT SEEN
tool calls
  1 tool call never reached a hook
    Bash in prompt 1 (probe Bash)
result: 4 problems, 0 inconclusive, 1 skipped (exit 1)
saved: examples/runs/after.json

$ hookdrill diff examples/runs/before.json examples/runs/after.json
hookdrill diff: examples/runs/before.json -> examples/runs/after.json
claude: 1.0.0-fake (Fake Claude) -> 1.1.0-fake (Fake Claude)
codex: codex-cli 1.0.0-fake -> codex-cli 1.1.0-fake
changed
  Stop verify: FIRED -> NOT FIRED
  AGENTS.md middle: SEEN -> NOT SEEN
  AGENTS.md end: SEEN -> NOT SEEN
  unreached tool calls: 0 -> 1
4 surfaces changed

Why

Rule files can be silently shortened, as described in Codex issue 13386. The AGENTS.md documentation defines how Codex finds project instructions. Hookdrill checks what a fresh CLI process actually exposes after an upgrade.

Quick start

From a checkout:

python -m pip install .
hookdrill --dry --project examples/project

In your own project, after pipx install git+https://github.com/mmelqadi/hookdrill:

hookdrill --dry
hookdrill run --yes

A live run spends real prompts from your plan, and drilled hooks really run in the sandbox. Read the dry plan first.

Usage

hookdrill [run] [--project DIR] [--vendor claude|codex]... [--dry] [--yes] [--claude PATH] [--codex PATH] [--include PATH]... [--out FILE] [--prompt-timeout S] [--keep-sandbox] runs a check. hookdrill diff RUN_A RUN_B compares saved results. With no subcommand, run is assumed. Vendor defaults to both; only vendors with a planned prompt need a CLI. --include copies additional project-relative paths for hooks that need them. --dry and a missing --yes never spawn a CLI. --keep-sandbox retains the temporary evidence folder.

Exit	Meaning
0	At least one surface verified; no problem or inconclusive result
1	A proven problem
2	Usage, config, safety or CLI error
3	Inconclusive evidence or no verified surface

HOOKDRILL_CLAUDE and HOOKDRILL_CODEX set CLI paths. Explicit flags take precedence. HOOKDRILL_FORBID_PATH_LOOKUP=1 disables fallback PATH discovery. CLAUDE_CODE_GIT_BASH_PATH can point to Git Bash on Windows.

--prompt-timeout must be a finite, positive number of seconds (default: 300). Projects with lifecycle hooks and no tool probes receive a separate session check before the canary prompt, so the dry plan includes both checks. Invalid comparison fields in saved run files produce a usage error when running diff.

What it reports

Hook state	Meaning
FIRED	The wrapped command ran
NOT FIRED	A trigger and project-hook activity were both proven, but the command did not run
ERRORED	The command failed or did not finish
INCONCLUSIVE	The trigger or project-hook activity could not be proven
SKIPPED	This hook form has no v0.1 drill

Canary state	Meaning
SEEN	Exact marker appeared in the answer
NOT SEEN	The answer was clean and echoed the prompt control marker, but omitted this file marker
INCONCLUSIVE	The answer or CLI activity did not justify a negative claim

Two sandbox-only control hooks prove project hooks ran. A drilled local-settings hook also gets local controls. Canary prompts run with the user's hooks removed from the sandbox config, leaving the controls. The tool-call report distinguishes successful calls without a matching PreToolUse shim record from INCONCLUSIVE calls and calls that were not counted.

How it works

Hookdrill copies agent config and rule files into a temporary sandbox, plants positional canaries, wraps supported commands, and runs fresh headless CLI prompts. Results describe headless -p sessions, not interactive ones. Claude Code can stream hook events with --include-hook-events, but the shim is vendor-neutral proof that YOUR command ran, with its exit code, and powers the unreached-call report. Control hooks turn bare mode, PowerShell hook shells and disabled hooks into INCONCLUSIVE, never NOT FIRED.

What this is not

Hookdrill is not a static config linter or a rule-file sync tool. It is not a mock-model conformance suite such as harness-test, which tests its own mock hooks and model server. It is not a security boundary. A canary does not prove that the model OBEYS a rule; it proves only that the text reached the model at session start. FIRED does not prove the CLI HONORED a hook decision. A blocking exit 2 whose call still succeeded is shown only as a note.

Limits

Read LIMITS.md for scope and live-run caveats, and METHOD.md for evidence rules and source references.

Part of the Exit Zero series.

License

MIT. See LICENSE.