Preserve hook execution and scope evidence to each drill

METHOD.md

@@ -2,6 +2,8 @@
 
 hookdrill observes a fresh, headless process in a copied project sandbox. A shim logs when each supported user command begins and ends, then runs that command with the original stdin, cwd, environment, stdout, stderr and exit code. Two extra project hooks prove that Claude Code loaded and executed sandbox hooks. The project files themselves are never rewritten.
 
+Copied files retain POSIX executable bits so hooks can invoke scripts directly. Their sandbox copies remain writable for canary insertion and probes. If a project has lifecycle hooks but no tool probe, a separate session check runs those hooks before the canary prompt.
+
 ## Pinned CLI facts
 
 The reference date is 2026-10-08. Claude Code hook locations, handler structure, matcher rules, input, exit codes and event timing come from the [hooks documentation](https://code.claude.com/docs/en/hooks). Project hook settings are in `.claude/settings.json` and `.claude/settings.local.json`; user hooks live in `~/.claude/settings.json`. Other hooks can originate in managed policy, plugins, skills and subagents. `PreToolUse` runs after tool parameters are made and before execution; `PostToolUse` follows a successful tool. `SessionStart` has a `source`; `UserPromptSubmit` and `Stop` ignore matchers. A shell command defaults to 600 seconds, while `args` changes it to direct execution. An exit of 2 can block a tool or keep a Stop hook active. Async hooks can be killed at print-mode teardown.
@@ -32,6 +34,8 @@ An unreached call is a main-conversation tool use targeted by a drilled PreToolU
 
 Canary positions are start, middle and end. Canary prompts run with the user's hooks removed from both sandbox settings files; only installed control hooks remain. Drill prompts keep the user's hooks. A clean answer must list marker candidates and echo the prompt's control marker. Failed prompts, tool use, unclean answers, missing controls or near misses are INCONCLUSIVE. Exact marker text is SEEN. A clean answer with control but without a position's marker is NOT SEEN, except that Claude needs a live project-hook control when controls were installed. A local hook can be NOT FIRED only after its local-settings control fires; otherwise local loading is INCONCLUSIVE. NOT SEEN means the model did not echo the marker; it does not prove why.
 
+Canary controls establish rule-file evidence only: user hooks are removed for those prompts, so their absence cannot establish NOT FIRED. If the separate hook drill fails, a successful canary prompt does not turn that failure into a negative hook verdict. Tool-use ids are paired with the prompt index when matching shim records, because each prompt is a fresh CLI session.
+
 ## Positional arithmetic
 
 For original length L and newline size s (LF 1, CRLF 2), a marker line has m = 28 + s bytes. `Project marker: ` is 16 bytes and `HDC-` plus eight hex characters is 12. If a final line lacks a newline, one is added. The middle line is inserted before the first original line whose start offset is at least floor(L/2). Let P be the bytes before that line; P is at least floor(L/2). Total planted length T is at most L + s + 3m. The start marker text ends at byte 28, while T is at least 3m, so the first half includes it. The end marker starts at T - m, at or beyond T/2. The middle marker text ends at E = m + P + 28. Since 55 > m + s (30 for LF, 32 for CRLF), E exceeds T/2. Thus a reader of the first half sees the start marker but cannot see the full middle or end markers.

README.md

@@ -82,6 +82,8 @@ A live run spends real prompts from your plan, and drilled hooks really run in t
 
 `HOOKDRILL_CLAUDE` and `HOOKDRILL_CODEX` set CLI paths. Explicit flags take precedence. `HOOKDRILL_FORBID_PATH_LOOKUP=1` disables fallback PATH discovery. `CLAUDE_CODE_GIT_BASH_PATH` can point to Git Bash on Windows.
 
+`--prompt-timeout` must be a finite, positive number of seconds (default: 300). Projects with lifecycle hooks and no tool probes receive a separate session check before the canary prompt, so the dry plan includes both checks. Invalid comparison fields in saved run files produce a usage error when running `diff`.
+
 ## What it reports
 
 | Hook state | Meaning |

src/hookdrill/cli.py

@@ -1,5 +1,6 @@
 import argparse
 import json
+import math
 import os
 import sys
 from datetime import datetime, timezone
@@ -55,8 +56,8 @@ def _run(args):
             for name in links:
                 print("  {}: SKIPPED: rule file is a link (not copied into the sandbox)".format(name))
         raise ValueError("nothing to drill")
-    if args.prompt_timeout <= 0:
-        raise ValueError("prompt timeout must be positive")
+    if not math.isfinite(args.prompt_timeout) or args.prompt_timeout <= 0:
+        raise ValueError("prompt timeout must be finite and positive")
     paths, versions = {}, {}
     for vendor in ("claude", "codex"):
         if any(p["vendor"] == vendor for p in prompts):

src/hookdrill/diff.py

@@ -7,6 +7,18 @@ def read(path):
             data = json.load(stream)
         if data.get("schema_version") != 1 or data.get("tool") != "hookdrill":
             raise ValueError()
+        vendors = data.get("vendors", {})
+        if not isinstance(vendors, dict) or any(not isinstance(value, dict) for value in vendors.values()):
+            raise ValueError()
+        for field, keys in (("hooks", ("key", "label", "state")),
+                            ("canaries", ("file", "position", "state"))):
+            rows = data.get(field, [])
+            if not isinstance(rows, list) or any(
+                    not isinstance(row, dict) or any(not isinstance(row.get(key), str) for key in keys)
+                    for row in rows):
+                raise ValueError()
+        if not isinstance(data.get("unreached", []), list):
+            raise ValueError()
         return data
     except (OSError, ValueError, AttributeError):
         raise ValueError("not a hookdrill run file: " + path)

src/hookdrill/plan.py

@@ -48,10 +48,10 @@ def make(project, vendors, handlers):
         for tool in TOOLS:
             if any(not h["skip"] and h["probe"] == tool for h in handlers):
                 prompts.append({"vendor": "claude", "purpose": "probe " + tool, "text": PROMPTS[tool]})
-        if not linked(os.path.join(project, "CLAUDE.md")) and os.path.isfile(os.path.join(project, "CLAUDE.md")):
-            prompts.append({"vendor": "claude", "purpose": "canaries CLAUDE.md", "text": PROMPTS["canary"]})
         if not prompts and any(not h["skip"] and h["event"] in ("SessionStart", "UserPromptSubmit", "Stop") for h in handlers):
             prompts.append({"vendor": "claude", "purpose": "session check", "text": PROMPTS["session"]})
+        if not linked(os.path.join(project, "CLAUDE.md")) and os.path.isfile(os.path.join(project, "CLAUDE.md")):
+            prompts.append({"vendor": "claude", "purpose": "canaries CLAUDE.md", "text": PROMPTS["canary"]})
     if "codex" in vendors and not linked(os.path.join(project, "AGENTS.md")) and os.path.isfile(os.path.join(project, "AGENTS.md")):
         prompts.append({"vendor": "codex", "purpose": "canaries AGENTS.md", "text": PROMPTS["canary"]})
     for index, prompt in enumerate(prompts, 1):

src/hookdrill/sandbox.py

@@ -77,6 +77,10 @@ def copy_item(relative):
                 if size > MAX_COPY_BYTES:
                     raise ValueError("sandbox copy exceeds 50 MB")
                 outgoing.write(chunk)
+            if os.name != "nt":
+                mode = stat.S_IMODE(os.fstat(outgoing.fileno()).st_mode)
+                executable = os.fstat(incoming.fileno()).st_mode & 0o111
+                os.fchmod(outgoing.fileno(), mode | executable)
 
     try:
         for index, item in enumerate([".claude", "CLAUDE.md", "AGENTS.md"] + list(include)):

src/hookdrill/shim.py

@@ -25,6 +25,8 @@ def main(argv=None):
     try:
         payload = json.loads(data)
         valid = isinstance(payload, dict)
+        if not valid:
+            payload = {}
     except (ValueError, UnicodeDecodeError):
         payload, valid = {}, False
     try:

src/hookdrill/verdict.py

@@ -43,10 +43,11 @@ def hook(item, records, prompts):
                 if prompt.get("results", {}).get(start.get("tool_use_id")) is True:
                     notes.append("exited 2 but the {} call in prompt {} succeeded".format(start.get("tool_name"), record.get("prompt")))
         return "FIRED", "{} {}".format(len(inv), "call" if len(inv) == 1 else "calls"), notes, inv
-    ok = [p for p in prompts if p["vendor"] == "claude" and p.get("ok")]
+    ok = [p for p in prompts if p["vendor"] == "claude" and p.get("ok")
+          and not p.get("purpose", "").startswith("canaries ")]
     live = [p for p in ok if p.get("live")]
     if not ok:
-        return "INCONCLUSIVE", "no prompt completed", notes, inv
+        return "INCONCLUSIVE", "no hook drill prompt completed", notes, inv
     if not live:
         if any(p["purpose"].startswith(("probe ", "session")) and p.get("tools") is not None and "Bash" not in p["tools"] for p in ok):
             return "INCONCLUSIVE", "the Bash tool is not registered, so on Windows the CLI is not using Git Bash for hooks", notes, inv
@@ -83,14 +84,15 @@ def calls(prompts, handlers, records):
     drilled = [h for h in handlers if not h["skip"] and h["event"] == "PreToolUse"]
     ids = {h["id"] for h in drilled}
     pre_records = [r for r in records if r.get("hook_id") in ids]
-    reached = {r.get("start", {}).get("tool_use_id") for r in pre_records if isinstance(r.get("start"), dict) and r["start"].get("tool_use_id")}
+    reached = {(r.get("prompt"), r["start"]["tool_use_id"]) for r in pre_records
+               if isinstance(r.get("start"), dict) and r["start"].get("tool_use_id")}
     missing, uncounted = [], []
     for prompt in prompts:
         if prompt["vendor"] != "claude":
             continue
         for call in prompt.get("calls", []):
             matching = [h for h in drilled if targets(h["matcher"], call.get("name"))]
-            if not matching or call.get("id") in reached:
+            if not matching or (prompt["index"], call.get("id")) in reached:
                 continue
             row = {"tool_name": call.get("name"), "tool_use_id": call.get("id"), "prompt": prompt["index"]}
             if prompt.get("tools") is not None and call["name"] not in prompt["tools"]:

tests/test_diff.py

@@ -50,6 +50,19 @@ def test_invalid_run_file(self):
         self.assertEqual(code, 2)
         self.assertIn("not a hookdrill run file:", errors)
 
+    def test_invalid_comparison_fields_are_usage_errors(self):
+        changes = ({"vendors": None}, {"vendors": {"claude": []}}, {"hooks": None},
+                   {"hooks": [{}]}, {"hooks": [{"key": [], "label": "hook", "state": "FIRED"}]},
+                   {"canaries": None}, {"canaries": [{}]}, {"unreached": None})
+        for changed in changes:
+            with self.subTest(changed=changed):
+                broken = self.sample()
+                broken.update(changed)
+                code, output, errors = self.run_diff(broken, self.sample())
+                self.assertEqual(code, 2)
+                self.assertIn("not a hookdrill run file:", errors)
+                self.assertEqual(output, "")
+
     def test_added_only_and_removed_only(self):
         original = self.sample()
         for changed, word in (({"hooks": original["hooks"] + [dict(original["hooks"][0], key="added")]}, "added"),

tests/test_e2e.py

@@ -100,6 +100,43 @@ def test_wrong_case_and_errored(self):
             self.assertEqual(code, 1, output)
             self.assertIn("PreToolUse broken: ERRORED (exit 1)", output)
 
+    def test_lifecycle_hooks_with_rule_file(self):
+        with tempfile.TemporaryDirectory() as temp:
+            project = Path(temp) / "project"
+            folder = project / ".claude"
+            folder.mkdir(parents=True)
+            (project / "CLAUDE.md").write_text("Keep the project tests deterministic.\n", encoding="utf-8")
+            events = ("SessionStart", "UserPromptSubmit", "Stop")
+            hooks = {event: [{"hooks": [{"type": "command", "command": "exit 0"}]}] for event in events}
+            (folder / "settings.json").write_text(json.dumps({"hooks": hooks}), encoding="utf-8")
+            before = tree_hashes(project)
+            output_path = Path(temp) / "run.json"
+            code, output, errors = invoke(project, vendors=["claude"], out=output_path)
+            self.assertEqual(code, 0, output + errors)
+            saved = json.loads(output_path.read_text(encoding="utf-8"))
+            self.assertEqual([p["purpose"] for p in saved["prompts"]],
+                             ["session check", "canaries CLAUDE.md"])
+            self.assertTrue(all(h["state"] == "FIRED" for h in saved["hooks"]))
+            self.assertTrue(all([r["prompt"] for r in h["invocations"]] == [1] for h in saved["hooks"]))
+            self.assertTrue(all(c["state"] == "SEEN" for c in saved["canaries"]))
+            self.assertEqual(tree_hashes(project), before)
+
+    @unittest.skipIf(os.name == "nt", "POSIX executable permission bits")
+    def test_executable_hook_survives_sandbox(self):
+        with tempfile.TemporaryDirectory() as temp:
+            project = Path(temp) / "project"
+            folder = project / ".claude/hooks"
+            folder.mkdir(parents=True)
+            script = folder / "verify.sh"
+            script.write_text("#!/bin/sh\nexit 0\n", encoding="utf-8")
+            script.chmod(0o755)
+            settings = {"hooks": {"Stop": [{"hooks": [{"type": "command", "command": ".claude/hooks/verify.sh"}]}]}}
+            (project / ".claude/settings.json").write_text(json.dumps(settings), encoding="utf-8")
+            code, output, errors = invoke(project, vendors=["claude"], out=Path(temp) / "run.json")
+            self.assertEqual(code, 0, output + errors)
+            self.assertIn("Stop verify: FIRED (1 call)", output)
+            self.assertEqual(script.stat().st_mode & 0o777, 0o755)
+
     def test_refusal_and_crash(self):
         with tempfile.TemporaryDirectory() as temp:
             scenario = self.scenario(temp, {"refuse_tools": True})

tests/test_round1.py

@@ -236,7 +236,7 @@ def test_timeout_is_recorded(self):
         process = Mock(returncode=-9, pid=1234)
         process.communicate.side_effect = [subprocess.TimeoutExpired("fake", 1), (b"", b"")]
         with patch("hookdrill.clis.subprocess.Popen", return_value=process), \
-             patch("hookdrill.clis.subprocess.run"):
+             patch("hookdrill.clis.subprocess.run"), patch("hookdrill.clis.os.killpg", create=True):
             code, stdout, stderr, timed_out = clis.prompt("fake.py", ["-p", "prompt"], "sandbox", 1, 1)
         self.assertEqual((code, stdout, stderr, timed_out), (-9, b"", b"", True))
         if os.name == "nt":

tests/test_safety.py

@@ -88,6 +88,18 @@ def test_version_spawn_args(self):
         self.assertEqual(kwargs["stdin"], clis.subprocess.DEVNULL)
         self.assertIsInstance(args[0], list)
 
+    def test_invalid_timeout_never_spawns_cli(self):
+        for value in ("nan", "inf", "-inf", "0", "-1"):
+            with self.subTest(value=value), patch("hookdrill.cli.clis.version", side_effect=AssertionError("spawned")) as version:
+                with contextlib.redirect_stderr(io.StringIO()) as errors:
+                    code = main(["run", "--yes", "--project", str(ROOT / "examples/project"),
+                                 "--vendor", "codex", "--codex", str(ROOT / "examples/fake-cli/fake_codex.py"),
+                                 "--prompt-timeout=" + value])
+                self.assertEqual(code, 2)
+                self.assertIn("prompt timeout must be finite and positive", errors.getvalue())
+                self.assertNotIn("internal error", errors.getvalue())
+                version.assert_not_called()
+
     def test_prompt_spawn_args(self):
         process = Mock()
         process.communicate.return_value = (b"", b"")

tests/test_shim.py

@@ -69,6 +69,18 @@ def test_control_and_unknown(self):
         self.assertEqual(result.returncode, 1)
         self.assertEqual(len(self.records()), 1)
 
+    def test_non_object_json_preserves_hook_stdin(self):
+        payloads = (b"[]\n", b"null\n", b'"text"\n', b"17\n", b"true\n")
+        for payload in payloads:
+            with self.subTest(payload=payload):
+                result = self.invoke("h1", "cat > received.bin; exit 2", prompt=4, payload=payload)
+                self.assertEqual(result.returncode, 2, result.stderr)
+                self.assertEqual((self.root / "received.bin").read_bytes(), payload)
+                self.assertNotIn(b"Traceback", result.stderr)
+        records = self.records()
+        self.assertEqual(len(records), len(payloads))
+        self.assertTrue(all(r["start"]["stdin_json"] is False and r["end"]["exit_code"] == 2 for r in records))
+
     def test_control_never_spawns(self):
         from hookdrill import shim
         (self.meta / "hooks.json").write_text(json.dumps({"c1": {"control": True}}), encoding="utf-8")

tests/test_verdict.py

@@ -44,6 +44,25 @@ def test_missing_stdin_fields(self):
                 self.assertEqual(missing, [])
                 self.assertTrue(all(row["reason"] == "hook stdin carried no tool_use_id" for row in uncounted))
 
+    def test_call_ids_are_scoped_to_prompt(self):
+        prompt, records, hook = fixture()
+        call = prompt["calls"][0]
+        first = dict(prompt, calls=[call], results={call["id"]: True})
+        second = dict(first, index=2)
+        missing, uncounted = verdict.calls([first, second], [hook], records)
+        self.assertEqual(missing, [{"tool_name": call["name"], "tool_use_id": call["id"], "prompt": 2}])
+        self.assertEqual(uncounted, [])
+
+    def test_canary_controls_do_not_prove_user_hook_absence(self):
+        canary = {"vendor": "claude", "index": 2, "purpose": "canaries CLAUDE.md", "ok": True,
+                  "live": True, "local_live": True, "c2": True, "sources": ["startup"]}
+        failed_drill = dict(canary, index=1, purpose="session check", ok=False)
+        for event in ("SessionStart", "UserPromptSubmit", "Stop"):
+            with self.subTest(event=event):
+                item = {"id": "h1", "event": event, "matcher": None, "probe": "any prompt", "skip": None}
+                state, _, _, _ = verdict.hook(item, [], [failed_drill, canary])
+                self.assertEqual(state, "INCONCLUSIVE")
+
     def test_none_match_targets_nothing(self):
         prompt, records, hook = fixture()
         prompt["calls"].append({"id": "extra", "name": None})