feat(hooks): Hookshot integration — cross-platform hook dispatcher, DIY tripwire, and L3 policy gate - #68
Open
basicScandal wants to merge 5 commits into
Open
feat(hooks): Hookshot integration — cross-platform hook dispatcher, DIY tripwire, and L3 policy gate#68basicScandal wants to merge 5 commits into
basicScandal wants to merge 5 commits into
Conversation
…pwire
Add the keystone of the Hookshot integration (.planning/hookshot-integration.md):
a portable agent-lifecycle hook contract, and wire its first handler into
`starlog init` so it fires in Claude Code.
- src/hook/dispatch.ts: the transport-agnostic `starlog hook <event>` contract —
{ event, cwd, payload } -> { decision: allow|warn|deny, message?, source }.
Fail-open (malformed input or any handler error -> allow) and warn-only until
policy opt-in. before_execution/prompt_submit/stop are forward-compatible
allow-stubs for later phases.
- after_file_edit MVP: a cheap, recurrence-gated tripwire. detectFile gates on a
single edited file (no fs walk, 512KB head-slice cap); the recurrence store is
the counter; one warn fires at the threshold crossing pointing at
starlog_advise. No search/LLM in the hot path (deviates from the plan's first
sketch, which paid a per-edit search even to watch).
- src/patterns/detect.ts: new detectFile(relPath, content) — the single-file gate.
- src/hook-runner.ts: the installed shim self-routes by tool_name —
Edit|Write|MultiEdit drive the tripwire (translating Claude's tool_input and
re-emitting warns as hookSpecificOutput); everything else keeps the existing
install-facts path.
- src/init.ts: register the shim under two matchers (Bash + Edit|Write|MultiEdit)
with matcher-aware dedup; hookAction reports `update` when either is missing so
pre-tripwire installs re-init; uninstall already strips both by filename.
- src/cli.ts: `starlog hook <event>` subcommand (reads stdin, always exits 0).
Tests (627/627): dispatch contract + fail-open + hostile-payload hardening +
independent per-category counters + cross-project accumulation; detectFile
false-positive floor / empty / oversized-edit; edit-tripwire e2e through the real
bundled shim; init upgrade-path e2e (Bash-only -> gains edit matcher, no dup,
foreign hooks preserved).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Verify the integration against the REAL Claude Code payload schema (primary
source), harden it with property/fuzz tests, and fix a fail-open bug the property
suite surfaced.
- fix(dispatch): dispatchRaw threw on any JSON that parses to a non-object
(e.g. `null`, `123`) — `parsed.event` ran outside the try/catch, breaking the
fail-open contract. Guard that the parsed value is a non-null object.
- Verified field names against Claude Code's real Edit/Write/MultiEdit tool
schemas: file_path / content / new_string / edits[].new_string are correct
(a docs-summary claim of path / new_text was a hallucination — not adopted).
- Export editPayload for direct unit testing of the tool_input translation.
- New tests (645/645):
* src/hook-runner.test.ts — editPayload translation (Write/Edit/MultiEdit,
missing/garbage, malformed edits[]).
* src/hook/dispatch.property.test.ts — fast-check invariants: never throws,
always a valid decision, no deny (warn-only), JSON round-trips, metamorphic
"exactly one warn per fresh store"; plus the non-object fail-open regression.
* src/hook/dispatch.test.ts — concurrent-edit robustness (no throw / no store
corruption), independent per-category counters, cross-project accumulation.
* src/init.hook.test.ts — golden real-envelope e2e (full PostToolUse payload,
all three edit tools + Bash no-regression), MultiEdit coverage.
- fast-check added as a dev dependency.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a $-gated behavioral eval that measures whether the after_file_edit tripwire's injected warn actually changes an agent's decision — the "LLM decision-flip eval" the bench-facts scope note names as deferred. Design (3 arms isolate the marginal contribution, per .planning/tripwire-eval.md): A control · B +tripwire-nudge · C +advice-inlined A->B lift in the adopt-library rate is the go/no-go number; blind LLM grader; decision rule (kill if A->B < 15pp) precommitted BEFORE running. - src/engine/tripwire-eval/scenarios.ts — 6 genuinely-ambiguous DIY scenarios. - src/engine/tripwire-eval/run.ts — arms, blind grader, scoring (LLM injected). - src/engine/tripwire-eval/run.test.ts — deterministic coverage (prompts differ only by injection, grader parser, scoring math via a stubbed LLM — no spend). - scripts/eval-tripwire.ts — harness (npm run eval:tripwire), reuses createLlmFn. Result (haiku, k=15, 90/arm): adopt-rate A 6.7% / B 66.7% / C 83.3%; A->B +60pp => KEEP. Honest boundary: control is a bare model, so this proves the nudge text works (upper bound), not marginal value over the existing MCP tool + CLAUDE.md — that needs a multi-turn eval. Surfaced: auth is immovable even with advice (0/7/0); inlining a brief hint (B->C +16.7pp) beats the "call starlog_advise" indirection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…icy gate
Two changes, both flowing from the decision-flip eval.
1. Inline hint (eval-motivated). The tripwire message now names example vetted
libraries per category ("like Clerk, Auth0, or better-auth") instead of a bare
"call starlog_advise" indirection — the eval showed a concrete named hint moves
behavior far more (B->C +16.7pp). The authoritative, safety/policy-checked
recommendation still comes from starlog_advise. (CATEGORY_HINTS in dispatch.ts.)
2. Phase 2 — before_execution L3 policy gate (the handler with teeth).
- src/patterns/install-parse.ts: shared install-command parser extracted from
hook-runner (reused by both the install-facts hook and the gate); hook-runner
refactored onto it (its regression tests are the safety net).
- dispatch.handleBeforeExecution: parse npm/pnpm/yarn/pip install, resolve L3
policy LOCAL-only (lookupFactView, no network in a blocking hot path; policy
path resolved from cwd via overlayPath). Any policy-DENY package -> warn
(default) or deny (STARLOG_HOOK_ENFORCE=deny). Dependency-introduction only,
not general command safety.
- hook-runner: PreToolUse/Bash -> gate; maps deny->permissionDecision "deny"
(hard block), warn->"ask" (surface + user decides), allow->emit nothing.
PreToolUse blocking format verified against code.claude.com/docs.
- init: registers 3 (event,matcher) pairs now, incl. PreToolUse/Bash; hookAction
flags 'update' when any is missing; uninstall strips both event arrays.
Tests (663/663): install-parse unit; before_execution policy gate (warn/deny/allow,
fail-open); PreToolUse e2e through the real shim (ask/deny/allow); init 3-registration
upgrade e2e. Verified end-to-end via `starlog hook before_execution`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…question The 3-arm eval measured the tripwire nudge vs a BARE model (+60pp) — but the real production baseline already carries the CLAUDE.md instruction Starlog ships. That A/B result proved the nudge text works, not that the hook adds value over the existing pull-surface (the honest boundary the eval doc flagged). Add two arms with an AMBIENT system prompt (simulating the CLAUDE.md guidance): D_ambient (instruction, no hook) · E_ambient_nudge (instruction + hook nudge) D→E lift is the honest go/no-go: a hook that adds nothing over the ambient instruction is redundant. Harness prints it and gates the verdict on BOTH A→B and D→E. Money-free parts covered by the stubbed-LLM test (6/6). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Member
Author
|
| Arm | adopt-library rate |
|---|---|
| A control (bare model) | 8.9% |
| B +tripwire (bare) | 65.6% |
| C +advice (bare) | 83.3% |
| D — CLAUDE.md instruction, no hook | 100% |
| E — instruction + tripwire | 98.9% |
D→E = −1.1pp. The +60pp (A→B) was nudge-vs-bare-model. But Starlog already ships the CLAUDE.md instruction, and in this single-turn probe that instruction alone gets 100% adoption — the after_file_edit tripwire adds nothing over it.
Caveat (both ways): D=100% means the probe ceilings on the instruction, so it structurally can't measure the hook's hypothesized value in a long multi-turn trajectory where instruction salience decays. So this is "could-not-prove-value + evidence-of-redundancy," not "useless." A multi-turn eval is the only way to settle it.
What this does and doesn't affect:
after_file_edittripwire (Phases 1/1.5): marginal value UNPROVEN. Kept warn-only (harmless), but should not be oversold. Multi-turn eval needed to justify.before_executionpolicy gate (Phase 2): unaffected — it's deterministic enforcement, not a nudge competing with an instruction. An instruction can't block an install; the hook can.- Phase 0 contract + the eval methodology: stand on their own.
Recommend reviewers weight Phase 0 + Phase 2 as the durable value, and treat the tripwire as experimental pending a multi-turn eval.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Integrates the Hookshot model into Starlog: a portable agent-lifecycle hook contract that turns Starlog from pull-based/advisory into a push + enforcement plane, wired into
starlog initso it fires in Claude Code today. Full design + decision log in.planning/hookshot-integration.md.Four handlers ride one contract (
starlog hook <event>, stdin{event,cwd,payload}→ stdout{decision: allow|warn|deny, message?, source}, fail-open):starlog hook <event>CLI;before_execution/prompt_submit/stopforward-compatibleafter_file_editdetectFilegate → warn once at threshold → points atstarlog_advise)before_executionDid the tripwire actually change behavior? (we measured)
Built a 3-arm decision-flip eval (
npm run eval:tripwire) with the decision rule precommitted before running (anti-confirmation-bias). Result (haiku, k=15, 90/arm):A→B = +60pp → KEEP. Honest boundary (documented): the control was a bare model, so this proves the nudge text works (upper bound), not marginal value over Starlog's existing MCP tool + CLAUDE.md — that needs a multi-turn eval. The eval also surfaced the inline-hint win (B→C +16.7pp), which Phase 1.5 acts on.
Key decisions
starlog hook <event>CLI; a thin Go/Hookshot shim (future, separate repo) fronts Cursor/Windsurf/Factory/Codex. L1/L2/L3 stays TypeScript.before_execution= dependency-introduction gating only, not general command safety.deny→permissionDecision:"deny"(hard block),warn→"ask"(surface + user decides),allow→nothing. Default warn; opt-in hard block viaSTARLOG_HOOK_ENFORCE=deny. Local-only policy resolution (no network in the blocking hot path).path/new_text; adopting it would have silently broken the feature.Testing — 663/663, typecheck clean
dispatchRaw('null')threw outside the try/catch).detectFilegate;editPayloadtranslation; install-command parser; policy gate (warn/deny/allow, fail-open); golden real-envelope e2e (all edit tools + Bash); PreToolUse e2e (ask/deny/allow);init3-registration upgrade path.starlog hookCLI and a realinit/uninstall round-trip.Notes for review
.planning/*is gitignored (design docs not in the diff).fast-checkadded as a dev-dependency (dev-only; not shipped).PostToolUseinstall-facts path is preserved;hook-runnernow self-routes byhook_event_name+tool_name.🤖 Generated with Claude Code