Restructure the agent-guidance tooling into an installable CLI package

[shunk031]

Why

The evaluator script (scripts/agent_guidance_eval.py) has grown into the shared infrastructure for three distinct jobs — guidance evaluation, calibration, and document slop review — while remaining a single ~1,500-line script. This shape has produced concrete problems, each observed during the 2026-08-14/15 hardening work:

• Internal-import coupling. The slop reviewer (scripts/doc_slop_review.py) imports the evaluator's private functions (invoke_codex, parse_trace, retry helpers) because there is no shared module boundary. Any evaluator refactor can silently break the reviewer.
• Path-based invocation. Skills and hooks must reference the scripts by checkout path, which is host-layout-dependent and cannot survive relocation.
• Interpreter incantations. The scripts require Python 3.11+, but uv run --no-project python resolves to a uv-managed 3.8 on some hosts. Today this is patched with repeated --python 3.14.6 pins in two hook entries and the Makefile, plus a runtime version guard (calibration PR).
• Mixed responsibilities in one file. Target discovery, the pass cache, failure-evidence persistence, per-case majority aggregation, trigger accounting, Codex subprocess management, and two CLI entry points live together, so every change risks the acceptance gate that guards everything else.

Proposal

Extract a proper uv project at tools/agent-guidance/ with a src/agent_guidance/ package and a single console script:

agent-guidance eval --staged --trials 3 --jobs 2 --timeout 600
agent-guidance validate --staged
agent-guidance slop-review <file|--diff|->
agent-guidance calibrate --all

Internal layout: codex_runner.py (the one definition of Codex subprocess management: model/effort arguments, ephemeral CODEX_HOME, transient retry, trace parsing — the gateway and auth themselves stay in ~/.codex/config.toml, not in this package), evaluator/ (discovery, cache, evidence, aggregation policies), slop/ (regex tier, rubric loading, judge flow), and cli.py.

Design constraints carried over from the hardening work:

• Acceptance policy stays in the hook lines. --trials 3 --jobs 2 --timeout 600 and the per-case majority behavior remain explicit in .pre-commit-config.yaml and the Makefile, because those files are the predeclared machine-policy source. The package must not bury policy in defaults.
• requires-python >= 3.11 in the package replaces the scattered interpreter pins and the runtime guard.
• uv tool install puts agent-guidance on PATH, so skills reference a stable command name instead of a checkout path.

Migration plan

1. Extract the package with the existing 90+ unit tests migrated; keep scripts/agent_guidance_eval.py and scripts/doc_slop_review.py as thin compatibility shims that delegate to the package.
2. Switch the hook entries and Makefile to the console script (separate PR — these lines are policy-owned).
3. Update the slop-review skill's command reference to the installed command; remove the shims once nothing invokes them.

Each step keeps the acceptance gate green: the gate itself runs on this code, so no step may change evaluation semantics (the pass cache will invalidate once from the runner-hash change; that is expected and harmless).

Open questions

• Package/command naming (agent-guidance vs something shorter).
• Whether calibration deserves a first-class subcommand or stays a documented invocation.
• Shim retention period.
• Conclusions from the ongoing design consultation (Herdr tab cli-design-consult) should be folded in before implementation starts.

[shunk031]

Design input from the user: consider replacing the hand-rolled Codex subprocess layer with the official Codex Python SDK (pip install openai-codex). Verified from its README: Codex() / thread_start() / thread.run() return a structured TurnResult (final response, items, token usage), existing Codex authentication is reused automatically, and workspace access control is exposed. If it also covers (1) sandbox-mode selection, (2) per-thread model/effort overrides, (3) per-trial isolation (ephemeral CODEX_HOME/eval repo), and (4) version pairing with the installed CLI, then codex_runner.py becomes a thin wrapper over the SDK instead of subprocess + trace parsing. Being evaluated in the design consultation.

[shunk031]

Proposal update: align the CLI with the Agent Skills eval workflow

Recommendation: keep the 2-PR migration, but make PR 1 define two stable contracts — the evals schema (adding a files field for input fixtures) and a durable on-disk workspace layout (iteration-N/<case>/{with_skill,without_skill}/{outputs,timing.json,grading.json} plus benchmark.json) — and make PR 2 port the runner/grader onto those contracts as subcommands (run, grade, aggregate, with analyze landing later). The pre-commit gate stays as a stricter profile chaining the same subcommands with the existing majority and negative-control policy.

The basis is the Agent Skills documentation on evaluating skill output quality, which prescribes an eval pipeline that matches what scripts/agent_guidance_eval.py already does in one interleaved script, and decomposes it more cleanly.

What the guide prescribes

• Test cases live in evals/evals.json per skill: prompt, expected_output, optional files (input fixtures), and assertions added after the first run.
• Every case runs twice — with the skill and without it, or against a snapshot of the previous skill version — in a clean context.
• Results live in the durable workspace above, with benchmark.json aggregating pass-rate/time/token means and a with-vs-without delta.
• Grading requires concrete evidence per assertion; mechanical checks go to scripts, judgment calls to an LLM; holistic quality can use a blind A/B judge.
• A pattern-analysis step audits the assertions themselves: drop assertions that always pass in both configurations, fix ones that always fail in both, and study the pass-with/fail-without set as the actual skill value.
• A human review step records per-case feedback, and an improvement loop feeds failed assertions, feedback, and transcripts back into the skill.

Why each adoption is worth it

1. Subcommand decomposition along the pipeline separates stages the current script interleaves, which is the core complaint motivating this issue.
2. The workspace contract turns in-process state into an inspectable interface between subcommands and gives skill authors an iteration history rather than only a gate verdict.
3. files in the evals schema eliminates the recurring "scenario assumes a populated checkout" eval-authoring defect at the schema level, instead of asking authors to inline material into prompts.
4. Timing/token capture adds the cost axis we currently ignore: a skill that wins assertions while tripling token cost is a different decision.
5. An analyze subcommand (always-pass-both / always-fail-both / pass-with-fail-without audit) would have mechanically caught the defective cases we found by hand during the recent calibration run.

What stays from the current design

The guide's loop is for authoring iteration; our pre-commit regression gate (per-case 2/3 majority, strict negative controls, sandbox policy) is a stricter consumer of the same pipeline, not something the guide replaces. The guide's assertion style rules (observable and countable checks good, literal-phrase matching brittle) independently confirm the anti-pattern catalog we derived from gate failures.

[shunk031]

Consultation verdict: scope split for the initial CLI

The design consultation reviewed both external references (Agent Skills evaluating-skills guide, anthropics/skills skill-creator) and settled the scope as follows.

In the initial CLI

• files input fixtures, the durable workspace layout, and a machine-readable JSON Schema for both.
• Judge/grader/comparator prompts externalized as package data files, with each prompt's digest included in provenance and cache keys (pattern: skill-creator grader.md).
• A policy-free triggers check: the existing target-read measurement becomes its own stage, and the pre-commit gate applies the per-case 2/3 majority and strict negative controls on top.
• Assertion grading and blind comparison stored as separate results (pattern: comparator.md).
• Deterministic assertion/variance/cost analysis over the workspace.

Deferred until the workspace contract is stable

• Trigger/description optimization loop (train/heldout split, 3 runs per query, up to 5 iterations, heldout selection; reference: run_loop.py).
• The unblinded improvement analyzer.
• report --static as a pure renderer over the workspace (reference: generate_review.py).

Decisions on open questions

• Existing trigger eval cases become a seed, not the optimizer's final query set: expand to roughly 20 queries and replace obviously-irrelevant negatives with near-misses. Splitting the current small set 60/40 would make the heldout unstable and overfit to the same gate cases.
• The skill-creator's Claude-specific trigger detection (temporary command file + claude -p stream parsing, run_eval.py) is not ported; the CLI keeps Codex structured traces and the existing target-read detection.

Pending user decision

The consultation also proposed a prevention for the "official guides were read only after the initial design" miss: no new prose rules, just one added eval case to shunk031-research-before-implementation asserting that designs distinguish documented workflows from representative implementations/APIs. This is not yet implemented and awaits explicit approval.

[shunk031]

Trigger measurement and policy ownership

Shuhari will keep one public command, shuhari check trigger. Internally, trigger execution will produce raw per-case, per-trial read observations before a separate policy function computes acceptance. The policy mechanism remains in Shuhari so every consuming repository gets the same majority and strict-negative semantics. Policy values such as trials, concurrency, timeout, sandbox, and strict-positive behavior remain owned by the consuming repository and must stay explicit in its pre-commit hook or other invocation.

This keeps the CLI small while preserving the important boundary: the runner does not hide repository policy in target-specific code, and negative controls always reject any observed read. We will not add separate public measure and gate subcommands in the initial release.

The adversarial review of Shuhari's initial CLI PR (#1) prompted this clarification because the earlier phrase “policy-free trigger measurement” did not identify the acceptance-policy owner.

[shunk031]

The subject of this issue no longer exists.

scripts/agent_guidance_eval.py was deleted in #672, along with tests/python/test_agent_guidance_eval.py and the two agent-guidance-* pre-commit hooks. scripts/ now holds run_benchmark.sh and run_unit_test.sh.

Each problem this issue names was resolved by that move rather than by restructuring:

• Internal-import coupling. doc_slop_review.py moved to shunk031/skills in feat(skills): remove the in-tree skills the subscription replaced #666 with its Codex invocation layer reimplemented inside the skill. It imports nothing from this repository.
• A single ~1,500-line script serving three jobs. Guidance evaluation is shuhari eval instructions, run by the two shuhari-*-instructions hooks. Skill evaluation belongs to shunk031/skills, which gates it with shuhari eval skill. Document slop review is a skill.

Ready to close unless the packaging idea is still wanted for something else.