[Feature] Skill effectiveness evaluation harness

[KbWen]

Problem (capability gap)

Skill value is currently asserted via description and popularity, not measured. There is no baseline/repeated-run evidence of a skill's effect on quality, cost, or trigger accuracy.

Acceptance direction

• An evaluation harness with: skill + baseline digests, assertions frozen before execution, model/harness/repository/config identity, isolation mode, real token/time/tool/error metrics (or explicit unknown), repeated runs + aggregation, held-out near-miss trigger tests, and separate human feedback.
• Kept as a separate suite from the governance eval; reuse governance-eval primitives only where semantics match.

Dependencies

Depends on stable task identities and artifact boundaries - backlog #77 and #78.

Scope guards / non-goals

• Must not absorb or rewrite the existing governance eval.
• Popularity, curation, "official" labels, or author benchmarks are not effectiveness evidence.

---

Surfaced by the 2026-06-19 external skill/workflow-practices research. Recording phase only. Tracked as backlog #79.

[KbWen]

Triage Analysis — #254 Skill effectiveness evaluation harness

Classification: feature (new eval suite + runner + cross-platform validator wiring + deploy manifest changes). Leaving a structured analysis rather than a direct fix per triage policy — this is blocked by two open issues and needs a design decision about metric provenance.

1. Priority Assessment

Backlog row #79, priority P2, dependencies #77 → #252 and #78 → #253 — both still Pending (docs/specs/_product-backlog.md:46-48). So the issue as written is genuinely gated, not merely sequenced.

What raises the value beyond P2 is a fact the existing eval already reports about itself: AGENTS.md §Skill Activation Triggers is currently a zero-coverage MUST-rule. Executed, not read:

$ python .agentcortex/tools/run_governance_eval.py --coverage
Rule inventory: 45 MUST-bearing section(s) across governance files
Cases evaluated: 28
Zero-coverage rules: 28

Rules with zero guarding cases:
  - AGENTS.md §Skill Activation Triggers
  - AGENTS.md §Shared Phase Contracts
  ...

The repo ships 14 skills (.agents/skills/*/SKILL.md) and a 16-entry trigger registry, and has no behavioral evidence for any of them — neither effectiveness nor activation correctness. The trigger-accuracy half of this issue is the part that closes a live governance blind spot; the cost/quality-delta half is the expensive, dependency-gated part.

2. Root Cause Analysis — what exists vs. what the AC needs

The governance eval provides a real, reusable substrate:

• .agentcortex/eval/governance.yaml — 28 cases; data-only, never loaded by a runtime workflow.
• .agentcortex/tools/run_governance_eval.py (475 lines) — case scoring (_score_case:136: expect_substrings / forbid_substrings / regex_present / regex_absent / max_lines), a live-agent runner (_run_agent:197, shlex.split on the template only, shell=False — safe against prompt-borne shell injection), secret redaction (_sanitize_diagnostic:182), JSON output, and a coverage inventory (_run_coverage:248).
• Already wired: WARN-level coverage check at validate.sh:2848 / validate.ps1:2697, and shipped downstream via the deploy.sh:739 runtime-tool whitelist.

Against the AC, the following are absent — verified in the runner's CLI and result shape:

AC element	Status today
skill + baseline digests (A/B pairing)	missing — no concept of a skill-off control run; cases are single-condition
repeated runs + aggregation	missing — one transcript per case id; _build_summary:297 counts only pass/fail/skipped/error
real token/time/tool/error metrics	missing — _run_agent returns (stdout, returncode, diagnostic) only; no timing, no token accounting
model / harness / repo / config identity	missing — nothing in the JSON output records model id, commit SHA, or .agent/config.yaml state
isolation mode	missing — no worktree/sandbox notion in the runner
held-out near-miss trigger tests	missing, but the input data already exists (see below)
separate human feedback	missing

The near-miss trigger tests are the outlier: .agentcortex/metadata/trigger-registry.yaml already encodes per-skill activation contracts (detect_by.intent_patterns, classification, phase_scope, load_policy, block_if_missed) — e.g. test-driven-development at :71-97. Held-out positives/near-misses can be authored and scored against that contract deterministically, with no A/B agent runs and no dependency on #77/#78. query_trigger_metadata.py and validate_trigger_metadata.py are the existing readers.

Identity has a partial primitive too: resolve_skill_lockfile.py (over trigger_runtime_core.build_skill_registry_snapshot) already produces a pinned skill snapshot — that is the skill half of run identity; model/harness/commit still need capturing.

3. Proposed Approach

Split into one unblocked increment and one gated increment. Do not ship them together.

Increment A — trigger-accuracy suite (unblocked, deterministic, recommend first).

1. New data file .agentcortex/eval/skills.yaml — separate from governance.yaml per the issue's scope guard. Cases: {skill_id, prompt, classification, phase, expect_activation: true|false}, with deliberate near-miss negatives per skill (paraphrases that must not trigger).
2. New runner .agentcortex/tools/run_skill_eval.py. Reuse _score_case-shaped assertions by extracting the shared scoring helpers into a small module rather than importing the governance runner wholesale (keeps the "must not absorb the governance eval" guard honest).
3. Freeze assertions before execution (AC requirement) — the YAML is the frozen artifact; the runner must never mutate it.
4. Emit a run-identity header in --format json: commit SHA, resolve_skill_lockfile.py snapshot digest, model/harness string supplied by the caller, and unknown where a value is genuinely unavailable (the AC explicitly permits explicit-unknown; silent omission is not acceptable).

Increment B — effectiveness measurement (gated on #252/#253).
5. A/B pairing: each case runs twice (skill loaded / skill withheld) under --isolation worktree, N repeats with aggregation (median + spread), quality scored by the frozen assertions and cost by real metrics.
6. Metrics provenance is the open design question — --agent-cmd captures stdout only, so token/time must come from a harness that reports them (e.g. a JSON-output mode of the driving CLI) or be recorded as unknown. Do not synthesize estimates; analyze_token_lifecycle.py produces static token estimates and must not be conflated with measured runtime cost.
7. Human feedback stays a separate, clearly-labelled channel — never averaged into the machine score.

Wiring note for whichever increment lands first (repo-specific traps, both previously bit us):

• A new validator check must be a Python tool invoked via run_python_check / Invoke-PythonCheck in both validate.sh and validate.ps1 (ADR-006 native-ratchet fails on a raw record_result / Add-Result).
• Any new shipped tool must be added to both deploy.sh whitelist spots (:739 and :946) and tests/ci/fixtures/deploy_manifest_golden.txt, or test_deploy_tiering.py fails.

4. Risk & Trade-offs

• WARN numbness (highest process risk). The governance eval's own coverage check already emits a standing WARN with 28 zero-coverage rules. A second suite that adds a second always-on WARN degrades both signals. Mitigation: the skill suite should be on-demand (invoked explicitly / in a dedicated job), not another line in the default validate summary — or, if surfaced there, it must be gated on a real regression rather than a coverage count.
• Measuring the wrong thing. With no reliable token/time source, "effectiveness" collapses into substring assertions — which is trigger correctness, not effectiveness. That is exactly why Increment A is honest to ship alone and Increment B is not. Naming must not overclaim: Increment A is a trigger-accuracy suite, and the issue title should not be marked satisfied by it.
• Non-determinism. Live-agent runs are stochastic; a single-run pass/fail will flap. Repeated runs + aggregation are mandatory before any result is treated as evidence — no N=1 claims in the Work Log.
• Scope-guard erosion. Reusing governance-eval primitives is permitted "only where semantics match"; a refactor that makes the governance runner generic risks changing its behavior. Mitigation: extract-and-keep-behavior with the existing governance eval re-run as the regression proof.
• Cost. Increment B burns real model tokens per case × conditions × repeats. It needs an explicit budget cap in the runner, defaulting small.
• Rollback. Increment A is purely additive (new YAML + new tool + optional check) → revert is a clean branch revert; nothing in the runtime path depends on it. Increment B inherits whatever task/artifact identity [Architecture] Reviewable Task -> Step hierarchy #252/[Feature] Task capsule + one read-only reviewer (two verdicts) #253 define, so its rollback surface is larger — another reason to keep the two increments in separate PRs.

5. Estimated Scope

• Increment A: ~4–5 files (.agentcortex/eval/skills.yaml, .agentcortex/tools/run_skill_eval.py, shared-scoring extraction in run_governance_eval.py, deploy.sh + deploy_manifest_golden.txt, plus tests) → feature (small). Independently shippable today.
• Increment B: 6+ files plus an isolation/metrics design decision → feature, and worth an ADR if the metrics contract (what counts as a measured token, what counts as unknown) becomes load-bearing for future claims.
• Overall issue classification: feature, matching the backlog row.

6. Blocked By

• #252 (backlog Fix R-1 (BLOCKER): wire ADR-003 hash chain into /ship INDEX.jsonl append #77 — Reviewable Task→Step hierarchy) — supplies the stable task identity that per-task metric attribution requires.
• #253 (backlog Fix R-2 + R-3: bootstrap §0 tiny-fix ADR-coverage guard + §0a Python-unavailable fallback #78 — Task capsule + read-only reviewer) — supplies the artifact boundary that makes a skill-on/skill-off diff comparable.

Both gate Increment B only. Increment A depends on neither and can proceed independently.

[KbWen]

Split: the trigger-accuracy half is now #398 / backlog #165

Following the triage analysis above, the unblocked half of this issue has been split out rather than left waiting on #252/#253:

• #398 (backlog docs: backlog issue-sync + verified 1-2 month optimization roadmap #165) — skill trigger-accuracy suite. Deterministic against the activation contract in trigger-registry.yaml; depends on neither Fix R-1 (BLOCKER): wire ADR-003 hash chain into /ship INDEX.jsonl append #77 nor Fix R-2 + R-3: bootstrap §0 tiny-fix ADR-coverage guard + §0a Python-unavailable fallback #78.
• This issue ([Feature] Skill effectiveness evaluation harness #254, backlog Fix R-4: ADR-001 lifecycle + ADR-002 applies_to self-backfill (close the irony) #79) — narrowed to the effectiveness half: A/B skill-on vs skill-off pairing, repeated-run aggregation, isolation mode, measured token/time/tool metrics, human-feedback channel. Still genuinely gated on [Architecture] Reviewable Task -> Step hierarchy #252/[Feature] Task capsule + one read-only reviewer (two verdicts) #253.

Shipped in PR #399 + #400.

One correction to the triage analysis above, found by a refute pass and worth recording here so it is not repeated: the claim that the shipped skills have zero behavioral evidence of correct activation was overstated. .agentcortex/tools/trigger_runtime_core.py already implements values_match:860 / skill_is_candidate:884 / skill_is_activated:918, and .agentcortex/tests/test_lifecycle_skill_activation.py collects 53 tests over registry-driven activation. The genuinely uncovered surface is narrower: free-text prompt → intent_patterns matching (values_match is only ever fed explicit manual_skills lists, never natural language) plus near-miss negatives. #398 carries the corrected scope and an explicit warning against building a second resolver.

Landing #398 does not close this issue.