Add the prove-the-test-fails skill

[walkerhughes]

Adds skills/prove-the-test-fails, a skill encoding one rule: a test you just wrote is not verified until you have watched it fail for the right reason.

Why

From a real session. A Python repo had two lexical retrieval strategies expected to satisfy one shared contract, and the test asserting it read:

assert {type(item) for item in bm25.search(...)} == {type(item) for item in lexical.search(...)}

Two empty lists satisfy that. The single test guarding the project's central design invariant could not fail for the reason it existed, and it had been green in CI for weeks. It was found only by deliberately breaking the implementation and noticing that nothing went red.

What the skill says

Pick the smallest mutation at the seam the test claims to guard, run the suite, confirm the right test fails and the message names the right thing, confirm the blast radius (breaking implementation A fails A's cases and leaves B's passing), then revert. It names the anti-patterns that make the exercise worthless, mutating too broadly or at the wrong layer or forgetting to revert, and the case where a surviving test is correct rather than decorative: an always-empty implementation legitimately satisfies "an empty query returns nothing".

Packaging

A top-level skills/ directory with its own marketplace entry, source: "./" plus a skills path, which is the documented pattern for several entries sharing one skills folder at the marketplace root. strict: false because the repository root has no plugin.json to be the authority. No version, so Claude Code resolves it from the commit SHA and installed copies refresh without a manual bump.

Verified end to end by installing from a local clone of this branch into a throwaway CLAUDE_CONFIG_DIR: the plugin installs, reports version 7aa20b2e69d9, and the skill lands at plugins/cache/walkerhughes/prove-the-test-fails/<sha>/skills/prove-the-test-fails/SKILL.md.

Evals

One fixture, a small retrieval library whose suite is green and whose contract test cannot fail, reproducing the defect above.

make check needs no credentials and runs in CI. It validates the frontmatter and the marketplace entry, then proves the fixture is still a trap: it applies the mutation an agent is meant to find and asserts the exact shape of the run that follows, including that the contract test survives, that only the [lexical] cases die, and that the empty-query case survives legitimately. Repairing the fixture's assertion turns it red.

make evals gives the fixture to a headless Claude Code run with the skill installed as a project skill, and grades on behaviour rather than prose: the verdict is only reachable by breaking a strategy, the tracked files must match the starting commit so any mutation was reverted, and the suite must be green again. The skill is never named in the prompt, so an automatic load also tests the description.

Honest result. --control runs the same task with the skill absent, and in the runs on record it passes too. So this is a regression guard on the guidance, not evidence of uplift, and evals/README.md says exactly that rather than implying otherwise.

Not a Harbor task, unlike the MCP plugins here. Those need a server exercised as a server against a live hub. A skill is a markdown file that has to sit in the agent's own skill directory, and the version under test is the one in the branch, which the existing tasks cannot reach because they install their subject from GitHub's default branch.

Checks run

• make check in skills/prove-the-test-fails, green.
• claude plugin validate . --strict, passes.
• make evals with --model sonnet, passes. The transcript shows the loop: two mutations, blast radius read correctly, reverted.
• Both graders verified by breaking them. Removing strict: false from the marketplace entry fails exactly one wiring check. Stub agents that leave the mutation, give the wrong verdict, or write nothing each fail exactly the corresponding grader check and no others.

🤖 Generated with Claude Code

[walkerhughes]

Generalised past the codebase that prompted it

Anything in this repo should be reusable on any project. The practice this skill states is
universal, but every illustration in it came from one Python retrieval codebase: two search
strategies agreeing, a stubbed lexical.search, a report counting [lexical] against
[bm25] cases. A reader in Go, TypeScript, Swift or Rust got a skill whose examples were
all one search engine.

What went. The two-strategies framing, the lexical/bm25 names, the stubbed search
function, and the pytest-only vocabulary in the general claims.

What the examples span now. A sentence up front says no step depends on a language or a
runner. The blast-radius step names the shape it needs in three ecosystems, because
table-driven subtests in Go, one describe per implementation in Jest and a parametrised
pytest case are three names for the same thing and each reports its cases separately. The
reporting example is now a cache contract under Vitest, keeping what made it sharp: a named
mutation, exact counts on both sides, explicit confirmation of what did not fail, and a
confirmed revert. The empty-result survivor stays, since it is still the best illustration
of a vacuous assertion, restated so it is not about a search function.

Length is roughly unchanged. This was substitution, not addition, and there is deliberately
no per-language matrix.

The fixture. Recast from two search strategies to two route matchers, a compiled-regex
one and a segment walk, over one route table. The defect is identical: a contract test
comparing sets of result types on an input that matches nothing, so it reduces to
set() == set() whichever implementation is broken. It stays one fixture in one language on
purpose; a fixture per ecosystem would cost a lot and measure the same method. evals/README.md
now says that explicitly.

Confirming the checks still bite. check_fixture.sh was run against three mutations it
exists to catch, and reverted after each:

Mutation	Result
Contract test given a path that actually matches	red, "the fixture's contract test can fail now"
segment_router delegating to regex_router	red, "the mutation leaked into the other implementation"
The anchored docstring reworded	red, "regex_router no longer looks the way this check expects"

check_wiring.py was checked the same way by dropping the marketplace entry; it went red and
was restored. make check is green with everything reverted.

Eval results. Both runs pass on the recast fixture, verdict UNGUARDED, tracked files
unchanged, suite green: make evals with the skill, and run_eval.sh --control without it.

The honest negative stands. The control still passes, so this remains a regression guard
on the guidance rather than evidence of uplift, and evals/README.md still says so in those
words. The recast did not raise the difficulty enough to change that, and I am not claiming
it did.

Untouched shared files. .claude-plugin/marketplace.json and README.md are not in this
diff: the description did not change, so the marketplace copy still matches. A sibling
branch generalising another skill may touch both, so this branch should not conflict there.

[walkerhughes]

Rewritten as principle and workflow

The file read as a procedure manual. It is now the principle, the workflow that principle
implies, and one example of each kind. Body: 84 lines to 47.

Cut

• The opening inventory of seven test runners. The point that the method is runner-agnostic
  survives by simply never naming a runner.
• The "when to run this" section. It restated the description frontmatter, which is what
  actually decides whether the skill loads.
• The "choosing the mutation" catalogue. Return empty, invert a comparison, drop a filter:
  three spellings of "smallest change at the seam", which the workflow step already says.
• The five-item anti-pattern list. Four of the five were the inverse of a step that was
  already there, so they added length without adding a decision.
• The dedicated section on legitimate survivors. The idea earns a place, the section did
  not; it is now one clause in the opening.

Kept, and why each earns it

• The loop. A workflow is skill content, not minutiae, and this is the workflow the skill
  exists to convey. Compressed, with the sub-catalogues under each step removed.
• Mutate at the seam the test claims to guard. The principle alone does not tell an agent
  where to break, and breaking the wrong layer yields a red run that is not evidence.
• Blast radius. "Watch it fail" does not by itself say to check what stayed green, which
  is the half that catches a test measuring something other than its name.
• Revert, always. Load-bearing. Nothing else in the skill prevents leaving a repository
  broken.
• One anti-pattern. The contract test comparing derived collections on an input that
  matches nothing, so it reduces to two empty sets. Sharpest illustration of green that
  guards nothing.
• One gold standard. The report naming the mutation, the cases that failed, the cases
  that did not, and the confirmed revert, against "50 tests pass".

One line was added rather than cut: the loop takes one mutation at a time, so independent
seams are independent runs and the work fans out across working copies. That is a true
property of the workflow and it is where this skill would grow an orchestration story.

Overfitting

No project names, session references, issue numbers, ports, or component names. No
language, runner, or framework is named anywhere in the body.

Wiring check

No retargeting needed. evals/check_wiring.py already asserts structure only: frontmatter
parses, name matches the directory, description is long enough to trigger on, and
exactly one marketplace entry loads the skill with the right source, strict, and
description. Nothing in it is pinned to prose. The description frontmatter is unchanged,
so marketplace.json and README.md are untouched.

Verification

make check passes: wiring plus the fixture-honesty check, which still reports the exact
mutation shape it asserts.

make evals passes, run twice against this version of the file. Both runs reached
UNGUARDED, reverted every mutation, and left the suite green. The second run found the
vacuous assertion by two independent routes and separately noticed that the fixture's
ordering rule is unreachable rather than merely untested.

evals/README.md is unchanged, including its admission that the control passes too, so
this remains a regression guard and not a measurement of uplift.