feat(verify-first): cover probes that pass for the wrong reason

[AndreJorgeLopes]

TL;DR

verify-first was thorough about false negatives and silent about their inverse. It now covers probes that pass for the wrong reason: liveness signals mistaken for correctness signals, and detectors that report nothing because they never ran.

Why

The skill's founding incident was a false negative (window.Relay undefined did not mean Relay was gone), and both of its stated lessons point that way. Nothing in it guarded the opposite direction, which is easier to miss precisely because it feels like success: the probe passes, the claim is marked [V], and the thing you cared about was broken all along.

The incident that prompted this, 2026-07-30: a skill whose only job was to hand a reviewer a running app was recorded as "fully verified end to end" on the strength of an HTTP 200 plus a <title> tag. The app was actually broken in the browser, does not provide an export named 'AIAgent', because a stale generated module never resolved. A vite dev server returns 200 for the HTML shell even when every module it references fails, so that probe could not have failed for the reason that mattered. A human opening the page found it in seconds.

What changed

Change	Where	Purpose
New section, "The inverse failure: a probe that passes for the wrong reason"	after the founding-incident section	gives the false positive the same treatment the false negative already had
Liveness vs correctness table	same section	a status code, a bound port, a live PID, exit code 0 and a startup banner each prove something started, never that it works
"Detectors fail silent-clean"	same section	"found no problems" and "never ran" are byte-identical output, so a green detector is evidence only after you have watched it go red on a known-bad fixture
Lesson 3: match claim precision to probe precision	same section	if all you polled was the port, the claim is "the server is up", not "the app works"
Falsifiability question added to Step 3	Step 3	before running a probe, ask whether it would look any different if the thing were broken; if not, it is the wrong probe
5 new anti-pattern rows	Anti-patterns table	the 200, the zero-violation scan, the bound port, the exit 0, and the passing probe laundered into [V]
Frontmatter description	frontmatter	names the new failure mode so the skill triggers on it

Verification

Check	Result
make test (skills-check, flows-check, smoke)	exit 0, 3 PASS, no error or fail lines
Generated plugin copies in sync	make skills-sync run and committed; skills-check passes, and only the three verify-first files changed, no collateral churn
make determinism SKILL=verify-first	ran on this branch AND on main for comparison: both 1 passed (100%), 0 failed, 0 errors, both emitting a real Verification Ledger with the pinned [V]/[B]/[R] table against the throwaway fixture, 44s here vs 51s on main

The gate asserts only the ledger shape, which this change preserves, and running it against both branches confirms the edit is inert to it rather than assuming so.

Worth knowing for anyone reading the gate output: promptfoo prints Could not find any valid files in the command: bash eval-run.sh on both branches. It is a cosmetic warning from globbing the provider string as a path before falling back to exec:, not a failure. The 44s duration and the real ledger in the output are what establish that the skill actually ran, and that distinction is the whole subject of this PR.

Note for reviewers

feat/eval-testmode (2 commits ahead, 19 behind, no PR) also modifies skills/verify-first/SKILL.md. It will conflict here and needs a rebase regardless of this change. Flagging rather than resolving, since it is not my branch.

Risk

Low. Additive prose to one skill plus its generated copies. No script, flow, or CLI behaviour changes.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.