A silently deleted test fails no gate — #1951 deleted a #1793 witness and nothing objected

[macanderson]

What happened

#1945 landed the two witnesses for #1793 (the mid-turn flip halt) plus the two doubles they share. Hours later #1951 — a PR about per-candidate verifier degradation — rewrote crates/stella-pipeline/src/pipeline/tests/verification_hardening.rs wholesale and dropped three of those items:

• a_revision_halts_at_the_step_where_the_tracked_test_flips — the configured-command witness for FlipHalt never arms on the authored-witness path, and revise turns pass None even for configured commands #1793
• PassingShell and shell_call_result — the doubles both witnesses need

Confirmed by git log -S:

$ git log --oneline -S"PassingShell" --all
2a142b26 fix(stella-pipeline,stella-protocol): ... (#1787) (#1951)   <- removed
eddf9700 fix(stella-pipeline): arm FlipHalt ... (#1793) (#1945)      <- added

#1951 had no reason to touch that file's flip-halt content at all — a same-seam clobber of the kind AGENTS.md already warns about.

Why nothing caught it

Deleting a test is invisible to every gate. file-size, left-behind, god-files, clippy, and the test suite all pass a tree with fewer tests in it — the suite is green because the test is gone.

Here it was masked twice over: the crate stopped compiling (E0425 for the two missing doubles), and a compile error in the lib masks the test target entirely, so the whole thing read as one build failure. #1793 shipped with half its witness silently absent, and the only reason anyone noticed is that a different merge (#1953) broke the build in a way that forced someone to read the file. Restored in PR #1971.

Why it matters here specifically

This repo's contract is "verified done, not claimed done". A witness test that can be deleted by an unrelated PR, with no gate objecting, means the evidence for a shipped fix has a weaker retention guarantee than the fix itself. #1793 is not special — any witness is deletable this way.

Possible directions (design decision, not a spec)

1. A test-count ratchet. Cheapest and dumbest: record per-crate #[test]/#[tokio::test] counts in a generated baseline like scripts/file-size-baseline.txt, and fail when a count drops without the baseline being regenerated in the same commit. Catches this exact case; noisy under legitimate consolidation, and the escape hatch (make test-count-update) is one command, so the cost is a reviewable diff rather than a block. Mirrors machinery the repo already has and trusts.
2. Named-witness registry. Stronger and narrower: let a PR closing an issue declare its witness by name, and gate on that named test continuing to exist. Ties directly to the witness contract and would have named a_revision_halts_at_the_step_where_the_tracked_test_flips as protected. More machinery; needs a decision on where the registry lives and who maintains it. crates/stella-parity already does something adjacent — every capability names a witness test per surface, and its tests fail when a named witness disappears. Read that first; this may be an extension of it rather than a new mechanism.
3. Accept and mitigate. Do nothing mechanical; rely on the co-location fix PR fix(stella-pipeline): unbreak main — four breaks two merges left behind, incl. a silently deleted #1793 witness #1971 already applied (both witnesses and their doubles in one file, so a wholesale rewrite is a merge conflict rather than a silent deletion). Cheapest, but only protects files someone has already thought about.

What "done" looks like

Either a gate that fails on the deletion demonstrated below, or a written decision on this issue recording that option 3 was chosen and why.

Repro / verify

git show eddf9700:crates/stella-pipeline/src/pipeline/tests/verification_hardening.rs \
  | rg -n "a_revision_halts_at_the_step_where_the_tracked_test_flips"   # present
git show 2a142b26:crates/stella-pipeline/src/pipeline/tests/verification_hardening.rs \
  | rg -n "a_revision_halts_at_the_step_where_the_tracked_test_flips"   # gone

A candidate gate must fail on the second tree. Note make gate on that tree fails for the compile error, not the deletion — the test to write is one where a test is deleted and nothing else is wrong.

Files

• scripts/ — where a new guard would live, alongside check-file-size.sh / check-left-behind.sh
• Makefile — GATE_STEPS. Adding a step is five coordinated edits: GATE_STEPS, the AGENTS.md gate block, CONTRIBUTING.md's copy, and scripts/check-gate-parity.sh enforces they match
• crates/stella-parity/ — prior art for named-witness enforcement; read before choosing option 2
• crates/stella-pipeline/src/pipeline/tests/verification_hardening/flip_halt_arming.rs — where the restored witnesses now live

Related

• PR fix(stella-pipeline): unbreak main — four breaks two merges left behind, incl. a silently deleted #1793 witness #1971 — the restore and the co-location mitigation
• FlipHalt never arms on the authored-witness path, and revise turns pass None even for configured commands #1793, fix(stella-pipeline): arm FlipHalt on the authored-witness path and in revise turns (#1793) #1945, fix(stella-pipeline,stella-protocol): record verifier degradation per candidate in a fan-out (#1787) #1951 — the fix, the witness, and the clobber