doctor answers for HEAD; check 18 publishes a reproducible figure; check 50 binds CI jobs to the release gate - #4
Merged
Conversation
…aces one
CHANGELOG.md cited "80 samples" for skill-collision suppression and "a uniform
0/5 across five fixtures" as the motivation for the 1.43.0 replay log. Neither
can be re-read. There is no 80-sample runlog in git history or on this machine;
the only collision arm on record is 55 samples on col-11 alone
(tests/README.md:188) and its instrument was an uncommitted edit to
tests/dispatch-fleet.sh, which that file's own fence comment at line 180 already
said. The 0/5 figure is worse than unsourced: at dispatch-fleet.sh:678-688 only
kind=="skill" and kind=="none" fixtures get a numeric k, so a CHAIN: or
AMBIGUOUS: fixture returns {k: null} and can never print a fraction. Two of the
five fixtures in that arm are exactly those kinds.
Both lines keep their place with a retraction note attached rather than being
deleted, the way tests/evals/RESULTS.md keeps its retracted run. The direction
each arm reported stands; the n is withdrawn from both.
tests/evals/collision/PREREGISTRATION.md registers the replacement before it
runs: 5 arms x n=5 = 25 calls, matched pairs so a silent skill can be told apart
from a suppressed one, thresholds and void conditions fixed in advance, fixtures
pinned by sha256 rather than by path, dispatch-fleet.sh not edited.
Also: docs/what-this-actually-does.md claimed 44 checks and falsifiability
totals of 60 and 61 declared; the tree declares 48 and 73. The old figures are
marked superseded with their run dates rather than replaced by a number nobody
re-ran for that document.
Gate: 48 declared, 47 ran, 1 skipped, VERIFIED, rc=0. The skip names its
dependency (v1.47.0 is not tagged yet, so there is no payload to compare the
README pin against; the pin moves in the release commit, as at 4a89761).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gh run list --branch main --limit 1 --json conclusion,status,... --branch main is a moving reference and --limit 1 takes whatever is newest under it. The projection carried no headSha at all, so nothing downstream could have filtered by commit even if it had wanted to. Standing on a commit whose CI had never run, or had failed, printed CI (main: an OLDER commit that passed) ✔ a true statement about a different commit. --limit 1 also picked one workflow arbitrarily when several run per commit: this repo runs verify and release, and on 2026-08-27 release completed with a failure while verify was still going, so which one spoke for the commit depended on timing. release.yml's own header warns against exactly this -- "a moving branch ref answers 'is the newest thing on this branch green', which silently drifts to a different commit" -- and the check whose job is to stop a release going out over red CI reproduced it. Three defects reached main in one day because nobody read a remote verdict, and this is the check that was supposed to catch that. It reads every run recorded against HEAD now, and orders the answer: a decided failure outranks any number still going, in-progress is a note, and no run for this commit is a note rather than a pass. An unpushed commit has no CI verdict; the newest run on the branch is not one. Check 49 gates the decision offline through a gh stub, five cases, both directions. The network call stays in bin/doctor -- verify.sh is hermetic by design, which is why the CI and release-reachability questions live there. What belongs in the gate is the decision, and row 49 deletes the commit filter to prove check 49 can fail. On a branch, not on main: a peer session holds uncommitted edits to README.md and docs/what-this-actually-does.md in the shared checkout, and two of the three files carrying the "48 checks" count are theirs right now. Committing those paths would have taken their work with it. verify.sh 49 declared, 48 ran, 1 skipped, VERIFIED (the skip is check 24's inter-release state: 1.47.0 declared, not yet tagged). inventory-contract 32 checks clean, digest ec03bbd7. tests/repro/ci-lane-answers-for-head.sh watched red on 2 of 5 cases first.
25 samples under thresholds committed before the first one, harness unedited, runlog committed. swarm fired 0/5 on its clean positive fixture and 0/5 on its collision fixture; principle-encode-lessons-in-structure did the same; col-01, whose prompt is a literal trigger string in two competing descriptions, fired interrogate once in five. One Skill call in 25 samples. H-C2 falsified for both skills, H-C3 confirmed for both. KEEP_WORKDIRS=1 kept all 25 transcripts and they name the mechanism. Every sample runs in an empty mktemp -d, so a prompt referring to a repository, a diff or a prior turn has no referent: all five pos-19 samples globbed, found no packages and asked which repository held them. The fence denies Write, so a skill whose action is to write a file gets as far as the attempt: all five pos-11 samples called Write and were refused. The routing was never missing - the model named swarm in prose without calling it in 7 of 10 swarm samples. So one mechanism explains both arms of the anomaly and it is the instrument. dispatch-fleet.sh records fired=[] identically for "did not route" and "asked which repository you meant", and deleted the transcripts that tell them apart until KEEP_WORKDIRS existed. H11 is confirmed wider than registered: this arm picked principle-encode-lessons-in-structure as the fence-immune control because its output is prose, and 5 of 5 samples disproved that by calling Write. H-C1's threshold was mis-specified in the pre-registration written to prevent that. A bare bound on one arm with no control clause returns CONFIRM on any run where nothing fires. col-01 met it and it means nothing. Fourteenth instance of the catalogued defect, in the instrument built to measure the thirteenth. docs/research/fake-greens-2026-08.md reports the whole evidence base. A code-reviewer pass caught four wrong figures in the draft: 18/18 not 15/15 for guard-quote-aware-split.sh, 4 SWE-bench instances not 3, six catalogue entries in the 2026-08-26/27 window not seven, and the sub-shape covers two entries not three. It also found docs/research/ is excluded by construction from check 12 and check 38, so neither gate would have caught any of them. Also: v1.46.0 was tagged locally and never pushed, which is the sole reason CI was red on main for three commits. Pushed; it names origin/main's own head so it published no commits. The committed runlog carries one edit, recorded in RESULTS.md with the pre-edit sha256: the header's fixtures= path rewritten to ~/ for the home-path scanner. The harness fix that would remove the need for it is deliberately not in this commit. Gate: 48 declared, 47 ran, 1 skipped, VERIFIED, rc=0. The skip names its dependency: v1.47.0 is not tagged yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
resolve_vstack_repo() prefers ~/.config/agents/vstack-repo -- wherever install.sh last ran from -- over the location of bin/doctor itself. So the harness computed HEAD and wrote its gh stub for the tree under test, while doctor answered about whatever tree the machine had installed. Both agreed for exactly as long as those were the same directory. Running the check from an isolated clone was enough to separate them: three of five cases went red, including both positive controls. Pin doctor with VSTACK_DIR so the check grades the copy it set up. Same shape as every other entry in the catalogue: the test could not disagree with itself because the environment was arranged so the two halves were one thing.
macos-latest no longer ships shellcheck preinstalled, so check 29 skipped and require-no-unexpected-skips.sh turned the lane red. That audit worked. Widening the approved-skip list would trade the primary platform's only static analysis for a green. Install it, and fail by name if the formula does not resolve.
The session hook splices three environment strings into the block check 18 measures: $root twice, $branch once, $base three times. The cap lane was normalized for the first two and explicitly left $base in, reasoning that the remote's default branch "does not vary with this checkout." It varies with the remote, which is no less environmental. The published-figure lane was never normalized at all and went on comparing README against the raw count. Measured: 4077 B at a 25-character checkout path on main -> 3.9 KB, inside the 0.15 KB tolerance. 4163 B in a clone three characters longer whose origin/HEAD named a 24-character branch -> 4.1 KB, outside it. Same commit, same bytes of prose. The gate was green on one directory and red everywhere else, which is how it stayed green while main was red for anyone who cloned it. Subtract all three splices. The invariant count is 3990 B in all three checkouts tested, and that is the number README publishes -- one a reader can reproduce. Cap re-derived from the new worst case (3990 + 320 + 80 + 240 = 4630, +25%, rounded to 5888). Regression first: the existing path-invariance control now varies the remote default branch as well as the path length, and went red at 4423 vs 4600 B before the fix. Row 18d splices $base a fourth time so the correction can never fall behind what the hook adds. Rows 18/18b/18c/18d/49 all falsifiable. Gate green from an isolated clone: 49 declared, 48 ran.
release.yml's REQUIRED_CHECKS is a hand-maintained list of four job names sitting beside a workflow that defines four jobs. Nothing connected the two. Add a lane to verify.yml and it can be red on every commit while the release publishes over it -- which is not hypothetical: install-macos was added, went red on its first run, and the failure was found by reading a job log by hand. Both directions, because each is a different lie. A job missing from REQUIRED_CHECKS is an unread verdict. A name in REQUIRED_CHECKS with no job behind it never produces a run at all, so require-checks-green.sh reports MISSING forever -- the release deadlock fixed earlier in this session, reachable again by a typo in a list. Scoped to the `jobs:` block rather than to indentation, because `on:` carries `push:` and `pull_request:` at the same indent and would otherwise be counted as CI jobs. Both selectors refuse to be empty: an empty list makes both comparisons vacuously true, which is the shape this repo's own catalogue exists to name. Rows 50 and 50b falsifiable, both watched red first (install-freebsd unread, install-plan9 phantom). Gate green at 50 checks, 49 ran, 1 skipped.
… nobody could run cleanup-on-failed-gate force-deletes a candidate tag from origin. Its entire rule lived in a GitHub Actions `if:` expression, which no test can execute -- so the most destructive step in this repository was also its least verified one. It was wrong in production on 2026-08-27: it deleted the tag for a gate that was UNDECIDED rather than failed, and since verify cannot go green until the tag is on origin, no tag could survive long enough to earn the green it needed. Move the rule into .github/scripts/should-delete-candidate-tag.sh, exit 0/10/2 so a crash can never be mistaken for a verdict. The job's `if:` is now deliberately BROADER than the rule and carries no part of it -- broad is safe (a job that declines to delete costs a runner minute), narrow would silently skip deletions the rule requires. tests/release-cleanup.sh is the truth table: the undecided carve-out keeps the tag, and all five shapes of an actual decision still delete, so "a failed required job cannot produce a published tag" is pinned rather than asserted. Two join assertions, because a tested decider the workflow does not call is a test of nothing: release.yml must invoke the script, and the job's `if:` must not mention the gate verdict. Check 51 runs that table both ways -- against the real decider, which must pass, and against a copy with the carve-out deleted, which must fail. It also refuses if the control mutation changed nothing, so the both-directions claim cannot go vacuous when the script is reworded. Rows 51 (restore the production defect) and 51b (disconnect the workflow from the script) both falsifiable. Gate: 51 declared, 50 ran, 1 skipped, VERIFIED.
Both lanes ran `./.claude/verify.sh | tee "$RUNNER_TEMP/gate.txt"`. GitHub's default shell for a run: block is `bash -e` with no pipefail, so the step's exit status is tee's, and tee succeeds whatever the gate did. Measured both directions against a seeded gate that prints FAIL and exits 1: `bash -e -c 'gate | tee f'` exits 0, the same command unpiped exits 1. And .github/scripts/require-no-unexpected-skips.sh, the only downstream step that reads the captured log, exits 0 on a log full of FAIL lines because it inspects skip lines and nothing else. A failing gate on either lane passed the job, passed the audit, and left no trace. This is the rule this repository wrote down after a local `./verify.sh | tail` produced a false green, applied everywhere except the two lanes that needed it. Both now redirect to a file, capture the status with `|| rc=$?` so -e cannot exit before the code is read, print the log, and exit "$rc" on its own line. Found because the macOS lane also stopped linting: macos-latest no longer ships shellcheck, check 29 skipped, and the lane's approved-skip list did not cover it. That audit is what surfaced this. Fixed by installing shellcheck rather than approving the skip, matching the alpine lane, whose comment already says a check that skips is measuring nothing. Approving it would have left 71 scripts unlinted on the one platform whose BSD tools this job exists to exercise, lane still green. Gate: 48 declared, 47 ran, 1 skipped, VERIFIED, rc=0. Falsifiability: 73 declared, 72 passed, 0 failed, 1 skipped, FALSIFIABLE, rc=0. Both skips name the same dependency: v1.47.0 is not tagged yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tests/bin-scripts.sh prints "38 passed, 0 failed" and nothing had asked whether it can print anything else. That is not an abstract worry about this file: it shipped claiming it never reaches the real `claude` CLI while two of its cases did exactly that, and the claim held until a poison stub was added earlier today. Check 52 runs its bg-args case both ways against a two-file copy -- unmutated it must pass, with claude-bg.sh's no-args guard deleted it must fail -- and refuses if the control mutation changed nothing, so the claim cannot go vacuous when the guard is reworded. About two seconds. Row 52 deletes the same guard in the tree. tests/install-matrix.sh gets no equivalent, and that is stated rather than skipped, in the check's own comment and in docs/checks-that-inherit-their-answer.md. Its cheapest single case measured 2m10s, which does not belong in an offline gate, and it is red right now for an unrelated reason (v1.47.0 declared but untagged), so a must-fail-when-broken control would pass without measuring anything. A vacuous control is worse than an absent one: the absent one is visible in a paragraph, the vacuous one reads as coverage. Gate: 52 declared, 51 ran, 1 skipped, VERIFIED.
…main Two sessions fixed the same regression independently and the merge kept both, leaving two identically-named steps installing the same formula. Theirs is the more specific write-up (it names the 71 scripts the lane stopped linting), so mine goes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four defects, each found by reading a verdict rather than by inspection, each with a regression watched red before the fix.
bin/doctor's CI lane answered for the branch, not the commit (ae6129f).gh run list --branch main --limit 1reads a moving reference, so an older commit's green spoke for the one you were standing on. Check 49 + row 49.Check 49 graded a different checkout than it stubbed (
cc35ae0).resolve_vstack_repo()prefers~/.config/agents/vstack-repoover the location ofbin/doctoritself. The harness stubbed this tree's HEAD while doctor answered about the installed one; they agreed only for as long as they were the same directory. Running from an isolated clone turned three of five cases red, including both positive controls.The macOS lane lost
shellcheck(736d49c).macos-lateststopped shipping it, check 29 skipped, and the lane's skip audit correctly called that a regression. Installing it rather than widening the approved-skip list: the primary platform's only static analysis is not something to trade for a green.Check 18 published a figure only one directory could reproduce (
0cba262). The session hook splices$roottwice,$branchonce and$basethree times into the block check 18 measures. The cap lane normalized the first two and left$basein, on the stated grounds that the remote's default branch "does not vary with this checkout" — it varies with the remote. The published-figure lane was never normalized at all. Measured 4077 B on a 25-character path onmain(3.9 KB, inside tolerance) and 4163 B in a clone three characters longer whoseorigin/HEADnamed a 24-character branch (4.1 KB, outside it). Same commit. Now subtracts all three and publishes the invariant count: 3990 B in every checkout tested. Row 18d splices$basea fourth time so the correction cannot fall behind the hook again.Check 50: a CI job nothing requires is a verdict nobody reads (
ff8bea5).REQUIRED_CHECKSwas a hand-maintained list beside a workflow that defines the jobs, with nothing connecting them. Both directions: a job missing from the list is an unread verdict, a name with no job behind it leaves the release gate UNDECIDED forever. Rows 50 and 50b.Gate: 50 declared, 49 ran, 1 skipped (check 24, the honest inter-release skip), VERIFIED — run from an isolated clone, not the authoring checkout. Rows 18/18b/18c/18d/49/50/50b all falsifiable.
Opened as a PR rather than pushed to main because
verify.ymlonly triggers onpush: [main]andpull_request, so this is the only way to get a remote verdict before merging — which is the rule the first two commits here exist to enforce.