From 224eef7ea22ca5b7c7c33a875fa013ff8e4176ad Mon Sep 17 00:00:00 2001 From: vk Date: Thu, 27 Aug 2026 18:01:52 +0530 Subject: [PATCH 01/11] Two published numbers had no artifact; pre-register the arm that replaces one CHANGELOG.md cited "80 samples" for skill-collision suppression and "a uniform 0/5 across five fixtures" as the motivation for the 1.43.0 replay log. Neither can be re-read. There is no 80-sample runlog in git history or on this machine; the only collision arm on record is 55 samples on col-11 alone (tests/README.md:188) and its instrument was an uncommitted edit to tests/dispatch-fleet.sh, which that file's own fence comment at line 180 already said. The 0/5 figure is worse than unsourced: at dispatch-fleet.sh:678-688 only kind=="skill" and kind=="none" fixtures get a numeric k, so a CHAIN: or AMBIGUOUS: fixture returns {k: null} and can never print a fraction. Two of the five fixtures in that arm are exactly those kinds. Both lines keep their place with a retraction note attached rather than being deleted, the way tests/evals/RESULTS.md keeps its retracted run. The direction each arm reported stands; the n is withdrawn from both. tests/evals/collision/PREREGISTRATION.md registers the replacement before it runs: 5 arms x n=5 = 25 calls, matched pairs so a silent skill can be told apart from a suppressed one, thresholds and void conditions fixed in advance, fixtures pinned by sha256 rather than by path, dispatch-fleet.sh not edited. Also: docs/what-this-actually-does.md claimed 44 checks and falsifiability totals of 60 and 61 declared; the tree declares 48 and 73. The old figures are marked superseded with their run dates rather than replaced by a number nobody re-ran for that document. Gate: 48 declared, 47 ran, 1 skipped, VERIFIED, rc=0. The skip names its dependency (v1.47.0 is not tagged yet, so there is no payload to compare the README pin against; the pin moves in the release commit, as at 4a89761). Co-Authored-By: Claude Opus 5 (1M context) --- .claude-plugin/marketplace.json | 2 +- CHANGELOG.md | 72 ++++++++++ claude/.claude-plugin/plugin.json | 2 +- claude/inventory.json | 4 +- docs/what-this-actually-does.md | 6 +- tests/evals/collision/PREREGISTRATION.md | 171 +++++++++++++++++++++++ 6 files changed, 250 insertions(+), 7 deletions(-) create mode 100644 tests/evals/collision/PREREGISTRATION.md diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index fca6c06..ead6795 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -10,7 +10,7 @@ "plugins": [ { "name": "vstack", - "version": "1.46.0", + "version": "1.47.0", "source": "./claude", "description": "28 skills that fire without a slash command, 14 agents, 15 commands, and the session hook that routes situations to skills. Most skills are ported from pstack and Superpowers — see claude/skills/ATTRIBUTION.md for per-skill source and license.", "category": "workflow" diff --git a/CHANGELOG.md b/CHANGELOG.md index ab9ad10..e5635a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,62 @@ Versions follow [semver](https://semver.org). The version lives in two manifests `.claude-plugin/marketplace.json` and `claude/.claude-plugin/plugin.json`, and check 13 of `.claude/verify.sh` fails when they disagree. +## 1.47.0 — 2026-08-27 + +### Two published measurements had no surviving artifact + +`CHANGELOG.md` cited "80 samples" for the skill-collision suppression finding and "a uniform 0/5 +across five fixtures" as the motivation for the 1.43.0 replay log. Neither number can be re-read. +There is no 80-sample runlog in git history or on this machine; the only collision arm on record +is 55 samples on `col-11` alone (`tests/README.md:188`), and its instrument was an uncommitted +edit to `tests/dispatch-fleet.sh` — the fence comment at line 180 of that file already said so. +Its runlog at `/private/tmp/vstack-dispatch-pilot-col.jsonl` no longer exists. The 0/5 figure is +worse than unsourced: the harness cannot produce that shape at all, because at +`tests/dispatch-fleet.sh:678-688` only `kind=="skill"` and `kind=="none"` fixtures receive a +numeric `k`, so a `CHAIN:` or `AMBIGUOUS:` fixture returns `{k: null}` and never prints a +fraction. Two of the five fixtures in that arm are exactly those kinds. + +This is the repository's own subject applied to its own evidence base. A check that inherits its +answer returns a true statement about the wrong question; a citation whose artifact is gone is the +same defect one level up — it reads as measured, and there is nothing left to disagree with. Both +lines are kept in place with a retraction note attached rather than deleted, which is how +`tests/evals/RESULTS.md` already handles its retracted pathway run. The direction each arm reported +stands as what it reported. The n is withdrawn from both. + +### Replacement arm, pre-registered before it runs + +`tests/evals/collision/PREREGISTRATION.md`. The question is deliberately not "why did those +fixtures score zero" — that is unanswerable now — but the one a single arm structurally cannot +answer: when a skill fails to fire on a collision-framed prompt, was it the collision, or does that +skill not fire under this harness on any prompt? Matched pairs, 5 arms × n=5 = 25 model calls: +`swarm` and `principle-encode-lessons-in-structure` each on their clean `pos-*` fixture and their +`col-*` fixture, plus `col-01` ("Tear this apart.") which is a literal trigger string in both +`grill-me`'s and `interrogate`'s descriptions. + +Thresholds, void conditions and named invalidators are fixed in that file before the first sample. +The fixture file is pinned by `sha256`, not by path, because `~/vstack-dispatch/` is not a git +repository and the path alone does not identify what ran. `tests/dispatch-fleet.sh` is **not +edited**: every parameter goes through the committed env overrides, which exist precisely because +a source edit is what destroyed the previous arm. The runlog is written to a tracked path in the +repository and committed with the results, and `KEEP_WORKDIRS=1` retains the transcripts. + +The arm also bears on H11 at no extra cost. `swarm`'s affordance is the `Agent` tool, which the +harness's own guard (`dispatch-fleet.sh:195-224`) requires to be denied; +`principle-encode-lessons-in-structure` produces prose and needs no denied tool. If `swarm` is +silent on both its fixtures while the prose skill fires on its positive, the fence explains that +zero and collision does not. This does not discharge the publication gate at +`dispatch-fleet.sh:277-283`; arm A5 of `tests/evals/build-the-lever/PREREGISTRATION.md` remains its +named condition, and no fleet-wide figure is derived here. + +### Stale counts in `docs/what-this-actually-does.md` + +The document stated 44 gate checks and falsifiability totals of `60 declared` and `61 declared`. +The tree declares 48 checks and 73 falsifiability rows (70 mutation + 3 fixed). The old figures +were true on their run dates and are now marked as superseded with those dates attached, rather +than replaced with a number nobody re-ran for this document — the document's own rule is that no +claim appears without a source and a date. + + ## 1.46.0 — 2026-08-27 ### claude-mem removed @@ -23,6 +79,14 @@ read path is a skill, `mem-search`, which would arrive with 18 others — `do`, and `find-skills`. Colliding triggers were measured over 80 samples to suppress both skills, not one, so the trade was working skills for a store nothing read back. +> **Retraction note added in 1.47.0: the "80 samples" above is unsourced.** +> No 80-sample runlog exists in git history or on this machine. The only collision arm on record +> is 55 samples on `col-11` alone (`tests/README.md:188`), produced by an uncommitted edit to +> `tests/dispatch-fleet.sh` — its own fence comment at line 180 says so — and its runlog at +> `/private/tmp/vstack-dispatch-pilot-col.jsonl` is gone. The direction of the finding is what +> that arm reported; its n is withdrawn. Replacement arm pre-registered at +> `tests/evals/collision/PREREGISTRATION.md`. + Removed from `setup-machine.sh` (the plugin list and the whole `hooks.json` async-flipper), `install.sh` (the `enabledPlugins` presence probe; the entry is now deleted unconditionally), `bin/doctor` (both checks), `claude/statusline.sh` (the indicator), `README.md` and @@ -602,6 +666,14 @@ created by any test in this suite. One gap stated rather than left to be discove matcher never sees `PostToolUseFailure`, which carries `error` instead of `tool_response`, so a dispatch that *failed* is invisible to the replay log. The rows record what ran, not what broke. +> **Retraction note added in 1.47.0: the "uniform 0/5 across five fixtures" above is unsourced.** +> No runlog, no transcripts and no fixture list survive for that arm, and the harness cannot +> produce that shape: at `tests/dispatch-fleet.sh:678-688` only `kind=="skill"` and +> `kind=="none"` fixtures receive a numeric `k`, so a `CHAIN:` or `AMBIGUOUS:` fixture returns +> `{k: null}` and can never print `0/5`. The defect this paragraph describes is real and the fix +> shipped; the measurement quoted as its motivation cannot be re-read. Kept with the reason +> attached rather than deleted, as `tests/evals/RESULTS.md` keeps its retracted run. + ## 1.42.0 — 2026-08-23 **The cloud sandbox gate has never armed, and has shipped inert since v1.30.0.** `overlay.sh` diff --git a/claude/.claude-plugin/plugin.json b/claude/.claude-plugin/plugin.json index 48aa426..fa17a72 100644 --- a/claude/.claude-plugin/plugin.json +++ b/claude/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "vstack", - "version": "1.46.0", + "version": "1.47.0", "description": "Skills that fire on the situation instead of a slash command, plus the subagents, commands, and session hook that make them fire. Verification gates, parallel fan-out, code review, and writing discipline.", "author": { "name": "Vedant Kumar" diff --git a/claude/inventory.json b/claude/inventory.json index 1bee19f..2350398 100644 --- a/claude/inventory.json +++ b/claude/inventory.json @@ -11,8 +11,8 @@ "repository": "https://github.com/itsvedantkumar/vstack" }, "derived_at": { - "head": "698bee3a436322801e74a67ae97c67c72c4a41b0", - "payload_digest": "d3c0428feef3b40010fca3062f3dd2930a520086df96fd8c12604b2e92ae80d0", + "head": "cc5bf690e1389a295de868c431ac5aa3c2bf7b7a", + "payload_digest": "3ac4425857bc4641c37db4c3b7c3244b777c4d67a1736e9e5fc73210518d73a4", "digest_recipe_source": "tests/inventory-contract.sh:payload_digest_compute()", "digest_recipe_note": "payload_digest hashes the working-tree BYTES of every payload file -- tracked and untracked-but-not-ignored -- together with its executable bit and path, then hashes that listing. Recompute it with `tests/inventory-contract.sh --print-digest` and nothing else: typing a recipe by hand into a shell is a second implementation. This field used to hold the recipe as an executable string that the validator eval'd, on the reasoning that the two could then never disagree. That made the artifact its own oracle -- editing the recipe and the digest together passed while measuring nothing -- and it also hashed index blob ids plus `git status --porcelain`, which names files and status letters but never their contents, so two different unstaged edits to the same path produced the same digest. Both were confirmed by running them.", "staleness_is_the_signal": "This file is a snapshot. When the payload digest no longer matches, the validator regenerates every derived field and diffs. A mismatch is the intended failure, not a maintenance chore: it means a component moved and at least one consumer has not been told.", diff --git a/docs/what-this-actually-does.md b/docs/what-this-actually-does.md index 2f541bf..89b4261 100644 --- a/docs/what-this-actually-does.md +++ b/docs/what-this-actually-does.md @@ -24,9 +24,9 @@ A number exists, sourced, dated. |---|---|---| | The gate declared 44 at v1.41.0 and every one ran green. | `./.claude/verify.sh` prints `checks: 44 declared, 44 ran, 0 skipped` and `VERIFIED`. | Run 2026-08-23 | | On 2026-08-26 the gate declared 48 and was **red**: `48 declared, 47 ran, 1 skipped` and `VERIFICATION FAILED`, on `referenced install paths exist`, `inventory contract matches the tree` and `payload_digest`. All three are fixed. | `./.claude/verify.sh`, unpiped, on this branch | Run 2026-08-26 | -| As of 2026-08-27 both gates are green: `48 declared, 47 ran, 1 skipped` and `VERIFIED`, and `60 declared, 59 passed, 0 failed, 1 skipped` with `FALSIFIABLE`. Run unpiped in an isolated worktree at the candidate commit, exit codes read on their own line. | `./.claude/verify.sh` and `./tests/gate-falsifiability.sh` | Run 2026-08-27 | +| As of 2026-08-27 both gates are green: `48 declared, 47 ran, 1 skipped` and `VERIFIED`, and falsifiability harness returned `60 declared, 59 passed, 0 failed, 1 skipped` with `FALSIFIABLE` (superseded by rows added in 1.46.0; tree now declares 73 rows, 70 mutation + 3 fixed, not re-run for this document). Run unpiped in an isolated worktree at the candidate commit, exit codes read on their own line. | `./.claude/verify.sh` and `./tests/gate-falsifiability.sh` | Run 2026-08-27 | | The one remaining skip in both gates is check 24, and it is the designed state between a version bump and its tag, not a gap. The check's own source says so: it does not demand that HEAD be a release, only that the payload match the tag if one exists with the declared version. **Clearing it by tagging would fabricate the green** — a tag cut at HEAD makes the compared range empty by construction, so the check would report success having compared a commit to itself. The label was narrowed to say that instead. | `.claude/verify.sh` check 24, and its three branches exercised in a scratch clone | Verified 2026-08-27 | -| Every one of those 44 has a falsifiability row that has been watched going red. `tests/gate-falsifiability.sh` breaks exactly what each check watches, requires the gate to name it, and restores the file byte for byte. Check 16 fails the gate if a check has no such row. | `tests/gate-falsifiability.sh`, `tests/README.md`, README.md "Checks that can fail" | Mechanism as of v1.41.0. Suite not re-run for this document; rows 40 and 44 verified by the scoped-row method only, see note below | +| At v1.41.0, all 44 of the checks then declared had falsifiability rows. The tree now declares 48 checks and 73 falsifiability rows (70 mutation + 3 fixed), all watched for red under mutation. `tests/gate-falsifiability.sh` breaks exactly what each check watches, requires the gate to name it, and restores the file byte for byte. Check 16 fails the gate if a check has no such row. | `tests/gate-falsifiability.sh`, `tests/README.md`, README.md "Checks that can fail" | v1.41.0 baseline as of 2026-08-23; current counts as of this write. Suite not re-run for this document; rows 40 and 44 verified by the scoped-row method only, see note below | | `principle-type-system-discipline` almost never fired: 1/10 at n=10. Rewriting its description around the literal nouns a user types ("a struct, enum, or type can hold an invalid combination of fields") moved it to 9/10, matching the control. The identical rewrite method applied to `principle-build-the-lever` did not move it. That skill scored 2/10 before and after, exactly at the pre-registered falsification floor, and the rewrite was reverted rather than shipped. | CHANGELOG.md, "1.38.0" | 2026-08-23 | | `principle-prove-it-works` scored 0/10 on its own fixture prompt, because its trigger condition is about the assistant's own closing claim, not anything a skill matcher can see in the user's prompt. Replaced with a direct Stop-hook check (`prove-it-works`) rather than a rewritten description. | CHANGELOG.md, "1.37.0" | 2026-08-23 | | The container matrix's first run against published GitHub tags found two shipped defects. `bin/doctor` exited 1 on a clean Alpine install because its 45-day-cutoff `date` fallback chain covered BSD and GNU but BusyBox understands neither `date -v-45d` nor `date -d '45 days ago'`. And `vstack update`, run by anyone following the README's own documented pin quickstart (`VSTACK_REF=vX.Y.Z bash bootstrap.sh`), reported "already up to date" forever regardless of how far behind `main` the pinned checkout had drifted, because the shallow clone's refspec never fetches `origin/main` and the comparison failed silently with stderr discarded. | CHANGELOG.md, "1.33.0", and `tests/container-matrix.sh` | 2026-08-23 | @@ -110,7 +110,7 @@ It was run on 2026-08-27, against `c37ce8c` in a detached worktree rather than t checkout: **61 declared rows, 60 passed, 0 failed, 1 skipped, `FALSIFIABLE`**, followed by `ok restore integrity: no concurrent edits during the run` and `ok tree unchanged by the run`. The one skip is check 24, which names its reason (no tag to compare against). 58 of those rows carry a -mutation; the other 3 are fixed rows that assert the harness's own accounting. The row total +mutation; the other 3 are fixed rows that assert the harness's own accounting. This run's row count (61) predates rows added in version 1.46.0; the tree now declares 73 rows (70 mutation + 3 fixed), and that count has not been re-run for this document. The row total exceeds the 48 declared checks because several checks can fail in more than one way. Read that as narrowly as it is written. Every row was watched going red under its own mutation and diff --git a/tests/evals/collision/PREREGISTRATION.md b/tests/evals/collision/PREREGISTRATION.md new file mode 100644 index 0000000..bc41afc --- /dev/null +++ b/tests/evals/collision/PREREGISTRATION.md @@ -0,0 +1,171 @@ +# Pre-registration: skill-collision arm, matched pairs + +Written before the first sample was drawn, and committed before the harness was invoked. The +instrument this arm replaces was an uncommitted local edit, so the finding it produced cannot be +rebuilt from git — see "What this arm replaces" below. Committing the thresholds first is the only +part of that failure this document can prevent. + +## The question + +Two claims are published in `CHANGELOG.md` and neither has a surviving artifact: + +- `CHANGELOG.md:23` — "Colliding triggers were measured over 80 samples to suppress both skills, + not one." No 80-sample runlog exists on this machine or in git history. The only collision arm + the repository records is **55 samples on `col-11` alone** (`tests/README.md:188`), produced by a + local uncommitted edit to `tests/dispatch-fleet.sh` (its own fence comment, line 180, says so). +- `CHANGELOG.md:572` — "a uniform 0/5 across five fixtures", cited as the motivation for building + the per-dispatch replay log in 1.43.0. No runlog, no transcripts, no fixture list. + +Both read as measured. Neither can be re-read. That is the same defect this repository catalogues +in `docs/checks-that-inherit-their-answer.md`, applied to its own evidence base rather than to a +check. + +So the question is not "why did those five fixtures score zero". That question is unanswerable: +the data is gone, and the harness cannot even produce the shape claimed — at +`tests/dispatch-fleet.sh:678-688` only `kind=="skill"` and `kind=="none"` fixtures get a numeric +`k`, so `col-11` (`CHAIN:brainstorming`) and `col-14` (`AMBIGUOUS:...`) return `{k: null}` and can +never print `0/5`. + +The question this arm asks instead: + +> When a skill fails to fire on a collision-framed prompt, is that because the collision +> suppressed it, or because that skill does not fire on any prompt under this harness? + +One arm cannot answer that. Matched pairs can. + +## Design + +Each of two skills is run on **both** its clean positive fixture and its collision fixture, from +the same frozen fixture file, in one invocation, under one runlog header. A fifth arm reruns the +strongest form of the collision claim: a prompt that is a literal trigger string in two skill +descriptions at once. + +| Arm | Fixture | Prompt | `expect` | n | +|---|---|---|---|---| +| P-swarm | `pos-19` | Audit these six packages for unused deps, all at once. | `swarm` | 5 | +| C-swarm | `col-13` | Try three different approaches to this and compare. | `swarm` | 5 | +| P-encode | `pos-11` | That's the second time I've told you to use pnpm not npm. | `principle-encode-lessons-in-structure` | 5 | +| C-encode | `col-15` | I keep having to remind you to run the tests. | `principle-encode-lessons-in-structure` | 5 | +| C-literal | `col-01` | Tear this apart. | `AMBIGUOUS:grill-me\|interrogate` | 5 | + +**25 model calls. No more without an explicit further authorisation.** + +n=5 is a pilot. At n=5 a two-sided 95% Wilson interval on 5/5 is [0.566, 1.000] and on 0/5 is +[0.000, 0.434]: the intervals do not overlap, so 5/5-versus-0/5 is a real separation and anything +between them is not. The thresholds below are set accordingly and are not negotiable after the run. + +## Instrument + +**Zero edits to `tests/dispatch-fleet.sh`.** Every parameter is passed by environment override. +That override path exists because a source edit is what destroyed the last arm; using it is the +point. A harness change would also invalidate the harness's prior findings +(`tests/README.md`: "A harness change invalidates its own prior findings"). + +```sh +N=5 MODEL=sonnet MAX_TURNS=20 KEEP_WORKDIRS=1 \ +FIXTURES="$HOME/vstack-dispatch/fixtures.jsonl" \ +RUNLOG=tests/evals/collision/runlog-2026-08-27.jsonl \ +./tests/dispatch-fleet.sh pos-19 col-13 pos-11 col-15 col-01 +``` + +- `MAX_TURNS=20`, not the suite default of 3: Stage 0 of the build-the-lever pre-registration + measured `ToolSearch` deferred-tool discovery consuming 2 turns, which is two thirds of a + 3-turn budget before the model has decided anything. +- `DISALLOWED_TOOLS` is left at the committed default. It is recorded in the runlog header either + way, and `check_runlog_params()` refuses to append samples drawn under different parameters. +- `KEEP_WORKDIRS=1`. The transcripts are the thing the lost arm did not have. +- The runlog is written to a **tracked path inside the repository** and committed with the + results. `/private/tmp` is where the last one went and is why there is nothing to read. + +**Fixture file pinned by content, not by path.** `~/vstack-dispatch/` is not a git repository and +carries no checksum, so the path alone does not identify what was run: + +``` +sha256 d940b4d37512bdf9085d2ee2184c4e936ac6095b02e455a9b8408c1815be4537 +lines 54 +classes pos 25, neg 8, col 15, var 6 +``` + +If that digest does not match at run time, the arm is void and the mismatch is published. + +## Hypotheses and thresholds, fixed in advance + +**H-C1 — a shared literal trigger suppresses both skills.** +The published claim, in its strongest form. `col-01`'s prompt ("Tear this apart.") is a literal +string in both `grill-me`'s and `interrogate`'s descriptions. +- CONFIRM: `fired` is empty in ≥4/5 samples. +- FALSIFY: `fired` is empty in ≤1/5 samples. +- INDETERMINATE: 2/5 or 3/5. Reported as indeterminate, not rounded toward the prior. + +**H-C2 — the collision framing fails, the skill is reachable.** +- CONFIRM, per skill: fires ≥4/5 on its `pos-*` fixture and ≤1/5 on its `col-*` fixture. +- Evaluated separately for `swarm` and `principle-encode-lessons-in-structure`. A split verdict + is a result, not a problem. + +**H-C3 — the skill does not fire under this harness at all.** +- CONFIRM, per skill: ≤1/5 on **both** its fixtures. The collision prompt then explains nothing, + and any prior zero attributed to collision was attributed wrongly. + +H-C2 and H-C3 are mutually exclusive by construction. If neither holds for a skill (e.g. 3/5 and +3/5), that skill's verdict is INDETERMINATE and is published as such. + +**Discriminator for H11 — the tool fence.** +`tests/dispatch-fleet.sh:277-283` carries a standing publication gate: the fence may be suppressing +every skill whose expected output is an artifact. This arm was chosen to bear on it at no extra +cost. `swarm`'s affordance is the `Agent` tool, which the harness's own hard guard **requires** to +be denied (`dispatch-fleet.sh:195-224` refuses to run if `Agent` or `Workflow` is absent from the +fence). `principle-encode-lessons-in-structure`'s output is prose and needs no denied tool. +- If `swarm` is ≤1/5 on both arms while `principle-encode-lessons-in-structure` fires ≥4/5 on + `pos-11`, H11 explains `swarm` and only `swarm`, and the fence — not collision — is the + mechanism behind at least one prior zero. +- This does **not** discharge the publication gate. Arm A5 of + `tests/evals/build-the-lever/PREREGISTRATION.md` is still the gate's named condition. + +## Void conditions + +- The fixture digest above does not match at run time. +- ≥3 of the 5 arms return `subtype` other than `success` (`no_output`, `error_max_turns`) in a + majority of their samples. The instrument is then dead and no hypothesis is scored. +- The runlog does not contain exactly 25 rows, or `--score-only` over the committed runlog does not + reproduce the same k/n with zero model calls. +- Any `fence_violations` entry. The workdir is supposed to be untouched. + +A void run is published as void. It is not re-run with different parameters and reported as one +result. + +## Named invalidators + +Stated in advance, in the form `~/vstack-dispatch/README.md` uses: + +- **Editing a fixture after seeing a score.** The fixture file is frozen by digest above. +- **Adding an arm after seeing results.** Five arms, 25 calls. A sixth arm is a new + pre-registration with its own digest, not an appendix to this one. +- **Scoring the `AMBIGUOUS` fixture as a pass/fail.** `col-01`'s split *is* the datum. A 50/50 + split and a 100/0 split are different findings and neither is an error + (`dispatch-fleet.sh:831`). +- **Choosing the threshold after the run.** ≥4/5 and ≤1/5 are fixed here. +- **Publishing a fleet-wide figure.** This arm publishes **per-fixture** numbers only. No `pos-*` + recall or `neg-*` precision figure may be derived from it; the publication gate at + `dispatch-fleet.sh:277-283` is untouched by this run. +- **Reading the exit code instead of the trailer.** The verdict is read off the printed k/n and + the retained transcripts, not off `$?`. + +## Declared limits + +- n=5 per arm. This is a pilot sized to separate 5/5 from 0/5 and nothing finer. +- Two skills. Whatever it finds about `swarm` and `principle-encode-lessons-in-structure` does not + generalise to the other 26 without running them. +- The visible skill set is whatever is installed under `~/.claude` at run time, since the harness + deliberately does not isolate `CLAUDE_CONFIG_DIR` (`dispatch-fleet.sh:83-90`). The installed + version is recorded in the results. +- The fixture labels are an author's judgement of what *should* fire, never validated against a + live matcher (`~/vstack-dispatch/README.md`). A miss may be a wrong label. +- `MODEL=sonnet` matches the parameters the lost arm is recorded as having used, so that this arm + is comparable to it in the one respect that record survives. It says nothing about any other + model. + +## What this arm replaces + +Nothing. Both prior claims stay in `CHANGELOG.md` with an annotation naming them unsourced, the +way `tests/evals/RESULTS.md` keeps its retracted run with the reason attached. A number deleted +is a number that cannot be argued with later. From ae6129f31c01e80c3380ee0c5b6930a9e287e3a5 Mon Sep 17 00:00:00 2001 From: vk Date: Thu, 27 Aug 2026 18:17:43 +0530 Subject: [PATCH 02/11] doctor's CI check answered for the branch, not for the commit you are on MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit gh run list --branch main --limit 1 --json conclusion,status,... --branch main is a moving reference and --limit 1 takes whatever is newest under it. The projection carried no headSha at all, so nothing downstream could have filtered by commit even if it had wanted to. Standing on a commit whose CI had never run, or had failed, printed CI (main: an OLDER commit that passed) ✔ a true statement about a different commit. --limit 1 also picked one workflow arbitrarily when several run per commit: this repo runs verify and release, and on 2026-08-27 release completed with a failure while verify was still going, so which one spoke for the commit depended on timing. release.yml's own header warns against exactly this -- "a moving branch ref answers 'is the newest thing on this branch green', which silently drifts to a different commit" -- and the check whose job is to stop a release going out over red CI reproduced it. Three defects reached main in one day because nobody read a remote verdict, and this is the check that was supposed to catch that. It reads every run recorded against HEAD now, and orders the answer: a decided failure outranks any number still going, in-progress is a note, and no run for this commit is a note rather than a pass. An unpushed commit has no CI verdict; the newest run on the branch is not one. Check 49 gates the decision offline through a gh stub, five cases, both directions. The network call stays in bin/doctor -- verify.sh is hermetic by design, which is why the CI and release-reachability questions live there. What belongs in the gate is the decision, and row 49 deletes the commit filter to prove check 49 can fail. On a branch, not on main: a peer session holds uncommitted edits to README.md and docs/what-this-actually-does.md in the shared checkout, and two of the three files carrying the "48 checks" count are theirs right now. Committing those paths would have taken their work with it. verify.sh 49 declared, 48 ran, 1 skipped, VERIFIED (the skip is check 24's inter-release state: 1.47.0 declared, not yet tagged). inventory-contract 32 checks clean, digest ec03bbd7. tests/repro/ci-lane-answers-for-head.sh watched red on 2 of 5 cases first. --- .claude/verify.sh | 57 +++++++++++++++ CHANGELOG.md | 36 +++++++++- README.md | 6 +- bin/doctor | 71 ++++++++++++++----- claude/inventory.json | 11 +-- docs/what-this-actually-does.md | 2 +- tests/gate-falsifiability.sh | 11 ++- tests/repro/ci-lane-answers-for-head.sh | 94 +++++++++++++++++++++++++ 8 files changed, 258 insertions(+), 30 deletions(-) create mode 100755 tests/repro/ci-lane-answers-for-head.sh diff --git a/.claude/verify.sh b/.claude/verify.sh index 6a7d300..ec4da32 100755 --- a/.claude/verify.sh +++ b/.claude/verify.sh @@ -3320,6 +3320,63 @@ else skip "inventory contract matches the tree" "jq not installed" fi +# --- 49. doctor's CI lane answers for the commit you are on, not for the branch --------------- +# The network call stays in bin/doctor -- this file is offline and hermetic by design, which is +# why the release-reachability and CI questions live there. What can be gated here is the +# DECISION, exercised through a `gh` stub with no network and no repository state. +# +# Both directions, because the failure this catches was a green: `gh run list --branch main +# --limit 1` reads a moving reference and takes whatever is newest under it, so a run belonging +# to an older commit spoke for the one you were standing on. Three of this session's defects +# reached main because a remote verdict was not read; this is the check that makes the reading +# itself falsifiable. +if command -v jq >/dev/null 2>&1 && [ -x bin/doctor ]; then + c49_errs="" + c49_dir=$(mktemp -d) + c49_head=$(git rev-parse HEAD 2>/dev/null) + c49_other=0000000000000000000000000000000000000000 + c49_line(){ # -> doctor's CI line + cat > "$c49_dir/gh" <&1 | grep -E '^CI' | head -1 + } + c49_row(){ printf '[{"headSha":"%s","name":"verify","status":"%s","conclusion":"%s","displayTitle":"t","databaseId":1}]' "$1" "$2" "$3"; } + + c49_n=0 + c49_want(){ #