feat: changed-file mutation gate as a second required PR check (D-027)

.github/workflows/mutation.yml

@@ -0,0 +1,86 @@
+# Portable mutation PR gate (StrykerJS). To adopt in another repo:
+#   1. copy scripts/mutation-gate.mjs and this file;
+#   2. replace the toolchain setup below (setup-bun + bun install) with your own
+#      (e.g. drop setup-bun, use actions/setup-node + `npm ci`);
+#   3. tune the MUTATION_GATE_* env knobs (full table in the script header);
+#   4. add the `mutation` check to branch protection; create the
+#      mutate-force / mutate-skip labels.
+# Modes: changed (default: mutates the PR's changed files) or incremental
+# (MUTATION_GATE_MODE: incremental; needs a baseline, see the commented-out
+# job at the bottom, plus an actions/cache restore step in this job).
+name: mutation
+on:
+  pull_request:
+    branches: [main]
+    types: [opened, synchronize, reopened, labeled, unlabeled]
+concurrency:
+  group: mutation-${{ github.event.pull_request.number }}
+  cancel-in-progress: true
+permissions:
+  contents: read
+  pull-requests: write # sticky comment
+jobs:
+  mutation:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15 # backstop against a mispredicted run
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          ref: ${{ github.event.pull_request.head.sha }} # the PR as authored, not the synthetic merge ref
+          fetch-depth: 0 # all branches: merge-base needs the real base branch
+      - uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: "1.3.14" # same pin rationale as gates (bunfig coverage semantics)
+      - uses: actions/setup-node@v5
+        with:
+          node-version: "24" # Stryker CLI host (D-010); >=20 required, 20 is EOL
+      - run: bun install --frozen-lockfile
+      - name: gate
+        id: gate
+        env:
+          MUTATION_GATE_BASE: origin/${{ github.event.pull_request.base.ref }} # the branch, never the payload SHA
+          MUTATION_GATE_FORCE: ${{ contains(github.event.pull_request.labels.*.name, 'mutate-force') && '1' || '' }}
+          MUTATION_GATE_SKIP: ${{ contains(github.event.pull_request.labels.*.name, 'mutate-skip') && '1' || '' }}
+          MUTATION_GATE_EXTRA_ARGS: "--concurrency 4" # Task 10 measurement: conc-1 exceeded the 15-min job timeout at 409 lines; conc-4 completed in 7.7 min (27.5 mutants/min) with a clean, non-corrupted report
+        run: node scripts/mutation-gate.mjs
+      - name: sticky comment
+        if: always() && steps.gate.outputs.decision != ''
+        continue-on-error: true # fork PRs get a read-only token; the verdict is the gate step's alone
+        env:
+          GH_TOKEN: ${{ github.token }}
+          DECISION: ${{ steps.gate.outputs.decision }}
+          SUMMARY: ${{ steps.gate.outputs.summary }}
+          PR: ${{ github.event.pull_request.number }}
+        run: |
+          BODY_FILE="$(mktemp)"
+          printf '<!-- mutation-gate -->\n\n%s\n' "$SUMMARY" > "$BODY_FILE"
+          EXISTING="$(gh api "repos/${GITHUB_REPOSITORY}/issues/${PR}/comments" --paginate \
+            --jq '[.[] | select(.body | startswith("<!-- mutation-gate -->"))][0].id // empty')"
+          case "$DECISION" in
+            pass|pass-empty) [ -z "$EXISTING" ] && exit 0 ;; # quiet pass: only update an existing comment
+          esac
+          if [ -n "$EXISTING" ]; then
+            gh api -X PATCH "repos/${GITHUB_REPOSITORY}/issues/comments/${EXISTING}" -F body=@"$BODY_FILE"
+          else
+            gh api -X POST "repos/${GITHUB_REPOSITORY}/issues/${PR}/comments" -F body=@"$BODY_FILE"
+          fi
+      - name: report artifact
+        if: failure()
+        uses: actions/upload-artifact@v7
+        with:
+          name: mutation-report
+          path: reports/mutation/
+          retention-days: 14
+          if-no-files-found: ignore
+#  baseline: # incremental-mode adopters: produce/refresh the baseline on pushes to main
+#    # (also add `push: { branches: [main] }` to `on:` and an `if: github.event_name == 'push'` guard)
+#    runs-on: ubuntu-latest
+#    steps:
+#      - uses: actions/checkout@v7
+#      - <your toolchain setup + dependency install>
+#      - uses: actions/cache@v4
+#        with:
+#          path: reports/stryker-incremental.json
+#          key: stryker-incremental-${{ github.sha }}
+#          restore-keys: stryker-incremental-
+#      - run: node_modules/.bin/stryker run --incremental --incrementalFile reports/stryker-incremental.json

AGENTS.md

@@ -45,12 +45,12 @@ Guidance for any agent (or human) working in this repo. `CLAUDE.md` is a symlink
 
 ## Working notes
 - **Git identity is the user's to set** — don't run `git config user.*` on their behalf. Commit/push **only when asked**.
-- **`bun run mutate` (Stryker) needs a Node >= 20 binary on `PATH`** to host the Stryker CLI process itself — Bun cannot (a `@babel/generator` CJS/ESM-interop crash: `TypeError: generator is not a function`); the runner plugin itself still drives `bun test`. Mutation testing is manual and never a gate. The runner sanitizes bunfig.toml for its child runs (forces coverage=false), which is why the always-on gate and mutation runs compose safely.
+- **`bun run mutate` (Stryker) needs a Node >= 20 binary on `PATH`** to host the Stryker CLI process itself — Bun cannot (a `@babel/generator` CJS/ESM-interop crash: `TypeError: generator is not a function`); the runner plugin itself still drives `bun test`. Mutation testing is manual full campaigns plus a changed-file PR gate (`.github/workflows/mutation.yml` + `scripts/mutation-gate.mjs`, D-027): small PRs are gated on zero undetected mutants in touched engine files; large PRs loud-skip with a sticky comment (labels `mutate-force`/`mutate-skip` override); after any Stryker/runner bump, run a full campaign before engine PRs resume. The runner sanitizes bunfig.toml for its child runs (forces coverage=false), which is why the always-on gate and mutation runs compose safely.
 - **`nvm use default` puts a Node 24 on PATH** for `bun run mutate` / focused `stryker run` invocations.
 - **`bun test <single-file>` may exit 1 with zero failures** — the per-file coverage floor (bunfig.toml) judges partially-imported files. Exit 1 with 0 fails = coverage floor, not a test failure; gate on full `bun test` runs.
 - **Internal imports**: upward reaches (anything needing `../`) use `#src/…`/`#scripts/…` (package.json `imports`); same-directory and downward stay relative, explicit `.ts` extensions. Enforced by `test/import-style.test.ts` (D-013).
 - **Never run `biome migrate` unattended** (D-021, D-023). On the v1 config it rewrote `"rules": { "recommended": true }` as `"rules": { "preset": "none" }`, which deletes the rule set instead of preserving it: `biome check .` then exits 0 on code containing `any`, `==` and unused vars, so the lint gate dies silently and CI stays green. The correct spelling is `"preset": "recommended"`, and `test/lint-gate.test.ts` now fails if it ever changes back. After any biome config change, re-verify with a planted violation and check the **exit code**, not the printed summary.
 - **Biome's "safe" fixes are not all safe here** (D-023). `noUselessEscapeInRegex` unescaped the `\.` in `MQTT_EXTENSION_KEY`, which is a no-op to the regex engine but breaks D-019's character-for-character transcription of the upstream schema key that `test/upstream-drift.test.ts` compares byte-for-byte. It is suppressed inline with that reason. Read what `--write` changed before trusting it; the test suite caught this one, but a less-covered invariant would have slipped through.
-- **TypeScript 7 ships `tsc` only** (D-022) — no `tsserver.js`, no programmatic `typescript` module API under `node_modules/typescript/lib`. Nothing in the repo imports it as a module, so gates and mutation runs are unaffected, but an editor set to "use workspace TypeScript version" finds no language server and silently falls back to its own bundled TypeScript. Expect the editor and the `typecheck` gate to be different compilers; when they disagree, `bun run typecheck` is the authority.
+- **TypeScript 7 ships `tsc` only** (D-022) — no `tsserver.js`, no programmatic `typescript` module API under `node_modules/typescript/lib`. Nothing in the repo imports it as a module, so the gates are unaffected, but Stryker's core sandbox preprocessing loads `typescript` regardless of `checkers` config (found when mutation first ran in CI, D-027), no-op'd via the `stryker.conf.json` `tsconfigFile` sentinel and pinned by `test/stryker-tsconfig-noop.test.ts`. An editor set to "use workspace TypeScript version" finds no language server and silently falls back to its own bundled TypeScript. Expect the editor and the `typecheck` gate to be different compilers; when they disagree, `bun run typecheck` is the authority.
 - **Dependency bumps: refresh ≠ range change.** Taking a newer build of an already-declared range is routine; requiring a version you previously did not is a decision. `bun update` conflates them — it rewrites `package.json` floors even for packages whose version did not move — so refresh with `bun update`, then `git checkout package.json && bun install` to keep the change lockfile-only. Range changes get their own entry in `DECISIONS.md`, their own PR, and a measurement (D-020, D-021).
-- **CI (GitHub Actions)**: `.github/workflows/ci.yml` runs the gate set (`check-docs` → `lint` → `typecheck` → `demo-app:build` → full `bun test`) on PRs and main pushes; main pushes also upload `demo-app/dist/` + `coverage/` artifacts. Bun is pinned there (1.3.14) so the bunfig coverage-gate semantics stay as verified; bump the pin deliberately, in its own PR. A `main` ruleset requires the `gates` check (repo-admin bypass keeps direct pushes possible). Mutation testing stays out of CI (D-017).
+- **CI (GitHub Actions)**: `.github/workflows/ci.yml` runs the gate set (`check-docs` → `lint` → `typecheck` → `demo-app:build` → full `bun test`) on PRs and main pushes; main pushes also upload `demo-app/dist/` + `coverage/` artifacts. Bun is pinned there (1.3.14) so the bunfig coverage-gate semantics stay as verified; bump the pin deliberately, in its own PR. A `main` ruleset requires the `gates` check (repo-admin bypass keeps direct pushes possible). The `mutation` required check runs the changed-file gate on PRs (D-027); full campaigns stay out of CI.

DECISIONS.md

@@ -256,3 +256,15 @@ Append-only. Each decision has a stable never-reused `D-###` id, what was decide
 **Obligations**: if the real client's profile ever changes (auth appears, QoS 2 shows up, the protocol level moves, the transport changes), re-run the capture per demo-app.md §9 and update `fixtures/connect/real-client.json` + this entry's facts; none blocking.
 **From**: the authoritative spike runs against the real browser application (reported 2026-08-03), executed per docs/specs/demo-app.md §9.
 **Folds into**: REQUIREMENTS.md (R-006, R-007, R-033 text, seeding note), fixtures/connect/real-client.json, fixtures/connect/README.md, src/broker/connect-profile.test.ts, docs/specs/build-plan.md §5, AGENTS.md (Status & next)
+
+### D-027: A changed-file mutation gate on PRs; full campaigns stay manual
+**Date**: 2026-08-04
+**What**: A second required check, `mutation` (`.github/workflows/mutation.yml` + dependency-free `scripts/mutation-gate.mjs`, the two-file portable unit): on every PR targeting main, mutate exactly the changed files that match `stryker.conf.json`'s `mutate` globs, plus sibling sources of changed or deleted test files (existence checked at HEAD), and fail below score 100. Score is Stryker's own metric: detected = Killed + Timeout, undetected = Survived + NoCoverage; Ignored and error statuses stay out of the verdict; zero valid mutants scores 100; the verdict is computed from the JSON report, never Stryker's exit code (thresholds.break defaults to null, so the exit code carries nothing). Above MUTATION_GATE_THRESHOLD_LINES (800 summed whole-file lines, unchanged — see Measured below) the gate loud-skips: green check, step summary, sticky PR comment nudging a local run; labels `mutate-force`/`mutate-skip` override in both directions, force wins. Diff base: the PR head SHA is checked out and merge-based against `origin/<base branch>`, never the payload base SHA or the synthetic merge ref (stale-payload diffs otherwise fail open via loud-skip). Report artifact uploads on failure only. An incremental mode ships for adopting projects (baseline required by default, loud-skip without it).
+**Measured** (ubuntu-latest, Task 10 rehearsal, 2026-08-04): mutating `src/engine/index.ts` + `instances.ts` (409 summed lines) at concurrency 1 hit the job's 15-minute `timeout-minutes` backstop — cancelled mid-campaign, no report written, no mutants/min computable. Concurrency 4 on the identical mutate set completed cleanly in 7m42s (462s): 212 mutants (195 killed, 3 timeout, 1 survived, 13 ignored; score 99.50), **27.53 mutants/min**, zero coverage-correlation warnings in the run log (no sign of the perTest-under-parallel-workers mis-correlation risk the design's Threshold rationale flagged as unverified). Concurrency 1 produced no report, so the design's planned "identical per-status counts" comparison is inapplicable; the timeout-vs-completes-cleanly outcome is itself the decision criterion, and it resolves unambiguously. **Shipped**: `MUTATION_GATE_EXTRA_ARGS: "--concurrency 4"` in `mutation.yml`. **Derived threshold**: 27.53 × 12 min / 0.5 mutants/line = 660.72 → 700 (nearest 100); 700 is within a factor of 2 of the provisional 800 (ratio ≈ 1.14, inside [400, 1600]) → `THRESHOLD_LINES` **stays 800** unchanged in `DEFAULTS`, its test expectations, and the spec's Threshold rationale.
+**Why**: Amends D-010 ("run manually, never a gate") and D-017 ("mutation testing is excluded from CI in any form"): both stances priced a full-campaign gate, and a changed-file gate prices per-PR work instead, catching test-strength regressions at merge time where they are cheapest. Whole-file (not changed-line) mutation keeps the D-011 ratchet reading: every file a PR touches ends the PR mutation-clean. Loud-skip keeps the obligation visible on large PRs without holding the check hostage; the label escape hatches keep "mandatory" from eroding at the first heuristic misfire.
+**Mitigations / notes**: The gate does not police annotation quality: `Ignored` is excluded, so a `// Stryker disable` comment silences a survivor, and the unobservability argument stays human review (D-011). A Stryker or runner bump can change the mutant set; run a full campaign (`bun run mutate`) after any such bump before engine PRs resume, or the drift lands on the next innocent PR. Test-helper and config changes do not trigger the gate (accepted residual; incremental mode is the answer for projects that care). Widening the `mutate` globs beyond `src/engine/` stays module-by-module, each behind its own kill-or-annotate campaign. **The gate's thesis was validated before it shipped**: the measurement run surfaced a genuine post-D-011 drift survivor, `src/engine/index.ts:311` (`OptionalChaining`), introduced by commit 245ab16 (the L2 scenario runtime, merged after the D-011 campaign closed) and invisible until this rehearsal exercised the file. It was killed by a test (commit 2f2342e), and a full `src/engine` recertification campaign then ran clean under the amended conf: 438 killed + 1 timeout, 0 survived, 0 no-coverage (local, concurrency 4, 3m30s) — exactly the D-011 clean-report bar. **Known flake, documented not fixed**: `@hughescr/stryker-bun-runner` carries an upstream-documented intermittent Bun `TestReporter` id-collision bug that aborts the dry run with a `ConfigError` unrelated to any real mutant. Frequent locally during this work (5 of 7 attempts in one session) but zero occurrences across the seven CI runs measured for this entry. The gate's infra-fail verdict is already distinct from a fail/pass verdict (edge case in the design doc), and a re-run clears it; filing the upstream issue is an open follow-up, not a blocker.
+**Discovered during implementation (2026-08-05)**: The first rehearsal run did not produce the expected verdict at all — it infra-failed before instrumenting a single mutant, with every subsequent CI invocation failing identically until fixed. `@stryker-mutator/core@9.6.1`'s `TSConfigPreprocessor` unconditionally calls `ts.parseConfigFileTextToJson` on the sandboxed `tsconfig.json` to rewrite `extends`/`references` paths, regardless of the conf's `checkers` setting; TypeScript 7 (D-022) ships `tsc` only, with no programmatic module API, so the call does not exist and every mutation run in CI crashed with `TypeError: ts.parseConfigFileTextToJson is not a function`. **This amends D-022's mitigation claim** ("stryker.conf.json configures no checkers, so mutation runs never load it either"): that was true of `checkers`, but not of Stryker's own core sandbox preprocessing, which loads `typescript` unconditionally and independent of the `checkers` config — invisible until mutation testing first ran in CI, which D-017 had kept from happening at all until this gate. **Fix**: `stryker.conf.json` sets `"tsconfigFile": "tsconfig.stryker-unused.json"`, a deliberately nonexistent sentinel path. The preprocessor's crash is reached only when the configured path exists in the sandbox's file set, so pointing it at an absent path makes the whole step a clean no-op — offbook's real `tsconfig.json` has no `extends`/`references` and no `typescript-checker` plugin is configured, so the step buys nothing here regardless. **Residual risk, stated explicitly**: a future `@stryker-mutator/core` bump could start validating the configured-but-absent path instead of silently skipping it; the invariant is pinned by `test/stryker-tsconfig-noop.test.ts`, and a focused `stryker run` re-verification is required after any Stryker or runner bump, before trusting the gate again. **Second fix**: past the tsconfig crash, the dry run (which must execute the whole suite once, for perTest coverage) was then killed by `@hughescr/stryker-bun-runner`'s own process-timeout: `bun.timeout` (default 10000ms, documented as "per test") plus a fixed 30000ms drain ceiling, a 40-second hard kill on the *entire* dry-run process — well short of Stryker core's own 5-minute `dryRunTimeoutMinutes`, which never got a chance to apply. The most recent plain `bun test` run in the `ci` workflow took ~68s for the full suite at measurement time, so 40s was never enough on a GitHub-hosted runner. `stryker.conf.json` sets `"bun": { "timeout": 120000 }`, raising the dry-run ceiling to 150s (~2.2x the observed baseline) while keeping the per-mutant ceiling (the same knob) well under the 15-minute job timeout.
+**Consequences for earlier entries**: Amends D-022's Mitigations claim as described above; D-022's decision to take TypeScript 7 is unaffected — only its "gates and mutation runs are unaffected" scope needed correcting, since mutation testing had never actually run in CI (D-017) at the time D-022 was written.
+**Obligations**: none blocking. Open, non-blocking: file the upstream `@hughescr/stryker-bun-runner` TestReporter id-collision issue; re-verify the `tsconfigFile` no-op with a focused `stryker run` after any future Stryker or runner bump.
+**From**: docs/superpowers/specs/2026-08-04-mutation-pr-gate-design.md (brainstorm dialog + adversarial agent review, 2026-08-04)
+**Folds into**: scripts/mutation-gate.mjs, scripts/mutation-gate.test.ts, .github/workflows/mutation.yml, stryker.conf.json, test/stryker-tsconfig-noop.test.ts, src/engine/index.test.ts, AGENTS.md (working notes), DECISIONS.md (D-022 amendment note)