docs(backlog): file the scoring gap that re-opens at the filing rate (BACKLOG #1444) #2540
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: quality-advisory | |
| # Advisory-only quality-measurement gates from the Code Quality and Anti-Slop Standards rubric | |
| # (docs/Code_Quality_Standards.md, signals 6-11). Every MEASUREMENT job here runs with `--exit-zero` | |
| # (or continue-on-error) so it can never fail on what it finds -- it surfaces findings for triage only. | |
| # This is the rubric's "advisory-first, ratchet to blocking once the baseline is trusted" rule: raw | |
| # complexity, coverage %, and clone counts are weak / gameable signals (rubric section 4.1) and must | |
| # never be a hard gate. This workflow is NOT in branch protection; adding a job here cannot freeze the repo. | |
| # | |
| # ONE DELIBERATE EXCEPTION: the `liveness` job at the bottom CAN go red, and is built to. It never rules | |
| # on findings -- it rules on whether each gate proved it MEASURED anything at all. A red mark there still | |
| # blocks nothing (this workflow holds no required contexts); it is simply the loudest honest way to say | |
| # a gate has stopped working. See that job's own comment for the three incidents that motivated it. | |
| # | |
| # Signals from the rubric (renumbered by tier -- docs/Code_Quality_Standards.md v0.7), each with how | |
| # it reaches a reviewer: | |
| # * signal 11 - complexity triage (ruff C901) -> BUILT (complexity job); PR-caused DELTA as | |
| # annotations + step summary, full list in the log | |
| # * signal 9 - clone / duplication (jscpd) -> BUILT (clone job); step summary only, see below | |
| # * signal 8 - diff-coverage visibility -> BUILT (coverage job; PR-only); INLINE `::notice` | |
| # annotations on the Files changed tab + summary | |
| # * signal 7 - mutation testing (highest leverage) -> BUILT and WORKING on mutmut 3 (2.5.1 crashed on | |
| # Python 3.14 before producing a mutant, and `|| true` | |
| # made that look green for months). Measured: 461 | |
| # mutants in 3s -- 87 killed, 19 survived. Runs on | |
| # PRs too; survivor table in the step summary. | |
| # signal 10 (lint breadth) is a coordinated one-shot sweep, not an advisory job; DORA is a context caveat, not a gate. | |
| # | |
| # ONE JOB HERE IS NOT A RUBRIC SIGNAL, AND IT IS HERE ON PURPOSE (BACKLOG #1431). `dangling-citations` | |
| # reports backlog `#N` references that name no filed item in either ledger. It is a LEDGER-hygiene check, | |
| # not a code-quality measurement, and its natural sibling -- backlog-hygiene.yml -- is the wrong home: | |
| # that workflow's only job IS a required context, so a step added there could block every merge. This | |
| # file is the repository's one place where a check can report without being able to gate, which is the | |
| # property that job needs. Promoting it to blocking is an owner decision and is deliberately untaken; | |
| # see the item for the reasoning and for what the nine current findings actually are. | |
| # | |
| # WHY THIS USES WORKFLOW-COMMAND ANNOTATIONS AND *NOT* CODE SCANNING / SARIF. This was measured, not | |
| # assumed, and the measurements are recorded here so a future session does not "helpfully" add a SARIF | |
| # upload back: | |
| # 1. All 122 C901 findings on this tree are SINGLE-LINE regions anchored on the function-name token of | |
| # the `def`. GitHub renders an alert inline only when its lines are in the diff, so a PR adding | |
| # branching INSIDE a function surfaces nothing, while a signature reflow fires on pre-existing debt. | |
| # 2. jscpd emits ONE location per clone pair, chosen by scan order, so ~8 files can never be the anchor | |
| # and roughly half of newly-introduced clones would anchor on the untouched twin. | |
| # 3. Uploading on pull_request only leaves NO default-branch baseline, so every PR forever would report | |
| # all ~161 findings as new -- not a one-off first-PR blip. | |
| # Workflow commands need no token and no permission grant, and behave identically on fork PRs. Every job | |
| # below therefore holds `contents: read` and nothing else -- least privilege, worth pinning on its own | |
| # merits given that two of these jobs execute third-party code fetched at run time. | |
| # | |
| # What actually keeps this advisory is TWO things, and permissions are neither of them: (a) these job | |
| # contexts are not in the required-checks set on `main`, and (b) every analysis step is | |
| # `continue-on-error: true` plus `--exit-zero` / `--fail-under=0` / `|| true`, so the job reports success | |
| # regardless of findings. Branch-protection membership is the only thing that determines gating. | |
| # tests/test_quality_advisory_invariants.py pins (b), the absent write scopes, and the absence of SARIF. | |
| # | |
| # HONEST LIMIT of the annotation mechanism: a workflow-command annotation renders INLINE on the Files | |
| # changed tab only when its line is in the diff. That always holds for diff-coverage (every line it | |
| # flags is a line the PR changed) but usually does NOT hold for the complexity delta, whose findings | |
| # anchor on a `def` line the PR often did not touch -- those land in the Checks tab and the step summary | |
| # instead. The summary table is the primary surface for complexity, not a nice-to-have. | |
| on: | |
| pull_request: | |
| workflow_dispatch: | |
| schedule: | |
| # Nightly sweep. Every job here is cheap (mutation is ~3s of actual mutating; complexity and clone | |
| # are seconds), so the cron is a safety net that catches drift on `main` rather than a cost dodge. | |
| # Coverage is PR-only -- it needs a base ref to diff against -- so it no-ops here. | |
| - cron: "23 4 * * *" | |
| # Deny-by-default at the workflow level; each job grants only what it needs. | |
| permissions: {} | |
| jobs: | |
| complexity: | |
| # Signal 11 - cyclomatic-complexity triage. Surfaces functions over the mccabe threshold (>10) so large | |
| # units are visible in review. Raw complexity is a WEAK defect predictor (rubric section 2), so this is | |
| # ADVISORY ONLY: --exit-zero reports but never fails, and this job is not a required status check. | |
| name: complexity triage (advisory) | |
| runs-on: ubuntu-latest | |
| permissions: | |
| contents: read | |
| outputs: | |
| receipt: ${{ steps.receipt.outputs.receipt }} | |
| steps: | |
| - name: Check out the source | |
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| # The delta step needs the base branch's history for `git merge-base`. | |
| fetch-depth: 0 | |
| - name: Install ruff (version derived from the lock) | |
| # DERIVED, not duplicated. This job used to hardcode 0.15.19 while all four locks pinned 0.15.22, | |
| # and that drift is load-bearing: c901_delta.py parses ruff's human-readable C901 message because | |
| # ruff exposes the complexity number nowhere else, so a skew can change the wording under the | |
| # parser. The obvious fix -- hardcode the right version and assert it matches the lock in a test -- | |
| # would leave a REQUIRED check (the pytest suite) hostage to a routine Dependabot ruff bump: | |
| # the lock moves, the workflow string does not, and a blocking context goes red for a purely | |
| # advisory concern. Reading the lock at run time removes the drift AND the coupling. | |
| run: | | |
| RUFF_PIN="$(sed -n 's/^ruff==\([^ ;]*\).*/\1/p' constraints.lock | head -1)" | |
| if [ -z "$RUFF_PIN" ]; then | |
| echo "::notice title=Ruff pin not found::falling back to the unpinned ruff from PyPI" | |
| pipx install ruff | |
| else | |
| echo "ruff pin from constraints.lock: $RUFF_PIN" | |
| pipx install "ruff==$RUFF_PIN" | |
| fi | |
| - name: Ruff C901 complexity triage (advisory - never gates) | |
| continue-on-error: true | |
| run: | | |
| echo "== functions over mccabe complexity 10 (advisory triage; NEVER a merge gate) ==" | |
| ruff check --select C901 --output-format=concise --exit-zero messagefoundry | |
| echo "" | |
| echo "== summary ==" | |
| ruff check --select C901 --statistics --exit-zero messagefoundry | |
| - name: Complexity delta vs the merge base (advisory - never gates) | |
| # PR-only: there is no base ref to diff against on the cron or on workflow_dispatch, where the | |
| # whole-repo console triage above remains the full picture. | |
| if: github.event_name == 'pull_request' | |
| continue-on-error: true | |
| env: | |
| BASE_REF: ${{ github.base_ref }} | |
| run: | | |
| echo "== complexity this PR CHANGED (advisory; pre-existing findings deliberately omitted) ==" | |
| git fetch --no-tags origin "$BASE_REF" || true | |
| # Steps run under `bash -e`, so an unresolvable merge base would abort this step with no | |
| # explanation at all. Say why instead: an advisory signal that goes quiet should at least | |
| # leave a reason in the log, since nobody watches a green advisory job for absence. | |
| if ! MERGE_BASE="$(git merge-base HEAD "origin/$BASE_REF" 2>/dev/null)"; then | |
| echo "::notice title=Complexity delta skipped::could not resolve a merge base against origin/$BASE_REF" | |
| # Record the bail so the liveness receipt can SEE it. Without this the step exits 0, the | |
| # receipt still reports "measured" off the whole-repo triage, and the PR-caused delta -- | |
| # the entire point of this step on a PR -- goes quiet behind a green check. That is the | |
| # failure mode this workflow's liveness job exists to catch, sitting inside it. | |
| # QUOTE the reason: the receipt SOURCES this file, and an unquoted value containing | |
| # spaces parses as a command, silently leaving the variable unset -- which reported | |
| # "no reason recorded" and threw away the one useful diagnostic. Caught by executing | |
| # the receipt shell, not by reading it. | |
| echo "C901_DELTA=skipped" > c901-delta.env | |
| echo "C901_DELTA_REASON='could not resolve a merge base against origin/$BASE_REF'" >> c901-delta.env | |
| exit 0 | |
| fi | |
| echo "merge base: $MERGE_BASE" | |
| ruff check --select C901 --output-format=json --exit-zero messagefoundry > c901-head.json | |
| # A second checkout of the merge base, so the "before" numbers come from real ruff output | |
| # rather than a reconstruction. Its absolute paths differ from HEAD's by the extra directory; | |
| # c901_delta.py collapses both to repo-relative keys (regression-tested -- if it did not, | |
| # every function would report as new). | |
| git worktree add --detach base-tree "$MERGE_BASE" | |
| ( cd base-tree && ruff check --select C901 --output-format=json --exit-zero messagefoundry ) \ | |
| > c901-base.json | |
| python3 scripts/quality/c901_delta.py \ | |
| --base c901-base.json \ | |
| --head c901-head.json \ | |
| --repo-root . \ | |
| --summary-file "$GITHUB_STEP_SUMMARY" | |
| # Written LAST, so it exists only if every step above actually completed. Its absence is | |
| # therefore proof the delta died somewhere, which the receipt reports as a dead gate. | |
| BASE_N="$(python3 -c 'import json,sys; print(len(json.load(open(sys.argv[1]))))' c901-base.json)" | |
| HEAD_N="$(python3 -c 'import json,sys; print(len(json.load(open(sys.argv[1]))))' c901-head.json)" | |
| { | |
| echo "C901_DELTA=ok" | |
| echo "C901_DELTA_MERGE_BASE=$MERGE_BASE" | |
| echo "C901_DELTA_BASE_N=$BASE_N" | |
| echo "C901_DELTA_HEAD_N=$HEAD_N" | |
| } > c901-delta.env | |
| - name: Record gate liveness | |
| id: receipt | |
| if: always() | |
| # Routed through env, not interpolated into the shell body (zizmor: template injection). | |
| env: | |
| IS_PR: ${{ github.event_name == 'pull_request' }} | |
| EVENT_NAME: ${{ github.event_name }} | |
| continue-on-error: true | |
| run: | | |
| # Units count what was EXAMINED, never what was found: a repo with zero functions over the | |
| # threshold is good news, and a liveness check that fires on good news gets muted. | |
| # `--show-files` lists exactly the files ruff would check, so it is non-zero iff ruff ran. | |
| FILES="$(ruff check --select C901 --show-files messagefoundry 2>/dev/null | wc -l)" | |
| FINDINGS="$(ruff check --select C901 --output-format=json --exit-zero messagefoundry \ | |
| | python3 -c 'import json,sys; print(len(json.load(sys.stdin)))')" | |
| # THE DELTA IS THE POINT ON A PR, and until now this receipt could not see it. `--show-files` | |
| # proves ruff enumerated files, which is true whether or not the delta step ran -- so a delta | |
| # that bailed on an unresolvable merge base left a green "measured" receipt and vanished. | |
| # That is this workflow's own failure mode reproduced inside its liveness control. | |
| # | |
| # On a non-PR event the delta legitimately does not run and the whole-repo triage IS the | |
| # measurement. On a PR its silence is a dead gate, not an absence of news. | |
| if [ "$IS_PR" != "true" ]; then | |
| python3 scripts/quality/liveness.py record \ | |
| --signal complexity --status measured \ | |
| --units "$FILES" --unit-name "files scanned" \ | |
| --evidence "ruff C901 scanned $FILES files; $FINDINGS over the threshold (whole-repo triage; the PR delta does not apply on $EVENT_NAME)" \ | |
| --extra "{\"findings\":$FINDINGS}" | |
| exit 0 | |
| fi | |
| if [ ! -s c901-delta.env ]; then | |
| python3 scripts/quality/liveness.py record --signal complexity --status failed \ | |
| --reason "the PR complexity delta produced no outcome marker; it did not complete (see the job log)" | |
| exit 0 | |
| fi | |
| . ./c901-delta.env | |
| if [ "$C901_DELTA" != "ok" ]; then | |
| python3 scripts/quality/liveness.py record --signal complexity --status failed \ | |
| --reason "the PR complexity delta did not run: ${C901_DELTA_REASON:-no reason recorded}" | |
| exit 0 | |
| fi | |
| python3 scripts/quality/liveness.py record \ | |
| --signal complexity --status measured \ | |
| --units "$FILES" --unit-name "files scanned" \ | |
| --evidence "ruff C901 scanned $FILES files ($FINDINGS over the threshold); delta compared ${C901_DELTA_BASE_N} base vs ${C901_DELTA_HEAD_N} head findings against ${C901_DELTA_MERGE_BASE}" \ | |
| --extra "{\"findings\":$FINDINGS,\"delta_base_n\":${C901_DELTA_BASE_N},\"delta_head_n\":${C901_DELTA_HEAD_N}}" | |
| clone: | |
| # Signal 9 - duplication / clone detection. Flags copy-pasted blocks (the "copy-instead-of-abstract" | |
| # AI-slop signature, rubric section 3). ADVISORY: reports duplicates, never fails. The justified | |
| # ~21k-LOC store-backend parity (store/sqlserver.py + store/postgres.py = deliberate T-SQL vs Postgres | |
| # dialect duplication, whitelisted in the rubric) is ignored so it does not drown the real signal. | |
| name: clone detection (advisory) | |
| runs-on: ubuntu-latest | |
| permissions: | |
| contents: read | |
| outputs: | |
| receipt: ${{ steps.receipt.outputs.receipt }} | |
| steps: | |
| - name: Check out the source | |
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - name: jscpd clone report (advisory - never gates) | |
| continue-on-error: true | |
| run: | | |
| echo "== copy-pasted blocks in messagefoundry/ (advisory; justified store-backend parity ignored) ==" | |
| # Summary only, DELIBERATELY. jscpd ships a SARIF reporter and it works, but each result | |
| # carries exactly ONE location -- whichever file jscpd scanned first -- so the case this | |
| # signal exists for (a PR copy-pasting an existing block into a new place) anchors on the | |
| # untouched twin about half the time. That is a coin flip, not a diff-scoped signal. | |
| # Keep the 4.x pin: npm `latest` is a 5.x Rust rewrite with a different CLI. | |
| npx --yes jscpd@4.0.5 messagefoundry \ | |
| --min-tokens 60 \ | |
| --ignore "**/store/sqlserver.py,**/store/postgres.py" \ | |
| --reporters console,markdown \ | |
| --output ./jscpd-report \ | |
| || true | |
| if [ -f jscpd-report/jscpd-report.md ]; then | |
| { | |
| echo "## Duplicate blocks (jscpd)" | |
| echo "" | |
| # $GITHUB_STEP_SUMMARY is capped at 1 MiB per step and an oversized write is DROPPED | |
| # ENTIRELY, so truncate rather than silently lose the whole summary. | |
| head -c 900000 jscpd-report/jscpd-report.md | |
| echo "" | |
| echo "<sub>Advisory only. Clone locations are in the job log — see the note in the" | |
| echo "workflow for why these are not annotated on the diff.</sub>" | |
| } >> "$GITHUB_STEP_SUMMARY" | |
| fi | |
| - name: Record gate liveness | |
| id: receipt | |
| if: always() | |
| continue-on-error: true | |
| run: | | |
| # Files ANALYSED, not clones found -- zero clones is a healthy repo, not a dead gate. | |
| REPORT=jscpd-report/jscpd-report.md | |
| FILES="$(grep -oE 'duplicated lines in [0-9]+' "$REPORT" 2>/dev/null | grep -oE '[0-9]+$' || true)" | |
| CLONES="$(grep -oE 'Found [0-9]+ exact clones' "$REPORT" 2>/dev/null | grep -oE '[0-9]+' || true)" | |
| python3 scripts/quality/liveness.py record \ | |
| --signal clone --status measured \ | |
| --units "${FILES:-0}" --unit-name "files analysed" \ | |
| --evidence "jscpd analysed ${FILES:-0} files; found ${CLONES:-0} clones" \ | |
| --extra "{\"clones\":${CLONES:-0}}" | |
| coverage: | |
| # Signal 8 - diff-coverage visibility. Runs the suite under coverage, then reports coverage of the | |
| # lines CHANGED by this PR (diff-cover) -- never a whole-repo % gate, which the rubric (section 4.1) | |
| # calls a gameable signal. ADVISORY: --fail-under=0 + `|| true` so it never fails a build, and this | |
| # job is not a required status check. PR-only (needs a base ref to diff against). It re-runs the | |
| # ubuntu suite (~a few min) per PR; add a paths filter if that cost bites. (This used to say | |
| # "move it to the mirror" -- there is no mirror since the cutover; development happens directly | |
| # on the public repo.) | |
| name: diff-coverage (advisory) | |
| if: github.event_name == 'pull_request' | |
| runs-on: ubuntu-latest | |
| # SIZED FROM A MEASUREMENT, NOT FROM A GUESS (BACKLOG #1409). This cap was 20 from the | |
| # 2026-07-26 publish snapshot until 2026-09-01, and it was never edited. The SUITE grew into | |
| # it: the job took 9.7 minutes on 2026-07-26, 14.8 on 2026-08-06, and 19m36s on its LAST GREEN | |
| # run (job 97249962177, 2026-08-23 15:18 CDT) -- 24 seconds of margin. | |
| # | |
| # 2026-08-23 IS THE LAST-GREEN BOUNDARY, NOT THE FIRST-KILL ONE, and the difference matters for | |
| # anyone reconstructing this. The first cap kill was 2026-08-08 (job 93153178205, cancelled at | |
| # 20m16s), a fortnight earlier. The job then sat ON the boundary, sometimes finishing and | |
| # sometimes not, until 2026-08-23 was simply the last time it made it. A cap is not crossed | |
| # cleanly; it is crossed intermittently first, and that window is the one where the signal is | |
| # already unreliable while still looking occasionally healthy. | |
| # | |
| # After 2026-08-23 the kill was consistent: `The operation was canceled` and no verdict. 714 | |
| # pull_request runs of this workflow were created between that green job and the fix; in a | |
| # 30-run sample, 28 of 28 COMPLETED coverage jobs were cancelled and none succeeded. (An | |
| # earlier draft said "over 700 of them across 8 days" as though every run were a confirmed | |
| # kill. 714 is the RUN population; the all-kills half is a sample and is now written as one.) | |
| # That is the failure this file's `liveness` job exists to catch, and it | |
| # could not see it (BACKLOG #1410, fixed in scripts/quality/liveness.py in the same change). | |
| # | |
| # WHERE THE TIME WENT, measured across four jobs (99554977847, 99578445996, 99580323074, | |
| # 99685011404): setup -- checkout, setup-python, setup-uv, the Qt apt install and the editable | |
| # install -- totals 23 to 36 SECONDS, 2 to 3 percent of the budget. The pytest step took | |
| # 19m37s to 19m49s and reached exactly [ 91%] every time. So splitting the install out of this | |
| # job buys nothing; the measurement is the whole cost, and the fix is the `-m 'not tooling'` | |
| # selection on the pytest line below, not a bigger box. | |
| # | |
| # THE CAP IS NOW SIZED ON A REAL GREEN RUN, and it replaces every estimate that stood here. | |
| # Job 99720329464 on PR 724 (2026-09-01 02:52:25Z to 03:06:58Z) finished SUCCESSFULLY in | |
| # 14m33s: `11510 passed, 839 skipped, 2731 deselected in 823.75s (0:13:43)`, and step 8 emitted | |
| # a diff-cover verdict -- the first this gate had produced in eight days. Against 30 minutes | |
| # that is 15m27s of headroom, about 2.1x. | |
| # | |
| # 2.1x IS DELIBERATE, NOT LEFTOVER SLACK. The convention elsewhere is ~1.5x, which here would | |
| # be 22 minutes. The defect this comment documents WAS a cap with thin margin that the suite | |
| # silently grew into, so the margin is the mitigation and shrinking it to convention would | |
| # rebuild the trap. Re-derive if the measured run approaches 20 minutes, not before. | |
| # | |
| # THREE ESTIMATES THAT STOOD HERE WERE WRONG, and they are named rather than deleted because | |
| # this block's whole authority is that its numbers were measured. | |
| # "no coverage run has ever finished" -- FALSE, and it contradicted this same comment two | |
| # paragraphs up. Job 97249962177, named above as the last green run, reached [100%] with | |
| # `13046 passed, 1380 skipped in 1130.96s (0:18:50)`. A completed run existed all along. | |
| # "the honest measured floor is 21m35s for the UNNARROWED suite" -- WRONG, wrong in the | |
| # UNSAFE direction, and not a measurement at all. 21m35s is 19m40s/0.91, which assumes the | |
| # last 9 percent costs what the first 91 did. pytest's percentage counts TESTS, not time, | |
| # and the tail is 15 to 26 times more expensive per point than the average, so that | |
| # division is invalid. Two independent extrapolations, by different routes, both land well | |
| # above it: applying the observed tail rate (0.35-0.45 s/test, from the four killed jobs' | |
| # own logs) to the unreached tests gives 26 to 28 minutes, and scaling from the last green | |
| # run's curve (91 percent at 602.9s of a 1110.3s span, so the final 9 percent took 45 | |
| # percent of the run) gives about 33. Call it 26 to 33 MINUTES, extrapolated -- and note | |
| # that the whole range sits at or above this 30-minute cap. So a bigger box alone would NOT | |
| # have fixed this. The narrowing is what brings it to a measured 14m33s, which is the | |
| # opposite of what 21m35s implied. | |
| # "ESTIMATED 18 to 20 minutes for the narrowed serial run" -- superseded by the 14m33s above. | |
| # | |
| # NEXT LEVER IF IT DRIFTS AGAIN: pytest-xdist. ci.yml's engine leg runs `-n 4 --dist loadfile` | |
| # and this job does not. Adding it here needs `[tool.coverage.run] parallel = true` plus a | |
| # combine before --cov-report=xml, so it is a separate change with its own verification -- | |
| # deliberately NOT bundled here, or a shifted coverage number would have two possible causes. | |
| timeout-minutes: 30 | |
| permissions: | |
| contents: read | |
| outputs: | |
| receipt: ${{ steps.receipt.outputs.receipt }} | |
| steps: | |
| - name: Check out the source | |
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| fetch-depth: 0 | |
| - name: Set up Python 3.14 | |
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.14" | |
| - name: Set up uv | |
| uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| - name: Qt headless system deps | |
| # BOUNDED AND RETRIED. Unguarded, this step turns an external apt-mirror hiccup into a | |
| # blocked merge queue: measured 2026-08-18, three hangs across two attempts on three | |
| # different jobs, one of them 27+ minutes on `test (ubuntu-latest, py3.14)` -- a REQUIRED | |
| # context, so the queue stops. Two levers, because they answer different failures. The | |
| # per-command `timeout` kills a HANG and lets the retry run; `timeout-minutes` is the | |
| # backstop that stops this step ever eating a job budget again if the loop is edited wrong. | |
| # The failure text names the cause, so the next reader is not sent hunting in their diff -- | |
| # a check whose label points at the wrong subject is what BACKLOG #1254 is about. | |
| timeout-minutes: 8 | |
| run: | | |
| for attempt in 1 2 3; do | |
| if sudo timeout 120 apt-get update && sudo timeout 180 apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3; then | |
| exit 0 | |
| fi | |
| echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying" | |
| sleep $((attempt * 5)) | |
| done | |
| echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test." | |
| exit 1 | |
| - name: Install project + coverage tools | |
| run: | | |
| uv pip install --system --constraint constraints.lock -e ".[dev,harness,fhir,dicom,x12,xml,webauthn]" -e packaging/messagefoundry-webconsole | |
| # HASH-PINNED from the CI toolchain lock (ADR 0034 §3). These tools now DO live in | |
| # pyproject.toml ([dependency-groups].ci-quality, where each pin's rationale sits) and DEP-1 | |
| # re-exports this lock by design -- superseding the note that used to sit here claiming they | |
| # were deliberately kept out of pyproject to avoid tripping that gate. | |
| # | |
| # `python -m pip`, not `uv pip install --system`, for ONE honest reason: every other hashed | |
| # install in this repo is a pip install, and keeping one spelling means one thing for the | |
| # guards in tests/test_ci_venv_pinning.py to match. Same setup-python interpreter either way. | |
| # | |
| # NOT because uv cannot: `uv pip install --require-hashes` exists (env UV_REQUIRE_HASHES), and | |
| # setup-uv is already in this job -- the earlier claim that "--require-hashes is pip's | |
| # contract" was simply wrong. NOR for Scorecard's benefit: its parser keys on | |
| # pip/pip3/`python -m pip`, so a `uv pip` line generates NO finding at all, and converting one | |
| # can only ADD parseable surface -- it cannot close an alert. This line is posture-neutral to | |
| # Scorecard and is here for uniformity. | |
| # | |
| # Safe against the editable install above: the group lock and constraints.lock agree on all 10 | |
| # shared packages (pytest and pytest-timeout among them), so this cannot re-point what the | |
| # coverage run executes under. | |
| python -m pip install --require-hashes -r ci/locks/ci-quality.lock | |
| - name: Tests under coverage (advisory - never fails) | |
| continue-on-error: true | |
| env: | |
| QT_QPA_PLATFORM: offscreen | |
| # `-m 'not tooling'` DESELECTS THE TOOLING TIER, AND IT COSTS THIS GATE NOTHING (#1409). | |
| # The mark is applied at runtime by tests/conftest.py from tests/tooling_manifest.txt (132 | |
| # files), not by decorators, so grepping for `pytest.mark.tooling` finds 2 hits and badly | |
| # under-reads its size. ci.yml carries the same flag on its engine leg and runs the tier as | |
| # its own parallel job; this job was the only place in CI running it unfiltered AND serial. | |
| # | |
| # WHY DESELECTING IT REMOVES NO SIGNAL: the report is scoped to `--cov=messagefoundry`, and | |
| # ZERO of the 132 manifest files import that package. Measured with the repo's OWN | |
| # instrument, the _ENGINE_IMPORT regex in tests/test_tooling_partition.py, against a | |
| # positive control of 507 files under tests/ that DO import it -- so the zero is a real | |
| # absence and not a broken pattern. A test that never imports the measured | |
| # package cannot contribute a covered line through its OWN test bodies. The tier is the | |
| # run's hard tail, and the honest form of that is a RANGE, not one job's pair of numbers: | |
| # across the four killed jobs the suite reaches 89 percent at 14.1 to 15.6 minutes, and the | |
| # next two percentage points then cost 3.5 to 4.7 minutes. An earlier draft said "89 | |
| # percent in 15.2 minutes, then the next two points cost 3.4 minutes", which is one job's | |
| # figures presented as the suite's -- and even for that job the second number is 3.47. | |
| # Note the percentages count TESTS, not time, which is exactly why the tail is so much | |
| # more expensive per point than the average. | |
| # | |
| # ONE QUALIFICATION, because "costs this gate NOTHING" is stronger than the grep supports. | |
| # The zero counts DIRECT imports. Reach is not the same thing: tests/test_dast_claims.py is | |
| # in the manifest and imports scripts.security.dast_auth_sweep, which pulls messagefoundry | |
| # modules transitively, and every import-time line of those counts as covered under --cov. | |
| # So deselecting the tier can drop some IMPORT-TIME coverage even though no tooling test | |
| # exercises the package directly. The measured effect is small and the trade is still | |
| # plainly worth it -- 14m33s against a job that produced no number at all -- but the honest | |
| # claim is "costs this gate almost nothing", not "nothing". | |
| # | |
| # THAT CONTROL WENT 510 -> 508 -> 507, AND THE LAST STEP IS THE INTERESTING ONE. | |
| # 510 is the same regex with the trailing word boundary dropped, so it also matched | |
| # messagefoundry_webconsole -- a DIFFERENT package. Adding \b gives 508. But 508 is still | |
| # one too many: tests/test_lifespan_startup_unwinds.py matches at lines 48-50, and those | |
| # lines sit INSIDE a 1256-character string literal beginning at line 42, which holds | |
| # child-process source written to a temp file. It is not an import this process executes. | |
| # An AST census (ast.Import / ast.ImportFrom over all 729 .py files under tests/, 0 | |
| # unparseable) returns 507, and the regex-minus-AST difference is exactly that one file, | |
| # with no files the regex missed in the other direction. | |
| # | |
| # WHY THIS IS RECORDED AND NOT JUST FIXED: 508 was reported by two separate verification | |
| # passes that agreed with each other, and both were wrong the same way, because both ran | |
| # the SAME line-anchored regex. Agreement between runs of one instrument is one | |
| # measurement, not two. Only a different METHOD -- parsing instead of matching -- could see | |
| # a string literal. The ZERO this controls holds under every spelling tried, so no | |
| # conclusion moved; the control's own size did, twice. | |
| # | |
| # THE WEB CONSOLE TIER STAYS, and that is the one place this job must NOT copy ci.yml. | |
| # Both ci.yml legs carry `--ignore-glob='*messagefoundry-webconsole*'` (lines 824 and 1134) | |
| # because that package has its own job over there. Here it would DELETE REAL COVERAGE: 16 of | |
| # the tier's 17 PYTHON files import `messagefoundry` -- the 15 test modules plus conftest.py, | |
| # i.e. every file except a WebAuthn test double. They stand up a live `Engine` via | |
| # `Engine.create()` (searching for the constructor `Engine(` returns ZERO, so that needle is | |
| # recorded here beside its zero), and 13 of them build `create_app` and `AuthService`. They | |
| # exercise AT LEAST api/, auth/, pipeline/, config/, store/ and transports/signing -- "at | |
| # least" because a closed enumeration is a liability (SDS-3.6), and __main__ is imported too. | |
| # Collecting the tier alone loads 174 messagefoundry.* modules, every import-time line of | |
| # which --cov records as covered, which is what makes this the ONLY place in CI where the | |
| # tier reaches a coverage number. Measured 130 to 140 seconds in its own ubuntu ci.yml leg | |
| # (177 to 237 on Windows), which is a cheap price for coverage this gate reports. | |
| # | |
| # THIS PARAGRAPH PREVIOUSLY SAID "16 of those TEST FILES" AND "114 to 134 seconds"; neither | |
| # reproduced, and both are named rather than silently edited. The tier holds 15 test modules, | |
| # so a reader who asks the tree for 16 test files does not find them. The web console leg | |
| # measures 130-140s on ubuntu across the last 10 green ci.yml runs on main, and 114 is below | |
| # every sample taken. The conclusion "a cheap price" survives at 140s. | |
| run: pytest -q -m 'not tooling' --cov=messagefoundry --cov-report=xml --cov-report=term-missing:skip-covered || true | |
| - name: Diff-coverage vs the PR base (advisory - never fails) | |
| continue-on-error: true | |
| env: | |
| BASE_REF: ${{ github.base_ref }} | |
| run: | | |
| echo "== coverage of the lines this PR changed (advisory; not a whole-repo % gate) ==" | |
| # NO --depth HERE. Running a shallow fetch against the COMPLETE clone that `fetch-depth: 0` | |
| # just produced writes .git/shallow and grafts away everything behind origin/<base>'s tip. | |
| # diff-cover diffs with the three-dot range `origin/<base>...HEAD`, which needs a merge base -- | |
| # so once the base branch advances past the PR's merge ref (another PR merging, or a re-run of | |
| # a stale check) the diff dies with "no merge base". Reproduced. It then fails in the worst | |
| # possible way: the markdown reporter runs BEFORE the annotations reporter and truncates | |
| # diff-cover.md on open, so `|| true` swallows the error, zero annotations are emitted, and the | |
| # summary shows a bare heading that reads as "nothing uncovered". | |
| git fetch --no-tags origin "$BASE_REF" || true | |
| if [ ! -s coverage.xml ]; then | |
| echo "::notice title=Diff coverage skipped::coverage.xml was not produced (the test step did not complete)" | |
| exit 0 | |
| fi | |
| # `github-annotations:notice` writes `::notice file=...,line=N,endLine=M` workflow commands to | |
| # STDOUT, which the runner turns into INLINE annotations on the Files changed tab -- no token, | |
| # no permission grant, identical behaviour on fork PRs. Adjacent uncovered lines are coalesced | |
| # into ranges, so a long uncovered block costs one annotation rather than one per line. Every | |
| # line flagged is a line this PR changed, so there is no pre-existing-debt noise by construction. | |
| # The console report still prints, so the job log is unchanged. | |
| diff-cover coverage.xml --compare-branch="origin/$BASE_REF" --fail-under=0 \ | |
| --format "github-annotations:notice,markdown:diff-cover.md" 2>&1 | tee diff-cover-console.txt || true | |
| # `-s` not `-f`: diff-cover creates and truncates this file BEFORE it can fail, so a crashed | |
| # run leaves a 0-byte report. Testing for existence alone would append an empty section that | |
| # looks like a clean result -- the exact silent-success shape this workflow keeps producing. | |
| if [ -s diff-cover.md ]; then | |
| { | |
| echo "## Diff coverage" | |
| echo "" | |
| # The markdown report embeds a source snippet per uncovered line, so it can get large. | |
| # $GITHUB_STEP_SUMMARY is capped at 1 MiB per step and an oversized write is DROPPED | |
| # ENTIRELY -- truncate rather than silently lose the whole summary. | |
| head -c 900000 diff-cover.md | |
| } >> "$GITHUB_STEP_SUMMARY" | |
| else | |
| echo "::notice title=Diff coverage unavailable::diff-cover produced no report (see the job log)" | |
| fi | |
| - name: Record gate liveness | |
| id: receipt | |
| if: always() | |
| # BASE_REF is needed here too -- without it the evidence string read "against origin/". | |
| env: | |
| BASE_REF: ${{ github.base_ref }} | |
| continue-on-error: true | |
| run: | | |
| # THE DISTINCTION THIS WHOLE CHECK RESTS ON, and the place it is easiest to get wrong. | |
| # | |
| # "No lines with coverage information in this diff" is a real, correct outcome -- it is what | |
| # PRs #18 and #19 produced, because neither touched measured source. But diff-cover prints | |
| # that SAME line when coverage.xml itself measured nothing (its console template branches on | |
| # whether any source has measured lines, and never consults the diff size). So the message | |
| # alone cannot tell "nothing to measure" from "the measurement is dead" -- exactly the | |
| # ambiguity this control exists to resolve, one layer up. Reproduced with a valid-but-empty | |
| # coverage.xml against a diff that really did change six lines. | |
| # | |
| # So: prove coverage.xml measured something before accepting the not-applicable branch, and | |
| # treat a MISSING coverage.xml as a dead gate (failed), never as inapplicable -- the same way | |
| # the mutation job treats a missing mutmut-counts.env. | |
| if [ ! -s coverage.xml ]; then | |
| python3 scripts/quality/liveness.py record --signal coverage --status failed \ | |
| --reason "coverage.xml was not produced; the measurement never happened" | |
| exit 0 | |
| fi | |
| MEASURED_FILES="$(grep -c '<class ' coverage.xml || true)" | |
| if [ "${MEASURED_FILES:-0}" -eq 0 ] 2>/dev/null; then | |
| python3 scripts/quality/liveness.py record --signal coverage --status failed \ | |
| --reason "coverage.xml contains no measured files; diff-cover had nothing to compare against" | |
| exit 0 | |
| fi | |
| if grep -q "No lines with coverage information" diff-cover-console.txt 2>/dev/null; then | |
| python3 scripts/quality/liveness.py record --signal coverage --status not-applicable \ | |
| --reason "no lines with coverage information in this diff; coverage.xml measured ${MEASURED_FILES} files, so the tooling is live and this PR simply changed no measured source" | |
| else | |
| LINES="$(grep -oE 'Total:[[:space:]]+[0-9]+' diff-cover-console.txt 2>/dev/null \ | |
| | grep -oE '[0-9]+' | head -1 || true)" | |
| python3 scripts/quality/liveness.py record --signal coverage --status measured \ | |
| --units "${LINES:-0}" --unit-name "changed lines analysed" \ | |
| --evidence "diff-cover reported on ${LINES:-0} changed lines against origin/${BASE_REF}; coverage.xml measured ${MEASURED_FILES} files" | |
| fi | |
| mutation: | |
| # Signal 7 - mutation testing (advisory; the HIGHEST-leverage gate -- it adversarially checks the | |
| # tests actually assert something, catching the "tests that never fail" AI-slop mode, rubric section 3). | |
| # | |
| # NOT expensive, contrary to the note that used to sit here. MEASURED on Python 3.14 (linux, the | |
| # bounded scope below): 461 mutants in 3 SECONDS -- 87 killed, 19 survived, 355 not covered by the | |
| # scoped test file. mutmut 3 only runs the tests that actually cover each mutant, so the old | |
| # "runs the suite once per mutant" cost model does not apply. That is why this now runs on PRs too: | |
| # a real signal for seconds of runtime is worth having in review, which is where survivors get fixed. | |
| name: mutation (advisory) | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 30 | |
| permissions: | |
| contents: read | |
| outputs: | |
| receipt: ${{ steps.receipt.outputs.receipt }} | |
| steps: | |
| - name: Check out the source | |
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - name: Set up Python 3.14 | |
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.14" | |
| - name: Set up uv | |
| uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| - name: Qt headless system deps | |
| # BOUNDED AND RETRIED. Unguarded, this step turns an external apt-mirror hiccup into a | |
| # blocked merge queue: measured 2026-08-18, three hangs across two attempts on three | |
| # different jobs, one of them 27+ minutes on `test (ubuntu-latest, py3.14)` -- a REQUIRED | |
| # context, so the queue stops. Two levers, because they answer different failures. The | |
| # per-command `timeout` kills a HANG and lets the retry run; `timeout-minutes` is the | |
| # backstop that stops this step ever eating a job budget again if the loop is edited wrong. | |
| # The failure text names the cause, so the next reader is not sent hunting in their diff -- | |
| # a check whose label points at the wrong subject is what BACKLOG #1254 is about. | |
| timeout-minutes: 8 | |
| run: | | |
| for attempt in 1 2 3; do | |
| if sudo timeout 120 apt-get update && sudo timeout 180 apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3; then | |
| exit 0 | |
| fi | |
| echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying" | |
| sleep $((attempt * 5)) | |
| done | |
| echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test." | |
| exit 1 | |
| - name: Install project + mutmut | |
| run: | | |
| uv pip install --system --constraint constraints.lock -e ".[dev,harness,fhir,dicom,x12,xml,webauthn]" -e packaging/messagefoundry-webconsole | |
| # mutmut 3 + pytest-timeout, HASH-PINNED from the CI toolchain lock (ADR 0034 §3). Both live in | |
| # pyproject.toml's [dependency-groups].ci-quality, which records why each is load-bearing: | |
| # `mutmut<3` resolved to 2.5.1, which CRASHES on Python 3.14 in its pony-ORM cache | |
| # (`cannot pickle 'itertools.count'`) before generating a single mutant -- verified from run | |
| # 30248096425; and pytest-timeout is REQUIRED because mutmut 3 always passes `--timeout`/ | |
| # `--timeout-method` to pytest, so without the plugin every invocation dies with an | |
| # unrecognised-argument error surfacing only as BadTestExecutionCommandsException. | |
| # `python -m pip` for UNIFORMITY with every other hashed install in this repo, not because uv | |
| # cannot do it (`uv pip install --require-hashes` exists) and not for Scorecard (a `uv pip` | |
| # line generates no finding at all) -- see the coverage job's install step for the full note. | |
| python -m pip install --require-hashes -r ci/locks/ci-quality.lock | |
| - name: Mutation-test a bounded scope (advisory - never fails) | |
| continue-on-error: true | |
| env: | |
| QT_QPA_PLATFORM: offscreen | |
| run: | | |
| echo "== mutation testing messagefoundry/parsing/binary.py (advisory; bounded scope) ==" | |
| # Ephemeral mutmut config (the repo ships none). THREE things here are load-bearing, each | |
| # found by a run that silently produced nothing: | |
| # * source_paths is the PACKAGE, not the one file. mutmut 3 copies source_paths into | |
| # `mutants/` and runs pytest THERE; with a single file copied, tests/conftest.py died on | |
| # `ModuleNotFoundError: messagefoundry.config` and every mutant came back "not checked". | |
| # * only_mutate keeps the bounded scope -- the whole package is copied, one module mutated. | |
| # * pytest_add_cli_args_test_selection replaces mutmut 2's `runner=`; the old `runner` and | |
| # `paths_to_mutate` keys are gone/deprecated in 3.x. | |
| printf '[mutmut]\nsource_paths=messagefoundry\nonly_mutate=messagefoundry/parsing/binary.py\npytest_add_cli_args_test_selection=tests/test_binary_carriage.py\n' > setup.cfg | |
| if mutmut run > mutmut-run.txt 2>&1; then | |
| MUTMUT_OK=1 | |
| else | |
| MUTMUT_OK=0 | |
| fi | |
| tail -c 20000 mutmut-run.txt | |
| if [ "$MUTMUT_OK" = "1" ]; then | |
| mutmut results > mutmut-results.txt 2>&1 || true | |
| # `mutmut results` lists ONLY the mutants worth looking at -- survived / no tests / | |
| # timeout / suspicious. Killed mutants are NEVER listed, so `grep -c ': killed'` is | |
| # always 0 and a naive table reports "Killed 0" on a perfectly healthy run. Derive it: | |
| # the run's final progress line carries the total (`461/461`), and every non-killed | |
| # mutant is exactly one line in the results file. Verified against run 30308667584: | |
| # 461 total - 374 listed = 87 killed, matching mutmut's own counter. | |
| TOTAL="$(tr '\r' '\n' < mutmut-run.txt | grep -oE '[0-9]+/[0-9]+' | tail -1 | cut -d/ -f2)" | |
| LISTED="$(grep -cE '^[[:space:]]+\S+: ' mutmut-results.txt || true)" | |
| SURVIVED="$(grep -c ': survived' mutmut-results.txt || true)" | |
| NOTESTS="$(grep -c ': no tests' mutmut-results.txt || true)" | |
| # Counted INDEPENDENTLY, never as `LISTED - SURVIVED - NOTESTS`. Deriving it as a | |
| # remainder would make the liveness reconciliation tautological -- the parts would add up | |
| # by construction and the check could never fire. | |
| # | |
| # The alternation must cover EVERY non-killed status mutmut can print, or the sum breaks | |
| # on a healthy run and the liveness job reddens for no reason. A check whose only | |
| # reachable failure is a false positive gets muted within a month, which would leave us | |
| # worse off than having no check. | |
| OTHER="$(grep -cE ': (timeout|suspicious|skipped|segfault|not checked|caught by type check|check was interrupted by user)$' mutmut-results.txt || true)" | |
| # mutmut's OWN killed counter, off its final progress line. KILLED above is derived a | |
| # completely different way (total minus listed); the liveness check reconciles the two. | |
| # Two independent derivations agreeing is what actually protects the number -- the sum | |
| # check alone cannot, because TOTAL cancels out of it algebraically. | |
| KILLED_REPORTED="$(tr '\r' '\n' < mutmut-run.txt | grep -oE '🎉 [0-9]+' | tail -1 \ | |
| | grep -oE '[0-9]+' || true)" | |
| if [ -n "$TOTAL" ] && [ "$TOTAL" -ge "$LISTED" ] 2>/dev/null; then | |
| KILLED=$((TOTAL - LISTED)) | |
| else | |
| # Never invent a number: if the total could not be read, say so rather than print 0. | |
| KILLED="?" | |
| fi | |
| echo "== total=$TOTAL killed=$KILLED survived=$SURVIVED no-tests=$NOTESTS ==" | |
| grep ': survived' mutmut-results.txt || true | |
| { | |
| echo "## Mutation testing — \`messagefoundry/parsing/binary.py\`" | |
| echo "" | |
| echo "| Killed | Survived | Not covered |" | |
| echo "| --- | --- | --- |" | |
| echo "| $KILLED | **$SURVIVED** | $NOTESTS |" | |
| echo "" | |
| if [ "$SURVIVED" != "0" ]; then | |
| echo "Survivors — injected bugs the tests did **not** catch:" | |
| echo "" | |
| echo '```' | |
| grep ': survived' mutmut-results.txt | head -c 100000 | |
| echo '```' | |
| fi | |
| echo "" | |
| echo "<sub>Advisory only. A survivor is a hint to strengthen an assertion, not a defect —" | |
| echo "and mutation score is a poor single number (rubric §2), so this never gates.</sub>" | |
| } >> "$GITHUB_STEP_SUMMARY" | |
| # Stash the breakdown for the liveness receipt in the next step. | |
| { | |
| echo "MUT_TOTAL=$TOTAL" | |
| echo "MUT_KILLED=$KILLED" | |
| echo "MUT_SURVIVED=$SURVIVED" | |
| echo "MUT_NOTESTS=$NOTESTS" | |
| echo "MUT_OTHER=$OTHER" | |
| echo "MUT_LISTED=$LISTED" | |
| echo "MUT_KILLED_REPORTED=${KILLED_REPORTED:-}" | |
| } > mutmut-counts.env | |
| else | |
| echo "::warning title=Mutation run failed::mutmut exited non-zero — see the job log" | |
| { | |
| echo "## Mutation testing — FAILED TO RUN" | |
| echo "" | |
| echo "\`mutmut run\` exited non-zero, so this signal produced nothing. Tail of the run:" | |
| echo "" | |
| echo '```' | |
| tail -c 4000 mutmut-run.txt | |
| echo '```' | |
| } >> "$GITHUB_STEP_SUMMARY" | |
| fi | |
| - name: Record gate liveness | |
| id: receipt | |
| if: always() | |
| continue-on-error: true | |
| run: | | |
| if [ ! -s mutmut-counts.env ]; then | |
| # mutmut crashed. This is the 2.5.1 case verbatim -- and it must NOT be silent. | |
| python3 scripts/quality/liveness.py record --signal mutation --status failed \ | |
| --reason "mutmut produced no counts; the run did not complete (see the job log)" | |
| exit 0 | |
| fi | |
| . ./mutmut-counts.env | |
| # `?` means the mutant total was unreadable, so KILLED is underivable. Interpolating it into | |
| # the --extra JSON would emit `"killed":?` -- malformed, and it would surface as "receipt is | |
| # not valid JSON" rather than the real diagnosis. | |
| case "$MUT_KILLED" in | |
| ''|*[!0-9]*) | |
| python3 scripts/quality/liveness.py record --signal mutation --status failed \ | |
| --reason "could not read mutmut's mutant total from the progress line; the killed count is underivable" | |
| exit 0 | |
| ;; | |
| esac | |
| # An EMPTY results file yields LISTED=0, hence killed=TOTAL, survived=0 -- a flawless score | |
| # that reconciles perfectly. A 100% kill rate with nothing listed is not a triumph, it is a | |
| # tool that produced no output. Refuse to report it as success. | |
| if [ "${MUT_LISTED:-0}" -eq 0 ] && [ "${MUT_TOTAL:-0}" -gt 0 ] 2>/dev/null; then | |
| python3 scripts/quality/liveness.py record --signal mutation --status failed \ | |
| --reason "mutmut listed no mutants at all while reporting $MUT_TOTAL processed; a perfect score here means the results file was empty, not that every mutant died" | |
| exit 0 | |
| fi | |
| EXTRA="{\"killed\":$MUT_KILLED,\"survived\":$MUT_SURVIVED,\"no_tests\":$MUT_NOTESTS,\"other\":$MUT_OTHER" | |
| case "$MUT_KILLED_REPORTED" in | |
| ''|*[!0-9]*) ;; | |
| *) EXTRA="$EXTRA,\"killed_reported\":$MUT_KILLED_REPORTED" ;; | |
| esac | |
| EXTRA="$EXTRA}" | |
| python3 scripts/quality/liveness.py record \ | |
| --signal mutation --status measured \ | |
| --units "$MUT_TOTAL" --unit-name "mutants processed" \ | |
| --evidence "mutmut processed $MUT_TOTAL mutants; $MUT_LISTED listed as non-killed" \ | |
| --extra "$EXTRA" | |
| - name: Upload the mutmut results (advisory) | |
| if: always() | |
| continue-on-error: true | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 | |
| with: | |
| name: mutmut-results | |
| path: | | |
| mutmut-results.txt | |
| mutmut-run.txt | |
| .mutmut-cache | |
| # .mutmut-cache is a DOTFILE and upload-artifact skips hidden files by DEFAULT. Without this | |
| # the step logs "No files were found", uploads nothing, and still reports success -- a silent | |
| # no-op that looks green. | |
| include-hidden-files: true | |
| # Deliberately `warn`, not `ignore`: if mutmut ever renames its cache, that must be visible in | |
| # the log rather than silently swallowed. | |
| if-no-files-found: warn | |
| dangling-citations: | |
| # BACKLOG #1431 -- a working detector that ran nowhere. | |
| # | |
| # `scripts/docs/dangling_citation_check.py` reports a `#N` backlog citation that names no item in | |
| # EITHER ledger. Until this job, the SCRIPT was invoked by no workflow and no pre-commit hook: | |
| # measured at 46ea10a78 with controls in the same pass, the script path occurs 0 times across | |
| # .github/ plus .pre-commit-config.yaml, while backlog_citation_check.py (a real `run:`) occurs | |
| # once and an invented path occurs 0 times. Only its TEST was wired, via ci.yml's DOC_GUARDS. | |
| # | |
| # WHY IT RUNS HERE AND CANNOT GATE. Four independent things keep it advisory, and | |
| # tests/test_dangling_citation_advisory.py pins the three a repository test can see: | |
| # 1. this workflow holds no required context (.github/required-contexts.txt records that it must | |
| # never be promoted) -- branch-protection membership is the only thing that decides gating; | |
| # 2. `--advisory` makes a live-shape finding exit 0 rather than 1, which is this tool's default; | |
| # 3. `continue-on-error: true` means even the checker's own empty-population REFUSAL cannot fail | |
| # the job -- it shows as a warning on a green job, which is the visibility this wants; | |
| # 4. this job is deliberately absent from `liveness`'s `needs`, so it cannot redden the one job | |
| # in this file that is built to go red. | |
| # | |
| # THE EXIT CODE IS NOT SWALLOWED WITH `|| true`, and that is the point of (3). With `--advisory` | |
| # the only way this tool exits non-zero is a GENUINE malfunction -- most plausibly its refusal to | |
| # report on an empty population, which is what running from the wrong directory looks like. A | |
| # clean scan and a scan that read nothing must not render alike, so that case stays visible as a | |
| # red step inside a green job rather than being turned into a silent success. | |
| # | |
| # NO PATHS FILTER ON THIS WORKFLOW, which is what makes the job worth having: a citation is | |
| # introduced by editing PROSE, and the pytest tier that holds this tool's unit test is skipped on | |
| # a documentation-only pull request. This job runs on all of them. | |
| name: dangling backlog citations (advisory) | |
| runs-on: ubuntu-latest | |
| permissions: | |
| contents: read | |
| steps: | |
| - name: Check out the source | |
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - name: Report backlog citations that name no filed item (advisory - never gates) | |
| continue-on-error: true | |
| run: | | |
| echo "== backlog citations under docs/ that name no item in either ledger ==" | |
| echo "== advisory: this reports, it never gates a merge ==" | |
| TOOL_STATUS=0 | |
| python3 scripts/docs/dangling_citation_check.py --advisory \ | |
| > dangling-citations.txt 2>&1 || TOOL_STATUS=$? | |
| cat dangling-citations.txt | |
| { | |
| echo "## Dangling backlog citations" | |
| echo "" | |
| echo "Advisory only. A reported token is an UPPER BOUND ON CITATIONS, not a count of" | |
| echo "defects: a number at or below the allocator high-water mark can never be issued," | |
| echo "and a PR, issue or foreign-repo reference is not a backlog citation at all. Both" | |
| echo "kinds are printed so a human judges them. See BACKLOG item 1431 for the triage." | |
| echo "" | |
| echo '```' | |
| # $GITHUB_STEP_SUMMARY is capped at 1 MiB per step and an oversized write is DROPPED | |
| # ENTIRELY, so truncate rather than silently lose the whole summary. | |
| head -c 200000 dangling-citations.txt | |
| echo '```' | |
| } >> "$GITHUB_STEP_SUMMARY" | |
| if [ "$TOOL_STATUS" -ne 0 ]; then | |
| echo "::warning title=Dangling-citation scan did not complete::the checker exited ${TOOL_STATUS} in advisory mode, which means a malfunction rather than a finding. See the job log." | |
| fi | |
| # Re-raise the checker's status. `continue-on-error` above keeps the JOB green; this keeps a | |
| # broken scan distinguishable from a clean one. | |
| exit "$TOOL_STATUS" | |
| liveness: | |
| # THE META-GATE. Every other job here is built so it can never fail; this one is built so it CAN. | |
| # | |
| # Three separate signals in this workflow spent months reporting success while measuring nothing: | |
| # diff-coverage (a shallow fetch destroyed its merge base, and the empty report looked clean), | |
| # mutation (mutmut 2.5.1 crashed before producing a mutant, and `|| true` made that green in 37s), | |
| # and the killed count (a grep for a line mutmut never prints, so a healthy run said "Killed 0"). | |
| # The rubric's anti-metric rule guards against trusting a NUMBER too much. Nothing guarded against | |
| # trusting a GREEN CHECK THAT NEVER RAN. This job is that control. | |
| # | |
| # It reads each job's liveness receipt and demands proof of EXECUTION -- a count of things | |
| # examined -- or an explicit, reasoned declaration that there was nothing to measure. It does not | |
| # demand findings: a clean repo legitimately has zero clones, and a gate that fires on good news | |
| # gets muted, which would leave us worse off than before. | |
| # | |
| # DELIBERATELY has no `continue-on-error` and no `|| true`. A dead gate should be loud, and a red | |
| # mark here blocks nothing -- this job is not, and must never become, a required status check. | |
| name: gate liveness (advisory) | |
| if: always() | |
| needs: [complexity, clone, coverage, mutation] | |
| runs-on: ubuntu-latest | |
| permissions: | |
| contents: read | |
| steps: | |
| - name: Check out the source | |
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false | |
| - name: Verify every gate proved it measured something | |
| # `needs` carries each job's result AND its receipt output. Routed through `env:` rather than | |
| # interpolated into the shell body -- the receipts are job-authored strings, and expanding | |
| # them into a command line is the template-injection shape zizmor exists to catch. | |
| env: | |
| NEEDS_JSON: ${{ toJSON(needs) }} | |
| run: python3 scripts/quality/liveness.py verify --needs-json "$NEEDS_JSON" |