Merge remote-tracking branch 'origin/main' into zizmor-1280-adoption

Author: wshallwshall

.github/workflows/ci.yml

@@ -222,8 +222,48 @@ jobs:
       # green in 775s against a 780s cap -- 5s of margin -- and the step was killed anyway, reported as
       # a failure on a run whose last line was "9598 passed, 855 skipped". A watchdog that cannot
       # separate "deadlocked" from "slow today" stops being a watchdog and becomes a coin flip, so the
-      # ubuntu budget is raised here. Re-check the margin when the suite grows; the Windows legs are
-      # unchanged because 26 min against the same suite is still ~2x headroom.
+      # ubuntu budget is raised here. Re-check the margin when the suite grows.
+      #
+      # It then happened again on WINDOWS, 2026-08-01, because the sentence that used to close this
+      # paragraph asserted the Windows legs held "~2x headroom" and nobody re-derived it. PR #119 was
+      # killed at 26:07 against the 26:00 cap with ZERO tests failing. What moved was the suite, not the
+      # code under test: #74 landed tests/test_worktree_prune_merged.py (1,506 lines) and windows-2025
+      # went 19:35 -> 26:07 on the same branch. The claim was already false when it was written.
+      #
+      # Measured over the 11 PASSING windows-2025 runs on 2026-08-01, timing the `Tests (pytest)` STEP,
+      # which is what step_timeout gates. NOT the job: the job is ~3 min longer and capped separately,
+      # and two sessions misread job durations as step durations while triaging this (c53f752b's JOB ran
+      # 28:41 and PASSED, because job cap 30 vs step cap 26).
+      #
+      #   leg              max passing step   old cap   old margin
+      #   ubuntu-latest    12:27              19:00     1.53x
+      #   windows-2022     18:39              26:00     1.39x
+      #   windows-2025     24:35              26:00     1.06x
+      #
+      # windows-2025 had already PASSED at 24:35 -- 85 seconds of margin -- before #119 died. The "2x"
+      # figure matched no leg. State the MEASURED value and its DATE, never a round multiple: a bare
+      # multiple gives the next reader no way to tell when it has rotted, which is how this one survived.
+      #
+      # PROOF IT WAS THE CAP, NOT THE BRANCH. #119's windows-2025 leg was RE-RUN on the SAME commit
+      # against the SAME 26:00 cap: attempt 1 was killed at the cap, attempt 2 concluded SUCCESS. Same
+      # code, same config, same ceiling, two outcomes. The leg was not failing -- it was coin-flipping
+      # against the cap, exactly the state the ubuntu note above names, and the reason "re-run it and
+      # see" is not a diagnosis here. A green re-run at 26:00 does not mean the suite fits; it means
+      # that runner was fast enough that time.
+      #
+      # 36:00 is 1.46x over that 24:35 maximum -- the margin ubuntu already runs with. Both Windows legs
+      # take the same number: windows-2022 is the faster of the two, so sizing on windows-2025 only
+      # leaves it more room, and one value is one thing to re-derive. job_timeout moves 30 -> 40 to keep
+      # the nesting invariant above: the step must still expire strictly BEFORE the job.
+      #
+      # This cap is NOT what catches a hung test; pytest_timeout (120s) is, per test. The step cap only
+      # catches a whole-process deadlock both in-process watchdogs miss, which is why it can sit well
+      # above a healthy run. Sizing it tight buys no detection and costs false failures on green suites,
+      # twice now.
+      #
+      # The remaining margin is a SHARED budget across every PR that lands and nothing accounts for it:
+      # three PRs each adding a minute of Windows time reproduce #119's death, individually blameless.
+      # A mechanical guard for that is BACKLOG #341; the underlying slowness is #320.
       - name: Tests (pytest)
         if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch'
         timeout-minutes: ${{ matrix.step_timeout }}
@@ -382,8 +422,8 @@ jobs:
           # expression interpolation into the run body), so it is zizmor-safe and cannot be misparsed as
           # an Actions expression the way a literal double-brace token in a run: block would be.
           U='{"os":"ubuntu-latest","python-version":"3.14","hosted":["ubuntu-latest"],"job_timeout":22,"step_timeout":19,"pytest_timeout":60,"fault_timeout":90}'
-          W22='{"os":"windows-2022","python-version":"3.14","hosted":["windows-2022"],"job_timeout":30,"step_timeout":26,"pytest_timeout":120,"fault_timeout":150}'
-          W25='{"os":"windows-2025","python-version":"3.14","hosted":["windows-2025"],"job_timeout":30,"step_timeout":26,"pytest_timeout":120,"fault_timeout":150}'
+          W22='{"os":"windows-2022","python-version":"3.14","hosted":["windows-2022"],"job_timeout":40,"step_timeout":36,"pytest_timeout":120,"fault_timeout":150}'
+          W25='{"os":"windows-2025","python-version":"3.14","hosted":["windows-2025"],"job_timeout":40,"step_timeout":36,"pytest_timeout":120,"fault_timeout":150}'
           if [ "${GITHUB_REPOSITORY:-}" = "MEFORORG/MessageFoundry" ]; then
             echo "matrix={\"include\":[$U,$W22,$W25]}" >> "$GITHUB_OUTPUT"
           else

.github/workflows/stalled-prs.yml

@@ -0,0 +1,57 @@
+name: Stalled PRs
+
+# A pull request can be finished, green, armed to merge -- and unable to merge, forever, silently.
+#
+# THE DEFECT THIS EXISTS FOR — measured on this repo, 2026-08-01: NINE open pull requests with zero
+# failing checks and zero pending checks, not one of which could merge. Six had auto-merge ARMED, which
+# will never fire. PR #74 had been in that state since 2026-07-30 and was found only because somebody
+# went looking for "stuck CI" by hand.
+#
+# THE MECHANISM. Branch protection sets `required_status_checks.strict = true` and there is no merge
+# queue, so a PR that goes green must finish while `main` holds still. When anything else lands first it
+# flips to BEHIND and stops. Armed auto-merge does NOT update a BEHIND branch — it waits on checks that
+# already passed. Nothing re-syncs it and nothing reports it.
+#
+# WHY NO EXISTING SIGNAL CATCHES IT. Every other signal is a check OUTCOME, and no check has failed —
+# that is the whole problem. `nightly-notice.yml` watches CI runs and there is no failing run to watch.
+# The author's last signal was a full pass, so they have no reason to look. A green dashboard and a
+# wedged repository are indistinguishable unless something asks "can this still merge at all?".
+#
+# SCHEDULED, not per-PR, deliberately: the stall arrives when a DIFFERENT pull request merges, so the
+# affected PR has no run in flight to hang a check on. It becomes true while the repo is idle.
+#
+# ADVISORY BY PLACEMENT — this must never become a required context. It reports on OTHER pull requests,
+# so a stall on #71 would block #128, which is both wrong and a way to wedge the repo with the very
+# tool meant to unwedge it. See .github/required-contexts.txt for the required-but-absent trap.
+#
+# This does NOT fix the race; only a merge queue does. It converts a SILENT failure into a LOUD one,
+# which is the part that let #74 sit for days. If a merge queue is enabled, this goes quiet on its own.
+on:
+  schedule:
+    # 07:05 UTC — just after `Required workflow state`, so the two merge-health checks report together.
+    - cron: "5 7 * * *"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  stalled:
+    name: open PRs can still reach a merge
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      pull-requests: read # list PRs + their merge state; no write scope anywhere
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+      # No pip install: the script is stdlib-only and shells out to `gh`, which is preinstalled on the
+      # runner. Nothing here parses the repo, so there is no PyYAML dependency to pin.
+      - name: Report pull requests that are green but cannot merge
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: python scripts/ci/check_stalled_prs.py --repo "$GITHUB_REPOSITORY"

docs/BACKLOG.md

@@ -8242,3 +8242,69 @@ Separately: the docstring argues *"the return value is the point"*, but all six
 **Source:** public-repo disclosure audit, 2026-08-01. Re-verified and re-measured at HEAD on the same date.
 
 ---
+
+## 340. Enable a GitHub merge queue: strict + no queue makes every merge a race that fails silently
+
+> 🔢 **Filed 2026-08-01 — not started.** Value **8/10** · Difficulty **3/10** · _fill-in_. `strict = true` with no merge queue and a ~20-minute suite means a PR must go green *and* have `main` hold still, or it flips to `BEHIND` and stops. Measured this date: **9 open PRs green (0 failing, 0 pending) and none able to merge**, 6 with auto-merge armed that will never fire.
+
+**Cluster:** Developer Experience & CI. **Priority:** P2. **Verdict:** build. **Severity:** medium.
+
+**What:** branch protection on `main` sets `required_status_checks.strict = true` — a PR must be up to date with the base to merge — and the repo has **no merge queue** (`repository.mergeQueue` is null; `allow_auto_merge` is true). The slowest required leg runs ~20–25 minutes. Those three facts compose into a race: a PR is mergeable only in the window between its checks going green and the next thing landing on `main`.
+
+Losing that race is **silent**. Armed auto-merge does *not* update a `BEHIND` branch — it waits on checks that already passed — so the PR sits armed and stalled with no failing check, no notification, and no run in flight. Measured 2026-08-01:
+
+```
+PR     mergeState  failing  pending  auto-merge
+#128   BEHIND      0        0        ARMED
+#125   BEHIND      0        0        ARMED
+#107   BEHIND      0        0        -
+#106   BEHIND      0        0        ARMED
+#101   BEHIND      0        0        ARMED
+#96    BEHIND      0        0        ARMED
+#71    BEHIND      0        0        ARMED
+#60    BEHIND      0        0        -
+```
+
+(Nine at the hand survey; eight when [`check_stalled_prs.py`](../scripts/ci/check_stalled_prs.py) ran ~20 minutes later, because #120 had been re-synced in between. The set moves — the condition does not.)
+
+Two worked instances the same day. **#74** went green on 2026-07-30 and sat unmergeable until 2026-08-01, found only by someone hunting "stuck CI" by hand; it took three merges from `main` to land. **#119** was green with 25 passing checks and armed, stalled, was re-synced, and lost the window again. A hand-coordinated merge freeze across five sessions *did* hold `main` still for a full window — and #119 still failed, on an unrelated timeout (#344) — which is the evidence that hand coordination is not the fix.
+
+**Why:** the cost is finished work sitting undelivered while everyone believes it is landing. This is the repo's recurring defect shape — a signal accurate about what it looks at and silent about what it does not ([`Secure_Development_Standards`](Secure_Development_Standards.md) §3) — but the worst variant, because every other instance has *someone waiting on a result*. Here the author already had their full pass and has no reason to look again. It also scales the wrong way: the more sessions working in parallel, the more often `main` moves, so the race gets harder to win exactly as throughput rises.
+
+**Proposed:**
+1. Enable a merge queue on `main` (branch protection → *Require merge queue*), squash method to match the existing history.
+2. Reconcile the required set against it: a queue runs checks on a `gh-readonly-queue/**` ref, so any workflow that must gate the queue needs a `merge_group:` trigger. Every context in [`.github/required-contexts.txt`](../.github/required-contexts.txt) lacking one will never report there — that file's own required-but-absent trap, in a new place.
+3. Decide the interaction with `strict = true`. A queue makes it largely redundant; leaving both on is safe but keeps the re-sync burden for anything bypassing the queue.
+4. Once landed, [`check_stalled_prs.py`](../scripts/ci/check_stalled_prs.py) goes quiet on its own. Keep it — it is the detector for this class returning.
+
+**Related:** [`scripts/ci/check_stalled_prs.py`](../scripts/ci/check_stalled_prs.py) + [`.github/workflows/stalled-prs.yml`](../.github/workflows/stalled-prs.yml) (built alongside this filing — it reports the condition, it does not remove it); [`.github/required-contexts.txt`](../.github/required-contexts.txt); [`scripts/ci/check_required_workflow_state.py`](../scripts/ci/check_required_workflow_state.py) (sibling: "can this context ever report?" to this one's "can this PR ever merge?"); #344 (the wall-clock bounds that actually killed #119); #320.
+
+**Source:** stuck-CI triage, 2026-08-01. Measured live against `MEFORORG/MessageFoundry` branch protection and the open-PR set that date; independently reached by three parallel sessions from separate evidence.
+
+---
+
+## 344. Fixed wall-clock bounds have drifted out of proportion to the work they bound
+
+> 🔢 **Filed 2026-08-01 — not started.** Value **6/10** · Difficulty **4/10** · _fill-in_. Hardcoded real-time budgets — a CI `step_timeout`, a test helper's poll deadline — were sized when the work was smaller or the machine faster, and nothing re-derives them. Two confirmed instances; each presents as an unrelated flake.
+
+**Cluster:** Developer Experience & CI. **Priority:** P2. **Verdict:** build. **Severity:** medium.
+
+**What:** a fixed wall-clock bound with no relationship to the work it bounds fails the day the work grows, and it fails as a **timeout with zero assertion failures** — which reads as a broken branch when nothing is broken.
+
+*Instance 1 (fixed 2026-08-01, this filing).* `ci.yml`'s Windows `step_timeout: 26`. windows-2025's max PASSING `Tests (pytest)` step was **24:35** over 11 runs — 1.06x margin — and PR #119 was killed at 26:07 with zero test failures after #74 added a 1,506-line test file. The comment beside the cap asserted "~2x headroom", a figure that matched no leg. Raised to 36:00 (1.46x) with the measurement and its date recorded in place of the multiple.
+
+*Instance 2 (open).* `tests/test_stage_dispatcher.py`'s `_wait_until` (:356) polls `loop.time()` against a hardcoded **8.0s** budget while the dispatcher under test is driven by an injected `ManualClock`. On PR #129 — whose diff is provably AST-identical to main, docstrings only — `test_adr0070_9_content_retry_is_not_an_infra_fault[sqlserver]` failed on that bound alone against a Dockerised SQL Server; every logic assertion in the same loop passed. A real-time deadline gating a virtual-clock system has no principled value.
+
+**Why:** these fail *individually blameless*. The remaining CI margin is a **shared budget nobody accounts for** — three PRs each adding a minute of Windows time reproduce #119's death, with no single PR at fault. And this repo has twice mislabelled such a failure: the two famous "flakes" turned out to be a livelock and a test that was right. A timeout with no failing assertion is the exact signature that invites the wrong diagnosis.
+
+**Proposed:**
+1. **A mechanical margin check** (suggested by the ASVS-scorecard session, whose framing this is): compare each leg's actual `Tests (pytest)` step duration against its configured `step_timeout` and fail below ~1.3x. Computable from data CI already emits. It would have flagged windows-2025 *before* #119 died — it was already at 1.06x and nothing said a word — and unlike the "re-check the margin when the suite grows" instruction in `ci.yml`, it does not depend on anyone remembering. Two traps for whoever builds it: time the **STEP**, not the job (the job is ~3 min longer with its own cap — `c53f752b`'s job ran 28:41 and passed against job cap 30 / step cap 26, and two sessions misread job for step while triaging this); and size against the **max passing** run, not the mean (windows-2025's mean ~21 min looks comfortable, its max passing 24:35 is what bites).
+2. Size the remaining bounds: `grep` hardcoded `timeout=` / deadline floats under `tests/` and judge each against the work it bounds.
+3. Where a virtual clock drives the system under test, the poll deadline should follow that clock, not `loop.time()` — instance 2 is the worked example.
+4. Prefer bounds expressed as a measured ratio with a date over round multiples, per instance 1's post-mortem.
+
+**Related:** [`.github/workflows/ci.yml`](../.github/workflows/ci.yml) §*Tests (pytest)* (instance 1 and its measurement table); `tests/test_stage_dispatcher.py`:356 (instance 2); #320 (windows-2025 slowness — the capacity fact that shrinks every Windows margin); #340 (the other half of this triage); [`Secure_Development_Standards`](Secure_Development_Standards.md) §3 (prose asserting a margin the numbers do not support — five instances found on 2026-08-01 alone).
+
+**Source:** stuck-CI triage, 2026-08-01. Instance 1 measured across 11 windows-2025 runs; instance 2 reported and diagnosed by the HA-construct-recheck session from PR #129's sqlserver leg.
+
+---