You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
backlog(#340): the merge-queue cost, re-derived — and two false statements in the item itself (#143)
* backlog(#340): the merge-queue cost, re-derived — and two false statements in the item itself
The #340 filing was agreed to need cost evidence. The hand account it was to be
built from ("#132 took four full CI cycles to land, none from a failure — green ->
main moved -> rebase, x3, ~80 min of runner time for a change correct on the first
pass") was wrong in every load-bearing figure, so this re-derives all of it from the
Actions API instead of relaying it.
What #132 actually did: nine CI runs / ten attempts across eleven head moves in a
3h28m window; two attempts failed on real defects (one on the head it opened at, so
it was not correct on the first pass — it opened at one commit and merged at five);
four runs cancelled in flight by cancel-in-progress; five head moves were rebases,
four of them onto a main tip that had landed 1-14 min earlier. Cost 212.7 min run
wall-clock / 442.5 min job wall-clock — quantities that differ by 2.08x, so the unit
is load-bearing. No billable figure exists: /timing reports 0 billable ms (self-hosted).
Two statements already in the item were false and are corrected:
- "A hand-coordinated merge freeze ... *did* hold main still for a full window."
It did not. main advanced four times between #119's first fully-green head and
its merge, one of them 8m26s after the freeze was recorded in a work claim.
- "#119 still failed" / "the bounds that actually killed #119" reads as the PR
dying. #119 merged at 2026-08-02T01:45:00Z, 12h15m after arming.
The caveat is stated level with the cost, per the standard this repo keeps
violating. Notably the overnight serialization is NOT evidence hand coordination
worked — it is entailed by strict + a 20-25 min suite — and strict does not always
serialize (#110/#111 merged 24s apart; mechanism not established). Also measured:
no workflow carries a merge_group: trigger, so zero of the 13 required contexts
would report on a queue ref, which makes Proposed step 2 a precondition rather than
a follow-up.
Adds the measurement-cost linkage to #344: supersession-and-re-run produces
multi-attempt runs, and the default actions/runs/{id}/jobs endpoint returns only the
latest attempt, so failed earlier attempts vanish from any duration table built from
it — at exactly the tight end where a margin is decided.
#344's own figures are untouched; another session holds that item.
* backlog(#340): the protocol cost, and a measurement claim of mine that was imprecise
Two amendments from peer review, both of which improve on what I wrote.
1. My measurement-cost paragraph conflated two different prunings. Corrected by
the ci-margin-correction session, who hit the same trap from the other side and
re-measured with ?filter=all. The precise statement:
- filtering on JOB conclusion deletes job-cancelled/step-succeeded rows, which
are the tightest by construction;
- the default latest-attempt view hides FAILED earlier attempts. It does NOT
move a step-success maximum -- so my implication that it changes the margin
was wrong. What it hides is that the sample is RIGHT-CENSORED: the largest
observable step is the largest that FIT under the cap.
Keying on the step's own conclusion (not the job's) and reading ?filter=all are
two separate fixes for two separate defects. Still no #344 figure is quoted here.
2. Adds the protocol cost, relayed independently by two sessions and assembled by
sandbox-codec. It is the strongest argument in the item and is not a throughput
argument: with no queue, sessions invent an ordering ritual, and the ritual is
less reliable than the mechanism it replaces. The self-reported instance -- a
session promising not to jump the queue while its own PR had auto-merge armed --
is already named as a failure mode in WORKTREES.md, which that session had read
about this very freeze hours earlier.
Also: re-read the open-PR set nine hours after the first measurement. 15 open, 10
armed, still 0 CLEAN. Membership churns; the condition has never lifted.
Removes a "20-25 min" suite duration I restated twice without deriving it -- the
What section already states it once, which is where it belongs.
(Nine at the hand survey; eight when [`check_stalled_prs.py`](../scripts/ci/check_stalled_prs.py) ran ~20 minutes later, because #120 had been re-synced in between. The set moves — the condition does not.)
8291
8291
8292
-
Two worked instances the same day. **#74** went green on 2026-07-30 and sat unmergeable until 2026-08-01, found only by someone hunting "stuck CI" by hand; it took three merges from `main` to land. **#119** was green with 25 passing checks and armed, stalled, was re-synced, and lost the window again. A hand-coordinated merge freeze across five sessions *did* hold `main` still for a full window — and #119 still failed, on an unrelated timeout (#344) — which is the evidence that hand coordination is not the fix.
8292
+
Two worked instances the same day. **#74** went green on 2026-07-30 and sat unmergeable until 2026-08-01, found only by someone hunting "stuck CI" by hand; it took three merges from `main` to land. **#119** was green with 25 passing checks and armed, stalled, was re-synced, and lost the window again — on two separate heads its last required check completed *after* another PR had already merged (by 2m11s on `2a2900cb`, by 2m38s on `8c407fb5`), so it was never simultaneously green and up to date. A hand-coordinated merge freeze was declared across the parallel sessions and did **not** hold `main` still: `main` advanced four times between #119's first fully-green head and its merge, one of those 8m26s after the freeze was recorded in a work claim. #119 did land in the end, at 2026-08-02T01:45:00Z, 12h15m after auto-merge was armed. That is the evidence that hand coordination is not the fix.
8293
8293
8294
8294
**Why:** the cost is finished work sitting undelivered while everyone believes it is landing. This is the repo's recurring defect shape — a signal accurate about what it looks at and silent about what it does not ([`Secure_Development_Standards`](Secure_Development_Standards.md) §3) — but the worst variant, because every other instance has *someone waiting on a result*. Here the author already had their full pass and has no reason to look again. It also scales the wrong way: the more sessions working in parallel, the more often `main` moves, so the race gets harder to win exactly as throughput rises.
8295
8295
8296
+
**Measured cost, re-derived 2026-08-02.** The first account of this instance was wrong in every load-bearing figure, and a peer caught it rather than its author — so every number below is re-derived from the Actions API and carries its unit.
8297
+
8298
+
*One worked instance.* #132 dispatched **nine CI runs — ten attempts — across eleven head moves** in a 3h28m open window. `main` moved **five times** underneath it. Five of the eleven head moves were rebases, four of them onto a tip that had landed 1–14 minutes earlier; **four runs were cancelled in flight** by `ci.yml`'s `cancel-in-progress` concurrency when a newer push superseded them. Cost: **212.7 min of run wall-clock**, or **442.5 min of job wall-clock** summed across matrix legs — the two differ by 2.08x, so neither is meaningful without its unit, and neither is *billable* minutes (`/timing` reports 0 billable ms on every run; these legs are self-hosted, so no billable figure exists to quote). Note what this instance is **not**: two of its attempts failed on real defects — one of them on the very head it opened at — so it is not a change that was right on the first pass and re-run anyway. The cost that generalises is the four superseded runs and the four rebase-driven re-runs — the ones that carried no new information about the change.
8299
+
8300
+
*The general form.* N ready PRs become N sequential suite-length cycles, because each merge invalidates every other. `main` moved **22 times on 2026-08-01**. Re-measured 2026-08-02T04:10Z: **0 of 14 open PRs were `CLEAN`** — not one could merge. Nine were green (0 failing, 0 pending) and stopped only by staleness or conflict; separately, nine of the fourteen carried armed auto-merge that could not fire. The cohort tabulated above **did not drain**: all eight are still open, and #96 has now been armed 39h at 26 commits behind. Re-read **nine hours later at 13:25Z** — 15 open, 10 armed, still **0 `CLEAN`**. The membership churns constantly; the condition has not lifted once.
8301
+
8302
+
*It is also a measurement cost, which is how this joins #344.* Supersession-and-re-run turns this population into one that **two different filters prune in two different ways**, and a margin read off either without naming it is not a margin. Filtering by **job** conclusion deletes rows where the job was cancelled but the *step* succeeded — the tightest rows, by construction, because a step near the cap is exactly what pushes its job past the job cap. The default **latest-attempt** view (`actions/runs/{id}/jobs`, `gh run list`) instead hides *failed earlier* attempts; it does not move a step-success maximum, but it conceals that the sample is **right-censored** — the largest step observable is the largest that *fit* under the cap, never the largest the suite wanted. Read `?filter=all` to see the censoring, and key on the **step's** own conclusion, not the job's.
8303
+
8304
+
*The largest cost is protocol, not throughput.* With no queue, sessions invent an ordering ritual to compensate — and the ritual is less reliable than the mechanism it replaces. Self-reported instance from the same night: a session assured a peer it would not jump the queue **while its own PR had auto-merge armed** and would have landed with nobody deciding anything. [`WORKTREES.md`](WORKTREES.md) already names that failure — *"'Don't do X' is the wrong primitive when automation already has X armed"* — and that session had read the line, about this very freeze, hours earlier. Effort spent negotiating a merge order is effort a queue spends for free, and it is where the night's stale facts and unenforceable promises came from. Wall-clock a reader can dismiss as impatience; this is not that.
8305
+
8306
+
**What a merge queue does not fix.** Stated level with the cost, because filing an overclaim inside the ticket about overclaiming would be its own instance:
8307
+
8308
+
- The overnight run of eight merges was strictly serialized (min gap 26m06s, mean 35m16s over 7 gaps) — but **serialization is not evidence that hand coordination worked.** It is entailed by `strict = true` plus a full-suite cycle, human or no human. `strict` also does not always serialize: #110 and #111 merged **24 seconds apart** on 2026-08-01, #111's squash commit parented directly on #110's. That mechanism is **not established** — only the timestamps and the parentage are.
8309
+
- A queue does not shorten the suite. It reorders the same cycles; the wall-clock bounds tracked in #344 are untouched by it.
8310
+
- **A queue would not report at all today.** Measured 2026-08-02: **no workflow under `.github/workflows/` carries a `merge_group:` trigger**, against 13 required contexts in branch protection — so by the mechanism step 2 describes, zero of them would report. Enabling the queue before step 2 lands would wedge every open PR. Step 2 is a precondition, not a follow-up.
8311
+
8296
8312
**Proposed:**
8297
8313
1. Enable a merge queue on `main` (branch protection → *Require merge queue*), squash method to match the existing history.
8298
8314
2. Reconcile the required set against it: a queue runs checks on a `gh-readonly-queue/**` ref, so any workflow that must gate the queue needs a `merge_group:` trigger. Every context in [`.github/required-contexts.txt`](../.github/required-contexts.txt) lacking one will never report there — that file's own required-but-absent trap, in a new place.
8299
8315
3. Decide the interaction with `strict = true`. A queue makes it largely redundant; leaving both on is safe but keeps the re-sync burden for anything bypassing the queue.
8300
8316
4. Once landed, [`check_stalled_prs.py`](../scripts/ci/check_stalled_prs.py) goes quiet on its own. Keep it — it is the detector for this class returning.
8301
8317
8302
-
**Related:** [`scripts/ci/check_stalled_prs.py`](../scripts/ci/check_stalled_prs.py) + [`.github/workflows/stalled-prs.yml`](../.github/workflows/stalled-prs.yml) (built alongside this filing — it reports the condition, it does not remove it); [`.github/required-contexts.txt`](../.github/required-contexts.txt); [`scripts/ci/check_required_workflow_state.py`](../scripts/ci/check_required_workflow_state.py) (sibling: "can this context ever report?" to this one's "can this PR ever merge?"); #344 (the wall-clock bounds that actually killed #119); #320.
8318
+
**Related:** [`scripts/ci/check_stalled_prs.py`](../scripts/ci/check_stalled_prs.py) + [`.github/workflows/stalled-prs.yml`](../.github/workflows/stalled-prs.yml) (built alongside this filing — it reports the condition, it does not remove it); [`.github/required-contexts.txt`](../.github/required-contexts.txt); [`scripts/ci/check_required_workflow_state.py`](../scripts/ci/check_required_workflow_state.py) (sibling: "can this context ever report?" to this one's "can this PR ever merge?"); #344 (the wall-clock bounds that killed one of #119's runs, and the sibling half of the measurement cost above); #320.
8303
8319
8304
-
**Source:** stuck-CI triage, 2026-08-01. Measured live against `MEFORORG/MessageFoundry` branch protection and the open-PR set that date; independently reached by three parallel sessions from separate evidence.
8320
+
**Source:** stuck-CI triage, 2026-08-01. Measured live against `MEFORORG/MessageFoundry` branch protection and the open-PR set that date; independently reached by three parallel sessions from separate evidence. Cost evidence and both corrections above re-derived from the Actions API on 2026-08-02 under adversarial verification, after the first hand account of #132 proved wrong in every load-bearing figure.
0 commit comments