@@ -274,7 +274,68 @@ jobs:
274274 name : diff-coverage (advisory)
275275 if : github.event_name == 'pull_request'
276276 runs-on : ubuntu-latest
277- timeout-minutes : 20
277+ # SIZED FROM A MEASUREMENT, NOT FROM A GUESS (BACKLOG #1409). This cap was 20 from the
278+ # 2026-07-26 publish snapshot until 2026-09-01, and it was never edited. The SUITE grew into
279+ # it: the job took 9.7 minutes on 2026-07-26, 14.8 on 2026-08-06, and 19m36s on its LAST GREEN
280+ # run (job 97249962177, 2026-08-23 15:18 CDT) -- 24 seconds of margin.
281+ #
282+ # 2026-08-23 IS THE LAST-GREEN BOUNDARY, NOT THE FIRST-KILL ONE, and the difference matters for
283+ # anyone reconstructing this. The first cap kill was 2026-08-08 (job 93153178205, cancelled at
284+ # 20m16s), a fortnight earlier. The job then sat ON the boundary, sometimes finishing and
285+ # sometimes not, until 2026-08-23 was simply the last time it made it. A cap is not crossed
286+ # cleanly; it is crossed intermittently first, and that window is the one where the signal is
287+ # already unreliable while still looking occasionally healthy.
288+ #
289+ # After 2026-08-23 the kill was consistent: `The operation was canceled` and no verdict. 714
290+ # pull_request runs of this workflow were created between that green job and the fix; in a
291+ # 30-run sample, 28 of 28 COMPLETED coverage jobs were cancelled and none succeeded. (An
292+ # earlier draft said "over 700 of them across 8 days" as though every run were a confirmed
293+ # kill. 714 is the RUN population; the all-kills half is a sample and is now written as one.)
294+ # That is the failure this file's `liveness` job exists to catch, and it
295+ # could not see it (BACKLOG #1410, fixed in scripts/quality/liveness.py in the same change).
296+ #
297+ # WHERE THE TIME WENT, measured across four jobs (99554977847, 99578445996, 99580323074,
298+ # 99685011404): setup -- checkout, setup-python, setup-uv, the Qt apt install and the editable
299+ # install -- totals 23 to 36 SECONDS, 2 to 3 percent of the budget. The pytest step took
300+ # 19m37s to 19m49s and reached exactly [ 91%] every time. So splitting the install out of this
301+ # job buys nothing; the measurement is the whole cost, and the fix is the `-m 'not tooling'`
302+ # selection on the pytest line below, not a bigger box.
303+ #
304+ # THE CAP IS NOW SIZED ON A REAL GREEN RUN, and it replaces every estimate that stood here.
305+ # Job 99720329464 on PR 724 (2026-09-01 02:52:25Z to 03:06:58Z) finished SUCCESSFULLY in
306+ # 14m33s: `11510 passed, 839 skipped, 2731 deselected in 823.75s (0:13:43)`, and step 8 emitted
307+ # a diff-cover verdict -- the first this gate had produced in eight days. Against 30 minutes
308+ # that is 15m27s of headroom, about 2.1x.
309+ #
310+ # 2.1x IS DELIBERATE, NOT LEFTOVER SLACK. The convention elsewhere is ~1.5x, which here would
311+ # be 22 minutes. The defect this comment documents WAS a cap with thin margin that the suite
312+ # silently grew into, so the margin is the mitigation and shrinking it to convention would
313+ # rebuild the trap. Re-derive if the measured run approaches 20 minutes, not before.
314+ #
315+ # THREE ESTIMATES THAT STOOD HERE WERE WRONG, and they are named rather than deleted because
316+ # this block's whole authority is that its numbers were measured.
317+ # "no coverage run has ever finished" -- FALSE, and it contradicted this same comment two
318+ # paragraphs up. Job 97249962177, named above as the last green run, reached [100%] with
319+ # `13046 passed, 1380 skipped in 1130.96s (0:18:50)`. A completed run existed all along.
320+ # "the honest measured floor is 21m35s for the UNNARROWED suite" -- WRONG, wrong in the
321+ # UNSAFE direction, and not a measurement at all. 21m35s is 19m40s/0.91, which assumes the
322+ # last 9 percent costs what the first 91 did. pytest's percentage counts TESTS, not time,
323+ # and the tail is 15 to 26 times more expensive per point than the average, so that
324+ # division is invalid. Two independent extrapolations, by different routes, both land well
325+ # above it: applying the observed tail rate (0.35-0.45 s/test, from the four killed jobs'
326+ # own logs) to the unreached tests gives 26 to 28 minutes, and scaling from the last green
327+ # run's curve (91 percent at 602.9s of a 1110.3s span, so the final 9 percent took 45
328+ # percent of the run) gives about 33. Call it 26 to 33 MINUTES, extrapolated -- and note
329+ # that the whole range sits at or above this 30-minute cap. So a bigger box alone would NOT
330+ # have fixed this. The narrowing is what brings it to a measured 14m33s, which is the
331+ # opposite of what 21m35s implied.
332+ # "ESTIMATED 18 to 20 minutes for the narrowed serial run" -- superseded by the 14m33s above.
333+ #
334+ # NEXT LEVER IF IT DRIFTS AGAIN: pytest-xdist. ci.yml's engine leg runs `-n 4 --dist loadfile`
335+ # and this job does not. Adding it here needs `[tool.coverage.run] parallel = true` plus a
336+ # combine before --cov-report=xml, so it is a separate change with its own verification --
337+ # deliberately NOT bundled here, or a shifted coverage number would have two possible causes.
338+ timeout-minutes : 30
278339 permissions :
279340 contents : read
280341 outputs :
@@ -338,7 +399,72 @@ jobs:
338399 continue-on-error : true
339400 env :
340401 QT_QPA_PLATFORM : offscreen
341- run : pytest -q --cov=messagefoundry --cov-report=xml --cov-report=term-missing:skip-covered || true
402+ # `-m 'not tooling'` DESELECTS THE TOOLING TIER, AND IT COSTS THIS GATE NOTHING (#1409).
403+ # The mark is applied at runtime by tests/conftest.py from tests/tooling_manifest.txt (132
404+ # files), not by decorators, so grepping for `pytest.mark.tooling` finds 2 hits and badly
405+ # under-reads its size. ci.yml carries the same flag on its engine leg and runs the tier as
406+ # its own parallel job; this job was the only place in CI running it unfiltered AND serial.
407+ #
408+ # WHY DESELECTING IT REMOVES NO SIGNAL: the report is scoped to `--cov=messagefoundry`, and
409+ # ZERO of the 132 manifest files import that package. Measured with the repo's OWN
410+ # instrument, the _ENGINE_IMPORT regex in tests/test_tooling_partition.py, against a
411+ # positive control of 507 files under tests/ that DO import it -- so the zero is a real
412+ # absence and not a broken pattern. A test that never imports the measured
413+ # package cannot contribute a covered line through its OWN test bodies. The tier is the
414+ # run's hard tail, and the honest form of that is a RANGE, not one job's pair of numbers:
415+ # across the four killed jobs the suite reaches 89 percent at 14.1 to 15.6 minutes, and the
416+ # next two percentage points then cost 3.5 to 4.7 minutes. An earlier draft said "89
417+ # percent in 15.2 minutes, then the next two points cost 3.4 minutes", which is one job's
418+ # figures presented as the suite's -- and even for that job the second number is 3.47.
419+ # Note the percentages count TESTS, not time, which is exactly why the tail is so much
420+ # more expensive per point than the average.
421+ #
422+ # ONE QUALIFICATION, because "costs this gate NOTHING" is stronger than the grep supports.
423+ # The zero counts DIRECT imports. Reach is not the same thing: tests/test_dast_claims.py is
424+ # in the manifest and imports scripts.security.dast_auth_sweep, which pulls messagefoundry
425+ # modules transitively, and every import-time line of those counts as covered under --cov.
426+ # So deselecting the tier can drop some IMPORT-TIME coverage even though no tooling test
427+ # exercises the package directly. The measured effect is small and the trade is still
428+ # plainly worth it -- 14m33s against a job that produced no number at all -- but the honest
429+ # claim is "costs this gate almost nothing", not "nothing".
430+ #
431+ # THAT CONTROL WENT 510 -> 508 -> 507, AND THE LAST STEP IS THE INTERESTING ONE.
432+ # 510 is the same regex with the trailing word boundary dropped, so it also matched
433+ # messagefoundry_webconsole -- a DIFFERENT package. Adding \b gives 508. But 508 is still
434+ # one too many: tests/test_lifespan_startup_unwinds.py matches at lines 48-50, and those
435+ # lines sit INSIDE a 1256-character string literal beginning at line 42, which holds
436+ # child-process source written to a temp file. It is not an import this process executes.
437+ # An AST census (ast.Import / ast.ImportFrom over all 729 .py files under tests/, 0
438+ # unparseable) returns 507, and the regex-minus-AST difference is exactly that one file,
439+ # with no files the regex missed in the other direction.
440+ #
441+ # WHY THIS IS RECORDED AND NOT JUST FIXED: 508 was reported by two separate verification
442+ # passes that agreed with each other, and both were wrong the same way, because both ran
443+ # the SAME line-anchored regex. Agreement between runs of one instrument is one
444+ # measurement, not two. Only a different METHOD -- parsing instead of matching -- could see
445+ # a string literal. The ZERO this controls holds under every spelling tried, so no
446+ # conclusion moved; the control's own size did, twice.
447+ #
448+ # THE WEB CONSOLE TIER STAYS, and that is the one place this job must NOT copy ci.yml.
449+ # Both ci.yml legs carry `--ignore-glob='*messagefoundry-webconsole*'` (lines 824 and 1134)
450+ # because that package has its own job over there. Here it would DELETE REAL COVERAGE: 16 of
451+ # the tier's 17 PYTHON files import `messagefoundry` -- the 15 test modules plus conftest.py,
452+ # i.e. every file except a WebAuthn test double. They stand up a live `Engine` via
453+ # `Engine.create()` (searching for the constructor `Engine(` returns ZERO, so that needle is
454+ # recorded here beside its zero), and 13 of them build `create_app` and `AuthService`. They
455+ # exercise AT LEAST api/, auth/, pipeline/, config/, store/ and transports/signing -- "at
456+ # least" because a closed enumeration is a liability (SDS-3.6), and __main__ is imported too.
457+ # Collecting the tier alone loads 174 messagefoundry.* modules, every import-time line of
458+ # which --cov records as covered, which is what makes this the ONLY place in CI where the
459+ # tier reaches a coverage number. Measured 130 to 140 seconds in its own ubuntu ci.yml leg
460+ # (177 to 237 on Windows), which is a cheap price for coverage this gate reports.
461+ #
462+ # THIS PARAGRAPH PREVIOUSLY SAID "16 of those TEST FILES" AND "114 to 134 seconds"; neither
463+ # reproduced, and both are named rather than silently edited. The tier holds 15 test modules,
464+ # so a reader who asks the tree for 16 test files does not find them. The web console leg
465+ # measures 130-140s on ubuntu across the last 10 green ci.yml runs on main, and 114 is below
466+ # every sample taken. The conclusion "a cheap price" survives at 140s.
467+ run : pytest -q -m 'not tooling' --cov=messagefoundry --cov-report=xml --cov-report=term-missing:skip-covered || true
342468 - name : Diff-coverage vs the PR base (advisory - never fails)
343469 continue-on-error : true
344470 env :
0 commit comments