Skip to content
CI

backlog: close 9 landed items, retire a stale marker, progress-note 3 that stay open #2390

Sign in to view logs

backlog: close 9 landed items, retire a stale marker, progress-note 3 that stay open

backlog: close 9 landed items, retire a stale marker, progress-note 3 that stay open #2390

Workflow file for this run

.github/workflows/ci.yml at 92c7960
name: CI
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
schedule:
# Nightly (03:17 UTC — off the top-of-hour rush) run of the HEAVY POST-MERGE legs: the load
# smokes (load-test, load-test-sqlserver x2 majors), the server-DB suites (sqlserver-store x2
# majors, postgres-store), windows-service-smoke (x2 SKUs, 2x-billed) and docker-smoke. These
# all used to run on EVERY push to main (~16 auto-merges/day re-validating content whose PR run
# had just passed); the nightly cadence + the PR path-gates (`changes`) keep the same coverage
# at a fraction of the runs. `gh workflow run ci.yml --ref main` (workflow_dispatch) still runs
# EVERYTHING on demand — use it to re-validate a specific merge immediately.
# IMPORTANT: the cron must still NOT drag the FUNCTIONAL matrix along. `test` and `ide` gate
# their full-coverage arm as `push || workflow_dispatch` — deliberately EXCLUDING `schedule` —
# so the nightly never re-executes them (they already ran on the merge itself; the original
# `!= 'pull_request'` form would have swept them into the cron). The nightly legs' `if:` is
# `schedule || workflow_dispatch` (+ a PR path-gate arm where `changes` provides one).
- cron: "17 3 * * *"
concurrency:
# Per-ref group so a new push/PR-update supersedes its own in-flight run. The nightly `schedule` run
# gets a SEPARATE group (…-nightly): without this it would share `ci-refs/heads/main` with push-to-main
# and a routine daytime push would cancel-in-progress the in-flight nightly LOAD run — silently killing
# the very long-running coverage the cron exists to provide. Scheduled and push runs no longer collide.
group: ci-${{ github.ref }}${{ github.event_name == 'schedule' && '-nightly' || '' }}
cancel-in-progress: true
# Least privilege: every job here only reads the repo and runs tests/builds (no gh writes, no git push,
# no secrets), so a read-only default token is sufficient. A job needing more declares its own block.
permissions:
contents: read
jobs:
# Lint + type-check + unit tests. The project supports a single Python — 3.14 — so every leg runs it.
# Linux carries the cheap breadth (1x minutes); Windows Server 2022 + 2025 is the primary deployment
# target, so 3.14 is also exercised on both Server SKUs even though Windows bills at 2x — testing the
# version we ship on the OS we ship it on is worth the minutes.
test:
name: test (${{ matrix.os }}, py${{ matrix.python-version }})
# `needs: changes` for the docs-only short-circuit ONLY — this REQUIRED job still RUNS on every PR
# and ALWAYS reports its context. On a docs-only PR `changes.outputs.code == 'false'` and the
# expensive steps (install/lint/type/test) are skipped, so the leg goes green in seconds; the
# required `test (…, py3.14)` context stays present + green. (changes never fails, so this needs
# edge can't wedge the required context.)
needs: changes
# Every leg runs on GitHub-HOSTED runners now — the self-hosted NucBox Windows runners are RETIRED.
# This repo (MEFORORG) runs the full ubuntu + windows-2022 + windows-2025 matrix, all FREE (hosted
# minutes are free on a public repo); a FORK building its own pushes gets the UBUNTU leg only, to
# keep a contributor's personal CI cheap. Hosted-everywhere is also safer: a self-hosted runner must
# never be reachable by a public repo (fork PR = RCE on the owner's LAN). `matrix.hosted` is the
# runner label set; the per-repo matrix is built in `changes`.
runs-on: ${{ matrix.hosted }}
# Wall-clock backstop: bound each leg so a hung test fails in minutes, not the 6h default. Three
# nested watchdogs catch a hang at increasing scope (#55): the pytest-timeout per-test cap (matrix
# `pytest_timeout`, 60s ubuntu / 120s Windows -- the step passes `--timeout=`, which overrides the
# pyproject addopts value) + the faulthandler belt fire FIRST and name the stuck frame; the pytest STEP
# has its own `timeout-minutes` (matrix `step_timeout`) so a process-level deadlock below pytest
# fails the step fast; this job cap is the outermost belt if even that is somehow out-raced.
timeout-minutes: ${{ matrix.job_timeout }}
strategy:
fail-fast: false
# Per-repo matrix from the `changes` job: the full ubuntu+windows matrix here, ubuntu-only on a
# fork's own pushes. Each leg carries os / python-version / hosted (the runner labels, an array) +
# the timeout knobs. NB: a fork *PR* against this repo still runs in THIS repo's context, so it
# gets the full matrix — the required `test (windows-2022/2025, py3.14)` contexts are always
# present on a PR here, and a required-but-absent context (which blocks a PR forever) can't occur.
matrix: ${{ fromJSON(needs.changes.outputs.matrix) }}
defaults:
run:
shell: bash # available on all runners (Git Bash on Windows) -> one set of commands
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ matrix.python-version }}
# No `cache: pip`: installs now go through uv (the "Set up uv" step below), which keeps its
# OWN wheel/download cache — a pip cache here would be a second, redundant cache competing
# for the repo's 10 GB cache budget. uv's cache is keyed on pyproject.toml + requirements.lock.
# Everything below is gated on `code` (docs-only short-circuit) OR a push-to-main / on-demand run
# (full coverage). On a docs-only PR these steps no-op and the REQUIRED leg reports green in seconds.
# The job itself (and the Python/cache setup above) still runs, so the required context is always
# present. NB: the gate is duplicated per-step (a job-level `if` would SKIP the whole job and remove
# the required context) — keep the condition identical on each. The push/dispatch arm is written
# `push || workflow_dispatch` (NOT `!= 'pull_request'`) so the nightly `schedule` event does NOT
# re-run this full functional matrix — the cron is reserved for the long load legs only.
# DELIBERATELY UNGATED — the one step in this job with no `code` condition.
#
# The ledger gate polices the ADR/BACKLOG number space, and an ADR-only PR is by definition
# DOCS-ONLY, so `code == 'false'` and every gated step above is skipped. Gating this one the same
# way would skip it on exactly the pull requests it exists to police. It needs only git + stdlib
# python (no install), so it costs seconds and runs on every PR.
#
# It rides inside the already-REQUIRED `test` leg rather than becoming a new required context: a
# brand-new required check wedges every PR opened before it existed (see the `ci-gate` roll-up).
#
# This is the backstop for `git commit --no-verify` and for a branch cut from a stale main — each
# branch is internally consistent, and the duplicate number only exists once BOTH have merged, so
# only a check against freshly-fetched main can see it.
- name: Ledger gate (ADR / BACKLOG number space)
if: runner.os == 'Linux'
run: |
# Only `origin/main` needs fetching: the gate reads it (`git show origin/main:…`, `ls-tree`) and
# TWO-dot-diffs it against HEAD. No ancestry is resolved, so the shallow checkout is fine and no
# deepening is required -- see the long note in ledger_check.py's tree-access section for why
# three-dot was wrong here, and why its failure was silent.
git fetch --no-tags --depth=200 origin main
python scripts/hooks/ledger_check.py --ci
# ALSO DELIBERATELY UNGATED, for exactly the reason the step above is — and it was NOT, which
# cost `main`.
#
# Two guards police docs/BACKLOG.md. The number-space gate above is ungated because "gating it
# would skip it on exactly the pull requests it exists to police". The STATUS invariant was
# reachable only through pytest, which IS gated on `code == 'true'` — so on a BACKLOG-only PR
# (the exact shape it exists to check) it did not run at all.
#
# On 2026-08-01 that landed a #320 entry whose banner used an emoji the invariant does not
# accept. The PR was docs-only, went green in seconds without compiling the suite, merged, and
# red `main` for every other session until someone else's PR tripped over it. A 0.6s markdown
# parse would have caught it. One hardened guard and one unhardened guard over the same file is
# worse than neither, because the hardened one makes it reasonable to assume BACKLOG changes are
# covered.
#
# No install needed, so the step above's "git + stdlib python, costs seconds" property holds:
# scripts/docs/backlog_status_check.py imports only argparse/re/sys/pathlib and has its own CLI.
# tests/test_backlog_status_check.py imports that SAME module, so there is one implementation and
# the unit tests keep covering its edge cases — this only adds the always-on invocation.
# --min-items is the anti-narrowing floor, and it is the point of this invocation as much as the
# banner check is. The item namespace spans docs/BACKLOG.md AND docs/archive/backlog/: retiring
# items moves them between those files, so every OTHER assertion here — one banner per item, no
# contradictions, no duplicates — is satisfied just as easily by scanning a remnant of the corpus
# as the whole of it. Without a floor, a change that stopped the archive being read would go green.
# Raise the number when the total legitimately grows; it must never be lowered to make CI pass.
# WARNING: THE FLOOR LIVES IN TWO PLACES AND NOTHING COMPARES THEM: here, and `_MIN_TOTAL_ITEMS` in
# tests/test_backlog_status_check.py. Raise BOTH, or the lower one becomes the only floor that
# binds. Found on 2026-08-05 at 277 against a corpus of 300 — 23 items of accumulated slack, in a
# guard whose entire purpose is to notice the corpus shrinking.
- name: Backlog status invariant (ungated — see above)
if: runner.os == 'Linux'
run: python scripts/docs/backlog_status_check.py --min-items 300
# THE SAME ARGUMENT, ONE STEP FURTHER — and the evidence is from 2026-08-04.
#
# Ten `*_doc_drift` / `docs_*` modules exist to police documents, and every one of them is
# reachable ONLY through pytest, which is gated on `code == 'true'` two steps below. So on a
# docs-only PR — the exact shape they exist to check — none of them runs. That is the same defect
# the status invariant above was added to fix, at ten times the surface.
#
# It is not hypothetical. On 2026-08-04 four docs-only PRs merged (#197, #198, #200, #201) and the
# doc guards ran on none of them. Two carried citation errors found only by reading: an ADR named
# ONE inbound citation of `docs/releases/` where there are 23 (12 via `docs/releases/...`, 15 via
# relative `(releases/...`, and neither grep form alone finds them all), and a `docs/SECURITY.md`
# route-table row asserted a refusal `DELETE /me/mfa` does not make.
#
# IT RECURRED ON 2026-08-11, AND THIS STEP WAS ALREADY HERE — the list was simply incomplete.
# `tests/test_dast_claims.py` scans documents for prose reading as though the independence gap
# were closed, but was never added, so a docs-only PR (#322) merged green and RED MAIN on the push
# afterwards. The blind mode is worse than a plain gap because of WHO PAYS: the failure surfaces
# on the next PR that touches code, so it is misattributed to whoever opens it. That happened —
# the next code PR inherited a red its author had no part in.
#
# => THE LESSON IS ABOUT THE LIST, NOT THE GATING. A curated allowlist silently omits; nothing in
# a green run says "a doc guard exists that I did not run." When adding a doc-scanning test
# module anywhere in `tests/`, add it HERE in the same commit, and confirm it needs no extras —
# a module that cannot run on `[dev]` alone would red every docs-only PR, which is the failure
# mode the minimal-install note below exists to avoid.
#
# WHY A SEPARATE MINIMAL INSTALL instead of ungating the install below. Unlike the status
# invariant, these modules import the package (`config.settings`, `config.models`,
# `parsing.binary`, `auth.service`) and one imports `fastapi.routing`, so they cannot run on
# stdlib alone. But they need NO extras — fhir/dicom/x12/xml are irrelevant to a doc scan — so a
# docs-only PR pays a base editable install and ~5s of tests rather than the full extras install.
# The gated steps below are deliberately UNTOUCHED, so a code PR is byte-identical to before.
#
# WARNING: 89 of these tests SKIP here and that is structural, not a gap to fix in this step:
# `tests/test_threat_model_doc_drift.py` asserts against `docs/security/THREAT-MODEL.md`, which is
# vault-only and absent from this tree. ADR 0156 records that class (six `*_doc_drift` modules
# assert against documents `git ls-files docs/security/` shows are not present) and ASVS 15.1.3 is
# open on it. 152 assertions DO run, which is the point; do not read the skips as coverage.
# `[dev]` and not a bare `pip install pytest`, and this is NOT belt-and-braces. pyproject sets
# `asyncio_mode = "auto"` (needs pytest-asyncio) and `addopts = "--timeout=60
# --timeout-method=thread"` (needs pytest-timeout), so a bare pytest ERRORS on an unknown option
# before collecting anything — a step that could never pass, reding every docs-only PR. `[dev]` is
# the canonical "can run this suite" extra and pins both plugins; it is still far lighter than the
# gated install below, which adds console + fhir + dicom + x12 + xml that no doc scan touches.
# `--constraint constraints.lock` for the same DEP-1 reason every other install here carries it.
- name: Install (minimal — for the doc guards on a docs-only PR)
if: runner.os == 'Linux' && needs.changes.outputs.code != 'true' && github.event_name == 'pull_request'
run: |
python -m pip install --upgrade pip
pip install -e ".[dev]" --constraint constraints.lock
- name: Doc guards (ungated — the docs-only blind spot; see above)
if: runner.os == 'Linux' && needs.changes.outputs.code != 'true' && github.event_name == 'pull_request'
run: |
# PRINT WHAT IS SCANNED BEFORE RUNNING IT. A list that silently shrinks is how this class of
# guard goes quiet, and `-rs` names every skip rather than letting one read as a pass.
#
# ONE LIST, USED TWICE. It used to be written out twice — once for the printf and once for
# pytest — which made the "print what you scanned" defence able to LIE: the two copies could
# drift, and then the step prints a module it does not run, or runs one it does not print.
# A defence implemented by duplication defeats itself. The variable is the fix, and the
# cross-check below turns a drift into a hard failure rather than a quiet one.
# MEMBERSHIP RULE (#1262). A module belongs here if it CAN FAIL on a change touching only
# allowlisted paths -- because those are exactly the changes that skip the suite, so those
# are exactly the gates that would be silently absent at the only moment they are needed.
#
# DERIVE IT BY TRACING WHAT A MODULE OPENS, NOT BY READING ITS NAME OR ITS SOURCE. Measured
# 2026-08-14, both static shortcuts fail, in opposite directions:
# "source mentions a docs-ish token" -> 135 candidates; the signal drowns
# "source quotes a real allowlisted path" -> MISSES 8 OF THE 16 then listed here, because
# gates build paths (REPO / "docs" / name)
# rather than writing one literal
# A criterion that cannot re-derive the KNOWN members cannot be trusted to find unknown
# ones. The check that works is to patch open/Path.read_text and record which allowlisted
# paths a module actually reads while running.
#
# THIS LIST IS A FLOOR, NOT A CENSUS. The trace that produced the two additions below ran in
# an environment missing five CI extras, where 89 tests SKIPPED -- and a skipped test reads
# nothing. More members may be missing. Re-derive with the full extras before believing the
# set is complete.
#
# test_doc_ref_handle.py READS NO DOCUMENTATION -- it tests mfdoc:v1:ref: document handles
# in the message store, and is here because its NAME reads like "documentation reference".
# KEPT DELIBERATELY (dispatcher ruling, #1262): removing a guard is a different act from
# adding one, and dropping it on a reading of its name is the same move that put it here.
# The cost is a few wasted seconds; the cost of the other error is a gate that stops running.
DOC_GUARDS="tests/test_asvs_file_surface_doc_drift.py tests/test_cloud_phi_hipaa_doc_drift.py
tests/test_crit2_inline_doc_drift.py tests/test_doc_ref_handle.py tests/test_docs_db_grants.py
tests/test_docs_runbooks.py tests/test_docs_security_pathways.py tests/test_security_doc_drift.py
tests/test_security_doc_rate_limits.py tests/test_threat_model_doc_drift.py
tests/test_backlog_status_check.py tests/test_sds_rule_ids_are_stable.py
tests/test_link_resolution.py tests/test_dast_claims.py
tests/test_claude_section_citations.py tests/test_write_share_denominator.py
tests/test_cutover_slug_rot.py tests/test_backlog_citation_check.py
tests/test_dangling_citation_check.py"
# Every named module must EXIST. A path typo would otherwise make pytest error on an unknown
# file, or — worse under a future -k/--ignore form — silently scan nothing and read as a pass.
for m in $DOC_GUARDS; do
test -f "$m" || { echo "doc-guard list names a missing module: $m"; exit 1; }
done
echo "doc guards, docs-only PR — scanning these modules:"
printf ' %s\n' $DOC_GUARDS
echo " ($(printf '%s\n' $DOC_GUARDS | wc -l) modules)"
pytest -q -rs $DOC_GUARDS
# PySide6's offscreen platform plugin needs a few system libraries even
# headless. Linux-only; Windows runners need no equivalent.
- name: Install Qt offscreen system libraries
if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux'
# BOUNDED AND RETRIED. Unguarded, this step turns an external apt-mirror hiccup into a
# blocked merge queue: measured 2026-08-18, three hangs across two attempts on three
# different jobs, one of them 27+ minutes on `test (ubuntu-latest, py3.14)` -- a REQUIRED
# context, so the queue stops. Two levers, because they answer different failures. The
# per-command `timeout` kills a HANG and lets the retry run; `timeout-minutes` is the
# backstop that stops this step ever eating a job budget again if the loop is edited wrong.
# The failure text names the cause, so the next reader is not sent hunting in their diff --
# a check whose label points at the wrong subject is what BACKLOG #1254 is about.
timeout-minutes: 8
run: |
for attempt in 1 2 3; do
if sudo timeout 120 apt-get update && sudo timeout 180 apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3; then
exit 0
fi
echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying"
sleep $((attempt * 5))
done
echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test."
exit 1
# uv is the installer for every leg (was pip): it resolves + installs the pyproject extras far
# faster than pip, into the same setup-python interpreter via `uv pip install --system`. Pinned
# by the same SHA dependabot-lock-resync.yml already uses. Gated identically to the install
# below (docs-only PRs skip both). enable-cache defaults to 'auto' — a GitHub-hosted cache on
# hosted runners, the persistent on-disk uv cache on a self-hosted runner — keyed on the glob.
#
# EVERY install below passes `--constraint constraints.lock` — the HASHLESS export of uv.lock, kept
# in sync by the DEP-1 gate. Without it, `uv pip install -e ".[extras]"` RE-RESOLVES from
# pyproject's `>=` floors and silently adopts whatever upstream published since: ruff 0.16.0 and
# annotated-types 0.8.0 (dropped `SLOTS`, broke fhir-core) each reddened EVERY open PR on the same
# day — see each cap's rationale in pyproject.toml. The constraint pins every version to the lock
# while each job still installs only ITS OWN extras (so windows-service-smoke's deliberately
# narrow set stays narrow; the docker-smoke job builds from docker/locks/* and installs nothing).
- name: Set up uv
if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch'
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
cache-dependency-glob: |
pyproject.toml
requirements.lock
- name: Install project (dev + console + fhir + dicom + x12 + xml extras)
if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch'
run: |
# [fhir] so the FHIR typed-tier tests (incl. the PHI-no-leak invariant, ADR 0022) actually
# run their assertions in CI rather than importorskip-skipping (extra-less local runs skip).
# [dicom] (pydicom + pynetdicom; ADR 0025) does the same for the DICOM codec + C-STORE SCP
# tests, and lets mypy type-check transports/dicom.py + parsing/dicom/ against the real
# (py.typed) libraries rather than treating them as Any.
# [x12] (pyx12) + [xml] (lxml/xmlschema/signxml) likewise so the strict X12 validation (#32)
# and the hardened XML/SOAP codec (#31) tests run their assertions instead of importorskip-skipping.
# [webauthn] (ADR 0068) so the passkey ceremony tests run real verify_* assertions instead of
# importorskip-skipping, and mypy sees the py.typed library rather than Any.
# + the browser ops console as a second editable wheel (Option B) so the moved /ui tests
# (tests/test_webui.py etc. now import messagefoundry_webconsole) run on this full-suite leg.
# It is NOT a published extra yet (would break uv lock), so install it by path.
uv pip install --system --constraint constraints.lock -e ".[dev,harness,fhir,dicom,x12,xml,webauthn]" -e packaging/messagefoundry-webconsole
# ruff (lint + format) and mypy are PLATFORM-INDEPENDENT results — a ruff/format outcome is
# identical on every OS, and mypy's platform-specific branches are covered by running it for
# BOTH platforms on the cheap Linux leg (below). Running them on the 2x-billed Windows legs was
# duplicate cost, so gate them to Linux. The Windows legs keep pytest — the one check whose
# result genuinely differs by OS (real sockets, ProactorEventLoop, Windows service paths).
# `.` — the WHOLE repo, matching the format check below and the pre-commit ruff hook. NARROWED
# 2026-08-18: that hook used to lint every changed Python file, because `language: system` ran a
# bare `ruff check` with no `--force-exclude`, so an explicitly-passed path was linted even when
# pyproject excluded it. The hook is now upstream's, whose entry carries `--force-exclude`, so
# BOTH sides honour extend-exclude and the two agree for the first time — the hook was previously
# STRICTER than CI on excluded paths, not looser. This step was an explicit allow-list of six paths,
# and the mismatch was not academic: harness/ (73 findings), tee/ (12), samples/ and docker/ were
# linted by the hook and by nothing in CI, so touching any file there blocked the commit on errors
# CI could not see and no CI run could ever have reported. Scope now lives in ONE place —
# pyproject's [tool.ruff] extend-exclude — instead of being re-stated, differently, per tool.
# tests/test_lint_scope_parity.py fails if the two drift apart again.
- name: Lint (ruff)
if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux'
run: ruff check .
- name: Format check (ruff)
if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux'
run: ruff format --check .
# Licence headers (BACKLOG #1010). Mirrors the `licence-header` pre-commit hook. Scope lives in
# the script's COMMENT_PREFIXES map rather than being restated here, so unlike ruff and bandit
# these two callers cannot drift apart in the first place.
#
# DELIBERATELY NOT PATHS-FILTERED, unlike the ruff steps above. `changes.code` keys on
# `alwayscode='\.(py|ps1|sh|ts|js|yml|yaml|toml|lock|cfg|ini)$'`, which omits `.go` -- so a
# Go-only PR sets code=false and this gate would skip on precisely the PR that introduced the
# headerless file. No `.go` source is tracked today, but BACKLOG #1011 rules on whether a Go tree
# stays, and a gate that silently stops covering a language the moment one is added is the exact
# failure this item exists to remove. The scan reads ~1,300 files in under a second, so filtering
# it would buy nothing and cost a blind spot.
- name: Licence headers (SPDX)
if: runner.os == 'Linux'
run: python scripts/quality/licence_header_check.py
# Control bytes. Mirrors the `control-char` pre-commit hook, and exists for the reason zizmor.yml
# states for actionlint: the hook is the load-bearing half and this step is the backstop for
# `git commit --no-verify` and for a fresh clone whose hooks are not installed until
# scripts/coord/install-git-hooks.ps1 runs. The hook shipped without this step, so for one day the
# gate ran only where hooks happened to be installed -- a gate against bytes no human view shows,
# itself invisible when skipped.
#
# Scope AND the ide/src NUL allowance live in the script, not here, for the reason the licence
# step above gives: two callers that restate a scope drift apart, and the drift is silent.
#
# DELIBERATELY NOT PATHS-FILTERED, and the argument is sharper than for licence headers. This
# gate covers .md, .json and .txt, none of which appear in `changes.code`'s alwayscode pattern --
# so a docs-only PR sets code=false, and a filter would skip the gate on exactly the prose where
# a pasted escape most often lands. ~1,850 files, stdlib only, well under a second.
- name: Control bytes (invisible)
if: runner.os == 'Linux'
run: python scripts/quality/control_char_check.py
# BACKLOG #1226. The screen shipped and NOTHING RAN IT, so it read as coverage and produced
# none -- a username reaching a WHERE clause was found three times by accident and zero times
# on purpose. --baseline is what makes this step able to FAIL without turning the screen into
# a verdict-emitter: it is silent on the judged sites and red on a key nobody has read yet.
# Several judged entries are deliberately CORRECT code and one is a confirmed defect, so the
# baseline records that a human looked, never that a site is fine. stdlib only.
- name: Username-as-access-key screen (new sites only)
if: runner.os == 'Linux'
run: >-
python scripts/quality/username_access_key_screen.py
--baseline scripts/quality/username_access_key_baseline.txt
# Two mypy passes on Linux so BOTH platforms' typed branches are checked without paying for a
# Windows mypy: the default linux-platform run, plus an explicit --platform win32 run that types
# the sys.platform=='win32' branches (ctypes.windll / DPAPI / service_control) the linux run
# skips. Previously those were typed only by the mypy on the Windows legs. (All win32 code uses
# stdlib ctypes — typeshed's win32-conditional stubs cover it; there are no win32-only deps — so
# this resolves on a Linux runner. Verified: both passes clean on 204 files.)
# `messagefoundry/tray` (the Windows notification-area service-manager, ADR 0113 — now shipped
# inside the wheel) is a win32-only ctypes package, so it is EXCLUDED here and typed on the win32
# pass only: its ungated ctypes.WinDLL method calls are unreachable-and-skipped under the linux
# platform but real under win32. Nothing in the engine imports it (it is a leaf entry point), so
# --exclude cleanly drops it from the linux build.
- name: Type-check (mypy, strict — linux platform)
if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux'
run: mypy messagefoundry messagefoundry_webconsole --exclude 'messagefoundry/tray/'
# The win32 pass types messagefoundry/tray (now inside the messagefoundry package, so no separate
# path argument) plus every other sys.platform=='win32' branch.
- name: Type-check (mypy, strict — win32 platform)
if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux'
run: mypy --platform win32 messagefoundry
# Step-level watchdog UNDER the job cap (#55): the windows-2022 leg intermittently
# wedges ~25% in (a Windows ProactorEventLoop listener-teardown / socket wait that the shared
# session event loop can't get past), emits no output for ~12 min, then the JOB cap CANCELS it
# — a red X with no stack and no named test. Two belts make that fail FAST and NAMED instead:
# * `--timeout-method=thread` (the ONLY method on Windows — SIGALRM is POSIX-only) dumps ALL
# thread stacks at the per-test cap (matrix `pytest_timeout`, 120s on these Windows legs),
# naming the stuck frame.
# * PYTHONFAULTHANDLER=1 + `-o faulthandler_timeout=` (matrix `fault_timeout`) is the belt that fires even when the
# thread-timer CANNOT interrupt a main-thread C-level wait (a Proactor overlapped op /
# blocking accept / subprocess.wait): pytest's faulthandler plugin arms a separate watchdog
# thread that dumps the NATIVE stack of every thread at `fault_timeout` (150s here, 90s on
# ubuntu -- always set ABOVE that leg's `pytest_timeout`, so the
# per-test dump is attributed first; this is the last-resort stack when the thread method is
# itself out-raced). NB the plugin exposes this as the `faulthandler_timeout` ini key, not a
# `--faulthandler-*` CLI flag, so it is passed with `-o`; it only DUMPS (never kills), safe.
# The step `timeout-minutes` (matrix `step_timeout`) is the outer backstop, held well under the
# job's own cap, so a hang both watchdogs somehow miss still fails the STEP (not the whole job at
# the silent cap). Keep the gap: the step must expire BEFORE the job, or the failure surfaces as
# an uninformative job-level kill with no step attribution.
#
# THAT GUARANTEE NOW COVERS BOTH GATED STEPS, and it did not until BACKLOG #344 proposal 5 landed.
# `Web console tests (pytest)` used to carry the SAME `step_timeout` while running AFTER
# `Tests (pytest)`, so reaching it had already spent setup + `Tests` and its own cap could not
# fire first at any job_timeout worth setting (~55 min on ubuntu, ~114 on Windows) -- a hang there
# was an unattributed job-level kill with no step conclusion. It now carries
# `matrix.webconsole_step_timeout`, sized to its own work, and the nesting arithmetic holds on all
# three legs; the derivation is in the note above that step.
#
# It only works while the cap stays clear of a HEALTHY run. On 2026-07-31 the ubuntu leg finished
# green in 775s against a 780s cap -- 5s of margin -- and the step was killed anyway, reported as
# a failure on a run whose last line was "9598 passed, 855 skipped". A watchdog that cannot
# separate "deadlocked" from "slow today" stops being a watchdog and becomes a coin flip, so the
# ubuntu budget is raised here. Re-check the margin when the suite grows.
#
# It then happened again on WINDOWS, 2026-08-01, because the sentence that used to close this
# paragraph asserted the Windows legs held "~2x headroom" and nobody re-derived it. PR #119 was
# killed at 26:07 against the 26:00 cap with ZERO tests failing. What moved was the suite, not the
# code under test: #74 landed tests/test_worktree_prune_merged.py (1,506 lines) and windows-2025
# went 19:35 -> 26:07 on the same branch. The claim was already false when it was written.
#
# Measured by timing the `Tests (pytest)` STEP, which is what step_timeout gates. NOT the job: the
# job runs several minutes longer and is capped separately, and at least three sessions misread job
# durations as step durations while triaging this (c53f752b's JOB ran 28:41 and PASSED, because
# job cap 30 vs step cap 26).
#
# POOL: every run of THIS workflow created 2026-08-01T00:00Z .. 2026-08-09T00:00Z UTC -- 688 runs.
# Enumerated with `gh api` over a deliberately WIDER page range (1,000 runs, back to 2026-07-26) and
# filtered locally on `created_at`. Jobs fetched with `?filter=all`, so an attempt killed at the cap
# is not hidden behind its passing re-run. Rows are each leg's `Tests (pytest)` STEP, kept when THAT
# STEP concluded, executions under 60s dropped (a docs-only PR skips the leg). 1,618 leg-executions;
# n below is per leg, passing rows only. Measured 2026-08-08 (BACKLOG #1096).
#
# leg max passing step n old cap old margin new cap new margin
# ubuntu-latest 20:28 35 25:00 1.221x 31:00 1.515x <- RE-DERIVED 2026-08-12
#
# THE UBUNTU ROW WAS RE-DERIVED 2026-08-12 AND THE OLD ONE IS KEPT ABOVE IN THE PROSE, NOT DELETED.
# Pool: every run of this workflow created after 2026-08-10 that contributed an ubuntu
# `Tests (pytest)` STEP -- 58 runs, 63 rows, 35 passing. Max passing 20:28; next highest 17:18;
# min 14:47; max non-success 17:01. Smaller pool than the 441-row original and stated as such.
#
# WHAT TRIGGERED IT, and the ordering matters: `main` ALREADY exceeded the old 16:08 anchor at
# 17:07 BEFORE the PR that reddened the gate. The record had rotted on its own; PR #343 (+274
# tests, +3:12) only carried the MARGIN across the 1.30x floor. So this is not "one PR was too
# big" -- the anchor was stale and the floor did its job by saying so. #1096 stays OPEN: the caps
# are re-derived, the DRIFT underneath them is still not fixed, and raising a cap is not fixing it.
#
# 20:28 IS PR #343's OWN RUN, and that is deliberate rather than an outlier being smuggled in: its
# 274 tests land on `main` when it merges, so 20:28 is the next baseline, not a one-off. Sizing on
# the pre-merge population would re-break the cap on contact -- the "three PRs each adding a
# minute" death this note already records.
# windows-2022 29:23 399 36:00 1.225x 55:00 1.872x
# windows-2025 35:47 CENSORED 360 36:00 1.006x 55:00 1.537x
#
# ONLY THE WINDOWS-2025 ROW IS CENSORED, AND THAT IS MEASURED, NOT ASSUMED. Largest ubuntu step
# FAILURE in 535 leg-executions is 14:21; largest windows-2022 failure in 542 is 25:48. Neither leg
# has ever touched its cap, so 16:08 and 29:23 are true maxima. windows-2025 was killed at the cap
# SEVEN times in this window -- 36:08 (31189317409, main b78214f5), 36:08 (31199321840, main
# 7ecff8ae), 36:08 (31202372465, PR #253), 36:07 (30955150892, 2026-08-04, the earliest), 36:07
# (31094875320, main fdec72ca), 36:07 (31192717684, PR #249), 36:01 (31149117314, PR #261) -- THREE
# of them push runs on main. So 35:47 is the largest step that FIT in 36:00, and 36:08 is a lower
# bound on the largest the suite wants. Smaller pools give smaller counts and each is right for its
# own pool: BACKLOG #1096's day pool has five, a 3-day pool has six. State the pool with the count.
#
# THE KILLS WERE SLOWNESS, NOT A WEDGE -- CHECKED, NOT ASSUMED. Run 31149117314's pytest finished
# GREEN and the step was killed 4.85 seconds later:
# 05:37:53.777Z 10666 passed, 831 skipped, 22 warnings in 2148.75s (0:35:48)
# 05:37:58.624Z ##[error]The action 'Tests (pytest)' has timed out after 36 minutes.
# No faulthandler native-stack dump appears in any kill log. That settles the hang-versus-slow
# question the #55 note above would otherwise leave open, and it has a second consequence that is
# easy to miss: the killed rows are genuine population members, so "max passing" is CENSORED, not
# conservative. Every ratio taken against it flatters itself.
#
# SUPERSEDED READING, kept because a number without its pool cannot be rechecked: the 2026-08-01
# pool (70 runs, pre-#131) read ubuntu 12:31 / W22 21:34 / W25 25:51 against the then-current
# 26:00 Windows cap. Those figures are correct for that pool and that code state. They are not
# comparable to the rows above, which are post-#131 and span eight days.
#
# THESE MAXIMA ARE LOWER BOUNDS, BECAUSE THE POOL IS RIGHT-CENSORED. Every run in it predates #131
# (28d186b5, landed 2026-08-02T00:35:28Z), so Windows ran under step 26:00 / job 30:00 and ubuntu
# under 19:00 / 22:00. A run that wanted longer than its cap was KILLED at the cap and then --
# correctly, by the filter above -- dropped for not concluding success. So 25:51 is the largest
# step that FIT in 26:00, not the largest the suite wants. Sizing a cap as a multiple of a
# censored maximum under-provisions by construction, and that is exactly how "1.06x" read as
# survivable right up until #119's leg was killed at 26:07. (#119 itself merged, 2026-08-02T01:45Z
# -- what the cap killed was a run, not the PR. Saying a PR "died" here reads as never-landed and
# had already propagated into docs/WORKTREES.md as exactly that claim.)
#
# THE CENSORED VALUES ARE VISIBLE IF YOU ASK FOR THEM, AND THEY MATTER. The jobs endpoint defaults
# to `filter=latest`, which hides a killed attempt behind its passing re-run; `?filter=all` shows
# both. #119's 26:07 is in this very pool that way (run 30717229521 attempt 1, `8c407fb5`, step
# conclusion FAILURE) beside its 22:25 attempt-2 success -- a 3:42 spread on identical code, and
# the first day AFTER the raise produced 26:23 TWICE, both concluding SUCCESS (runs 30728793103
# and 30731407003). So the population really does exceed the old 26:00 cap: the largest windows-2025
# `Tests (pytest)` execution observed to date is 26:23, not 25:51.
#
# THIS TABLE HAS BEEN WRONG TWICE. #131 published 12:27 / 18:39 / 24:35 over "11 passing runs";
# the first correction published the maxima above but over "101 runs" with n = 57 / 52 / 49. The
# maxima re-derive exactly; BOTH pool sizes were unreproducible. Three mechanisms produced that,
# all cheap to repeat:
#
# * THE POOL WAS A PAGE, NOT A QUESTION. #131's came from `gh run list --limit 20` -- a default
# page size, reported as though it had been chosen.
# * FILTERING ON *JOB* CONCLUSION DROPS THE TIGHTEST STEPS BY CONSTRUCTION. A step that nearly
# exhausts step_timeout is the most likely to push its job into job_timeout, so the job is
# cancelled while the step itself concluded success. Re-running this pool with a job-conclusion
# filter reproduces #131's 24:35 exactly; the step-conclusion filter gives 25:51. Measure the
# step; filter on the step. Two such rows exist on 2026-08-01 and one of them is the maximum.
# * A COUNT NOBODY RE-DERIVED. Neither "101 runs" nor n = 57 / 52 / 49 is reproducible under any
# pool definition tried. A table whose own point is "state your pool and your n" has to carry
# an n the next reader can recompute -- that column is how they would tell it had rotted.
#
# Only the windows-2025 row was ever a maximum; #131's ubuntu 12:27 was that same run's ubuntu leg,
# quoted as if it were that leg's worst case. Its 18:39 was not even that: that run's windows-2022
# step is 18:29, and 18:39 belongs to a different run entirely.
#
# State the MEASURED value, its POOL, its n and its DATE -- never a bare multiple. A multiple gives
# the next reader no way to tell when it has rotted; a multiple without its pool cannot even be
# rechecked.
#
# PROOF IT WAS THE CAP, NOT THE BRANCH. #119's windows-2025 leg was RE-RUN on the SAME commit
# against the SAME 26:00 cap: attempt 1 was killed at the cap, attempt 2 concluded SUCCESS. Same
# code, same config, same ceiling, two outcomes. The leg was not failing -- it was coin-flipping
# against the cap, exactly the state the ubuntu note above names, and the reason "re-run it and
# see" is not a diagnosis here. A green re-run at 26:00 does not mean the suite fits; it means
# that runner was fast enough that time.
#
# THE CAP IS SIZED ON THE POST-#1027 POPULATION, NOT ON WHAT RUNS TODAY. PR #253 (BACKLOG #1027) is
# open and MERGEABLE and sets testpaths = ["tests", "packaging/messagefoundry-webconsole/tests"],
# while keeping the `Web console tests (pytest)` step -- so once it lands the web console suite runs
# inside `Tests (pytest)` AS WELL. Sizing on the current population would let #253 land and re-break
# the cap on contact, which is the "three PRs each adding a minute" death recorded at the foot of
# this note. The one windows-2025 execution of that branch is one of the seven kills above, so the
# post-#1027 windows-2025 duration HAS NEVER BEEN OBSERVED UNCENSORED -- and windows-2022's sizing
# maximum (29:23) IS a post-#1027 row. One shared value, two code states, until this is fixed.
#
# leg largest observed Tests + largest observed web-console = post-#1027 anchor
# ubuntu 20:28 (uncensored) 2:22 (n=441) 22:50 <- RE-DERIVED 2026-08-12
# W22 29:23 (uncensored; already a post-#1027 row) 29:23
# W25 36:08 (RIGHT-CENSORED) 3:59 (n=360) 40:07 lower bound
#
# SIZING RULE: step_timeout = ceil_minute(1.35 x that leg's post-#1027 anchor). 1.35 is the multiple
# #131 used when it set 36:00 (1.364x over 26:23, the largest execution then observed).
# ubuntu 22:50 x 1.35 = 30:50 -> 31:00 (1.358x over the anchor, 1.515x over max passing)
# W22 29:23 x 1.35 = 39:40 -> 40:00 (takes the shared Windows value below)
# W25 40:07 x 1.35 = 54:09 -> 55:00 (1.371x over the anchor, 1.537x over max passing)
#
# UBUNTU IS CHANGED IN THE SAME ACT BECAUSE IT IS THE NEXT INSTANCE, NOT AS TIDYING. At 19:00 against
# its post-#1027 anchor of 18:30 it stands at 1.027x -- within seconds of where windows-2025 is now --
# and its job cap is ALREADY NEGATIVE on measured maxima (see the job paragraph below). Publishing
# "ubuntu unchanged, still positive" beside a corrected Windows row would be a compensating control
# resting on a false premise, which is the defect this repo names specifically.
#
# THE SPREAD RULE NOW HOLDS, WHERE AT 36:00 IT DID NOT. The rule is "headroom must exceed observed
# spread". At 36:00 it FAILED on windows-2025 (headroom 9:37 against a 10:27 spread) and the old
# revision kept the value anyway and said so. At 55:00, on the eight-day pool:
# windows-2025 headroom 55:00 - 35:47 = 19:13 spread 35:47 - 24:48 = 10:59 HOLDS (+8:14)
# windows-2022 headroom 55:00 - 29:23 = 25:37 spread 29:23 - 18:42 = 10:41 HOLDS (+14:56)
# The 24:48 low is same-suite runner variance, not a smaller suite: three consecutive push-to-main
# runs inside 2h42m gave windows-2025 34:15 / 24:48 / 34:29 while windows-2022 held a 1:00 band.
#
# RE-DERIVE IF a windows-2025 `Tests (pytest)` STEP is ever seen above 40:00, or if any Windows
# `test` JOB is ever seen above 50:00 -- those are the triggers, not a calendar reminder. Both are
# the 1.2x decay point of their cap (55:00 / 1.2 = 45:50, floored to 40:00 for margin against the
# measured drift; 66:00 / 1.2 = 55:00, floored to 50:00). The OLD trigger of 28:00 is retired
# because it was already universally exceeded: 58 of 59 passing steps on the day pool cleared it and
# the median itself sat above it, so it had become a permanently-tripped alarm, which is
# indistinguishable from no alarm at all.
#
# A ratio against one run says nothing about a distribution, and #119's leg was killed by the
# distribution, not by its own duration. Both Windows legs take the same values, but they are sized
# from DIFFERENT legs and that asymmetry is deliberate: the STEP cap is sized on windows-2025 (the
# slower step, 35:47 censored against 29:23), while the JOB cap is sized on windows-2022 (the worse
# non-step addends, setup 4:05 against 2:32). An earlier revision said "sized on windows-2025 as the
# worse of the pair" without qualification -- that is FALSE for the job cap.
#
# THE JOB CAP IS NOT A ROUNDING-UP OF THE STEP CAP, AND IT HAS FIRED. Two steps in this job carry
# a `timeout-minutes`, so the job must contain the SUM of two gated budgets plus setup -- a
# quantity neither step cap can bound on its own.
#
# UNTIL BACKLOG #344 PROPOSAL 5 THAT SUM WAS UNBOUNDABLE, because both steps drew on the SAME
# `step_timeout`: the worst case the caps admitted was 2 x step_timeout + setup, which no
# job_timeout worth setting could cover (2 x 25 = 50 > 37 on ubuntu, 2 x 55 = 110 > 66 on
# Windows). The caps were sized against the OBSERVED sum instead, which is a weaker guarantee and
# was recorded here as such. The web console step now carries `webconsole_step_timeout`, so the
# worst case the caps admit is finite and positive on every leg -- the arithmetic is in the note
# above that step, and it is the first time this file has been able to state it.
# Observed for real on run 30724385719 (main @ 8f01cef8, 2026-08-01):
#
# Tests (pytest) 00:01:21 -> 00:27:12 25:51 SUCCESS (9s under the 26:00 cap)
# Web console tests (pytest) 00:27:12 -> 00:30:14 CANCELLED
# JOB 00:00:06 -> 00:30:19 30:13 CANCELLED <- job_timeout 30 fired
#
# A GREEN first step, then an unattributed job-level kill during the second -- exactly the failure
# the nesting note above exists to prevent, arriving by the path that note does not consider. It
# does NOT happen when a step is killed (that ends the job and skips what follows, so those
# durations never sum); it happens when the first step PASSES near its budget.
#
# READ THAT EXHIBIT FOR ITS MECHANISM, NOT AS A VERDICT ON THE CAP IT REPLACED: it ran under the
# retired 26/30 pair and would have passed under #131's 40:00. What condemns 40:00 is the
# arithmetic below -- which is arithmetic, not an observed kill; no job has yet hit 40:00.
#
# job_timeout is therefore sized against BOTH gated steps plus setup, rather than as step_timeout
# plus a constant. Each addend below is that addend's MEASURED MAXIMUM over the same pool, taken
# over rows where `Tests` concluded success -- NOT over rows where both steps did, which silently
# drops the very run that motivates this whole paragraph (its web-console step was cancelled) and
# is the same censoring mistake as filtering the step table by job conclusion. Two earlier
# revisions used a 0:41 setup (the MEDIAN), web-console values that were each only third-highest
# on their leg, and a 1:04 Windows setup that the exhibit above contradicts on its face:
#
# THE BAND MOVED OFF step_timeout AND ONTO THE OVERHEAD, 2026-08-08 (BACKLOG #1096). The earlier
# formula multiplied the WHOLE sum (step_timeout + addends) by ~1.13-1.16. That applies a safety
# band to `step_timeout`, which is a HARD BOUND the runner enforces -- the largest step overrun in
# 1,618 measured executions is EIGHT SECONDS (36:08 against a 36:00 cap). Banding it is slack for an
# event that cannot happen, it grows with the cap (4:41 at 36:00, 7:09 at 55:00), and it makes the
# published ratio pool-dependent because the step term dominates it. Corrected rule:
#
# job_timeout = step_timeout + ceil_minute(1.5 x that leg's worst measured job OVERHEAD)
#
# where overhead = job wall clock minus the `Tests (pytest)` step, taken as the larger of (a) the
# worst SAME-ROW overhead on non-cancelled rows and (b) that leg's setup(max) + web-console(max).
# The 1.5 band is on the overhead only, and is 1.5 rather than 1.35 because setup is hosted-runner
# provisioning outside this repo's control: `Set up job` alone has been observed at 4:30 (ubuntu run
# 31109989006, inside a 5:22 setup).
#
# leg worst same-row setup(max)+wc(max) overhead used x1.5 room job
# ubuntu 7:12 (n=451) 5:22 + 2:22 = 7:44 7:44 11:36 12:00 31+12 = 43
# W22 6:16 (n=413) 4:05 + 2:51 = 6:56 6:56 10:24 11:00 55+11 = 66
# W25 6:29 (n=372) 2:32 + 3:59 = 6:31 6:31 9:47 10:00 (takes W22's 66)
#
# INSTRUMENT PRESERVATION, the whole point of the pairing -- worst-case job wall built from
# independent maxima with `Tests` passing one second under its cap, new values against old:
#
# ubuntu 5:22 + 30:59 + 2:22 = 38:43 vs 43:00 +4:17 | old: 32:43 vs 37:00 +4:17 (band held)
# W22 4:05 + 54:59 + 2:51 = 61:55 vs 66:00 +4:05 | old: 42:55 vs 46:00 +3:04
# W25 2:32 + 54:59 + 3:59 = 61:30 vs 66:00 +4:30 | old: 42:30 vs 46:00 +3:30
#
# UBUNTU WAS THE LEG ACTUALLY BROKEN, at -0:43, and nobody had measured it: BACKLOG #1096 was filed
# as a Windows problem. A negative row here means the JOB cap can fire before the STEP cap, and a
# job-level kill reports NO step conclusion -- so the instrument this item was measured with is
# destroyed exactly when it is needed. Max measured post-step teardown is 0:24, so all three new
# rows stay positive with room.
#
# THE THIRD ADDEND ABOVE IS THE OBSERVED web-console maximum, which is a DIFFERENT QUANTITY from
# the worst case the caps now admit -- one describes what has happened, the other what is
# permitted. Capping that step (BACKLOG #344 proposal 5) does not change how long it takes, so the
# rows above stand and `job_timeout` is UNCHANGED by it. The cap-based worst case is stated once,
# in the note above the `Web console tests (pytest)` step, and it is positive on every leg.
#
# NESTING INVARIANT, first gated step -- setup(max) + step_timeout must stay under job_timeout, so a
# `Tests` kill is a NAMED STEP failure rather than an unattributed job cancellation:
# ubuntu 5:22 + 25:00 = 30:22 < 37:00 (gap 6:38)
# W22 4:05 + 55:00 = 59:05 < 66:00 (gap 6:55)
# W25 2:32 + 55:00 = 57:32 < 66:00 (gap 8:28)
#
# 66 IS DERIVED, NOT ROUNDED UP FROM 55. The bare minimum whole minute that clears the worst sum is
# 61:00; it is rejected as thinner than any row this file has ever carried, and thinner than the
# three-addend model's own known error -- the worst observed single row sums to 40:16 against a real
# job wall clock of 40:22, so inter-step gaps make the model an UNDER-estimate by ~6s, not a
# conservative one.
#
# SUPERSEDED, kept because a number without its pool cannot be rechecked. The previous table read:
# ubuntu 19:00 + 2:00 + 1:20 = 22:20 (26:00, +3:40) | W22 36:00 + 2:33 + 1:09 = 39:42 (46:00,
# +6:18) | W25 36:00 + 3:33 + 1:20 = 40:53 (46:00, +5:07). Correct for its pool; superseded above.
#
# The `+4` habit that produced 40 was Windows-only
# and was carried through two cap changes unchecked (30/26 then 40/36); ubuntu has never been +4 --
# it went 15/13 to 22/19, so +2 then +3. Either way the number was derived from the OTHER BOUND
# rather than from the work, which is the defect, not the particular constant.
#
# WHAT THIS SIZING DID NOT FIX, AND WHAT SINCE FIXED IT. This sizing only made the job cap cover
# the OBSERVED sum. The nesting invariant held for `Tests (pytest)` on every leg and for
# `Web console tests (pytest)` on NONE of them, because both steps drew on one `step_timeout`:
# guaranteeing the second would have needed job_timeout above setup + 2 x step_timeout -- about
# 55:22 on ubuntu and 114:05 on Windows -- so a hang in the web console step surfaced as an
# unattributed job-level kill, the very failure the invariant is there to prevent. BACKLOG #344
# proposal 5 closed it the other way round: the web console step now has `webconsole_step_timeout`
# sized to its own work (a 2-to-4-minute suite has no business holding 55:00), which brings
# setup + step_timeout + webconsole_step_timeout under job_timeout on all three legs. The
# arithmetic, and the trigger for re-deriving it, live in the note above that step.
#
# AND KNOW WHAT IS NOT CAPPED AT ALL. `test` is the ONLY one of this file's ten jobs carrying any
# `timeout-minutes`; the other nine -- `changes`, `ide`, `sqlserver-store`, `postgres-store`,
# `load-test`, `load-test-sqlserver`, `windows-service-smoke`, `docker-smoke`, `ci-gate` -- run on
# GitHub's 6-HOUR default. That is recorded here so the omission is known rather than assumed
# deliberate; sizing them is out of scope for this change and belongs with BACKLOG #344.
#
# This cap is NOT what catches a hung test; pytest_timeout (120s) is, per test. The step cap only
# catches a whole-process deadlock both in-process watchdogs miss, which is why it can sit well
# above a healthy run. Sizing it tight buys no detection and costs false failures on green suites,
# twice now.
#
# The remaining margin is a SHARED budget across every PR that lands, and it is now ACCOUNTED
# FOR: `scripts/ci/step_margin.py` runs after the gated step below and reds the leg when a
# step's own duration comes within 1.30x of its own cap (BACKLOG #344 proposal 1). It used to run
# after BOTH gated steps; the web console suite moved to its own `webconsole` job and took its
# half of the check with it, so each job now measures the one gated step it owns. Three PRs
# each adding a minute of Windows time is still individually blameless -- the difference is that
# the third one now says so instead of the fourth one dying at the cap with zero failing
# assertions. It is NOT a request to raise a cap; the underlying slowness is #320.
#
# THE CLOCK IS TAKEN IN ITS OWN STEP so neither gated `run:` body has to carry timing code that
# a kill would skip anyway. Elapsed is therefore the step plus its two transitions -- an UPPER
# bound on the step, so the reported margin is a LOWER bound on the true margin, and the error
# runs toward firing early rather than toward a false green.
#
# It marks THROUGH THE SCRIPT rather than `echo ... >> "$GITHUB_ENV"`, deliberately. That
# spelling routes a Windows path through a bash redirection on two of the three legs -- a
# construct no local run can exercise, whose failure mode is a silently absent variable, i.e. a
# check that stops measuring while still printing a verdict. Marking in Python keeps the path
# handling where the tests reach it, and a missing mark is a REFUSAL (exit 2), never a zero.
- name: Step margin -- start the clock
if: always()
run: python scripts/ci/step_margin.py --mark before-tests
- name: Tests (pytest)
id: tests
if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch'
timeout-minutes: ${{ matrix.step_timeout }}
env:
QT_QPA_PLATFORM: offscreen
PYTHONFAULTHANDLER: "1"
# Pass the matrix-derived timeout knobs through env instead of expanding ${{ }} straight into
# the run: shell. The per-repo matrix is now built at runtime in `changes` (fromJSON), so zizmor
# can no longer statically prove these are literal ints and flags the in-run expansion as
# template-injection; env-passing removes the script sink. `defaults.run.shell: bash` applies on
# every OS, so $VAR resolves the same on the Windows legs (Git Bash).
FAULT_TIMEOUT: ${{ matrix.fault_timeout }}
PYTEST_TIMEOUT: ${{ matrix.pytest_timeout }}
PYTEST_WORKERS: ${{ matrix.pytest_workers }}
# BACKLOG #1260: empty on purpose -- see the comment above the run line. The wrapper's
# default clause names the pyodbc class, which is not established for this leg.
RETRY_NATIVE_CRASH_CAUSE: ""
# `--ignore-glob` subtracts the web console package, which runs as its OWN step below. BACKLOG
# #1027 added that package to the root `testpaths` so a bare local `pytest` stops silently
# excluding it; without this subtraction the SAME 356 tests would then run TWICE on every leg.
# Measured 2026-08-08 on this branch: bare collection 11,956, console-only 356, with this flag
# 11,600.
# NOT `--ignore=<path>` -- measured, it does NOT prune a directory that `testpaths` names as a
# collection root, and still collected all 11,956. The obvious spelling is the one that fails.
# NOT `pytest tests` either: hardcoding the engine path would make CI silently miss any future
# third entry in `testpaths`, which is the exact drift #1027 exists to prevent. Subtracting FROM
# `testpaths` keeps it the single source of truth for what the suite is.
# `tests/test_ci_engine_step_excludes_webconsole.py` pins all three of those decisions.
#
# `-n` RUNS THE SUITE ACROSS PROCESSES (pytest-xdist), and it is the whole reason this step
# stopped being 40 minutes. Measured 2026-08-16 on this branch: serial 2191s vs `-n 4
# --dist loadfile` 511s locally, with the failure set IDENTICAL in both runs (the same six
# pre-existing environment failures, zero parallelism-induced). That box has 20 cores and a
# hosted runner has 4, so the CI figure is expected to be lower than that 4.29x -- re-read
# the step duration from the first runs rather than quoting the local number.
#
# `--dist loadfile` IS LOAD-BEARING, NOT A PREFERENCE. xdist's default is `--dist load`, which
# scatters individual tests from one file across workers, and this suite has two things that
# breaks. (1) 44 module-scoped fixtures live outside conftest and would be rebuilt once PER
# WORKER instead of once -- `tests/test_worktree_prune_merged.py` alone pays ~3s of setup per
# test. (2) `tests/test_multishard_smoke.py` is the sole consumer of MEFOR_TEST_PORT_BASE and
# binds a real port window; keeping a file's tests together keeps that window inside one
# process. The cost of `loadfile` is that the run cannot finish faster than its single largest
# FILE, which is why the per-file hot spots are worth attacking next.
# `-m 'not tooling'` DESELECTS the repo-harness tier, which the `tooling` job runs instead.
# Measured 2026-08-16 on a 20-core box: 1,882 of 12,990 tests carry the mark but they are 66
# percent of the suite's TIME, and the single largest file in the tier (test_worktree_prune_
# merged.py, 391.4s) was setting this step's `--dist loadfile` floor on every engine leg.
#
# Deselection is by MARKER, applied in tests/conftest.py from tests/tooling_manifest.txt. Not by
# path, and not by a second `--ignore-glob`: the webconsole exclusion right above already shows
# what happens when the selection lives in the run body -- the obvious spelling (`--ignore`) was
# the one that silently did nothing. One list, read in one place, asserted by
# tests/test_tooling_partition.py, which also pins the two halves of this wiring.
# BACKLOG #1260: WRAPPED SO A NATIVE CRASH IS NOT REPORTED AS A TEST FAILURE. A segfault kills
# the interpreter, so pytest returns 139/134 with no verdict and THREE layers of naming then
# say "tests failed" -- the check name, the step name, and `steps.tests.outcome`, which is what
# `step_margin.py` below consumes. None of them is true: the engine was fine and a process died.
# The wrapper re-runs ONLY on 139/134 and re-raises exit 1 immediately, so it cannot mask a
# regression.
#
# RETRY_NATIVE_CRASH_CAUSE IS DELIBERATELY EMPTY HERE. The wrapper's default clause names the
# pyodbc py3.14 class, which is ESTABLISHED for the database legs and is NOT established for
# this one -- the item filing this refuses to conclude it. An empty value makes the annotation
# say CAUSE NOT ESTABLISHED rather than assert a mechanism nobody has measured on this leg.
run: bash scripts/ci/retry-native-crash.sh pytest -q -n "$PYTEST_WORKERS" --dist loadfile -m 'not tooling' --ignore-glob='*messagefoundry-webconsole*' -o faulthandler_timeout="$FAULT_TIMEOUT" --timeout="$PYTEST_TIMEOUT"
# THE MARGIN CHECK (BACKLOG #344 proposal 1). Last in the job so a LOW margin cannot skip a suite
# that has not run yet -- a step `if:` with no status function carries an implicit `success()`,
# so a failing check placed ahead of a gated step would silently skip it.
#
# IT NOW GUARDS ONE STEP, NOT TWO. The web console suite moved to its own `webconsole` job and
# took its half of this check with it, so the `between` mark is gone: this reads
# `before-tests .. now`, which is the whole of the only gated step left in this job. The
# invariant did not weaken -- it was RE-SCOPED. Each job now holds exactly one gated step, its
# own cap, and its own check, which is a stronger shape than one check straddling two steps
# whose budgets had to be summed under a single job cap.
#
# It keys on `steps.<id>.outcome` -- THE STEP'S OWN CONCLUSION, never the job's. That substitution
# is not pedantry: a step that nearly exhausts its cap is the most likely to push its job into
# `job_timeout`, so filtering on the job deletes the tightest rows by construction, and doing
# exactly that reproduced a published maximum of 24:35 where the truth was 25:51.
#
# A skipped step (docs-only PR) reports NO OBSERVATION in words rather than a healthy-looking
# ratio, and the script runs its own red/green control pair on every invocation and prints both
# into the job summary -- a gate that has never been red is a claim, not a control.
- name: Step margin -- engine suite
if: always()
env:
TESTS_OUTCOME: ${{ steps.tests.outcome }}
STEP_CAP: ${{ matrix.step_timeout }}
LEG: ${{ matrix.os }}
run: |
rc=0
# `|| rc=$?` OUTSIDE the command, so a non-zero exit is captured rather than swallowed by
# `set -e` or absorbed into a pipeline.
python scripts/ci/step_margin.py --step "Tests (pytest)" --leg "$LEG" \
--cap-minutes "$STEP_CAP" --outcome "$TESTS_OUTCOME" \
--since before-tests --until now || rc=$?
exit $rc
# THE WEB CONSOLE SUITE, IN ITS OWN PARALLEL JOB (was a second step on every `test` leg).
#
# WHY IT MOVED. It was ~170s sitting SERIALLY behind a ~1090s engine suite on the critical-path leg,
# for no reason other than sharing a checkout. Run alongside instead of after, it costs zero
# wall-clock: its whole job (setup + suite) finishes far inside the engine job it now runs beside.
#
# THIS REVERSES THE PLACEMENT HALF OF BACKLOG #344 PROPOSAL 5, AND KEEPS ITS SUBSTANCE. That proposal
# gave this suite its own `webconsole_step_timeout` because two steps sharing one `timeout-minutes`
# meant the second one's cap could never fire first -- reaching it had already spent setup plus the
# engine suite. Moving it to its own job does not undo that; it completes it. The suite keeps its own
# STEP cap, gains its own JOB cap (`webconsole_job_timeout`), and keeps its own margin check. The
# nesting invariant is now setup(max) + webconsole_step_timeout < webconsole_job_timeout in ONE job,
# instead of a sum of two step budgets straddling a single job cap -- strictly easier to satisfy and
# to check, which is why the guard in tests/test_ci_step_margin.py is re-scoped per job rather than
# relaxed.
#
# IT IS NOT A REQUIRED CONTEXT AND DOES NOT NEED TO BE. `ci-gate` lists it in `needs:` and fails on
# any needed job's `failure` or `cancelled`, and `CI gate` IS required -- the same mechanism that
# already gates the six path-gated legs. Adding a new required context would instead have meant a
# branch-protection change and the required-but-absent trap on every docs-only PR.
#
# THE DOCS-ONLY GATE IS PER-STEP, NOT PER-JOB, exactly as in `test` and for the same reason: a
# job-level `if:` would skip the job outright, and `ci-gate` reads `needs.*.result`, where a skipped
# job reports `skipped` rather than `success`. Keeping the job running and no-oping its steps keeps
# that roll-up honest on a docs-only PR.
webconsole:
name: web console tests (${{ matrix.os }}, py${{ matrix.python-version }})
needs: changes
strategy:
fail-fast: false
# Same per-repo matrix as `test`: this suite is as platform-sensitive as the engine one (it runs
# under the package's OWN pytest config, against a real ASGI test client), so it gets the same
# legs rather than a cheaper ubuntu-only run.
matrix: ${{ fromJSON(needs.changes.outputs.matrix) }}
runs-on: ${{ matrix.hosted }}
timeout-minutes: ${{ matrix.webconsole_job_timeout }}
defaults:
run:
shell: bash
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ matrix.python-version }}
- name: Install Qt offscreen system libraries
if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux'
# BOUNDED AND RETRIED. Unguarded, this step turns an external apt-mirror hiccup into a
# blocked merge queue: measured 2026-08-18, three hangs across two attempts on three
# different jobs, one of them 27+ minutes on `test (ubuntu-latest, py3.14)` -- a REQUIRED
# context, so the queue stops. Two levers, because they answer different failures. The
# per-command `timeout` kills a HANG and lets the retry run; `timeout-minutes` is the
# backstop that stops this step ever eating a job budget again if the loop is edited wrong.
# The failure text names the cause, so the next reader is not sent hunting in their diff --
# a check whose label points at the wrong subject is what BACKLOG #1254 is about.
timeout-minutes: 8
run: |
for attempt in 1 2 3; do
if sudo timeout 120 apt-get update && sudo timeout 180 apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3; then
exit 0
fi
echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying"
sleep $((attempt * 5))
done
echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test."
exit 1
- name: Set up uv
if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch'
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
cache-dependency-glob: |
pyproject.toml
requirements.lock
# THE SAME INSTALL LINE AS `test`, deliberately. The console suite imports the engine, so it must
# be exercised against the same build; keeping the commands identical (rather than trimming to a
# console-only set) means the two jobs cannot drift into testing different resolutions of the same
# lock. It also shares the uv cache entry -- same key, already warm from the sibling job.
- name: Install project (dev + console + fhir + dicom + x12 + xml extras)
if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch'
run: |
uv pip install --system --constraint constraints.lock -e ".[dev,harness,fhir,dicom,x12,xml,webauthn]" -e packaging/messagefoundry-webconsole
- name: Step margin -- start the clock
if: always()
run: python scripts/ci/step_margin.py --mark before-webconsole
# The web console's OWN suite (Option B, ADR 0065): the moved /ui tests live in the package's
# tests/. They ARE in the root `testpaths` since BACKLOG #1027, so a bare local `pytest` collects
# them -- the engine step in `test` subtracts them explicitly via `--ignore-glob` rather than
# relying on `testpaths` to exclude them, which it no longer does. The package is installed
# editable above, so both suites exercise the same engine source.
#
# NO `-n` HERE, deliberately. xdist buys the engine suite ~2x across ~13,000 tests; this is ~356
# tests in 2-to-4 minutes, where worker startup would eat most of the gain, and the job now runs
# concurrently with the engine one anyway -- its wall-clock is already off the critical path.
# Adding workers would spend the contention risk the -n 8 trial measured to buy nothing.
- name: Web console tests (pytest)
id: webconsole
if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch'
timeout-minutes: ${{ matrix.webconsole_step_timeout }}
env:
QT_QPA_PLATFORM: offscreen
PYTHONFAULTHANDLER: "1"
# env-passed (not a ${{ }} expansion into run:) -- see the engine Tests step for why.
FAULT_TIMEOUT: ${{ matrix.fault_timeout }}
PYTEST_TIMEOUT: ${{ matrix.pytest_timeout }}
run: pytest packaging/messagefoundry-webconsole/tests -q -o faulthandler_timeout="$FAULT_TIMEOUT" --timeout="$PYTEST_TIMEOUT"
# This job's half of the margin check (BACKLOG #344 proposal 1), moved here with the suite it
# measures. Same contract as the engine one: last in the job, `if: always()` so a step killed at
# its cap is still reported, and keyed on `steps.webconsole.outcome` -- the STEP's own conclusion,
# never the job's, because a step that nearly exhausts its cap is the most likely to push its job
# into `timeout-minutes` and filtering on the job would delete exactly those rows.
- name: Step margin -- web console suite
if: always()
env:
WEBCONSOLE_OUTCOME: ${{ steps.webconsole.outcome }}
WEBCONSOLE_CAP: ${{ matrix.webconsole_step_timeout }}
LEG: ${{ matrix.os }}
run: |
rc=0
python scripts/ci/step_margin.py --step "Web console tests (pytest)" --leg "$LEG" \
--cap-minutes "$WEBCONSOLE_CAP" --outcome "$WEBCONSOLE_OUTCOME" \
--since before-webconsole --until now || rc=$?
exit $rc
# The repo-HARNESS tier, lifted off the engine legs' critical path. Membership is
# tests/tooling_manifest.txt; the `test` legs deselect it with `-m 'not tooling'` and this job is the
# only place it runs. Measured 2026-08-16: 94 files, 1,882 tests, 66 percent of the suite's time.
#
# UBUNTU + WINDOWS, and the ubuntu-only version of this job was WRONG -- recorded because the
# reasoning that produced it was superficially sound. The first cut argued the tier is mostly
# OS-agnostic shell/git behaviour and that Windows is where its child-spawn cost hurts most. Measured
# instead of argued: 18 of the 89 manifest entries carry a module-level skipif on os.name/sys.platform
# and they hold 337 of the tier's 1,882 tests -- 18 percent. Those ran on windows-2022 and
# windows-2025 before the split; under ubuntu-only they would have executed on NO leg, on any event.
# They are the coordination and worktree-gate core: session_mail (89), prune_merged (71),
# announce_hook (45), collision_gate (31), every coord_*.
#
# The sizing number in the first cut was invalid on its own terms too: the 391.4s attributed to
# test_worktree_prune_merged.py is a WINDOWS figure for a file that has always skipped on ubuntu, so
# it could never have been that leg's `--dist loadfile` floor.
#
# windows-2025 only, not both Windows legs. The engine matrix carries 2022 and 2025 because the
# ENGINE ships onto both; this tier tests scripts/ against the local git and pwsh, where the
# 2022-vs-2025 difference is not a surface any listed test asserts on. That is a deliberate reduction
# from the pre-split state and the honest cost of the split -- if a harness regression ever lands that
# only windows-2022 would have caught, THIS is the sentence that was wrong.
#
# NOT a required context of its own: it rides the `ci-gate` roll-up, exactly as `webconsole` does and
# for the same reason recorded there -- a new required context needs a branch-protection change, and
# until that lands every PR carries a check it can never satisfy.
tooling:
name: repo harness tests (${{ matrix.os }})
needs: changes
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-2025]
# The push arm is the safety net, not redundancy: the PR arm only fires when the harness itself
# changed, so an ENGINE change that breaks a harness test is invisible on its own PR by
# construction, and push-to-main is where it surfaces. See the `tooling` output on `changes`.
if: needs.changes.outputs.tooling == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch'
runs-on: ${{ matrix.os }}
# SIZED FOR THE WINDOWS LEG, NOT THE UBUNTU ONE, and the first cut (25 job / 18 step) was too
# tight to survive it. Derivation, stated so it can be re-derived rather than trusted: the tier
# runs 431s at -n 4 on a 20-PHYSICAL-core box, with the pwsh/git children this tier spawns landing
# on spare cores. A hosted runner has 4 vCPU and no spare -- ci.yml already records ~1.12 cores of
# average child occupancy for exactly this work, and the -n 8 trial documented above shows what
# oversubscription does here. Windows adds its ~2.4x process-spawn tax on top, and the Windows leg
# is the ONLY leg that executes the 337 platform-gated tests at all. A 2.5-3x factor on 431s is
# 18-21 minutes, which the old 18-minute step cap sat directly on top of.
#
# These are DEADLOCK BACKSTOPS, not tightness gates -- the same reasoning the engine matrix records
# for its own 66/55. Sizing a cap close to the observed run buys no detection and costs red builds
# on green suites, and a gate that cries wolf gets deleted. The engine Windows legs carry 66/55 for
# a bigger suite; 40/30 here is proportionate and still well clear.
#
# RE-DERIVE IF the tier's -n 4 wall clock is ever observed above 25 minutes on the Windows leg, or
# if pytest_workers changes.
timeout-minutes: 40
defaults:
run:
shell: bash
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0 # several of these tests read git history (ledger, claim, cutover guards)
- name: Set up Python 3.14
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
# NEEDED EVEN THOUGH NO HARNESS TEST TOUCHES Qt, because `-m tooling` FILTERS AFTER COLLECTION.
# pytest imports every module under `testpaths` to collect it and only then applies the marker
# expression, so this job still imports the four PySide6 modules (test_harness_compose,
# test_harness_file, test_harness_monitor, test_console_messages_refresh) and dies on
# `ImportError: libEGL.so.1` before a single tooling test runs. Measured on this job's first real
# CI run: 1494 passed, 388 skipped, 4 COLLECTION ERRORS -- deselection does not mean not-imported.
#
# That is also why the install line below cannot be trimmed to a harness-only set: collection
# imports the whole suite regardless of what this job intends to execute.
- name: Install Qt offscreen system libraries
if: runner.os == 'Linux'
# BOUNDED AND RETRIED. Unguarded, this step turns an external apt-mirror hiccup into a
# blocked merge queue: measured 2026-08-18, three hangs across two attempts on three
# different jobs, one of them 27+ minutes on `test (ubuntu-latest, py3.14)` -- a REQUIRED
# context, so the queue stops. Two levers, because they answer different failures. The
# per-command `timeout` kills a HANG and lets the retry run; `timeout-minutes` is the
# backstop that stops this step ever eating a job budget again if the loop is edited wrong.
# The failure text names the cause, so the next reader is not sent hunting in their diff --
# a check whose label points at the wrong subject is what BACKLOG #1254 is about.
timeout-minutes: 8
run: |
for attempt in 1 2 3; do
if sudo timeout 120 apt-get update && sudo timeout 180 apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3; then
exit 0
fi
echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying"
sleep $((attempt * 5))
done
echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test."
exit 1
- name: Set up uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
cache-dependency-glob: |
pyproject.toml
requirements.lock
# THE SAME INSTALL LINE AS `test`, and it cannot be trimmed even though no listed test imports the
# engine: tests/conftest.py itself does (`from messagefoundry.config.settings import ...`), so the
# package must be importable for collection to happen at all. Keeping the line identical also
# shares the uv cache entry with the sibling jobs.
- name: Install project (dev + console + fhir + dicom + x12 + xml extras)
run: |
uv pip install --system --constraint constraints.lock -e ".[dev,harness,fhir,dicom,x12,xml,webauthn]" -e packaging/messagefoundry-webconsole
# `-m tooling` SELECTS what the engine legs deselected. The two spellings are a pair and
# tests/test_tooling_partition.py asserts BOTH are present in this file, because either one alone
# is a silent half-failure: without the engine-side `not tooling` the split buys nothing, and
# without this the tier stops running anywhere while both jobs still report green.
#
# `--dist loadfile` for the same reason as the engine step: these tests bind real port windows and
# own module-scoped fixtures, so a file's tests must stay in one worker.
- name: Harness tests (pytest)
timeout-minutes: 30 # see the job cap above for the derivation; must stay under it with setup
env:
# Same reason as the apt step above: collection imports the Qt modules whether or not this
# job executes them, and the shipped ones construct QApplication at import on some paths.
QT_QPA_PLATFORM: offscreen
run: pytest -q -n 4 --dist loadfile -m tooling --ignore-glob='*messagefoundry-webconsole*' --timeout=120 --junitxml=tooling-junit.xml
# A marker typo, a manifest rename, or a conftest hook that silently stops firing all produce the
# same thing: zero selected tests and a GREEN job. Deselection cannot be distinguished from success
# by exit code -- pytest exits 5 on "no tests collected", which `-q` reports in one line nobody
# reads.
#
# COUNT WHAT EXECUTED, NOT WHAT WAS COLLECTED. The first version of this step grepped
# `--collect-only`, and pytest COLLECTS skipif-marked tests -- verified against a synthetic file
# whose every test was skipped: "2 tests collected". So it would have printed a healthy ~1,900 and
# passed while 337 Windows-gated tests skipped on an ubuntu-only job. A receipt that cannot tell
# green-because-it-ran from green-because-it-skipped is not a receipt. This parses the junit the
# run itself produced, so the number is the number that happened.
#
# The floor is deliberately loose (1,000 against ~1,545 executing on ubuntu and ~1,882 on
# Windows). It is a DEAD-MARKER backstop, not a coverage assertion: a tight floor would go red
# every time the tier legitimately grows or a platform gate moves, and a gate that cries wolf gets
# deleted. Coverage drift is tests/test_tooling_partition.py's job, on evidence, not this one's.
# A SCRIPT, NOT AN INLINE HEREDOC. The logic and its reasoning live in the script, where they can
# be read, linted, type-checked and run locally -- and where a shell that handles heredocs
# differently cannot change the answer. This job is the first in this workflow to run a `run:`
# body on BOTH ubuntu and Windows, so an inline heredoc here would have been the first of its
# kind on the Windows leg; that is not a thing to discover from a red build.
- name: Prove the tier actually ran
if: always() # a timeout or a crash is exactly when the receipt matters most
run: python scripts/ci/tooling_receipt.py --report tooling-junit.xml
# Build + type-check the VS Code extension, and run its integration tests. The Python jobs never
# touch ide/, so a dep bump or IDE code change that breaks the bundle or the types would otherwise
# pass CI unbuilt (this job exists because an esbuild bump merged "green" without anything having
# compiled it). The build runs on both OSes; the @vscode/test-electron suite (which downloads and
# launches a real headless VS Code to assert the extension activates and its commands register/run)
# runs on the Windows leg — Electron launches headless there with no xvfb, and Windows is a
# deployment target.
ide:
name: ide build (${{ matrix.os }})
# `ide` is NOT a required check and ci-gate does NOT `needs: ide`, so skipping it — or varying its
# matrix per repo (below) — can't wedge any required context. It runs when it can be affected:
# push-to-main + workflow_dispatch (full coverage), and PRs that touch ide/** or this workflow (the
# `changes.ide` path-gate) — a pure-Python PR can't change the isolated npm project's result, so it's
# skipped there. Not on the nightly cron: the engine heavy legs need that safety net, an isolated npm
# build gated on its own paths does not. The per-repo os matrix below runs the 2x-billed
# windows-latest electron leg here (this repo is public, so hosted runners are free) and drops it on
# a fork, where those minutes come out of the fork owner's own allowance.
needs: changes
if: github.event_name == 'workflow_dispatch' || needs.changes.outputs.ide == 'true'
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
# Per-repo os matrix (built in `changes`): ubuntu + windows-latest on THIS repo, ubuntu-only on a
# fork. The windows-latest leg is the 2x-billed one that runs the @vscode/test-electron integration
# suite; this repo is public so hosted minutes are free, while on a fork they are billed to the
# fork owner. `ide` is not a required check and ci-gate does not `needs` it, so dropping the
# windows leg on a fork can't wedge anything.
matrix: ${{ fromJSON(needs.changes.outputs.ide_matrix) }}
defaults:
run:
working-directory: ide
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Set up Node
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: "24"
cache: npm
cache-dependency-path: ide/package-lock.json
- name: Install (clean, from lockfile)
run: npm ci
- name: Type-check (tsc --noEmit)
run: npm run typecheck
- name: Bundle (esbuild)
run: npm run compile
# The vscode-free suites (the pure model layer: the engine link state + its two frozen boundary
# allowlists, the settings-scope SEC-005 invariant, the graph/steps/HL7 models, …). They need no
# Extension Host, so they run on EVERY leg — including the ubuntu-only leg a fork gets.
# Before this step they ran NOWHERE on an ubuntu-only matrix: `npm test` is Windows-only, so the
# entire node-side estate was type-checked and never executed there.
# ADR 0110's allowlists are only "asserted in CI" because of this line.
- name: Unit tests (node-side, no VS Code)
run: npm run test:unit
- name: Integration tests (headless VS Code)
if: runner.os == 'Windows'
run: npm test
# Path filter for the container-pulling server-DB leg(s). A docs/unrelated PR shouldn't pull the SQL
# Server image (mcr.microsoft.com anonymous pulls are rate-limited — repeated PRs flake on
# "toomanyrequests"). This job decides whether the SQL Server leg needs to run: ALWAYS true on
# workflow_dispatch (full coverage); FALSE on push-to-main (the container legs run nightly +
# PR-path-gated, not per merge); on a PR, true only when store / cluster / server-DB tests /
# this workflow changed. (NB: if `sqlserver-store` is later made a *required* check in branch
# protection, a path-skipped run reports "skipped" — use a ruleset that treats skips as success, or
# switch to an always-run job whose heavy steps are step-gated, since the service container starts
# before steps.)
changes:
name: detect server-DB + docker changes
runs-on: ubuntu-latest
outputs:
serverdb: ${{ steps.f.outputs.serverdb }}
docker: ${{ steps.f.outputs.docker }}
# `code` is the docs-only short-circuit signal: TRUE unless this PR touches ONLY non-code paths
# (Markdown / docs / licence / editorconfig / git metadata). It gates the EXPENSIVE STEPS of the
# required `test` legs (and the whole `ide` job) — the jobs still run + report their (required)
# context, but on a docs-only PR they skip install+lint+type+test and go green in seconds. TRUE on
# push-to-main / workflow_dispatch (full coverage); FALSE on the nightly `schedule` (load-legs-only
# cron — the functional matrix must not re-run nightly).
code: ${{ steps.f.outputs.code }}
# `ide`: run the ide build job only when a PR touches ide/** or this workflow (true on
# push/dispatch for full coverage; false on the nightly cron). Not a required check.
ide: ${{ steps.f.outputs.ide }}
# `tooling`: the repo-HARNESS test tier — worktree gate, coordination claims, session mail,
# ledger, announce, and the workflow files themselves. Measured 2026-08-16: 94 files and 1,882
# tests, 17 percent of the suite by count but 66 percent by TIME, because every one of them
# spawns real pwsh/git children (~1.0s per test against 0.10s for an engine test). Its SUBJECT is
# the development harness, so an engine-only PR cannot change its result — yet before this it set
# the `--dist loadfile` floor for all three engine legs. The `test` legs now deselect it with
# `-m 'not tooling'` and the `tooling` job below runs it, path-gated. Membership is
# tests/tooling_manifest.txt, NOT a filename pattern — see that file for why.
tooling: ${{ steps.f.outputs.tooling }}
# `matrix`: the `test` job's include list, per-repo — full ubuntu+windows matrix HERE (this repo is
# public, so hosted runners are free; the self-hosted Windows runners are retired), ubuntu-only on
# a fork. Consumed as `matrix: ${{ fromJSON(needs.changes.outputs.matrix) }}` in `test`.
matrix: ${{ steps.f.outputs.matrix }}
# `ide_matrix`: the `ide` job's os matrix, per-repo — ubuntu + windows-latest HERE, where the
# 2x-billed VS Code electron leg runs free; ubuntu-only on a fork. Consumed as
# `matrix: ${{ fromJSON(needs.changes.outputs.ide_matrix) }}` in `ide`.
ide_matrix: ${{ steps.f.outputs.ide_matrix }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
- id: f
# Hoist the two workflow expressions into env (zizmor: no `${{ }}` interpolated into the run body
# — a branch name / base sha is attacker-influenceable on a fork PR, so it must arrive as data).
env:
EVENT_NAME: ${{ github.event_name }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
# test matrix, per-repo — computed FIRST (independent of the event, before any early exit).
# THIS repo (MEFORORG/MessageFoundry, the source since the cutover): full ubuntu + windows-2022
# + windows-2025 on FREE hosted runners — free because the repo is public.
# Anywhere else (a FORK): the UBUNTU leg ONLY, so a contributor's own minutes are not spent on
# the 2x-billed Windows legs. This branch used to mean "the private source repo"; that repo is
# now an inactive archive with Actions disabled, so a fork is the only thing that reaches it.
# The self-hosted Windows NucBox runners are retired either way.
# `hosted` = the runner labels (an array; see the `test` job's runs-on).
# $GITHUB_REPOSITORY is a built-in runner env var, read here as plain shell (NOT a workflow-
# expression interpolation into the run body), so it is zizmor-safe and cannot be misparsed as
# an Actions expression the way a literal double-brace token in a run: block would be.
# `webconsole_step_timeout` is the web console suite's OWN cap (BACKLOG #344 proposal 5). It
# used to take `step_timeout` -- a 2-to-4-minute suite holding a 25-to-55-minute budget --
# which is why the nesting invariant could not hold for it on any leg. Derived in the note
# above the `Web console tests (pytest)` step, which is also where to re-derive it.
# `pytest_workers` is the xdist worker count for the engine suite. It is a MATRIX knob rather
# than a literal in the run body for the same reason every other per-leg number here is one:
# the trial dispatches that size it are then a one-line edit, and a leg can differ from its
# siblings without touching the step. Sized at 4 because GitHub's hosted ubuntu and Windows
# runners are 4-vCPU; `-n auto` would resolve to the same 4 today but would silently
# over-subscribe if the runner size ever changes, and this suite spawns pwsh/git children
# that compete for those same cores (measured: ~1.12 cores of average child occupancy), so
# the count is stated rather than inferred. IF TIMING-SENSITIVE TESTS START FLAKING, 2 IS THE
# CONSERVATIVE RUNG -- it leaves 2 vCPU of headroom for the dispatcher and connscale
# assertions that measure elapsed time, at roughly two thirds of the saving.
#
# DO NOT RE-RUN THE "MORE WORKERS MIGHT BE FREE" EXPERIMENT: 8 WAS MEASURED AND IT IS WORSE ON
# EVERY LEG. The tempting argument is that a worker blocked waiting on a pwsh/git child is not
# holding a core, so 4 workers on 4 vCPU are under-subscribed. It is wrong. Dispatched
# 2026-08-16 on branch claude/ci-worker-count-trial at 2bb733915 (run 31968796353), all three
# legs raised to 8 together so each compares against its own baseline:
#
# leg -n 4 baseline (n=2) -n 8 (n=1) delta -n 8 outcome
# ubuntu-latest 630 / 633s 731s +16% RED, 2 timing assertions
# windows-2022 967 / 1004s 1067s +8% green
# windows-2025 1084 / 1092s 1366s +25% RED, 3 tests
#
# The -n 4 baselines are two runs agreeing within 4 percent (PR #411 and its post-merge push).
#
# -n 4 IS NOT CLEAR OF THE TIMING FLOOR, ONLY FURTHER FROM IT. Recorded because the paragraph
# below reads as "8 is unsafe, 4 is settled" and the second half is now known to be too
# strong. Observed 2026-08-17 on PR #420, run 32005389065, ubuntu-latest, at -n 4:
# FAILED tests/test_send_pacing.py::test_single_message_seam_is_paced
# assert 0.03876582899999903 >= (0.05 * 0.8)
# Missed by 1.2 ms -- 0.0388 against a required 0.0400 -- on a PR whose diff does not touch
# that file (two coordination files, one of them pwsh-gated and skipped on ubuntu). So a 20
# percent tolerance on a 50 ms interval does not reliably clear a shared runner even at the
# merged worker count. If this is to be fixed, the lever is the TOLERANCE or the pacing
# mechanism, not pytest_workers -- lowering workers to 2 would buy margin by accident and
# cost wall clock everywhere.
#
# ONE THING THAT SHOULD HELP, and is stated as a prediction rather than a result: the tooling
# partition takes ~1,880 subprocess-spawning tests off this leg, and those are precisely the
# contention this floor is sensitive to. All 7 pacing tests passed on a local -n 4 engine leg
# under the partition, the seam test at 0.194s. That is one green run on a 20-core box, which
# is not evidence about a 4-vCPU runner -- if the flake recurs on a partitioned leg, this
# prediction was wrong and the tolerance is the only remaining lever.
# What went red is the mechanism, not bad luck: ubuntu lost the two elapsed-time assertions
# (test_outbound_batch.py's pacing floor and a super-linear-scaling ratio), and windows-2025
# lost the three heaviest multi-process tests -- connscale_smoke, multishard_smoke and
# session_mail's claim-verdict test, which is the slowest single test in the suite. Contention
# hits exactly the tests that measure time or bind real ports, which is why the failures
# cluster there rather than scattering.
# `webconsole_job_timeout` caps the SEPARATE web console job. It exists because that suite
# moved out of `test` into its own parallel job, so it now needs a job cap of its own -- and
# the nesting invariant has to hold there too: setup(max) + webconsole_step_timeout must fit
# under it. Recorded setup maxima are 5:22 ubuntu / 4:05 W22 / 2:32 W25, and ci.yml records
# provisioning noise alone at 4:30, so the worst case is about 5:22 + 4:30 + 5:00 = 14:52 on
# the widest leg. 20 clears that with room. It is a DEADLOCK BACKSTOP, not a tightness gate:
# the drift detector is the margin check against `webconsole_step_timeout`, so sizing this
# close to the observed run would buy no detection and cost false reds on green suites.
U='{"os":"ubuntu-latest","python-version":"3.14","hosted":["ubuntu-latest"],"job_timeout":43,"step_timeout":31,"webconsole_step_timeout":5,"webconsole_job_timeout":20,"pytest_timeout":60,"fault_timeout":90,"pytest_workers":4}'
W22='{"os":"windows-2022","python-version":"3.14","hosted":["windows-2022"],"job_timeout":66,"step_timeout":55,"webconsole_step_timeout":6,"webconsole_job_timeout":20,"pytest_timeout":120,"fault_timeout":150,"pytest_workers":4}'
W25='{"os":"windows-2025","python-version":"3.14","hosted":["windows-2025"],"job_timeout":66,"step_timeout":55,"webconsole_step_timeout":6,"webconsole_job_timeout":20,"pytest_timeout":120,"fault_timeout":150,"pytest_workers":4}'
if [ "${GITHUB_REPOSITORY:-}" = "MEFORORG/MessageFoundry" ]; then
echo "matrix={\"include\":[$U,$W22,$W25]}" >> "$GITHUB_OUTPUT"
else
echo "matrix={\"include\":[$U]}" >> "$GITHUB_OUTPUT"
fi
# ide matrix, per-repo (computed here with the test matrix, before any early exit): the
# 2x-billed windows-latest electron leg runs only HERE, where hosted minutes are free; a fork
# builds + type-checks the extension on ubuntu only. Consumed as
# matrix: fromJSON(needs.changes.outputs.ide_matrix) in the ide job. (No shell vars here, so a
# single-quoted literal JSON needs no escaping and carries no run-block workflow expression.)
if [ "${GITHUB_REPOSITORY:-}" = "MEFORORG/MessageFoundry" ]; then
echo 'ide_matrix={"os":["ubuntu-latest","windows-latest"]}' >> "$GITHUB_OUTPUT"
else
echo 'ide_matrix={"os":["ubuntu-latest"]}' >> "$GITHUB_OUTPUT"
fi
# Manual dispatch = full coverage: force every gate true so the whole workflow runs.
if [ "$EVENT_NAME" = "workflow_dispatch" ]; then
echo "serverdb=true" >> "$GITHUB_OUTPUT"
echo "docker=true" >> "$GITHUB_OUTPUT"
echo "code=true" >> "$GITHUB_OUTPUT"
echo "ide=true" >> "$GITHUB_OUTPUT"
echo "tooling=true" >> "$GITHUB_OUTPUT"
exit 0
fi
# Push-to-main: the FUNCTIONAL gate (code) stays true — the required test legs re-run on
# the merge commit — but serverdb/docker are FALSE: the container legs (sqlserver-store,
# postgres-store, docker-smoke) run nightly + on dispatch + via the PR path-gates, so a
# push must no longer fire them. BOTH halves of that guard matter: their `if:` has no push
# arm AND these outputs are false on push — restoring EITHER half re-adds the per-merge
# cost. A merge touching those surfaces was already path-gated on its own PR minutes ago.
if [ "$EVENT_NAME" = "push" ]; then
echo "serverdb=false" >> "$GITHUB_OUTPUT"
echo "docker=false" >> "$GITHUB_OUTPUT"
echo "code=true" >> "$GITHUB_OUTPUT"
echo "ide=true" >> "$GITHUB_OUTPUT"
# TRUE on push, unlike serverdb/docker above, and the asymmetry is deliberate. Those two are
# re-runs of container legs a PR already path-gated minutes earlier. This one is the SAFETY
# NET for the whole gating scheme: the PR arm below only fires when the harness itself
# changed, so an ENGINE change that breaks a harness test is invisible on its PR by
# construction. Push-to-main is where that gets caught. Set this false and the tier is
# effectively unguarded against everything except its own edits.
echo "tooling=true" >> "$GITHUB_OUTPUT"
exit 0
fi
if [ "$EVENT_NAME" = "schedule" ]; then
# Nightly run: the heavy legs fire via their own `schedule` if-arm; these outputs stay
# false so the functional matrix and the PR path-gate arms don't ALSO fire.
echo "serverdb=false" >> "$GITHUB_OUTPUT"
echo "docker=false" >> "$GITHUB_OUTPUT"
echo "code=false" >> "$GITHUB_OUTPUT"
echo "ide=false" >> "$GITHUB_OUTPUT"
echo "tooling=false" >> "$GITHUB_OUTPUT"
exit 0
fi
# Remaining case: pull_request. Compute the path-based filters from the PR diff.
changed=$(git diff --name-only "$BASE_SHA...HEAD")
# NB: the tests/test_(...) alternation MUST list every file the sqlserver-store /
# postgres-store pytest steps below run (the ADR 0057/0058/0060/0066 throughput levers AND
# the ADR 0013/0016 RTE capture/re-ingress case); those suites are MEFOR_TEST_SQLSERVER-gated
# and run ONLY on the server-DB legs, so a file not matched here edits with NO real SS/PG
# coverage (it merely skips the gated leg). Keep this in sync with those steps
# (docs/archive/throughput/throughput-build-plan.md).
#
# AND THE INVARIANT BINDS THE SOURCES THOSE SUITES ASSERT AGAINST, NOT ONLY THE TEST FILES
# THEY RUN (BACKLOG #1322). Binding the asserting file but not the asserted-on file cannot
# see this class, and it has already fired: #514 (BACKLOG #1187) added a `masked` key to the
# summary_access audit detail built by messagefoundry/api/app.py::_emit. It touched api/ only,
# so serverdb evaluated FALSE, the DB legs never ran on its PR, and push never runs them. It
# merged green leaving tests/test_postgres_store.py and tests/test_sqlserver_store.py
# asserting the old two-key shape, and the NEXT PR to select those legs inherited three reds
# it did not cause.
#
# MEASURED, not reasoned about: of the 36 messagefoundry modules the gated suites import,
# EIGHTEEN were unmatched here -- not just api/app, but __main__, parsing/, six pipeline/
# modules and config/models. The source arms below now admit all 36, and
# tests/test_serverdb_ci_coverage.py::test_serverdb_path_gate_admits_every_source_those_suites_import
# DERIVES that set from the suites' own imports and fails if one stops being admitted -- so
# this is enforced rather than merely written down here. Import-reachability is a PROXY for
# "asserts against": it is what a machine can check, and it is deliberately wider than the
# assertion set rather than narrower.
#
# This does NOT widen the push/schedule arms. Skipping these legs on push is deliberate
# (see the no-per-merge-run guard below); the PR arm is the only one that can catch a
# change BEFORE it lands, which is why the hole mattered.
# The transports/ + config/wiring arms are the CAPTURE SURFACE: capture_response / reingress_to
# are declared there (ADR 0013), and SQL Server / Postgres both declare supports_response_capture
# + supports_pt_reingress True — so a regression in that surface is a SERVER-DB regression and must
# pull these legs, not just the SQLite suite.
if echo "$changed" | grep -qE '^(messagefoundry/store/|messagefoundry/__main__|messagefoundry/api/app|messagefoundry/parsing/(__init__|binary|message|peek|x12/)|messagefoundry/pipeline/(__init__|alerts|cluster|config_convergence|dr|leader_tasks|sharding|stage_dispatcher|wiring_runner)|messagefoundry/config/(models|response|settings|wiring)|messagefoundry/transports/(__init__|base|database|dicomweb|fhir|http_auth|mllp|rest|soap|tcp|x12)|tests/test_(sqlserver|postgres|cluster|database_connector|database_source|pooled|stage_dispatcher|batch_claim|claim_fifo|inline_fast_path|seq_only_fifo|fifo_index|per_lane_wake|response_capture|reingress|x12_rte|shard_recovery|shard_cert|adr0071|adr0075|adr0114|adr0157|dr_server_seed_gate|dr7_server_config_only_backup|backup_runner_server_db|connscale|load_failover|load_runner)|\.github/workflows/ci\.yml)'; then
echo "serverdb=true" >> "$GITHUB_OUTPUT"
else
echo "serverdb=false" >> "$GITHUB_OUTPUT"
fi
# The container image smoke runs on PRs that touch the image, the per-profile locks, packaging,
# or this workflow (engine-wide regressions are already covered by the `test` job + push-to-main).
if echo "$changed" | grep -qE '^(docker/|\.dockerignore|pyproject\.toml|requirements\.lock|\.github/workflows/ci\.yml)'; then
echo "docker=true" >> "$GITHUB_OUTPUT"
else
echo "docker=false" >> "$GITHUB_OUTPUT"
fi
# The ide job (VS Code extension build+test) only depends on ide/** (its own npm project)
# and this workflow; a pure-Python PR can't change its result, so don't run it there.
if echo "$changed" | grep -qE '^(ide/|\.github/workflows/ci\.yml)'; then
echo "ide=true" >> "$GITHUB_OUTPUT"
else
echo "ide=false" >> "$GITHUB_OUTPUT"
fi
# `tooling`: the harness tier's SUBJECT is scripts/, the workflows, and the ledger, so those are
# what can change its result on a PR. Plus the two files that define the partition itself: edit
# the manifest or the conftest hook that reads it and the tier must re-run to prove the new
# split still holds.
#
# THE SECOND ARM MATCHES THE MANIFEST AGAINST ITSELF, and that is the point. Editing one of the
# 94 listed tests has to run them, and the obvious spelling -- an alternation of test-name
# prefixes -- is a SECOND definition of the tier that drifts from the first silently, in the
# direction where an edited test simply stops being covered. `grep -xF -f` reads the one
# definition there is. Comments and blanks are stripped first: a blank pattern under -F -x
# would match nothing here, but an unfiltered `#` line is still a pattern nobody intended.
#
# NOT gated on `code`: a docs-only PR that rewrites docs/BACKLOG.md must still face the ledger
# and backlog-status tests, and those live in this tier.
# `.gitignore` IS IN THIS LIST AND IT IS NOT COSMETIC -- it closes a reintroduction of BACKLOG
# #327. tests/test_private_paths_stay_ignored.py is a manifest entry, and six .gitignore rules
# are the sole control keeping maintainer-internal material out of a public commit. #327's fix
# was to force-classify .gitignore as CODE so the test legs run it. This split broke that at a
# layer above: code=true still runs the legs, but they now DESELECT the guard, and .gitignore
# matched no arm here -- so the tooling job skipped, ci-gate reads a skipped need as pass, and
# a .gitignore-only PR faced the guard on no leg at all. Same defect as #327, new route.
# `.gitattributes` rides along for the same reason its sibling does in `alwayscodepath`.
# docs/** in full, plus CLAUDE.md, pyproject.toml, ide/ and LICENSE: each is the declared
# SUBJECT of a listed test (link_resolution and docs_runbooks read all of docs/;
# claude_section_citations reads the root CLAUDE.md, which `\.claude/` does NOT match --
# that arm is the DIRECTORY; new_dependency_check reads pyproject; ide_licence_packaging reads
# ide/ and LICENSE). Docs-only PRs were never the gap -- code=false already skipped the pytest
# step. The gap is MIXED code+docs PRs, which are the common shape.
# `.pre-commit-config.yaml` and `constraints.lock` close that same shape one layer further
# out, added 2026-08-18. tests/test_lint_scope_parity.py is a manifest entry, and its subject
# is a three-way agreement: the ruff `rev:` the hook pins, the `ruff==` constraints.lock
# installs, and pyproject's cap. Measured against this regex as it stood that day, NEITHER file
# matched EITHER arm. So on a `pre-commit autoupdate` PR -- which by construction rewrites revs
# in .pre-commit-config.yaml and touches nothing else -- the engine legs deselected that guard
# with `-m 'not tooling'`, this job skipped, ci-gate read the skipped need as a pass, and the PR
# merged green with the guard having run nowhere. The push arm above is no backstop for it:
# push fires on `branches: [main]`, i.e. AFTER the merge the gate existed to hold.
#
# `constraints.lock` IS THE EXPENSIVE ENTRY, AND THE COST IS ACCEPTED -- recorded here so it is
# not rediscovered later as a surprise. The lock moves on essentially every dependency PR, so
# this job now fires on essentially every dependency PR. The hole if it is dropped is concrete
# rather than theoretical: `uv lock --upgrade-package ruff` edits the lockfiles and does NOT
# edit pyproject.toml, so a lock-only ruff bump past the cap changes exactly what that test
# asserts while matching no other arm here. This lock is listed rather than its siblings
# because it is the file the test READS -- uv.lock and requirements.lock move with it in one
# re-export, but a uv.lock edit that was never exported has not changed the guard's subject.
if echo "$changed" | grep -qE '^(scripts/|\.github/|docs/|\.claude/|CLAUDE\.md|LICENSE|ide/|pyproject\.toml|\.gitignore|\.gitattributes|\.pre-commit-config\.yaml|constraints\.lock|tests/(conftest\.py|tooling_manifest\.txt|test_tooling_partition\.py))' \
|| echo "$changed" | grep -qxFf <(grep -vE '^[[:space:]]*(#|$)' tests/tooling_manifest.txt); then
echo "tooling=true" >> "$GITHUB_OUTPUT"
else
echo "tooling=false" >> "$GITHUB_OUTPUT"
fi
# `code`: CONSERVATIVE docs-only detector. Default to running everything (code=true), and only
# short-circuit when EVERY changed path matches the docs/non-code allowlist below. We compute it
# by inverting: list the files that are NOT docs-only; if that list is empty (and the diff is
# non-empty), it's a docs-only PR. Anything outside the allowlist — any *.py, ide/**, config,
# lockfiles, OTHER workflows, scripts, samples, harness — counts as CODE and runs the full suite.
# A path appears in `changed` even if only deleted/renamed, so a pure doc rename still short-
# circuits, and a code deletion still runs.
# Allowlisted (docs-only) paths: any *.md anywhere, docs/**, top-level LICENSE/NOTICE/AUTHORS,
# .editorconfig, .gitattributes, .github/{ISSUE_TEMPLATE,PULL_REQUEST_TEMPLATE,...}.md.
# NOT .gitignore -- #327 removed it from the regex and this line went on listing it for
# weeks, the THIRD instance in this file of a comment stating the opposite of its code.
# `.gitattributes` IS still listed here and is nonetheless CODE: `alwayscodepath` below is
# checked first and wins. Precedence, not deletion -- see the reasoning there.
# Be conservative: when in doubt a path is CODE. Empty diff (shouldn't happen on a PR) => code=true.
# `.gitignore` is NOT in this allowlist, deliberately (BACKLOG #327). Six of its rules are the
# sole control keeping maintainer-internal material out of a public commit, and
# tests/test_private_paths_stay_ignored.py is what asserts they still match. That test runs
# under pytest, which is gated on code == 'true' -- so allowlisting `.gitignore` as docs-only
# meant a `.gitignore`-only PR set code=false and the one guard that would catch the rule
# being deleted DID NOT RUN, on exactly the PR shape it exists to catch. Same defect the two
# ungated backlog guards above were added for. Treating it as code costs one suite run on a
# rare PR; the alternative costs the publishing boundary, silently.
noncode='(\.md$|^docs/|^LICENSE$|^NOTICE$|^AUTHORS$|^\.editorconfig$|^\.gitattributes$|^\.github/(ISSUE_TEMPLATE/|PULL_REQUEST_TEMPLATE))'
# EXTENSIONLESS CONFIG THAT IS ALWAYS CODE (BACKLOG #1200). The extension rule below cannot
# reach a file with no extension, and `.gitattributes` is exactly that: it was still in the
# docs-only allowlist above, so a `.gitattributes`-only PR skipped lint, mypy and the whole
# suite. It is not cosmetic -- the vault's own asvs-scorecard.yml records that a change here
# "silently alters how the corpus is materialized, which is exactly what makes the digest
# differ", so it is an input to the ASVS corpus pin.
#
# Stated as a POSITIVE list and CHECKED FIRST, rather than by deleting the entry from
# `noncode`, for two reasons. "Is code" should not depend on the ABSENCE of a line somewhere
# else -- a future edit re-adding a path to `noncode` would silently undo this with nothing
# to say so. And keeping the historical `noncode` intact is what lets
# tests/test_ci_docs_only_detector.py assert the regression in BOTH directions: it can still
# reconstruct the old classification by disabling this rule alone. `.gitignore` is named for
# the first reason even though #327 already removed it -- it was code only by falling through.
alwayscodepath='^(\.gitattributes|\.gitignore)$'
# EXTENSION OVERRIDE, EVALUATED FIRST (BACKLOG #1200). An executable file is CODE wherever it
# lives, including under docs/.
#
# The paragraph above states the intent exactly -- "any *.py ... counts as CODE" -- and the
# regex did not implement it, because `^docs/` is an alternation branch that matches a .py
# under docs/ and short-circuits before the *.py rule is ever reached. An auditor reads the
# comment, agrees with it, and moves on. Measured on 2026-08-09:
# docs/security/asvs-apply-cells.py classified NON-CODE -- the tool that WRITES the ASVS
# record of record, able to silently un-close an owner-closed cell, exempt from lint, mypy
# and the entire pytest suite by virtue of its directory. Two mypy errors had been sitting in
# it since it was written; they could not have survived a single check.
#
# THE PRECEDENT IS FOUR LINES ABOVE THIS ONE. BACKLOG #327 fixed exactly this shape for
# `.gitignore` and wrote the lesson down -- and the identical defect for docs/**/*.py sat in
# the regex immediately below the paragraph explaining it. The instance was fixed and the
# class left open, with the reasoning that would have closed it preserved in place. Hence an
# EXTENSION rule rather than another one-path exception: the next executable file someone
# puts under docs/ must not need this discovered a third time.
#
# The docs-only optimisation is deliberately preserved for actual documents -- deleting
# `^docs/` outright would run the full suite on every prose edit, which is the cost this
# short-circuit exists to avoid. Order matters: this is checked BEFORE `noncode`.
alwayscode='\.(py|ps1|sh|ts|js|yml|yaml|toml|lock|cfg|ini)$'
if [ -z "$changed" ]; then
echo "code=true" >> "$GITHUB_OUTPUT"
elif echo "$changed" | grep -qE "$alwayscode"; then
# An executable/config file changed, wherever it lives -> run the full suite.
echo "code=true" >> "$GITHUB_OUTPUT"
elif echo "$changed" | grep -qE "$alwayscodepath"; then
# Extensionless config the extension rule cannot see -> run the full suite.
echo "code=true" >> "$GITHUB_OUTPUT"
elif echo "$changed" | grep -qvE "$noncode"; then
# At least one changed path is NOT docs-only -> run the full suite.
echo "code=true" >> "$GITHUB_OUTPUT"
else
# Every changed path is docs-only -> short-circuit the expensive steps.
echo "code=false" >> "$GITHUB_OUTPUT"
fi
# SQL Server service-container leg: runs the gated suites that need a real SQL Server (Linux, so 1x
# minutes) — the PRODUCTION store backend suite AND the PRODUCTION DATABASE connector round-trip
# (ADR 0003/0010/0013). Runs NIGHTLY + on-demand + on PRs that touch the server-DB surface (gated
# by `changes` above, so docs/unrelated PRs don't pull the rate-limited mcr image). It no longer
# runs per push-to-main: a merge touching this surface was path-gated on its own PR minutes
# earlier, so the per-merge re-run re-validated unchanged content (~16x/day); a regression that
# somehow slips past a green PR (e.g. a semantic conflict between two concurrently-green PRs) is
# caught by the nightly, fix-forward. Exercises the T-SQL + the live aioodbc round-trip the
# SQLite / faked-driver tests can't. NOT a required check — ci-gate rolls it up (skip = pass).
sqlserver-store:
name: sql server (store + connector) ${{ matrix.label }}
needs: changes
# Nightly / on-demand, or a PR that touches the server-DB surface. `changes` sets serverdb=false
# on push (both halves of the no-per-merge-run guard live there — see its comment) and on
# schedule, where the first arm below already covers the run.
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || needs.changes.outputs.serverdb == 'true'
runs-on: ubuntu-latest
# Matrix the gated suite across every supported SQL Server major (2022 = 16.x, 2025 = 17.x) so a
# version-specific T-SQL / engine regression can't merge. fail-fast: false keeps a 2025-only hiccup
# from cancelling the 2022 leg (and vice-versa). The `changes` path-filter still gates this on PRs, so
# matrixing only doubles the (rate-limited) mcr pulls on server-DB PRs + the nightly run.
strategy:
fail-fast: false
matrix:
include:
- { image: "mcr.microsoft.com/mssql/server:2022-latest", label: "2022" }
- { image: "mcr.microsoft.com/mssql/server:2025-latest", label: "2025" }
services:
mssql:
image: ${{ matrix.image }}
env:
ACCEPT_EULA: "Y"
MSSQL_SA_PASSWORD: "Str0ng_P@ssw0rd!"
ports:
- 1433:1433
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Install Microsoft ODBC Driver 18 + sqlcmd
# THIS IS THE DANGEROUS HALF OF THE apt CLASS, and it is the opposite of the one that was
# visibly hurting. The Qt sites sit in jobs carrying a job-level `timeout-minutes`, so a
# mirror hang there dies at the job cap (20-40 min, measured). THIS JOB HAS NO JOB-LEVEL
# TIMEOUT, so the same hang runs to GitHub's 360-minute default -- a whole runner-hour
# budget burned on a stalled mirror, on a leg nobody is watching because it is path-gated.
#
# The step cap bounds EVERY command here, including the two curls, which are equally
# unguarded network calls. The retry wraps only the apt pair: re-running that is idempotent,
# whereas re-fetching the signing key and repo list is not the part that hangs.
#
# THE DEEPER FIX IS A JOB-LEVEL `timeout-minutes` ON THIS JOB, which would bound every step
# rather than the one that bit. Deliberately NOT done here: choosing that number needs a
# measurement of how long this job legitimately runs, and a guessed cap on a server-DB leg
# would kill real work. Recorded as a direction, not applied as a guess.
timeout-minutes: 10
run: |
curl -fsSL --max-time 60 https://packages.microsoft.com/keys/microsoft.asc \
| sudo tee /etc/apt/trusted.gpg.d/microsoft.asc > /dev/null
curl -fsSL --max-time 60 "https://packages.microsoft.com/config/ubuntu/$(. /etc/os-release; echo "$VERSION_ID")/prod.list" \
| sudo tee /etc/apt/sources.list.d/mssql-release.list > /dev/null
for attempt in 1 2 3; do
if sudo timeout 120 apt-get update && sudo ACCEPT_EULA=Y timeout 240 apt-get install -y msodbcsql18 mssql-tools18 unixodbc-dev; then
exit 0
fi
echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying"
sleep $((attempt * 5))
done
echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test."
exit 1
- name: Wait for SQL Server and create the database
run: |
sqlcmd=/opt/mssql-tools18/bin/sqlcmd
for i in $(seq 1 60); do # ~120s cap: 2025's first boot can run slower than 2022's
if "$sqlcmd" -S localhost -U sa -P 'Str0ng_P@ssw0rd!' -C -Q "SELECT 1" > /dev/null 2>&1; then
echo "SQL Server is up"; break
fi
echo "waiting for SQL Server ($i)"; sleep 2
done
"$sqlcmd" -S localhost -U sa -P 'Str0ng_P@ssw0rd!' -C \
-Q "IF DB_ID('MessageFoundry') IS NULL CREATE DATABASE MessageFoundry"
- name: Set up uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
cache-dependency-glob: |
pyproject.toml
requirements.lock
- name: Install project (dev + sqlserver extras)
run: uv pip install --system --constraint constraints.lock -e ".[dev,sqlserver]"
- name: Run the SQL Server store suite (against the real DB)
env:
MEFOR_TEST_SQLSERVER: "1"
MEFOR_STORE_BACKEND: sqlserver
MEFOR_STORE_SERVER: localhost
MEFOR_STORE_PORT: "1433"
MEFOR_STORE_DATABASE: MessageFoundry
MEFOR_STORE_AUTH: sql
MEFOR_STORE_USERNAME: sa
MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!"
MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true"
# The service container uses a self-signed cert, so trust_server_certificate=true is
# required to connect. The store's TLS-hardening guard refuses that MITM-able combination
# unless this escape is set — exactly the trusted-network dev/test case it exists for.
MEFOR_ALLOW_INSECURE_TLS: "1"
# Dump the native (C-level) frame if a retry still crashes (faulthandler only DUMPS, never kills).
PYTHONFAULTHANDLER: "1"
# pyodbc 5.3.0 + py3.14 native-crash retry (#973 follow-up, upstream pyodbc#1459): re-runs ONLY on a 139/134 crash exit, never on exit 1, so it can't mask a regression. See scripts/ci/retry-native-crash.sh.
run: >-
bash scripts/ci/retry-native-crash.sh
pytest -v tests/test_sqlserver_store.py
- name: Run the SQL Server coordinator suite (active-passive HA leader election)
env:
MEFOR_TEST_SQLSERVER: "1"
MEFOR_STORE_BACKEND: sqlserver
MEFOR_STORE_SERVER: localhost
MEFOR_STORE_PORT: "1433"
MEFOR_STORE_DATABASE: MessageFoundry
MEFOR_STORE_AUTH: sql
MEFOR_STORE_USERNAME: sa
MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!"
MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true"
# Same trusted-network TLS escape as the store suite (self-signed container cert): ODBC Driver
# 18 defaults to Encrypt=yes + verify, so trust_server_certificate=true is required to connect,
# and the store's TLS-hardening guard needs MEFOR_ALLOW_INSECURE_TLS to permit it. Exercises the
# SqlServerCoordinator leader-lease (MERGE WITH(HOLDLOCK) acquire/renew/take-over, self-fence,
# membership) against the real T-SQL the faked-driver tests can't.
MEFOR_ALLOW_INSECURE_TLS: "1"
# Dump the native (C-level) frame if a retry still crashes (faulthandler only DUMPS, never kills).
PYTHONFAULTHANDLER: "1"
# pyodbc 5.3.0 + py3.14 native-crash retry (#973 follow-up, upstream pyodbc#1459): re-runs ONLY on a 139/134 crash exit, never on exit 1, so it can't mask a regression. See scripts/ci/retry-native-crash.sh.
run: >-
bash scripts/ci/retry-native-crash.sh
pytest -v tests/test_sqlserver_coordinator.py
- name: Run the SQL Server failover suite (real TTL takeover + 2-node lifecycle)
env:
MEFOR_TEST_SQLSERVER: "1"
MEFOR_STORE_BACKEND: sqlserver
MEFOR_STORE_SERVER: localhost
MEFOR_STORE_PORT: "1433"
MEFOR_STORE_DATABASE: MessageFoundry
MEFOR_STORE_AUTH: sql
MEFOR_STORE_USERNAME: sa
MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!"
MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true"
# Same trusted-network TLS escape (self-signed container cert). Drives two coordinators sharing
# the store through real DB-clock lease expiry + the full start()/stop() election + failover.
MEFOR_ALLOW_INSECURE_TLS: "1"
# Dump the native (C-level) frame if a retry still crashes (faulthandler only DUMPS, never kills).
PYTHONFAULTHANDLER: "1"
# pyodbc 5.3.0 + py3.14 native-crash retry (#973 follow-up, upstream pyodbc#1459): re-runs ONLY on a 139/134 crash exit, never on exit 1, so it can't mask a regression. See scripts/ci/retry-native-crash.sh.
run: >-
bash scripts/ci/retry-native-crash.sh
pytest -v tests/test_cluster_failover_sqlserver.py
- name: Run the DATABASE connector round-trip (against the real DB)
env:
MEFOR_TEST_SQLSERVER: "1"
MEFOR_STORE_SERVER: localhost
MEFOR_STORE_PORT: "1433"
MEFOR_STORE_DATABASE: MessageFoundry
MEFOR_STORE_USERNAME: sa
MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!"
# The connector suite reuses the same MEFOR_STORE_* connection env + the trusted-network TLS
# escape (self-signed container cert), exactly like the store suite above. This is the live
# aioodbc/ODBC round-trip backing the connector's "production" status (PR D).
MEFOR_ALLOW_INSECURE_TLS: "1"
# Dump the native (C-level) frame if a retry still crashes (faulthandler only DUMPS, never kills).
PYTHONFAULTHANDLER: "1"
# pyodbc 5.3.0 + py3.14 native-crash retry (#973 follow-up, upstream pyodbc#1459): re-runs ONLY on a 139/134 crash exit, never on exit 1, so it can't mask a regression. See scripts/ci/retry-native-crash.sh.
run: >-
bash scripts/ci/retry-native-crash.sh
pytest -v tests/test_database_connector_integration.py
- name: Run the failover-LOAD test (two nodes, SIGKILL the primary mid-load, against the real DB)
env:
MEFOR_TEST_SQLSERVER: "1"
MEFOR_STORE_BACKEND: sqlserver
MEFOR_STORE_SERVER: localhost
MEFOR_STORE_PORT: "1433"
MEFOR_STORE_DATABASE: MessageFoundry
MEFOR_STORE_AUTH: sql
MEFOR_STORE_USERNAME: sa
MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!"
MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true"
# Gate #3 capstone + the FIRST LIVE PROOF of the SQL Server on-promotion recovery under a real
# crash: two `serve` nodes share this container, the harness SIGKILLs the primary mid-load and
# hard-asserts the host-independent CONFORMANCE invariants (zero acknowledged loss, per-lane FIFO,
# no split-brain, bounded duplicates, a recovered + drained pipeline). Recovery *time* is reported,
# not hard-gated (host-variable — see docs/LOAD-TESTING.md). This run guards the #285 claim-FIFO
# fix. The orchestrator spawns the nodes with auth off + the cluster env; they inherit MEFOR_STORE_*.
MEFOR_ALLOW_INSECURE_TLS: "1"
# Dump the native (C-level) frame if a retry still crashes (faulthandler only DUMPS, never kills).
PYTHONFAULTHANDLER: "1"
# pyodbc 5.3.0 + py3.14 native-crash retry (#973 follow-up, upstream pyodbc#1459): re-runs ONLY on a 139/134 crash exit, never on exit 1, so it can't mask a regression. See scripts/ci/retry-native-crash.sh.
run: >-
bash scripts/ci/retry-native-crash.sh
pytest -v tests/test_load_failover_sqlserver.py
- name: Run the throughput-lever backend invariants on real SQL Server (B1 inline, B2 batch-claim, ...)
env:
MEFOR_TEST_SQLSERVER: "1"
MEFOR_STORE_BACKEND: sqlserver
MEFOR_STORE_SERVER: localhost
MEFOR_STORE_PORT: "1433"
MEFOR_STORE_DATABASE: MessageFoundry
MEFOR_STORE_AUTH: sql
MEFOR_STORE_USERNAME: sa
MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!"
MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true"
MEFOR_ALLOW_INSECURE_TLS: "1"
# If every retry below still crashes, dump the native (C-level) frame so the upstream
# pyodbc bug can be diagnosed from the log. faulthandler only DUMPS — it never kills.
PYTHONFAULTHANDLER: "1"
# The staged-pipeline throughput levers (ADR 0057 inline fast-path, ADR 0058 batch-claim, ...) carry
# reliability-core invariant tests — crash-replay, poison-bound, and the per-lane-FIFO locked-head-
# BLOCKS (#285) gate — that are MEFOR_TEST_SQLSERVER-gated but live in their OWN files, so the named
# suites above never picked them up and they ran only on SQLite. Run them here against the real
# backend. APPEND each new lever's test file per docs/archive/throughput/throughput-build-plan.md.
#
# NOT only throughput levers: this step is the catch-all for ANY MEFOR_TEST_SQLSERVER-gated file
# the named suites above do not collect. CI runs an explicit file LIST here, not the whole suite,
# so a new gated file that nobody appends never runs in CI at all — it just reports as skipped on
# the plain legs, which reads identical to passing. `tests/test_sqlserver_legacy_outbox_migration.py`
# (ASVS 14.2.7) is here for exactly that reason: it verifies a one-way schema migration that can
# only be observed against a real server.
#
# RETRY WRAPPER: pyodbc 5.3.0 intermittently segfaults in its C parameter-binding path under
# py3.14 against the SQL Server 2025 container (upstream mkleehammer/pyodbc#1459, unfixed; 5.3.0
# is the newest pyodbc and the first with py3.14 wheels, so there is nothing to bump to). The
# crash kills the interpreter, so pytest-rerunfailures can't recover it — retry the WHOLE step at
# the process level, but ONLY on a native-crash exit (139 SIGSEGV / 134 SIGABRT); a real failure
# (exit 1) is NOT retried, so this never masks a regression. Remove the wrapper when #1459 ships a
# fix and the pyodbc floor moves to it. See scripts/ci/retry-native-crash.sh.
run: >-
bash scripts/ci/retry-native-crash.sh
pytest -v
tests/test_inline_fast_path.py
tests/test_batch_claim_fifo.py
tests/test_batch_claim_worker.py
tests/test_batch_claim_locking.py
tests/test_claim_fifo_heads.py
tests/test_seq_only_fifo.py
tests/test_fifo_index_migration.py
tests/test_per_lane_wake.py
tests/test_stage_dispatcher.py
tests/test_pooled_rider.py
tests/test_sqlserver_legacy_outbox_migration.py
- name: Run the RTE capture / re-ingress loop on real SQL Server (ADR 0013/0016)
env:
MEFOR_TEST_SQLSERVER: "1"
MEFOR_STORE_BACKEND: sqlserver
MEFOR_STORE_SERVER: localhost
MEFOR_STORE_PORT: "1433"
MEFOR_STORE_DATABASE: MessageFoundry
MEFOR_STORE_AUTH: sql
MEFOR_STORE_USERNAME: sa
MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!"
MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true"
MEFOR_ALLOW_INSECURE_TLS: "1"
PYTHONFAULTHANDLER: "1"
# SQL Server has declared supports_response_capture + supports_pt_reingress True since ADR 0013
# landed, and the store suite above proves the T-SQL — but the CONNECTOR->RUNNER half of the RTE
# loop (a capturing outbound -> Stage.RESPONSE -> the response worker re-ingressing into a Loopback
# inbound) had only ever run on SQLite, so a server-DB regression in it could not be caught. The
# backend-parametrized case in this file closes that gap on the real backend.
# Same pyodbc 5.3.0 + py3.14 native-crash retry as the steps above (upstream pyodbc#1459).
run: >-
bash scripts/ci/retry-native-crash.sh
pytest -v tests/test_x12_rte.py
- name: Run the engine-shard + statement-dispatch suites on real SQL Server
env:
MEFOR_TEST_SQLSERVER: "1"
MEFOR_STORE_BACKEND: sqlserver
MEFOR_STORE_SERVER: localhost
MEFOR_STORE_PORT: "1433"
MEFOR_STORE_DATABASE: MessageFoundry
MEFOR_STORE_AUTH: sql
MEFOR_STORE_USERNAME: sa
MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!"
MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true"
MEFOR_ALLOW_INSECURE_TLS: "1"
PYTHONFAULTHANDLER: "1"
# Every file here is MEFOR_TEST_SQLSERVER-gated at MODULE level, so before this step they were
# collected nowhere and reported nothing — not a pass, not a skip, on any trigger. That covered
# engine-shard crash recovery and the shard TLS-cert ladder (ADR 0037 over the ADR 0063 unified
# store), the ADR 0071 statement dispatch/fusion wiring, the ADR 0075 batch backend, the ADR 0114
# live claim procedure, and the synchronous handoff path. tests/test_serverdb_ci_coverage.py now
# fails if a module-gated suite is added without being named here.
# Same pyodbc 5.3.0 + py3.14 native-crash retry as the steps above (upstream pyodbc#1459).
run: >-
bash scripts/ci/retry-native-crash.sh
pytest -v
tests/test_shard_recovery_sqlserver.py
tests/test_shard_cert_sqlserver.py
tests/test_adr0071_dispatch_wiring_sqlserver.py
tests/test_adr0071_fused_callables_sqlserver.py
tests/test_adr0075_batch_sqlserver.py
tests/test_adr0114_claim_proc_live.py
tests/test_sqlserver_sync_handoff.py
tests/test_database_source_integration.py
- name: Run the DR seed-gate + backup suites on real SQL Server
env:
MEFOR_TEST_SQLSERVER: "1"
MEFOR_STORE_BACKEND: sqlserver
MEFOR_STORE_SERVER: localhost
MEFOR_STORE_PORT: "1433"
MEFOR_STORE_DATABASE: MessageFoundry
MEFOR_STORE_AUTH: sql
MEFOR_STORE_USERNAME: sa
MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!"
MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true"
MEFOR_ALLOW_INSECURE_TLS: "1"
PYTHONFAULTHANDLER: "1"
# ADR 0048/0049 disaster recovery, SQL Server side: the seed gate that must REFUSE to activate a
# standby onto a non-empty store, the .mfbak backup runner against a real server DB, and the DR7
# config-only backup + DbaDelegatedError path. Module-gated and never executed; a DR control that
# has never run is a DR control that has never been proven.
# Same pyodbc 5.3.0 + py3.14 native-crash retry as the steps above (upstream pyodbc#1459).
run: >-
bash scripts/ci/retry-native-crash.sh
pytest -v
tests/test_dr_server_seed_gate_sqlserver.py
tests/test_backup_runner_server_db_sqlserver.py
tests/test_dr7_server_config_only_backup_sqlserver.py
# Postgres store backend (Track B): run the gated store suite against a real PostgreSQL service
# container (Linux, so 1x minutes). Runs NIGHTLY + on-demand (workflow_dispatch — use
# `gh workflow run ci.yml --ref <branch>` to exercise it on a feature branch) + on PRs that touch
# the server-DB surface (NEW: the same `changes` path-gate as sqlserver-store — a store/cluster PR
# now gets real-Postgres coverage PRE-merge, which it never had). No longer per push-to-main; same
# rationale as sqlserver-store above. This is what actually drives the SELECT ... FOR UPDATE SKIP
# LOCKED + advisory-lock concurrency the SQLite tests can't reach.
postgres-store:
name: postgres store
needs: changes
# Nightly / on-demand / server-DB PRs (serverdb=false on push and schedule — see `changes`).
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || needs.changes.outputs.serverdb == 'true'
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: mefor
POSTGRES_DB: messagefoundry
ports:
- 5432:5432
# Gate steps on readiness so the suite never races the container's first-boot init.
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
cache-dependency-glob: |
pyproject.toml
requirements.lock
- name: Install project (dev + postgres extras)
run: uv pip install --system --constraint constraints.lock -e ".[dev,postgres]"
- name: Run the Postgres store suite (against the real DB)
env:
MEFOR_TEST_POSTGRES: "1"
MEFOR_STORE_BACKEND: postgres
MEFOR_STORE_SERVER: localhost
MEFOR_STORE_PORT: "5432"
MEFOR_STORE_DATABASE: messagefoundry
MEFOR_STORE_USERNAME: postgres
MEFOR_STORE_PASSWORD: mefor
# The service container speaks plaintext (no TLS), so encrypt=false; the store's
# TLS-hardening guard refuses that MITM-able combination unless this trusted-network
# dev/test escape is set.
MEFOR_STORE_ENCRYPT: "false"
MEFOR_ALLOW_INSECURE_TLS: "1"
run: pytest tests/test_postgres_store.py tests/test_adr0157_postgres_fence.py -v
- name: Run the failover-LOAD test (two nodes, SIGKILL the primary mid-load, against the real DB)
env:
MEFOR_TEST_POSTGRES: "1"
MEFOR_STORE_BACKEND: postgres
MEFOR_STORE_SERVER: localhost
MEFOR_STORE_PORT: "5432"
MEFOR_STORE_DATABASE: messagefoundry
MEFOR_STORE_USERNAME: postgres
MEFOR_STORE_PASSWORD: mefor
# Gate #3 capstone: two `serve` nodes share this container; the harness SIGKILLs the primary
# mid-load and hard-asserts the host-independent CONFORMANCE invariants (zero acknowledged loss,
# per-lane FIFO, no split-brain, bounded duplicates, a recovered + drained pipeline). Recovery
# time is reported, not hard-gated. The orchestrator spawns the nodes with auth off + the cluster
# env itself; they inherit MEFOR_STORE_* (incl. the plaintext-TLS escape) from here.
MEFOR_STORE_ENCRYPT: "false"
MEFOR_ALLOW_INSECURE_TLS: "1"
run: pytest tests/test_load_failover_postgres.py -v
- name: Run the throughput-lever backend invariants on real Postgres (B1 inline, B2 batch-claim, ...)
env:
MEFOR_TEST_POSTGRES: "1"
MEFOR_STORE_BACKEND: postgres
MEFOR_STORE_SERVER: localhost
MEFOR_STORE_PORT: "5432"
MEFOR_STORE_DATABASE: messagefoundry
MEFOR_STORE_USERNAME: postgres
MEFOR_STORE_PASSWORD: mefor
MEFOR_STORE_ENCRYPT: "false"
MEFOR_ALLOW_INSECURE_TLS: "1"
# The throughput levers' MEFOR_TEST_POSTGRES-gated invariants (esp. the FOR-UPDATE no-SKIP-LOCKED
# locked-head-BLOCKS twin of #285, and the batch FIFO/crash tests) live in their own files. APPEND
# each new lever's test file per docs/archive/throughput/throughput-build-plan.md.
run: >-
pytest -v
tests/test_inline_fast_path.py
tests/test_batch_claim_fifo.py
tests/test_batch_claim_worker.py
tests/test_batch_claim_locking.py
tests/test_claim_fifo_heads.py
tests/test_seq_only_fifo.py
tests/test_fifo_index_migration.py
tests/test_per_lane_wake.py
tests/test_stage_dispatcher.py
tests/test_pooled_rider.py
- name: Run the connection-scale pool-wait smoke on real Postgres (B11 wall #2, tiny pool)
env:
MEFOR_TEST_POSTGRES: "1"
MEFOR_STORE_BACKEND: postgres
MEFOR_STORE_SERVER: localhost
MEFOR_STORE_PORT: "5432"
MEFOR_STORE_DATABASE: messagefoundry
MEFOR_STORE_USERNAME: postgres
MEFOR_STORE_PASSWORD: mefor
MEFOR_STORE_ENCRYPT: "false"
MEFOR_ALLOW_INSECURE_TLS: "1"
# FORCE a below-default pool (4 connections) so the perf_counter-measured acquire-WAIT
# histogram (the PRIMARY pool-wait signal) is non-trivial even at small N: ~3N inbound workers
# contend for 4 connections, so acquires actually queue and the percentiles populate. The
# default pool (40) would mask the wall. 4 (not 1-2) is deliberate: GET /status runs
# db_status()'s four sequential COUNT(*) queries on every poll, and an extremely tiny pool
# starves those acquires past the poller's HTTP timeout under the empty-claim herd on a slow
# runner — every poll then fails and the drain loops burn the test budget (the PR #675 hang).
# This still gives the B11 pool-wait instrumentation real regression coverage SQLite (no pool)
# cannot — the connscale harness OWNS the engine subprocess and inherits this env.
MEFOR_STORE_POOL_SIZE: "4"
run: pytest tests/test_connscale_postgres.py -v
- name: Run the RTE capture / re-ingress loop on real Postgres (ADR 0013/0016)
env:
MEFOR_TEST_POSTGRES: "1"
MEFOR_STORE_BACKEND: postgres
MEFOR_STORE_SERVER: localhost
MEFOR_STORE_PORT: "5432"
MEFOR_STORE_DATABASE: messagefoundry
MEFOR_STORE_USERNAME: postgres
MEFOR_STORE_PASSWORD: mefor
MEFOR_STORE_ENCRYPT: "false"
MEFOR_ALLOW_INSECURE_TLS: "1"
# The Postgres twin of the sqlserver-store leg's RTE step (same backend-parametrized case, the
# `postgres` param). Postgres declares the same two capability flags True, so the same
# connector->runner gap existed here.
run: pytest -v tests/test_x12_rte.py
- name: Run the failover + engine-shard recovery suites on real Postgres
env:
MEFOR_TEST_POSTGRES: "1"
MEFOR_STORE_BACKEND: postgres
MEFOR_STORE_SERVER: localhost
MEFOR_STORE_PORT: "5432"
MEFOR_STORE_DATABASE: messagefoundry
MEFOR_STORE_USERNAME: postgres
MEFOR_STORE_PASSWORD: mefor
MEFOR_STORE_ENCRYPT: "false"
MEFOR_ALLOW_INSECURE_TLS: "1"
# These are MEFOR_TEST_POSTGRES-gated at MODULE level, so until this step existed they were
# collected nowhere and reported nothing — not a pass, not a skip. The SQL Server failover twin
# (test_cluster_failover_sqlserver.py) has run for some time; its Postgres counterpart never had.
# Engine-shard crash recovery (ADR 0037 over the ADR 0063 unified store) was dark on BOTH server
# backends. tests/test_serverdb_ci_coverage.py now fails if a module-gated suite is added without
# being named here.
run: >-
pytest -v
tests/test_cluster_failover_postgres.py
tests/test_shard_recovery_postgres.py
- name: Run the DR seed-gate + backup suites on real Postgres
env:
MEFOR_TEST_POSTGRES: "1"
MEFOR_STORE_BACKEND: postgres
MEFOR_STORE_SERVER: localhost
MEFOR_STORE_PORT: "5432"
MEFOR_STORE_DATABASE: messagefoundry
MEFOR_STORE_USERNAME: postgres
MEFOR_STORE_PASSWORD: mefor
MEFOR_STORE_ENCRYPT: "false"
MEFOR_ALLOW_INSECURE_TLS: "1"
# ADR 0048/0049 disaster recovery: the seed gate that must REFUSE to activate a standby onto a
# non-empty store, the .mfbak backup runner against a real server DB, and the DR7 config-only
# backup + DbaDelegatedError path. All three are module-gated and were never executed against
# PostgreSQL; a DR control that has never run is a DR control that has never been proven.
run: >-
pytest -v
tests/test_dr_server_seed_gate_postgres.py
tests/test_backup_runner_server_db_postgres.py
tests/test_dr7_server_config_only_backup_postgres.py
# Headless load test (Track B / throughput): serve the synthetic high-fan-out load config (auth
# off, small fan-out) and drive the smoke profile through the real `python -m harness --load` CLI,
# asserting zero message loss + all SLOs (exit 0) and uploading the JSON/CSV report. Skipped on PRs
# to save spend — the in-process load integration test (tests/test_load_runner.py, in the `test`
# job) is the PR gate. Runs on push to main + on-demand (workflow_dispatch). Heavier profiles
# (fanout-baseline / soak) and the SQLite-vs-Postgres comparison are run manually — see
# docs/LOAD-TESTING.md.
load-test:
name: load test (smoke, sqlite)
# Nightly schedule + on-demand only (was: every push to main). The PR gate is the in-process
# tests/test_load_runner.py in the `test` job; push-to-main keeps the full functional matrix
# (the server-DB suites are nightly + PR-path-gated too, as of the same cost pass). This
# multi-minute serve+load run no longer fires once per commit. NOT a required check,
# so dropping it from push is safe; ci-gate `needs` it but counts a `skipped` leg as a pass.
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
cache-dependency-glob: |
pyproject.toml
requirements.lock
- name: Install project (dev — engine + httpx client; no Qt needed for the headless load path)
run: uv pip install --system --constraint constraints.lock -e ".[dev]"
- name: Serve the load config (auth off) and run the smoke profile
env:
MEFOR_SECURITY_REQUIRE_SIGN_IN: "false" # the load CLI polls /stats etc. without a bearer token
MEFOR_LOAD_FANOUT: "8"
MEFOR_LOAD_RESULTS_FANOUT: "2"
MEFOR_LOAD_TRANSFORM: edit
MEFOR_LOAD_SINK_PORT: "2700" # matches --sink-port below
# GIVEN 1 (ADR 0148): the default env `dev` now derives PHI, so a bare `serve --env dev` runs the
# secure PHI posture (secure-by-default). This leg validates that PHI+enforce serves green — so
# provision the production posture rather than opting out to synthetic: a store encryption key
# (minted at runtime below — never a committed literal), bounded PHI-body retention windows, and a
# locked-down egress allowlisting the synthetic loopback sink the load graph delivers to. The
# security-notification gate is skipped because auth is off (no accounts to notify).
MEFOR_SECURITY_DELETE_MESSAGE_BODIES_AFTER_DAYS: "30" # bound inbound PHI bodies at rest
MEFOR_RETENTION_DEAD_LETTER_DAYS: "30" # bound dead-lettered PHI bodies at rest
MEFOR_SECURITY_BLOCK_UNLISTED_OUTBOUND: "true" # deny-by-default egress
MEFOR_EGRESS_ALLOWED_MLLP: "127.0.0.1" # the load graph fans out to the loopback correlation sink
run: |
set -o pipefail
mkdir -p out/load
# Mint a throwaway AES-256 store key at runtime (PHI-at-rest must never be plaintext); never a
# committed literal (gitleaks). 32 random bytes, base64 — the MEFOR_STORE_ENCRYPTION_KEY format.
export MEFOR_STORE_ENCRYPTION_KEY="$(python -c 'import base64, os; print(base64.b64encode(os.urandom(32)).decode())')"
python -m messagefoundry serve --config harness/config/load --env dev --db ./load-ci.db \
--host 127.0.0.1 --port 8765 > engine.log 2>&1 &
engine_pid=$!
for _ in $(seq 1 60); do
curl -sf http://127.0.0.1:8765/health > /dev/null && break || sleep 0.5
done
set +e
python -m harness --load smoke --engine http://127.0.0.1:8765 --sink-port 2700 \
--report-json out/load/smoke-ci.json --report-csv out/load/smoke-ci.csv
rc=$?
set -e
kill "$engine_pid" 2>/dev/null || true
echo "===== engine log (tail) ====="; tail -50 engine.log || true
exit $rc
- name: Upload load report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: load-report
path: out/load/
if-no-files-found: ignore
# SQL Server throughput smoke: the same zero-loss fan-out wiring check as `load test (smoke, sqlite)`,
# but driven through the **SQL Server store** end-to-end (ingress→routed→outbound→delivered, each a
# committed server-DB round-trip) on a real mssql service container. This is the load path that caught
# the routed-stage `handler_name`-drop regression the in-process tests missed. Uses the `smoke-sqlserver`
# profile (correctness SLOs strict; drain bound sized for a server DB). Skipped on PRs to save spend —
# runs on push to main + on-demand (workflow_dispatch). Heavier SQL Server profiles run manually.
load-test-sqlserver:
name: load test (smoke, sqlserver) ${{ matrix.label }}
# Nightly schedule + on-demand only (was: every push to main); same rationale as `load-test`. The
# end-to-end SQL Server store path is still smoked nightly (x2 majors) + on demand, and the SQL
# Server functional coverage stays in `sqlserver-store` (nightly + server-DB-PR path-gate). NOT
# required; ci-gate treats skip as pass.
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
# Smoke the end-to-end SQL Server store path on every supported major (2022/2025). Push/dispatch-only,
# so no PR cost; fail-fast: false keeps the two legs independent.
strategy:
fail-fast: false
matrix:
include:
- { image: "mcr.microsoft.com/mssql/server:2022-latest", label: "2022" }
- { image: "mcr.microsoft.com/mssql/server:2025-latest", label: "2025" }
services:
mssql:
image: ${{ matrix.image }}
env:
ACCEPT_EULA: "Y"
MSSQL_SA_PASSWORD: "Str0ng_P@ssw0rd!"
ports:
- 1433:1433
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Install Microsoft ODBC Driver 18 + sqlcmd
# THIS IS THE DANGEROUS HALF OF THE apt CLASS, and it is the opposite of the one that was
# visibly hurting. The Qt sites sit in jobs carrying a job-level `timeout-minutes`, so a
# mirror hang there dies at the job cap (20-40 min, measured). THIS JOB HAS NO JOB-LEVEL
# TIMEOUT, so the same hang runs to GitHub's 360-minute default -- a whole runner-hour
# budget burned on a stalled mirror, on a leg nobody is watching because it is path-gated.
#
# The step cap bounds EVERY command here, including the two curls, which are equally
# unguarded network calls. The retry wraps only the apt pair: re-running that is idempotent,
# whereas re-fetching the signing key and repo list is not the part that hangs.
#
# THE DEEPER FIX IS A JOB-LEVEL `timeout-minutes` ON THIS JOB, which would bound every step
# rather than the one that bit. Deliberately NOT done here: choosing that number needs a
# measurement of how long this job legitimately runs, and a guessed cap on a server-DB leg
# would kill real work. Recorded as a direction, not applied as a guess.
timeout-minutes: 10
run: |
curl -fsSL --max-time 60 https://packages.microsoft.com/keys/microsoft.asc \
| sudo tee /etc/apt/trusted.gpg.d/microsoft.asc > /dev/null
curl -fsSL --max-time 60 "https://packages.microsoft.com/config/ubuntu/$(. /etc/os-release; echo "$VERSION_ID")/prod.list" \
| sudo tee /etc/apt/sources.list.d/mssql-release.list > /dev/null
for attempt in 1 2 3; do
if sudo timeout 120 apt-get update && sudo ACCEPT_EULA=Y timeout 240 apt-get install -y msodbcsql18 mssql-tools18 unixodbc-dev; then
exit 0
fi
echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying"
sleep $((attempt * 5))
done
echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test."
exit 1
- name: Wait for SQL Server and create the database (RCSI on, like the store suite)
run: |
sqlcmd=/opt/mssql-tools18/bin/sqlcmd
for i in $(seq 1 60); do # ~120s cap: 2025's first boot can run slower than 2022's
if "$sqlcmd" -S localhost -U sa -P 'Str0ng_P@ssw0rd!' -C -Q "SELECT 1" > /dev/null 2>&1; then
echo "SQL Server is up"; break
fi
echo "waiting for SQL Server ($i)"; sleep 2
done
"$sqlcmd" -S localhost -U sa -P 'Str0ng_P@ssw0rd!' -C \
-Q "IF DB_ID('MessageFoundry') IS NULL CREATE DATABASE MessageFoundry; ALTER DATABASE MessageFoundry SET READ_COMMITTED_SNAPSHOT ON WITH ROLLBACK IMMEDIATE;"
- name: Set up uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
cache-dependency-glob: |
pyproject.toml
requirements.lock
- name: Install project (dev + sqlserver extras)
run: uv pip install --system --constraint constraints.lock -e ".[dev,sqlserver]"
- name: Serve on the SQL Server store and run the smoke-sqlserver profile
env:
MEFOR_SECURITY_REQUIRE_SIGN_IN: "false" # the load CLI polls /stats etc. without a bearer token
MEFOR_LOAD_FANOUT: "8"
MEFOR_LOAD_RESULTS_FANOUT: "2"
MEFOR_LOAD_TRANSFORM: edit
MEFOR_LOAD_SINK_PORT: "2700" # matches --sink-port below
MEFOR_STORE_BACKEND: sqlserver # no --db: the backend + connection come from MEFOR_STORE_*
MEFOR_STORE_SERVER: localhost
MEFOR_STORE_PORT: "1433"
MEFOR_STORE_DATABASE: MessageFoundry
MEFOR_STORE_AUTH: sql
MEFOR_STORE_USERNAME: sa
MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!"
MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true"
MEFOR_ALLOW_INSECURE_TLS: "1" # trusted-network escape for the container's self-signed cert
# DECLARE THE DATA CLASS HONESTLY (ADR 0148 GIVEN 1 + docs/SECURITY-LOOSENING.md
# "`handles_real_patient_data = false`"). This leg processes harness-GENERATED synthetic HL7 on
# a throwaway service container; asserting PHI was a false declaration about the instance, and
# the register names exactly this case as acceptable: "a CI runner ... that only ever processes
# synthetic / sample HL7". Since GIVEN 1 the built-in `dev` env derives PHI, so a genuinely
# throwaway CI box must now set this EXPLICITLY — it is no longer the `dev` default.
#
# It is also what makes the leg run at all. This job was RED for four consecutive nights
# (2026-07-27..30) on:
# ValueError: SQL Server TLS is weakened (trust_server_certificate=true or encrypt=false)
# Under enforcing-PHI the `MEFOR_ALLOW_INSECURE_TLS` escape above is clamped INERT by design
# (#200 / ADR 0092 decision 2 — `weakened_tls_escape_permitted`), so a self-signed container
# can never be trusted from a PHI instance. The previous comment here said this leg "validates
# PHI+enforce green, not a synthetic opt-out" — an intent that is UNREACHABLE with a
# `services:` container: GitHub starts it before any step runs, so a cert generated in a step
# cannot be mounted into it. Whoever set that provisioned the PHI posture's retention and
# egress requirements but not its TLS one.
#
# WHAT THIS GIVES UP, stated plainly: the leg no longer exercises the PHI+enforce path. Its
# actual job is a load/throughput smoke of the SQL Server store, and `sqlserver-store` carries
# the functional coverage. Restoring PHI+enforce here needs a REAL certificate — the job moved
# off `services:` onto a `docker run` with a generated cert mounted and trusted — which is
# worth doing deliberately, not as a side effect of unbreaking a nightly.
#
# The PHI-shaped provisions below are KEPT even though synthetic does not require them: they
# cost nothing and keep the leg measuring a realistic configuration.
MEFOR_SECURITY_HANDLES_REAL_PATIENT_DATA: "false"
# Bounded PHI-body retention windows + a locked-down egress allowlisting the loopback sink.
# The store key is minted at runtime below. The security-notification gate is skipped because
# auth is off (no accounts to notify).
MEFOR_SECURITY_DELETE_MESSAGE_BODIES_AFTER_DAYS: "30"
MEFOR_RETENTION_DEAD_LETTER_DAYS: "30"
MEFOR_SECURITY_BLOCK_UNLISTED_OUTBOUND: "true"
MEFOR_EGRESS_ALLOWED_MLLP: "127.0.0.1" # the load graph fans out to the loopback correlation sink
run: |
set -o pipefail
mkdir -p out/load
# Mint a throwaway AES-256 store key at runtime (never a committed literal — gitleaks).
export MEFOR_STORE_ENCRYPTION_KEY="$(python -c 'import base64, os; print(base64.b64encode(os.urandom(32)).decode())')"
python -m messagefoundry serve --config harness/config/load --env dev \
--host 127.0.0.1 --port 8765 > engine.log 2>&1 &
engine_pid=$!
for _ in $(seq 1 60); do
curl -sf http://127.0.0.1:8765/health > /dev/null && break || sleep 0.5
done
set +e
python -m harness --load smoke-sqlserver --engine http://127.0.0.1:8765 --sink-port 2700 \
--report-json out/load/smoke-sqlserver-ci.json --report-csv out/load/smoke-sqlserver-ci.csv
rc=$?
set -e
kill "$engine_pid" 2>/dev/null || true
echo "===== engine log (tail) ====="; tail -50 engine.log || true
exit $rc
- name: Upload load report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: load-report-sqlserver
path: out/load/
if-no-files-found: ignore
# End-to-end check of the real Windows service path on both target Server
# versions: install via NSSM -> start -> /health -> send an MLLP message ->
# confirm it was recorded -> stop -> uninstall. Nightly + on-demand only
# (2x-billed Windows legs): this validates the NSSM/service PLUMBING, which
# changes when scripts/service/ or the serve path changes — not per merge —
# and it was never a pre-merge gate (it already ran post-merge). Per-push it
# was the single biggest push-run line item (~3.6 Windows min x ~16 merges/
# day); the nightly keeps both Server SKUs covered daily and workflow_dispatch
# re-validates a specific merge on demand.
windows-service-smoke:
name: windows service smoke (${{ matrix.os }}, py${{ matrix.python-version }})
# This repo only: it spins up 2x-billed hosted Windows runners on BOTH Server SKUs, which is free
# here (public) and billed to the owner of a fork. Nightly schedule + manual dispatch only. ci-gate
# `needs` this job, but a `skipped` leg counts as a pass there, so gating it off a fork cannot wedge
# the gate.
if: (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && github.repository == 'MEFORORG/MessageFoundry'
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
# Windows Server 2022 + 2025 on the supported deploy Python (3.14), so the real NSSM
# install->serve->MLLP path is validated on the version we ship, on each Server SKU. This job
# is push/dispatch-only, so the Windows minutes cost no PR time.
include:
- { os: windows-2022, python-version: "3.14" }
- { os: windows-2025, python-version: "3.14" }
defaults:
run:
shell: pwsh
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ matrix.python-version }}
- name: Set up uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
cache-dependency-glob: |
pyproject.toml
requirements.lock
- name: Install the engine (fhir + dicom extras)
run: |
# The samples/config graph this smoke serves uses the FHIR and DICOM connectors, and the
# DICOM inbound binds a DIMSE C-STORE SCP at startup (transports/dicom.py imports pynetdicom
# eagerly on start) — so a bare `uv pip install --system -e .` makes the engine fail-close at
# wiring with ModuleNotFoundError and /health never comes up. Install the connector extras the
# graph needs (the "real Windows service path" a user running this graph would install). Keep
# in sync with the connectors used in samples/config.
uv pip install --system --constraint constraints.lock -e ".[fhir,dicom]"
# Run the install/uninstall scripts under Windows PowerShell 5.1 (shell: powershell) to
# match what the console launches on an end-user machine — catches 5.1-only behaviour
# (e.g. native-command stderr aborting under ErrorActionPreference=Stop). NSSM happens to
# be on the runner PATH, so the auto-download path is validated locally / on the user's box.
- name: Install the service
shell: powershell
run: |
$exe = (Get-Command messagefoundry).Source
# Run as the 'prod' environment (ADR 0017: `serve` requires an explicit active environment;
# install-service.ps1 -Environment threads it into the service command + resolves
# environments/prod.toml). INFO not DEBUG: `serve` refuses DEBUG in prod (Gate #1 — DEBUG can
# surface PHI); INFO is enough to smoke service start + MLLP.
# -LockConfigDir: the runner checks out samples/config with the default ACL, which grants
# BUILTIN\Users (S-1-5-32-545) write. The in-process config-source trust guard (SEC-003,
# wiring._assert_safe_config_source_windows) then REFUSES to load it ("writable-by-others
# path") and `serve` crash-loops to SERVICE_PAUSED, so /health never comes up. Locking the
# config dir (strip inheritance -> SYSTEM/Administrators + the run-as account, RX) is the
# production-correct posture docs/SERVICE.md prescribes, and exercises the real -LockConfigDir
# install path rather than the MEFOR_ALLOW_INSECURE_CONFIG_SOURCE dev escape.
# #224: NO -ServiceAccount and NO -AllowLocalSystem, so this install exercises the NEW DEFAULT -
# the least-privilege virtual account NT SERVICE\MessageFoundry (no password). This leg is the
# gate for that flip: it proves the virtual account is granted config-read (RX, via the
# -LockConfigDir path that now runs AFTER ObjectName so the per-service SID resolves) + data-dir
# read/write, gets SeServiceLogonRight, and actually starts and serves /health + MLLP. Pass
# -AllowLocalSystem here if a run-as-LocalSystem smoke is ever needed instead.
.\scripts\service\install-service.ps1 -AppExe $exe -LogLevel INFO -Environment prod -LockConfigDir
# The shipped samples graph includes a mutual-TLS WS-* SOAP feed (IB_IMMUNIZATION_VXU) whose
# connector LOADS its client cert at wiring time, so the engine refuses to start the graph
# without a readable cert+key. Mint a throwaway pair for the smoke — the smoke sends only an
# MLLP ADT, never a VXU, so the cert is loaded but never used in a handshake. Run openssl via
# `cmd /c "... 2>nul"`: it streams RSA key-gen progress to stderr, and a bare `2>$null` did NOT
# suppress it on the runner's openssl build — PowerShell 5.1 wrapped that stderr as a
# NativeCommandError and aborted the step. Redirecting at the cmd level keeps PowerShell out of
# openssl's stderr path entirely, so no NativeCommandError regardless of the openssl build.
$certDir = Join-Path $env:RUNNER_TEMP "registry-cert"
New-Item -ItemType Directory -Force -Path $certDir | Out-Null
$cert = Join-Path $certDir "client.pem"
$key = Join-Path $certDir "client.key"
cmd /c "openssl req -x509 -newkey rsa:2048 -keyout ""$key"" -out ""$cert"" -days 1 -subj /CN=mefor-smoke -passout pass:smoke-key-pass 2>nul"
if ($LASTEXITCODE -ne 0 -or -not (Test-Path $cert)) { throw "openssl failed to mint the smoke client cert" }
# `prod` is a PHI environment, so the engine FAIL-CLOSES at startup unless a store encryption
# key is set (PHI-at-rest must never be plaintext). Without one, `serve` refuses to start, NSSM
# throttles the crash-loop to SERVICE_PAUSED, and /health never comes up. Mint a throwaway AES
# key for the smoke — `gen-key` prints only the key (stdout, pipeable; no stderr to trip PS 5.1).
$storeKey = (& $exe gen-key).Trim()
if (-not $storeKey) { throw "messagefoundry gen-key produced no store encryption key" }
# This smoke validates the Windows service + MLLP plumbing, not authentication (auth is
# covered by the pytest suite: test_api_auth / test_console_auth). Run with auth disabled so
# the /messages call below needs no bearer token, and supply the registry endpoint's secrets
# (cert/key/password + WS-Security creds) so the SOAP feed wires — placeholders, never used
# since no VXU is sent; the non-secret endpoints live in environments/prod.toml (selected by
# the -Environment prod passed to install-service.ps1 above).
# `prod` also FAIL-CLOSES on unrestricted egress (a transform could send PHI anywhere): lock it
# down (deny_by_default) AND allowlist the samples/config graph's 8 outbound destinations, or the
# engine refuses to start at wiring. [egress] is SERVICE settings (MEFOR_EGRESS_*), NOT an
# environments/<env>.toml value; lists are comma-separated host:port matching the resolved
# prod.toml endpoints (MLLP->allowed_mllp; X12->allowed_tcp; SOAP+FHIR->allowed_http; File->allowed_file_dirs).
# NOTE keep this list in sync with samples/config when an outbound is added (e.g. OB_FHIR_SERVER) —
# the egress check fails fast on the FIRST denied dest, so a missing entry wedges service start.
# fhir-prod.example.org is allowlisted HOST-ONLY: its prod URL (https://fhir-prod.example.org/fhir)
# has no explicit port, so urlsplit().port is None and a ":443" suffix would NOT match (see
# _http_egress_allowed). A bare host matches any port, which is what we want for a default-port URL.
# `prod` FAIL-CLOSES on unbounded PHI RETENTION too (#186a, ASVS 14.2.4): both PHI-body windows —
# [retention].messages_days (inbound bodies) and [retention].dead_letter_days (dead-lettered
# outbound bodies, replayable until purged) — default to 0 = keep-forever, and a PRODUCTION PHI
# instance with EITHER unbounded refuses to start ("no data-retention window is configured ...
# refusing to start"), so NSSM crash-loops to SERVICE_PAUSED and /health never comes up. BOUND both
# (30d) rather than setting [retention].allow_unbounded_phi=true: a real prod deployment configures
# a window, so the smoke exercises the path it is meant to validate instead of opting out of it.
nssm set MessageFoundry AppEnvironmentExtra `
"MEFOR_SECURITY_REQUIRE_SIGN_IN=false" `
"MEFOR_STORE_ENCRYPTION_KEY=$storeKey" `
"MEFOR_SECURITY_DELETE_MESSAGE_BODIES_AFTER_DAYS=30" `
"MEFOR_RETENTION_DEAD_LETTER_DAYS=30" `
"MEFOR_SECURITY_BLOCK_UNLISTED_OUTBOUND=true" `
"MEFOR_EGRESS_ALLOWED_MLLP=receiver-prod.example.org:6661,powerscribe-prod.example.org:6665" `
"MEFOR_EGRESS_ALLOWED_TCP=payer-prod.example.org:6662,rte-prod.example.org:6663,rte-result-prod.example.org:6664" `
"MEFOR_EGRESS_ALLOWED_HTTP=iis-prod.example.org:8443,fhir-prod.example.org" `
"MEFOR_EGRESS_ALLOWED_FILE_DIRS=out" `
"MEFOR_VALUE_REGISTRY_CLIENT_CERT=$cert" `
"MEFOR_VALUE_REGISTRY_CLIENT_KEY=$key" `
"MEFOR_VALUE_REGISTRY_KEY_PASSWORD=smoke-key-pass" `
"MEFOR_VALUE_REGISTRY_USER=smoke-user" `
"MEFOR_VALUE_REGISTRY_PASSWORD=smoke-pass"
- name: Start and verify /health
run: |
nssm start MessageFoundry
$ok = $null
foreach ($i in 1..30) {
try { $ok = Invoke-RestMethod http://127.0.0.1:8765/health -TimeoutSec 2; break }
catch { Start-Sleep -Milliseconds 500 }
}
if (-not $ok) { throw "engine /health never came up" }
"health: $($ok | ConvertTo-Json -Compress)"
- name: Send an MLLP message and confirm it was recorded
run: |
python samples\send_mllp.py samples\messages\adt_a01.hl7
$msgs = $null
foreach ($i in 1..20) {
$msgs = Invoke-RestMethod http://127.0.0.1:8765/messages
if ($msgs.total -ge 1) { break }
Start-Sleep -Milliseconds 500
}
if ($msgs.total -lt 1) { throw "message was not recorded by the engine" }
"messages.total = $($msgs.total)"
- name: Stop the service (graceful)
if: always()
run: nssm stop MessageFoundry
- name: Show service logs
if: always()
run: |
$dir = "C:\ProgramData\MessageFoundry\logs"
foreach ($f in "service.out.log", "service.err.log") {
$p = Join-Path $dir $f
"===== $f ====="
if (Test-Path $p) { Get-Content $p } else { "(none)" }
}
- name: Uninstall the service
if: always()
shell: powershell
run: .\scripts\service\uninstall-service.ps1
- name: Upload service logs
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: service-logs-${{ matrix.os }}-py${{ matrix.python-version }}
path: C:\ProgramData\MessageFoundry\logs\
if-no-files-found: ignore
# Container image smoke (the engine's FIRST non-Windows runtime target — it is Windows-service/NSSM
# first today). Builds the slim engine image, builds a test-only image that bakes a minimal ADT->file
# config + a self-contained MLLP sender (config baked, not mounted, so it is owned by the engine's UID
# and not group/world-writable — _assert_safe_config_source refuses otherwise), serves it loopback +
# auth-off, sends one synthetic ADT^A01 over the in-container MLLP listener, and asserts it FINALIZES
# to PROCESSED (not merely RECEIVED — that distinguishes ACK-on-receipt from final disposition). Then
# it verifies a graceful `docker stop`: tini forwards SIGTERM -> uvicorn lifespan -> engine.stop().
# Runs NIGHTLY + on dispatch; on PRs only when the image/locks/packaging/this workflow change
# (gated by `changes`), so docs/unrelated PRs don't pay the build. No longer per push-to-main —
# an image-touching merge was path-gated on its own PR minutes earlier (same rationale as the
# server-DB legs; `changes` sets docker=false on push, the other half of this guard).
docker-smoke:
name: docker image smoke (slim, linux)
needs: changes
# Nightly / on-demand, or a PR that touches the image/locks/packaging.
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || needs.changes.outputs.docker == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Build the slim engine image
run: docker build -f docker/Dockerfile -t messagefoundry:ci .
- name: Build the -sqlserver variant (ODBC apt layer + sqlserver lock must resolve)
# Build-only: it pulls the Microsoft ODBC repo + installs from requirements-sqlserver.lock, so a
# broken hashed dep or the Debian/MS-key issue the Dockerfile warns about turns the job red here.
run: docker build -f docker/Dockerfile --target runtime-sqlserver -t messagefoundry:sqlserver-ci .
- name: Build the test smoke image (baked minimal config + MLLP sender)
run: docker build -f docker/smoke/Dockerfile --build-arg BASE=messagefoundry:ci -t messagefoundry:smoke .
- name: Serve (loopback, auth off) and assert an MLLP message reaches PROCESSED
run: |
set -euo pipefail
# Bind the API + MLLP to loopback INSIDE the container and drive the test from inside it
# (docker exec). A non-loopback bind without TLS would be refused by the startup bind guard
# (and auth-off is refused off-loopback regardless of --allow-insecure-bind), so loopback +
# in-container traffic is the correct posture for an auth-disabled smoke — no insecure override.
# GIVEN 1 (ADR 0148): `dev` now derives PHI, so `serve --env dev` runs the secure PHI posture.
# This smoke validates PHI+enforce green (not a synthetic opt-out): provision a runtime-minted
# store key (never a committed literal — gitleaks), bounded PHI-body retention windows, and a
# locked-down egress allowlisting the smoke config's File archive dir (/var/lib/mefor/out/adt).
# The security-notification gate is skipped because auth is off (no accounts to notify).
storeKey="$(python3 -c 'import base64, os; print(base64.b64encode(os.urandom(32)).decode())')"
docker run -d --name mefor \
-e MEFOR_SECURITY_REQUIRE_SIGN_IN=false \
-e MEFOR_STORE_ENCRYPTION_KEY="$storeKey" \
-e MEFOR_SECURITY_DELETE_MESSAGE_BODIES_AFTER_DAYS=30 \
-e MEFOR_RETENTION_DEAD_LETTER_DAYS=30 \
-e MEFOR_SECURITY_BLOCK_UNLISTED_OUTBOUND=true \
-e MEFOR_EGRESS_ALLOWED_FILE_DIRS=/var/lib/mefor \
messagefoundry:smoke \
serve --config /config --env dev --host 127.0.0.1 --port 8765
# Readiness: /health is tokenless and always 200 once the listener is up.
for _ in $(seq 1 60); do
docker exec mefor curl -fsS http://127.0.0.1:8765/health >/dev/null 2>&1 && break || sleep 0.5
done
docker exec mefor curl -fsS http://127.0.0.1:8765/health
# Send one synthetic ADT^A01 over the in-container MLLP listener (loopback:2575); fails on a non-AA ACK.
docker exec mefor python /smoke/send_adt.py 127.0.0.1 2575
# Poll the disposition via the container's own python (stdlib urllib — no host python/jq needed).
total() { docker exec mefor python -c "import urllib.request,json;print(json.load(urllib.request.urlopen('http://127.0.0.1:8765/messages?status=$1'))['total'])"; }
processed=0
for _ in $(seq 1 60); do
# `|| processed=0`: a transient non-zero before the row finalizes must NOT abort the loop under set -e.
processed=$(total processed) || processed=0
[ "$processed" -ge 1 ] && break || sleep 0.5
done
received=$(total received) || received=0
echo "processed=$processed received=$received"
# The whole point of the assert: confirm it FINALIZED, not just that it was acknowledged-on-receipt.
if [ "$processed" -lt 1 ]; then
echo "FAIL: message did not finalize to PROCESSED (received=$received) — stuck at ACK-on-receipt"
exit 1
fi
- name: Verify graceful shutdown (tini -> SIGTERM -> engine.stop drains)
run: |
set -euo pipefail
docker stop --time 30 mefor
code=$(docker inspect -f '{{.State.ExitCode}}' mefor)
echo "container exit code: $code"
# A fully graceful shutdown exits 0 (uvicorn handles SIGTERM, runs the lifespan, returns); 143
# (128+SIGTERM) is acceptable too. 137 = SIGKILL after the grace expired = ungraceful.
if [ "$code" = "137" ]; then echo "FAIL: ungraceful shutdown (SIGKILL after grace)"; exit 1; fi
# Poll for the marker: `docker stop` returns the instant the container exits, but the final
# buffered log line ("Application shutdown complete") can land in `docker logs` a beat later, so
# grepping once immediately is racy (a clean 143 shutdown then read as a missing marker). Retry
# for ~10s so a slow log flush is not misread as an incomplete lifespan shutdown.
for _ in $(seq 1 20); do
if docker logs mefor 2>&1 | grep -q "Application shutdown complete"; then
echo "graceful shutdown verified"; exit 0
fi
sleep 0.5
done
echo "FAIL: no clean-shutdown marker (lifespan shutdown did not complete)"
docker logs mefor 2>&1 | tail -20
exit 1
- name: Engine log (always)
if: always()
run: docker logs mefor 2>&1 | tail -60 || true
- name: Tear down
if: always()
run: docker rm -f mefor || true
# Single STABLE required status check standing in for the conditional / matrix heavy legs (SQL
# Server, Postgres, load, Windows-service smoke). Those legs CANNOT be required directly:
# * the nightly/path-gated legs (`postgres-store`, `load-test`, `load-test-sqlserver`,
# `windows-service-smoke`) report `skipped` on most or all PRs — never a stable required
# context a PR can wait on — so requiring them either no-ops or wedges; and
# * the matrix legs report an UNEXPANDED name when skipped (`... ${{ matrix.label }}`) but
# EXPANDED names when they run (`... 2022`/`2025`, `(windows-2022, py3.14)`), so no single
# context string matches both the skipped and the run state.
# This gate ALWAYS runs (if: always()), so it reports ONE fixed context ("CI gate") regardless of
# which legs ran, and fails iff a gated leg actually FAILED or was cancelled — a `skipped` leg
# counts as a pass. Require "CI gate" in branch protection INSTEAD of the individual legs; a leg
# that does run on a PR (e.g. `sqlserver-store` on a server-DB PR) thus still blocks the merge.
# NB: `docker-smoke` is intentionally NOT gated here (out of the named set) — add it to `needs`
# below to make a container-smoke failure block merge too.
ci-gate:
name: CI gate
if: always()
needs:
- changes
- sqlserver-store
- postgres-store
- load-test
- load-test-sqlserver
- windows-service-smoke
# `webconsole` rides this roll-up rather than taking a required context of its own. It is NOT
# path-gated like the five above -- it runs on every PR -- so it COULD have been required
# directly; riding here is the cheaper correct choice, because adding a context means a
# branch-protection change and, until that lands, a PR that can never satisfy it. The coverage is
# identical either way: this job is `if: always()` and fails on any needed job's `failure` or
# `cancelled`, and `CI gate` is required. Before this, the console suite gated merges by being a
# step inside the required `test` legs; it must not silently stop gating them now that it is not.
- webconsole
# `tooling` rides here for the same reason, and the precedent above is the whole argument: it was
# a set of steps inside the required `test` legs until the manifest split it out, so without this
# line the harness tier would stop gating merges the moment it stopped being part of `test`. It IS
# path-gated (unlike webconsole), so it skips on most PRs -- and a skipped need is not a failure
# to the `contains(needs.*.result, ...)` check below, which is the behaviour wanted.
- tooling
runs-on: ubuntu-latest
steps:
- name: Fail if any gated leg failed or was cancelled
if: >-
contains(needs.*.result, 'failure') ||
contains(needs.*.result, 'cancelled')
env:
# Pass the needs context via env, not inline ${{ }} in the run block (defense-in-depth against
# template injection — a job output could in theory carry a quote that breaks out of echo).
NEEDS_JSON: ${{ toJSON(needs) }}
run: |
echo "::error::A gated CI leg failed or was cancelled."
echo "$NEEDS_JSON"
exit 1
- name: Gated legs OK
run: echo "All gated CI legs succeeded or were skipped."