feat(security): screen for a requirement identifier PAIRED WITH A VERDICT, not for identifiers (BACKLOG #1337) #2530
Workflow file for this run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: CI | |
| on: | |
| push: | |
| branches: [main] | |
| pull_request: | |
| workflow_dispatch: | |
| schedule: | |
| # Nightly (03:17 UTC — off the top-of-hour rush) run of the HEAVY POST-MERGE legs: the load | |
| # smokes (load-test, load-test-sqlserver x2 majors), the server-DB suites (sqlserver-store x2 | |
| # majors, postgres-store), windows-service-smoke (x2 SKUs, 2x-billed) and docker-smoke. These | |
| # all used to run on EVERY push to main (~16 auto-merges/day re-validating content whose PR run | |
| # had just passed); the nightly cadence + the PR path-gates (`changes`) keep the same coverage | |
| # at a fraction of the runs. `gh workflow run ci.yml --ref main` (workflow_dispatch) still runs | |
| # EVERYTHING on demand — use it to re-validate a specific merge immediately. | |
| # IMPORTANT: the cron must still NOT drag the FUNCTIONAL matrix along. `test` and `ide` gate | |
| # their full-coverage arm as `push || workflow_dispatch` — deliberately EXCLUDING `schedule` — | |
| # so the nightly never re-executes them (they already ran on the merge itself; the original | |
| # `!= 'pull_request'` form would have swept them into the cron). The nightly legs' `if:` is | |
| # `schedule || workflow_dispatch` (+ a PR path-gate arm where `changes` provides one). | |
| - cron: "17 3 * * *" | |
| concurrency: | |
| # Per-ref group so a new push/PR-update supersedes its own in-flight run. The nightly `schedule` run | |
| # gets a SEPARATE group (…-nightly): without this it would share `ci-refs/heads/main` with push-to-main | |
| # and a routine daytime push would cancel-in-progress the in-flight nightly LOAD run — silently killing | |
| # the very long-running coverage the cron exists to provide. Scheduled and push runs no longer collide. | |
| group: ci-${{ github.ref }}${{ github.event_name == 'schedule' && '-nightly' || '' }} | |
| cancel-in-progress: true | |
| # Least privilege: every job here only reads the repo and runs tests/builds (no gh writes, no git push, | |
| # no secrets), so a read-only default token is sufficient. A job needing more declares its own block. | |
| permissions: | |
| contents: read | |
| jobs: | |
| # Lint + type-check + unit tests. The project supports a single Python — 3.14 — so every leg runs it. | |
| # Linux carries the cheap breadth (1x minutes); Windows Server 2022 + 2025 is the primary deployment | |
| # target, so 3.14 is also exercised on both Server SKUs even though Windows bills at 2x — testing the | |
| # version we ship on the OS we ship it on is worth the minutes. | |
| test: | |
| name: test (${{ matrix.os }}, py${{ matrix.python-version }}) | |
| # `needs: changes` for the docs-only short-circuit ONLY — this REQUIRED job still RUNS on every PR | |
| # and ALWAYS reports its context. On a docs-only PR `changes.outputs.code == 'false'` and the | |
| # expensive steps (install/lint/type/test) are skipped, so the leg goes green in seconds; the | |
| # required `test (…, py3.14)` context stays present + green. (changes never fails, so this needs | |
| # edge can't wedge the required context.) | |
| needs: changes | |
| # Every leg runs on GitHub-HOSTED runners now — the self-hosted NucBox Windows runners are RETIRED. | |
| # This repo (MEFORORG) runs the full ubuntu + windows-2022 + windows-2025 matrix, all FREE (hosted | |
| # minutes are free on a public repo); a FORK building its own pushes gets the UBUNTU leg only, to | |
| # keep a contributor's personal CI cheap. Hosted-everywhere is also safer: a self-hosted runner must | |
| # never be reachable by a public repo (fork PR = RCE on the owner's LAN). `matrix.hosted` is the | |
| # runner label set; the per-repo matrix is built in `changes`. | |
| runs-on: ${{ matrix.hosted }} | |
| # Wall-clock backstop: bound each leg so a hung test fails in minutes, not the 6h default. Three | |
| # nested watchdogs catch a hang at increasing scope (#55): the pytest-timeout per-test cap (matrix | |
| # `pytest_timeout`, 60s ubuntu / 120s Windows -- the step passes `--timeout=`, which overrides the | |
| # pyproject addopts value) + the faulthandler belt fire FIRST and name the stuck frame; the pytest STEP | |
| # has its own `timeout-minutes` (matrix `step_timeout`) so a process-level deadlock below pytest | |
| # fails the step fast; this job cap is the outermost belt if even that is somehow out-raced. | |
| timeout-minutes: ${{ matrix.job_timeout }} | |
| strategy: | |
| fail-fast: false | |
| # Per-repo matrix from the `changes` job: the full ubuntu+windows matrix here, ubuntu-only on a | |
| # fork's own pushes. Each leg carries os / python-version / hosted (the runner labels, an array) + | |
| # the timeout knobs. NB: a fork *PR* against this repo still runs in THIS repo's context, so it | |
| # gets the full matrix — the required `test (windows-2022/2025, py3.14)` contexts are always | |
| # present on a PR here, and a required-but-absent context (which blocks a PR forever) can't occur. | |
| matrix: ${{ fromJSON(needs.changes.outputs.matrix) }} | |
| defaults: | |
| run: | |
| shell: bash # available on all runners (Git Bash on Windows) -> one set of commands | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Set up Python ${{ matrix.python-version }} | |
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: ${{ matrix.python-version }} | |
| # No `cache: pip`: installs now go through uv (the "Set up uv" step below), which keeps its | |
| # OWN wheel/download cache — a pip cache here would be a second, redundant cache competing | |
| # for the repo's 10 GB cache budget. uv's cache is keyed on pyproject.toml + requirements.lock. | |
| # Everything below is gated on `code` (docs-only short-circuit) OR a push-to-main / on-demand run | |
| # (full coverage). On a docs-only PR these steps no-op and the REQUIRED leg reports green in seconds. | |
| # The job itself (and the Python/cache setup above) still runs, so the required context is always | |
| # present. NB: the gate is duplicated per-step (a job-level `if` would SKIP the whole job and remove | |
| # the required context) — keep the condition identical on each. The push/dispatch arm is written | |
| # `push || workflow_dispatch` (NOT `!= 'pull_request'`) so the nightly `schedule` event does NOT | |
| # re-run this full functional matrix — the cron is reserved for the long load legs only. | |
| # DELIBERATELY UNGATED — the one step in this job with no `code` condition. | |
| # | |
| # The ledger gate polices the ADR/BACKLOG number space, and an ADR-only PR is by definition | |
| # DOCS-ONLY, so `code == 'false'` and every gated step above is skipped. Gating this one the same | |
| # way would skip it on exactly the pull requests it exists to police. It needs only git + stdlib | |
| # python (no install), so it costs seconds and runs on every PR. | |
| # | |
| # It rides inside the already-REQUIRED `test` leg rather than becoming a new required context: a | |
| # brand-new required check wedges every PR opened before it existed (see the `ci-gate` roll-up). | |
| # | |
| # This is the backstop for `git commit --no-verify` and for a branch cut from a stale main — each | |
| # branch is internally consistent, and the duplicate number only exists once BOTH have merged, so | |
| # only a check against freshly-fetched main can see it. | |
| - name: Ledger gate (ADR / BACKLOG number space) | |
| if: runner.os == 'Linux' | |
| run: | | |
| # Only `origin/main` needs fetching: the gate reads it (`git show origin/main:…`, `ls-tree`) and | |
| # TWO-dot-diffs it against HEAD. No ancestry is resolved, so the shallow checkout is fine and no | |
| # deepening is required -- see the long note in ledger_check.py's tree-access section for why | |
| # three-dot was wrong here, and why its failure was silent. | |
| git fetch --no-tags --depth=200 origin main | |
| python scripts/hooks/ledger_check.py --ci | |
| # ALSO DELIBERATELY UNGATED, for exactly the reason the step above is — and it was NOT, which | |
| # cost `main`. | |
| # | |
| # Two guards police docs/BACKLOG.md. The number-space gate above is ungated because "gating it | |
| # would skip it on exactly the pull requests it exists to police". The STATUS invariant was | |
| # reachable only through pytest, which IS gated on `code == 'true'` — so on a BACKLOG-only PR | |
| # (the exact shape it exists to check) it did not run at all. | |
| # | |
| # On 2026-08-01 that landed a #320 entry whose banner used an emoji the invariant does not | |
| # accept. The PR was docs-only, went green in seconds without compiling the suite, merged, and | |
| # red `main` for every other session until someone else's PR tripped over it. A 0.6s markdown | |
| # parse would have caught it. One hardened guard and one unhardened guard over the same file is | |
| # worse than neither, because the hardened one makes it reasonable to assume BACKLOG changes are | |
| # covered. | |
| # | |
| # No install needed, so the step above's "git + stdlib python, costs seconds" property holds: | |
| # scripts/docs/backlog_status_check.py imports only argparse/re/sys/pathlib and has its own CLI. | |
| # tests/test_backlog_status_check.py imports that SAME module, so there is one implementation and | |
| # the unit tests keep covering its edge cases — this only adds the always-on invocation. | |
| # --min-items is the anti-narrowing floor, and it is the point of this invocation as much as the | |
| # banner check is. The item namespace spans docs/BACKLOG.md AND docs/archive/backlog/: retiring | |
| # items moves them between those files, so every OTHER assertion here — one banner per item, no | |
| # contradictions, no duplicates — is satisfied just as easily by scanning a remnant of the corpus | |
| # as the whole of it. Without a floor, a change that stopped the archive being read would go green. | |
| # Raise the number when the total legitimately grows; it must never be lowered to make CI pass. | |
| # WARNING: THE FLOOR LIVES IN TWO PLACES AND NOTHING COMPARES THEM: here, and `_MIN_TOTAL_ITEMS` in | |
| # tests/test_backlog_status_check.py. Raise BOTH, or the lower one becomes the only floor that | |
| # binds. Found on 2026-08-05 at 277 against a corpus of 300 — 23 items of accumulated slack, in a | |
| # guard whose entire purpose is to notice the corpus shrinking. | |
| - name: Backlog status invariant (ungated — see above) | |
| if: runner.os == 'Linux' | |
| run: python scripts/docs/backlog_status_check.py --min-items 300 | |
| # THE SAME ARGUMENT, ONE STEP FURTHER — and the evidence is from 2026-08-04. | |
| # | |
| # Ten `*_doc_drift` / `docs_*` modules exist to police documents, and every one of them is | |
| # reachable ONLY through pytest, which is gated on `code == 'true'` two steps below. So on a | |
| # docs-only PR — the exact shape they exist to check — none of them runs. That is the same defect | |
| # the status invariant above was added to fix, at ten times the surface. | |
| # | |
| # It is not hypothetical. On 2026-08-04 four docs-only PRs merged (#197, #198, #200, #201) and the | |
| # doc guards ran on none of them. Two carried citation errors found only by reading: an ADR named | |
| # ONE inbound citation of `docs/releases/` where there are 23 (12 via `docs/releases/...`, 15 via | |
| # relative `(releases/...`, and neither grep form alone finds them all), and a `docs/SECURITY.md` | |
| # route-table row asserted a refusal `DELETE /me/mfa` does not make. | |
| # | |
| # IT RECURRED ON 2026-08-11, AND THIS STEP WAS ALREADY HERE — the list was simply incomplete. | |
| # `tests/test_dast_claims.py` scans documents for prose reading as though the independence gap | |
| # were closed, but was never added, so a docs-only PR (#322) merged green and RED MAIN on the push | |
| # afterwards. The blind mode is worse than a plain gap because of WHO PAYS: the failure surfaces | |
| # on the next PR that touches code, so it is misattributed to whoever opens it. That happened — | |
| # the next code PR inherited a red its author had no part in. | |
| # | |
| # => THE LESSON IS ABOUT THE LIST, NOT THE GATING. A curated allowlist silently omits; nothing in | |
| # a green run says "a doc guard exists that I did not run." When adding a doc-scanning test | |
| # module anywhere in `tests/`, add it HERE in the same commit, and confirm it needs no extras — | |
| # a module that cannot run on `[dev]` alone would red every docs-only PR, which is the failure | |
| # mode the minimal-install note below exists to avoid. | |
| # | |
| # WHY A SEPARATE MINIMAL INSTALL instead of ungating the install below. Unlike the status | |
| # invariant, these modules import the package (`config.settings`, `config.models`, | |
| # `parsing.binary`, `auth.service`) and one imports `fastapi.routing`, so they cannot run on | |
| # stdlib alone. But they need NO extras — fhir/dicom/x12/xml are irrelevant to a doc scan — so a | |
| # docs-only PR pays a base editable install and ~5s of tests rather than the full extras install. | |
| # The gated steps below are deliberately UNTOUCHED, so a code PR is byte-identical to before. | |
| # | |
| # WARNING: 89 of these tests SKIP here and that is structural, not a gap to fix in this step: | |
| # `tests/test_threat_model_doc_drift.py` asserts against `docs/security/THREAT-MODEL.md`, which is | |
| # vault-only and absent from this tree. ADR 0156 records that class (six `*_doc_drift` modules | |
| # assert against documents `git ls-files docs/security/` shows are not present) and ASVS 15.1.3 is | |
| # open on it. 152 assertions DO run, which is the point; do not read the skips as coverage. | |
| # `[dev]` and not a bare `pip install pytest`, and this is NOT belt-and-braces. pyproject sets | |
| # `asyncio_mode = "auto"` (needs pytest-asyncio) and `addopts = "--timeout=60 | |
| # --timeout-method=thread"` (needs pytest-timeout), so a bare pytest ERRORS on an unknown option | |
| # before collecting anything — a step that could never pass, reding every docs-only PR. `[dev]` is | |
| # the canonical "can run this suite" extra and pins both plugins; it is still far lighter than the | |
| # gated install below, which adds console + fhir + dicom + x12 + xml that no doc scan touches. | |
| # `--constraint constraints.lock` for the same DEP-1 reason every other install here carries it. | |
| - name: Install (minimal — for the doc guards on a docs-only PR) | |
| if: runner.os == 'Linux' && needs.changes.outputs.code != 'true' && github.event_name == 'pull_request' | |
| run: | | |
| python -m pip install --upgrade pip | |
| pip install -e ".[dev]" --constraint constraints.lock | |
| - name: Doc guards (ungated — the docs-only blind spot; see above) | |
| if: runner.os == 'Linux' && needs.changes.outputs.code != 'true' && github.event_name == 'pull_request' | |
| run: | | |
| # PRINT WHAT IS SCANNED BEFORE RUNNING IT. A list that silently shrinks is how this class of | |
| # guard goes quiet, and `-rs` names every skip rather than letting one read as a pass. | |
| # | |
| # ONE LIST, USED TWICE. It used to be written out twice — once for the printf and once for | |
| # pytest — which made the "print what you scanned" defence able to LIE: the two copies could | |
| # drift, and then the step prints a module it does not run, or runs one it does not print. | |
| # A defence implemented by duplication defeats itself. The variable is the fix, and the | |
| # cross-check below turns a drift into a hard failure rather than a quiet one. | |
| # MEMBERSHIP RULE (#1262). A module belongs here if it CAN FAIL on a change touching only | |
| # allowlisted paths -- because those are exactly the changes that skip the suite, so those | |
| # are exactly the gates that would be silently absent at the only moment they are needed. | |
| # | |
| # DERIVE IT BY TRACING WHAT A MODULE OPENS, NOT BY READING ITS NAME OR ITS SOURCE. Measured | |
| # 2026-08-14, both static shortcuts fail, in opposite directions: | |
| # "source mentions a docs-ish token" -> 135 candidates; the signal drowns | |
| # "source quotes a real allowlisted path" -> MISSES 8 OF THE 16 then listed here, because | |
| # gates build paths (REPO / "docs" / name) | |
| # rather than writing one literal | |
| # A criterion that cannot re-derive the KNOWN members cannot be trusted to find unknown | |
| # ones. The check that works is to patch open/Path.read_text and record which allowlisted | |
| # paths a module actually reads while running. | |
| # | |
| # THIS LIST IS A FLOOR, NOT A CENSUS. The trace that produced the two additions below ran in | |
| # an environment missing five CI extras, where 89 tests SKIPPED -- and a skipped test reads | |
| # nothing. More members may be missing. Re-derive with the full extras before believing the | |
| # set is complete. | |
| # | |
| # test_doc_ref_handle.py READS NO DOCUMENTATION -- it tests mfdoc:v1:ref: document handles | |
| # in the message store, and is here because its NAME reads like "documentation reference". | |
| # KEPT DELIBERATELY (dispatcher ruling, #1262): removing a guard is a different act from | |
| # adding one, and dropping it on a reading of its name is the same move that put it here. | |
| # The cost is a few wasted seconds; the cost of the other error is a gate that stops running. | |
| DOC_GUARDS="tests/test_asvs_file_surface_doc_drift.py tests/test_cloud_phi_hipaa_doc_drift.py | |
| tests/test_crit2_inline_doc_drift.py tests/test_doc_ref_handle.py tests/test_docs_db_grants.py | |
| tests/test_docs_runbooks.py tests/test_docs_security_pathways.py tests/test_security_doc_drift.py | |
| tests/test_security_doc_rate_limits.py tests/test_threat_model_doc_drift.py | |
| tests/test_backlog_status_check.py tests/test_sds_rule_ids_are_stable.py | |
| tests/test_link_resolution.py tests/test_dast_claims.py | |
| tests/test_claude_section_citations.py tests/test_write_share_denominator.py | |
| tests/test_cutover_slug_rot.py tests/test_backlog_citation_check.py | |
| tests/test_dangling_citation_check.py" | |
| # Every named module must EXIST. A path typo would otherwise make pytest error on an unknown | |
| # file, or — worse under a future -k/--ignore form — silently scan nothing and read as a pass. | |
| for m in $DOC_GUARDS; do | |
| test -f "$m" || { echo "doc-guard list names a missing module: $m"; exit 1; } | |
| done | |
| echo "doc guards, docs-only PR — scanning these modules:" | |
| printf ' %s\n' $DOC_GUARDS | |
| echo " ($(printf '%s\n' $DOC_GUARDS | wc -l) modules)" | |
| pytest -q -rs $DOC_GUARDS | |
| # PySide6's offscreen platform plugin needs a few system libraries even | |
| # headless. Linux-only; Windows runners need no equivalent. | |
| - name: Install Qt offscreen system libraries | |
| if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux' | |
| # BOUNDED AND RETRIED. Unguarded, this step turns an external apt-mirror hiccup into a | |
| # blocked merge queue: measured 2026-08-18, three hangs across two attempts on three | |
| # different jobs, one of them 27+ minutes on `test (ubuntu-latest, py3.14)` -- a REQUIRED | |
| # context, so the queue stops. Two levers, because they answer different failures. The | |
| # per-command `timeout` kills a HANG and lets the retry run; `timeout-minutes` is the | |
| # backstop that stops this step ever eating a job budget again if the loop is edited wrong. | |
| # The failure text names the cause, so the next reader is not sent hunting in their diff -- | |
| # a check whose label points at the wrong subject is what BACKLOG #1254 is about. | |
| timeout-minutes: 8 | |
| run: | | |
| for attempt in 1 2 3; do | |
| if sudo timeout 120 apt-get update && sudo timeout 180 apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3; then | |
| exit 0 | |
| fi | |
| echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying" | |
| sleep $((attempt * 5)) | |
| done | |
| echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test." | |
| exit 1 | |
| # uv is the installer for every leg (was pip): it resolves + installs the pyproject extras far | |
| # faster than pip, into the same setup-python interpreter via `uv pip install --system`. Pinned | |
| # by the same SHA dependabot-lock-resync.yml already uses. Gated identically to the install | |
| # below (docs-only PRs skip both). enable-cache defaults to 'auto' — a GitHub-hosted cache on | |
| # hosted runners, the persistent on-disk uv cache on a self-hosted runner — keyed on the glob. | |
| # | |
| # EVERY install below passes `--constraint constraints.lock` — the HASHLESS export of uv.lock, kept | |
| # in sync by the DEP-1 gate. Without it, `uv pip install -e ".[extras]"` RE-RESOLVES from | |
| # pyproject's `>=` floors and silently adopts whatever upstream published since: ruff 0.16.0 and | |
| # annotated-types 0.8.0 (dropped `SLOTS`, broke fhir-core) each reddened EVERY open PR on the same | |
| # day — see each cap's rationale in pyproject.toml. The constraint pins every version to the lock | |
| # while each job still installs only ITS OWN extras (so windows-service-smoke's deliberately | |
| # narrow set stays narrow; the docker-smoke job builds from docker/locks/* and installs nothing). | |
| - name: Set up uv | |
| if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch' | |
| uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| requirements.lock | |
| - name: Install project (dev + console + fhir + dicom + x12 + xml extras) | |
| if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch' | |
| run: | | |
| # [fhir] so the FHIR typed-tier tests (incl. the PHI-no-leak invariant, ADR 0022) actually | |
| # run their assertions in CI rather than importorskip-skipping (extra-less local runs skip). | |
| # [dicom] (pydicom + pynetdicom; ADR 0025) does the same for the DICOM codec + C-STORE SCP | |
| # tests, and lets mypy type-check transports/dicom.py + parsing/dicom/ against the real | |
| # (py.typed) libraries rather than treating them as Any. | |
| # [x12] (pyx12) + [xml] (lxml/xmlschema/signxml) likewise so the strict X12 validation (#32) | |
| # and the hardened XML/SOAP codec (#31) tests run their assertions instead of importorskip-skipping. | |
| # [webauthn] (ADR 0068) so the passkey ceremony tests run real verify_* assertions instead of | |
| # importorskip-skipping, and mypy sees the py.typed library rather than Any. | |
| # + the browser ops console as a second editable wheel (Option B) so the moved /ui tests | |
| # (tests/test_webui.py etc. now import messagefoundry_webconsole) run on this full-suite leg. | |
| # It is NOT a published extra yet (would break uv lock), so install it by path. | |
| uv pip install --system --constraint constraints.lock -e ".[dev,harness,fhir,dicom,x12,xml,webauthn]" -e packaging/messagefoundry-webconsole | |
| # ruff (lint + format) and mypy are PLATFORM-INDEPENDENT results — a ruff/format outcome is | |
| # identical on every OS, and mypy's platform-specific branches are covered by running it for | |
| # BOTH platforms on the cheap Linux leg (below). Running them on the 2x-billed Windows legs was | |
| # duplicate cost, so gate them to Linux. The Windows legs keep pytest — the one check whose | |
| # result genuinely differs by OS (real sockets, ProactorEventLoop, Windows service paths). | |
| # `.` — the WHOLE repo, matching the format check below and the pre-commit ruff hook. NARROWED | |
| # 2026-08-18: that hook used to lint every changed Python file, because `language: system` ran a | |
| # bare `ruff check` with no `--force-exclude`, so an explicitly-passed path was linted even when | |
| # pyproject excluded it. The hook is now upstream's, whose entry carries `--force-exclude`, so | |
| # BOTH sides honour extend-exclude and the two agree for the first time — the hook was previously | |
| # STRICTER than CI on excluded paths, not looser. This step was an explicit allow-list of six paths, | |
| # and the mismatch was not academic: harness/ (73 findings), tee/ (12), samples/ and docker/ were | |
| # linted by the hook and by nothing in CI, so touching any file there blocked the commit on errors | |
| # CI could not see and no CI run could ever have reported. Scope now lives in ONE place — | |
| # pyproject's [tool.ruff] extend-exclude — instead of being re-stated, differently, per tool. | |
| # tests/test_lint_scope_parity.py fails if the two drift apart again. | |
| - name: Lint (ruff) | |
| if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux' | |
| run: ruff check . | |
| - name: Format check (ruff) | |
| if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux' | |
| run: ruff format --check . | |
| # Licence headers (BACKLOG #1010). Mirrors the `licence-header` pre-commit hook. Scope lives in | |
| # the script's COMMENT_PREFIXES map rather than being restated here, so unlike ruff and bandit | |
| # these two callers cannot drift apart in the first place. | |
| # | |
| # DELIBERATELY NOT PATHS-FILTERED, unlike the ruff steps above. `changes.code` keys on | |
| # `alwayscode='\.(py|ps1|sh|ts|js|yml|yaml|toml|lock|cfg|ini)$'`, which omits `.go` -- so a | |
| # Go-only PR sets code=false and this gate would skip on precisely the PR that introduced the | |
| # headerless file. No `.go` source is tracked today, but BACKLOG #1011 rules on whether a Go tree | |
| # stays, and a gate that silently stops covering a language the moment one is added is the exact | |
| # failure this item exists to remove. The scan reads ~1,300 files in under a second, so filtering | |
| # it would buy nothing and cost a blind spot. | |
| - name: Licence headers (SPDX) | |
| if: runner.os == 'Linux' | |
| run: python scripts/quality/licence_header_check.py | |
| # Control bytes. Mirrors the `control-char` pre-commit hook, and exists for the reason zizmor.yml | |
| # states for actionlint: the hook is the load-bearing half and this step is the backstop for | |
| # `git commit --no-verify` and for a fresh clone whose hooks are not installed until | |
| # scripts/coord/install-git-hooks.ps1 runs. The hook shipped without this step, so for one day the | |
| # gate ran only where hooks happened to be installed -- a gate against bytes no human view shows, | |
| # itself invisible when skipped. | |
| # | |
| # Scope AND the ide/src NUL allowance live in the script, not here, for the reason the licence | |
| # step above gives: two callers that restate a scope drift apart, and the drift is silent. | |
| # | |
| # DELIBERATELY NOT PATHS-FILTERED, and the argument is sharper than for licence headers. This | |
| # gate covers .md, .json and .txt, none of which appear in `changes.code`'s alwayscode pattern -- | |
| # so a docs-only PR sets code=false, and a filter would skip the gate on exactly the prose where | |
| # a pasted escape most often lands. ~1,850 files, stdlib only, well under a second. | |
| - name: Control bytes (invisible) | |
| if: runner.os == 'Linux' | |
| run: python scripts/quality/control_char_check.py | |
| # BACKLOG #1226. The screen shipped and NOTHING RAN IT, so it read as coverage and produced | |
| # none -- a username reaching a WHERE clause was found three times by accident and zero times | |
| # on purpose. --baseline is what makes this step able to FAIL without turning the screen into | |
| # a verdict-emitter: it is silent on the judged sites and red on a key nobody has read yet. | |
| # Several judged entries are deliberately CORRECT code and one is a confirmed defect, so the | |
| # baseline records that a human looked, never that a site is fine. stdlib only. | |
| - name: Username-as-access-key screen (new sites only) | |
| if: runner.os == 'Linux' | |
| run: >- | |
| python scripts/quality/username_access_key_screen.py | |
| --baseline scripts/quality/username_access_key_baseline.txt | |
| # Two mypy passes on Linux so BOTH platforms' typed branches are checked without paying for a | |
| # Windows mypy: the default linux-platform run, plus an explicit --platform win32 run that types | |
| # the sys.platform=='win32' branches (ctypes.windll / DPAPI / service_control) the linux run | |
| # skips. Previously those were typed only by the mypy on the Windows legs. (All win32 code uses | |
| # stdlib ctypes — typeshed's win32-conditional stubs cover it; there are no win32-only deps — so | |
| # this resolves on a Linux runner. Verified: both passes clean on 204 files.) | |
| # `messagefoundry/tray` (the Windows notification-area service-manager, ADR 0113 — now shipped | |
| # inside the wheel) is a win32-only ctypes package, so it is EXCLUDED here and typed on the win32 | |
| # pass only: its ungated ctypes.WinDLL method calls are unreachable-and-skipped under the linux | |
| # platform but real under win32. Nothing in the engine imports it (it is a leaf entry point), so | |
| # --exclude cleanly drops it from the linux build. | |
| - name: Type-check (mypy, strict — linux platform) | |
| if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux' | |
| run: mypy messagefoundry messagefoundry_webconsole --exclude 'messagefoundry/tray/' | |
| # The win32 pass types messagefoundry/tray (now inside the messagefoundry package, so no separate | |
| # path argument) plus every other sys.platform=='win32' branch. | |
| - name: Type-check (mypy, strict — win32 platform) | |
| if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux' | |
| run: mypy --platform win32 messagefoundry | |
| # Step-level watchdog UNDER the job cap (#55): the windows-2022 leg intermittently | |
| # wedges ~25% in (a Windows ProactorEventLoop listener-teardown / socket wait that the shared | |
| # session event loop can't get past), emits no output for ~12 min, then the JOB cap CANCELS it | |
| # — a red X with no stack and no named test. Two belts make that fail FAST and NAMED instead: | |
| # * `--timeout-method=thread` (the ONLY method on Windows — SIGALRM is POSIX-only) dumps ALL | |
| # thread stacks at the per-test cap (matrix `pytest_timeout`, 120s on these Windows legs), | |
| # naming the stuck frame. | |
| # * PYTHONFAULTHANDLER=1 + `-o faulthandler_timeout=` (matrix `fault_timeout`) is the belt that fires even when the | |
| # thread-timer CANNOT interrupt a main-thread C-level wait (a Proactor overlapped op / | |
| # blocking accept / subprocess.wait): pytest's faulthandler plugin arms a separate watchdog | |
| # thread that dumps the NATIVE stack of every thread at `fault_timeout` (150s here, 90s on | |
| # ubuntu -- always set ABOVE that leg's `pytest_timeout`, so the | |
| # per-test dump is attributed first; this is the last-resort stack when the thread method is | |
| # itself out-raced). NB the plugin exposes this as the `faulthandler_timeout` ini key, not a | |
| # `--faulthandler-*` CLI flag, so it is passed with `-o`; it only DUMPS (never kills), safe. | |
| # The step `timeout-minutes` (matrix `step_timeout`) is the outer backstop, held well under the | |
| # job's own cap, so a hang both watchdogs somehow miss still fails the STEP (not the whole job at | |
| # the silent cap). Keep the gap: the step must expire BEFORE the job, or the failure surfaces as | |
| # an uninformative job-level kill with no step attribution. | |
| # | |
| # THAT GUARANTEE NOW COVERS BOTH GATED STEPS, and it did not until BACKLOG #344 proposal 5 landed. | |
| # `Web console tests (pytest)` used to carry the SAME `step_timeout` while running AFTER | |
| # `Tests (pytest)`, so reaching it had already spent setup + `Tests` and its own cap could not | |
| # fire first at any job_timeout worth setting (~55 min on ubuntu, ~114 on Windows) -- a hang there | |
| # was an unattributed job-level kill with no step conclusion. It now carries | |
| # `matrix.webconsole_step_timeout`, sized to its own work, and the nesting arithmetic holds on all | |
| # three legs; the derivation is in the note above that step. | |
| # | |
| # It only works while the cap stays clear of a HEALTHY run. On 2026-07-31 the ubuntu leg finished | |
| # green in 775s against a 780s cap -- 5s of margin -- and the step was killed anyway, reported as | |
| # a failure on a run whose last line was "9598 passed, 855 skipped". A watchdog that cannot | |
| # separate "deadlocked" from "slow today" stops being a watchdog and becomes a coin flip, so the | |
| # ubuntu budget is raised here. Re-check the margin when the suite grows. | |
| # | |
| # It then happened again on WINDOWS, 2026-08-01, because the sentence that used to close this | |
| # paragraph asserted the Windows legs held "~2x headroom" and nobody re-derived it. PR #119 was | |
| # killed at 26:07 against the 26:00 cap with ZERO tests failing. What moved was the suite, not the | |
| # code under test: #74 landed tests/test_worktree_prune_merged.py (1,506 lines) and windows-2025 | |
| # went 19:35 -> 26:07 on the same branch. The claim was already false when it was written. | |
| # | |
| # Measured by timing the `Tests (pytest)` STEP, which is what step_timeout gates. NOT the job: the | |
| # job runs several minutes longer and is capped separately, and at least three sessions misread job | |
| # durations as step durations while triaging this (c53f752b's JOB ran 28:41 and PASSED, because | |
| # job cap 30 vs step cap 26). | |
| # | |
| # POOL: every run of THIS workflow created 2026-08-01T00:00Z .. 2026-08-09T00:00Z UTC -- 688 runs. | |
| # Enumerated with `gh api` over a deliberately WIDER page range (1,000 runs, back to 2026-07-26) and | |
| # filtered locally on `created_at`. Jobs fetched with `?filter=all`, so an attempt killed at the cap | |
| # is not hidden behind its passing re-run. Rows are each leg's `Tests (pytest)` STEP, kept when THAT | |
| # STEP concluded, executions under 60s dropped (a docs-only PR skips the leg). 1,618 leg-executions; | |
| # n below is per leg, passing rows only. Measured 2026-08-08 (BACKLOG #1096). | |
| # | |
| # leg max passing step n old cap old margin new cap new margin | |
| # ubuntu-latest 20:28 35 25:00 1.221x 31:00 1.515x <- RE-DERIVED 2026-08-12 | |
| # | |
| # THE UBUNTU ROW WAS RE-DERIVED 2026-08-12 AND THE OLD ONE IS KEPT ABOVE IN THE PROSE, NOT DELETED. | |
| # Pool: every run of this workflow created after 2026-08-10 that contributed an ubuntu | |
| # `Tests (pytest)` STEP -- 58 runs, 63 rows, 35 passing. Max passing 20:28; next highest 17:18; | |
| # min 14:47; max non-success 17:01. Smaller pool than the 441-row original and stated as such. | |
| # | |
| # WHAT TRIGGERED IT, and the ordering matters: `main` ALREADY exceeded the old 16:08 anchor at | |
| # 17:07 BEFORE the PR that reddened the gate. The record had rotted on its own; PR #343 (+274 | |
| # tests, +3:12) only carried the MARGIN across the 1.30x floor. So this is not "one PR was too | |
| # big" -- the anchor was stale and the floor did its job by saying so. #1096 stays OPEN: the caps | |
| # are re-derived, the DRIFT underneath them is still not fixed, and raising a cap is not fixing it. | |
| # | |
| # 20:28 IS PR #343's OWN RUN, and that is deliberate rather than an outlier being smuggled in: its | |
| # 274 tests land on `main` when it merges, so 20:28 is the next baseline, not a one-off. Sizing on | |
| # the pre-merge population would re-break the cap on contact -- the "three PRs each adding a | |
| # minute" death this note already records. | |
| # windows-2022 29:23 399 36:00 1.225x 55:00 1.872x | |
| # windows-2025 35:47 CENSORED 360 36:00 1.006x 55:00 1.537x | |
| # | |
| # ONLY THE WINDOWS-2025 ROW IS CENSORED, AND THAT IS MEASURED, NOT ASSUMED. Largest ubuntu step | |
| # FAILURE in 535 leg-executions is 14:21; largest windows-2022 failure in 542 is 25:48. Neither leg | |
| # has ever touched its cap, so 16:08 and 29:23 are true maxima. windows-2025 was killed at the cap | |
| # SEVEN times in this window -- 36:08 (31189317409, main b78214f5), 36:08 (31199321840, main | |
| # 7ecff8ae), 36:08 (31202372465, PR #253), 36:07 (30955150892, 2026-08-04, the earliest), 36:07 | |
| # (31094875320, main fdec72ca), 36:07 (31192717684, PR #249), 36:01 (31149117314, PR #261) -- THREE | |
| # of them push runs on main. So 35:47 is the largest step that FIT in 36:00, and 36:08 is a lower | |
| # bound on the largest the suite wants. Smaller pools give smaller counts and each is right for its | |
| # own pool: BACKLOG #1096's day pool has five, a 3-day pool has six. State the pool with the count. | |
| # | |
| # THE KILLS WERE SLOWNESS, NOT A WEDGE -- CHECKED, NOT ASSUMED. Run 31149117314's pytest finished | |
| # GREEN and the step was killed 4.85 seconds later: | |
| # 05:37:53.777Z 10666 passed, 831 skipped, 22 warnings in 2148.75s (0:35:48) | |
| # 05:37:58.624Z ##[error]The action 'Tests (pytest)' has timed out after 36 minutes. | |
| # No faulthandler native-stack dump appears in any kill log. That settles the hang-versus-slow | |
| # question the #55 note above would otherwise leave open, and it has a second consequence that is | |
| # easy to miss: the killed rows are genuine population members, so "max passing" is CENSORED, not | |
| # conservative. Every ratio taken against it flatters itself. | |
| # | |
| # SUPERSEDED READING, kept because a number without its pool cannot be rechecked: the 2026-08-01 | |
| # pool (70 runs, pre-#131) read ubuntu 12:31 / W22 21:34 / W25 25:51 against the then-current | |
| # 26:00 Windows cap. Those figures are correct for that pool and that code state. They are not | |
| # comparable to the rows above, which are post-#131 and span eight days. | |
| # | |
| # THESE MAXIMA ARE LOWER BOUNDS, BECAUSE THE POOL IS RIGHT-CENSORED. Every run in it predates #131 | |
| # (28d186b5, landed 2026-08-02T00:35:28Z), so Windows ran under step 26:00 / job 30:00 and ubuntu | |
| # under 19:00 / 22:00. A run that wanted longer than its cap was KILLED at the cap and then -- | |
| # correctly, by the filter above -- dropped for not concluding success. So 25:51 is the largest | |
| # step that FIT in 26:00, not the largest the suite wants. Sizing a cap as a multiple of a | |
| # censored maximum under-provisions by construction, and that is exactly how "1.06x" read as | |
| # survivable right up until #119's leg was killed at 26:07. (#119 itself merged, 2026-08-02T01:45Z | |
| # -- what the cap killed was a run, not the PR. Saying a PR "died" here reads as never-landed and | |
| # had already propagated into docs/WORKTREES.md as exactly that claim.) | |
| # | |
| # THE CENSORED VALUES ARE VISIBLE IF YOU ASK FOR THEM, AND THEY MATTER. The jobs endpoint defaults | |
| # to `filter=latest`, which hides a killed attempt behind its passing re-run; `?filter=all` shows | |
| # both. #119's 26:07 is in this very pool that way (run 30717229521 attempt 1, `8c407fb5`, step | |
| # conclusion FAILURE) beside its 22:25 attempt-2 success -- a 3:42 spread on identical code, and | |
| # the first day AFTER the raise produced 26:23 TWICE, both concluding SUCCESS (runs 30728793103 | |
| # and 30731407003). So the population really does exceed the old 26:00 cap: the largest windows-2025 | |
| # `Tests (pytest)` execution observed to date is 26:23, not 25:51. | |
| # | |
| # THIS TABLE HAS BEEN WRONG TWICE. #131 published 12:27 / 18:39 / 24:35 over "11 passing runs"; | |
| # the first correction published the maxima above but over "101 runs" with n = 57 / 52 / 49. The | |
| # maxima re-derive exactly; BOTH pool sizes were unreproducible. Three mechanisms produced that, | |
| # all cheap to repeat: | |
| # | |
| # * THE POOL WAS A PAGE, NOT A QUESTION. #131's came from `gh run list --limit 20` -- a default | |
| # page size, reported as though it had been chosen. | |
| # * FILTERING ON *JOB* CONCLUSION DROPS THE TIGHTEST STEPS BY CONSTRUCTION. A step that nearly | |
| # exhausts step_timeout is the most likely to push its job into job_timeout, so the job is | |
| # cancelled while the step itself concluded success. Re-running this pool with a job-conclusion | |
| # filter reproduces #131's 24:35 exactly; the step-conclusion filter gives 25:51. Measure the | |
| # step; filter on the step. Two such rows exist on 2026-08-01 and one of them is the maximum. | |
| # * A COUNT NOBODY RE-DERIVED. Neither "101 runs" nor n = 57 / 52 / 49 is reproducible under any | |
| # pool definition tried. A table whose own point is "state your pool and your n" has to carry | |
| # an n the next reader can recompute -- that column is how they would tell it had rotted. | |
| # | |
| # Only the windows-2025 row was ever a maximum; #131's ubuntu 12:27 was that same run's ubuntu leg, | |
| # quoted as if it were that leg's worst case. Its 18:39 was not even that: that run's windows-2022 | |
| # step is 18:29, and 18:39 belongs to a different run entirely. | |
| # | |
| # State the MEASURED value, its POOL, its n and its DATE -- never a bare multiple. A multiple gives | |
| # the next reader no way to tell when it has rotted; a multiple without its pool cannot even be | |
| # rechecked. | |
| # | |
| # PROOF IT WAS THE CAP, NOT THE BRANCH. #119's windows-2025 leg was RE-RUN on the SAME commit | |
| # against the SAME 26:00 cap: attempt 1 was killed at the cap, attempt 2 concluded SUCCESS. Same | |
| # code, same config, same ceiling, two outcomes. The leg was not failing -- it was coin-flipping | |
| # against the cap, exactly the state the ubuntu note above names, and the reason "re-run it and | |
| # see" is not a diagnosis here. A green re-run at 26:00 does not mean the suite fits; it means | |
| # that runner was fast enough that time. | |
| # | |
| # THE CAP IS SIZED ON THE POST-#1027 POPULATION, NOT ON WHAT RUNS TODAY. PR #253 (BACKLOG #1027) is | |
| # open and MERGEABLE and sets testpaths = ["tests", "packaging/messagefoundry-webconsole/tests"], | |
| # while keeping the `Web console tests (pytest)` step -- so once it lands the web console suite runs | |
| # inside `Tests (pytest)` AS WELL. Sizing on the current population would let #253 land and re-break | |
| # the cap on contact, which is the "three PRs each adding a minute" death recorded at the foot of | |
| # this note. The one windows-2025 execution of that branch is one of the seven kills above, so the | |
| # post-#1027 windows-2025 duration HAS NEVER BEEN OBSERVED UNCENSORED -- and windows-2022's sizing | |
| # maximum (29:23) IS a post-#1027 row. One shared value, two code states, until this is fixed. | |
| # | |
| # leg largest observed Tests + largest observed web-console = post-#1027 anchor | |
| # ubuntu 20:28 (uncensored) 2:22 (n=441) 22:50 <- RE-DERIVED 2026-08-12 | |
| # W22 29:23 (uncensored; already a post-#1027 row) 29:23 | |
| # W25 36:08 (RIGHT-CENSORED) 3:59 (n=360) 40:07 lower bound | |
| # | |
| # SIZING RULE: step_timeout = ceil_minute(1.35 x that leg's post-#1027 anchor). 1.35 is the multiple | |
| # #131 used when it set 36:00 (1.364x over 26:23, the largest execution then observed). | |
| # ubuntu 22:50 x 1.35 = 30:50 -> 31:00 (1.358x over the anchor, 1.515x over max passing) | |
| # W22 29:23 x 1.35 = 39:40 -> 40:00 (takes the shared Windows value below) | |
| # W25 40:07 x 1.35 = 54:09 -> 55:00 (1.371x over the anchor, 1.537x over max passing) | |
| # | |
| # UBUNTU IS CHANGED IN THE SAME ACT BECAUSE IT IS THE NEXT INSTANCE, NOT AS TIDYING. At 19:00 against | |
| # its post-#1027 anchor of 18:30 it stands at 1.027x -- within seconds of where windows-2025 is now -- | |
| # and its job cap is ALREADY NEGATIVE on measured maxima (see the job paragraph below). Publishing | |
| # "ubuntu unchanged, still positive" beside a corrected Windows row would be a compensating control | |
| # resting on a false premise, which is the defect this repo names specifically. | |
| # | |
| # THE SPREAD RULE NOW HOLDS, WHERE AT 36:00 IT DID NOT. The rule is "headroom must exceed observed | |
| # spread". At 36:00 it FAILED on windows-2025 (headroom 9:37 against a 10:27 spread) and the old | |
| # revision kept the value anyway and said so. At 55:00, on the eight-day pool: | |
| # windows-2025 headroom 55:00 - 35:47 = 19:13 spread 35:47 - 24:48 = 10:59 HOLDS (+8:14) | |
| # windows-2022 headroom 55:00 - 29:23 = 25:37 spread 29:23 - 18:42 = 10:41 HOLDS (+14:56) | |
| # The 24:48 low is same-suite runner variance, not a smaller suite: three consecutive push-to-main | |
| # runs inside 2h42m gave windows-2025 34:15 / 24:48 / 34:29 while windows-2022 held a 1:00 band. | |
| # | |
| # RE-DERIVE IF a windows-2025 `Tests (pytest)` STEP is ever seen above 40:00, or if any Windows | |
| # `test` JOB is ever seen above 50:00 -- those are the triggers, not a calendar reminder. Both are | |
| # the 1.2x decay point of their cap (55:00 / 1.2 = 45:50, floored to 40:00 for margin against the | |
| # measured drift; 66:00 / 1.2 = 55:00, floored to 50:00). The OLD trigger of 28:00 is retired | |
| # because it was already universally exceeded: 58 of 59 passing steps on the day pool cleared it and | |
| # the median itself sat above it, so it had become a permanently-tripped alarm, which is | |
| # indistinguishable from no alarm at all. | |
| # | |
| # A ratio against one run says nothing about a distribution, and #119's leg was killed by the | |
| # distribution, not by its own duration. Both Windows legs take the same values, but they are sized | |
| # from DIFFERENT legs and that asymmetry is deliberate: the STEP cap is sized on windows-2025 (the | |
| # slower step, 35:47 censored against 29:23), while the JOB cap is sized on windows-2022 (the worse | |
| # non-step addends, setup 4:05 against 2:32). An earlier revision said "sized on windows-2025 as the | |
| # worse of the pair" without qualification -- that is FALSE for the job cap. | |
| # | |
| # THE JOB CAP IS NOT A ROUNDING-UP OF THE STEP CAP, AND IT HAS FIRED. Two steps in this job carry | |
| # a `timeout-minutes`, so the job must contain the SUM of two gated budgets plus setup -- a | |
| # quantity neither step cap can bound on its own. | |
| # | |
| # UNTIL BACKLOG #344 PROPOSAL 5 THAT SUM WAS UNBOUNDABLE, because both steps drew on the SAME | |
| # `step_timeout`: the worst case the caps admitted was 2 x step_timeout + setup, which no | |
| # job_timeout worth setting could cover (2 x 25 = 50 > 37 on ubuntu, 2 x 55 = 110 > 66 on | |
| # Windows). The caps were sized against the OBSERVED sum instead, which is a weaker guarantee and | |
| # was recorded here as such. The web console step now carries `webconsole_step_timeout`, so the | |
| # worst case the caps admit is finite and positive on every leg -- the arithmetic is in the note | |
| # above that step, and it is the first time this file has been able to state it. | |
| # Observed for real on run 30724385719 (main @ 8f01cef8, 2026-08-01): | |
| # | |
| # Tests (pytest) 00:01:21 -> 00:27:12 25:51 SUCCESS (9s under the 26:00 cap) | |
| # Web console tests (pytest) 00:27:12 -> 00:30:14 CANCELLED | |
| # JOB 00:00:06 -> 00:30:19 30:13 CANCELLED <- job_timeout 30 fired | |
| # | |
| # A GREEN first step, then an unattributed job-level kill during the second -- exactly the failure | |
| # the nesting note above exists to prevent, arriving by the path that note does not consider. It | |
| # does NOT happen when a step is killed (that ends the job and skips what follows, so those | |
| # durations never sum); it happens when the first step PASSES near its budget. | |
| # | |
| # READ THAT EXHIBIT FOR ITS MECHANISM, NOT AS A VERDICT ON THE CAP IT REPLACED: it ran under the | |
| # retired 26/30 pair and would have passed under #131's 40:00. What condemns 40:00 is the | |
| # arithmetic below -- which is arithmetic, not an observed kill; no job has yet hit 40:00. | |
| # | |
| # job_timeout is therefore sized against BOTH gated steps plus setup, rather than as step_timeout | |
| # plus a constant. Each addend below is that addend's MEASURED MAXIMUM over the same pool, taken | |
| # over rows where `Tests` concluded success -- NOT over rows where both steps did, which silently | |
| # drops the very run that motivates this whole paragraph (its web-console step was cancelled) and | |
| # is the same censoring mistake as filtering the step table by job conclusion. Two earlier | |
| # revisions used a 0:41 setup (the MEDIAN), web-console values that were each only third-highest | |
| # on their leg, and a 1:04 Windows setup that the exhibit above contradicts on its face: | |
| # | |
| # THE BAND MOVED OFF step_timeout AND ONTO THE OVERHEAD, 2026-08-08 (BACKLOG #1096). The earlier | |
| # formula multiplied the WHOLE sum (step_timeout + addends) by ~1.13-1.16. That applies a safety | |
| # band to `step_timeout`, which is a HARD BOUND the runner enforces -- the largest step overrun in | |
| # 1,618 measured executions is EIGHT SECONDS (36:08 against a 36:00 cap). Banding it is slack for an | |
| # event that cannot happen, it grows with the cap (4:41 at 36:00, 7:09 at 55:00), and it makes the | |
| # published ratio pool-dependent because the step term dominates it. Corrected rule: | |
| # | |
| # job_timeout = step_timeout + ceil_minute(1.5 x that leg's worst measured job OVERHEAD) | |
| # | |
| # where overhead = job wall clock minus the `Tests (pytest)` step, taken as the larger of (a) the | |
| # worst SAME-ROW overhead on non-cancelled rows and (b) that leg's setup(max) + web-console(max). | |
| # The 1.5 band is on the overhead only, and is 1.5 rather than 1.35 because setup is hosted-runner | |
| # provisioning outside this repo's control: `Set up job` alone has been observed at 4:30 (ubuntu run | |
| # 31109989006, inside a 5:22 setup). | |
| # | |
| # leg worst same-row setup(max)+wc(max) overhead used x1.5 room job | |
| # ubuntu 7:12 (n=451) 5:22 + 2:22 = 7:44 7:44 11:36 12:00 31+12 = 43 | |
| # W22 6:16 (n=413) 4:05 + 2:51 = 6:56 6:56 10:24 11:00 55+11 = 66 | |
| # W25 6:29 (n=372) 2:32 + 3:59 = 6:31 6:31 9:47 10:00 (takes W22's 66) | |
| # | |
| # INSTRUMENT PRESERVATION, the whole point of the pairing -- worst-case job wall built from | |
| # independent maxima with `Tests` passing one second under its cap, new values against old: | |
| # | |
| # ubuntu 5:22 + 30:59 + 2:22 = 38:43 vs 43:00 +4:17 | old: 32:43 vs 37:00 +4:17 (band held) | |
| # W22 4:05 + 54:59 + 2:51 = 61:55 vs 66:00 +4:05 | old: 42:55 vs 46:00 +3:04 | |
| # W25 2:32 + 54:59 + 3:59 = 61:30 vs 66:00 +4:30 | old: 42:30 vs 46:00 +3:30 | |
| # | |
| # UBUNTU WAS THE LEG ACTUALLY BROKEN, at -0:43, and nobody had measured it: BACKLOG #1096 was filed | |
| # as a Windows problem. A negative row here means the JOB cap can fire before the STEP cap, and a | |
| # job-level kill reports NO step conclusion -- so the instrument this item was measured with is | |
| # destroyed exactly when it is needed. Max measured post-step teardown is 0:24, so all three new | |
| # rows stay positive with room. | |
| # | |
| # THE THIRD ADDEND ABOVE IS THE OBSERVED web-console maximum, which is a DIFFERENT QUANTITY from | |
| # the worst case the caps now admit -- one describes what has happened, the other what is | |
| # permitted. Capping that step (BACKLOG #344 proposal 5) does not change how long it takes, so the | |
| # rows above stand and `job_timeout` is UNCHANGED by it. The cap-based worst case is stated once, | |
| # in the note above the `Web console tests (pytest)` step, and it is positive on every leg. | |
| # | |
| # NESTING INVARIANT, first gated step -- setup(max) + step_timeout must stay under job_timeout, so a | |
| # `Tests` kill is a NAMED STEP failure rather than an unattributed job cancellation: | |
| # ubuntu 5:22 + 25:00 = 30:22 < 37:00 (gap 6:38) | |
| # W22 4:05 + 55:00 = 59:05 < 66:00 (gap 6:55) | |
| # W25 2:32 + 55:00 = 57:32 < 66:00 (gap 8:28) | |
| # | |
| # 66 IS DERIVED, NOT ROUNDED UP FROM 55. The bare minimum whole minute that clears the worst sum is | |
| # 61:00; it is rejected as thinner than any row this file has ever carried, and thinner than the | |
| # three-addend model's own known error -- the worst observed single row sums to 40:16 against a real | |
| # job wall clock of 40:22, so inter-step gaps make the model an UNDER-estimate by ~6s, not a | |
| # conservative one. | |
| # | |
| # SUPERSEDED, kept because a number without its pool cannot be rechecked. The previous table read: | |
| # ubuntu 19:00 + 2:00 + 1:20 = 22:20 (26:00, +3:40) | W22 36:00 + 2:33 + 1:09 = 39:42 (46:00, | |
| # +6:18) | W25 36:00 + 3:33 + 1:20 = 40:53 (46:00, +5:07). Correct for its pool; superseded above. | |
| # | |
| # The `+4` habit that produced 40 was Windows-only | |
| # and was carried through two cap changes unchecked (30/26 then 40/36); ubuntu has never been +4 -- | |
| # it went 15/13 to 22/19, so +2 then +3. Either way the number was derived from the OTHER BOUND | |
| # rather than from the work, which is the defect, not the particular constant. | |
| # | |
| # WHAT THIS SIZING DID NOT FIX, AND WHAT SINCE FIXED IT. This sizing only made the job cap cover | |
| # the OBSERVED sum. The nesting invariant held for `Tests (pytest)` on every leg and for | |
| # `Web console tests (pytest)` on NONE of them, because both steps drew on one `step_timeout`: | |
| # guaranteeing the second would have needed job_timeout above setup + 2 x step_timeout -- about | |
| # 55:22 on ubuntu and 114:05 on Windows -- so a hang in the web console step surfaced as an | |
| # unattributed job-level kill, the very failure the invariant is there to prevent. BACKLOG #344 | |
| # proposal 5 closed it the other way round: the web console step now has `webconsole_step_timeout` | |
| # sized to its own work (a 2-to-4-minute suite has no business holding 55:00), which brings | |
| # setup + step_timeout + webconsole_step_timeout under job_timeout on all three legs. The | |
| # arithmetic, and the trigger for re-deriving it, live in the note above that step. | |
| # | |
| # AND KNOW WHAT IS NOT CAPPED AT ALL. `test` is the ONLY one of this file's ten jobs carrying any | |
| # `timeout-minutes`; the other nine -- `changes`, `ide`, `sqlserver-store`, `postgres-store`, | |
| # `load-test`, `load-test-sqlserver`, `windows-service-smoke`, `docker-smoke`, `ci-gate` -- run on | |
| # GitHub's 6-HOUR default. That is recorded here so the omission is known rather than assumed | |
| # deliberate; sizing them is out of scope for this change and belongs with BACKLOG #344. | |
| # | |
| # This cap is NOT what catches a hung test; pytest_timeout (120s) is, per test. The step cap only | |
| # catches a whole-process deadlock both in-process watchdogs miss, which is why it can sit well | |
| # above a healthy run. Sizing it tight buys no detection and costs false failures on green suites, | |
| # twice now. | |
| # | |
| # The remaining margin is a SHARED budget across every PR that lands, and it is now ACCOUNTED | |
| # FOR: `scripts/ci/step_margin.py` runs after the gated step below and reds the leg when a | |
| # step's own duration comes within 1.30x of its own cap (BACKLOG #344 proposal 1). It used to run | |
| # after BOTH gated steps; the web console suite moved to its own `webconsole` job and took its | |
| # half of the check with it, so each job now measures the one gated step it owns. Three PRs | |
| # each adding a minute of Windows time is still individually blameless -- the difference is that | |
| # the third one now says so instead of the fourth one dying at the cap with zero failing | |
| # assertions. It is NOT a request to raise a cap; the underlying slowness is #320. | |
| # | |
| # THE CLOCK IS TAKEN IN ITS OWN STEP so neither gated `run:` body has to carry timing code that | |
| # a kill would skip anyway. Elapsed is therefore the step plus its two transitions -- an UPPER | |
| # bound on the step, so the reported margin is a LOWER bound on the true margin, and the error | |
| # runs toward firing early rather than toward a false green. | |
| # | |
| # It marks THROUGH THE SCRIPT rather than `echo ... >> "$GITHUB_ENV"`, deliberately. That | |
| # spelling routes a Windows path through a bash redirection on two of the three legs -- a | |
| # construct no local run can exercise, whose failure mode is a silently absent variable, i.e. a | |
| # check that stops measuring while still printing a verdict. Marking in Python keeps the path | |
| # handling where the tests reach it, and a missing mark is a REFUSAL (exit 2), never a zero. | |
| - name: Step margin -- start the clock | |
| if: always() | |
| run: python scripts/ci/step_margin.py --mark before-tests | |
| - name: Tests (pytest) | |
| id: tests | |
| if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch' | |
| timeout-minutes: ${{ matrix.step_timeout }} | |
| env: | |
| QT_QPA_PLATFORM: offscreen | |
| PYTHONFAULTHANDLER: "1" | |
| # Pass the matrix-derived timeout knobs through env instead of expanding ${{ }} straight into | |
| # the run: shell. The per-repo matrix is now built at runtime in `changes` (fromJSON), so zizmor | |
| # can no longer statically prove these are literal ints and flags the in-run expansion as | |
| # template-injection; env-passing removes the script sink. `defaults.run.shell: bash` applies on | |
| # every OS, so $VAR resolves the same on the Windows legs (Git Bash). | |
| FAULT_TIMEOUT: ${{ matrix.fault_timeout }} | |
| PYTEST_TIMEOUT: ${{ matrix.pytest_timeout }} | |
| PYTEST_WORKERS: ${{ matrix.pytest_workers }} | |
| # BACKLOG #1260: empty on purpose -- see the comment above the run line. The wrapper's | |
| # default clause names the pyodbc class, which is not established for this leg. | |
| RETRY_NATIVE_CRASH_CAUSE: "" | |
| # `--ignore-glob` subtracts the web console package, which runs as its OWN step below. BACKLOG | |
| # #1027 added that package to the root `testpaths` so a bare local `pytest` stops silently | |
| # excluding it; without this subtraction the SAME 356 tests would then run TWICE on every leg. | |
| # Measured 2026-08-08 on this branch: bare collection 11,956, console-only 356, with this flag | |
| # 11,600. | |
| # NOT `--ignore=<path>` -- measured, it does NOT prune a directory that `testpaths` names as a | |
| # collection root, and still collected all 11,956. The obvious spelling is the one that fails. | |
| # NOT `pytest tests` either: hardcoding the engine path would make CI silently miss any future | |
| # third entry in `testpaths`, which is the exact drift #1027 exists to prevent. Subtracting FROM | |
| # `testpaths` keeps it the single source of truth for what the suite is. | |
| # `tests/test_ci_engine_step_excludes_webconsole.py` pins all three of those decisions. | |
| # | |
| # `-n` RUNS THE SUITE ACROSS PROCESSES (pytest-xdist), and it is the whole reason this step | |
| # stopped being 40 minutes. Measured 2026-08-16 on this branch: serial 2191s vs `-n 4 | |
| # --dist loadfile` 511s locally, with the failure set IDENTICAL in both runs (the same six | |
| # pre-existing environment failures, zero parallelism-induced). That box has 20 cores and a | |
| # hosted runner has 4, so the CI figure is expected to be lower than that 4.29x -- re-read | |
| # the step duration from the first runs rather than quoting the local number. | |
| # | |
| # `--dist loadfile` IS LOAD-BEARING, NOT A PREFERENCE. xdist's default is `--dist load`, which | |
| # scatters individual tests from one file across workers, and this suite has two things that | |
| # breaks. (1) 44 module-scoped fixtures live outside conftest and would be rebuilt once PER | |
| # WORKER instead of once -- `tests/test_worktree_prune_merged.py` alone pays ~3s of setup per | |
| # test. (2) `tests/test_multishard_smoke.py` is the sole consumer of MEFOR_TEST_PORT_BASE and | |
| # binds a real port window; keeping a file's tests together keeps that window inside one | |
| # process. The cost of `loadfile` is that the run cannot finish faster than its single largest | |
| # FILE, which is why the per-file hot spots are worth attacking next. | |
| # `-m 'not tooling'` DESELECTS the repo-harness tier, which the `tooling` job runs instead. | |
| # Measured 2026-08-16 on a 20-core box: 1,882 of 12,990 tests carry the mark but they are 66 | |
| # percent of the suite's TIME, and the single largest file in the tier (test_worktree_prune_ | |
| # merged.py, 391.4s) was setting this step's `--dist loadfile` floor on every engine leg. | |
| # | |
| # Deselection is by MARKER, applied in tests/conftest.py from tests/tooling_manifest.txt. Not by | |
| # path, and not by a second `--ignore-glob`: the webconsole exclusion right above already shows | |
| # what happens when the selection lives in the run body -- the obvious spelling (`--ignore`) was | |
| # the one that silently did nothing. One list, read in one place, asserted by | |
| # tests/test_tooling_partition.py, which also pins the two halves of this wiring. | |
| # BACKLOG #1260: WRAPPED SO A NATIVE CRASH IS NOT REPORTED AS A TEST FAILURE. A segfault kills | |
| # the interpreter, so pytest returns 139/134 with no verdict and THREE layers of naming then | |
| # say "tests failed" -- the check name, the step name, and `steps.tests.outcome`, which is what | |
| # `step_margin.py` below consumes. None of them is true: the engine was fine and a process died. | |
| # The wrapper re-runs ONLY on 139/134 and re-raises exit 1 immediately, so it cannot mask a | |
| # regression. | |
| # | |
| # RETRY_NATIVE_CRASH_CAUSE IS DELIBERATELY EMPTY HERE. The wrapper's default clause names the | |
| # pyodbc py3.14 class, which is ESTABLISHED for the database legs and is NOT established for | |
| # this one -- the item filing this refuses to conclude it. An empty value makes the annotation | |
| # say CAUSE NOT ESTABLISHED rather than assert a mechanism nobody has measured on this leg. | |
| run: bash scripts/ci/retry-native-crash.sh pytest -q -n "$PYTEST_WORKERS" --dist loadfile -m 'not tooling' --ignore-glob='*messagefoundry-webconsole*' -o faulthandler_timeout="$FAULT_TIMEOUT" --timeout="$PYTEST_TIMEOUT" | |
| # THE MARGIN CHECK (BACKLOG #344 proposal 1). Last in the job so a LOW margin cannot skip a suite | |
| # that has not run yet -- a step `if:` with no status function carries an implicit `success()`, | |
| # so a failing check placed ahead of a gated step would silently skip it. | |
| # | |
| # IT NOW GUARDS ONE STEP, NOT TWO. The web console suite moved to its own `webconsole` job and | |
| # took its half of this check with it, so the `between` mark is gone: this reads | |
| # `before-tests .. now`, which is the whole of the only gated step left in this job. The | |
| # invariant did not weaken -- it was RE-SCOPED. Each job now holds exactly one gated step, its | |
| # own cap, and its own check, which is a stronger shape than one check straddling two steps | |
| # whose budgets had to be summed under a single job cap. | |
| # | |
| # It keys on `steps.<id>.outcome` -- THE STEP'S OWN CONCLUSION, never the job's. That substitution | |
| # is not pedantry: a step that nearly exhausts its cap is the most likely to push its job into | |
| # `job_timeout`, so filtering on the job deletes the tightest rows by construction, and doing | |
| # exactly that reproduced a published maximum of 24:35 where the truth was 25:51. | |
| # | |
| # A skipped step (docs-only PR) reports NO OBSERVATION in words rather than a healthy-looking | |
| # ratio, and the script runs its own red/green control pair on every invocation and prints both | |
| # into the job summary -- a gate that has never been red is a claim, not a control. | |
| - name: Step margin -- engine suite | |
| if: always() | |
| env: | |
| TESTS_OUTCOME: ${{ steps.tests.outcome }} | |
| STEP_CAP: ${{ matrix.step_timeout }} | |
| LEG: ${{ matrix.os }} | |
| run: | | |
| rc=0 | |
| # `|| rc=$?` OUTSIDE the command, so a non-zero exit is captured rather than swallowed by | |
| # `set -e` or absorbed into a pipeline. | |
| python scripts/ci/step_margin.py --step "Tests (pytest)" --leg "$LEG" \ | |
| --cap-minutes "$STEP_CAP" --outcome "$TESTS_OUTCOME" \ | |
| --since before-tests --until now || rc=$? | |
| exit $rc | |
| # THE WEB CONSOLE SUITE, IN ITS OWN PARALLEL JOB (was a second step on every `test` leg). | |
| # | |
| # WHY IT MOVED. It was ~170s sitting SERIALLY behind a ~1090s engine suite on the critical-path leg, | |
| # for no reason other than sharing a checkout. Run alongside instead of after, it costs zero | |
| # wall-clock: its whole job (setup + suite) finishes far inside the engine job it now runs beside. | |
| # | |
| # THIS REVERSES THE PLACEMENT HALF OF BACKLOG #344 PROPOSAL 5, AND KEEPS ITS SUBSTANCE. That proposal | |
| # gave this suite its own `webconsole_step_timeout` because two steps sharing one `timeout-minutes` | |
| # meant the second one's cap could never fire first -- reaching it had already spent setup plus the | |
| # engine suite. Moving it to its own job does not undo that; it completes it. The suite keeps its own | |
| # STEP cap, gains its own JOB cap (`webconsole_job_timeout`), and keeps its own margin check. The | |
| # nesting invariant is now setup(max) + webconsole_step_timeout < webconsole_job_timeout in ONE job, | |
| # instead of a sum of two step budgets straddling a single job cap -- strictly easier to satisfy and | |
| # to check, which is why the guard in tests/test_ci_step_margin.py is re-scoped per job rather than | |
| # relaxed. | |
| # | |
| # IT IS NOT A REQUIRED CONTEXT AND DOES NOT NEED TO BE. `ci-gate` lists it in `needs:` and fails on | |
| # any needed job's `failure` or `cancelled`, and `CI gate` IS required -- the same mechanism that | |
| # already gates the six path-gated legs. Adding a new required context would instead have meant a | |
| # branch-protection change and the required-but-absent trap on every docs-only PR. | |
| # | |
| # THE DOCS-ONLY GATE IS PER-STEP, NOT PER-JOB, exactly as in `test` and for the same reason: a | |
| # job-level `if:` would skip the job outright, and `ci-gate` reads `needs.*.result`, where a skipped | |
| # job reports `skipped` rather than `success`. Keeping the job running and no-oping its steps keeps | |
| # that roll-up honest on a docs-only PR. | |
| webconsole: | |
| name: web console tests (${{ matrix.os }}, py${{ matrix.python-version }}) | |
| needs: changes | |
| strategy: | |
| fail-fast: false | |
| # Same per-repo matrix as `test`: this suite is as platform-sensitive as the engine one (it runs | |
| # under the package's OWN pytest config, against a real ASGI test client), so it gets the same | |
| # legs rather than a cheaper ubuntu-only run. | |
| matrix: ${{ fromJSON(needs.changes.outputs.matrix) }} | |
| runs-on: ${{ matrix.hosted }} | |
| timeout-minutes: ${{ matrix.webconsole_job_timeout }} | |
| defaults: | |
| run: | |
| shell: bash | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Set up Python ${{ matrix.python-version }} | |
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: ${{ matrix.python-version }} | |
| - name: Install Qt offscreen system libraries | |
| if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux' | |
| # BOUNDED AND RETRIED. Unguarded, this step turns an external apt-mirror hiccup into a | |
| # blocked merge queue: measured 2026-08-18, three hangs across two attempts on three | |
| # different jobs, one of them 27+ minutes on `test (ubuntu-latest, py3.14)` -- a REQUIRED | |
| # context, so the queue stops. Two levers, because they answer different failures. The | |
| # per-command `timeout` kills a HANG and lets the retry run; `timeout-minutes` is the | |
| # backstop that stops this step ever eating a job budget again if the loop is edited wrong. | |
| # The failure text names the cause, so the next reader is not sent hunting in their diff -- | |
| # a check whose label points at the wrong subject is what BACKLOG #1254 is about. | |
| timeout-minutes: 8 | |
| run: | | |
| for attempt in 1 2 3; do | |
| if sudo timeout 120 apt-get update && sudo timeout 180 apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3; then | |
| exit 0 | |
| fi | |
| echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying" | |
| sleep $((attempt * 5)) | |
| done | |
| echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test." | |
| exit 1 | |
| - name: Set up uv | |
| if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch' | |
| uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| requirements.lock | |
| # THE SAME INSTALL LINE AS `test`, deliberately. The console suite imports the engine, so it must | |
| # be exercised against the same build; keeping the commands identical (rather than trimming to a | |
| # console-only set) means the two jobs cannot drift into testing different resolutions of the same | |
| # lock. It also shares the uv cache entry -- same key, already warm from the sibling job. | |
| - name: Install project (dev + console + fhir + dicom + x12 + xml extras) | |
| if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch' | |
| run: | | |
| uv pip install --system --constraint constraints.lock -e ".[dev,harness,fhir,dicom,x12,xml,webauthn]" -e packaging/messagefoundry-webconsole | |
| - name: Step margin -- start the clock | |
| if: always() | |
| run: python scripts/ci/step_margin.py --mark before-webconsole | |
| # The web console's OWN suite (Option B, ADR 0065): the moved /ui tests live in the package's | |
| # tests/. They ARE in the root `testpaths` since BACKLOG #1027, so a bare local `pytest` collects | |
| # them -- the engine step in `test` subtracts them explicitly via `--ignore-glob` rather than | |
| # relying on `testpaths` to exclude them, which it no longer does. The package is installed | |
| # editable above, so both suites exercise the same engine source. | |
| # | |
| # NO `-n` HERE, deliberately. xdist buys the engine suite ~2x across ~13,000 tests; this is ~356 | |
| # tests in 2-to-4 minutes, where worker startup would eat most of the gain, and the job now runs | |
| # concurrently with the engine one anyway -- its wall-clock is already off the critical path. | |
| # Adding workers would spend the contention risk the -n 8 trial measured to buy nothing. | |
| - name: Web console tests (pytest) | |
| id: webconsole | |
| if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch' | |
| timeout-minutes: ${{ matrix.webconsole_step_timeout }} | |
| env: | |
| QT_QPA_PLATFORM: offscreen | |
| PYTHONFAULTHANDLER: "1" | |
| # env-passed (not a ${{ }} expansion into run:) -- see the engine Tests step for why. | |
| FAULT_TIMEOUT: ${{ matrix.fault_timeout }} | |
| PYTEST_TIMEOUT: ${{ matrix.pytest_timeout }} | |
| run: pytest packaging/messagefoundry-webconsole/tests -q -o faulthandler_timeout="$FAULT_TIMEOUT" --timeout="$PYTEST_TIMEOUT" | |
| # This job's half of the margin check (BACKLOG #344 proposal 1), moved here with the suite it | |
| # measures. Same contract as the engine one: last in the job, `if: always()` so a step killed at | |
| # its cap is still reported, and keyed on `steps.webconsole.outcome` -- the STEP's own conclusion, | |
| # never the job's, because a step that nearly exhausts its cap is the most likely to push its job | |
| # into `timeout-minutes` and filtering on the job would delete exactly those rows. | |
| - name: Step margin -- web console suite | |
| if: always() | |
| env: | |
| WEBCONSOLE_OUTCOME: ${{ steps.webconsole.outcome }} | |
| WEBCONSOLE_CAP: ${{ matrix.webconsole_step_timeout }} | |
| LEG: ${{ matrix.os }} | |
| run: | | |
| rc=0 | |
| python scripts/ci/step_margin.py --step "Web console tests (pytest)" --leg "$LEG" \ | |
| --cap-minutes "$WEBCONSOLE_CAP" --outcome "$WEBCONSOLE_OUTCOME" \ | |
| --since before-webconsole --until now || rc=$? | |
| exit $rc | |
| # The repo-HARNESS tier, lifted off the engine legs' critical path. Membership is | |
| # tests/tooling_manifest.txt; the `test` legs deselect it with `-m 'not tooling'` and this job is the | |
| # only place it runs. Measured 2026-08-16: 94 files, 1,882 tests, 66 percent of the suite's time. | |
| # | |
| # UBUNTU + WINDOWS, and the ubuntu-only version of this job was WRONG -- recorded because the | |
| # reasoning that produced it was superficially sound. The first cut argued the tier is mostly | |
| # OS-agnostic shell/git behaviour and that Windows is where its child-spawn cost hurts most. Measured | |
| # instead of argued: 18 of the 89 manifest entries carry a module-level skipif on os.name/sys.platform | |
| # and they hold 337 of the tier's 1,882 tests -- 18 percent. Those ran on windows-2022 and | |
| # windows-2025 before the split; under ubuntu-only they would have executed on NO leg, on any event. | |
| # They are the coordination and worktree-gate core: session_mail (89), prune_merged (71), | |
| # announce_hook (45), collision_gate (31), every coord_*. | |
| # | |
| # The sizing number in the first cut was invalid on its own terms too: the 391.4s attributed to | |
| # test_worktree_prune_merged.py is a WINDOWS figure for a file that has always skipped on ubuntu, so | |
| # it could never have been that leg's `--dist loadfile` floor. | |
| # | |
| # windows-2025 only, not both Windows legs. The engine matrix carries 2022 and 2025 because the | |
| # ENGINE ships onto both; this tier tests scripts/ against the local git and pwsh, where the | |
| # 2022-vs-2025 difference is not a surface any listed test asserts on. That is a deliberate reduction | |
| # from the pre-split state and the honest cost of the split -- if a harness regression ever lands that | |
| # only windows-2022 would have caught, THIS is the sentence that was wrong. | |
| # | |
| # NOT a required context of its own: it rides the `ci-gate` roll-up, exactly as `webconsole` does and | |
| # for the same reason recorded there -- a new required context needs a branch-protection change, and | |
| # until that lands every PR carries a check it can never satisfy. | |
| tooling: | |
| name: repo harness tests (${{ matrix.os }}) | |
| needs: changes | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| os: [ubuntu-latest, windows-2025] | |
| # The push arm is the safety net, not redundancy: the PR arm only fires when the harness itself | |
| # changed, so an ENGINE change that breaks a harness test is invisible on its own PR by | |
| # construction, and push-to-main is where it surfaces. See the `tooling` output on `changes`. | |
| if: needs.changes.outputs.tooling == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch' | |
| runs-on: ${{ matrix.os }} | |
| # SIZED FOR THE WINDOWS LEG, NOT THE UBUNTU ONE, and the first cut (25 job / 18 step) was too | |
| # tight to survive it. Derivation, stated so it can be re-derived rather than trusted: the tier | |
| # runs 431s at -n 4 on a 20-PHYSICAL-core box, with the pwsh/git children this tier spawns landing | |
| # on spare cores. A hosted runner has 4 vCPU and no spare -- ci.yml already records ~1.12 cores of | |
| # average child occupancy for exactly this work, and the -n 8 trial documented above shows what | |
| # oversubscription does here. Windows adds its ~2.4x process-spawn tax on top, and the Windows leg | |
| # is the ONLY leg that executes the 337 platform-gated tests at all. A 2.5-3x factor on 431s is | |
| # 18-21 minutes, which the old 18-minute step cap sat directly on top of. | |
| # | |
| # These are DEADLOCK BACKSTOPS, not tightness gates -- the same reasoning the engine matrix records | |
| # for its own 66/55. Sizing a cap close to the observed run buys no detection and costs red builds | |
| # on green suites, and a gate that cries wolf gets deleted. The engine Windows legs carry 66/55 for | |
| # a bigger suite; 40/30 here is proportionate and still well clear. | |
| # | |
| # RE-DERIVE IF the tier's -n 4 wall clock is ever observed above 25 minutes on the Windows leg, or | |
| # if pytest_workers changes. | |
| timeout-minutes: 40 | |
| defaults: | |
| run: | |
| shell: bash | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| fetch-depth: 0 # several of these tests read git history (ledger, claim, cutover guards) | |
| - name: Set up Python 3.14 | |
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.14" | |
| # NEEDED EVEN THOUGH NO HARNESS TEST TOUCHES Qt, because `-m tooling` FILTERS AFTER COLLECTION. | |
| # pytest imports every module under `testpaths` to collect it and only then applies the marker | |
| # expression, so this job still imports the four PySide6 modules (test_harness_compose, | |
| # test_harness_file, test_harness_monitor, test_console_messages_refresh) and dies on | |
| # `ImportError: libEGL.so.1` before a single tooling test runs. Measured on this job's first real | |
| # CI run: 1494 passed, 388 skipped, 4 COLLECTION ERRORS -- deselection does not mean not-imported. | |
| # | |
| # That is also why the install line below cannot be trimmed to a harness-only set: collection | |
| # imports the whole suite regardless of what this job intends to execute. | |
| - name: Install Qt offscreen system libraries | |
| if: runner.os == 'Linux' | |
| # BOUNDED AND RETRIED. Unguarded, this step turns an external apt-mirror hiccup into a | |
| # blocked merge queue: measured 2026-08-18, three hangs across two attempts on three | |
| # different jobs, one of them 27+ minutes on `test (ubuntu-latest, py3.14)` -- a REQUIRED | |
| # context, so the queue stops. Two levers, because they answer different failures. The | |
| # per-command `timeout` kills a HANG and lets the retry run; `timeout-minutes` is the | |
| # backstop that stops this step ever eating a job budget again if the loop is edited wrong. | |
| # The failure text names the cause, so the next reader is not sent hunting in their diff -- | |
| # a check whose label points at the wrong subject is what BACKLOG #1254 is about. | |
| timeout-minutes: 8 | |
| run: | | |
| for attempt in 1 2 3; do | |
| if sudo timeout 120 apt-get update && sudo timeout 180 apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3; then | |
| exit 0 | |
| fi | |
| echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying" | |
| sleep $((attempt * 5)) | |
| done | |
| echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test." | |
| exit 1 | |
| - name: Set up uv | |
| uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| requirements.lock | |
| # THE SAME INSTALL LINE AS `test`, and it cannot be trimmed even though no listed test imports the | |
| # engine: tests/conftest.py itself does (`from messagefoundry.config.settings import ...`), so the | |
| # package must be importable for collection to happen at all. Keeping the line identical also | |
| # shares the uv cache entry with the sibling jobs. | |
| - name: Install project (dev + console + fhir + dicom + x12 + xml extras) | |
| run: | | |
| uv pip install --system --constraint constraints.lock -e ".[dev,harness,fhir,dicom,x12,xml,webauthn]" -e packaging/messagefoundry-webconsole | |
| # `-m tooling` SELECTS what the engine legs deselected. The two spellings are a pair and | |
| # tests/test_tooling_partition.py asserts BOTH are present in this file, because either one alone | |
| # is a silent half-failure: without the engine-side `not tooling` the split buys nothing, and | |
| # without this the tier stops running anywhere while both jobs still report green. | |
| # | |
| # `--dist loadfile` for the same reason as the engine step: these tests bind real port windows and | |
| # own module-scoped fixtures, so a file's tests must stay in one worker. | |
| - name: Harness tests (pytest) | |
| timeout-minutes: 30 # see the job cap above for the derivation; must stay under it with setup | |
| env: | |
| # Same reason as the apt step above: collection imports the Qt modules whether or not this | |
| # job executes them, and the shipped ones construct QApplication at import on some paths. | |
| QT_QPA_PLATFORM: offscreen | |
| run: pytest -q -n 4 --dist loadfile -m tooling --ignore-glob='*messagefoundry-webconsole*' --timeout=120 --junitxml=tooling-junit.xml | |
| # A marker typo, a manifest rename, or a conftest hook that silently stops firing all produce the | |
| # same thing: zero selected tests and a GREEN job. Deselection cannot be distinguished from success | |
| # by exit code -- pytest exits 5 on "no tests collected", which `-q` reports in one line nobody | |
| # reads. | |
| # | |
| # COUNT WHAT EXECUTED, NOT WHAT WAS COLLECTED. The first version of this step grepped | |
| # `--collect-only`, and pytest COLLECTS skipif-marked tests -- verified against a synthetic file | |
| # whose every test was skipped: "2 tests collected". So it would have printed a healthy ~1,900 and | |
| # passed while 337 Windows-gated tests skipped on an ubuntu-only job. A receipt that cannot tell | |
| # green-because-it-ran from green-because-it-skipped is not a receipt. This parses the junit the | |
| # run itself produced, so the number is the number that happened. | |
| # | |
| # The floor is deliberately loose (1,000 against ~1,545 executing on ubuntu and ~1,882 on | |
| # Windows). It is a DEAD-MARKER backstop, not a coverage assertion: a tight floor would go red | |
| # every time the tier legitimately grows or a platform gate moves, and a gate that cries wolf gets | |
| # deleted. Coverage drift is tests/test_tooling_partition.py's job, on evidence, not this one's. | |
| # A SCRIPT, NOT AN INLINE HEREDOC. The logic and its reasoning live in the script, where they can | |
| # be read, linted, type-checked and run locally -- and where a shell that handles heredocs | |
| # differently cannot change the answer. This job is the first in this workflow to run a `run:` | |
| # body on BOTH ubuntu and Windows, so an inline heredoc here would have been the first of its | |
| # kind on the Windows leg; that is not a thing to discover from a red build. | |
| - name: Prove the tier actually ran | |
| if: always() # a timeout or a crash is exactly when the receipt matters most | |
| run: python scripts/ci/tooling_receipt.py --report tooling-junit.xml | |
| # Build + type-check the VS Code extension, and run its integration tests. The Python jobs never | |
| # touch ide/, so a dep bump or IDE code change that breaks the bundle or the types would otherwise | |
| # pass CI unbuilt (this job exists because an esbuild bump merged "green" without anything having | |
| # compiled it). The build runs on both OSes; the @vscode/test-electron suite (which downloads and | |
| # launches a real headless VS Code to assert the extension activates and its commands register/run) | |
| # runs on the Windows leg — Electron launches headless there with no xvfb, and Windows is a | |
| # deployment target. | |
| ide: | |
| name: ide build (${{ matrix.os }}) | |
| # `ide` is NOT a required check and ci-gate does NOT `needs: ide`, so skipping it — or varying its | |
| # matrix per repo (below) — can't wedge any required context. It runs when it can be affected: | |
| # push-to-main + workflow_dispatch (full coverage), and PRs that touch ide/** or this workflow (the | |
| # `changes.ide` path-gate) — a pure-Python PR can't change the isolated npm project's result, so it's | |
| # skipped there. Not on the nightly cron: the engine heavy legs need that safety net, an isolated npm | |
| # build gated on its own paths does not. The per-repo os matrix below runs the 2x-billed | |
| # windows-latest electron leg here (this repo is public, so hosted runners are free) and drops it on | |
| # a fork, where those minutes come out of the fork owner's own allowance. | |
| needs: changes | |
| if: github.event_name == 'workflow_dispatch' || needs.changes.outputs.ide == 'true' | |
| runs-on: ${{ matrix.os }} | |
| strategy: | |
| fail-fast: false | |
| # Per-repo os matrix (built in `changes`): ubuntu + windows-latest on THIS repo, ubuntu-only on a | |
| # fork. The windows-latest leg is the 2x-billed one that runs the @vscode/test-electron integration | |
| # suite; this repo is public so hosted minutes are free, while on a fork they are billed to the | |
| # fork owner. `ide` is not a required check and ci-gate does not `needs` it, so dropping the | |
| # windows leg on a fork can't wedge anything. | |
| matrix: ${{ fromJSON(needs.changes.outputs.ide_matrix) }} | |
| defaults: | |
| run: | |
| working-directory: ide | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Set up Node | |
| uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 | |
| with: | |
| node-version: "24" | |
| cache: npm | |
| cache-dependency-path: ide/package-lock.json | |
| - name: Install (clean, from lockfile) | |
| run: npm ci | |
| - name: Type-check (tsc --noEmit) | |
| run: npm run typecheck | |
| - name: Bundle (esbuild) | |
| run: npm run compile | |
| # The vscode-free suites (the pure model layer: the engine link state + its two frozen boundary | |
| # allowlists, the settings-scope SEC-005 invariant, the graph/steps/HL7 models, …). They need no | |
| # Extension Host, so they run on EVERY leg — including the ubuntu-only leg a fork gets. | |
| # Before this step they ran NOWHERE on an ubuntu-only matrix: `npm test` is Windows-only, so the | |
| # entire node-side estate was type-checked and never executed there. | |
| # ADR 0110's allowlists are only "asserted in CI" because of this line. | |
| - name: Unit tests (node-side, no VS Code) | |
| run: npm run test:unit | |
| - name: Integration tests (headless VS Code) | |
| if: runner.os == 'Windows' | |
| run: npm test | |
| # Path filter for the container-pulling server-DB leg(s). A docs/unrelated PR shouldn't pull the SQL | |
| # Server image (mcr.microsoft.com anonymous pulls are rate-limited — repeated PRs flake on | |
| # "toomanyrequests"). This job decides whether the SQL Server leg needs to run: ALWAYS true on | |
| # workflow_dispatch (full coverage); FALSE on push-to-main (the container legs run nightly + | |
| # PR-path-gated, not per merge); on a PR, true only when store / cluster / server-DB tests / | |
| # this workflow changed. (NB: if `sqlserver-store` is later made a *required* check in branch | |
| # protection, a path-skipped run reports "skipped" — use a ruleset that treats skips as success, or | |
| # switch to an always-run job whose heavy steps are step-gated, since the service container starts | |
| # before steps.) | |
| changes: | |
| name: detect server-DB + docker changes | |
| runs-on: ubuntu-latest | |
| outputs: | |
| serverdb: ${{ steps.f.outputs.serverdb }} | |
| docker: ${{ steps.f.outputs.docker }} | |
| # `code` is the docs-only short-circuit signal: TRUE unless this PR touches ONLY non-code paths | |
| # (Markdown / docs / licence / editorconfig / git metadata). It gates the EXPENSIVE STEPS of the | |
| # required `test` legs (and the whole `ide` job) — the jobs still run + report their (required) | |
| # context, but on a docs-only PR they skip install+lint+type+test and go green in seconds. TRUE on | |
| # push-to-main / workflow_dispatch (full coverage); FALSE on the nightly `schedule` (load-legs-only | |
| # cron — the functional matrix must not re-run nightly). | |
| code: ${{ steps.f.outputs.code }} | |
| # `ide`: run the ide build job only when a PR touches ide/** or this workflow (true on | |
| # push/dispatch for full coverage; false on the nightly cron). Not a required check. | |
| ide: ${{ steps.f.outputs.ide }} | |
| # `tooling`: the repo-HARNESS test tier — worktree gate, coordination claims, session mail, | |
| # ledger, announce, and the workflow files themselves. Measured 2026-08-16: 94 files and 1,882 | |
| # tests, 17 percent of the suite by count but 66 percent by TIME, because every one of them | |
| # spawns real pwsh/git children (~1.0s per test against 0.10s for an engine test). Its SUBJECT is | |
| # the development harness, so an engine-only PR cannot change its result — yet before this it set | |
| # the `--dist loadfile` floor for all three engine legs. The `test` legs now deselect it with | |
| # `-m 'not tooling'` and the `tooling` job below runs it, path-gated. Membership is | |
| # tests/tooling_manifest.txt, NOT a filename pattern — see that file for why. | |
| tooling: ${{ steps.f.outputs.tooling }} | |
| # `matrix`: the `test` job's include list, per-repo — full ubuntu+windows matrix HERE (this repo is | |
| # public, so hosted runners are free; the self-hosted Windows runners are retired), ubuntu-only on | |
| # a fork. Consumed as `matrix: ${{ fromJSON(needs.changes.outputs.matrix) }}` in `test`. | |
| matrix: ${{ steps.f.outputs.matrix }} | |
| # `ide_matrix`: the `ide` job's os matrix, per-repo — ubuntu + windows-latest HERE, where the | |
| # 2x-billed VS Code electron leg runs free; ubuntu-only on a fork. Consumed as | |
| # `matrix: ${{ fromJSON(needs.changes.outputs.ide_matrix) }}` in `ide`. | |
| ide_matrix: ${{ steps.f.outputs.ide_matrix }} | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| fetch-depth: 0 | |
| - id: f | |
| # Hoist the two workflow expressions into env (zizmor: no `${{ }}` interpolated into the run body | |
| # — a branch name / base sha is attacker-influenceable on a fork PR, so it must arrive as data). | |
| env: | |
| EVENT_NAME: ${{ github.event_name }} | |
| BASE_SHA: ${{ github.event.pull_request.base.sha }} | |
| run: | | |
| # test matrix, per-repo — computed FIRST (independent of the event, before any early exit). | |
| # THIS repo (MEFORORG/MessageFoundry, the source since the cutover): full ubuntu + windows-2022 | |
| # + windows-2025 on FREE hosted runners — free because the repo is public. | |
| # Anywhere else (a FORK): the UBUNTU leg ONLY, so a contributor's own minutes are not spent on | |
| # the 2x-billed Windows legs. This branch used to mean "the private source repo"; that repo is | |
| # now an inactive archive with Actions disabled, so a fork is the only thing that reaches it. | |
| # The self-hosted Windows NucBox runners are retired either way. | |
| # `hosted` = the runner labels (an array; see the `test` job's runs-on). | |
| # $GITHUB_REPOSITORY is a built-in runner env var, read here as plain shell (NOT a workflow- | |
| # expression interpolation into the run body), so it is zizmor-safe and cannot be misparsed as | |
| # an Actions expression the way a literal double-brace token in a run: block would be. | |
| # `webconsole_step_timeout` is the web console suite's OWN cap (BACKLOG #344 proposal 5). It | |
| # used to take `step_timeout` -- a 2-to-4-minute suite holding a 25-to-55-minute budget -- | |
| # which is why the nesting invariant could not hold for it on any leg. Derived in the note | |
| # above the `Web console tests (pytest)` step, which is also where to re-derive it. | |
| # `pytest_workers` is the xdist worker count for the engine suite. It is a MATRIX knob rather | |
| # than a literal in the run body for the same reason every other per-leg number here is one: | |
| # the trial dispatches that size it are then a one-line edit, and a leg can differ from its | |
| # siblings without touching the step. Sized at 4 because GitHub's hosted ubuntu and Windows | |
| # runners are 4-vCPU; `-n auto` would resolve to the same 4 today but would silently | |
| # over-subscribe if the runner size ever changes, and this suite spawns pwsh/git children | |
| # that compete for those same cores (measured: ~1.12 cores of average child occupancy), so | |
| # the count is stated rather than inferred. IF TIMING-SENSITIVE TESTS START FLAKING, 2 IS THE | |
| # CONSERVATIVE RUNG -- it leaves 2 vCPU of headroom for the dispatcher and connscale | |
| # assertions that measure elapsed time, at roughly two thirds of the saving. | |
| # | |
| # DO NOT RE-RUN THE "MORE WORKERS MIGHT BE FREE" EXPERIMENT: 8 WAS MEASURED AND IT IS WORSE ON | |
| # EVERY LEG. The tempting argument is that a worker blocked waiting on a pwsh/git child is not | |
| # holding a core, so 4 workers on 4 vCPU are under-subscribed. It is wrong. Dispatched | |
| # 2026-08-16 on branch claude/ci-worker-count-trial at 2bb733915 (run 31968796353), all three | |
| # legs raised to 8 together so each compares against its own baseline: | |
| # | |
| # leg -n 4 baseline (n=2) -n 8 (n=1) delta -n 8 outcome | |
| # ubuntu-latest 630 / 633s 731s +16% RED, 2 timing assertions | |
| # windows-2022 967 / 1004s 1067s +8% green | |
| # windows-2025 1084 / 1092s 1366s +25% RED, 3 tests | |
| # | |
| # The -n 4 baselines are two runs agreeing within 4 percent (PR #411 and its post-merge push). | |
| # | |
| # -n 4 IS NOT CLEAR OF THE TIMING FLOOR, ONLY FURTHER FROM IT. Recorded because the paragraph | |
| # below reads as "8 is unsafe, 4 is settled" and the second half is now known to be too | |
| # strong. Observed 2026-08-17 on PR #420, run 32005389065, ubuntu-latest, at -n 4: | |
| # FAILED tests/test_send_pacing.py::test_single_message_seam_is_paced | |
| # assert 0.03876582899999903 >= (0.05 * 0.8) | |
| # Missed by 1.2 ms -- 0.0388 against a required 0.0400 -- on a PR whose diff does not touch | |
| # that file (two coordination files, one of them pwsh-gated and skipped on ubuntu). So a 20 | |
| # percent tolerance on a 50 ms interval does not reliably clear a shared runner even at the | |
| # merged worker count. If this is to be fixed, the lever is the TOLERANCE or the pacing | |
| # mechanism, not pytest_workers -- lowering workers to 2 would buy margin by accident and | |
| # cost wall clock everywhere. | |
| # | |
| # ONE THING THAT SHOULD HELP, and is stated as a prediction rather than a result: the tooling | |
| # partition takes ~1,880 subprocess-spawning tests off this leg, and those are precisely the | |
| # contention this floor is sensitive to. All 7 pacing tests passed on a local -n 4 engine leg | |
| # under the partition, the seam test at 0.194s. That is one green run on a 20-core box, which | |
| # is not evidence about a 4-vCPU runner -- if the flake recurs on a partitioned leg, this | |
| # prediction was wrong and the tolerance is the only remaining lever. | |
| # What went red is the mechanism, not bad luck: ubuntu lost the two elapsed-time assertions | |
| # (test_outbound_batch.py's pacing floor and a super-linear-scaling ratio), and windows-2025 | |
| # lost the three heaviest multi-process tests -- connscale_smoke, multishard_smoke and | |
| # session_mail's claim-verdict test, which is the slowest single test in the suite. Contention | |
| # hits exactly the tests that measure time or bind real ports, which is why the failures | |
| # cluster there rather than scattering. | |
| # `webconsole_job_timeout` caps the SEPARATE web console job. It exists because that suite | |
| # moved out of `test` into its own parallel job, so it now needs a job cap of its own -- and | |
| # the nesting invariant has to hold there too: setup(max) + webconsole_step_timeout must fit | |
| # under it. Recorded setup maxima are 5:22 ubuntu / 4:05 W22 / 2:32 W25, and ci.yml records | |
| # provisioning noise alone at 4:30, so the worst case is about 5:22 + 4:30 + 5:00 = 14:52 on | |
| # the widest leg. 20 clears that with room. It is a DEADLOCK BACKSTOP, not a tightness gate: | |
| # the drift detector is the margin check against `webconsole_step_timeout`, so sizing this | |
| # close to the observed run would buy no detection and cost false reds on green suites. | |
| U='{"os":"ubuntu-latest","python-version":"3.14","hosted":["ubuntu-latest"],"job_timeout":43,"step_timeout":31,"webconsole_step_timeout":5,"webconsole_job_timeout":20,"pytest_timeout":60,"fault_timeout":90,"pytest_workers":4}' | |
| W22='{"os":"windows-2022","python-version":"3.14","hosted":["windows-2022"],"job_timeout":66,"step_timeout":55,"webconsole_step_timeout":6,"webconsole_job_timeout":20,"pytest_timeout":120,"fault_timeout":150,"pytest_workers":4}' | |
| W25='{"os":"windows-2025","python-version":"3.14","hosted":["windows-2025"],"job_timeout":66,"step_timeout":55,"webconsole_step_timeout":6,"webconsole_job_timeout":20,"pytest_timeout":120,"fault_timeout":150,"pytest_workers":4}' | |
| if [ "${GITHUB_REPOSITORY:-}" = "MEFORORG/MessageFoundry" ]; then | |
| echo "matrix={\"include\":[$U,$W22,$W25]}" >> "$GITHUB_OUTPUT" | |
| else | |
| echo "matrix={\"include\":[$U]}" >> "$GITHUB_OUTPUT" | |
| fi | |
| # ide matrix, per-repo (computed here with the test matrix, before any early exit): the | |
| # 2x-billed windows-latest electron leg runs only HERE, where hosted minutes are free; a fork | |
| # builds + type-checks the extension on ubuntu only. Consumed as | |
| # matrix: fromJSON(needs.changes.outputs.ide_matrix) in the ide job. (No shell vars here, so a | |
| # single-quoted literal JSON needs no escaping and carries no run-block workflow expression.) | |
| if [ "${GITHUB_REPOSITORY:-}" = "MEFORORG/MessageFoundry" ]; then | |
| echo 'ide_matrix={"os":["ubuntu-latest","windows-latest"]}' >> "$GITHUB_OUTPUT" | |
| else | |
| echo 'ide_matrix={"os":["ubuntu-latest"]}' >> "$GITHUB_OUTPUT" | |
| fi | |
| # Manual dispatch = full coverage: force every gate true so the whole workflow runs. | |
| if [ "$EVENT_NAME" = "workflow_dispatch" ]; then | |
| echo "serverdb=true" >> "$GITHUB_OUTPUT" | |
| echo "docker=true" >> "$GITHUB_OUTPUT" | |
| echo "code=true" >> "$GITHUB_OUTPUT" | |
| echo "ide=true" >> "$GITHUB_OUTPUT" | |
| echo "tooling=true" >> "$GITHUB_OUTPUT" | |
| exit 0 | |
| fi | |
| # Push-to-main: the FUNCTIONAL gate (code) stays true — the required test legs re-run on | |
| # the merge commit — but serverdb/docker are FALSE: the container legs (sqlserver-store, | |
| # postgres-store, docker-smoke) run nightly + on dispatch + via the PR path-gates, so a | |
| # push must no longer fire them. BOTH halves of that guard matter: their `if:` has no push | |
| # arm AND these outputs are false on push — restoring EITHER half re-adds the per-merge | |
| # cost. A merge touching those surfaces was already path-gated on its own PR minutes ago. | |
| if [ "$EVENT_NAME" = "push" ]; then | |
| echo "serverdb=false" >> "$GITHUB_OUTPUT" | |
| echo "docker=false" >> "$GITHUB_OUTPUT" | |
| echo "code=true" >> "$GITHUB_OUTPUT" | |
| echo "ide=true" >> "$GITHUB_OUTPUT" | |
| # TRUE on push, unlike serverdb/docker above, and the asymmetry is deliberate. Those two are | |
| # re-runs of container legs a PR already path-gated minutes earlier. This one is the SAFETY | |
| # NET for the whole gating scheme: the PR arm below only fires when the harness itself | |
| # changed, so an ENGINE change that breaks a harness test is invisible on its PR by | |
| # construction. Push-to-main is where that gets caught. Set this false and the tier is | |
| # effectively unguarded against everything except its own edits. | |
| echo "tooling=true" >> "$GITHUB_OUTPUT" | |
| exit 0 | |
| fi | |
| if [ "$EVENT_NAME" = "schedule" ]; then | |
| # Nightly run: the heavy legs fire via their own `schedule` if-arm; these outputs stay | |
| # false so the functional matrix and the PR path-gate arms don't ALSO fire. | |
| echo "serverdb=false" >> "$GITHUB_OUTPUT" | |
| echo "docker=false" >> "$GITHUB_OUTPUT" | |
| echo "code=false" >> "$GITHUB_OUTPUT" | |
| echo "ide=false" >> "$GITHUB_OUTPUT" | |
| echo "tooling=false" >> "$GITHUB_OUTPUT" | |
| exit 0 | |
| fi | |
| # Remaining case: pull_request. Compute the path-based filters from the PR diff. | |
| changed=$(git diff --name-only "$BASE_SHA...HEAD") | |
| # NB: the tests/test_(...) alternation MUST list every file the sqlserver-store / | |
| # postgres-store pytest steps below run (the ADR 0057/0058/0060/0066 throughput levers AND | |
| # the ADR 0013/0016 RTE capture/re-ingress case); those suites are MEFOR_TEST_SQLSERVER-gated | |
| # and run ONLY on the server-DB legs, so a file not matched here edits with NO real SS/PG | |
| # coverage (it merely skips the gated leg). Keep this in sync with those steps | |
| # (docs/archive/throughput/throughput-build-plan.md). | |
| # | |
| # AND THE INVARIANT BINDS THE SOURCES THOSE SUITES ASSERT AGAINST, NOT ONLY THE TEST FILES | |
| # THEY RUN (BACKLOG #1322). Binding the asserting file but not the asserted-on file cannot | |
| # see this class, and it has already fired: #514 (BACKLOG #1187) added a `masked` key to the | |
| # summary_access audit detail built by messagefoundry/api/app.py::_emit. It touched api/ only, | |
| # so serverdb evaluated FALSE, the DB legs never ran on its PR, and push never runs them. It | |
| # merged green leaving tests/test_postgres_store.py and tests/test_sqlserver_store.py | |
| # asserting the old two-key shape, and the NEXT PR to select those legs inherited three reds | |
| # it did not cause. | |
| # | |
| # MEASURED, not reasoned about: of the 36 messagefoundry modules the gated suites import, | |
| # EIGHTEEN were unmatched here -- not just api/app, but __main__, parsing/, six pipeline/ | |
| # modules and config/models. The source arms below now admit all 36, and | |
| # tests/test_serverdb_ci_coverage.py::test_serverdb_path_gate_admits_every_source_those_suites_import | |
| # DERIVES that set from the suites' own imports and fails if one stops being admitted -- so | |
| # this is enforced rather than merely written down here. Import-reachability is a PROXY for | |
| # "asserts against": it is what a machine can check, and it is deliberately wider than the | |
| # assertion set rather than narrower. | |
| # | |
| # This does NOT widen the push/schedule arms. Skipping these legs on push is deliberate | |
| # (see the no-per-merge-run guard below); the PR arm is the only one that can catch a | |
| # change BEFORE it lands, which is why the hole mattered. | |
| # The transports/ + config/wiring arms are the CAPTURE SURFACE: capture_response / reingress_to | |
| # are declared there (ADR 0013), and SQL Server / Postgres both declare supports_response_capture | |
| # + supports_pt_reingress True — so a regression in that surface is a SERVER-DB regression and must | |
| # pull these legs, not just the SQLite suite. | |
| if echo "$changed" | grep -qE '^(messagefoundry/store/|messagefoundry/__main__|messagefoundry/api/app|messagefoundry/parsing/(__init__|binary|message|peek|x12/)|messagefoundry/pipeline/(__init__|alerts|cluster|config_convergence|dr|leader_tasks|sharding|stage_dispatcher|wiring_runner)|messagefoundry/config/(models|response|settings|wiring)|messagefoundry/transports/(__init__|base|database|dicomweb|fhir|http_auth|mllp|rest|soap|tcp|x12)|tests/test_(sqlserver|postgres|cluster|database_connector|database_source|pooled|stage_dispatcher|batch_claim|claim_fifo|inline_fast_path|seq_only_fifo|fifo_index|per_lane_wake|response_capture|reingress|x12_rte|shard_recovery|shard_cert|adr0071|adr0075|adr0114|adr0157|dr_server_seed_gate|dr7_server_config_only_backup|backup_runner_server_db|connscale|load_failover|load_runner)|\.github/workflows/ci\.yml)'; then | |
| echo "serverdb=true" >> "$GITHUB_OUTPUT" | |
| else | |
| echo "serverdb=false" >> "$GITHUB_OUTPUT" | |
| fi | |
| # The container image smoke runs on PRs that touch the image, the per-profile locks, packaging, | |
| # or this workflow (engine-wide regressions are already covered by the `test` job + push-to-main). | |
| if echo "$changed" | grep -qE '^(docker/|\.dockerignore|pyproject\.toml|requirements\.lock|\.github/workflows/ci\.yml)'; then | |
| echo "docker=true" >> "$GITHUB_OUTPUT" | |
| else | |
| echo "docker=false" >> "$GITHUB_OUTPUT" | |
| fi | |
| # The ide job (VS Code extension build+test) only depends on ide/** (its own npm project) | |
| # and this workflow; a pure-Python PR can't change its result, so don't run it there. | |
| if echo "$changed" | grep -qE '^(ide/|\.github/workflows/ci\.yml)'; then | |
| echo "ide=true" >> "$GITHUB_OUTPUT" | |
| else | |
| echo "ide=false" >> "$GITHUB_OUTPUT" | |
| fi | |
| # `tooling`: the harness tier's SUBJECT is scripts/, the workflows, and the ledger, so those are | |
| # what can change its result on a PR. Plus the two files that define the partition itself: edit | |
| # the manifest or the conftest hook that reads it and the tier must re-run to prove the new | |
| # split still holds. | |
| # | |
| # THE SECOND ARM MATCHES THE MANIFEST AGAINST ITSELF, and that is the point. Editing one of the | |
| # 94 listed tests has to run them, and the obvious spelling -- an alternation of test-name | |
| # prefixes -- is a SECOND definition of the tier that drifts from the first silently, in the | |
| # direction where an edited test simply stops being covered. `grep -xF -f` reads the one | |
| # definition there is. Comments and blanks are stripped first: a blank pattern under -F -x | |
| # would match nothing here, but an unfiltered `#` line is still a pattern nobody intended. | |
| # | |
| # NOT gated on `code`: a docs-only PR that rewrites docs/BACKLOG.md must still face the ledger | |
| # and backlog-status tests, and those live in this tier. | |
| # `.gitignore` IS IN THIS LIST AND IT IS NOT COSMETIC -- it closes a reintroduction of BACKLOG | |
| # #327. tests/test_private_paths_stay_ignored.py is a manifest entry, and six .gitignore rules | |
| # are the sole control keeping maintainer-internal material out of a public commit. #327's fix | |
| # was to force-classify .gitignore as CODE so the test legs run it. This split broke that at a | |
| # layer above: code=true still runs the legs, but they now DESELECT the guard, and .gitignore | |
| # matched no arm here -- so the tooling job skipped, ci-gate reads a skipped need as pass, and | |
| # a .gitignore-only PR faced the guard on no leg at all. Same defect as #327, new route. | |
| # `.gitattributes` rides along for the same reason its sibling does in `alwayscodepath`. | |
| # docs/** in full, plus CLAUDE.md, pyproject.toml, ide/ and LICENSE: each is the declared | |
| # SUBJECT of a listed test (link_resolution and docs_runbooks read all of docs/; | |
| # claude_section_citations reads the root CLAUDE.md, which `\.claude/` does NOT match -- | |
| # that arm is the DIRECTORY; new_dependency_check reads pyproject; ide_licence_packaging reads | |
| # ide/ and LICENSE). Docs-only PRs were never the gap -- code=false already skipped the pytest | |
| # step. The gap is MIXED code+docs PRs, which are the common shape. | |
| # `.pre-commit-config.yaml` and `constraints.lock` close that same shape one layer further | |
| # out, added 2026-08-18. tests/test_lint_scope_parity.py is a manifest entry, and its subject | |
| # is a three-way agreement: the ruff `rev:` the hook pins, the `ruff==` constraints.lock | |
| # installs, and pyproject's cap. Measured against this regex as it stood that day, NEITHER file | |
| # matched EITHER arm. So on a `pre-commit autoupdate` PR -- which by construction rewrites revs | |
| # in .pre-commit-config.yaml and touches nothing else -- the engine legs deselected that guard | |
| # with `-m 'not tooling'`, this job skipped, ci-gate read the skipped need as a pass, and the PR | |
| # merged green with the guard having run nowhere. The push arm above is no backstop for it: | |
| # push fires on `branches: [main]`, i.e. AFTER the merge the gate existed to hold. | |
| # | |
| # `constraints.lock` IS THE EXPENSIVE ENTRY, AND THE COST IS ACCEPTED -- recorded here so it is | |
| # not rediscovered later as a surprise. The lock moves on essentially every dependency PR, so | |
| # this job now fires on essentially every dependency PR. The hole if it is dropped is concrete | |
| # rather than theoretical: `uv lock --upgrade-package ruff` edits the lockfiles and does NOT | |
| # edit pyproject.toml, so a lock-only ruff bump past the cap changes exactly what that test | |
| # asserts while matching no other arm here. This lock is listed rather than its siblings | |
| # because it is the file the test READS -- uv.lock and requirements.lock move with it in one | |
| # re-export, but a uv.lock edit that was never exported has not changed the guard's subject. | |
| if echo "$changed" | grep -qE '^(scripts/|\.github/|docs/|\.claude/|CLAUDE\.md|LICENSE|ide/|pyproject\.toml|\.gitignore|\.gitattributes|\.pre-commit-config\.yaml|constraints\.lock|tests/(conftest\.py|tooling_manifest\.txt|test_tooling_partition\.py))' \ | |
| || echo "$changed" | grep -qxFf <(grep -vE '^[[:space:]]*(#|$)' tests/tooling_manifest.txt); then | |
| echo "tooling=true" >> "$GITHUB_OUTPUT" | |
| else | |
| echo "tooling=false" >> "$GITHUB_OUTPUT" | |
| fi | |
| # `code`: CONSERVATIVE docs-only detector. Default to running everything (code=true), and only | |
| # short-circuit when EVERY changed path matches the docs/non-code allowlist below. We compute it | |
| # by inverting: list the files that are NOT docs-only; if that list is empty (and the diff is | |
| # non-empty), it's a docs-only PR. Anything outside the allowlist — any *.py, ide/**, config, | |
| # lockfiles, OTHER workflows, scripts, samples, harness — counts as CODE and runs the full suite. | |
| # A path appears in `changed` even if only deleted/renamed, so a pure doc rename still short- | |
| # circuits, and a code deletion still runs. | |
| # Allowlisted (docs-only) paths: any *.md anywhere, docs/**, top-level LICENSE/NOTICE/AUTHORS, | |
| # .editorconfig, .gitattributes, .github/{ISSUE_TEMPLATE,PULL_REQUEST_TEMPLATE,...}.md. | |
| # NOT .gitignore -- #327 removed it from the regex and this line went on listing it for | |
| # weeks, the THIRD instance in this file of a comment stating the opposite of its code. | |
| # `.gitattributes` IS still listed here and is nonetheless CODE: `alwayscodepath` below is | |
| # checked first and wins. Precedence, not deletion -- see the reasoning there. | |
| # Be conservative: when in doubt a path is CODE. Empty diff (shouldn't happen on a PR) => code=true. | |
| # `.gitignore` is NOT in this allowlist, deliberately (BACKLOG #327). Six of its rules are the | |
| # sole control keeping maintainer-internal material out of a public commit, and | |
| # tests/test_private_paths_stay_ignored.py is what asserts they still match. That test runs | |
| # under pytest, which is gated on code == 'true' -- so allowlisting `.gitignore` as docs-only | |
| # meant a `.gitignore`-only PR set code=false and the one guard that would catch the rule | |
| # being deleted DID NOT RUN, on exactly the PR shape it exists to catch. Same defect the two | |
| # ungated backlog guards above were added for. Treating it as code costs one suite run on a | |
| # rare PR; the alternative costs the publishing boundary, silently. | |
| noncode='(\.md$|^docs/|^LICENSE$|^NOTICE$|^AUTHORS$|^\.editorconfig$|^\.gitattributes$|^\.github/(ISSUE_TEMPLATE/|PULL_REQUEST_TEMPLATE))' | |
| # EXTENSIONLESS CONFIG THAT IS ALWAYS CODE (BACKLOG #1200). The extension rule below cannot | |
| # reach a file with no extension, and `.gitattributes` is exactly that: it was still in the | |
| # docs-only allowlist above, so a `.gitattributes`-only PR skipped lint, mypy and the whole | |
| # suite. It is not cosmetic -- the vault's own asvs-scorecard.yml records that a change here | |
| # "silently alters how the corpus is materialized, which is exactly what makes the digest | |
| # differ", so it is an input to the ASVS corpus pin. | |
| # | |
| # Stated as a POSITIVE list and CHECKED FIRST, rather than by deleting the entry from | |
| # `noncode`, for two reasons. "Is code" should not depend on the ABSENCE of a line somewhere | |
| # else -- a future edit re-adding a path to `noncode` would silently undo this with nothing | |
| # to say so. And keeping the historical `noncode` intact is what lets | |
| # tests/test_ci_docs_only_detector.py assert the regression in BOTH directions: it can still | |
| # reconstruct the old classification by disabling this rule alone. `.gitignore` is named for | |
| # the first reason even though #327 already removed it -- it was code only by falling through. | |
| alwayscodepath='^(\.gitattributes|\.gitignore)$' | |
| # EXTENSION OVERRIDE, EVALUATED FIRST (BACKLOG #1200). An executable file is CODE wherever it | |
| # lives, including under docs/. | |
| # | |
| # The paragraph above states the intent exactly -- "any *.py ... counts as CODE" -- and the | |
| # regex did not implement it, because `^docs/` is an alternation branch that matches a .py | |
| # under docs/ and short-circuits before the *.py rule is ever reached. An auditor reads the | |
| # comment, agrees with it, and moves on. Measured on 2026-08-09: | |
| # docs/security/asvs-apply-cells.py classified NON-CODE -- the tool that WRITES the ASVS | |
| # record of record, able to silently un-close an owner-closed cell, exempt from lint, mypy | |
| # and the entire pytest suite by virtue of its directory. Two mypy errors had been sitting in | |
| # it since it was written; they could not have survived a single check. | |
| # | |
| # THE PRECEDENT IS FOUR LINES ABOVE THIS ONE. BACKLOG #327 fixed exactly this shape for | |
| # `.gitignore` and wrote the lesson down -- and the identical defect for docs/**/*.py sat in | |
| # the regex immediately below the paragraph explaining it. The instance was fixed and the | |
| # class left open, with the reasoning that would have closed it preserved in place. Hence an | |
| # EXTENSION rule rather than another one-path exception: the next executable file someone | |
| # puts under docs/ must not need this discovered a third time. | |
| # | |
| # The docs-only optimisation is deliberately preserved for actual documents -- deleting | |
| # `^docs/` outright would run the full suite on every prose edit, which is the cost this | |
| # short-circuit exists to avoid. Order matters: this is checked BEFORE `noncode`. | |
| alwayscode='\.(py|ps1|sh|ts|js|yml|yaml|toml|lock|cfg|ini)$' | |
| if [ -z "$changed" ]; then | |
| echo "code=true" >> "$GITHUB_OUTPUT" | |
| elif echo "$changed" | grep -qE "$alwayscode"; then | |
| # An executable/config file changed, wherever it lives -> run the full suite. | |
| echo "code=true" >> "$GITHUB_OUTPUT" | |
| elif echo "$changed" | grep -qE "$alwayscodepath"; then | |
| # Extensionless config the extension rule cannot see -> run the full suite. | |
| echo "code=true" >> "$GITHUB_OUTPUT" | |
| elif echo "$changed" | grep -qvE "$noncode"; then | |
| # At least one changed path is NOT docs-only -> run the full suite. | |
| echo "code=true" >> "$GITHUB_OUTPUT" | |
| else | |
| # Every changed path is docs-only -> short-circuit the expensive steps. | |
| echo "code=false" >> "$GITHUB_OUTPUT" | |
| fi | |
| # SQL Server service-container leg: runs the gated suites that need a real SQL Server (Linux, so 1x | |
| # minutes) — the PRODUCTION store backend suite AND the PRODUCTION DATABASE connector round-trip | |
| # (ADR 0003/0010/0013). Runs NIGHTLY + on-demand + on PRs that touch the server-DB surface (gated | |
| # by `changes` above, so docs/unrelated PRs don't pull the rate-limited mcr image). It no longer | |
| # runs per push-to-main: a merge touching this surface was path-gated on its own PR minutes | |
| # earlier, so the per-merge re-run re-validated unchanged content (~16x/day); a regression that | |
| # somehow slips past a green PR (e.g. a semantic conflict between two concurrently-green PRs) is | |
| # caught by the nightly, fix-forward. Exercises the T-SQL + the live aioodbc round-trip the | |
| # SQLite / faked-driver tests can't. NOT a required check — ci-gate rolls it up (skip = pass). | |
| sqlserver-store: | |
| name: sql server (store + connector) ${{ matrix.label }} | |
| needs: changes | |
| # Nightly / on-demand, or a PR that touches the server-DB surface. `changes` sets serverdb=false | |
| # on push (both halves of the no-per-merge-run guard live there — see its comment) and on | |
| # schedule, where the first arm below already covers the run. | |
| if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || needs.changes.outputs.serverdb == 'true' | |
| runs-on: ubuntu-latest | |
| # Matrix the gated suite across every supported SQL Server major (2022 = 16.x, 2025 = 17.x) so a | |
| # version-specific T-SQL / engine regression can't merge. fail-fast: false keeps a 2025-only hiccup | |
| # from cancelling the 2022 leg (and vice-versa). The `changes` path-filter still gates this on PRs, so | |
| # matrixing only doubles the (rate-limited) mcr pulls on server-DB PRs + the nightly run. | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| include: | |
| - { image: "mcr.microsoft.com/mssql/server:2022-latest", label: "2022" } | |
| - { image: "mcr.microsoft.com/mssql/server:2025-latest", label: "2025" } | |
| services: | |
| mssql: | |
| image: ${{ matrix.image }} | |
| env: | |
| ACCEPT_EULA: "Y" | |
| MSSQL_SA_PASSWORD: "Str0ng_P@ssw0rd!" | |
| ports: | |
| - 1433:1433 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Set up Python | |
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.14" | |
| - name: Install Microsoft ODBC Driver 18 + sqlcmd | |
| # THIS IS THE DANGEROUS HALF OF THE apt CLASS, and it is the opposite of the one that was | |
| # visibly hurting. The Qt sites sit in jobs carrying a job-level `timeout-minutes`, so a | |
| # mirror hang there dies at the job cap (20-40 min, measured). THIS JOB HAS NO JOB-LEVEL | |
| # TIMEOUT, so the same hang runs to GitHub's 360-minute default -- a whole runner-hour | |
| # budget burned on a stalled mirror, on a leg nobody is watching because it is path-gated. | |
| # | |
| # The step cap bounds EVERY command here, including the two curls, which are equally | |
| # unguarded network calls. The retry wraps only the apt pair: re-running that is idempotent, | |
| # whereas re-fetching the signing key and repo list is not the part that hangs. | |
| # | |
| # THE DEEPER FIX IS A JOB-LEVEL `timeout-minutes` ON THIS JOB, which would bound every step | |
| # rather than the one that bit. Deliberately NOT done here: choosing that number needs a | |
| # measurement of how long this job legitimately runs, and a guessed cap on a server-DB leg | |
| # would kill real work. Recorded as a direction, not applied as a guess. | |
| timeout-minutes: 10 | |
| run: | | |
| curl -fsSL --max-time 60 https://packages.microsoft.com/keys/microsoft.asc \ | |
| | sudo tee /etc/apt/trusted.gpg.d/microsoft.asc > /dev/null | |
| curl -fsSL --max-time 60 "https://packages.microsoft.com/config/ubuntu/$(. /etc/os-release; echo "$VERSION_ID")/prod.list" \ | |
| | sudo tee /etc/apt/sources.list.d/mssql-release.list > /dev/null | |
| for attempt in 1 2 3; do | |
| if sudo timeout 120 apt-get update && sudo ACCEPT_EULA=Y timeout 240 apt-get install -y msodbcsql18 mssql-tools18 unixodbc-dev; then | |
| exit 0 | |
| fi | |
| echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying" | |
| sleep $((attempt * 5)) | |
| done | |
| echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test." | |
| exit 1 | |
| - name: Wait for SQL Server and create the database | |
| run: | | |
| sqlcmd=/opt/mssql-tools18/bin/sqlcmd | |
| for i in $(seq 1 60); do # ~120s cap: 2025's first boot can run slower than 2022's | |
| if "$sqlcmd" -S localhost -U sa -P 'Str0ng_P@ssw0rd!' -C -Q "SELECT 1" > /dev/null 2>&1; then | |
| echo "SQL Server is up"; break | |
| fi | |
| echo "waiting for SQL Server ($i)"; sleep 2 | |
| done | |
| "$sqlcmd" -S localhost -U sa -P 'Str0ng_P@ssw0rd!' -C \ | |
| -Q "IF DB_ID('MessageFoundry') IS NULL CREATE DATABASE MessageFoundry" | |
| - name: Set up uv | |
| uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| requirements.lock | |
| - name: Install project (dev + sqlserver extras) | |
| run: uv pip install --system --constraint constraints.lock -e ".[dev,sqlserver]" | |
| - name: Run the SQL Server store suite (against the real DB) | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_BACKEND: sqlserver | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| # The service container uses a self-signed cert, so trust_server_certificate=true is | |
| # required to connect. The store's TLS-hardening guard refuses that MITM-able combination | |
| # unless this escape is set — exactly the trusted-network dev/test case it exists for. | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # Dump the native (C-level) frame if a retry still crashes (faulthandler only DUMPS, never kills). | |
| PYTHONFAULTHANDLER: "1" | |
| # pyodbc 5.3.0 + py3.14 native-crash retry (#973 follow-up, upstream pyodbc#1459): re-runs ONLY on a 139/134 crash exit, never on exit 1, so it can't mask a regression. See scripts/ci/retry-native-crash.sh. | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v tests/test_sqlserver_store.py | |
| - name: Run the SQL Server coordinator suite (active-passive HA leader election) | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_BACKEND: sqlserver | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| # Same trusted-network TLS escape as the store suite (self-signed container cert): ODBC Driver | |
| # 18 defaults to Encrypt=yes + verify, so trust_server_certificate=true is required to connect, | |
| # and the store's TLS-hardening guard needs MEFOR_ALLOW_INSECURE_TLS to permit it. Exercises the | |
| # SqlServerCoordinator leader-lease (MERGE WITH(HOLDLOCK) acquire/renew/take-over, self-fence, | |
| # membership) against the real T-SQL the faked-driver tests can't. | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # Dump the native (C-level) frame if a retry still crashes (faulthandler only DUMPS, never kills). | |
| PYTHONFAULTHANDLER: "1" | |
| # pyodbc 5.3.0 + py3.14 native-crash retry (#973 follow-up, upstream pyodbc#1459): re-runs ONLY on a 139/134 crash exit, never on exit 1, so it can't mask a regression. See scripts/ci/retry-native-crash.sh. | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v tests/test_sqlserver_coordinator.py | |
| - name: Run the SQL Server failover suite (real TTL takeover + 2-node lifecycle) | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_BACKEND: sqlserver | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| # Same trusted-network TLS escape (self-signed container cert). Drives two coordinators sharing | |
| # the store through real DB-clock lease expiry + the full start()/stop() election + failover. | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # Dump the native (C-level) frame if a retry still crashes (faulthandler only DUMPS, never kills). | |
| PYTHONFAULTHANDLER: "1" | |
| # pyodbc 5.3.0 + py3.14 native-crash retry (#973 follow-up, upstream pyodbc#1459): re-runs ONLY on a 139/134 crash exit, never on exit 1, so it can't mask a regression. See scripts/ci/retry-native-crash.sh. | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v tests/test_cluster_failover_sqlserver.py | |
| - name: Run the DATABASE connector round-trip (against the real DB) | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| # The connector suite reuses the same MEFOR_STORE_* connection env + the trusted-network TLS | |
| # escape (self-signed container cert), exactly like the store suite above. This is the live | |
| # aioodbc/ODBC round-trip backing the connector's "production" status (PR D). | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # Dump the native (C-level) frame if a retry still crashes (faulthandler only DUMPS, never kills). | |
| PYTHONFAULTHANDLER: "1" | |
| # pyodbc 5.3.0 + py3.14 native-crash retry (#973 follow-up, upstream pyodbc#1459): re-runs ONLY on a 139/134 crash exit, never on exit 1, so it can't mask a regression. See scripts/ci/retry-native-crash.sh. | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v tests/test_database_connector_integration.py | |
| - name: Run the failover-LOAD test (two nodes, SIGKILL the primary mid-load, against the real DB) | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_BACKEND: sqlserver | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| # Gate #3 capstone + the FIRST LIVE PROOF of the SQL Server on-promotion recovery under a real | |
| # crash: two `serve` nodes share this container, the harness SIGKILLs the primary mid-load and | |
| # hard-asserts the host-independent CONFORMANCE invariants (zero acknowledged loss, per-lane FIFO, | |
| # no split-brain, bounded duplicates, a recovered + drained pipeline). Recovery *time* is reported, | |
| # not hard-gated (host-variable — see docs/LOAD-TESTING.md). This run guards the #285 claim-FIFO | |
| # fix. The orchestrator spawns the nodes with auth off + the cluster env; they inherit MEFOR_STORE_*. | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # Dump the native (C-level) frame if a retry still crashes (faulthandler only DUMPS, never kills). | |
| PYTHONFAULTHANDLER: "1" | |
| # pyodbc 5.3.0 + py3.14 native-crash retry (#973 follow-up, upstream pyodbc#1459): re-runs ONLY on a 139/134 crash exit, never on exit 1, so it can't mask a regression. See scripts/ci/retry-native-crash.sh. | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v tests/test_load_failover_sqlserver.py | |
| - name: Run the throughput-lever backend invariants on real SQL Server (B1 inline, B2 batch-claim, ...) | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_BACKEND: sqlserver | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # If every retry below still crashes, dump the native (C-level) frame so the upstream | |
| # pyodbc bug can be diagnosed from the log. faulthandler only DUMPS — it never kills. | |
| PYTHONFAULTHANDLER: "1" | |
| # The staged-pipeline throughput levers (ADR 0057 inline fast-path, ADR 0058 batch-claim, ...) carry | |
| # reliability-core invariant tests — crash-replay, poison-bound, and the per-lane-FIFO locked-head- | |
| # BLOCKS (#285) gate — that are MEFOR_TEST_SQLSERVER-gated but live in their OWN files, so the named | |
| # suites above never picked them up and they ran only on SQLite. Run them here against the real | |
| # backend. APPEND each new lever's test file per docs/archive/throughput/throughput-build-plan.md. | |
| # | |
| # NOT only throughput levers: this step is the catch-all for ANY MEFOR_TEST_SQLSERVER-gated file | |
| # the named suites above do not collect. CI runs an explicit file LIST here, not the whole suite, | |
| # so a new gated file that nobody appends never runs in CI at all — it just reports as skipped on | |
| # the plain legs, which reads identical to passing. `tests/test_sqlserver_legacy_outbox_migration.py` | |
| # (ASVS 14.2.7) is here for exactly that reason: it verifies a one-way schema migration that can | |
| # only be observed against a real server. | |
| # | |
| # RETRY WRAPPER: pyodbc 5.3.0 intermittently segfaults in its C parameter-binding path under | |
| # py3.14 against the SQL Server 2025 container (upstream mkleehammer/pyodbc#1459, unfixed; 5.3.0 | |
| # is the newest pyodbc and the first with py3.14 wheels, so there is nothing to bump to). The | |
| # crash kills the interpreter, so pytest-rerunfailures can't recover it — retry the WHOLE step at | |
| # the process level, but ONLY on a native-crash exit (139 SIGSEGV / 134 SIGABRT); a real failure | |
| # (exit 1) is NOT retried, so this never masks a regression. Remove the wrapper when #1459 ships a | |
| # fix and the pyodbc floor moves to it. See scripts/ci/retry-native-crash.sh. | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v | |
| tests/test_inline_fast_path.py | |
| tests/test_batch_claim_fifo.py | |
| tests/test_batch_claim_worker.py | |
| tests/test_batch_claim_locking.py | |
| tests/test_claim_fifo_heads.py | |
| tests/test_seq_only_fifo.py | |
| tests/test_fifo_index_migration.py | |
| tests/test_per_lane_wake.py | |
| tests/test_stage_dispatcher.py | |
| tests/test_pooled_rider.py | |
| tests/test_sqlserver_legacy_outbox_migration.py | |
| - name: Run the RTE capture / re-ingress loop on real SQL Server (ADR 0013/0016) | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_BACKEND: sqlserver | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| PYTHONFAULTHANDLER: "1" | |
| # SQL Server has declared supports_response_capture + supports_pt_reingress True since ADR 0013 | |
| # landed, and the store suite above proves the T-SQL — but the CONNECTOR->RUNNER half of the RTE | |
| # loop (a capturing outbound -> Stage.RESPONSE -> the response worker re-ingressing into a Loopback | |
| # inbound) had only ever run on SQLite, so a server-DB regression in it could not be caught. The | |
| # backend-parametrized case in this file closes that gap on the real backend. | |
| # Same pyodbc 5.3.0 + py3.14 native-crash retry as the steps above (upstream pyodbc#1459). | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v tests/test_x12_rte.py | |
| - name: Run the engine-shard + statement-dispatch suites on real SQL Server | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_BACKEND: sqlserver | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| PYTHONFAULTHANDLER: "1" | |
| # Every file here is MEFOR_TEST_SQLSERVER-gated at MODULE level, so before this step they were | |
| # collected nowhere and reported nothing — not a pass, not a skip, on any trigger. That covered | |
| # engine-shard crash recovery and the shard TLS-cert ladder (ADR 0037 over the ADR 0063 unified | |
| # store), the ADR 0071 statement dispatch/fusion wiring, the ADR 0075 batch backend, the ADR 0114 | |
| # live claim procedure, and the synchronous handoff path. tests/test_serverdb_ci_coverage.py now | |
| # fails if a module-gated suite is added without being named here. | |
| # Same pyodbc 5.3.0 + py3.14 native-crash retry as the steps above (upstream pyodbc#1459). | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v | |
| tests/test_shard_recovery_sqlserver.py | |
| tests/test_shard_cert_sqlserver.py | |
| tests/test_adr0071_dispatch_wiring_sqlserver.py | |
| tests/test_adr0071_fused_callables_sqlserver.py | |
| tests/test_adr0075_batch_sqlserver.py | |
| tests/test_adr0114_claim_proc_live.py | |
| tests/test_sqlserver_sync_handoff.py | |
| tests/test_database_source_integration.py | |
| - name: Run the DR seed-gate + backup suites on real SQL Server | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_BACKEND: sqlserver | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| PYTHONFAULTHANDLER: "1" | |
| # ADR 0048/0049 disaster recovery, SQL Server side: the seed gate that must REFUSE to activate a | |
| # standby onto a non-empty store, the .mfbak backup runner against a real server DB, and the DR7 | |
| # config-only backup + DbaDelegatedError path. Module-gated and never executed; a DR control that | |
| # has never run is a DR control that has never been proven. | |
| # Same pyodbc 5.3.0 + py3.14 native-crash retry as the steps above (upstream pyodbc#1459). | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v | |
| tests/test_dr_server_seed_gate_sqlserver.py | |
| tests/test_backup_runner_server_db_sqlserver.py | |
| tests/test_dr7_server_config_only_backup_sqlserver.py | |
| # Postgres store backend (Track B): run the gated store suite against a real PostgreSQL service | |
| # container (Linux, so 1x minutes). Runs NIGHTLY + on-demand (workflow_dispatch — use | |
| # `gh workflow run ci.yml --ref <branch>` to exercise it on a feature branch) + on PRs that touch | |
| # the server-DB surface (NEW: the same `changes` path-gate as sqlserver-store — a store/cluster PR | |
| # now gets real-Postgres coverage PRE-merge, which it never had). No longer per push-to-main; same | |
| # rationale as sqlserver-store above. This is what actually drives the SELECT ... FOR UPDATE SKIP | |
| # LOCKED + advisory-lock concurrency the SQLite tests can't reach. | |
| postgres-store: | |
| name: postgres store | |
| needs: changes | |
| # Nightly / on-demand / server-DB PRs (serverdb=false on push and schedule — see `changes`). | |
| if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || needs.changes.outputs.serverdb == 'true' | |
| runs-on: ubuntu-latest | |
| services: | |
| postgres: | |
| image: postgres:16 | |
| env: | |
| POSTGRES_PASSWORD: mefor | |
| POSTGRES_DB: messagefoundry | |
| ports: | |
| - 5432:5432 | |
| # Gate steps on readiness so the suite never races the container's first-boot init. | |
| options: >- | |
| --health-cmd pg_isready | |
| --health-interval 10s | |
| --health-timeout 5s | |
| --health-retries 5 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Set up Python | |
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.14" | |
| - name: Set up uv | |
| uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| requirements.lock | |
| - name: Install project (dev + postgres extras) | |
| run: uv pip install --system --constraint constraints.lock -e ".[dev,postgres]" | |
| - name: Run the Postgres store suite (against the real DB) | |
| env: | |
| MEFOR_TEST_POSTGRES: "1" | |
| MEFOR_STORE_BACKEND: postgres | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "5432" | |
| MEFOR_STORE_DATABASE: messagefoundry | |
| MEFOR_STORE_USERNAME: postgres | |
| MEFOR_STORE_PASSWORD: mefor | |
| # The service container speaks plaintext (no TLS), so encrypt=false; the store's | |
| # TLS-hardening guard refuses that MITM-able combination unless this trusted-network | |
| # dev/test escape is set. | |
| MEFOR_STORE_ENCRYPT: "false" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| run: pytest tests/test_postgres_store.py tests/test_adr0157_postgres_fence.py -v | |
| - name: Run the failover-LOAD test (two nodes, SIGKILL the primary mid-load, against the real DB) | |
| env: | |
| MEFOR_TEST_POSTGRES: "1" | |
| MEFOR_STORE_BACKEND: postgres | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "5432" | |
| MEFOR_STORE_DATABASE: messagefoundry | |
| MEFOR_STORE_USERNAME: postgres | |
| MEFOR_STORE_PASSWORD: mefor | |
| # Gate #3 capstone: two `serve` nodes share this container; the harness SIGKILLs the primary | |
| # mid-load and hard-asserts the host-independent CONFORMANCE invariants (zero acknowledged loss, | |
| # per-lane FIFO, no split-brain, bounded duplicates, a recovered + drained pipeline). Recovery | |
| # time is reported, not hard-gated. The orchestrator spawns the nodes with auth off + the cluster | |
| # env itself; they inherit MEFOR_STORE_* (incl. the plaintext-TLS escape) from here. | |
| MEFOR_STORE_ENCRYPT: "false" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| run: pytest tests/test_load_failover_postgres.py -v | |
| - name: Run the throughput-lever backend invariants on real Postgres (B1 inline, B2 batch-claim, ...) | |
| env: | |
| MEFOR_TEST_POSTGRES: "1" | |
| MEFOR_STORE_BACKEND: postgres | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "5432" | |
| MEFOR_STORE_DATABASE: messagefoundry | |
| MEFOR_STORE_USERNAME: postgres | |
| MEFOR_STORE_PASSWORD: mefor | |
| MEFOR_STORE_ENCRYPT: "false" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # The throughput levers' MEFOR_TEST_POSTGRES-gated invariants (esp. the FOR-UPDATE no-SKIP-LOCKED | |
| # locked-head-BLOCKS twin of #285, and the batch FIFO/crash tests) live in their own files. APPEND | |
| # each new lever's test file per docs/archive/throughput/throughput-build-plan.md. | |
| run: >- | |
| pytest -v | |
| tests/test_inline_fast_path.py | |
| tests/test_batch_claim_fifo.py | |
| tests/test_batch_claim_worker.py | |
| tests/test_batch_claim_locking.py | |
| tests/test_claim_fifo_heads.py | |
| tests/test_seq_only_fifo.py | |
| tests/test_fifo_index_migration.py | |
| tests/test_per_lane_wake.py | |
| tests/test_stage_dispatcher.py | |
| tests/test_pooled_rider.py | |
| - name: Run the connection-scale pool-wait smoke on real Postgres (B11 wall #2, tiny pool) | |
| env: | |
| MEFOR_TEST_POSTGRES: "1" | |
| MEFOR_STORE_BACKEND: postgres | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "5432" | |
| MEFOR_STORE_DATABASE: messagefoundry | |
| MEFOR_STORE_USERNAME: postgres | |
| MEFOR_STORE_PASSWORD: mefor | |
| MEFOR_STORE_ENCRYPT: "false" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # FORCE a below-default pool (4 connections) so the perf_counter-measured acquire-WAIT | |
| # histogram (the PRIMARY pool-wait signal) is non-trivial even at small N: ~3N inbound workers | |
| # contend for 4 connections, so acquires actually queue and the percentiles populate. The | |
| # default pool (40) would mask the wall. 4 (not 1-2) is deliberate: GET /status runs | |
| # db_status()'s four sequential COUNT(*) queries on every poll, and an extremely tiny pool | |
| # starves those acquires past the poller's HTTP timeout under the empty-claim herd on a slow | |
| # runner — every poll then fails and the drain loops burn the test budget (the PR #675 hang). | |
| # This still gives the B11 pool-wait instrumentation real regression coverage SQLite (no pool) | |
| # cannot — the connscale harness OWNS the engine subprocess and inherits this env. | |
| MEFOR_STORE_POOL_SIZE: "4" | |
| run: pytest tests/test_connscale_postgres.py -v | |
| - name: Run the RTE capture / re-ingress loop on real Postgres (ADR 0013/0016) | |
| env: | |
| MEFOR_TEST_POSTGRES: "1" | |
| MEFOR_STORE_BACKEND: postgres | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "5432" | |
| MEFOR_STORE_DATABASE: messagefoundry | |
| MEFOR_STORE_USERNAME: postgres | |
| MEFOR_STORE_PASSWORD: mefor | |
| MEFOR_STORE_ENCRYPT: "false" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # The Postgres twin of the sqlserver-store leg's RTE step (same backend-parametrized case, the | |
| # `postgres` param). Postgres declares the same two capability flags True, so the same | |
| # connector->runner gap existed here. | |
| run: pytest -v tests/test_x12_rte.py | |
| - name: Run the failover + engine-shard recovery suites on real Postgres | |
| env: | |
| MEFOR_TEST_POSTGRES: "1" | |
| MEFOR_STORE_BACKEND: postgres | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "5432" | |
| MEFOR_STORE_DATABASE: messagefoundry | |
| MEFOR_STORE_USERNAME: postgres | |
| MEFOR_STORE_PASSWORD: mefor | |
| MEFOR_STORE_ENCRYPT: "false" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # These are MEFOR_TEST_POSTGRES-gated at MODULE level, so until this step existed they were | |
| # collected nowhere and reported nothing — not a pass, not a skip. The SQL Server failover twin | |
| # (test_cluster_failover_sqlserver.py) has run for some time; its Postgres counterpart never had. | |
| # Engine-shard crash recovery (ADR 0037 over the ADR 0063 unified store) was dark on BOTH server | |
| # backends. tests/test_serverdb_ci_coverage.py now fails if a module-gated suite is added without | |
| # being named here. | |
| run: >- | |
| pytest -v | |
| tests/test_cluster_failover_postgres.py | |
| tests/test_shard_recovery_postgres.py | |
| - name: Run the DR seed-gate + backup suites on real Postgres | |
| env: | |
| MEFOR_TEST_POSTGRES: "1" | |
| MEFOR_STORE_BACKEND: postgres | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "5432" | |
| MEFOR_STORE_DATABASE: messagefoundry | |
| MEFOR_STORE_USERNAME: postgres | |
| MEFOR_STORE_PASSWORD: mefor | |
| MEFOR_STORE_ENCRYPT: "false" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # ADR 0048/0049 disaster recovery: the seed gate that must REFUSE to activate a standby onto a | |
| # non-empty store, the .mfbak backup runner against a real server DB, and the DR7 config-only | |
| # backup + DbaDelegatedError path. All three are module-gated and were never executed against | |
| # PostgreSQL; a DR control that has never run is a DR control that has never been proven. | |
| run: >- | |
| pytest -v | |
| tests/test_dr_server_seed_gate_postgres.py | |
| tests/test_backup_runner_server_db_postgres.py | |
| tests/test_dr7_server_config_only_backup_postgres.py | |
| # Headless load test (Track B / throughput): serve the synthetic high-fan-out load config (auth | |
| # off, small fan-out) and drive the smoke profile through the real `python -m harness --load` CLI, | |
| # asserting zero message loss + all SLOs (exit 0) and uploading the JSON/CSV report. Skipped on PRs | |
| # to save spend — the in-process load integration test (tests/test_load_runner.py, in the `test` | |
| # job) is the PR gate. Runs on push to main + on-demand (workflow_dispatch). Heavier profiles | |
| # (fanout-baseline / soak) and the SQLite-vs-Postgres comparison are run manually — see | |
| # docs/LOAD-TESTING.md. | |
| load-test: | |
| name: load test (smoke, sqlite) | |
| # Nightly schedule + on-demand only (was: every push to main). The PR gate is the in-process | |
| # tests/test_load_runner.py in the `test` job; push-to-main keeps the full functional matrix | |
| # (the server-DB suites are nightly + PR-path-gated too, as of the same cost pass). This | |
| # multi-minute serve+load run no longer fires once per commit. NOT a required check, | |
| # so dropping it from push is safe; ci-gate `needs` it but counts a `skipped` leg as a pass. | |
| if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Set up Python | |
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.14" | |
| - name: Set up uv | |
| uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| requirements.lock | |
| - name: Install project (dev — engine + httpx client; no Qt needed for the headless load path) | |
| run: uv pip install --system --constraint constraints.lock -e ".[dev]" | |
| - name: Serve the load config (auth off) and run the smoke profile | |
| env: | |
| MEFOR_SECURITY_REQUIRE_SIGN_IN: "false" # the load CLI polls /stats etc. without a bearer token | |
| MEFOR_LOAD_FANOUT: "8" | |
| MEFOR_LOAD_RESULTS_FANOUT: "2" | |
| MEFOR_LOAD_TRANSFORM: edit | |
| MEFOR_LOAD_SINK_PORT: "2700" # matches --sink-port below | |
| # GIVEN 1 (ADR 0148): the default env `dev` now derives PHI, so a bare `serve --env dev` runs the | |
| # secure PHI posture (secure-by-default). This leg validates that PHI+enforce serves green — so | |
| # provision the production posture rather than opting out to synthetic: a store encryption key | |
| # (minted at runtime below — never a committed literal), bounded PHI-body retention windows, and a | |
| # locked-down egress allowlisting the synthetic loopback sink the load graph delivers to. The | |
| # security-notification gate is skipped because auth is off (no accounts to notify). | |
| MEFOR_SECURITY_DELETE_MESSAGE_BODIES_AFTER_DAYS: "30" # bound inbound PHI bodies at rest | |
| MEFOR_RETENTION_DEAD_LETTER_DAYS: "30" # bound dead-lettered PHI bodies at rest | |
| MEFOR_SECURITY_BLOCK_UNLISTED_OUTBOUND: "true" # deny-by-default egress | |
| MEFOR_EGRESS_ALLOWED_MLLP: "127.0.0.1" # the load graph fans out to the loopback correlation sink | |
| run: | | |
| set -o pipefail | |
| mkdir -p out/load | |
| # Mint a throwaway AES-256 store key at runtime (PHI-at-rest must never be plaintext); never a | |
| # committed literal (gitleaks). 32 random bytes, base64 — the MEFOR_STORE_ENCRYPTION_KEY format. | |
| export MEFOR_STORE_ENCRYPTION_KEY="$(python -c 'import base64, os; print(base64.b64encode(os.urandom(32)).decode())')" | |
| python -m messagefoundry serve --config harness/config/load --env dev --db ./load-ci.db \ | |
| --host 127.0.0.1 --port 8765 > engine.log 2>&1 & | |
| engine_pid=$! | |
| for _ in $(seq 1 60); do | |
| curl -sf http://127.0.0.1:8765/health > /dev/null && break || sleep 0.5 | |
| done | |
| set +e | |
| python -m harness --load smoke --engine http://127.0.0.1:8765 --sink-port 2700 \ | |
| --report-json out/load/smoke-ci.json --report-csv out/load/smoke-ci.csv | |
| rc=$? | |
| set -e | |
| kill "$engine_pid" 2>/dev/null || true | |
| echo "===== engine log (tail) ====="; tail -50 engine.log || true | |
| exit $rc | |
| - name: Upload load report | |
| if: always() | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 | |
| with: | |
| name: load-report | |
| path: out/load/ | |
| if-no-files-found: ignore | |
| # SQL Server throughput smoke: the same zero-loss fan-out wiring check as `load test (smoke, sqlite)`, | |
| # but driven through the **SQL Server store** end-to-end (ingress→routed→outbound→delivered, each a | |
| # committed server-DB round-trip) on a real mssql service container. This is the load path that caught | |
| # the routed-stage `handler_name`-drop regression the in-process tests missed. Uses the `smoke-sqlserver` | |
| # profile (correctness SLOs strict; drain bound sized for a server DB). Skipped on PRs to save spend — | |
| # runs on push to main + on-demand (workflow_dispatch). Heavier SQL Server profiles run manually. | |
| load-test-sqlserver: | |
| name: load test (smoke, sqlserver) ${{ matrix.label }} | |
| # Nightly schedule + on-demand only (was: every push to main); same rationale as `load-test`. The | |
| # end-to-end SQL Server store path is still smoked nightly (x2 majors) + on demand, and the SQL | |
| # Server functional coverage stays in `sqlserver-store` (nightly + server-DB-PR path-gate). NOT | |
| # required; ci-gate treats skip as pass. | |
| if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' | |
| runs-on: ubuntu-latest | |
| # Smoke the end-to-end SQL Server store path on every supported major (2022/2025). Push/dispatch-only, | |
| # so no PR cost; fail-fast: false keeps the two legs independent. | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| include: | |
| - { image: "mcr.microsoft.com/mssql/server:2022-latest", label: "2022" } | |
| - { image: "mcr.microsoft.com/mssql/server:2025-latest", label: "2025" } | |
| services: | |
| mssql: | |
| image: ${{ matrix.image }} | |
| env: | |
| ACCEPT_EULA: "Y" | |
| MSSQL_SA_PASSWORD: "Str0ng_P@ssw0rd!" | |
| ports: | |
| - 1433:1433 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Set up Python | |
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.14" | |
| - name: Install Microsoft ODBC Driver 18 + sqlcmd | |
| # THIS IS THE DANGEROUS HALF OF THE apt CLASS, and it is the opposite of the one that was | |
| # visibly hurting. The Qt sites sit in jobs carrying a job-level `timeout-minutes`, so a | |
| # mirror hang there dies at the job cap (20-40 min, measured). THIS JOB HAS NO JOB-LEVEL | |
| # TIMEOUT, so the same hang runs to GitHub's 360-minute default -- a whole runner-hour | |
| # budget burned on a stalled mirror, on a leg nobody is watching because it is path-gated. | |
| # | |
| # The step cap bounds EVERY command here, including the two curls, which are equally | |
| # unguarded network calls. The retry wraps only the apt pair: re-running that is idempotent, | |
| # whereas re-fetching the signing key and repo list is not the part that hangs. | |
| # | |
| # THE DEEPER FIX IS A JOB-LEVEL `timeout-minutes` ON THIS JOB, which would bound every step | |
| # rather than the one that bit. Deliberately NOT done here: choosing that number needs a | |
| # measurement of how long this job legitimately runs, and a guessed cap on a server-DB leg | |
| # would kill real work. Recorded as a direction, not applied as a guess. | |
| timeout-minutes: 10 | |
| run: | | |
| curl -fsSL --max-time 60 https://packages.microsoft.com/keys/microsoft.asc \ | |
| | sudo tee /etc/apt/trusted.gpg.d/microsoft.asc > /dev/null | |
| curl -fsSL --max-time 60 "https://packages.microsoft.com/config/ubuntu/$(. /etc/os-release; echo "$VERSION_ID")/prod.list" \ | |
| | sudo tee /etc/apt/sources.list.d/mssql-release.list > /dev/null | |
| for attempt in 1 2 3; do | |
| if sudo timeout 120 apt-get update && sudo ACCEPT_EULA=Y timeout 240 apt-get install -y msodbcsql18 mssql-tools18 unixodbc-dev; then | |
| exit 0 | |
| fi | |
| echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying" | |
| sleep $((attempt * 5)) | |
| done | |
| echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test." | |
| exit 1 | |
| - name: Wait for SQL Server and create the database (RCSI on, like the store suite) | |
| run: | | |
| sqlcmd=/opt/mssql-tools18/bin/sqlcmd | |
| for i in $(seq 1 60); do # ~120s cap: 2025's first boot can run slower than 2022's | |
| if "$sqlcmd" -S localhost -U sa -P 'Str0ng_P@ssw0rd!' -C -Q "SELECT 1" > /dev/null 2>&1; then | |
| echo "SQL Server is up"; break | |
| fi | |
| echo "waiting for SQL Server ($i)"; sleep 2 | |
| done | |
| "$sqlcmd" -S localhost -U sa -P 'Str0ng_P@ssw0rd!' -C \ | |
| -Q "IF DB_ID('MessageFoundry') IS NULL CREATE DATABASE MessageFoundry; ALTER DATABASE MessageFoundry SET READ_COMMITTED_SNAPSHOT ON WITH ROLLBACK IMMEDIATE;" | |
| - name: Set up uv | |
| uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| requirements.lock | |
| - name: Install project (dev + sqlserver extras) | |
| run: uv pip install --system --constraint constraints.lock -e ".[dev,sqlserver]" | |
| - name: Serve on the SQL Server store and run the smoke-sqlserver profile | |
| env: | |
| MEFOR_SECURITY_REQUIRE_SIGN_IN: "false" # the load CLI polls /stats etc. without a bearer token | |
| MEFOR_LOAD_FANOUT: "8" | |
| MEFOR_LOAD_RESULTS_FANOUT: "2" | |
| MEFOR_LOAD_TRANSFORM: edit | |
| MEFOR_LOAD_SINK_PORT: "2700" # matches --sink-port below | |
| MEFOR_STORE_BACKEND: sqlserver # no --db: the backend + connection come from MEFOR_STORE_* | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" # trusted-network escape for the container's self-signed cert | |
| # DECLARE THE DATA CLASS HONESTLY (ADR 0148 GIVEN 1 + docs/SECURITY-LOOSENING.md | |
| # "`handles_real_patient_data = false`"). This leg processes harness-GENERATED synthetic HL7 on | |
| # a throwaway service container; asserting PHI was a false declaration about the instance, and | |
| # the register names exactly this case as acceptable: "a CI runner ... that only ever processes | |
| # synthetic / sample HL7". Since GIVEN 1 the built-in `dev` env derives PHI, so a genuinely | |
| # throwaway CI box must now set this EXPLICITLY — it is no longer the `dev` default. | |
| # | |
| # It is also what makes the leg run at all. This job was RED for four consecutive nights | |
| # (2026-07-27..30) on: | |
| # ValueError: SQL Server TLS is weakened (trust_server_certificate=true or encrypt=false) | |
| # Under enforcing-PHI the `MEFOR_ALLOW_INSECURE_TLS` escape above is clamped INERT by design | |
| # (#200 / ADR 0092 decision 2 — `weakened_tls_escape_permitted`), so a self-signed container | |
| # can never be trusted from a PHI instance. The previous comment here said this leg "validates | |
| # PHI+enforce green, not a synthetic opt-out" — an intent that is UNREACHABLE with a | |
| # `services:` container: GitHub starts it before any step runs, so a cert generated in a step | |
| # cannot be mounted into it. Whoever set that provisioned the PHI posture's retention and | |
| # egress requirements but not its TLS one. | |
| # | |
| # WHAT THIS GIVES UP, stated plainly: the leg no longer exercises the PHI+enforce path. Its | |
| # actual job is a load/throughput smoke of the SQL Server store, and `sqlserver-store` carries | |
| # the functional coverage. Restoring PHI+enforce here needs a REAL certificate — the job moved | |
| # off `services:` onto a `docker run` with a generated cert mounted and trusted — which is | |
| # worth doing deliberately, not as a side effect of unbreaking a nightly. | |
| # | |
| # The PHI-shaped provisions below are KEPT even though synthetic does not require them: they | |
| # cost nothing and keep the leg measuring a realistic configuration. | |
| MEFOR_SECURITY_HANDLES_REAL_PATIENT_DATA: "false" | |
| # Bounded PHI-body retention windows + a locked-down egress allowlisting the loopback sink. | |
| # The store key is minted at runtime below. The security-notification gate is skipped because | |
| # auth is off (no accounts to notify). | |
| MEFOR_SECURITY_DELETE_MESSAGE_BODIES_AFTER_DAYS: "30" | |
| MEFOR_RETENTION_DEAD_LETTER_DAYS: "30" | |
| MEFOR_SECURITY_BLOCK_UNLISTED_OUTBOUND: "true" | |
| MEFOR_EGRESS_ALLOWED_MLLP: "127.0.0.1" # the load graph fans out to the loopback correlation sink | |
| run: | | |
| set -o pipefail | |
| mkdir -p out/load | |
| # Mint a throwaway AES-256 store key at runtime (never a committed literal — gitleaks). | |
| export MEFOR_STORE_ENCRYPTION_KEY="$(python -c 'import base64, os; print(base64.b64encode(os.urandom(32)).decode())')" | |
| python -m messagefoundry serve --config harness/config/load --env dev \ | |
| --host 127.0.0.1 --port 8765 > engine.log 2>&1 & | |
| engine_pid=$! | |
| for _ in $(seq 1 60); do | |
| curl -sf http://127.0.0.1:8765/health > /dev/null && break || sleep 0.5 | |
| done | |
| set +e | |
| python -m harness --load smoke-sqlserver --engine http://127.0.0.1:8765 --sink-port 2700 \ | |
| --report-json out/load/smoke-sqlserver-ci.json --report-csv out/load/smoke-sqlserver-ci.csv | |
| rc=$? | |
| set -e | |
| kill "$engine_pid" 2>/dev/null || true | |
| echo "===== engine log (tail) ====="; tail -50 engine.log || true | |
| exit $rc | |
| - name: Upload load report | |
| if: always() | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 | |
| with: | |
| name: load-report-sqlserver | |
| path: out/load/ | |
| if-no-files-found: ignore | |
| # End-to-end check of the real Windows service path on both target Server | |
| # versions: install via NSSM -> start -> /health -> send an MLLP message -> | |
| # confirm it was recorded -> stop -> uninstall. Nightly + on-demand only | |
| # (2x-billed Windows legs): this validates the NSSM/service PLUMBING, which | |
| # changes when scripts/service/ or the serve path changes — not per merge — | |
| # and it was never a pre-merge gate (it already ran post-merge). Per-push it | |
| # was the single biggest push-run line item (~3.6 Windows min x ~16 merges/ | |
| # day); the nightly keeps both Server SKUs covered daily and workflow_dispatch | |
| # re-validates a specific merge on demand. | |
| windows-service-smoke: | |
| name: windows service smoke (${{ matrix.os }}, py${{ matrix.python-version }}) | |
| # This repo only: it spins up 2x-billed hosted Windows runners on BOTH Server SKUs, which is free | |
| # here (public) and billed to the owner of a fork. Nightly schedule + manual dispatch only. ci-gate | |
| # `needs` this job, but a `skipped` leg counts as a pass there, so gating it off a fork cannot wedge | |
| # the gate. | |
| if: (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && github.repository == 'MEFORORG/MessageFoundry' | |
| runs-on: ${{ matrix.os }} | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| # Windows Server 2022 + 2025 on the supported deploy Python (3.14), so the real NSSM | |
| # install->serve->MLLP path is validated on the version we ship, on each Server SKU. This job | |
| # is push/dispatch-only, so the Windows minutes cost no PR time. | |
| include: | |
| - { os: windows-2022, python-version: "3.14" } | |
| - { os: windows-2025, python-version: "3.14" } | |
| defaults: | |
| run: | |
| shell: pwsh | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Set up Python | |
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: ${{ matrix.python-version }} | |
| - name: Set up uv | |
| uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 | |
| with: | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| requirements.lock | |
| - name: Install the engine (fhir + dicom extras) | |
| run: | | |
| # The samples/config graph this smoke serves uses the FHIR and DICOM connectors, and the | |
| # DICOM inbound binds a DIMSE C-STORE SCP at startup (transports/dicom.py imports pynetdicom | |
| # eagerly on start) — so a bare `uv pip install --system -e .` makes the engine fail-close at | |
| # wiring with ModuleNotFoundError and /health never comes up. Install the connector extras the | |
| # graph needs (the "real Windows service path" a user running this graph would install). Keep | |
| # in sync with the connectors used in samples/config. | |
| uv pip install --system --constraint constraints.lock -e ".[fhir,dicom]" | |
| # Run the install/uninstall scripts under Windows PowerShell 5.1 (shell: powershell) to | |
| # match what the console launches on an end-user machine — catches 5.1-only behaviour | |
| # (e.g. native-command stderr aborting under ErrorActionPreference=Stop). NSSM happens to | |
| # be on the runner PATH, so the auto-download path is validated locally / on the user's box. | |
| - name: Install the service | |
| shell: powershell | |
| run: | | |
| $exe = (Get-Command messagefoundry).Source | |
| # Run as the 'prod' environment (ADR 0017: `serve` requires an explicit active environment; | |
| # install-service.ps1 -Environment threads it into the service command + resolves | |
| # environments/prod.toml). INFO not DEBUG: `serve` refuses DEBUG in prod (Gate #1 — DEBUG can | |
| # surface PHI); INFO is enough to smoke service start + MLLP. | |
| # -LockConfigDir: the runner checks out samples/config with the default ACL, which grants | |
| # BUILTIN\Users (S-1-5-32-545) write. The in-process config-source trust guard (SEC-003, | |
| # wiring._assert_safe_config_source_windows) then REFUSES to load it ("writable-by-others | |
| # path") and `serve` crash-loops to SERVICE_PAUSED, so /health never comes up. Locking the | |
| # config dir (strip inheritance -> SYSTEM/Administrators + the run-as account, RX) is the | |
| # production-correct posture docs/SERVICE.md prescribes, and exercises the real -LockConfigDir | |
| # install path rather than the MEFOR_ALLOW_INSECURE_CONFIG_SOURCE dev escape. | |
| # #224: NO -ServiceAccount and NO -AllowLocalSystem, so this install exercises the NEW DEFAULT - | |
| # the least-privilege virtual account NT SERVICE\MessageFoundry (no password). This leg is the | |
| # gate for that flip: it proves the virtual account is granted config-read (RX, via the | |
| # -LockConfigDir path that now runs AFTER ObjectName so the per-service SID resolves) + data-dir | |
| # read/write, gets SeServiceLogonRight, and actually starts and serves /health + MLLP. Pass | |
| # -AllowLocalSystem here if a run-as-LocalSystem smoke is ever needed instead. | |
| .\scripts\service\install-service.ps1 -AppExe $exe -LogLevel INFO -Environment prod -LockConfigDir | |
| # The shipped samples graph includes a mutual-TLS WS-* SOAP feed (IB_IMMUNIZATION_VXU) whose | |
| # connector LOADS its client cert at wiring time, so the engine refuses to start the graph | |
| # without a readable cert+key. Mint a throwaway pair for the smoke — the smoke sends only an | |
| # MLLP ADT, never a VXU, so the cert is loaded but never used in a handshake. Run openssl via | |
| # `cmd /c "... 2>nul"`: it streams RSA key-gen progress to stderr, and a bare `2>$null` did NOT | |
| # suppress it on the runner's openssl build — PowerShell 5.1 wrapped that stderr as a | |
| # NativeCommandError and aborted the step. Redirecting at the cmd level keeps PowerShell out of | |
| # openssl's stderr path entirely, so no NativeCommandError regardless of the openssl build. | |
| $certDir = Join-Path $env:RUNNER_TEMP "registry-cert" | |
| New-Item -ItemType Directory -Force -Path $certDir | Out-Null | |
| $cert = Join-Path $certDir "client.pem" | |
| $key = Join-Path $certDir "client.key" | |
| cmd /c "openssl req -x509 -newkey rsa:2048 -keyout ""$key"" -out ""$cert"" -days 1 -subj /CN=mefor-smoke -passout pass:smoke-key-pass 2>nul" | |
| if ($LASTEXITCODE -ne 0 -or -not (Test-Path $cert)) { throw "openssl failed to mint the smoke client cert" } | |
| # `prod` is a PHI environment, so the engine FAIL-CLOSES at startup unless a store encryption | |
| # key is set (PHI-at-rest must never be plaintext). Without one, `serve` refuses to start, NSSM | |
| # throttles the crash-loop to SERVICE_PAUSED, and /health never comes up. Mint a throwaway AES | |
| # key for the smoke — `gen-key` prints only the key (stdout, pipeable; no stderr to trip PS 5.1). | |
| $storeKey = (& $exe gen-key).Trim() | |
| if (-not $storeKey) { throw "messagefoundry gen-key produced no store encryption key" } | |
| # This smoke validates the Windows service + MLLP plumbing, not authentication (auth is | |
| # covered by the pytest suite: test_api_auth / test_console_auth). Run with auth disabled so | |
| # the /messages call below needs no bearer token, and supply the registry endpoint's secrets | |
| # (cert/key/password + WS-Security creds) so the SOAP feed wires — placeholders, never used | |
| # since no VXU is sent; the non-secret endpoints live in environments/prod.toml (selected by | |
| # the -Environment prod passed to install-service.ps1 above). | |
| # `prod` also FAIL-CLOSES on unrestricted egress (a transform could send PHI anywhere): lock it | |
| # down (deny_by_default) AND allowlist the samples/config graph's 8 outbound destinations, or the | |
| # engine refuses to start at wiring. [egress] is SERVICE settings (MEFOR_EGRESS_*), NOT an | |
| # environments/<env>.toml value; lists are comma-separated host:port matching the resolved | |
| # prod.toml endpoints (MLLP->allowed_mllp; X12->allowed_tcp; SOAP+FHIR->allowed_http; File->allowed_file_dirs). | |
| # NOTE keep this list in sync with samples/config when an outbound is added (e.g. OB_FHIR_SERVER) — | |
| # the egress check fails fast on the FIRST denied dest, so a missing entry wedges service start. | |
| # fhir-prod.example.org is allowlisted HOST-ONLY: its prod URL (https://fhir-prod.example.org/fhir) | |
| # has no explicit port, so urlsplit().port is None and a ":443" suffix would NOT match (see | |
| # _http_egress_allowed). A bare host matches any port, which is what we want for a default-port URL. | |
| # `prod` FAIL-CLOSES on unbounded PHI RETENTION too (#186a, ASVS 14.2.4): both PHI-body windows — | |
| # [retention].messages_days (inbound bodies) and [retention].dead_letter_days (dead-lettered | |
| # outbound bodies, replayable until purged) — default to 0 = keep-forever, and a PRODUCTION PHI | |
| # instance with EITHER unbounded refuses to start ("no data-retention window is configured ... | |
| # refusing to start"), so NSSM crash-loops to SERVICE_PAUSED and /health never comes up. BOUND both | |
| # (30d) rather than setting [retention].allow_unbounded_phi=true: a real prod deployment configures | |
| # a window, so the smoke exercises the path it is meant to validate instead of opting out of it. | |
| nssm set MessageFoundry AppEnvironmentExtra ` | |
| "MEFOR_SECURITY_REQUIRE_SIGN_IN=false" ` | |
| "MEFOR_STORE_ENCRYPTION_KEY=$storeKey" ` | |
| "MEFOR_SECURITY_DELETE_MESSAGE_BODIES_AFTER_DAYS=30" ` | |
| "MEFOR_RETENTION_DEAD_LETTER_DAYS=30" ` | |
| "MEFOR_SECURITY_BLOCK_UNLISTED_OUTBOUND=true" ` | |
| "MEFOR_EGRESS_ALLOWED_MLLP=receiver-prod.example.org:6661,powerscribe-prod.example.org:6665" ` | |
| "MEFOR_EGRESS_ALLOWED_TCP=payer-prod.example.org:6662,rte-prod.example.org:6663,rte-result-prod.example.org:6664" ` | |
| "MEFOR_EGRESS_ALLOWED_HTTP=iis-prod.example.org:8443,fhir-prod.example.org" ` | |
| "MEFOR_EGRESS_ALLOWED_FILE_DIRS=out" ` | |
| "MEFOR_VALUE_REGISTRY_CLIENT_CERT=$cert" ` | |
| "MEFOR_VALUE_REGISTRY_CLIENT_KEY=$key" ` | |
| "MEFOR_VALUE_REGISTRY_KEY_PASSWORD=smoke-key-pass" ` | |
| "MEFOR_VALUE_REGISTRY_USER=smoke-user" ` | |
| "MEFOR_VALUE_REGISTRY_PASSWORD=smoke-pass" | |
| - name: Start and verify /health | |
| run: | | |
| nssm start MessageFoundry | |
| $ok = $null | |
| foreach ($i in 1..30) { | |
| try { $ok = Invoke-RestMethod http://127.0.0.1:8765/health -TimeoutSec 2; break } | |
| catch { Start-Sleep -Milliseconds 500 } | |
| } | |
| if (-not $ok) { throw "engine /health never came up" } | |
| "health: $($ok | ConvertTo-Json -Compress)" | |
| - name: Send an MLLP message and confirm it was recorded | |
| run: | | |
| python samples\send_mllp.py samples\messages\adt_a01.hl7 | |
| $msgs = $null | |
| foreach ($i in 1..20) { | |
| $msgs = Invoke-RestMethod http://127.0.0.1:8765/messages | |
| if ($msgs.total -ge 1) { break } | |
| Start-Sleep -Milliseconds 500 | |
| } | |
| if ($msgs.total -lt 1) { throw "message was not recorded by the engine" } | |
| "messages.total = $($msgs.total)" | |
| - name: Stop the service (graceful) | |
| if: always() | |
| run: nssm stop MessageFoundry | |
| - name: Show service logs | |
| if: always() | |
| run: | | |
| $dir = "C:\ProgramData\MessageFoundry\logs" | |
| foreach ($f in "service.out.log", "service.err.log") { | |
| $p = Join-Path $dir $f | |
| "===== $f =====" | |
| if (Test-Path $p) { Get-Content $p } else { "(none)" } | |
| } | |
| - name: Uninstall the service | |
| if: always() | |
| shell: powershell | |
| run: .\scripts\service\uninstall-service.ps1 | |
| - name: Upload service logs | |
| if: always() | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 | |
| with: | |
| name: service-logs-${{ matrix.os }}-py${{ matrix.python-version }} | |
| path: C:\ProgramData\MessageFoundry\logs\ | |
| if-no-files-found: ignore | |
| # Container image smoke (the engine's FIRST non-Windows runtime target — it is Windows-service/NSSM | |
| # first today). Builds the slim engine image, builds a test-only image that bakes a minimal ADT->file | |
| # config + a self-contained MLLP sender (config baked, not mounted, so it is owned by the engine's UID | |
| # and not group/world-writable — _assert_safe_config_source refuses otherwise), serves it loopback + | |
| # auth-off, sends one synthetic ADT^A01 over the in-container MLLP listener, and asserts it FINALIZES | |
| # to PROCESSED (not merely RECEIVED — that distinguishes ACK-on-receipt from final disposition). Then | |
| # it verifies a graceful `docker stop`: tini forwards SIGTERM -> uvicorn lifespan -> engine.stop(). | |
| # Runs NIGHTLY + on dispatch; on PRs only when the image/locks/packaging/this workflow change | |
| # (gated by `changes`), so docs/unrelated PRs don't pay the build. No longer per push-to-main — | |
| # an image-touching merge was path-gated on its own PR minutes earlier (same rationale as the | |
| # server-DB legs; `changes` sets docker=false on push, the other half of this guard). | |
| docker-smoke: | |
| name: docker image smoke (slim, linux) | |
| needs: changes | |
| # Nightly / on-demand, or a PR that touches the image/locks/packaging. | |
| if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || needs.changes.outputs.docker == 'true' | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Build the slim engine image | |
| run: docker build -f docker/Dockerfile -t messagefoundry:ci . | |
| - name: Build the -sqlserver variant (ODBC apt layer + sqlserver lock must resolve) | |
| # Build-only: it pulls the Microsoft ODBC repo + installs from requirements-sqlserver.lock, so a | |
| # broken hashed dep or the Debian/MS-key issue the Dockerfile warns about turns the job red here. | |
| run: docker build -f docker/Dockerfile --target runtime-sqlserver -t messagefoundry:sqlserver-ci . | |
| - name: Build the test smoke image (baked minimal config + MLLP sender) | |
| run: docker build -f docker/smoke/Dockerfile --build-arg BASE=messagefoundry:ci -t messagefoundry:smoke . | |
| - name: Serve (loopback, auth off) and assert an MLLP message reaches PROCESSED | |
| run: | | |
| set -euo pipefail | |
| # Bind the API + MLLP to loopback INSIDE the container and drive the test from inside it | |
| # (docker exec). A non-loopback bind without TLS would be refused by the startup bind guard | |
| # (and auth-off is refused off-loopback regardless of --allow-insecure-bind), so loopback + | |
| # in-container traffic is the correct posture for an auth-disabled smoke — no insecure override. | |
| # GIVEN 1 (ADR 0148): `dev` now derives PHI, so `serve --env dev` runs the secure PHI posture. | |
| # This smoke validates PHI+enforce green (not a synthetic opt-out): provision a runtime-minted | |
| # store key (never a committed literal — gitleaks), bounded PHI-body retention windows, and a | |
| # locked-down egress allowlisting the smoke config's File archive dir (/var/lib/mefor/out/adt). | |
| # The security-notification gate is skipped because auth is off (no accounts to notify). | |
| storeKey="$(python3 -c 'import base64, os; print(base64.b64encode(os.urandom(32)).decode())')" | |
| docker run -d --name mefor \ | |
| -e MEFOR_SECURITY_REQUIRE_SIGN_IN=false \ | |
| -e MEFOR_STORE_ENCRYPTION_KEY="$storeKey" \ | |
| -e MEFOR_SECURITY_DELETE_MESSAGE_BODIES_AFTER_DAYS=30 \ | |
| -e MEFOR_RETENTION_DEAD_LETTER_DAYS=30 \ | |
| -e MEFOR_SECURITY_BLOCK_UNLISTED_OUTBOUND=true \ | |
| -e MEFOR_EGRESS_ALLOWED_FILE_DIRS=/var/lib/mefor \ | |
| messagefoundry:smoke \ | |
| serve --config /config --env dev --host 127.0.0.1 --port 8765 | |
| # Readiness: /health is tokenless and always 200 once the listener is up. | |
| for _ in $(seq 1 60); do | |
| docker exec mefor curl -fsS http://127.0.0.1:8765/health >/dev/null 2>&1 && break || sleep 0.5 | |
| done | |
| docker exec mefor curl -fsS http://127.0.0.1:8765/health | |
| # Send one synthetic ADT^A01 over the in-container MLLP listener (loopback:2575); fails on a non-AA ACK. | |
| docker exec mefor python /smoke/send_adt.py 127.0.0.1 2575 | |
| # Poll the disposition via the container's own python (stdlib urllib — no host python/jq needed). | |
| total() { docker exec mefor python -c "import urllib.request,json;print(json.load(urllib.request.urlopen('http://127.0.0.1:8765/messages?status=$1'))['total'])"; } | |
| processed=0 | |
| for _ in $(seq 1 60); do | |
| # `|| processed=0`: a transient non-zero before the row finalizes must NOT abort the loop under set -e. | |
| processed=$(total processed) || processed=0 | |
| [ "$processed" -ge 1 ] && break || sleep 0.5 | |
| done | |
| received=$(total received) || received=0 | |
| echo "processed=$processed received=$received" | |
| # The whole point of the assert: confirm it FINALIZED, not just that it was acknowledged-on-receipt. | |
| if [ "$processed" -lt 1 ]; then | |
| echo "FAIL: message did not finalize to PROCESSED (received=$received) — stuck at ACK-on-receipt" | |
| exit 1 | |
| fi | |
| - name: Verify graceful shutdown (tini -> SIGTERM -> engine.stop drains) | |
| run: | | |
| set -euo pipefail | |
| docker stop --time 30 mefor | |
| code=$(docker inspect -f '{{.State.ExitCode}}' mefor) | |
| echo "container exit code: $code" | |
| # A fully graceful shutdown exits 0 (uvicorn handles SIGTERM, runs the lifespan, returns); 143 | |
| # (128+SIGTERM) is acceptable too. 137 = SIGKILL after the grace expired = ungraceful. | |
| if [ "$code" = "137" ]; then echo "FAIL: ungraceful shutdown (SIGKILL after grace)"; exit 1; fi | |
| # Poll for the marker: `docker stop` returns the instant the container exits, but the final | |
| # buffered log line ("Application shutdown complete") can land in `docker logs` a beat later, so | |
| # grepping once immediately is racy (a clean 143 shutdown then read as a missing marker). Retry | |
| # for ~10s so a slow log flush is not misread as an incomplete lifespan shutdown. | |
| for _ in $(seq 1 20); do | |
| if docker logs mefor 2>&1 | grep -q "Application shutdown complete"; then | |
| echo "graceful shutdown verified"; exit 0 | |
| fi | |
| sleep 0.5 | |
| done | |
| echo "FAIL: no clean-shutdown marker (lifespan shutdown did not complete)" | |
| docker logs mefor 2>&1 | tail -20 | |
| exit 1 | |
| - name: Engine log (always) | |
| if: always() | |
| run: docker logs mefor 2>&1 | tail -60 || true | |
| - name: Tear down | |
| if: always() | |
| run: docker rm -f mefor || true | |
| # Single STABLE required status check standing in for the conditional / matrix heavy legs (SQL | |
| # Server, Postgres, load, Windows-service smoke). Those legs CANNOT be required directly: | |
| # * the nightly/path-gated legs (`postgres-store`, `load-test`, `load-test-sqlserver`, | |
| # `windows-service-smoke`) report `skipped` on most or all PRs — never a stable required | |
| # context a PR can wait on — so requiring them either no-ops or wedges; and | |
| # * the matrix legs report an UNEXPANDED name when skipped (`... ${{ matrix.label }}`) but | |
| # EXPANDED names when they run (`... 2022`/`2025`, `(windows-2022, py3.14)`), so no single | |
| # context string matches both the skipped and the run state. | |
| # This gate ALWAYS runs (if: always()), so it reports ONE fixed context ("CI gate") regardless of | |
| # which legs ran, and fails iff a gated leg actually FAILED or was cancelled — a `skipped` leg | |
| # counts as a pass. Require "CI gate" in branch protection INSTEAD of the individual legs; a leg | |
| # that does run on a PR (e.g. `sqlserver-store` on a server-DB PR) thus still blocks the merge. | |
| # NB: `docker-smoke` is intentionally NOT gated here (out of the named set) — add it to `needs` | |
| # below to make a container-smoke failure block merge too. | |
| ci-gate: | |
| name: CI gate | |
| if: always() | |
| needs: | |
| - changes | |
| - sqlserver-store | |
| - postgres-store | |
| - load-test | |
| - load-test-sqlserver | |
| - windows-service-smoke | |
| # `webconsole` rides this roll-up rather than taking a required context of its own. It is NOT | |
| # path-gated like the five above -- it runs on every PR -- so it COULD have been required | |
| # directly; riding here is the cheaper correct choice, because adding a context means a | |
| # branch-protection change and, until that lands, a PR that can never satisfy it. The coverage is | |
| # identical either way: this job is `if: always()` and fails on any needed job's `failure` or | |
| # `cancelled`, and `CI gate` is required. Before this, the console suite gated merges by being a | |
| # step inside the required `test` legs; it must not silently stop gating them now that it is not. | |
| - webconsole | |
| # `tooling` rides here for the same reason, and the precedent above is the whole argument: it was | |
| # a set of steps inside the required `test` legs until the manifest split it out, so without this | |
| # line the harness tier would stop gating merges the moment it stopped being part of `test`. It IS | |
| # path-gated (unlike webconsole), so it skips on most PRs -- and a skipped need is not a failure | |
| # to the `contains(needs.*.result, ...)` check below, which is the behaviour wanted. | |
| - tooling | |
| runs-on: ubuntu-latest | |
| steps: | |
| - name: Fail if any gated leg failed or was cancelled | |
| if: >- | |
| contains(needs.*.result, 'failure') || | |
| contains(needs.*.result, 'cancelled') | |
| env: | |
| # Pass the needs context via env, not inline ${{ }} in the run block (defense-in-depth against | |
| # template injection — a job output could in theory carry a quote that breaks out of echo). | |
| NEEDS_JSON: ${{ toJSON(needs) }} | |
| run: | | |
| echo "::error::A gated CI leg failed or was cancelled." | |
| echo "$NEEDS_JSON" | |
| exit 1 | |
| - name: Gated legs OK | |
| run: echo "All gated CI legs succeeded or were skipped." |