feat(retention): ASVS 14.2.7 — retention classification, automatic deletion, and the legacy outbox PHI leak #695
Workflow file for this run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: CI | |
| on: | |
| push: | |
| branches: [main] | |
| pull_request: | |
| workflow_dispatch: | |
| schedule: | |
| # Nightly (03:17 UTC — off the top-of-hour rush) run of the HEAVY POST-MERGE legs: the load | |
| # smokes (load-test, load-test-sqlserver x2 majors), the server-DB suites (sqlserver-store x2 | |
| # majors, postgres-store), windows-service-smoke (x2 SKUs, 2x-billed) and docker-smoke. These | |
| # all used to run on EVERY push to main (~16 auto-merges/day re-validating content whose PR run | |
| # had just passed); the nightly cadence + the PR path-gates (`changes`) keep the same coverage | |
| # at a fraction of the runs. `gh workflow run ci.yml --ref main` (workflow_dispatch) still runs | |
| # EVERYTHING on demand — use it to re-validate a specific merge immediately. | |
| # IMPORTANT: the cron must still NOT drag the FUNCTIONAL matrix along. `test` and `ide` gate | |
| # their full-coverage arm as `push || workflow_dispatch` — deliberately EXCLUDING `schedule` — | |
| # so the nightly never re-executes them (they already ran on the merge itself; the original | |
| # `!= 'pull_request'` form would have swept them into the cron). The nightly legs' `if:` is | |
| # `schedule || workflow_dispatch` (+ a PR path-gate arm where `changes` provides one). | |
| - cron: "17 3 * * *" | |
| concurrency: | |
| # Per-ref group so a new push/PR-update supersedes its own in-flight run. The nightly `schedule` run | |
| # gets a SEPARATE group (…-nightly): without this it would share `ci-refs/heads/main` with push-to-main | |
| # and a routine daytime push would cancel-in-progress the in-flight nightly LOAD run — silently killing | |
| # the very long-running coverage the cron exists to provide. Scheduled and push runs no longer collide. | |
| group: ci-${{ github.ref }}${{ github.event_name == 'schedule' && '-nightly' || '' }} | |
| cancel-in-progress: true | |
| # Least privilege: every job here only reads the repo and runs tests/builds (no gh writes, no git push, | |
| # no secrets), so a read-only default token is sufficient. A job needing more declares its own block. | |
| permissions: | |
| contents: read | |
| jobs: | |
| # Lint + type-check + unit tests. The project supports a single Python — 3.14 — so every leg runs it. | |
| # Linux carries the cheap breadth (1x minutes); Windows Server 2022 + 2025 is the primary deployment | |
| # target, so 3.14 is also exercised on both Server SKUs even though Windows bills at 2x — testing the | |
| # version we ship on the OS we ship it on is worth the minutes. | |
| test: | |
| name: test (${{ matrix.os }}, py${{ matrix.python-version }}) | |
| # `needs: changes` for the docs-only short-circuit ONLY — this REQUIRED job still RUNS on every PR | |
| # and ALWAYS reports its context. On a docs-only PR `changes.outputs.code == 'false'` and the | |
| # expensive steps (install/lint/type/test) are skipped, so the leg goes green in seconds; the | |
| # required `test (…, py3.14)` context stays present + green. (changes never fails, so this needs | |
| # edge can't wedge the required context.) | |
| needs: changes | |
| # Every leg runs on GitHub-HOSTED runners now — the self-hosted NucBox Windows runners are RETIRED. | |
| # This repo (MEFORORG) runs the full ubuntu + windows-2022 + windows-2025 matrix, all FREE (hosted | |
| # minutes are free on a public repo); a FORK building its own pushes gets the UBUNTU leg only, to | |
| # keep a contributor's personal CI cheap. Hosted-everywhere is also safer: a self-hosted runner must | |
| # never be reachable by a public repo (fork PR = RCE on the owner's LAN). `matrix.hosted` is the | |
| # runner label set; the per-repo matrix is built in `changes`. | |
| runs-on: ${{ matrix.hosted }} | |
| # Wall-clock backstop: bound each leg so a hung test fails in minutes, not the 6h default. Three | |
| # nested watchdogs catch a hang at increasing scope (#55): the pytest-timeout per-test cap (matrix | |
| # `pytest_timeout`, 60s ubuntu / 120s Windows -- the step passes `--timeout=`, which overrides the | |
| # pyproject addopts value) + the faulthandler belt fire FIRST and name the stuck frame; the pytest STEP | |
| # has its own `timeout-minutes` (matrix `step_timeout`) so a process-level deadlock below pytest | |
| # fails the step fast; this job cap is the outermost belt if even that is somehow out-raced. | |
| timeout-minutes: ${{ matrix.job_timeout }} | |
| strategy: | |
| fail-fast: false | |
| # Per-repo matrix from the `changes` job: the full ubuntu+windows matrix here, ubuntu-only on a | |
| # fork's own pushes. Each leg carries os / python-version / hosted (the runner labels, an array) + | |
| # the timeout knobs. NB: a fork *PR* against this repo still runs in THIS repo's context, so it | |
| # gets the full matrix — the required `test (windows-2022/2025, py3.14)` contexts are always | |
| # present on a PR here, and a required-but-absent context (which blocks a PR forever) can't occur. | |
| matrix: ${{ fromJSON(needs.changes.outputs.matrix) }} | |
| defaults: | |
| run: | |
| shell: bash # available on all runners (Git Bash on Windows) -> one set of commands | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Set up Python ${{ matrix.python-version }} | |
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: ${{ matrix.python-version }} | |
| # No `cache: pip`: installs now go through uv (the "Set up uv" step below), which keeps its | |
| # OWN wheel/download cache — a pip cache here would be a second, redundant cache competing | |
| # for the repo's 10 GB cache budget. uv's cache is keyed on pyproject.toml + requirements.lock. | |
| # Everything below is gated on `code` (docs-only short-circuit) OR a push-to-main / on-demand run | |
| # (full coverage). On a docs-only PR these steps no-op and the REQUIRED leg reports green in seconds. | |
| # The job itself (and the Python/cache setup above) still runs, so the required context is always | |
| # present. NB: the gate is duplicated per-step (a job-level `if` would SKIP the whole job and remove | |
| # the required context) — keep the condition identical on each. The push/dispatch arm is written | |
| # `push || workflow_dispatch` (NOT `!= 'pull_request'`) so the nightly `schedule` event does NOT | |
| # re-run this full functional matrix — the cron is reserved for the long load legs only. | |
| # DELIBERATELY UNGATED — the one step in this job with no `code` condition. | |
| # | |
| # The ledger gate polices the ADR/BACKLOG number space, and an ADR-only PR is by definition | |
| # DOCS-ONLY, so `code == 'false'` and every gated step above is skipped. Gating this one the same | |
| # way would skip it on exactly the pull requests it exists to police. It needs only git + stdlib | |
| # python (no install), so it costs seconds and runs on every PR. | |
| # | |
| # It rides inside the already-REQUIRED `test` leg rather than becoming a new required context: a | |
| # brand-new required check wedges every PR opened before it existed (see the `ci-gate` roll-up). | |
| # | |
| # This is the backstop for `git commit --no-verify` and for a branch cut from a stale main — each | |
| # branch is internally consistent, and the duplicate number only exists once BOTH have merged, so | |
| # only a check against freshly-fetched main can see it. | |
| - name: Ledger gate (ADR / BACKLOG number space) | |
| if: runner.os == 'Linux' | |
| run: | | |
| # Only `origin/main` needs fetching: the gate reads it (`git show origin/main:…`, `ls-tree`) and | |
| # TWO-dot-diffs it against HEAD. No ancestry is resolved, so the shallow checkout is fine and no | |
| # deepening is required -- see the long note in ledger_check.py's tree-access section for why | |
| # three-dot was wrong here, and why its failure was silent. | |
| git fetch --no-tags --depth=200 origin main | |
| python scripts/hooks/ledger_check.py --ci | |
| # ALSO DELIBERATELY UNGATED, for exactly the reason the step above is — and it was NOT, which | |
| # cost `main`. | |
| # | |
| # Two guards police docs/BACKLOG.md. The number-space gate above is ungated because "gating it | |
| # would skip it on exactly the pull requests it exists to police". The STATUS invariant was | |
| # reachable only through pytest, which IS gated on `code == 'true'` — so on a BACKLOG-only PR | |
| # (the exact shape it exists to check) it did not run at all. | |
| # | |
| # On 2026-08-01 that landed a #320 entry whose banner used an emoji the invariant does not | |
| # accept. The PR was docs-only, went green in seconds without compiling the suite, merged, and | |
| # red `main` for every other session until someone else's PR tripped over it. A 0.6s markdown | |
| # parse would have caught it. One hardened guard and one unhardened guard over the same file is | |
| # worse than neither, because the hardened one makes it reasonable to assume BACKLOG changes are | |
| # covered. | |
| # | |
| # No install needed, so the step above's "git + stdlib python, costs seconds" property holds: | |
| # scripts/docs/backlog_status_check.py imports only argparse/re/sys/pathlib and has its own CLI. | |
| # tests/test_backlog_status_check.py imports that SAME module, so there is one implementation and | |
| # the unit tests keep covering its edge cases — this only adds the always-on invocation. | |
| - name: Backlog status invariant (ungated — see above) | |
| if: runner.os == 'Linux' | |
| run: python scripts/docs/backlog_status_check.py | |
| # PySide6's offscreen platform plugin needs a few system libraries even | |
| # headless. Linux-only; Windows runners need no equivalent. | |
| - name: Install Qt offscreen system libraries | |
| if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux' | |
| run: | | |
| sudo apt-get update | |
| sudo apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3 | |
| # uv is the installer for every leg (was pip): it resolves + installs the pyproject extras far | |
| # faster than pip, into the same setup-python interpreter via `uv pip install --system`. Pinned | |
| # by the same SHA dependabot-lock-resync.yml already uses. Gated identically to the install | |
| # below (docs-only PRs skip both). enable-cache defaults to 'auto' — a GitHub-hosted cache on | |
| # hosted runners, the persistent on-disk uv cache on a self-hosted runner — keyed on the glob. | |
| # | |
| # EVERY install below passes `--constraint constraints.lock` — the HASHLESS export of uv.lock, kept | |
| # in sync by the DEP-1 gate. Without it, `uv pip install -e ".[extras]"` RE-RESOLVES from | |
| # pyproject's `>=` floors and silently adopts whatever upstream published since: ruff 0.16.0 and | |
| # annotated-types 0.8.0 (dropped `SLOTS`, broke fhir-core) each reddened EVERY open PR on the same | |
| # day — see each cap's rationale in pyproject.toml. The constraint pins every version to the lock | |
| # while each job still installs only ITS OWN extras (so windows-service-smoke's deliberately | |
| # narrow set stays narrow; the docker-smoke job builds from docker/locks/* and installs nothing). | |
| - name: Set up uv | |
| if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch' | |
| uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 | |
| with: | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| requirements.lock | |
| - name: Install project (dev + console + fhir + dicom + x12 + xml extras) | |
| if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch' | |
| run: | | |
| # [fhir] so the FHIR typed-tier tests (incl. the PHI-no-leak invariant, ADR 0022) actually | |
| # run their assertions in CI rather than importorskip-skipping (extra-less local runs skip). | |
| # [dicom] (pydicom + pynetdicom; ADR 0025) does the same for the DICOM codec + C-STORE SCP | |
| # tests, and lets mypy type-check transports/dicom.py + parsing/dicom/ against the real | |
| # (py.typed) libraries rather than treating them as Any. | |
| # [x12] (pyx12) + [xml] (lxml/xmlschema/signxml) likewise so the strict X12 validation (#32) | |
| # and the hardened XML/SOAP codec (#31) tests run their assertions instead of importorskip-skipping. | |
| # [webauthn] (ADR 0068) so the passkey ceremony tests run real verify_* assertions instead of | |
| # importorskip-skipping, and mypy sees the py.typed library rather than Any. | |
| # + the browser ops console as a second editable wheel (Option B) so the moved /ui tests | |
| # (tests/test_webui.py etc. now import messagefoundry_webconsole) run on this full-suite leg. | |
| # It is NOT a published extra yet (would break uv lock), so install it by path. | |
| uv pip install --system --constraint constraints.lock -e ".[dev,harness,fhir,dicom,x12,xml,webauthn]" -e packaging/messagefoundry-webconsole | |
| # ruff (lint + format) and mypy are PLATFORM-INDEPENDENT results — a ruff/format outcome is | |
| # identical on every OS, and mypy's platform-specific branches are covered by running it for | |
| # BOTH platforms on the cheap Linux leg (below). Running them on the 2x-billed Windows legs was | |
| # duplicate cost, so gate them to Linux. The Windows legs keep pytest — the one check whose | |
| # result genuinely differs by OS (real sockets, ProactorEventLoop, Windows service paths). | |
| # `.` — the WHOLE repo, matching the format check below and the pre-commit ruff hook, which has no | |
| # `exclude` and so lints every changed Python file. This was an explicit allow-list of six paths, | |
| # and the mismatch was not academic: harness/ (73 findings), tee/ (12), samples/ and docker/ were | |
| # linted by the hook and by nothing in CI, so touching any file there blocked the commit on errors | |
| # CI could not see and no CI run could ever have reported. Scope now lives in ONE place — | |
| # pyproject's [tool.ruff] extend-exclude — instead of being re-stated, differently, per tool. | |
| # tests/test_lint_scope_parity.py fails if the two drift apart again. | |
| - name: Lint (ruff) | |
| if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux' | |
| run: ruff check . | |
| - name: Format check (ruff) | |
| if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux' | |
| run: ruff format --check . | |
| # Two mypy passes on Linux so BOTH platforms' typed branches are checked without paying for a | |
| # Windows mypy: the default linux-platform run, plus an explicit --platform win32 run that types | |
| # the sys.platform=='win32' branches (ctypes.windll / DPAPI / service_control) the linux run | |
| # skips. Previously those were typed only by the mypy on the Windows legs. (All win32 code uses | |
| # stdlib ctypes — typeshed's win32-conditional stubs cover it; there are no win32-only deps — so | |
| # this resolves on a Linux runner. Verified: both passes clean on 204 files.) | |
| # `messagefoundry/tray` (the Windows notification-area service-manager, ADR 0113 — now shipped | |
| # inside the wheel) is a win32-only ctypes package, so it is EXCLUDED here and typed on the win32 | |
| # pass only: its ungated ctypes.WinDLL method calls are unreachable-and-skipped under the linux | |
| # platform but real under win32. Nothing in the engine imports it (it is a leaf entry point), so | |
| # --exclude cleanly drops it from the linux build. | |
| - name: Type-check (mypy, strict — linux platform) | |
| if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux' | |
| run: mypy messagefoundry messagefoundry_webconsole --exclude 'messagefoundry/tray/' | |
| # The win32 pass types messagefoundry/tray (now inside the messagefoundry package, so no separate | |
| # path argument) plus every other sys.platform=='win32' branch. | |
| - name: Type-check (mypy, strict — win32 platform) | |
| if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux' | |
| run: mypy --platform win32 messagefoundry | |
| # Step-level watchdog UNDER the job cap (#55): the windows-2022 leg intermittently | |
| # wedges ~25% in (a Windows ProactorEventLoop listener-teardown / socket wait that the shared | |
| # session event loop can't get past), emits no output for ~12 min, then the JOB cap CANCELS it | |
| # — a red ✗ with no stack and no named test. Two belts make that fail FAST and NAMED instead: | |
| # * `--timeout-method=thread` (the ONLY method on Windows — SIGALRM is POSIX-only) dumps ALL | |
| # thread stacks at the per-test cap (matrix `pytest_timeout`, 120s on these Windows legs), | |
| # naming the stuck frame. | |
| # * PYTHONFAULTHANDLER=1 + `-o faulthandler_timeout=` (matrix `fault_timeout`) is the belt that fires even when the | |
| # thread-timer CANNOT interrupt a main-thread C-level wait (a Proactor overlapped op / | |
| # blocking accept / subprocess.wait): pytest's faulthandler plugin arms a separate watchdog | |
| # thread that dumps the NATIVE stack of every thread at `fault_timeout` (150s here, 90s on | |
| # ubuntu -- always set ABOVE that leg's `pytest_timeout`, so the | |
| # per-test dump is attributed first; this is the last-resort stack when the thread method is | |
| # itself out-raced). NB the plugin exposes this as the `faulthandler_timeout` ini key, not a | |
| # `--faulthandler-*` CLI flag, so it is passed with `-o`; it only DUMPS (never kills), safe. | |
| # The step `timeout-minutes` (matrix `step_timeout`) is the outer backstop, held well under the | |
| # job's own cap, so a hang both watchdogs somehow miss still fails the STEP (not the whole job at | |
| # the silent cap). Keep the gap: the step must expire BEFORE the job, or the failure surfaces as | |
| # an uninformative job-level kill with no step attribution. | |
| # | |
| # THAT GUARANTEE COVERS THE FIRST GATED STEP ONLY, and saying otherwise would be a control resting | |
| # on a false premise. `Web console tests (pytest)` carries the SAME `step_timeout` but runs after | |
| # `Tests (pytest)`, so reaching it has already spent setup + `Tests`; its own cap cannot fire first | |
| # at any job_timeout worth setting (it would take ~39 min on ubuntu, ~73 min on Windows). A hang | |
| # THERE is still an unattributed job-level kill. Measured under `Tests (pytest)` below; the | |
| # structural fix is BACKLOG #344 proposal 5. | |
| # | |
| # It only works while the cap stays clear of a HEALTHY run. On 2026-07-31 the ubuntu leg finished | |
| # green in 775s against a 780s cap -- 5s of margin -- and the step was killed anyway, reported as | |
| # a failure on a run whose last line was "9598 passed, 855 skipped". A watchdog that cannot | |
| # separate "deadlocked" from "slow today" stops being a watchdog and becomes a coin flip, so the | |
| # ubuntu budget is raised here. Re-check the margin when the suite grows. | |
| # | |
| # It then happened again on WINDOWS, 2026-08-01, because the sentence that used to close this | |
| # paragraph asserted the Windows legs held "~2x headroom" and nobody re-derived it. PR #119 was | |
| # killed at 26:07 against the 26:00 cap with ZERO tests failing. What moved was the suite, not the | |
| # code under test: #74 landed tests/test_worktree_prune_merged.py (1,506 lines) and windows-2025 | |
| # went 19:35 -> 26:07 on the same branch. The claim was already false when it was written. | |
| # | |
| # Measured by timing the `Tests (pytest)` STEP, which is what step_timeout gates. NOT the job: the | |
| # job runs several minutes longer and is capped separately, and at least three sessions misread job | |
| # durations as step durations while triaging this (c53f752b's JOB ran 28:41 and PASSED, because | |
| # job cap 30 vs step cap 26). | |
| # | |
| # POOL: every run of THIS workflow created on 2026-08-01 UTC -- 70 runs. Enumerated with | |
| # `gh api --paginate` over a deliberately WIDER window and filtered locally on `created_at`, then | |
| # cross-checked against the narrow `?created=` query (same 70 ids, symmetric difference 0). Rows | |
| # are each leg's `Tests (pytest)` step, kept when THAT STEP concluded success. n is per leg. | |
| # | |
| # leg max passing step n old cap old margin | |
| # ubuntu-latest 12:31 42 19:00 1.518x | |
| # windows-2022 21:34 39 26:00 1.206x | |
| # windows-2025 25:51 36 26:00 1.006x <- NINE SECONDS | |
| # | |
| # THESE MAXIMA ARE LOWER BOUNDS, BECAUSE THE POOL IS RIGHT-CENSORED. Every run in it predates #131 | |
| # (28d186b5, landed 2026-08-02T00:35:28Z), so Windows ran under step 26:00 / job 30:00 and ubuntu | |
| # under 19:00 / 22:00. A run that wanted longer than its cap was KILLED at the cap and then -- | |
| # correctly, by the filter above -- dropped for not concluding success. So 25:51 is the largest | |
| # step that FIT in 26:00, not the largest the suite wants. Sizing a cap as a multiple of a | |
| # censored maximum under-provisions by construction, and that is exactly how "1.06x" read as | |
| # survivable right up until #119's leg was killed at 26:07. (#119 itself merged, 2026-08-02T01:45Z | |
| # -- what the cap killed was a run, not the PR. Saying a PR "died" here reads as never-landed and | |
| # had already propagated into docs/WORKTREES.md as exactly that claim.) | |
| # | |
| # THE CENSORED VALUES ARE VISIBLE IF YOU ASK FOR THEM, AND THEY MATTER. The jobs endpoint defaults | |
| # to `filter=latest`, which hides a killed attempt behind its passing re-run; `?filter=all` shows | |
| # both. #119's 26:07 is in this very pool that way (run 30717229521 attempt 1, `8c407fb5`, step | |
| # conclusion FAILURE) beside its 22:25 attempt-2 success -- a 3:42 spread on identical code, and | |
| # the first day AFTER the raise produced 26:23 TWICE, both concluding SUCCESS (runs 30728793103 | |
| # and 30731407003). So the population really does exceed the old 26:00 cap: the largest windows-2025 | |
| # `Tests (pytest)` execution observed to date is 26:23, not 25:51. | |
| # | |
| # THIS TABLE HAS BEEN WRONG TWICE. #131 published 12:27 / 18:39 / 24:35 over "11 passing runs"; | |
| # the first correction published the maxima above but over "101 runs" with n = 57 / 52 / 49. The | |
| # maxima re-derive exactly; BOTH pool sizes were unreproducible. Three mechanisms produced that, | |
| # all cheap to repeat: | |
| # | |
| # * THE POOL WAS A PAGE, NOT A QUESTION. #131's came from `gh run list --limit 20` -- a default | |
| # page size, reported as though it had been chosen. | |
| # * FILTERING ON *JOB* CONCLUSION DROPS THE TIGHTEST STEPS BY CONSTRUCTION. A step that nearly | |
| # exhausts step_timeout is the most likely to push its job into job_timeout, so the job is | |
| # cancelled while the step itself concluded success. Re-running this pool with a job-conclusion | |
| # filter reproduces #131's 24:35 exactly; the step-conclusion filter gives 25:51. Measure the | |
| # step; filter on the step. Two such rows exist on 2026-08-01 and one of them is the maximum. | |
| # * A COUNT NOBODY RE-DERIVED. Neither "101 runs" nor n = 57 / 52 / 49 is reproducible under any | |
| # pool definition tried. A table whose own point is "state your pool and your n" has to carry | |
| # an n the next reader can recompute -- that column is how they would tell it had rotted. | |
| # | |
| # Only the windows-2025 row was ever a maximum; #131's ubuntu 12:27 was that same run's ubuntu leg, | |
| # quoted as if it were that leg's worst case. Its 18:39 was not even that: that run's windows-2022 | |
| # step is 18:29, and 18:39 belongs to a different run entirely. | |
| # | |
| # State the MEASURED value, its POOL, its n and its DATE -- never a bare multiple. A multiple gives | |
| # the next reader no way to tell when it has rotted; a multiple without its pool cannot even be | |
| # rechecked. | |
| # | |
| # PROOF IT WAS THE CAP, NOT THE BRANCH. #119's windows-2025 leg was RE-RUN on the SAME commit | |
| # against the SAME 26:00 cap: attempt 1 was killed at the cap, attempt 2 concluded SUCCESS. Same | |
| # code, same config, same ceiling, two outcomes. The leg was not failing -- it was coin-flipping | |
| # against the cap, exactly the state the ubuntu note above names, and the reason "re-run it and | |
| # see" is not a diagnosis here. A green re-run at 26:00 does not mean the suite fits; it means | |
| # that runner was fast enough that time. | |
| # | |
| # 36:00 is 1.393x over the 25:51 that fit under the old cap, and 1.364x over 26:23, the largest | |
| # windows-2025 execution actually observed. Use the second number: a ratio against a censored | |
| # maximum flatters itself. | |
| # | |
| # BE HONEST ABOUT THE SPREAD RULE -- 36:00 DOES NOT MEET IT. An earlier revision argued "headroom | |
| # must exceed observed spread". Against real values that rule FAILS here: headroom is 36:00 - 26:23 | |
| # = 9:37, and the windows-2025 spread is 26:23 - 15:56 = 10:27. It would need roughly 37:00 to hold. | |
| # 36:00 is kept anyway, and the reason is stated rather than dressed up: this cap exists to catch a | |
| # whole-process DEADLOCK, not slowness (see the paragraph below), so 1.364x over the worst observed | |
| # run is ample for its actual job, and #131 already set this value. What the spread rule is good | |
| # for is telling you the margin is thinner than the ratio suggests. RE-DERIVE IF a windows-2025 | |
| # `Tests (pytest)` step is ever seen above 28:00 -- that is the trigger, not a calendar reminder. | |
| # | |
| # A ratio against one run says nothing about a distribution, and #119's leg was killed by the | |
| # distribution, not by its own duration. Both Windows legs take the same number: windows-2022 is the faster, so | |
| # sizing on windows-2025 only leaves it more room, and one value is one thing to re-derive. | |
| # | |
| # THE JOB CAP IS NOT A ROUNDING-UP OF THE STEP CAP, AND IT HAS FIRED. Two steps in this job carry | |
| # `step_timeout` -- `Tests (pytest)` and `Web console tests (pytest)` -- so the job can contain | |
| # 2 x step_timeout of gated work that step_timeout cannot bound. Note the caps here do NOT cover | |
| # that worst case and are not sized to (2 x 19 = 38 > 26 on ubuntu, 2 x 36 = 72 > 46 on Windows); | |
| # they are sized against the OBSERVED sum, which is a weaker guarantee. See the closing paragraph. | |
| # Observed for real on run 30724385719 (main @ 8f01cef8, 2026-08-01): | |
| # | |
| # Tests (pytest) 00:01:21 -> 00:27:12 25:51 SUCCESS (9s under the 26:00 cap) | |
| # Web console tests (pytest) 00:27:12 -> 00:30:14 CANCELLED | |
| # JOB 00:00:06 -> 00:30:19 30:13 CANCELLED <- job_timeout 30 fired | |
| # | |
| # A GREEN first step, then an unattributed job-level kill during the second -- exactly the failure | |
| # the nesting note above exists to prevent, arriving by the path that note does not consider. It | |
| # does NOT happen when a step is killed (that ends the job and skips what follows, so those | |
| # durations never sum); it happens when the first step PASSES near its budget. | |
| # | |
| # READ THAT EXHIBIT FOR ITS MECHANISM, NOT AS A VERDICT ON THE CAP IT REPLACED: it ran under the | |
| # retired 26/30 pair and would have passed under #131's 40:00. What condemns 40:00 is the | |
| # arithmetic below -- which is arithmetic, not an observed kill; no job has yet hit 40:00. | |
| # | |
| # job_timeout is therefore sized against BOTH gated steps plus setup, rather than as step_timeout | |
| # plus a constant. Each addend below is that addend's MEASURED MAXIMUM over the same pool, taken | |
| # over rows where `Tests` concluded success -- NOT over rows where both steps did, which silently | |
| # drops the very run that motivates this whole paragraph (its web-console step was cancelled) and | |
| # is the same censoring mistake as filtering the step table by job conclusion. Two earlier | |
| # revisions used a 0:41 setup (the MEDIAN), web-console values that were each only third-highest | |
| # on their leg, and a 1:04 Windows setup that the exhibit above contradicts on its face: | |
| # | |
| # leg step_timeout + web-console(max) + setup(max) old job new job | |
| # ubuntu 19:00 + 2:00 + 1:20 = 22:20 22:00 -> -0:20 26:00 -> +3:40 (1.16x) | |
| # W22 36:00 + 2:33 + 1:09 = 39:42 40:00 -> +0:18 46:00 -> +6:18 (1.16x) | |
| # W25 36:00 + 3:33 + 1:20 = 40:53 40:00 -> -0:53 46:00 -> +5:07 (1.13x) | |
| # | |
| # ubuntu and windows-2025 were ALREADY NEGATIVE -- ubuntu too, which an earlier revision had the | |
| # wrong way round; windows-2022 was the one row genuinely in the black. Windows takes one value, | |
| # sized on windows-2025 as the worse of the pair. The `+4` habit that produced 40 was Windows-only | |
| # and was carried through two cap changes unchecked (30/26 then 40/36); ubuntu has never been +4 -- | |
| # it went 15/13 to 22/19, so +2 then +3. Either way the number was derived from the OTHER BOUND | |
| # rather than from the work, which is the defect, not the particular constant. | |
| # | |
| # WHAT THIS DOES NOT FIX. The nesting invariant at the top of this note holds for `Tests (pytest)` | |
| # on every leg and for `Web console tests (pytest)` on NONE of them: reaching that step already | |
| # spends setup plus `Tests`, so its own cap can never fire first. Guaranteeing it would need | |
| # job_timeout above setup + 2 x step_timeout -- about 39:20 on ubuntu and 73:20 on Windows at the | |
| # measured setup maxima above, i.e. far beyond anything worth setting. A hang in the web-console | |
| # step therefore still surfaces as an unattributed job-level kill, which is the very failure the | |
| # invariant is there to prevent. The structural fix is to stop | |
| # the two steps sharing one budget (a 3:33 suite has no business holding 36:00) and is filed as | |
| # BACKLOG #344; this sizing only makes the job cap cover the sum. Re-derive it if either suite's | |
| # duration moves. | |
| # | |
| # AND KNOW WHAT IS NOT CAPPED AT ALL. `test` is the ONLY one of this file's ten jobs carrying any | |
| # `timeout-minutes`; the other nine -- `changes`, `ide`, `sqlserver-store`, `postgres-store`, | |
| # `load-test`, `load-test-sqlserver`, `windows-service-smoke`, `docker-smoke`, `ci-gate` -- run on | |
| # GitHub's 6-HOUR default. That is recorded here so the omission is known rather than assumed | |
| # deliberate; sizing them is out of scope for this change and belongs with BACKLOG #344. | |
| # | |
| # This cap is NOT what catches a hung test; pytest_timeout (120s) is, per test. The step cap only | |
| # catches a whole-process deadlock both in-process watchdogs miss, which is why it can sit well | |
| # above a healthy run. Sizing it tight buys no detection and costs false failures on green suites, | |
| # twice now. | |
| # | |
| # The remaining margin is a SHARED budget across every PR that lands and nothing accounts for it: | |
| # three PRs each adding a minute of Windows time reproduce #119's kill, individually blameless. | |
| # A mechanical guard for that is BACKLOG #344 item 1; the underlying slowness is #320. | |
| - name: Tests (pytest) | |
| if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch' | |
| timeout-minutes: ${{ matrix.step_timeout }} | |
| env: | |
| QT_QPA_PLATFORM: offscreen | |
| PYTHONFAULTHANDLER: "1" | |
| # Pass the matrix-derived timeout knobs through env instead of expanding ${{ }} straight into | |
| # the run: shell. The per-repo matrix is now built at runtime in `changes` (fromJSON), so zizmor | |
| # can no longer statically prove these are literal ints and flags the in-run expansion as | |
| # template-injection; env-passing removes the script sink. `defaults.run.shell: bash` applies on | |
| # every OS, so $VAR resolves the same on the Windows legs (Git Bash). | |
| FAULT_TIMEOUT: ${{ matrix.fault_timeout }} | |
| PYTEST_TIMEOUT: ${{ matrix.pytest_timeout }} | |
| run: pytest -q -o faulthandler_timeout="$FAULT_TIMEOUT" --timeout="$PYTEST_TIMEOUT" | |
| # The web console's OWN suite (Option B, ADR 0065): the moved /ui tests live in the package's | |
| # tests/ (the engine `pytest` above no longer collects them — engine testpaths = ["tests"]), so | |
| # run them as a SECOND step on the SAME leg. The package was installed editable in the install | |
| # step above (`-e packaging/messagefoundry-webconsole`), so both suites exercise the same | |
| # engine build. Its pyproject sets asyncio_mode="auto" + session loop scopes; the timeout flags | |
| # mirror the engine step (a hung ASGI test fails fast + named, not at the silent job cap). | |
| # SEAM MATRIX — KNOWN GAP, deliberately recorded rather than silently carried. This runs the | |
| # package against the ONE installed engine (whatever ENGINE_UI_SEAM is at HEAD). The original | |
| # comment said "only ENGINE_UI_SEAM==1 exists today ... when a seam 2 lands, expand this into a | |
| # matrix" — that trigger fired long ago and was never actioned: the console now supports a | |
| # RANGE (SUPPORTED_ENGINE_SEAMS), so the back-compat claim that an older engine still renders | |
| # is NOT exercised anywhere. Closing it means installing the MIN and MAX supported engine | |
| # builds and running the package suite against each. | |
| - name: Web console tests (pytest) | |
| if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch' | |
| timeout-minutes: ${{ matrix.step_timeout }} | |
| env: | |
| QT_QPA_PLATFORM: offscreen | |
| PYTHONFAULTHANDLER: "1" | |
| # env-passed (not a ${{ }} expansion into run:) — see the Tests step above for why. | |
| FAULT_TIMEOUT: ${{ matrix.fault_timeout }} | |
| PYTEST_TIMEOUT: ${{ matrix.pytest_timeout }} | |
| run: pytest packaging/messagefoundry-webconsole/tests -q -o faulthandler_timeout="$FAULT_TIMEOUT" --timeout="$PYTEST_TIMEOUT" | |
| # Build + type-check the VS Code extension, and run its integration tests. The Python jobs never | |
| # touch ide/, so a dep bump or IDE code change that breaks the bundle or the types would otherwise | |
| # pass CI unbuilt (this job exists because an esbuild bump merged "green" without anything having | |
| # compiled it). The build runs on both OSes; the @vscode/test-electron suite (which downloads and | |
| # launches a real headless VS Code to assert the extension activates and its commands register/run) | |
| # runs on the Windows leg — Electron launches headless there with no xvfb, and Windows is a | |
| # deployment target. | |
| ide: | |
| name: ide build (${{ matrix.os }}) | |
| # `ide` is NOT a required check and ci-gate does NOT `needs: ide`, so skipping it — or varying its | |
| # matrix per repo (below) — can't wedge any required context. It runs when it can be affected: | |
| # push-to-main + workflow_dispatch (full coverage), and PRs that touch ide/** or this workflow (the | |
| # `changes.ide` path-gate) — a pure-Python PR can't change the isolated npm project's result, so it's | |
| # skipped there. Not on the nightly cron: the engine heavy legs need that safety net, an isolated npm | |
| # build gated on its own paths does not. The per-repo os matrix below runs the 2x-billed | |
| # windows-latest electron leg here (this repo is public, so hosted runners are free) and drops it on | |
| # a fork, where those minutes come out of the fork owner's own allowance. | |
| needs: changes | |
| if: github.event_name == 'workflow_dispatch' || needs.changes.outputs.ide == 'true' | |
| runs-on: ${{ matrix.os }} | |
| strategy: | |
| fail-fast: false | |
| # Per-repo os matrix (built in `changes`): ubuntu + windows-latest on THIS repo, ubuntu-only on a | |
| # fork. The windows-latest leg is the 2x-billed one that runs the @vscode/test-electron integration | |
| # suite; this repo is public so hosted minutes are free, while on a fork they are billed to the | |
| # fork owner. `ide` is not a required check and ci-gate does not `needs` it, so dropping the | |
| # windows leg on a fork can't wedge anything. | |
| matrix: ${{ fromJSON(needs.changes.outputs.ide_matrix) }} | |
| defaults: | |
| run: | |
| working-directory: ide | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Set up Node | |
| uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 | |
| with: | |
| node-version: "24" | |
| cache: npm | |
| cache-dependency-path: ide/package-lock.json | |
| - name: Install (clean, from lockfile) | |
| run: npm ci | |
| - name: Type-check (tsc --noEmit) | |
| run: npm run typecheck | |
| - name: Bundle (esbuild) | |
| run: npm run compile | |
| # The vscode-free suites (the pure model layer: the engine link state + its two frozen boundary | |
| # allowlists, the settings-scope SEC-005 invariant, the graph/steps/HL7 models, …). They need no | |
| # Extension Host, so they run on EVERY leg — including the ubuntu-only leg a fork gets. | |
| # Before this step they ran NOWHERE on an ubuntu-only matrix: `npm test` is Windows-only, so the | |
| # entire node-side estate was type-checked and never executed there. | |
| # ADR 0110's allowlists are only "asserted in CI" because of this line. | |
| - name: Unit tests (node-side, no VS Code) | |
| run: npm run test:unit | |
| - name: Integration tests (headless VS Code) | |
| if: runner.os == 'Windows' | |
| run: npm test | |
| # Path filter for the container-pulling server-DB leg(s). A docs/unrelated PR shouldn't pull the SQL | |
| # Server image (mcr.microsoft.com anonymous pulls are rate-limited — repeated PRs flake on | |
| # "toomanyrequests"). This job decides whether the SQL Server leg needs to run: ALWAYS true on | |
| # workflow_dispatch (full coverage); FALSE on push-to-main (the container legs run nightly + | |
| # PR-path-gated, not per merge); on a PR, true only when store / cluster / server-DB tests / | |
| # this workflow changed. (NB: if `sqlserver-store` is later made a *required* check in branch | |
| # protection, a path-skipped run reports "skipped" — use a ruleset that treats skips as success, or | |
| # switch to an always-run job whose heavy steps are step-gated, since the service container starts | |
| # before steps.) | |
| changes: | |
| name: detect server-DB + docker changes | |
| runs-on: ubuntu-latest | |
| outputs: | |
| serverdb: ${{ steps.f.outputs.serverdb }} | |
| docker: ${{ steps.f.outputs.docker }} | |
| # `code` is the docs-only short-circuit signal: TRUE unless this PR touches ONLY non-code paths | |
| # (Markdown / docs / licence / editorconfig / git metadata). It gates the EXPENSIVE STEPS of the | |
| # required `test` legs (and the whole `ide` job) — the jobs still run + report their (required) | |
| # context, but on a docs-only PR they skip install+lint+type+test and go green in seconds. TRUE on | |
| # push-to-main / workflow_dispatch (full coverage); FALSE on the nightly `schedule` (load-legs-only | |
| # cron — the functional matrix must not re-run nightly). | |
| code: ${{ steps.f.outputs.code }} | |
| # `ide`: run the ide build job only when a PR touches ide/** or this workflow (true on | |
| # push/dispatch for full coverage; false on the nightly cron). Not a required check. | |
| ide: ${{ steps.f.outputs.ide }} | |
| # `matrix`: the `test` job's include list, per-repo — full ubuntu+windows matrix HERE (this repo is | |
| # public, so hosted runners are free; the self-hosted Windows runners are retired), ubuntu-only on | |
| # a fork. Consumed as `matrix: ${{ fromJSON(needs.changes.outputs.matrix) }}` in `test`. | |
| matrix: ${{ steps.f.outputs.matrix }} | |
| # `ide_matrix`: the `ide` job's os matrix, per-repo — ubuntu + windows-latest HERE, where the | |
| # 2x-billed VS Code electron leg runs free; ubuntu-only on a fork. Consumed as | |
| # `matrix: ${{ fromJSON(needs.changes.outputs.ide_matrix) }}` in `ide`. | |
| ide_matrix: ${{ steps.f.outputs.ide_matrix }} | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| fetch-depth: 0 | |
| - id: f | |
| # Hoist the two workflow expressions into env (zizmor: no `${{ }}` interpolated into the run body | |
| # — a branch name / base sha is attacker-influenceable on a fork PR, so it must arrive as data). | |
| env: | |
| EVENT_NAME: ${{ github.event_name }} | |
| BASE_SHA: ${{ github.event.pull_request.base.sha }} | |
| run: | | |
| # test matrix, per-repo — computed FIRST (independent of the event, before any early exit). | |
| # THIS repo (MEFORORG/MessageFoundry, the source since the cutover): full ubuntu + windows-2022 | |
| # + windows-2025 on FREE hosted runners — free because the repo is public. | |
| # Anywhere else (a FORK): the UBUNTU leg ONLY, so a contributor's own minutes are not spent on | |
| # the 2x-billed Windows legs. This branch used to mean "the private source repo"; that repo is | |
| # now an inactive archive with Actions disabled, so a fork is the only thing that reaches it. | |
| # The self-hosted Windows NucBox runners are retired either way. | |
| # `hosted` = the runner labels (an array; see the `test` job's runs-on). | |
| # $GITHUB_REPOSITORY is a built-in runner env var, read here as plain shell (NOT a workflow- | |
| # expression interpolation into the run body), so it is zizmor-safe and cannot be misparsed as | |
| # an Actions expression the way a literal double-brace token in a run: block would be. | |
| U='{"os":"ubuntu-latest","python-version":"3.14","hosted":["ubuntu-latest"],"job_timeout":26,"step_timeout":19,"pytest_timeout":60,"fault_timeout":90}' | |
| W22='{"os":"windows-2022","python-version":"3.14","hosted":["windows-2022"],"job_timeout":46,"step_timeout":36,"pytest_timeout":120,"fault_timeout":150}' | |
| W25='{"os":"windows-2025","python-version":"3.14","hosted":["windows-2025"],"job_timeout":46,"step_timeout":36,"pytest_timeout":120,"fault_timeout":150}' | |
| if [ "${GITHUB_REPOSITORY:-}" = "MEFORORG/MessageFoundry" ]; then | |
| echo "matrix={\"include\":[$U,$W22,$W25]}" >> "$GITHUB_OUTPUT" | |
| else | |
| echo "matrix={\"include\":[$U]}" >> "$GITHUB_OUTPUT" | |
| fi | |
| # ide matrix, per-repo (computed here with the test matrix, before any early exit): the | |
| # 2x-billed windows-latest electron leg runs only HERE, where hosted minutes are free; a fork | |
| # builds + type-checks the extension on ubuntu only. Consumed as | |
| # matrix: fromJSON(needs.changes.outputs.ide_matrix) in the ide job. (No shell vars here, so a | |
| # single-quoted literal JSON needs no escaping and carries no run-block workflow expression.) | |
| if [ "${GITHUB_REPOSITORY:-}" = "MEFORORG/MessageFoundry" ]; then | |
| echo 'ide_matrix={"os":["ubuntu-latest","windows-latest"]}' >> "$GITHUB_OUTPUT" | |
| else | |
| echo 'ide_matrix={"os":["ubuntu-latest"]}' >> "$GITHUB_OUTPUT" | |
| fi | |
| # Manual dispatch = full coverage: force every gate true so the whole workflow runs. | |
| if [ "$EVENT_NAME" = "workflow_dispatch" ]; then | |
| echo "serverdb=true" >> "$GITHUB_OUTPUT" | |
| echo "docker=true" >> "$GITHUB_OUTPUT" | |
| echo "code=true" >> "$GITHUB_OUTPUT" | |
| echo "ide=true" >> "$GITHUB_OUTPUT" | |
| exit 0 | |
| fi | |
| # Push-to-main: the FUNCTIONAL gate (code) stays true — the required test legs re-run on | |
| # the merge commit — but serverdb/docker are FALSE: the container legs (sqlserver-store, | |
| # postgres-store, docker-smoke) run nightly + on dispatch + via the PR path-gates, so a | |
| # push must no longer fire them. BOTH halves of that guard matter: their `if:` has no push | |
| # arm AND these outputs are false on push — restoring EITHER half re-adds the per-merge | |
| # cost. A merge touching those surfaces was already path-gated on its own PR minutes ago. | |
| if [ "$EVENT_NAME" = "push" ]; then | |
| echo "serverdb=false" >> "$GITHUB_OUTPUT" | |
| echo "docker=false" >> "$GITHUB_OUTPUT" | |
| echo "code=true" >> "$GITHUB_OUTPUT" | |
| echo "ide=true" >> "$GITHUB_OUTPUT" | |
| exit 0 | |
| fi | |
| if [ "$EVENT_NAME" = "schedule" ]; then | |
| # Nightly run: the heavy legs fire via their own `schedule` if-arm; these outputs stay | |
| # false so the functional matrix and the PR path-gate arms don't ALSO fire. | |
| echo "serverdb=false" >> "$GITHUB_OUTPUT" | |
| echo "docker=false" >> "$GITHUB_OUTPUT" | |
| echo "code=false" >> "$GITHUB_OUTPUT" | |
| echo "ide=false" >> "$GITHUB_OUTPUT" | |
| exit 0 | |
| fi | |
| # Remaining case: pull_request. Compute the path-based filters from the PR diff. | |
| changed=$(git diff --name-only "$BASE_SHA...HEAD") | |
| # NB: the tests/test_(...) alternation MUST list every file the sqlserver-store / | |
| # postgres-store pytest steps below run (the ADR 0057/0058/0060/0066 throughput levers AND | |
| # the ADR 0013/0016 RTE capture/re-ingress case); those suites are MEFOR_TEST_SQLSERVER-gated | |
| # and run ONLY on the server-DB legs, so a file not matched here edits with NO real SS/PG | |
| # coverage (it merely skips the gated leg). Keep this in sync with those steps | |
| # (docs/archive/throughput/throughput-build-plan.md). | |
| # The transports/ + config/wiring arms are the CAPTURE SURFACE: capture_response / reingress_to | |
| # are declared there (ADR 0013), and SQL Server / Postgres both declare supports_response_capture | |
| # + supports_pt_reingress True — so a regression in that surface is a SERVER-DB regression and must | |
| # pull these legs, not just the SQLite suite. | |
| if echo "$changed" | grep -qE '^(messagefoundry/store/|messagefoundry/pipeline/(cluster|wiring_runner|stage_dispatcher)|messagefoundry/config/(settings|wiring)|messagefoundry/transports/(base|database|dicomweb|fhir|http_auth|mllp|rest|soap|tcp|x12)|tests/test_(sqlserver|postgres|cluster|database_connector|database_source|pooled|stage_dispatcher|batch_claim|claim_fifo|inline_fast_path|seq_only_fifo|fifo_index|per_lane_wake|response_capture|reingress|x12_rte|shard_recovery|shard_cert|adr0071|adr0075|adr0114|adr0157|dr_server_seed_gate|dr7_server_config_only_backup|backup_runner_server_db|connscale|load_failover|load_runner)|\.github/workflows/ci\.yml)'; then | |
| echo "serverdb=true" >> "$GITHUB_OUTPUT" | |
| else | |
| echo "serverdb=false" >> "$GITHUB_OUTPUT" | |
| fi | |
| # The container image smoke runs on PRs that touch the image, the per-profile locks, packaging, | |
| # or this workflow (engine-wide regressions are already covered by the `test` job + push-to-main). | |
| if echo "$changed" | grep -qE '^(docker/|\.dockerignore|pyproject\.toml|requirements\.lock|\.github/workflows/ci\.yml)'; then | |
| echo "docker=true" >> "$GITHUB_OUTPUT" | |
| else | |
| echo "docker=false" >> "$GITHUB_OUTPUT" | |
| fi | |
| # The ide job (VS Code extension build+test) only depends on ide/** (its own npm project) | |
| # and this workflow; a pure-Python PR can't change its result, so don't run it there. | |
| if echo "$changed" | grep -qE '^(ide/|\.github/workflows/ci\.yml)'; then | |
| echo "ide=true" >> "$GITHUB_OUTPUT" | |
| else | |
| echo "ide=false" >> "$GITHUB_OUTPUT" | |
| fi | |
| # `code`: CONSERVATIVE docs-only detector. Default to running everything (code=true), and only | |
| # short-circuit when EVERY changed path matches the docs/non-code allowlist below. We compute it | |
| # by inverting: list the files that are NOT docs-only; if that list is empty (and the diff is | |
| # non-empty), it's a docs-only PR. Anything outside the allowlist — any *.py, ide/**, config, | |
| # lockfiles, OTHER workflows, scripts, samples, harness — counts as CODE and runs the full suite. | |
| # A path appears in `changed` even if only deleted/renamed, so a pure doc rename still short- | |
| # circuits, and a code deletion still runs. | |
| # Allowlisted (docs-only) paths: any *.md anywhere, docs/**, top-level LICENSE/NOTICE/AUTHORS, | |
| # .editorconfig, .gitignore/.gitattributes, .github/{ISSUE_TEMPLATE,PULL_REQUEST_TEMPLATE,...}.md. | |
| # Be conservative: when in doubt a path is CODE. Empty diff (shouldn't happen on a PR) => code=true. | |
| noncode='(\.md$|^docs/|^LICENSE$|^NOTICE$|^AUTHORS$|^\.editorconfig$|^\.gitignore$|^\.gitattributes$|^\.github/(ISSUE_TEMPLATE/|PULL_REQUEST_TEMPLATE))' | |
| if [ -z "$changed" ]; then | |
| echo "code=true" >> "$GITHUB_OUTPUT" | |
| elif echo "$changed" | grep -qvE "$noncode"; then | |
| # At least one changed path is NOT docs-only -> run the full suite. | |
| echo "code=true" >> "$GITHUB_OUTPUT" | |
| else | |
| # Every changed path is docs-only -> short-circuit the expensive steps. | |
| echo "code=false" >> "$GITHUB_OUTPUT" | |
| fi | |
| # SQL Server service-container leg: runs the gated suites that need a real SQL Server (Linux, so 1x | |
| # minutes) — the PRODUCTION store backend suite AND the PRODUCTION DATABASE connector round-trip | |
| # (ADR 0003/0010/0013). Runs NIGHTLY + on-demand + on PRs that touch the server-DB surface (gated | |
| # by `changes` above, so docs/unrelated PRs don't pull the rate-limited mcr image). It no longer | |
| # runs per push-to-main: a merge touching this surface was path-gated on its own PR minutes | |
| # earlier, so the per-merge re-run re-validated unchanged content (~16x/day); a regression that | |
| # somehow slips past a green PR (e.g. a semantic conflict between two concurrently-green PRs) is | |
| # caught by the nightly, fix-forward. Exercises the T-SQL + the live aioodbc round-trip the | |
| # SQLite / faked-driver tests can't. NOT a required check — ci-gate rolls it up (skip = pass). | |
| sqlserver-store: | |
| name: sql server (store + connector) ${{ matrix.label }} | |
| needs: changes | |
| # Nightly / on-demand, or a PR that touches the server-DB surface. `changes` sets serverdb=false | |
| # on push (both halves of the no-per-merge-run guard live there — see its comment) and on | |
| # schedule, where the first arm below already covers the run. | |
| if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || needs.changes.outputs.serverdb == 'true' | |
| runs-on: ubuntu-latest | |
| # Matrix the gated suite across every supported SQL Server major (2022 = 16.x, 2025 = 17.x) so a | |
| # version-specific T-SQL / engine regression can't merge. fail-fast: false keeps a 2025-only hiccup | |
| # from cancelling the 2022 leg (and vice-versa). The `changes` path-filter still gates this on PRs, so | |
| # matrixing only doubles the (rate-limited) mcr pulls on server-DB PRs + the nightly run. | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| include: | |
| - { image: "mcr.microsoft.com/mssql/server:2022-latest", label: "2022" } | |
| - { image: "mcr.microsoft.com/mssql/server:2025-latest", label: "2025" } | |
| services: | |
| mssql: | |
| image: ${{ matrix.image }} | |
| env: | |
| ACCEPT_EULA: "Y" | |
| MSSQL_SA_PASSWORD: "Str0ng_P@ssw0rd!" | |
| ports: | |
| - 1433:1433 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Set up Python | |
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.14" | |
| - name: Install Microsoft ODBC Driver 18 + sqlcmd | |
| run: | | |
| curl -fsSL https://packages.microsoft.com/keys/microsoft.asc \ | |
| | sudo tee /etc/apt/trusted.gpg.d/microsoft.asc > /dev/null | |
| curl -fsSL "https://packages.microsoft.com/config/ubuntu/$(. /etc/os-release; echo "$VERSION_ID")/prod.list" \ | |
| | sudo tee /etc/apt/sources.list.d/mssql-release.list > /dev/null | |
| sudo apt-get update | |
| sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18 mssql-tools18 unixodbc-dev | |
| - name: Wait for SQL Server and create the database | |
| run: | | |
| sqlcmd=/opt/mssql-tools18/bin/sqlcmd | |
| for i in $(seq 1 60); do # ~120s cap: 2025's first boot can run slower than 2022's | |
| if "$sqlcmd" -S localhost -U sa -P 'Str0ng_P@ssw0rd!' -C -Q "SELECT 1" > /dev/null 2>&1; then | |
| echo "SQL Server is up"; break | |
| fi | |
| echo "waiting for SQL Server ($i)"; sleep 2 | |
| done | |
| "$sqlcmd" -S localhost -U sa -P 'Str0ng_P@ssw0rd!' -C \ | |
| -Q "IF DB_ID('MessageFoundry') IS NULL CREATE DATABASE MessageFoundry" | |
| - name: Set up uv | |
| uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 | |
| with: | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| requirements.lock | |
| - name: Install project (dev + sqlserver extras) | |
| run: uv pip install --system --constraint constraints.lock -e ".[dev,sqlserver]" | |
| - name: Run the SQL Server store suite (against the real DB) | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_BACKEND: sqlserver | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| # The service container uses a self-signed cert, so trust_server_certificate=true is | |
| # required to connect. The store's TLS-hardening guard refuses that MITM-able combination | |
| # unless this escape is set — exactly the trusted-network dev/test case it exists for. | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # Dump the native (C-level) frame if a retry still crashes (faulthandler only DUMPS, never kills). | |
| PYTHONFAULTHANDLER: "1" | |
| # pyodbc 5.3.0 + py3.14 native-crash retry (#973 follow-up, upstream pyodbc#1459): re-runs ONLY on a 139/134 crash exit, never on exit 1, so it can't mask a regression. See scripts/ci/retry-native-crash.sh. | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v tests/test_sqlserver_store.py | |
| - name: Run the SQL Server coordinator suite (active-passive HA leader election) | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_BACKEND: sqlserver | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| # Same trusted-network TLS escape as the store suite (self-signed container cert): ODBC Driver | |
| # 18 defaults to Encrypt=yes + verify, so trust_server_certificate=true is required to connect, | |
| # and the store's TLS-hardening guard needs MEFOR_ALLOW_INSECURE_TLS to permit it. Exercises the | |
| # SqlServerCoordinator leader-lease (MERGE WITH(HOLDLOCK) acquire/renew/take-over, self-fence, | |
| # membership) against the real T-SQL the faked-driver tests can't. | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # Dump the native (C-level) frame if a retry still crashes (faulthandler only DUMPS, never kills). | |
| PYTHONFAULTHANDLER: "1" | |
| # pyodbc 5.3.0 + py3.14 native-crash retry (#973 follow-up, upstream pyodbc#1459): re-runs ONLY on a 139/134 crash exit, never on exit 1, so it can't mask a regression. See scripts/ci/retry-native-crash.sh. | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v tests/test_sqlserver_coordinator.py | |
| - name: Run the SQL Server failover suite (real TTL takeover + 2-node lifecycle) | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_BACKEND: sqlserver | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| # Same trusted-network TLS escape (self-signed container cert). Drives two coordinators sharing | |
| # the store through real DB-clock lease expiry + the full start()/stop() election + failover. | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # Dump the native (C-level) frame if a retry still crashes (faulthandler only DUMPS, never kills). | |
| PYTHONFAULTHANDLER: "1" | |
| # pyodbc 5.3.0 + py3.14 native-crash retry (#973 follow-up, upstream pyodbc#1459): re-runs ONLY on a 139/134 crash exit, never on exit 1, so it can't mask a regression. See scripts/ci/retry-native-crash.sh. | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v tests/test_cluster_failover_sqlserver.py | |
| - name: Run the DATABASE connector round-trip (against the real DB) | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| # The connector suite reuses the same MEFOR_STORE_* connection env + the trusted-network TLS | |
| # escape (self-signed container cert), exactly like the store suite above. This is the live | |
| # aioodbc/ODBC round-trip backing the connector's "production" status (PR D). | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # Dump the native (C-level) frame if a retry still crashes (faulthandler only DUMPS, never kills). | |
| PYTHONFAULTHANDLER: "1" | |
| # pyodbc 5.3.0 + py3.14 native-crash retry (#973 follow-up, upstream pyodbc#1459): re-runs ONLY on a 139/134 crash exit, never on exit 1, so it can't mask a regression. See scripts/ci/retry-native-crash.sh. | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v tests/test_database_connector_integration.py | |
| - name: Run the failover-LOAD test (two nodes, SIGKILL the primary mid-load, against the real DB) | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_BACKEND: sqlserver | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| # Gate #3 capstone + the FIRST LIVE PROOF of the SQL Server on-promotion recovery under a real | |
| # crash: two `serve` nodes share this container, the harness SIGKILLs the primary mid-load and | |
| # hard-asserts the host-independent CONFORMANCE invariants (zero acknowledged loss, per-lane FIFO, | |
| # no split-brain, bounded duplicates, a recovered + drained pipeline). Recovery *time* is reported, | |
| # not hard-gated (host-variable — see docs/LOAD-TESTING.md). This run guards the #285 claim-FIFO | |
| # fix. The orchestrator spawns the nodes with auth off + the cluster env; they inherit MEFOR_STORE_*. | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # Dump the native (C-level) frame if a retry still crashes (faulthandler only DUMPS, never kills). | |
| PYTHONFAULTHANDLER: "1" | |
| # pyodbc 5.3.0 + py3.14 native-crash retry (#973 follow-up, upstream pyodbc#1459): re-runs ONLY on a 139/134 crash exit, never on exit 1, so it can't mask a regression. See scripts/ci/retry-native-crash.sh. | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v tests/test_load_failover_sqlserver.py | |
| - name: Run the throughput-lever backend invariants on real SQL Server (B1 inline, B2 batch-claim, ...) | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_BACKEND: sqlserver | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # If every retry below still crashes, dump the native (C-level) frame so the upstream | |
| # pyodbc bug can be diagnosed from the log. faulthandler only DUMPS — it never kills. | |
| PYTHONFAULTHANDLER: "1" | |
| # The staged-pipeline throughput levers (ADR 0057 inline fast-path, ADR 0058 batch-claim, ...) carry | |
| # reliability-core invariant tests — crash-replay, poison-bound, and the per-lane-FIFO locked-head- | |
| # BLOCKS (#285) gate — that are MEFOR_TEST_SQLSERVER-gated but live in their OWN files, so the named | |
| # suites above never picked them up and they ran only on SQLite. Run them here against the real | |
| # backend. APPEND each new lever's test file per docs/archive/throughput/throughput-build-plan.md. | |
| # | |
| # NOT only throughput levers: this step is the catch-all for ANY MEFOR_TEST_SQLSERVER-gated file | |
| # the named suites above do not collect. CI runs an explicit file LIST here, not the whole suite, | |
| # so a new gated file that nobody appends never runs in CI at all — it just reports as skipped on | |
| # the plain legs, which reads identical to passing. `tests/test_sqlserver_legacy_outbox_migration.py` | |
| # (ASVS 14.2.7) is here for exactly that reason: it verifies a one-way schema migration that can | |
| # only be observed against a real server. | |
| # | |
| # RETRY WRAPPER: pyodbc 5.3.0 intermittently segfaults in its C parameter-binding path under | |
| # py3.14 against the SQL Server 2025 container (upstream mkleehammer/pyodbc#1459, unfixed; 5.3.0 | |
| # is the newest pyodbc and the first with py3.14 wheels, so there is nothing to bump to). The | |
| # crash kills the interpreter, so pytest-rerunfailures can't recover it — retry the WHOLE step at | |
| # the process level, but ONLY on a native-crash exit (139 SIGSEGV / 134 SIGABRT); a real failure | |
| # (exit 1) is NOT retried, so this never masks a regression. Remove the wrapper when #1459 ships a | |
| # fix and the pyodbc floor moves to it. See scripts/ci/retry-native-crash.sh. | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v | |
| tests/test_inline_fast_path.py | |
| tests/test_batch_claim_fifo.py | |
| tests/test_batch_claim_worker.py | |
| tests/test_batch_claim_locking.py | |
| tests/test_claim_fifo_heads.py | |
| tests/test_seq_only_fifo.py | |
| tests/test_fifo_index_migration.py | |
| tests/test_per_lane_wake.py | |
| tests/test_stage_dispatcher.py | |
| tests/test_pooled_rider.py | |
| tests/test_sqlserver_legacy_outbox_migration.py | |
| - name: Run the RTE capture / re-ingress loop on real SQL Server (ADR 0013/0016) | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_BACKEND: sqlserver | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| PYTHONFAULTHANDLER: "1" | |
| # SQL Server has declared supports_response_capture + supports_pt_reingress True since ADR 0013 | |
| # landed, and the store suite above proves the T-SQL — but the CONNECTOR->RUNNER half of the RTE | |
| # loop (a capturing outbound -> Stage.RESPONSE -> the response worker re-ingressing into a Loopback | |
| # inbound) had only ever run on SQLite, so a server-DB regression in it could not be caught. The | |
| # backend-parametrized case in this file closes that gap on the real backend. | |
| # Same pyodbc 5.3.0 + py3.14 native-crash retry as the steps above (upstream pyodbc#1459). | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v tests/test_x12_rte.py | |
| - name: Run the engine-shard + statement-dispatch suites on real SQL Server | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_BACKEND: sqlserver | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| PYTHONFAULTHANDLER: "1" | |
| # Every file here is MEFOR_TEST_SQLSERVER-gated at MODULE level, so before this step they were | |
| # collected nowhere and reported nothing — not a pass, not a skip, on any trigger. That covered | |
| # engine-shard crash recovery and the shard TLS-cert ladder (ADR 0037 over the ADR 0063 unified | |
| # store), the ADR 0071 statement dispatch/fusion wiring, the ADR 0075 batch backend, the ADR 0114 | |
| # live claim procedure, and the synchronous handoff path. tests/test_serverdb_ci_coverage.py now | |
| # fails if a module-gated suite is added without being named here. | |
| # Same pyodbc 5.3.0 + py3.14 native-crash retry as the steps above (upstream pyodbc#1459). | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v | |
| tests/test_shard_recovery_sqlserver.py | |
| tests/test_shard_cert_sqlserver.py | |
| tests/test_adr0071_dispatch_wiring_sqlserver.py | |
| tests/test_adr0071_fused_callables_sqlserver.py | |
| tests/test_adr0075_batch_sqlserver.py | |
| tests/test_adr0114_claim_proc_live.py | |
| tests/test_sqlserver_sync_handoff.py | |
| tests/test_database_source_integration.py | |
| - name: Run the DR seed-gate + backup suites on real SQL Server | |
| env: | |
| MEFOR_TEST_SQLSERVER: "1" | |
| MEFOR_STORE_BACKEND: sqlserver | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| PYTHONFAULTHANDLER: "1" | |
| # ADR 0048/0049 disaster recovery, SQL Server side: the seed gate that must REFUSE to activate a | |
| # standby onto a non-empty store, the .mfbak backup runner against a real server DB, and the DR7 | |
| # config-only backup + DbaDelegatedError path. Module-gated and never executed; a DR control that | |
| # has never run is a DR control that has never been proven. | |
| # Same pyodbc 5.3.0 + py3.14 native-crash retry as the steps above (upstream pyodbc#1459). | |
| run: >- | |
| bash scripts/ci/retry-native-crash.sh | |
| pytest -v | |
| tests/test_dr_server_seed_gate_sqlserver.py | |
| tests/test_backup_runner_server_db_sqlserver.py | |
| tests/test_dr7_server_config_only_backup_sqlserver.py | |
| # Postgres store backend (Track B): run the gated store suite against a real PostgreSQL service | |
| # container (Linux, so 1x minutes). Runs NIGHTLY + on-demand (workflow_dispatch — use | |
| # `gh workflow run ci.yml --ref <branch>` to exercise it on a feature branch) + on PRs that touch | |
| # the server-DB surface (NEW: the same `changes` path-gate as sqlserver-store — a store/cluster PR | |
| # now gets real-Postgres coverage PRE-merge, which it never had). No longer per push-to-main; same | |
| # rationale as sqlserver-store above. This is what actually drives the SELECT ... FOR UPDATE SKIP | |
| # LOCKED + advisory-lock concurrency the SQLite tests can't reach. | |
| postgres-store: | |
| name: postgres store | |
| needs: changes | |
| # Nightly / on-demand / server-DB PRs (serverdb=false on push and schedule — see `changes`). | |
| if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || needs.changes.outputs.serverdb == 'true' | |
| runs-on: ubuntu-latest | |
| services: | |
| postgres: | |
| image: postgres:16 | |
| env: | |
| POSTGRES_PASSWORD: mefor | |
| POSTGRES_DB: messagefoundry | |
| ports: | |
| - 5432:5432 | |
| # Gate steps on readiness so the suite never races the container's first-boot init. | |
| options: >- | |
| --health-cmd pg_isready | |
| --health-interval 10s | |
| --health-timeout 5s | |
| --health-retries 5 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Set up Python | |
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.14" | |
| - name: Set up uv | |
| uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 | |
| with: | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| requirements.lock | |
| - name: Install project (dev + postgres extras) | |
| run: uv pip install --system --constraint constraints.lock -e ".[dev,postgres]" | |
| - name: Run the Postgres store suite (against the real DB) | |
| env: | |
| MEFOR_TEST_POSTGRES: "1" | |
| MEFOR_STORE_BACKEND: postgres | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "5432" | |
| MEFOR_STORE_DATABASE: messagefoundry | |
| MEFOR_STORE_USERNAME: postgres | |
| MEFOR_STORE_PASSWORD: mefor | |
| # The service container speaks plaintext (no TLS), so encrypt=false; the store's | |
| # TLS-hardening guard refuses that MITM-able combination unless this trusted-network | |
| # dev/test escape is set. | |
| MEFOR_STORE_ENCRYPT: "false" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| run: pytest tests/test_postgres_store.py tests/test_adr0157_postgres_fence.py -v | |
| - name: Run the failover-LOAD test (two nodes, SIGKILL the primary mid-load, against the real DB) | |
| env: | |
| MEFOR_TEST_POSTGRES: "1" | |
| MEFOR_STORE_BACKEND: postgres | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "5432" | |
| MEFOR_STORE_DATABASE: messagefoundry | |
| MEFOR_STORE_USERNAME: postgres | |
| MEFOR_STORE_PASSWORD: mefor | |
| # Gate #3 capstone: two `serve` nodes share this container; the harness SIGKILLs the primary | |
| # mid-load and hard-asserts the host-independent CONFORMANCE invariants (zero acknowledged loss, | |
| # per-lane FIFO, no split-brain, bounded duplicates, a recovered + drained pipeline). Recovery | |
| # time is reported, not hard-gated. The orchestrator spawns the nodes with auth off + the cluster | |
| # env itself; they inherit MEFOR_STORE_* (incl. the plaintext-TLS escape) from here. | |
| MEFOR_STORE_ENCRYPT: "false" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| run: pytest tests/test_load_failover_postgres.py -v | |
| - name: Run the throughput-lever backend invariants on real Postgres (B1 inline, B2 batch-claim, ...) | |
| env: | |
| MEFOR_TEST_POSTGRES: "1" | |
| MEFOR_STORE_BACKEND: postgres | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "5432" | |
| MEFOR_STORE_DATABASE: messagefoundry | |
| MEFOR_STORE_USERNAME: postgres | |
| MEFOR_STORE_PASSWORD: mefor | |
| MEFOR_STORE_ENCRYPT: "false" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # The throughput levers' MEFOR_TEST_POSTGRES-gated invariants (esp. the FOR-UPDATE no-SKIP-LOCKED | |
| # locked-head-BLOCKS twin of #285, and the batch FIFO/crash tests) live in their own files. APPEND | |
| # each new lever's test file per docs/archive/throughput/throughput-build-plan.md. | |
| run: >- | |
| pytest -v | |
| tests/test_inline_fast_path.py | |
| tests/test_batch_claim_fifo.py | |
| tests/test_batch_claim_worker.py | |
| tests/test_batch_claim_locking.py | |
| tests/test_claim_fifo_heads.py | |
| tests/test_seq_only_fifo.py | |
| tests/test_fifo_index_migration.py | |
| tests/test_per_lane_wake.py | |
| tests/test_stage_dispatcher.py | |
| tests/test_pooled_rider.py | |
| - name: Run the connection-scale pool-wait smoke on real Postgres (B11 wall #2, tiny pool) | |
| env: | |
| MEFOR_TEST_POSTGRES: "1" | |
| MEFOR_STORE_BACKEND: postgres | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "5432" | |
| MEFOR_STORE_DATABASE: messagefoundry | |
| MEFOR_STORE_USERNAME: postgres | |
| MEFOR_STORE_PASSWORD: mefor | |
| MEFOR_STORE_ENCRYPT: "false" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # FORCE a below-default pool (4 connections) so the perf_counter-measured acquire-WAIT | |
| # histogram (the PRIMARY pool-wait signal) is non-trivial even at small N: ~3N inbound workers | |
| # contend for 4 connections, so acquires actually queue and the percentiles populate. The | |
| # default pool (40) would mask the wall. 4 (not 1-2) is deliberate: GET /status runs | |
| # db_status()'s four sequential COUNT(*) queries on every poll, and an extremely tiny pool | |
| # starves those acquires past the poller's HTTP timeout under the empty-claim herd on a slow | |
| # runner — every poll then fails and the drain loops burn the test budget (the PR #675 hang). | |
| # This still gives the B11 pool-wait instrumentation real regression coverage SQLite (no pool) | |
| # cannot — the connscale harness OWNS the engine subprocess and inherits this env. | |
| MEFOR_STORE_POOL_SIZE: "4" | |
| run: pytest tests/test_connscale_postgres.py -v | |
| - name: Run the RTE capture / re-ingress loop on real Postgres (ADR 0013/0016) | |
| env: | |
| MEFOR_TEST_POSTGRES: "1" | |
| MEFOR_STORE_BACKEND: postgres | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "5432" | |
| MEFOR_STORE_DATABASE: messagefoundry | |
| MEFOR_STORE_USERNAME: postgres | |
| MEFOR_STORE_PASSWORD: mefor | |
| MEFOR_STORE_ENCRYPT: "false" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # The Postgres twin of the sqlserver-store leg's RTE step (same backend-parametrized case, the | |
| # `postgres` param). Postgres declares the same two capability flags True, so the same | |
| # connector->runner gap existed here. | |
| run: pytest -v tests/test_x12_rte.py | |
| - name: Run the failover + engine-shard recovery suites on real Postgres | |
| env: | |
| MEFOR_TEST_POSTGRES: "1" | |
| MEFOR_STORE_BACKEND: postgres | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "5432" | |
| MEFOR_STORE_DATABASE: messagefoundry | |
| MEFOR_STORE_USERNAME: postgres | |
| MEFOR_STORE_PASSWORD: mefor | |
| MEFOR_STORE_ENCRYPT: "false" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # These are MEFOR_TEST_POSTGRES-gated at MODULE level, so until this step existed they were | |
| # collected nowhere and reported nothing — not a pass, not a skip. The SQL Server failover twin | |
| # (test_cluster_failover_sqlserver.py) has run for some time; its Postgres counterpart never had. | |
| # Engine-shard crash recovery (ADR 0037 over the ADR 0063 unified store) was dark on BOTH server | |
| # backends. tests/test_serverdb_ci_coverage.py now fails if a module-gated suite is added without | |
| # being named here. | |
| run: >- | |
| pytest -v | |
| tests/test_cluster_failover_postgres.py | |
| tests/test_shard_recovery_postgres.py | |
| - name: Run the DR seed-gate + backup suites on real Postgres | |
| env: | |
| MEFOR_TEST_POSTGRES: "1" | |
| MEFOR_STORE_BACKEND: postgres | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "5432" | |
| MEFOR_STORE_DATABASE: messagefoundry | |
| MEFOR_STORE_USERNAME: postgres | |
| MEFOR_STORE_PASSWORD: mefor | |
| MEFOR_STORE_ENCRYPT: "false" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" | |
| # ADR 0048/0049 disaster recovery: the seed gate that must REFUSE to activate a standby onto a | |
| # non-empty store, the .mfbak backup runner against a real server DB, and the DR7 config-only | |
| # backup + DbaDelegatedError path. All three are module-gated and were never executed against | |
| # PostgreSQL; a DR control that has never run is a DR control that has never been proven. | |
| run: >- | |
| pytest -v | |
| tests/test_dr_server_seed_gate_postgres.py | |
| tests/test_backup_runner_server_db_postgres.py | |
| tests/test_dr7_server_config_only_backup_postgres.py | |
| # Headless load test (Track B / throughput): serve the synthetic high-fan-out load config (auth | |
| # off, small fan-out) and drive the smoke profile through the real `python -m harness --load` CLI, | |
| # asserting zero message loss + all SLOs (exit 0) and uploading the JSON/CSV report. Skipped on PRs | |
| # to save spend — the in-process load integration test (tests/test_load_runner.py, in the `test` | |
| # job) is the PR gate. Runs on push to main + on-demand (workflow_dispatch). Heavier profiles | |
| # (fanout-baseline / soak) and the SQLite-vs-Postgres comparison are run manually — see | |
| # docs/LOAD-TESTING.md. | |
| load-test: | |
| name: load test (smoke, sqlite) | |
| # Nightly schedule + on-demand only (was: every push to main). The PR gate is the in-process | |
| # tests/test_load_runner.py in the `test` job; push-to-main keeps the full functional matrix | |
| # (the server-DB suites are nightly + PR-path-gated too, as of the same cost pass). This | |
| # multi-minute serve+load run no longer fires once per commit. NOT a required check, | |
| # so dropping it from push is safe; ci-gate `needs` it but counts a `skipped` leg as a pass. | |
| if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Set up Python | |
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.14" | |
| - name: Set up uv | |
| uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 | |
| with: | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| requirements.lock | |
| - name: Install project (dev — engine + httpx client; no Qt needed for the headless load path) | |
| run: uv pip install --system --constraint constraints.lock -e ".[dev]" | |
| - name: Serve the load config (auth off) and run the smoke profile | |
| env: | |
| MEFOR_SECURITY_REQUIRE_SIGN_IN: "false" # the load CLI polls /stats etc. without a bearer token | |
| MEFOR_LOAD_FANOUT: "8" | |
| MEFOR_LOAD_RESULTS_FANOUT: "2" | |
| MEFOR_LOAD_TRANSFORM: edit | |
| MEFOR_LOAD_SINK_PORT: "2700" # matches --sink-port below | |
| # GIVEN 1 (ADR 0148): the default env `dev` now derives PHI, so a bare `serve --env dev` runs the | |
| # secure PHI posture (secure-by-default). This leg validates that PHI+enforce serves green — so | |
| # provision the production posture rather than opting out to synthetic: a store encryption key | |
| # (minted at runtime below — never a committed literal), bounded PHI-body retention windows, and a | |
| # locked-down egress allowlisting the synthetic loopback sink the load graph delivers to. The | |
| # security-notification gate is skipped because auth is off (no accounts to notify). | |
| MEFOR_SECURITY_DELETE_MESSAGE_BODIES_AFTER_DAYS: "30" # bound inbound PHI bodies at rest | |
| MEFOR_RETENTION_DEAD_LETTER_DAYS: "30" # bound dead-lettered PHI bodies at rest | |
| MEFOR_SECURITY_BLOCK_UNLISTED_OUTBOUND: "true" # deny-by-default egress | |
| MEFOR_EGRESS_ALLOWED_MLLP: "127.0.0.1" # the load graph fans out to the loopback correlation sink | |
| run: | | |
| set -o pipefail | |
| mkdir -p out/load | |
| # Mint a throwaway AES-256 store key at runtime (PHI-at-rest must never be plaintext); never a | |
| # committed literal (gitleaks). 32 random bytes, base64 — the MEFOR_STORE_ENCRYPTION_KEY format. | |
| export MEFOR_STORE_ENCRYPTION_KEY="$(python -c 'import base64, os; print(base64.b64encode(os.urandom(32)).decode())')" | |
| python -m messagefoundry serve --config harness/config/load --env dev --db ./load-ci.db \ | |
| --host 127.0.0.1 --port 8765 > engine.log 2>&1 & | |
| engine_pid=$! | |
| for _ in $(seq 1 60); do | |
| curl -sf http://127.0.0.1:8765/health > /dev/null && break || sleep 0.5 | |
| done | |
| set +e | |
| python -m harness --load smoke --engine http://127.0.0.1:8765 --sink-port 2700 \ | |
| --report-json out/load/smoke-ci.json --report-csv out/load/smoke-ci.csv | |
| rc=$? | |
| set -e | |
| kill "$engine_pid" 2>/dev/null || true | |
| echo "===== engine log (tail) ====="; tail -50 engine.log || true | |
| exit $rc | |
| - name: Upload load report | |
| if: always() | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 | |
| with: | |
| name: load-report | |
| path: out/load/ | |
| if-no-files-found: ignore | |
| # SQL Server throughput smoke: the same zero-loss fan-out wiring check as `load test (smoke, sqlite)`, | |
| # but driven through the **SQL Server store** end-to-end (ingress→routed→outbound→delivered, each a | |
| # committed server-DB round-trip) on a real mssql service container. This is the load path that caught | |
| # the routed-stage `handler_name`-drop regression the in-process tests missed. Uses the `smoke-sqlserver` | |
| # profile (correctness SLOs strict; drain bound sized for a server DB). Skipped on PRs to save spend — | |
| # runs on push to main + on-demand (workflow_dispatch). Heavier SQL Server profiles run manually. | |
| load-test-sqlserver: | |
| name: load test (smoke, sqlserver) ${{ matrix.label }} | |
| # Nightly schedule + on-demand only (was: every push to main); same rationale as `load-test`. The | |
| # end-to-end SQL Server store path is still smoked nightly (x2 majors) + on demand, and the SQL | |
| # Server functional coverage stays in `sqlserver-store` (nightly + server-DB-PR path-gate). NOT | |
| # required; ci-gate treats skip as pass. | |
| if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' | |
| runs-on: ubuntu-latest | |
| # Smoke the end-to-end SQL Server store path on every supported major (2022/2025). Push/dispatch-only, | |
| # so no PR cost; fail-fast: false keeps the two legs independent. | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| include: | |
| - { image: "mcr.microsoft.com/mssql/server:2022-latest", label: "2022" } | |
| - { image: "mcr.microsoft.com/mssql/server:2025-latest", label: "2025" } | |
| services: | |
| mssql: | |
| image: ${{ matrix.image }} | |
| env: | |
| ACCEPT_EULA: "Y" | |
| MSSQL_SA_PASSWORD: "Str0ng_P@ssw0rd!" | |
| ports: | |
| - 1433:1433 | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Set up Python | |
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: "3.14" | |
| - name: Install Microsoft ODBC Driver 18 + sqlcmd | |
| run: | | |
| curl -fsSL https://packages.microsoft.com/keys/microsoft.asc \ | |
| | sudo tee /etc/apt/trusted.gpg.d/microsoft.asc > /dev/null | |
| curl -fsSL "https://packages.microsoft.com/config/ubuntu/$(. /etc/os-release; echo "$VERSION_ID")/prod.list" \ | |
| | sudo tee /etc/apt/sources.list.d/mssql-release.list > /dev/null | |
| sudo apt-get update | |
| sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18 mssql-tools18 unixodbc-dev | |
| - name: Wait for SQL Server and create the database (RCSI on, like the store suite) | |
| run: | | |
| sqlcmd=/opt/mssql-tools18/bin/sqlcmd | |
| for i in $(seq 1 60); do # ~120s cap: 2025's first boot can run slower than 2022's | |
| if "$sqlcmd" -S localhost -U sa -P 'Str0ng_P@ssw0rd!' -C -Q "SELECT 1" > /dev/null 2>&1; then | |
| echo "SQL Server is up"; break | |
| fi | |
| echo "waiting for SQL Server ($i)"; sleep 2 | |
| done | |
| "$sqlcmd" -S localhost -U sa -P 'Str0ng_P@ssw0rd!' -C \ | |
| -Q "IF DB_ID('MessageFoundry') IS NULL CREATE DATABASE MessageFoundry; ALTER DATABASE MessageFoundry SET READ_COMMITTED_SNAPSHOT ON WITH ROLLBACK IMMEDIATE;" | |
| - name: Set up uv | |
| uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 | |
| with: | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| requirements.lock | |
| - name: Install project (dev + sqlserver extras) | |
| run: uv pip install --system --constraint constraints.lock -e ".[dev,sqlserver]" | |
| - name: Serve on the SQL Server store and run the smoke-sqlserver profile | |
| env: | |
| MEFOR_SECURITY_REQUIRE_SIGN_IN: "false" # the load CLI polls /stats etc. without a bearer token | |
| MEFOR_LOAD_FANOUT: "8" | |
| MEFOR_LOAD_RESULTS_FANOUT: "2" | |
| MEFOR_LOAD_TRANSFORM: edit | |
| MEFOR_LOAD_SINK_PORT: "2700" # matches --sink-port below | |
| MEFOR_STORE_BACKEND: sqlserver # no --db: the backend + connection come from MEFOR_STORE_* | |
| MEFOR_STORE_SERVER: localhost | |
| MEFOR_STORE_PORT: "1433" | |
| MEFOR_STORE_DATABASE: MessageFoundry | |
| MEFOR_STORE_AUTH: sql | |
| MEFOR_STORE_USERNAME: sa | |
| MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!" | |
| MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true" | |
| MEFOR_ALLOW_INSECURE_TLS: "1" # trusted-network escape for the container's self-signed cert | |
| # DECLARE THE DATA CLASS HONESTLY (ADR 0148 GIVEN 1 + docs/SECURITY-LOOSENING.md | |
| # "`handles_real_patient_data = false`"). This leg processes harness-GENERATED synthetic HL7 on | |
| # a throwaway service container; asserting PHI was a false declaration about the instance, and | |
| # the register names exactly this case as acceptable: "a CI runner ... that only ever processes | |
| # synthetic / sample HL7". Since GIVEN 1 the built-in `dev` env derives PHI, so a genuinely | |
| # throwaway CI box must now set this EXPLICITLY — it is no longer the `dev` default. | |
| # | |
| # It is also what makes the leg run at all. This job was RED for four consecutive nights | |
| # (2026-07-27..30) on: | |
| # ValueError: SQL Server TLS is weakened (trust_server_certificate=true or encrypt=false) | |
| # Under enforcing-PHI the `MEFOR_ALLOW_INSECURE_TLS` escape above is clamped INERT by design | |
| # (#200 / ADR 0092 decision 2 — `weakened_tls_escape_permitted`), so a self-signed container | |
| # can never be trusted from a PHI instance. The previous comment here said this leg "validates | |
| # PHI+enforce green, not a synthetic opt-out" — an intent that is UNREACHABLE with a | |
| # `services:` container: GitHub starts it before any step runs, so a cert generated in a step | |
| # cannot be mounted into it. Whoever set that provisioned the PHI posture's retention and | |
| # egress requirements but not its TLS one. | |
| # | |
| # WHAT THIS GIVES UP, stated plainly: the leg no longer exercises the PHI+enforce path. Its | |
| # actual job is a load/throughput smoke of the SQL Server store, and `sqlserver-store` carries | |
| # the functional coverage. Restoring PHI+enforce here needs a REAL certificate — the job moved | |
| # off `services:` onto a `docker run` with a generated cert mounted and trusted — which is | |
| # worth doing deliberately, not as a side effect of unbreaking a nightly. | |
| # | |
| # The PHI-shaped provisions below are KEPT even though synthetic does not require them: they | |
| # cost nothing and keep the leg measuring a realistic configuration. | |
| MEFOR_SECURITY_HANDLES_REAL_PATIENT_DATA: "false" | |
| # Bounded PHI-body retention windows + a locked-down egress allowlisting the loopback sink. | |
| # The store key is minted at runtime below. The security-notification gate is skipped because | |
| # auth is off (no accounts to notify). | |
| MEFOR_SECURITY_DELETE_MESSAGE_BODIES_AFTER_DAYS: "30" | |
| MEFOR_RETENTION_DEAD_LETTER_DAYS: "30" | |
| MEFOR_SECURITY_BLOCK_UNLISTED_OUTBOUND: "true" | |
| MEFOR_EGRESS_ALLOWED_MLLP: "127.0.0.1" # the load graph fans out to the loopback correlation sink | |
| run: | | |
| set -o pipefail | |
| mkdir -p out/load | |
| # Mint a throwaway AES-256 store key at runtime (never a committed literal — gitleaks). | |
| export MEFOR_STORE_ENCRYPTION_KEY="$(python -c 'import base64, os; print(base64.b64encode(os.urandom(32)).decode())')" | |
| python -m messagefoundry serve --config harness/config/load --env dev \ | |
| --host 127.0.0.1 --port 8765 > engine.log 2>&1 & | |
| engine_pid=$! | |
| for _ in $(seq 1 60); do | |
| curl -sf http://127.0.0.1:8765/health > /dev/null && break || sleep 0.5 | |
| done | |
| set +e | |
| python -m harness --load smoke-sqlserver --engine http://127.0.0.1:8765 --sink-port 2700 \ | |
| --report-json out/load/smoke-sqlserver-ci.json --report-csv out/load/smoke-sqlserver-ci.csv | |
| rc=$? | |
| set -e | |
| kill "$engine_pid" 2>/dev/null || true | |
| echo "===== engine log (tail) ====="; tail -50 engine.log || true | |
| exit $rc | |
| - name: Upload load report | |
| if: always() | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 | |
| with: | |
| name: load-report-sqlserver | |
| path: out/load/ | |
| if-no-files-found: ignore | |
| # End-to-end check of the real Windows service path on both target Server | |
| # versions: install via NSSM -> start -> /health -> send an MLLP message -> | |
| # confirm it was recorded -> stop -> uninstall. Nightly + on-demand only | |
| # (2x-billed Windows legs): this validates the NSSM/service PLUMBING, which | |
| # changes when scripts/service/ or the serve path changes — not per merge — | |
| # and it was never a pre-merge gate (it already ran post-merge). Per-push it | |
| # was the single biggest push-run line item (~3.6 Windows min x ~16 merges/ | |
| # day); the nightly keeps both Server SKUs covered daily and workflow_dispatch | |
| # re-validates a specific merge on demand. | |
| windows-service-smoke: | |
| name: windows service smoke (${{ matrix.os }}, py${{ matrix.python-version }}) | |
| # This repo only: it spins up 2x-billed hosted Windows runners on BOTH Server SKUs, which is free | |
| # here (public) and billed to the owner of a fork. Nightly schedule + manual dispatch only. ci-gate | |
| # `needs` this job, but a `skipped` leg counts as a pass there, so gating it off a fork cannot wedge | |
| # the gate. | |
| if: (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && github.repository == 'MEFORORG/MessageFoundry' | |
| runs-on: ${{ matrix.os }} | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| # Windows Server 2022 + 2025 on the supported deploy Python (3.14), so the real NSSM | |
| # install->serve->MLLP path is validated on the version we ship, on each Server SKU. This job | |
| # is push/dispatch-only, so the Windows minutes cost no PR time. | |
| include: | |
| - { os: windows-2022, python-version: "3.14" } | |
| - { os: windows-2025, python-version: "3.14" } | |
| defaults: | |
| run: | |
| shell: pwsh | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Set up Python | |
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: ${{ matrix.python-version }} | |
| - name: Set up uv | |
| uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 | |
| with: | |
| cache-dependency-glob: | | |
| pyproject.toml | |
| requirements.lock | |
| - name: Install the engine (fhir + dicom extras) | |
| run: | | |
| # The samples/config graph this smoke serves uses the FHIR and DICOM connectors, and the | |
| # DICOM inbound binds a DIMSE C-STORE SCP at startup (transports/dicom.py imports pynetdicom | |
| # eagerly on start) — so a bare `uv pip install --system -e .` makes the engine fail-close at | |
| # wiring with ModuleNotFoundError and /health never comes up. Install the connector extras the | |
| # graph needs (the "real Windows service path" a user running this graph would install). Keep | |
| # in sync with the connectors used in samples/config. | |
| uv pip install --system --constraint constraints.lock -e ".[fhir,dicom]" | |
| # Run the install/uninstall scripts under Windows PowerShell 5.1 (shell: powershell) to | |
| # match what the console launches on an end-user machine — catches 5.1-only behaviour | |
| # (e.g. native-command stderr aborting under ErrorActionPreference=Stop). NSSM happens to | |
| # be on the runner PATH, so the auto-download path is validated locally / on the user's box. | |
| - name: Install the service | |
| shell: powershell | |
| run: | | |
| $exe = (Get-Command messagefoundry).Source | |
| # Run as the 'prod' environment (ADR 0017: `serve` requires an explicit active environment; | |
| # install-service.ps1 -Environment threads it into the service command + resolves | |
| # environments/prod.toml). INFO not DEBUG: `serve` refuses DEBUG in prod (Gate #1 — DEBUG can | |
| # surface PHI); INFO is enough to smoke service start + MLLP. | |
| # -LockConfigDir: the runner checks out samples/config with the default ACL, which grants | |
| # BUILTIN\Users (S-1-5-32-545) write. The in-process config-source trust guard (SEC-003, | |
| # wiring._assert_safe_config_source_windows) then REFUSES to load it ("writable-by-others | |
| # path") and `serve` crash-loops to SERVICE_PAUSED, so /health never comes up. Locking the | |
| # config dir (strip inheritance -> SYSTEM/Administrators + the run-as account, RX) is the | |
| # production-correct posture docs/SERVICE.md prescribes, and exercises the real -LockConfigDir | |
| # install path rather than the MEFOR_ALLOW_INSECURE_CONFIG_SOURCE dev escape. | |
| # #224: NO -ServiceAccount and NO -AllowLocalSystem, so this install exercises the NEW DEFAULT - | |
| # the least-privilege virtual account NT SERVICE\MessageFoundry (no password). This leg is the | |
| # gate for that flip: it proves the virtual account is granted config-read (RX, via the | |
| # -LockConfigDir path that now runs AFTER ObjectName so the per-service SID resolves) + data-dir | |
| # read/write, gets SeServiceLogonRight, and actually starts and serves /health + MLLP. Pass | |
| # -AllowLocalSystem here if a run-as-LocalSystem smoke is ever needed instead. | |
| .\scripts\service\install-service.ps1 -AppExe $exe -LogLevel INFO -Environment prod -LockConfigDir | |
| # The shipped samples graph includes a mutual-TLS WS-* SOAP feed (IB_IMMUNIZATION_VXU) whose | |
| # connector LOADS its client cert at wiring time, so the engine refuses to start the graph | |
| # without a readable cert+key. Mint a throwaway pair for the smoke — the smoke sends only an | |
| # MLLP ADT, never a VXU, so the cert is loaded but never used in a handshake. Run openssl via | |
| # `cmd /c "... 2>nul"`: it streams RSA key-gen progress to stderr, and a bare `2>$null` did NOT | |
| # suppress it on the runner's openssl build — PowerShell 5.1 wrapped that stderr as a | |
| # NativeCommandError and aborted the step. Redirecting at the cmd level keeps PowerShell out of | |
| # openssl's stderr path entirely, so no NativeCommandError regardless of the openssl build. | |
| $certDir = Join-Path $env:RUNNER_TEMP "registry-cert" | |
| New-Item -ItemType Directory -Force -Path $certDir | Out-Null | |
| $cert = Join-Path $certDir "client.pem" | |
| $key = Join-Path $certDir "client.key" | |
| cmd /c "openssl req -x509 -newkey rsa:2048 -keyout ""$key"" -out ""$cert"" -days 1 -subj /CN=mefor-smoke -passout pass:smoke-key-pass 2>nul" | |
| if ($LASTEXITCODE -ne 0 -or -not (Test-Path $cert)) { throw "openssl failed to mint the smoke client cert" } | |
| # `prod` is a PHI environment, so the engine FAIL-CLOSES at startup unless a store encryption | |
| # key is set (PHI-at-rest must never be plaintext). Without one, `serve` refuses to start, NSSM | |
| # throttles the crash-loop to SERVICE_PAUSED, and /health never comes up. Mint a throwaway AES | |
| # key for the smoke — `gen-key` prints only the key (stdout, pipeable; no stderr to trip PS 5.1). | |
| $storeKey = (& $exe gen-key).Trim() | |
| if (-not $storeKey) { throw "messagefoundry gen-key produced no store encryption key" } | |
| # This smoke validates the Windows service + MLLP plumbing, not authentication (auth is | |
| # covered by the pytest suite: test_api_auth / test_console_auth). Run with auth disabled so | |
| # the /messages call below needs no bearer token, and supply the registry endpoint's secrets | |
| # (cert/key/password + WS-Security creds) so the SOAP feed wires — placeholders, never used | |
| # since no VXU is sent; the non-secret endpoints live in environments/prod.toml (selected by | |
| # the -Environment prod passed to install-service.ps1 above). | |
| # `prod` also FAIL-CLOSES on unrestricted egress (a transform could send PHI anywhere): lock it | |
| # down (deny_by_default) AND allowlist the samples/config graph's 8 outbound destinations, or the | |
| # engine refuses to start at wiring. [egress] is SERVICE settings (MEFOR_EGRESS_*), NOT an | |
| # environments/<env>.toml value; lists are comma-separated host:port matching the resolved | |
| # prod.toml endpoints (MLLP->allowed_mllp; X12->allowed_tcp; SOAP+FHIR->allowed_http; File->allowed_file_dirs). | |
| # NOTE keep this list in sync with samples/config when an outbound is added (e.g. OB_FHIR_SERVER) — | |
| # the egress check fails fast on the FIRST denied dest, so a missing entry wedges service start. | |
| # fhir-prod.example.org is allowlisted HOST-ONLY: its prod URL (https://fhir-prod.example.org/fhir) | |
| # has no explicit port, so urlsplit().port is None and a ":443" suffix would NOT match (see | |
| # _http_egress_allowed). A bare host matches any port, which is what we want for a default-port URL. | |
| # `prod` FAIL-CLOSES on unbounded PHI RETENTION too (#186a, ASVS 14.2.4): both PHI-body windows — | |
| # [retention].messages_days (inbound bodies) and [retention].dead_letter_days (dead-lettered | |
| # outbound bodies, replayable until purged) — default to 0 = keep-forever, and a PRODUCTION PHI | |
| # instance with EITHER unbounded refuses to start ("no data-retention window is configured ... | |
| # refusing to start"), so NSSM crash-loops to SERVICE_PAUSED and /health never comes up. BOUND both | |
| # (30d) rather than setting [retention].allow_unbounded_phi=true: a real prod deployment configures | |
| # a window, so the smoke exercises the path it is meant to validate instead of opting out of it. | |
| nssm set MessageFoundry AppEnvironmentExtra ` | |
| "MEFOR_SECURITY_REQUIRE_SIGN_IN=false" ` | |
| "MEFOR_STORE_ENCRYPTION_KEY=$storeKey" ` | |
| "MEFOR_SECURITY_DELETE_MESSAGE_BODIES_AFTER_DAYS=30" ` | |
| "MEFOR_RETENTION_DEAD_LETTER_DAYS=30" ` | |
| "MEFOR_SECURITY_BLOCK_UNLISTED_OUTBOUND=true" ` | |
| "MEFOR_EGRESS_ALLOWED_MLLP=receiver-prod.example.org:6661,powerscribe-prod.example.org:6665" ` | |
| "MEFOR_EGRESS_ALLOWED_TCP=payer-prod.example.org:6662,rte-prod.example.org:6663,rte-result-prod.example.org:6664" ` | |
| "MEFOR_EGRESS_ALLOWED_HTTP=iis-prod.example.org:8443,fhir-prod.example.org" ` | |
| "MEFOR_EGRESS_ALLOWED_FILE_DIRS=out" ` | |
| "MEFOR_VALUE_REGISTRY_CLIENT_CERT=$cert" ` | |
| "MEFOR_VALUE_REGISTRY_CLIENT_KEY=$key" ` | |
| "MEFOR_VALUE_REGISTRY_KEY_PASSWORD=smoke-key-pass" ` | |
| "MEFOR_VALUE_REGISTRY_USER=smoke-user" ` | |
| "MEFOR_VALUE_REGISTRY_PASSWORD=smoke-pass" | |
| - name: Start and verify /health | |
| run: | | |
| nssm start MessageFoundry | |
| $ok = $null | |
| foreach ($i in 1..30) { | |
| try { $ok = Invoke-RestMethod http://127.0.0.1:8765/health -TimeoutSec 2; break } | |
| catch { Start-Sleep -Milliseconds 500 } | |
| } | |
| if (-not $ok) { throw "engine /health never came up" } | |
| "health: $($ok | ConvertTo-Json -Compress)" | |
| - name: Send an MLLP message and confirm it was recorded | |
| run: | | |
| python samples\send_mllp.py samples\messages\adt_a01.hl7 | |
| $msgs = $null | |
| foreach ($i in 1..20) { | |
| $msgs = Invoke-RestMethod http://127.0.0.1:8765/messages | |
| if ($msgs.total -ge 1) { break } | |
| Start-Sleep -Milliseconds 500 | |
| } | |
| if ($msgs.total -lt 1) { throw "message was not recorded by the engine" } | |
| "messages.total = $($msgs.total)" | |
| - name: Stop the service (graceful) | |
| if: always() | |
| run: nssm stop MessageFoundry | |
| - name: Show service logs | |
| if: always() | |
| run: | | |
| $dir = "C:\ProgramData\MessageFoundry\logs" | |
| foreach ($f in "service.out.log", "service.err.log") { | |
| $p = Join-Path $dir $f | |
| "===== $f =====" | |
| if (Test-Path $p) { Get-Content $p } else { "(none)" } | |
| } | |
| - name: Uninstall the service | |
| if: always() | |
| shell: powershell | |
| run: .\scripts\service\uninstall-service.ps1 | |
| - name: Upload service logs | |
| if: always() | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 | |
| with: | |
| name: service-logs-${{ matrix.os }}-py${{ matrix.python-version }} | |
| path: C:\ProgramData\MessageFoundry\logs\ | |
| if-no-files-found: ignore | |
| # Container image smoke (the engine's FIRST non-Windows runtime target — it is Windows-service/NSSM | |
| # first today). Builds the slim engine image, builds a test-only image that bakes a minimal ADT->file | |
| # config + a self-contained MLLP sender (config baked, not mounted, so it is owned by the engine's UID | |
| # and not group/world-writable — _assert_safe_config_source refuses otherwise), serves it loopback + | |
| # auth-off, sends one synthetic ADT^A01 over the in-container MLLP listener, and asserts it FINALIZES | |
| # to PROCESSED (not merely RECEIVED — that distinguishes ACK-on-receipt from final disposition). Then | |
| # it verifies a graceful `docker stop`: tini forwards SIGTERM -> uvicorn lifespan -> engine.stop(). | |
| # Runs NIGHTLY + on dispatch; on PRs only when the image/locks/packaging/this workflow change | |
| # (gated by `changes`), so docs/unrelated PRs don't pay the build. No longer per push-to-main — | |
| # an image-touching merge was path-gated on its own PR minutes earlier (same rationale as the | |
| # server-DB legs; `changes` sets docker=false on push, the other half of this guard). | |
| docker-smoke: | |
| name: docker image smoke (slim, linux) | |
| needs: changes | |
| # Nightly / on-demand, or a PR that touches the image/locks/packaging. | |
| if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || needs.changes.outputs.docker == 'true' | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Build the slim engine image | |
| run: docker build -f docker/Dockerfile -t messagefoundry:ci . | |
| - name: Build the -sqlserver variant (ODBC apt layer + sqlserver lock must resolve) | |
| # Build-only: it pulls the Microsoft ODBC repo + installs from requirements-sqlserver.lock, so a | |
| # broken hashed dep or the Debian/MS-key issue the Dockerfile warns about turns the job red here. | |
| run: docker build -f docker/Dockerfile --target runtime-sqlserver -t messagefoundry:sqlserver-ci . | |
| - name: Build the test smoke image (baked minimal config + MLLP sender) | |
| run: docker build -f docker/smoke/Dockerfile --build-arg BASE=messagefoundry:ci -t messagefoundry:smoke . | |
| - name: Serve (loopback, auth off) and assert an MLLP message reaches PROCESSED | |
| run: | | |
| set -euo pipefail | |
| # Bind the API + MLLP to loopback INSIDE the container and drive the test from inside it | |
| # (docker exec). A non-loopback bind without TLS would be refused by the startup bind guard | |
| # (and auth-off is refused off-loopback regardless of --allow-insecure-bind), so loopback + | |
| # in-container traffic is the correct posture for an auth-disabled smoke — no insecure override. | |
| # GIVEN 1 (ADR 0148): `dev` now derives PHI, so `serve --env dev` runs the secure PHI posture. | |
| # This smoke validates PHI+enforce green (not a synthetic opt-out): provision a runtime-minted | |
| # store key (never a committed literal — gitleaks), bounded PHI-body retention windows, and a | |
| # locked-down egress allowlisting the smoke config's File archive dir (/var/lib/mefor/out/adt). | |
| # The security-notification gate is skipped because auth is off (no accounts to notify). | |
| storeKey="$(python3 -c 'import base64, os; print(base64.b64encode(os.urandom(32)).decode())')" | |
| docker run -d --name mefor \ | |
| -e MEFOR_SECURITY_REQUIRE_SIGN_IN=false \ | |
| -e MEFOR_STORE_ENCRYPTION_KEY="$storeKey" \ | |
| -e MEFOR_SECURITY_DELETE_MESSAGE_BODIES_AFTER_DAYS=30 \ | |
| -e MEFOR_RETENTION_DEAD_LETTER_DAYS=30 \ | |
| -e MEFOR_SECURITY_BLOCK_UNLISTED_OUTBOUND=true \ | |
| -e MEFOR_EGRESS_ALLOWED_FILE_DIRS=/var/lib/mefor \ | |
| messagefoundry:smoke \ | |
| serve --config /config --env dev --host 127.0.0.1 --port 8765 | |
| # Readiness: /health is tokenless and always 200 once the listener is up. | |
| for _ in $(seq 1 60); do | |
| docker exec mefor curl -fsS http://127.0.0.1:8765/health >/dev/null 2>&1 && break || sleep 0.5 | |
| done | |
| docker exec mefor curl -fsS http://127.0.0.1:8765/health | |
| # Send one synthetic ADT^A01 over the in-container MLLP listener (loopback:2575); fails on a non-AA ACK. | |
| docker exec mefor python /smoke/send_adt.py 127.0.0.1 2575 | |
| # Poll the disposition via the container's own python (stdlib urllib — no host python/jq needed). | |
| total() { docker exec mefor python -c "import urllib.request,json;print(json.load(urllib.request.urlopen('http://127.0.0.1:8765/messages?status=$1'))['total'])"; } | |
| processed=0 | |
| for _ in $(seq 1 60); do | |
| # `|| processed=0`: a transient non-zero before the row finalizes must NOT abort the loop under set -e. | |
| processed=$(total processed) || processed=0 | |
| [ "$processed" -ge 1 ] && break || sleep 0.5 | |
| done | |
| received=$(total received) || received=0 | |
| echo "processed=$processed received=$received" | |
| # The whole point of the assert: confirm it FINALIZED, not just that it was acknowledged-on-receipt. | |
| if [ "$processed" -lt 1 ]; then | |
| echo "FAIL: message did not finalize to PROCESSED (received=$received) — stuck at ACK-on-receipt" | |
| exit 1 | |
| fi | |
| - name: Verify graceful shutdown (tini -> SIGTERM -> engine.stop drains) | |
| run: | | |
| set -euo pipefail | |
| docker stop --time 30 mefor | |
| code=$(docker inspect -f '{{.State.ExitCode}}' mefor) | |
| echo "container exit code: $code" | |
| # A fully graceful shutdown exits 0 (uvicorn handles SIGTERM, runs the lifespan, returns); 143 | |
| # (128+SIGTERM) is acceptable too. 137 = SIGKILL after the grace expired = ungraceful. | |
| if [ "$code" = "137" ]; then echo "FAIL: ungraceful shutdown (SIGKILL after grace)"; exit 1; fi | |
| # Poll for the marker: `docker stop` returns the instant the container exits, but the final | |
| # buffered log line ("Application shutdown complete") can land in `docker logs` a beat later, so | |
| # grepping once immediately is racy (a clean 143 shutdown then read as a missing marker). Retry | |
| # for ~10s so a slow log flush is not misread as an incomplete lifespan shutdown. | |
| for _ in $(seq 1 20); do | |
| if docker logs mefor 2>&1 | grep -q "Application shutdown complete"; then | |
| echo "graceful shutdown verified"; exit 0 | |
| fi | |
| sleep 0.5 | |
| done | |
| echo "FAIL: no clean-shutdown marker (lifespan shutdown did not complete)" | |
| docker logs mefor 2>&1 | tail -20 | |
| exit 1 | |
| - name: Engine log (always) | |
| if: always() | |
| run: docker logs mefor 2>&1 | tail -60 || true | |
| - name: Tear down | |
| if: always() | |
| run: docker rm -f mefor || true | |
| # Single STABLE required status check standing in for the conditional / matrix heavy legs (SQL | |
| # Server, Postgres, load, Windows-service smoke). Those legs CANNOT be required directly: | |
| # * the nightly/path-gated legs (`postgres-store`, `load-test`, `load-test-sqlserver`, | |
| # `windows-service-smoke`) report `skipped` on most or all PRs — never a stable required | |
| # context a PR can wait on — so requiring them either no-ops or wedges; and | |
| # * the matrix legs report an UNEXPANDED name when skipped (`... ${{ matrix.label }}`) but | |
| # EXPANDED names when they run (`... 2022`/`2025`, `(windows-2022, py3.14)`), so no single | |
| # context string matches both the skipped and the run state. | |
| # This gate ALWAYS runs (if: always()), so it reports ONE fixed context ("CI gate") regardless of | |
| # which legs ran, and fails iff a gated leg actually FAILED or was cancelled — a `skipped` leg | |
| # counts as a pass. Require "CI gate" in branch protection INSTEAD of the individual legs; a leg | |
| # that does run on a PR (e.g. `sqlserver-store` on a server-DB PR) thus still blocks the merge. | |
| # NB: `docker-smoke` is intentionally NOT gated here (out of the named set) — add it to `needs` | |
| # below to make a container-smoke failure block merge too. | |
| ci-gate: | |
| name: CI gate | |
| if: always() | |
| needs: | |
| - changes | |
| - sqlserver-store | |
| - postgres-store | |
| - load-test | |
| - load-test-sqlserver | |
| - windows-service-smoke | |
| runs-on: ubuntu-latest | |
| steps: | |
| - name: Fail if any gated leg failed or was cancelled | |
| if: >- | |
| contains(needs.*.result, 'failure') || | |
| contains(needs.*.result, 'cancelled') | |
| env: | |
| # Pass the needs context via env, not inline ${{ }} in the run block (defense-in-depth against | |
| # template injection — a job output could in theory carry a quote that breaks out of echo). | |
| NEEDS_JSON: ${{ toJSON(needs) }} | |
| run: | | |
| echo "::error::A gated CI leg failed or was cancelled." | |
| echo "$NEEDS_JSON" | |
| exit 1 | |
| - name: Gated legs OK | |
| run: echo "All gated CI legs succeeded or were skipped." |