feat(api): the engine always serves TLS, minting on first run (BACKLOG #1276 part A, ADR 0172) #2421

Workflow file for this run

.github/workflows/security.yml at 4a3f7bd

	name: Security

	# Security gate: Python SAST (bandit), Python dependency audit (pip-audit), npm dependency audit
	# (npm-audit, for the ide/ extension), secret scan (gitleaks), and project SAST rules (semgrep).
	# All four are BLOCKING — their first CI run was clean (bandit: 0; gitleaks: 0; semgrep: 0 across
	# 5 rules; pip-audit: no known CVEs), so each now fails the build on a regression rather than
	# merging unnoticed. To temporarily downgrade one, add `continue-on-error: true` back to its job.
	#
	# READ THAT LAST SENTENCE AS A TRAP, not a procedure. Every job here except `sbom` and `trivy` is a
	# REQUIRED context, and GitHub reports a continue-on-error job as SUCCESS — so that one line leaves
	# branch protection green while the scanner's findings are discarded, and with auto-merge armed and
	# zero required approvals the PR merges unread. `tests/test_security_posture.py` now REFUSES it (and
	# `\|\| true`, `--exit-zero`, and a skippable job-level `if:`) for any job in the required set. To take
	# a gate off the merge path, remove its context from branch protection AND
	# `.github/required-contexts.txt` — deliberately, in a reviewed diff. Never neuter it in place.
	# This repo is PUBLIC, so CodeQL runs free and IS a required context (codeql.yml) — it is not gated
	# behind GitHub Advanced Security. This comment previously said "this private repo", written when it
	# was; CodeQL reporting on every PR is itself the disproof. gitleaks below is retained deliberately:
	# it scans on the same run as the other gates rather than relying on native push protection.
	# See docs/SECURITY.md.
	#
	# THE TRIGGER SET IS THE `on:` BLOCK BELOW AND NOTHING ELSE. Each arm carries its own reason there,
	# so this header states none of them and must not start. A header paragraph that ALSO describes the
	# triggers is a second definition, free to drift from the first — and one did: for months a paragraph
	# here denied a trigger the `on:` block declared ten lines beneath it, at length and with costings.
	# It was DELETED rather than corrected, because correcting it would have left the second definition
	# in place to drift again. The cost of that drift is not to CI, which behaved as the `on:` block says;
	# it is that the rest of this header is load-bearing (the continue-on-error trap above), and a reader
	# who finds one paragraph of it demonstrably false has no way to tell which of the others still hold.
	# tests/test_security_posture.py refuses the return of a header claim that denies a declared trigger.
	on:
	pull_request:
	# Post-merge re-scan (main only). A fork PR is scanned STRUCTURAL-ONLY by design -- the secret is
	# unavailable to it -- so without this arm no fully-loaded scan ever sees fork-contributed content.
	# Scoped to main so branch pushes do not double-run alongside their own PR.
	push:
	branches: [main]
	schedule:
	# Daily, so a CVE freshly disclosed against an UNCHANGED pinned dep (no push/PR to trigger a scan)
	# is caught within ~24h instead of up to 7 days — a weekly cadence can't meet a 72h-class remediation
	# clock. pip-audit/npm-audit here are a second detector independent of GitHub's advisory-review lag.
	- cron: "0 6 * * *"
	workflow_dispatch:
	inputs:
	released_line_audit_tag:
	description: "released-line-audit: audit THIS release tag instead of the selected latest. Leave blank for normal runs; used to re-run the positive control against an older tag."
	required: false
	type: string

	concurrency:
	# Per-ref for PR events so a rapid re-push supersedes its own in-flight scan (ci.yml's pattern —
	# auto-merge tolerates cancelled contexts on superseded SHAs; it gates on the latest head SHA).
	# schedule/dispatch runs get a unique group (run_id): never superseded, never cancelled.
	group: security-${{ github.event_name == 'pull_request' && github.ref \|\| github.run_id }}
	cancel-in-progress: ${{ github.event_name == 'pull_request' }}

	permissions:
	contents: read

	jobs:
	pip-audit:
	name: pip-audit (dependency vulnerabilities)
	runs-on: ubuntu-latest
	# BLOCKING: the lockfile audit is clean. A newly-disclosed CVE in a pinned dep will now turn CI
	# red (the intended forcing function) — bump the lockfile, or `pip-audit --ignore-vuln <ID>` to
	# accept a triaged advisory.
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
	with:
	python-version: "3.14"
	- name: Check the lockfile is in sync with pyproject (DEP-1)
	run: \|
	# uv is PINNED: it is the resolver that produces every lockfile this job then audits, so an
	# unpinned upgrade lets a new resolver silently change the exported set mid-PR and reds the
	# `git diff --exit-code` below for a reason unrelated to the change. Bump deliberately.
	python -m pip install --upgrade pip "uv==0.12.0"
	# Fails if pyproject changed without re-running `uv lock` (drift guard).
	uv lock --check
	# Re-export to the same path and fail if it differs from what's committed (keeps the
	# hashed requirements.lock in sync with uv.lock; same -o name = deterministic header).
	uv export --all-extras --no-emit-project --format requirements.txt -o requirements.lock
	# The container image installs from SCOPED, per-profile locks (not the all-extras lock above,
	# which drags PySide6/dev/etc. into a slim runtime image) — keep them in sync too. The -o paths
	# match the header recorded in each committed file, so re-export is byte-identical when in sync.
	uv export --no-emit-project --no-dev --format requirements.txt -o docker/locks/requirements-core.lock
	uv export --no-emit-project --no-dev --extra sqlserver --format requirements.txt -o docker/locks/requirements-sqlserver.lock
	# CI's own installs pass `--constraint constraints.lock` (see ci.yml): a HASHLESS export of this
	# same lock, so a fresh `uv pip install -e ".[extras]"` cannot drift to a newer upstream
	# release. Re-export it here so it can never fall out of step with uv.lock.
	uv export --all-extras --no-emit-project --no-hashes --format requirements.txt -o constraints.lock
	# The CI TOOLCHAIN locks (ADR 0034 §3): PEP 735 `[dependency-groups]`, so the scanners and the
	# quality tools flow through uv.lock and are consumed HASH-PINNED. Non-default groups, so they
	# stay out of the four exports above (and out of the SBOM / image locks / audited runtime).
	uv export --only-group ci-scanners --format requirements.txt -o ci/locks/ci-scanners.lock
	uv export --only-group ci-quality --format requirements.txt -o ci/locks/ci-quality.lock
	git diff --exit-code -- requirements.lock docker/locks/requirements-core.lock docker/locks/requirements-sqlserver.lock constraints.lock ci/locks/ci-scanners.lock ci/locks/ci-quality.lock
	- name: Install from the hashed lockfile (DEP-1)
	run: \|
	# --require-hashes enforces a hash for every requirement (the lockfile carries them): a
	# reproducible, tamper-evident install. Exercises the lockfile as an actual install path
	# instead of only auditing it, so a lockfile that doesn't resolve/install is caught (low-26).
	python -m venv /tmp/lockcheck
	# No unpinned pip bootstrap: --require-hashes resolves nothing (tests/test_ci_venv_pinning.py).
	/tmp/lockcheck/bin/pip install --require-hashes -r requirements.lock
	- name: Audit the locked dependencies (DEP-1)
	run: \|
	# HASH-PINNED from the lock, not merely `==`-pinned: a version pin does not satisfy Scorecard
	# PinnedDependenciesID, and the transitive closure floated regardless. Rationale for each pin
	# lives in pyproject.toml's [dependency-groups]. THIS step's `--upgrade pip` bootstrap is gone
	# rather than pinned — the lock hash-pins `pip` itself (a pip-audit → pip-api dependency).
	# Scoped deliberately, because the unqualified claim was FALSE for this job: the DEP-1 step
	# above still runs `pip install --upgrade pip "uv=="` in this same interpreter two steps
	# earlier, and that unverified pip does the load-bearing work (the six exports + the diff
	# gate); the line below then DOWNGRADES pip to the locked version after the fact. So this
	# job's posture improves for the AUDITOR, not for the exporter. The `uv` bootstrap is circular
	# by construction (ADR 0034 section 3's residuals) and is why `pip` stays registered in
	# SECURITY_YML_ACCEPTED_UNPINNED. The claim holds unqualified in the bandit job and zizmor.yml.
	python -m pip install --require-hashes -r ci/locks/ci-scanners.lock
	# Audit the committed, pinned set — reproducible (vs auditing a fresh latest-resolve).
	pip-audit -r requirements.lock --desc
	# Audit the TOOLCHAIN locks too. NOT bookkeeping: hash-pinning makes the toolchain STICKY, so
	# without this a CVE in a pinned scanner is invisible to every gate — the "pinned, stale,
	# unpatched is worse than floating" failure ADR 0034 section 3 names. `--ignore-vuln <ID>` is
	# the escape hatch for a triaged advisory, exactly as for the lockfile audit above.
	#
	# CONSEQUENCE, stated rather than discovered later: this is a REQUIRED context, so a CVE in
	# mutmut's or diff-cover's closure reds the merge gate over an ADVISORY tool. That is not a new
	# posture — requirements.lock is exported `--all-extras`, so ruff, mypy and pytest already do
	# exactly this — but it does add 40 distributions to the blocking set (23 from ci-scanners, 17
	# from ci-quality; measured). Zero advisories at the time of writing. If the owner would rather
	# the advisory half not block, move the ci-quality line alone to its own step.
	pip-audit -r ci/locks/ci-scanners.lock --desc
	pip-audit -r ci/locks/ci-quality.lock --desc
	# ANTI-SLOPSQUAT. Deliberately a STEP in this already-REQUIRED job rather than a new context: it
	# blocks today with no branch-protection change, the same reasoning as the ledger-gate backstop
	# in ci.yml. (A hard-failing job that is NOT a required context does not stop auto-merge — it
	# only looks like it does.)
	#
	# This answers the question pip-audit above structurally cannot. pip-audit asks "does this pinned
	# version have a known CVE"; a freshly registered hallucinated name has no advisory, so it
	# resolves, locks, hashes and installs through every DEP-1 control clean. This project's
	# dependencies are chosen by an AI, and docs/Secure_AI_Development_Standards.md has carried the
	# check as the "highest-priority deferred gate" while verify-before-add stayed "enforced only by
	# the human remembering".
	#
	# Stdlib-only (no install). Fails CLOSED: if PyPI is unreachable it exits 2 rather than passing,
	# and it exits 2 if it examined zero distributions, so a schema change cannot read as a clean
	# sweep. Its known blind spot — a real package that is not the INTENDED one — is documented in
	# the script and pinned by tests/test_new_dependency_check.py.
	- name: Verify every declared dependency is a real, established distribution
	run: python scripts/security/new_dependency_check.py

	npm-audit:
	name: npm-audit (ide dependency vulnerabilities)
	runs-on: ubuntu-latest
	# BLOCKING: the ide/ npm tree (the VS Code extension's build-time toolchain) audits clean today, so a
	# newly-disclosed advisory in a pinned npm dep now turns CI red — the same forcing function as
	# pip-audit, extended to the extension's supply chain (previously only pip + github-actions were
	# watched). `overrides` in ide/package.json is where a triaged transitive advisory gets pinned out.
	defaults:
	run:
	working-directory: ide
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	- name: Set up Node
	uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
	with:
	node-version: "24"
	- name: Audit the locked npm dependencies (install-free)
	# --package-lock-only audits straight from the committed package-lock.json (no install needed) —
	# fast and reproducible. Default level fails on ANY severity, matching pip-audit's strict posture.
	run: npm audit --package-lock-only

	released-line-audit:
	name: released-line-audit (latest release's pinned runtime)
	runs-on: ubuntu-latest
	# WHAT THIS ANSWERS, and why the `pip-audit` job above does not. That job audits the CHECKED-OUT
	# tree, so on the daily cron it audits `main`. "Is what we would ship next current" is the right
	# question and it is a DIFFERENT question from "does the version we already shipped carry a known
	# advisory". The two answers diverge from the moment a fix lands on main until a release carries
	# it, and nothing watched that window.
	#
	# MEASURED 2026-08-06, stated in the conditional because there are ZERO deployments (CLAUDE.md
	# section 0). v0.3.2 was tagged 2026-07-28 pinning cryptography 49.0.0. GHSA-g6cj-pr64-35w5 /
	# CVE-2026-69247 / PYSEC-2026-3552 (fixed in 50.0.0) published 2026-08-03. The required `pip-audit`
	# gate above caught it on main THE SAME DAY and the pin was bumped -- that gate worked, and this job
	# is NOT a second detector for it. What did not exist was anything noticing that days later no
	# release carries the fix, so an operator scanning the published v0.3.2 SBOM WOULD see an advisory
	# our shipped VEX says nothing about (security/vex/messagefoundry.openvex.json has no statements).
	#
	# SCOPE = the latest released line only. docs/SUPPORT-POLICY.md -- only the latest released version
	# is supported pre-1.0 and there is no back-port. A finding against an older tag has no supported
	# remedy, so scanning one would manufacture unactionable work.
	#
	# ADVISORY BY PLACEMENT, NOT BY continue-on-error -- the dast.yml posture recorded in
	# .github/required-contexts.txt. It goes RED on a finding. It is schedule/dispatch-only, so it can
	# never report on a PR and must never become a required context (the required-but-absent trap).
	# tests/test_security_posture.py pins both halves of that.
	#
	# NO VEX IS APPLIED HERE, deliberately. Applying our maintained OpenVEX would let a `fixed` or
	# `not_affected` statement written on main suppress the finding against the ALREADY-SHIPPED
	# artifact -- security/vex/README.md's own worked example names the product with no version
	# qualifier, so a statement applies to every release forever. That would turn this gate green at
	# the exact moment the assessment was written and before any release carried the fix. To accept a
	# triaged advisory, add `--ignore-vuln <ID>` below with the reason in a comment: explicit,
	# greppable and per-advisory, the same escape hatch the pip-audit job documents.
	if: github.event_name == 'schedule' \|\| github.event_name == 'workflow_dispatch'
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	# Nothing in this job pushes or uploads the tree, so the persisted git credential has no work
	# to do. Fixed at the source rather than leaning on this file's artipacked entry in
	# .github/zizmor.yml, which that config's own header asks for.
	persist-credentials: false
	- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
	with:
	python-version: "3.14"
	- name: Install pip-audit (hash-pinned, from the CI toolchain lock)
	# Same install as the pip-audit job above: no new tool, no new pin, no second vulnerability
	# database, nothing that can drift out of step with an operator-facing command.
	run: python -m pip install --require-hashes -r ci/locks/ci-scanners.lock
	- name: Audit the latest released line's pinned core runtime
	env:
	GH_TOKEN: ${{ github.token }}
	GH_REPO: ${{ github.repository }}
	# Hoisted, never interpolated into the script body (zizmor: template injection). A dispatch
	# input is untrusted text and is shape-checked before it reaches `gh`.
	TAG_OVERRIDE: ${{ inputs.released_line_audit_tag }}
	run: \|
	set -euo pipefail

	# WHICH TAG. Deliberately NOT `/releases/latest`: that endpoint excludes pre-releases, and
	# release.yml publishes pre-release `-*` tags to PRODUCTION PyPI -- so its default answers a
	# narrower question than the one asked. The web console is separately versioned under a
	# `webconsole-v*` namespace and ships no engine runtime lock, so it is excluded by shape.
	# Enumerate, filter, and PRINT the rule applied plus every candidate considered.
	if [ -n "${TAG_OVERRIDE:-}" ]; then
	case "$TAG_OVERRIDE" in
	v[0-9].[0-9].[0-9]*) targets="$TAG_OVERRIDE" ;;
	*) echo "::error::refusing tag override '$TAG_OVERRIDE' -- engine tags are vX.Y.Z[-suffix]" >&2 ; exit 2 ;;
	esac
	echo "selection rule: OPERATOR OVERRIDE via workflow_dispatch input"
	else
	releases="$(gh release list --limit 100 --json tagName,isDraft,isPrerelease,publishedAt)"
	echo "--- candidates considered ---"
	printf '%s' "$releases" \| jq -r '.[] \| "\(.publishedAt) \(.tagName) draft=\(.isDraft) prerelease=\(.isPrerelease)"'
	echo "selection rule: newest non-draft engine tag, PLUS the newest pre-release when it is newer than that"
	targets="$(printf '%s' "$releases" \| jq -r '
	[ .[]
	\| select(.isDraft \| not)
	\| select(.tagName \| test("^v[0-9]+\\.[0-9]+\\.[0-9]+(-.+)?$")) ]
	\| sort_by(.publishedAt) \| reverse
	\| ( (map(select(.isPrerelease \| not)) \| .[0:1]) + (.[0:1] \| map(select(.isPrerelease))) )
	\| map(.tagName) \| unique \| .[]')"
	fi

	if [ -z "$targets" ]; then
	echo "::error::no published engine release matched the selection rule -- this gate examined NOTHING. Failing closed." >&2
	exit 2
	fi

	examined=0
	for tag in $targets; do
	echo "=== $tag ==="
	# Read the lock AT THE TAG. This is the exact file release.yml installs into a clean venv to
	# generate the shipped CycloneDX SBOM, so it is the same component population the published
	# messagefoundry-sbom.cdx.json inventories -- not a fresh resolve of main.
	gh api "repos/$GH_REPO/contents/docker/locks/requirements-core.lock?ref=$tag" \
	-H "Accept: application/vnd.github.raw" > "core-$tag.lock"
	pinned="$(awk '/^[^#[:space:]-]/ && /==/ {n++} END {print n+0}' "core-$tag.lock")"
	echo "$tag: docker/locks/requirements-core.lock carries $pinned pinned requirements"
	# A run that examined nothing must never read as clean. 41 at the time of writing; this
	# floor is a collapse detector, not a pin.
	if [ "$pinned" -lt 25 ]; then
	echo "::error::$tag: read only $pinned pinned requirements -- the audit input is missing or truncated. Failing closed." >&2
	exit 2
	fi
	pip-audit -r "core-$tag.lock" --desc
	echo "$tag: CLEAN -- $pinned pinned requirements of the CORE RUNTIME closure, no known advisory"
	examined=$((examined + 1))
	done

	echo "released-line-audit: examined $examined release(s):"
	printf '%s\n' "$targets"
	echo "SCOPE: the CORE RUNTIME closure only (docker/locks/requirements-core.lock), which is"
	echo "what the shipped SBOM inventories. Extras (webauthn, dicom, sqlserver) and the CI"
	echo "toolchain are covered against main by the pip-audit job, not here. A wheel adopter"
	echo "resolves against pyproject.toml's FLOORS, so a finding here is a statement about the"
	echo "published SBOM's inventory, not about every install."
	- name: What to do about a finding
	if: failure()
	run: \|
	set -euo pipefail
	echo "The latest released line pins a component with a known advisory. The order matters."
	echo
	echo "1. TRIAGE FIRST. .github/SECURITY.md -- the clock starts at UPSTREAM-FIX availability,"
	echo " not at this run. Priority is KEV first, then EPSS >= 0.7, with CVSS only as a"
	echo " tiebreaker, weighing reachability. Pick the row from that file's SLA table. A HIGH"
	echo " CVSS that is unreachable lands on the LAST row, not the High row."
	echo
	echo "2. THE REMEDY IS A RELEASE. docs/SUPPORT-POLICY.md -- there is no back-port, and"
	echo " adopting a fix is a one-line pin bump, so a patch release on the latest line is the"
	echo " only vendor action that changes an adopter's state."
	echo
	echo "3. IF IT IS UNREACHABLE, record the assessment as an OpenVEX statement in"
	echo " security/vex/messagefoundry.openvex.json (security/vex/README.md) and bump the"
	echo " document version. It ships with the NEXT release: release.yml copies that file from"
	echo " the working tree at tag time, so the VEX commit must PRECEDE the tag."
	echo
	echo "4. NEVER edit the shipped artifact. The released SBOM and VEX are Sigstore-signed and"
	echo " SLSA-attested at tag time. A post-tag assessment reaches consumers via the next"
	echo " release, never by amending an old one."
	echo
	echo "5. NO UPSTREAM FIX YET? .github/SECURITY.md -- apply a documented compensating control"
	echo " and track to the fix. To silence a triaged advisory here, add '--ignore-vuln <ID>'"
	echo " to the pip-audit call in the previous step, with the reason."

	sbom:
	name: SBOMs (CycloneDX, multi-ecosystem)
	runs-on: ubuntu-latest
	# Advisory (non-blocking): generate machine-readable, LICENSE-complete CycloneDX SBOMs for the two code
	# artifacts whose dependency trees pip-audit/npm-audit already watch — the Python engine (from the
	# hash-locked core runtime lock) and the VS Code extension (from its npm lockfile) — score their quality,
	# and retain them, so "are we exposed to CVE-X? under what license?" is answerable from a recorded bill of
	# materials rather than a fresh resolve (ASVS 15.1.2). The container image's SBOM is emitted by the `trivy`
	# job below (which builds the image). Full rationale: docs/SUPPLY-CHAIN.md + ADR 0149. Each step still
	# fails if its SBOM can't be produced (a lock that won't render is caught), but continue-on-error keeps
	# the job off the merge path.
	continue-on-error: true
	# Daily cron + on-demand only (CI cost): the SBOMs derive from COMMITTED lockfiles, so a per-PR build adds
	# no information over a daily one — a freshness question, not a merge gate. Advisory (continue-on-error)
	# and NOT a required check, so this gate can never wedge a PR.
	if: github.event_name == 'schedule' \|\| github.event_name == 'workflow_dispatch'
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
	with:
	python-version: "3.14"
	- name: Set up Node
	uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
	with:
	node-version: "24"
	- name: Generate the Python engine SBOM (license-complete, hash-locked core runtime)
	run: \|
	# environment mode reads each installed dist's METADATA so component LICENSES populate — the
	# `requirements <lockfile>` parser has no metadata and emits license-less components. The core lock
	# (docker/locks/requirements-core.lock) is the honest runtime closure `pip install messagefoundry`
	# pulls; the all-extras requirements.lock stays covered by the pip-audit job above. cyclonedx-bom
	# ~=7.3.1 → lxml 6.x (cp314 wheels) so the 3.14 runner doesn't source-build lxml. ADR 0149.
	# PINNED and BYTE-IDENTICAL to release.yml's SBOM install (enforced by
	# tests/test_ci_venv_pinning.py). ~=7.3.1 rather than ~=7.3 so a 7.4 cannot change the JSON
	# shape sbom_finalize.py parses. This job is ADR 0034's pre-tag dry-run for the release SBOM
	# step, so the two must stay the SAME command — if they drift, dispatching this one proves
	# nothing about the release.
	python -m pip install "pip==26.1.2" "cyclonedx-bom~=7.3.1"
	# No unpinned pip bootstrap: --require-hashes resolves nothing (tests/test_ci_venv_pinning.py).
	python -m venv /tmp/sbomenv
	/tmp/sbomenv/bin/pip install --require-hashes -r docker/locks/requirements-core.lock
	python -m cyclonedx_py environment /tmp/sbomenv/bin/python \
	--pyproject pyproject.toml --mc-type application \
	--output-format JSON --output-file sbom-python.cdx.json
	# Declare the SBOM lifecycle (CISA "Build" type) + backfill the hatchling-dynamic root version.
	python scripts/security/sbom_finalize.py sbom-python.cdx.json \
	--phase build --set-version-from messagefoundry/__init__.py
	- name: Generate the VS Code extension SBOM (npm, install-free)
	# if: always() so a Python-SBOM failure above doesn't skip this independent generator.
	if: always()
	working-directory: ide
	run: \|
	# --package-lock-only reads package-lock.json directly (no `npm install`), matching the npm-audit
	# job's install-free posture. The extension has NO runtime npm dependencies (esbuild bundles the
	# payload), so we capture the FULL tree — the build toolchain (esbuild/typescript/…) is the npm
	# supply chain worth inventorying, and this mirrors npm-audit's full-tree scan. Tool version pinned.
	npx --yes @cyclonedx/cyclonedx-npm@6.0.0 --package-lock-only \
	--output-format JSON --output-file ../sbom-ide.cdx.json
	python ../scripts/security/sbom_finalize.py ../sbom-ide.cdx.json --phase build
	- name: Score SBOM quality (sbomqs — advisory)
	# continue-on-error (matching the release.yml twin): sbomqs is advisory, so a download/checksum/parse
	# hiccup here must never skip the SBOM-artifact upload below (the actual deliverable).
	continue-on-error: true
	run: \|
	# sbomqs grades an SBOM 0-10 across structural/semantic/quality completeness and prints the NTIA
	# minimum-elements breakdown, so an SBOM-completeness regression is visible. Pinned + checksum-
	# verified against the release's own goreleaser checksums.txt. ADR 0149.
	VER=2.0.11
	asset="sbomqs_${VER}_Linux_x86_64.tar.gz"
	base="https://github.com/interlynk-io/sbomqs/releases/download/v${VER}"
	curl -sSfL "${base}/${asset}" -o "${asset}"
	curl -sSfL "${base}/checksums.txt" -o sbomqs-checksums.txt
	grep " ${asset}$" sbomqs-checksums.txt \| sha256sum -c -
	tar -xzf "${asset}" sbomqs
	sudo install -m 0755 sbomqs /usr/local/bin/sbomqs
	sbomqs score -b sbom-python.cdx.json
	sbomqs score -b sbom-ide.cdx.json
	sbomqs compliance --ntia sbom-python.cdx.json \|\| true
	- name: Upload the SBOM artifacts
	# if: always() + if-no-files-found: ignore so whichever SBOM(s) generated are retained even if a
	# sibling generator failed. SHA-pinned for supply-chain integrity, matching ci.yml's upload pin.
	if: always()
	uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
	with:
	name: sbom-cyclonedx
	path: \|
	sbom-python.cdx.json
	sbom-ide.cdx.json
	if-no-files-found: ignore

	trivy:
	name: trivy (container image vulnerabilities)
	runs-on: ubuntu-latest
	# ADVISORY for its first cycle (continue-on-error): scan the built engine image for OS-package and
	# Python CVEs — the supply-chain layer pip-audit (lockfile only) and bandit/semgrep (source only)
	# never see: the Debian base, system libs, and (on the -sqlserver variant) the msodbcsql18/unixODBC
	# apt layer. Once a baseline run confirms it's clean, DELETE `continue-on-error` to make it BLOCKING
	# like the other gates here. There's now a real docker/Dockerfile, so the image is a live surface.
	continue-on-error: true
	# Daily cron + on-demand only (CI cost): the scan INPUT (Dockerfile + base image + locks) changes
	# rarely while trivy's vuln DB changes daily — the cron is the natural cadence, and ci.yml's
	# docker-smoke still covers image buildability on merges. Advisory and NOT a required check, so
	# this gate can never wedge a PR. If trivy is later promoted to BLOCKING (deleting
	# continue-on-error per the note above), REMOVE this gate too so it runs pre-merge again.
	if: github.event_name == 'schedule' \|\| github.event_name == 'workflow_dispatch'
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	- name: Build the slim engine image
	# The slim default the docker-smoke leg (ci.yml) and the Dockerfile's `docker build` docs produce.
	run: docker build -f docker/Dockerfile -t messagefoundry:scan .
	- name: Install Trivy (pinned)
	run: \|
	# Pin the release; bump deliberately (same posture as the gitleaks job). Verify the tag exists
	# at https://github.com/aquasecurity/trivy/releases if this step 404s.
	VER=0.71.2
	curl -sSfL "https://github.com/aquasecurity/trivy/releases/download/v${VER}/trivy_${VER}_Linux-64bit.tar.gz" \
	\| tar -xz -C /tmp trivy
	sudo install -m 0755 /tmp/trivy /usr/local/bin/trivy
	trivy --version
	- name: Generate the container-image SBOM (CycloneDX — OS + Python layers)
	# The image SBOM is the "container contents" manifest NTIA's Consumer Playbook expects for a
	# containerized artifact: the Debian base + system libs AND the installed Python runtime, none of
	# which the lockfile/npm SBOMs capture. Retained so an operator can inventory the running image.
	# RUN BEFORE the vuln scan on purpose: the scan's `--exit-code 1` fails its step on a fixable CVE,
	# which (default `if: success()`) would SKIP any SBOM step placed after it — so the inventory would
	# vanish exactly on the vulnerable runs that need it. ubuntu-latest ships python3; sbom_finalize is
	# stdlib-only. ADR 0149.
	run: \|
	trivy image --no-progress --format cyclonedx --output sbom-image.cdx.json messagefoundry:scan
	python3 scripts/security/sbom_finalize.py sbom-image.cdx.json --phase build
	- name: Upload the image SBOM
	uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
	with:
	name: sbom-container-image
	path: sbom-image.cdx.json
	- name: Scan the image (fixable HIGH/CRITICAL fail the build)
	# --ignore-unfixed keeps the gate ACTIONABLE: a Debian CVE with no upstream fix can't be
	# remediated by us and shouldn't redden CI; only patchable HIGH/CRITICAL fail. --scanners vuln
	# keeps this to CVEs (secrets stay gitleaks' job). --vex applies our maintained OpenVEX so a
	# vulnerability we've assessed as not_affected/fixed is suppressed here too — the SAME file
	# operators feed their scanners (--show-suppressed logs what was suppressed and why). Extend to
	# `--target runtime-sqlserver` (more OS surface) once green. Trivy auto-pulls its vuln DB from ghcr.io.
	run: \|
	trivy image --no-progress --scanners vuln \
	--vex security/vex/messagefoundry.openvex.json --show-suppressed \
	--severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 \
	messagefoundry:scan
	- name: Lint the Dockerfile for misconfig (informational)
	# IaC checks (non-root USER, pinned base, secrets in layers). Informational only — never fails.
	if: always()
	run: trivy config --no-progress --severity HIGH,CRITICAL docker/Dockerfile \|\| true

	bandit:
	name: bandit (Python SAST)
	runs-on: ubuntu-latest
	# BLOCKING: the baseline is clean, so any NEW insecure pattern (that isn't a reviewed per-line
	# `# nosec`) fails the build instead of merging unnoticed.
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
	with:
	python-version: "3.14"
	- name: Scan source for insecure patterns
	run: \|
	# HASH-PINNED from the lock (ADR 0034 §3); bandit's version is the findings baseline of a
	# blocking gate — the `# nosec`-parsing incident behind that pin is recorded beside the pin
	# itself, in pyproject.toml's [dependency-groups].ci-scanners. Bump there, then re-export.
	python -m pip install --require-hashes -r ci/locks/ci-scanners.lock
	# Skips are reviewed-safe project idioms. The subprocess sites (B603/B607) are annotated
	# per-line with `# nosec` at the call instead of a wholesale skip, so a NEW unreviewed
	# subprocess call is still flagged (review low-27):
	# B101 assert — type-narrowing invariants, not security checks
	# B404 import subprocess — the calls themselves are per-line nosec'd (icacls/sc/ruff/mypy)
	# B311 random — synthetic-data generators only; tokens use `secrets`
	# B110 try/except/pass — best-effort resource cleanup
	# B608 hardcoded SQL — f-strings interpolate only literal column names; values are bound
	# params (verified — see security review STORE-4)
	# `tee/` is in-tree vendored SOUP (a standalone relay) — scanned to the SAME bar as the
	# engine. Its one urllib GET (tee/mefor_api.py) is per-line `# nosec B310`: the URL scheme is
	# fixed by operator config (default localhost), never message-derived.
	#
	# SCOPE = the whole repo minus the excludes below, which mirrors the pre-commit bandit hook
	# (.pre-commit-config.yaml) EXACTLY. It was `-r messagefoundry tee` while the hook scanned
	# everything except tests/harness/samples — so scripts/ (security tooling, subprocess-heavy)
	# was gated locally and by nothing in CI. Touching a file there failed the commit on findings
	# no CI run could report. Widening cost nothing: the repo is already clean at this bar.
	# tests/harness/samples — intentional non-production idioms (asserts, synthetic-data RNG)
	# packaging/messagefoundry-webconsole/tests — the same, one directory deeper: 20 B105/B106
	# "hardcoded password" hits that are all literal test credentials
	# ide/ — TypeScript; no Python to scan
	# docs/benchmarks/results — archived measurement artifacts, not maintained source
	# tests/test_lint_scope_parity.py fails if this and the hook drift apart again.
	bandit -r . --skip B101,B110,B311,B404,B608 \
	--exclude ./tests,./harness,./samples,./ide,./docs/benchmarks/results,./packaging/messagefoundry-webconsole/tests,./.venv,./node_modules

	gitleaks:
	name: gitleaks (secret scan)
	runs-on: ubuntu-latest
	# BLOCKING: full-history scan came back clean; a newly-introduced secret now fails the build.
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	fetch-depth: 0 # full history so the scan also covers earlier commits, not just the tip
	- name: Install gitleaks (pinned + checksum-verified)
	run: \|
	# Pin the release; bump deliberately. Verify the tag exists if this step 404s.
	#
	# VERIFIED, not just version-pinned. This was `curl ... \| tar -xz` straight into a pipe: a
	# tag pin says WHICH artifact to fetch, not that the bytes received are that artifact, and
	# this runs inside a REQUIRED gate. The sbomqs step in this same file already had the answer
	# (download the release's own checksums file and sha256sum -c it), so this was unfinished
	# scope rather than an accepted risk. Mirroring that shape exactly.
	VER=8.18.4
	asset="gitleaks_${VER}_linux_x64.tar.gz"
	base="https://github.com/gitleaks/gitleaks/releases/download/v${VER}"
	curl -sSfL "${base}/${asset}" -o "$asset"
	curl -sSfL "${base}/gitleaks_${VER}_checksums.txt" -o gitleaks_checksums.txt
	# Keep the asset's canonical name: sha256sum -c verifies BY the filename in the line, so a
	# renamed download would silently verify nothing.
	grep " ${asset}$" gitleaks_checksums.txt \| sha256sum -c -
	tar -xzf "$asset" gitleaks
	sudo install -m 0755 gitleaks /usr/local/bin/gitleaks
	gitleaks version
	- name: Scan repository for secrets
	run: gitleaks detect --config .gitleaks.toml --redact --verbose --no-banner

	semgrep:
	name: semgrep (project SAST rules)
	runs-on: ubuntu-latest
	# BLOCKING: the first run was 0 findings; a new match for any rule now fails the build.
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
	with:
	python-version: "3.14"
	- name: Run the MessageFoundry rules
	run: \|
	# PINNED to an exact version, not `~=1.90`. A compatible-release range silently adopts every
	# new 1.x, and a new semgrep release can add rules or change taint propagation — which reds a
	# green PR for a reason that has nothing to do with its diff. Same posture as bandit==1.9.4
	# above and zizmor==1.5.2: bump deliberately, in a PR that also clears any new findings.
	# (1.172.0 is what `~=1.90` resolved to on 2026-07-29, so this pin is a no-op today.)
	python -m pip install --upgrade pip "semgrep==1.172.0"
	# --error: any finding fails the step — and the job, now that this gate is blocking.
	#
	# SCOPE = the whole repo minus the excludes below, which is the bandit exclude set in this
	# same file EXACTLY. It was the allow-list `messagefoundry tee`, which never covered
	# scripts/ (the security tooling itself), messagefoundry_webconsole/ (the operator UI) or
	# docker/ — 59 tracked .py files that the sibling bandit gate already scans, held to none of
	# this project's own dangerous-sink rules. An allow-list cannot be kept in step with "the
	# project" by hand; `.` minus explicit excludes cannot go stale when a package is added.
	# `tee/` (in-tree vendored SOUP) stays in scope for the reason it always was: the standalone
	# relay is held to the SAME bar as the engine. Widening cost nothing — clean at this bar.
	# tests/harness/samples — intentional non-production idioms. This exclude is here for
	# PARITY WITH BANDIT — which is what the parity test enforces —
	# NOT as a prediction about findings. semgrep's own default
	# .semgrepignore already excludes tests/: that is the recorded
	# fact scripts/ci/assert_semgrep_handler_taint.py:41 depends on,
	# and the reason that step copies its fixture to a temp dir to
	# get it analysed at all. So this flag is explicit and probably
	# redundant, stated that way on purpose — a gate's scope must be
	# readable from its own argv, never inherited from a tool default
	# a version bump can change. (The only two sinks in the whole
	# tree that match these rules do sit under tests/: an `eval`
	# taint fixture and a `pickle.loads` sandbox test.)
	# packaging/messagefoundry-webconsole/tests — the same, one directory deeper
	# ide/ — TypeScript; no Python to scan
	# docs/benchmarks/results — archived measurement artifacts, not maintained source
	# The `./` prefix bandit uses is DROPPED on purpose: semgrep matches --exclude as a GLOB,
	# not a path, so `./tests` matches nothing and the flag would be inert — copying bandit's
	# string byte-for-byte yields an exclude list that excludes nothing. (Whether tests/ would
	# then actually be SCANNED is the separate question the default .semgrepignore above
	# answers; the point is that this gate's scope must not rest on that answer.)
	# Whether those globs are root-anchored or match a bare directory NAME at any depth is not
	# settled here, and is not load-bearing — measured over the WHOLE tracked tree, which is
	# the population this scan now covers: the only nested directory matching any of these
	# names is packaging/messagefoundry-webconsole/tests, excluded explicitly on both sides
	# anyway (2026-08-04).
	# tests/test_lint_scope_parity.py fails if this scope and bandit's drift apart again. It
	# also asserts, separately, that no --exclude here carries the `./` prefix (its normalised
	# set comparison would otherwise call `./tests` and `tests` identical), that no `--include`
	# re-narrows the scan behind the positional `.`, and that `--error` survives.
	semgrep --config .semgrep --error --metrics off \
	--exclude tests --exclude harness --exclude samples --exclude ide \
	--exclude docs/benchmarks/results --exclude packaging/messagefoundry-webconsole/tests \
	--exclude .venv --exclude node_modules \
	.
	- name: Handler-config taint rules — validate, regression gate, scan samples (ADR 0144 Inc 3)
	run: \|
	# semgrep is already installed by the step above (same job). This is why the packaged rules
	# need NO project dependency to validate — the CI-installed semgrep is the authority.
	RULES=messagefoundry/security/semgrep/handler-security.yml
	semgrep --validate --config "$RULES" # rule-syntax gate
	# Behavioural gate: the two recovered false-negatives (inter-statement taint + aliased-import)
	# and the other rules fire exactly as annotated; the `# ok` cases stay clean. A --json assert
	# (not `semgrep --test`, which crashes on Windows path-pairing) so it is deterministic.
	python scripts/ci/assert_semgrep_handler_taint.py
	semgrep --config "$RULES" --error --metrics off samples/config # shipped samples stay clean

	# zizmor (GitHub Actions static analysis) moved to its own paths-filtered workflow
	# (.github/workflows/zizmor.yml): it lints .github/** only, so running it on every PR was pure
	# cost — it now runs when the workflow surface changes, plus the same daily cron + dispatch.

	crypto-inventory:
	name: crypto-inventory (ASVS 11.1.3 discovery gate)
	runs-on: ubuntu-latest
	# BLOCKING: enumerate every crypto call site (hashlib/secrets/hmac/ssl/argon2/cryptography) and diff
	# against the maintained inventory in scripts/security/crypto_inventory_check.py (machine-readable
	# companion to docs/ASVS-L2-PHASE0-CHANGES.md §4). A new/moved/undocumented crypto usage —
	# or a stale inventory entry — fails the build, so the inventory can't silently drift (WP-L3-02).
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
	with:
	python-version: "3.14"
	- name: Diff crypto call sites against the inventory
	# Stdlib-only (no install). Exits non-zero — failing the build — on any drift.
	run: python scripts/security/crypto_inventory_check.py

	forbidden-content:
	name: forbidden-content (customer/PHI leak guard)
	runs-on: ubuntu-latest
	# BLOCKING (required context): scan the WHOLE tracked tree for customer/PHI-adjacent strings --
	# partner/site names, the real production estate, routable host IPs, internal worktree slugs and
	# absolute home paths. These are not secrets (gitleaks will not flag them) but must never land on
	# this public repo. The committed scanner ships only STRUCTURAL detectors plus a synthetic
	# .example; the real token list is NEVER committed -- it arrives from the MEFOR_FORBIDDEN_TOKENS
	# secret here, and locally from a git-ignored scripts/security/scan-tokens.local.txt.
	steps:
	- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
	with:
	python-version: "3.14"
	- name: Scan the tracked tree for forbidden content
	# Stdlib-only scanner (no install).
	#
	# MEFOR_MIN_DETECTORS is what makes "fail-closed" mean anything. Requiring a token SOURCE only
	# proves one arrived: a partially-mangled secret still loaded as few as 1 of 21 detectors and
	# passed with a green tick, and losing just the final line silently disabled every site-code
	# detector. The floor is PER-SECTION, not a bare total -- a total is a SUM, so growth in a cheap
	# section masks collapse in an expensive one (names 7->1 alongside estate 13->19 still totals 21).
	# It is a FLOOR: adding tokens needs no CI change, losing them fails the build.
	#
	# zizmor: the secret is never interpolated into this run: body. It arrives as the step-level env
	# var (an opaque value, not an Actions-expression sink).
	env:
	MEFOR_FORBIDDEN_TOKENS: ${{ secrets.MEFOR_FORBIDDEN_TOKENS }}
	# Whether this run CANNOT legitimately see the secret. Empty on pushes and same-repo PRs.
	IS_FORK_PR: ${{ github.event.pull_request.head.repo.fork }}
	run: \|
	if [ -n "$MEFOR_FORBIDDEN_TOKENS" ]; then
	# Deliberately NOT written to scripts/security/scan-tokens.local.txt: the scanner resolves
	# the env var first, so that file would never be read -- it would only drop the full real
	# token list into the job workspace for every later step to see.
	export MEFOR_REQUIRE_TOKENS=1
	export MEFOR_MIN_DETECTORS=names=7,estate=13,site_prefixes=1
	echo "token list loaded from the MEFOR_FORBIDDEN_TOKENS secret (fail-closed, per-section floor)."
	elif [ "$IS_FORK_PR" = "true" ]; then
	echo "fork PR -- the secret is unavailable BY DESIGN; structural-only scan."
	else
	# Degrading here would be SILENT and PERMANENT: a renamed, deleted, environment-scoped or
	# rotated secret takes the fork branch forever, scanning nothing while the REQUIRED context
	# stays green. Absent-on-a-non-fork is a broken gate, not a mode.
	echo "::error::MEFOR_FORBIDDEN_TOKENS is absent on a non-fork run. The customer-leak gate cannot run. Check the repository secret (repo-scoped, not environment-scoped)." >&2
	exit 2
	fi
	python scripts/security/scan_forbidden.py --path .

Navigation Menu

Uh oh!

feat(api): the engine always serves TLS, minting on first run (BACKLOG #1276 part A, ADR 0172) #2421

Workflow file

feat(api): the engine always serves TLS, minting on first run (BACKLOG #1276 part A, ADR 0172) #2421

Uh oh!

Workflow file for this run