feat(api): the engine always serves TLS, minting on first run (BACKLOG #1276 part A, ADR 0172)

docs/ASVS-L2-PHASE0-CHANGES.md

@@ -125,6 +125,7 @@ for, not how it is protected before it gets there.
 | AD transport | LDAPS (TLS) with `CERT_REQUIRED` by default; optional internal CA via `ad_tls_ca_cert_file` | OS / configured CA trust | Managed by the directory / OS trust store |
 | SQL Server transport | TLS via ODBC Driver 18 (`Encrypt=yes`, `TrustServerCertificate=no` by default) | Server certificate | Managed by SQL Server / OS trust store |
 | Console → engine TLS (a remote native client — the Qt-free `apiclient`, today the test harness; the PySide6 desktop console is retired) | Verifies the engine API server cert: **OS trust store** by default (`truststore.SSLContext`) or a pinned PEM via `cacert` (`ssl.create_default_context`); opt-in client cert (mTLS) via `load_cert_chain`; `ssl` in `apiclient/client.py` (CONSOLE-3; extracted from the since-deleted `console/client.py` per ADR 0088) | OS trust store / operator-supplied CA PEM (`cacert`) | Managed by the OS trust store; `cacert` for a self-signed / internal-CA engine |
+| Load-test harness → spawned-engine TLS (BACKLOG #1276 part A) | The engine always serves TLS now and mints a self-signed placeholder when no operator cert is configured — which every harness driver hardcoded `http://` against, so no spawned node ever became healthy. The harness supplies its own certificate instead of chasing the one the engine mints: one EC P-256 pair minted per process via `pki.make_self_signed`, handed to each node as `[api].tls_cert_file`/`tls_key_file` (`ensure_api_tls_material` honours it and never mints over it), and pinned by every client that talks to that node (`ssl.create_default_context(cafile=...)` in `harness/load/tlsmat.py`) — the same posture `apiclient.EngineClient`'s `cacert` already supports, just resolved locally instead of by an operator. **Usage scope: non-prod only.** The pair lives in a per-process temp directory, covers loopback names alone (`127.0.0.1`/`localhost`/`::1`), and is inherited by a spawned child harness process (`connscale-remote`) via environment so a parent and its child never mint two different anchors for the same engines. A second host in a two-box shardcert rig is out of scope by design — this anchor cannot cover a certificate it never minted, so those URLs stay `http://` and say why inline. | Minted in-process (non-persistent, never written outside a per-run temp dir) | Regenerated every harness run; nothing to rotate |
 | Tray → engine TLS (local status probe, [ADR 0113](adr/0113-windows-tray-service-manager-stdlib-ctypes-tokenless.md) 2026-07-22 amendment) | Verifies the engine API server cert on the tray's **tokenless** `/health` + `/ui` probes when `[api].tls_cert_file` makes the loopback bind serve https: **OS trust store only** (`truststore.SSLContext`); `ssl` in `tray/probe.py`. **No pinned-PEM option and no `verify=False` escape** — an AST test in `tests/test_tray_probe.py` freezes that | Windows machine trust store (an internal-CA/AD-CS cert verifies as-is; a self-signed engine cert is installed under Trusted Root) | Managed by the OS trust store; a verification failure renders the engine `DOWN`, never an unverified connection |
 | Cert tooling — `.pfx` import / read-only inventory / self-signed dev cert (BACKLOG #71/#72) | `cryptography` in [`pki.py`](../messagefoundry/pki.py) — the single PKI call site for the `cert` CLI group: PKCS#12/.pfx import (`pkcs12.load_key_and_certificates`) writes the leaf cert + private key + CA chain to the PEM files the TLS loaders already read; a **read-only** inventory reads only **public** cert facts (subject/issuer/notAfter/SAN/days) via `x509`; `make_self_signed` mints an **EC P-256 / SHA-256** self-signed cert for **non-prod** bring-up. `pipeline/cert_expiry.py` shares this module's `read_cert_facts` (so it no longer imports `cryptography` itself). **Usage scope:** an operator CLI utility — it imports/serializes/inspects operator-supplied cert material and mints throwaway dev certs; it holds no long-lived engine key, signs no message, and encrypts nothing at rest. The imported/minted **private-key** PEM is written `O_EXCL` + `0o600` + the `_secure_file` DACL; the `.pfx` passphrase is env-only (`MEFOR_PFX_PASSWORD`), never a CLI arg, never logged/echoed/put in an exception | Operator-supplied `.pfx` bundle → cert/key/CA PEM files on disk (`--out-dir`) | Managed by the operator / PKI; self-signed dev certs are disposable (default 365-day validity) |
 | Engine/console seam identity (BACKLOG #1220) | SHA-256 over the **discovered** engine/console contract surface, truncated to 16 hex characters, published as `ENGINE_UI_SEAM` in [`api/_ui_seam.py`](../messagefoundry/api/_ui_seam.py) and derived by `hashlib` in [`scripts/webconsole_seam_snapshot.py`](../scripts/webconsole_seam_snapshot.py). **A change detector, not a security control** — no secret, no key, no message authentication, and nothing user- or PHI-derived is hashed; the input is a serialization of public type signatures, field names, enum members and `Literal` values. It replaced a hand-picked incrementing integer, which two unlanded branches had both claimed for two different contract changes while the golden snapshot auto-merged clean under one value. What it needs is accidental-collision avoidance across the contract surfaces this project will ever produce: at 64 bits the birthday bound is 2.7e-12 for 10,000 distinct surfaces, about 500x the ~20 seam moves to date. Preimage resistance buys nothing — anyone able to craft a colliding surface already has commit access to the file holding the constant. SHA-256 rather than BLAKE2 or a non-approved digest only because the engine renders a `fips_mode` attestation and a non-approved hash in the shipped surface invites a FIPS question for no gain | Not a secret: the digest is committed in source and mirrored in the console's `SUPPORTED_ENGINE_SEAMS` | Recomputed by `scripts/webconsole_seam_snapshot.py --write` whenever the contract changes; a stale value fails `tests/test_webconsole_seam_snapshot.py` |

docs/BACKLOG.md

@@ -13310,6 +13310,7 @@ measurement from this row's subject and it is named here rather than performed.*
 > 🔢 **Re-scored 2026-08-20 -> P2.** Value **6/10** · Difficulty **6/10** · _big bet_. Nothing mints a certificate on first start: tls_enabled is literally bool(tls_cert_file) with both keys defaulting None, so an unconfigured engine opens a cleartext socket. Value 6 because the minting primitive and its CLI verb already ship, giving an operator a real if awkward workaround; difficulty 6 because the change spans the serve path, five written DEV ONLY prohibitions that must be rewritten in the same PR, and the scheme-inference seam at tray/config.py:207-219 that the harness, apiclient and IDE share, which is in-repo client work rather than deployment migration. _(previously unscored.)_
 >
 > **Filed 2026-08-16 - not started. THE ENGINE SERVES PLAIN HTTP WHEN NOBODY HAS SUPPLIED A CERTIFICATE, AND IT ALREADY OWNS EVERY PIECE NEEDED TO MINT ONE.** `[api].tls_cert_file` and `[api].tls_key_file` both ship `None` ([`config/settings.py:758-759`](../messagefoundry/config/settings.py)), `tls_enabled` is literally `bool(self.tls_cert_file)` (`settings.py:828-831`), and [`__main__.py:2840`](../messagefoundry/__main__.py) builds an SSL context **only** when that property is true. With no certificate configured, `uvicorn.run` at `__main__.py:2868` opens a cleartext socket. **THE CHANGE: when no certificate is configured, mint a self-signed one on first start, persist it, and serve HTTPS.**
+> **PART A BUILT 2026-08-25 -- PR 575, ADR 0172. PROGRESS NOTE, NOT A CLOSURE: the item stays OPEN.** `ensure_api_tls_material` mints a self-signed pair beside the store on first run, beneath any operator-supplied `[api].tls_cert_file`, and returns `None` under `tls_terminated_upstream` so it cannot break a proxy hop that already terminates in front. **So the "not started" above is superseded for part A only.** STILL OPEN: nothing re-mints an EXPIRED generated pair -- `build_api_ssl_context` performs no expiry check, so a site past day 365 would serve an expired certificate; the rotation shape is undecided, and `CertExpiryRunner` alarms on this path meanwhile. Written by the LANDER to satisfy the required backlog-hygiene gate on PR 575, whose author deliberately did not touch this banner. **Status glyph untouched; correct this text freely.**
 > **OPERATOR-SUPPLIED CERTIFICATES KEEP PRECEDENCE, and the mechanism is already that way round.** The generated certificate is a **first-run fallback beneath** `[api].tls_cert_file`/`tls_key_file`, never a replacement: the fallback is reached only when `tls_enabled` is false, which is exactly the state in which no operator certificate exists. A site that sets those keys sees no behaviour change at all.
 > **THE MINTING PRIMITIVE IS BUILT AND ALREADY DRIVEN END TO END BY A CLI VERB -- this is wiring, not cryptography:**
 >     messagefoundry/pki.py:136                 make_self_signed(cn, sans, days) -> (cert_pem, key_pem)

docs/adr/0172-the-engine-always-serves-tls-minting-a-self-signed-certificate-on-first-run.md

@@ -0,0 +1,99 @@
+<!-- SPDX-License-Identifier: AGPL-3.0-or-later -->
+<!-- Copyright (C) 2026 MessageFoundry Organization and contributors -->
+
+# ADR 0172 — The engine always serves TLS, minting a self-signed certificate on first run
+
+- **Status:** Accepted (2026-08-22)
+- **Date:** 2026-08-22
+- **Supersedes:** [ADR 0143](0143-web-console-on-by-default-disableable-with-loopback-secure-context-browser-hardening.md)'s *decision*, not its analysis — see "What of 0143 survives" below
+- **Related:** [ADR 0002](0002-phase2-transport-security-and-strong-auth.md) · [ADR 0065](0065-web-ops-dashboard.md) · [ADR 0118](0118-secure-by-default-security-configuration-section.md) · BACKLOG #1276
+
+## Context
+
+`[api].tls_cert_file` and `tls_key_file` both shipped `None`, and `tls_enabled` was literally
+`bool(self.tls_cert_file)`. An engine nobody had configured therefore opened a **cleartext
+socket** — `uvicorn.run` with no `ssl_context_factory`.
+
+The minting primitive already shipped and was already driven end to end by a CLI verb:
+`pki.make_self_signed`, and `_write_private_key` with its `O_EXCL` + `0o600` + Windows-DACL
+sequence. Nothing needed inventing; the gap was wiring.
+
+**ADR 0143 considered exactly this change and declined it.** Its own words: *"A full fix —
+terminate TLS on the loopback bind so `effective_https` is true and everything (headers +
+secure cookie + HSTS) engages — is an **XL**: it means moving the whole API to https by default
+and migrating every client (harness, `apiclient`, tray, IDE) in lockstep. Out of scope here."*
+It shipped an http-safe hardening subset over the loopback secure-context **without** auto-TLS.
+
+That decline was reasonable on the information it had. **The sizing claim it rests on is
+measurably false**, which is why this ADR supersedes the decision rather than merely amending it.
+
+## The measurement that overturns the sizing
+
+0143 sized the client migration as four clients moving in lockstep. Measured on `origin/main`:
+
+| Client | How it decides the scheme |
+|---|---|
+| tray | **Infers** — `service_toml_uses_tls`, exactly ONE caller (`tray/config.py`) |
+| `apiclient` | **Does not.** Zero references to `tls_cert_file`; it is *given* a base URL and only validates the scheme |
+| IDE | **Does not.** Its `tls_cert_file` hits are MLLP *connector* schema — the same name for a different setting |
+| harness | **Does not infer — it assumes.** Hardcoded `http://127.0.0.1:8765` |
+
+So it is **one inference site plus a set of hardcoded defaults**, not a four-way lockstep
+migration. Each default is a one-line flip. The XL that justified declining the full fix does
+not exist.
+
+## Decision
+
+**The engine always serves TLS.** An operator-supplied `[api].tls_cert_file` always wins; with
+none configured the engine mints a self-signed pair on first run, persists it, and serves HTTPS.
+
+1. **Unconditional, deliberately.** A *conditional* scheme is what let the tray, the harness and
+   the DAST target each decide it their own way. Clients cannot disagree about a scheme that has
+   no conditional — the divergence is removed rather than managed.
+2. **Beneath the operator, never instead of.** The fallback is reached only when no certificate
+   is configured, so a site with its own chain sees no behaviour change at all.
+3. **NOT in every topology.** `tls_terminated_upstream` (+ `trusted_proxies`) declares a reverse
+   proxy terminating TLS *in front* of the engine and speaking plaintext to it. Minting there
+   would break the proxy's own hop rather than harden anything. **"Always serves TLS" means the
+   engine never leaves a hop unprotected — not that it terminates TLS in every deployment.**
+4. **The generated pair is a placeholder to be replaced.** Self-signed, so no chain of trust:
+   strictly better than cleartext, strictly worse than an operator chain. A browser shows a trust
+   interstitial until it is imported.
+5. **Mint-once, then reuse.** `_write_private_key` refuses to overwrite, so a second start loads
+   rather than rotating.
+6. **Re-minting an expired pair is AUDITED, never silent** (owner ruling, 2026-08-22). Nothing
+   re-mints today and `build_api_ssl_context` performs no expiry check, so an unrefreshed pair
+   would serve an expired certificate every client rejects. *Silent* is the defect in replacing a
+   key on disk, not *replaces*: an audited re-mint keeps this decision true without a human and
+   leaves a trail. Timing (at startup versus inside the expiry warn window) is a build detail —
+   both mutate disk identically, so the security question is settled for both.
+
+**Storage:** beside the store database. That directory is already the engine's own writable
+state, already operator-controlled via `--db` / `[store].path`, and is **not** operator-authored
+configuration. *Rejected:* a new `[api].tls_generated_dir` setting — a knob for a question with
+one sensible answer. *Rejected outright:* the engine writing `tls_cert_file` into the operator's
+service TOML. An engine that edits operator configuration is a surprising side effect, and it was
+not needed once the scheme stopped being conditional.
+
+**Lifetime:** 365 days, inheriting the `cert self-signed` CLI default rather than inventing a
+second lifetime for the same primitive.
+
+## What of ADR 0143 survives
+
+**Its analysis stands; only its decision is superseded.** 0143's diagnosis — that
+`effective_https` gated two coupled concerns on one signal, and that a secure cookie over
+cleartext http is dropped by Chrome and Safari and *breaks login* — is correct and is precisely
+why this change is the better end state. Its `app.state.loopback` mechanism becomes vestigial
+where the engine terminates TLS, because `effective_https` is now true on that bind.
+
+It is not vestigial everywhere: the `tls_terminated_upstream` topology in decision 3 still
+reaches the engine over plaintext, and 0143's http-safe subset is what covers it.
+
+## Consequences
+
+- An operator reaching the console for the first time gets a **trust interstitial** until the
+  generated certificate is imported. `docs/TRAY.md` already documents that import.
+- Every first-party client default becomes `https`. `service_toml_uses_tls` becomes vestigial.
+- **No deployment axis** ([§0](../../CLAUDE.md)) — zero instances, so nothing is served in the
+  clear today and no upgrade breaks anyone. The change is cheap now and gets dearer with every
+  client that learns the scheme its own way.