Merge branch 'main' into claude/builder-2-1211-capture

Author: wshallwshall

docs/BACKLOG.md

@@ -9416,6 +9416,22 @@ Nothing here touches the TLS/FTPS context in the same module, which was already
 > 🔢 **Re-scored 2026-08-20 -> P2.** Value **6/10** · Difficulty **3/10** · _quick win_. The concrete IDE defect is closed by cspNonce.ts and its negative test, so what remains is the recovery-code shortfall (totp.py:58-60, 31^15 or about 74.3 bits, ten issued by default at settings.py:1798) plus the cross-language inventory question the crypto gate still cannot answer (crypto_inventory_check.py:456-472 excludes ide/ by invariant rather than inventorying it). An operator can only work around the shortfall by setting the count to 0, which is the awkward-workaround band. _(was 6/10 · 4/10.)_
 > Research: done 2026-08-20
 >
+> **PARTIAL 2026-08-25 -- THE PROPOSED BUILD SHIPPED (PR #603). THE ROW STAYS OPEN: the item's own
+> closing act is a scorecard re-score, not this build, and that re-score is explicitly not run.**
+> `_RECOVERY_GROUPS` went 3 -> 6 over the existing alphabet (148.63 bits, 142.98 after the
+> multiplicity adjustment -- 15 bits of margin against the 128-bit floor rather than 59 bits short
+> of it); the bare 26-character widening named below as a trap was NOT taken, matching this item's
+> own reasoning. `_generate_policy_password` now draws 24 bytes rather than 16, so the
+> `token_urlsafe` byte/character mismatch this research flagged as "safe by accident" no longer
+> depends on accident. An entropy-floor regression test derives its bit count from the three
+> shipped constants and the validator's own maximum count rather than transcribing either, with a
+> mutation control run against the shipped constant (not just asserted): reverting
+> `_RECOVERY_GROUPS` to 3 fails the new test, restored by byte-copy with the hash verified.
+> **Still explicitly open, all named in this item's own proposed-work list and none touched by
+> PR #603:** the ASVS 11.5.1 scorecard re-score itself; the scope-completeness adversarial pass this
+> item's own caveat says was never run; recording the five zero-margin values its census already
+> flags; and the cross-language randomness inventory arm.
+>
 > **Filed 2026-08-08 - not started. RESEARCH item: the goal is an HONEST pass, and "cannot honestly reach pass" is a valid finding.** ASVS **11.5.1** (L2) currently scores **partial**. The pinned verb sets a 128-bit floor on every non-guessable random value. TOTP recovery codes are CSPRNG-drawn but reach only about 74.3 bits, and the IDE extension mints its webview CSP nonces from `Math.random()`.
 > Verdict: research
 > Closing-act: scorecard-rescore
@@ -13218,6 +13234,7 @@ measurement from this row's subject and it is named here rather than performed.*
 
 > 🔢 **Re-scored 2026-08-20 -> P3.** Value **4/10** · Difficulty **2/10** · _fill-in_. One limb remains and it is a pure refactor: logging_setup.py:66-70 still derives its own table from range(0x20) plus 0x7F and does not import controlchars at all, while the module now states the alphabet once at :67 with both arms reading it. Value 4 because nothing is mis-screened today, two of the three named copies already agree, and at least two further independent derivations sit outside this item's scope (spreadsheet.py:37, soap.py:263), so folding logging_setup in buys the stated cross-module leverage for one file rather than repo-wide. Difficulty 2 for one import, one derivation with an explicit tab subtraction, a test pinning that tab stays excluded, and renaming the acceptance test's scrub_control_chars, which returns zero code hits. _(previously unscored.)_
 >
+> **PARTIAL 2026-08-25 -- THE THIRD SPELLING IS NOW GONE TOO. STILL STAYS OPEN** (spreadsheet.py:37 and soap.py:263 are named out-of-scope above, and remain so). `logging_setup.py` now imports `controlchars` and states its own table as that alphabet minus TAB, proved byte-identical to the pre-change table entry-for-entry (32 entries both sides) rather than asserted. Verified before writing this note, not taken from the branch's own claim.
 > **PARTIAL 2026-08-20 -- ONE OF THE TWO SPELLINGS INSIDE THE MODULE IS GONE; THE THIRD, IN ANOTHER MODULE, IS NOT. THIS ITEM STAYS OPEN.** `controlchars.py` now states the set once, in `_is_control_char` at `:67`, and both `has_control_char` and `strip_control_chars` read it -- so the module's own interior no longer contradicts its docstring. **But `logging_setup.py:66-70` still re-derives the same set independently** (`for _i in range(0x20)` plus `[0x7F]`), and the landing branch does not touch that file at all. The item names **three** spellings and asks for **one definition of the SET**; two of three now share one. **The leverage the module exists to provide is still absent across the module boundary:** widen `_is_control_char` and it reaches neither `_CTRL_TRANSLATION` nor anything reading it, and nothing reports the omission. Measured on the landing branch, not inferred. Recorded by the lander under ADR 0165, because a closing banner that lists what a change fixed and not what it left is half a record.
 > 🔢 **Filed 2026-08-15 - not started. THE CONSOLIDATION DOES NOT CONSOLIDATE ITS OWN TWO FUNCTIONS.** [`controlchars.py`](../messagefoundry/controlchars.py) was created by [#1253](BACKLOG.md) to write the C0/DEL test **once**; its docstring is titled *"The C0/DEL control-character test, written once"* and ends *"THE POINT IS THE COPYING PRACTICE, not the seven known lines. If you need this test, import it."* **It then spells the predicate out twice inside itself**, and a third statement of the same set lives in `logging_setup`.
 > **MEASURED, three independent spellings of one set:**

messagefoundry/auth/service.py

@@ -602,7 +602,11 @@ def _generate_policy_password(self) -> str:
         is held to the same bar operators are. ``token_urlsafe(n)`` yields ~1.33·n chars (so length is
         guaranteed ≥ ``min_length``); the loop covers the astronomically-unlikely breach/context hit
         or an opt-in character-class requirement a given token happens to miss."""
-        length = max(16, self._policy.min_length)
+        # 24 BYTES (192 bits), not 16. token_urlsafe's argument is a byte count, and the floor is
+        # raised here rather than left at the policy minimum because min_length is a CHARACTER count
+        # -- passing it as bytes happens to be safe but ties an entropy floor to a legibility knob an
+        # operator may lower (BACKLOG #1172).
+        length = max(24, self._policy.min_length)
         for _ in range(16):
             candidate = secrets.token_urlsafe(length)
             if not self._policy.violations(candidate):

messagefoundry/auth/totp.py

@@ -57,7 +57,14 @@
 # Recovery codes: human-legible groups from an unambiguous alphabet (no 0/O/1/I/L confusion).
 _RECOVERY_ALPHABET = "ABCDEFGHJKMNPQRSTUVWXYZ23456789"
 _RECOVERY_GROUP_LEN = 5
-_RECOVERY_GROUPS = 3  # e.g. "K7QF2-9DMNA-3XZP4" → ~74 bits of entropy
+# SIX groups, not three. Three gave 74.31 bits per code; against the 50-code ceiling
+# `[auth].mfa_recovery_code_count` permits, an attacker needs any ONE of them, so the guessing
+# strength is 74.31 - log2(50) = 68.67 bits -- 59 bits UNDER a 128-bit floor. Six gives 148.63,
+# i.e. 142.98 after the same adjustment, clearing the floor by 15 bits (BACKLOG #1172).
+# The floor is asserted in tests/test_recovery_code_entropy.py, DERIVED from these three constants
+# and from the validator's own ceiling, so none of them can drift away from it silently.
+# e.g. "K7QF2-9DMNA-3XZP4-B8VWR-6JTCH-2YQD5"
+_RECOVERY_GROUPS = 6
 
 
 def generate_secret() -> str:

messagefoundry/logging_setup.py

@@ -32,6 +32,10 @@
 from typing import Any
 
 from messagefoundry.config.tls_policy import harden_cipher_suites
+
+# A LEAF MODULE, imported for its DEFINITION rather than its behaviour (BACKLOG #1273). controlchars
+# imports nothing from this package, so there is no cycle -- checked by import, not assumed.
+from messagefoundry.controlchars import _is_control_char
 from messagefoundry.redaction import redact
 
 __all__ = [
@@ -64,11 +68,33 @@
 
 # C0 control characters (and DEL) escaped to keep one log record on one line. CR/LF are the
 # log-injection vector; tab (0x09) is left intact as benign whitespace.
+#
+# THE ALPHABET IS controlchars._is_control_char's, MINUS TAB (BACKLOG #1273, limb 3). It used to be
+# re-derived here as `range(0x20)` plus a separate `0x7F` line -- a second statement of the same set
+# in a codebase whose controlchars module exists precisely to state it once. The two agreed, so
+# nothing was mis-escaped; the cost is the future-tense one #1239 named and #1253 acted on, that a
+# later widening applied to one copy silently does not apply to the other.
+#
+# THE SUBTRACTION IS THE POINT, so it is written as one. Documenting this as "excluded" and leaving
+# the copy was considered and is refuted by the residual block on #1273: the parsing/sniff.py
+# carve-out earns its separate definition by being BYTE-wise and subtracting a whole allowlist,
+# while this is CHARACTER-wise, escapes CR/LF rather than tolerating them, and differs by EXACTLY
+# ONE code point. Measured: controlchars 33 code points, this table 32, symmetric difference {0x09}.
+# One code point of divergence is a subtraction, not a different predicate.
 _CTRL_TRANSLATION: dict[int, str] = {0x0A: "\\n", 0x0D: "\\r"}
-for _i in range(0x20):
-    if _i not in (0x09, 0x0A, 0x0D):
+# RANGE 0x100, NOT 0x80, AND THAT IS THE DIFFERENCE BETWEEN A REAL FOLD AND A COSMETIC ONE. The
+# alphabet is C0+DEL today, so both bounds produce the identical 32 entries -- proved by the
+# byte-identity check in the commit. But `_is_control_char`'s docstring names widening to C1
+# (U+0080-U+009F) as the deliberate change this shared module exists to make cheap, and a 0x80 bound
+# would silently NOT follow it: the escape table would keep the old alphabet while every other call
+# site moved, which is the exact two-copy drift limb 3 removes. Iterating past the current boundary
+# costs 128 predicate calls at import and makes the widening propagate by construction.
+for _i in range(0x100):
+    # TAB IS THE ONLY SUBTRACTION and test_tab_is_the_only_control_character_left_intact pins it.
+    # CR/LF are excluded from this loop because they get readable escapes above, not because they
+    # are tolerated -- they are the injection vector this whole table exists for.
+    if _is_control_char(chr(_i)) and _i not in (0x09, 0x0A, 0x0D):
         _CTRL_TRANSLATION[_i] = f"\\x{_i:02x}"
-_CTRL_TRANSLATION[0x7F] = "\\x7f"
 
 #: Stamped on every physical line of a record's ``exc_text``/``stack_info`` (BACKLOG #335). A traceback
 #: is multi-line by nature, so collapsing it the way the rendered message is collapsed would cost the

tests/test_logging.py

@@ -927,3 +927,66 @@ def test_serve_time_sync_ok_within_threshold_starts_clean(
         ["serve", "--config", str(tmp_path), "--db", str(tmp_path / "x.db"), "--env", "dev"]
     )
     assert rc == 0
+
+
+# --- BACKLOG #1273 limb 3: ONE definition of the alphabet, with the tab subtraction pinned -------
+#
+# `_CTRL_TRANSLATION` used to re-derive the control-character set as `range(0x20)` plus a separate
+# `0x7F` line -- a second statement of the set that `controlchars` exists to state once. The two
+# agreed, so nothing was mis-escaped. The cost was the future-tense one: a later widening applied to
+# one copy silently does not apply to the other, and nothing reports the omission.
+#
+# These tests pin the RELATIONSHIP rather than either set's contents, which is what survives a
+# deliberate widening: widen `_is_control_char` and the table follows automatically, and if it does
+# not, the first test goes red naming the code points that drifted.
+
+
+def test_the_log_escape_table_is_the_controlchars_alphabet_minus_tab() -> None:
+    """The whole of limb 3, as one assertion about the DIFFERENCE.
+
+    Not "the table has 32 entries" -- that pins a number and would have to be edited by whoever
+    widens the alphabet, which is precisely the person who should be told rather than asked to
+    update a constant. This pins the SUBTRACTION, so a legitimate widening passes untouched and a
+    divergence names its own code points.
+    """
+    from messagefoundry.controlchars import _is_control_char
+    from messagefoundry.logging_setup import _CTRL_TRANSLATION
+
+    alphabet = {cp for cp in range(0x80) if _is_control_char(chr(cp))}
+    escaped = set(_CTRL_TRANSLATION)
+
+    assert alphabet - escaped == {0x09}, (
+        f"the log escape table and controlchars have drifted: "
+        f"{sorted(hex(c) for c in (alphabet - escaped) - {0x09})} are screened as control "
+        f"characters but not escaped in a log line"
+    )
+    assert not escaped - alphabet, (
+        f"the log table escapes {sorted(hex(c) for c in escaped - alphabet)}, which controlchars "
+        f"does not treat as control characters -- one of the two has been widened alone"
+    )
+
+
+def test_tab_is_the_only_control_character_left_intact() -> None:
+    """Tab is benign whitespace in a log line; CR/LF are the injection vector and must not join it.
+
+    The asymmetry is the reason this is a separate test from the one above: that one would still
+    pass if tab were swapped for CR in the subtraction, because the difference would still be a
+    single code point.
+    """
+    from messagefoundry.logging_setup import _CTRL_TRANSLATION
+
+    assert 0x09 not in _CTRL_TRANSLATION, "tab must survive a log line unescaped"
+    assert _CTRL_TRANSLATION[0x0A] == "\\n", "LF is the injection vector and must be escaped"
+    assert _CTRL_TRANSLATION[0x0D] == "\\r", "CR is the injection vector and must be escaped"
+    assert _CTRL_TRANSLATION[0x00] == "\\x00"
+    assert _CTRL_TRANSLATION[0x7F] == "\\x7f", "DEL is in the alphabet and must still be escaped"
+
+
+def test_a_tab_survives_the_real_scrub_and_a_newline_does_not() -> None:
+    """Drives the shipped filter rather than the table, so the two cannot agree while the code differs."""
+    from messagefoundry.logging_setup import _CTRL_TRANSLATION
+
+    scrubbed = "before\tafter\nnext".translate(_CTRL_TRANSLATION)
+    assert "\t" in scrubbed, "the tab was escaped; a log line lost its benign whitespace"
+    assert "\n" not in scrubbed, "a real newline survived; one record can now forge a second line"
+    assert scrubbed == "before\tafter\\nnext"