From 0b12fc0d0704f7dce0ddec84d8180cb41732d464 Mon Sep 17 00:00:00 2001 From: wilfordgrimley <2397930+WilfordGrimley@users.noreply.github.com> Date: Wed, 29 Jul 2026 17:41:59 +0000 Subject: [PATCH 1/2] Derive the PROTECTED CORE roster from the policy doc; gate the decrypt tool MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/upstreaming/license-provenance.md §2 declared `decrypt-saved-deck-export/decrypt.mjs` and its test part of the vote/federation trust anchor, and instructed that both be added to check_protected_core_license.py's PROTECTED_CORE_FILES "in the PR that merges #242 (or immediately after)". #242 merged (5ddf109c), both files landed on master, and the CI list was never updated. Two files the policy calls a trust anchor have carried no gate at all since then. The one-line fix is to add them to the list. That is not what this does, because the defect is not the missing entry — it is that two hand-maintained lists were kept in sync by a convention written in prose. check_protected_core_license.py now holds NO list: it parses the marker-bounded roster region in §2 and derives its file set from the doc, which §2 already declared the source of truth. They cannot disagree because there is only one list. Adding the .mjs entries also exposed that neither existing rule could have fired on them: - the provenance-marker regex required a `#` comment leader, so a `// PROVENANCE: ..., AGPL-3.0` line in a JS file was invisible. It now accepts `#`, `//` and `*`. - the import walk was ast.parse()-only, which raises SyntaxError on .mjs and returned []. ES-module `import`/`export ... from` / dynamic `import()` / `require()` specifiers are now extracted, with only relative specifiers resolving (a bare specifier is a node: builtin or an npm package — out of scope for the same reason the Python side does not scan PyPI metadata). A roster check's characteristic failure is to check nothing and pass, so a missing/out-of-order marker, an empty region, a listed path that does not exist, and a roster entry in an unwalkable language are each HARD findings rather than silent skips. §2 also spells out the two "(+ its test)" prose parentheticals as explicit paths, since only backticked spans are machine-read, and replaces the stale "not yet in the CI script" bullet with a dated record of the gap it described. Roster derived: 11 files (was 9 hardcoded). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013NhYmT1PxCcyemA16dFDxN --- .../scripts/check_protected_core_license.py | 231 +++++++++++--- .../test_check_protected_core_license.py | 285 ++++++++++++++++-- docs/upstreaming/license-provenance.md | 84 ++++-- 3 files changed, 521 insertions(+), 79 deletions(-) diff --git a/.github/scripts/check_protected_core_license.py b/.github/scripts/check_protected_core_license.py index a84d20722..00d0d799c 100644 --- a/.github/scripts/check_protected_core_license.py +++ b/.github/scripts/check_protected_core_license.py @@ -2,22 +2,50 @@ """ PROTECTED CORE license lint, per docs/upstreaming/license-provenance.md §2. -Fails if any PROTECTED_CORE_FILES entry (a) carries an AGPL provenance -marker on itself, or (b) locally imports another in-repo module that -does. The real invariant this enforces is "no AGPL-derived code in -protected core" — NOT "everything here must be GPL-3.0": one entry -(federation-hash-tool/hash_my_cards.py) is deliberately MIT-licensed -(docs/federation/public-export-v1.md §5), and AGPL would poison either -license, not just GPL-3.0. - -Provenance marker convention (docs/upstreaming/license-provenance.md -§3's absorption protocol): a `# PROVENANCE: , , -` comment near the top of a vendored file. This lint only looks -for the substring "AGPL" in that line - it does not attempt to scan -transitive PyPI/npm dependency license metadata (a separate, much larger -problem; tools like `pip-licenses` exist for that). Nothing in this repo -is AGPL-marked as of this writing - this lint passes with zero findings -today, correctly, and exists to catch the day that stops being true. +Fails if any protected-core file (a) carries an AGPL provenance marker on +itself, (b) locally imports another in-repo module that does, or (c) is +listed in the policy but does not exist on disk. + +The real invariant this enforces is "no AGPL-derived code in protected +core" — NOT "everything here must be GPL-3.0": two entries +(federation-hash-tool/hash_my_cards.py and decrypt-saved-deck-export/ +decrypt.mjs, plus their tests) are deliberately MIT-licensed +(docs/federation/public-export-v1.md §5; PR #242), and AGPL would poison +either license, not just GPL-3.0. + +THE ROSTER IS DERIVED FROM THE POLICY DOC, NOT RESTATED HERE +------------------------------------------------------------ +This script holds no file list. It parses the marker-bounded region in +docs/upstreaming/license-provenance.md §2 and treats every backtick-quoted +repo path inside it as a protected-core file. + +That is a deliberate correction of a real, dated failure. §2 declared +`decrypt-saved-deck-export/decrypt.mjs` and its test part of the trust +anchor, and instructed that they be added to this script's hand-maintained +`PROTECTED_CORE_FILES` "in the PR that merges #242 (or immediately +after)". #242 merged (`5ddf109c`), both files landed on master, and the +list was never updated — so two files the policy calls a trust anchor +carried no gate at all until 2026-07-29. Two hand-maintained lists kept in +sync by a prose convention is the defect; adding a third entry to the +second list would only have deferred it. With the roster derived, the doc +and the check cannot disagree, because there is only one list. + +If the marker region is missing or yields no paths, that is a HARD +FINDING, not a quiet pass — a roster check that silently checks nothing is +worse than no check. + +PROVENANCE MARKER CONVENTION (docs/upstreaming/license-provenance.md §3's +absorption protocol): a `PROVENANCE: , , ` +comment near the top of a vendored file. The comment leader may be `#` +(Python/shell), `//` (JS) or `*` (inside a JS block comment) — the roster +spans two languages, and a regex that only recognised `#` would have let a +`// PROVENANCE: ..., AGPL-3.0` line in a `.mjs` roster file pass unseen. +This lint only looks for the substring "AGPL" in that line; it does not +attempt to scan transitive PyPI/npm dependency license metadata (a +separate, much larger problem; tools like `pip-licenses` exist for that). +Nothing in this repo is AGPL-marked as of this writing - this lint passes +with zero findings today, correctly, and exists to catch the day that +stops being true. Exit code is the number of findings (0 = clean), matching docs_lint.py's own convention. @@ -29,19 +57,21 @@ REPO_ROOT = Path(__file__).resolve().parents[2] -# The exact PROTECTED CORE file list - docs/upstreaming/license-provenance.md -# §2 is the source of truth; keep these in sync in the same PR. -PROTECTED_CORE_FILES = [ - "MPCAutofill/cardpicker/vote_consensus.py", - "MPCAutofill/cardpicker/printing_consensus.py", - "MPCAutofill/cardpicker/tag_consensus.py", - "MPCAutofill/cardpicker/artist_consensus.py", - "MPCAutofill/cardpicker/local_phash.py", - "MPCAutofill/cardpicker/local_fallback.py", - "federation-hash-tool/hash_my_cards.py", - "federation-hash-tool/tests/test_hash_my_cards.py", - "MPCAutofill/cardpicker/tests/test_federation_hash_tool_parity.py", -] +# docs/upstreaming/license-provenance.md §2 is the source of truth for the +# protected-core roster, and this script READS it (see the module docstring) +# rather than restating it. These three constants are the entire contract +# with the doc: the markers bound the region, and every backtick-quoted +# path inside it is a roster entry. +POLICY_DOC_REL = "docs/upstreaming/license-provenance.md" +ROSTER_BEGIN_MARKER = "" +ROSTER_END_MARKER = "" + +# A backtick span is a roster path if it contains "/" and ends in one of +# these. Prose inside the region routinely backticks non-paths (constant +# names, PR refs); requiring both a separator and a known source extension +# keeps those out without needing a second allowlist. +ROSTER_PATH_EXTENSIONS = (".py", ".mjs", ".cjs", ".js", ".ts", ".tsx") +ROSTER_PATH_RE = re.compile(r"`([\w./\-]+/[\w./\-]+)`") # Local import roots: a dotted import prefix maps to a directory that acts # as its own package root, mirroring how MPCAutofill/manage.py makes @@ -52,7 +82,29 @@ REPO_ROOT / "federation-hash-tool", ] -PROVENANCE_RE = re.compile(r"#\s*PROVENANCE:.*", re.IGNORECASE) +# Accepts `#`, `//` and `*` comment leaders — see the module docstring's +# marker-convention note for why the `#`-only form was a real hole. +PROVENANCE_RE = re.compile(r"(?:#|//|\*)\s*PROVENANCE:.*", re.IGNORECASE) + +PY_SUFFIXES = (".py",) +JS_SUFFIXES = (".mjs", ".cjs", ".js") + +# ES-module / CommonJS specifier extraction. Deliberately regex, not a JS +# parser: the roster's JS entries are by policy zero-dependency, single-file +# tools, and this repo's stated "narrow v1, no heavyweight library" +# philosophy (docs_lint.py's own limitation note) applies. The cost is that +# an exotic construct could be missed; the alternative is vendoring a JS +# parser into CI to check two files. +JS_IMPORT_RE = re.compile( + r"""(?: + \bimport\s+[^;'"]*?\bfrom\s*["']([^"']+)["'] # import x from "y" + | \bexport\s+[^;'"]*?\bfrom\s*["']([^"']+)["'] # export * from "y" + | \bimport\s*["']([^"']+)["'] # import "y" (side effect) + | \bimport\s*\(\s*["']([^"']+)["'] # dynamic import("y") + | \brequire\s*\(\s*["']([^"']+)["'] # require("y") + )""", + re.VERBOSE, +) def is_agpl_marked(text: str) -> bool: @@ -62,6 +114,54 @@ def is_agpl_marked(text: str) -> bool: return False +def protected_core_files() -> tuple[list[str], list[str]]: + """ + DERIVE the protected-core roster from the policy doc. Returns + (paths, findings) — findings is non-empty when the region itself is + broken, which must fail loudly rather than yield an empty roster. + + Order is preserved and duplicates collapsed, so a path that appears + twice in the prose is checked once. + """ + doc = REPO_ROOT / POLICY_DOC_REL + if not doc.is_file(): + return [], [ + f"::error::check_protected_core_license.py: policy doc {POLICY_DOC_REL} " + f"is missing — the protected-core roster is derived from its §2 marker " + f"region and cannot be built without it" + ] + + text = doc.read_text() + start = text.find(ROSTER_BEGIN_MARKER) + end = text.find(ROSTER_END_MARKER) + if start == -1 or end == -1 or end < start: + return [], [ + f"::error file={POLICY_DOC_REL}::protected-core roster markers " + f"{ROSTER_BEGIN_MARKER} / {ROSTER_END_MARKER} not found (or out of " + f"order) in §2. This script derives its file list from that region; " + f"without the markers it would check NOTHING and pass, so the missing " + f"markers are themselves the finding." + ] + + region = text[start + len(ROSTER_BEGIN_MARKER) : end] + paths: list[str] = [] + for m in ROSTER_PATH_RE.finditer(region): + candidate = m.group(1) + if not candidate.endswith(ROSTER_PATH_EXTENSIONS): + continue + if candidate not in paths: + paths.append(candidate) + + if not paths: + return [], [ + f"::error file={POLICY_DOC_REL}::protected-core roster region is " + f"present but contains no backtick-quoted source paths. An empty " + f"roster means this lint checks nothing; that is the finding." + ] + + return paths, [] + + def resolve_local_import(module: str) -> Path | None: """module is a dotted path like 'cardpicker.models' - resolve it against each import root, trying both '.py' and @@ -93,22 +193,77 @@ def local_imports(path: Path) -> list[str]: return modules +def js_imports(path: Path) -> list[str]: + """Every import/require specifier in a JS/ESM file, in source order.""" + specifiers = [] + for m in JS_IMPORT_RE.finditer(path.read_text()): + specifiers.append(next(g for g in m.groups() if g is not None)) + return specifiers + + +def resolve_js_import(specifier: str, importer: Path) -> Path | None: + """ + Resolve a JS specifier to an in-repo file, or None. + + Only RELATIVE specifiers resolve. A bare specifier is either a `node:` + builtin or an npm package — the same out-of-scope category as a PyPI + import on the Python side, and the roster's JS entries are by policy + dependency-free anyway. Tries the literal path first (ESM requires the + extension), then the extension-less CommonJS / index forms so a future + roster addition written in either style still resolves. + """ + if not specifier.startswith("."): + return None + base = (importer.parent / specifier).resolve() + candidates = [base] + for ext in JS_SUFFIXES: + candidates.append(base.with_name(base.name + ext)) + candidates.append(base / ("index" + ext)) + for candidate in candidates: + if candidate.is_file(): + try: + candidate.relative_to(REPO_ROOT) + except ValueError: + return None # escaped the repo — not an in-repo module + return candidate + return None + + def check_file(rel_path: str) -> list[str]: findings = [] path = REPO_ROOT / rel_path if not path.is_file(): - return [f"::error::check_protected_core_license.py: PROTECTED_CORE_FILES entry {rel_path!r} does not exist"] + return [ + f"::error file={POLICY_DOC_REL}::protected-core roster lists " + f"{rel_path!r}, which does not exist in the repo" + ] text = path.read_text() if is_agpl_marked(text): findings.append(f"::error file={rel_path}::PROTECTED CORE file itself carries an AGPL provenance marker") - for module in local_imports(path): - resolved = resolve_local_import(module) + if path.suffix in PY_SUFFIXES: + dependencies = [(module, resolve_local_import(module)) for module in local_imports(path)] + elif path.suffix in JS_SUFFIXES: + dependencies = [(spec, resolve_js_import(spec, path)) for spec in js_imports(path)] + else: + # A roster entry in a language this lint cannot walk is a hard + # finding, not a silent skip: it would otherwise be an entry that + # LOOKS gated and is not — the exact failure this script was + # rewritten to eliminate. + findings.append( + f"::error file={rel_path}::protected-core roster entry has unsupported " + f"suffix {path.suffix!r} — this lint can only walk imports for " + f"{list(PY_SUFFIXES + JS_SUFFIXES)}. Extend " + f"check_protected_core_license.py rather than leaving the entry " + f"half-checked." + ) + return findings + + for module, resolved in dependencies: if resolved is None: continue - imported_text = resolved.read_text() - if is_agpl_marked(imported_text): + if is_agpl_marked(resolved.read_text()): findings.append( f"::error file={rel_path}::imports {module!r} " f"({resolved.relative_to(REPO_ROOT)}), which carries an AGPL provenance marker" @@ -118,8 +273,8 @@ def check_file(rel_path: str) -> list[str]: def main() -> int: - all_findings = [] - for rel_path in PROTECTED_CORE_FILES: + roster, all_findings = protected_core_files() + for rel_path in roster: all_findings.extend(check_file(rel_path)) for finding in all_findings: @@ -128,7 +283,7 @@ def main() -> int: if all_findings: print(f"\n{len(all_findings)} PROTECTED CORE license violation(s) found.") else: - print(f"protected-core-license: clean ({len(PROTECTED_CORE_FILES)} files checked).") + print(f"protected-core-license: clean ({len(roster)} files checked, derived from {POLICY_DOC_REL} §2).") return len(all_findings) diff --git a/.github/scripts/tests/test_check_protected_core_license.py b/.github/scripts/tests/test_check_protected_core_license.py index 6dcd8a66f..b4bb89ee0 100644 --- a/.github/scripts/tests/test_check_protected_core_license.py +++ b/.github/scripts/tests/test_check_protected_core_license.py @@ -2,11 +2,25 @@ Unit tests for check_protected_core_license.py, per docs/upstreaming/license-provenance.md §2. Proves the lint actually catches a violation (not just "passes with zero findings against the -real repo, trust us") via a real fixture case in a scratch directory, -and separately confirms the real repo's own PROTECTED_CORE_FILES list is -clean today - the property docs-lint.yml's protected-core-license job +real repo, trust us") via real fixture cases in a scratch directory, +and separately confirms the real repo's own derived roster is clean +today - the property docs-lint.yml's protected-core-license job enforces on every PR. +Covers three classes of failure, each of which has actually happened or +was structurally possible before 2026-07-29: + 1. an AGPL marker on a protected-core file or one it imports (the + original purpose); + 2. the DERIVED ROSTER going wrong - missing markers, an empty region, a + path that does not exist, an unwalkable file type. Each must be a + hard finding, because the failure mode of a roster check is to check + nothing and pass; + 3. JS/ESM entries being invisible - before this rewrite the marker + regex required a `#` leader and the import walk was Python-only, so + `decrypt-saved-deck-export/decrypt.mjs` (a policy-declared trust + anchor) would have been "checked" without either rule being able to + fire on it. + Run: python3 .github/scripts/tests/test_check_protected_core_license.py """ @@ -22,6 +36,24 @@ import check_protected_core_license as lint # noqa: E402 +class _patched_roots: + """Context manager: temporarily repoint REPO_ROOT/IMPORT_ROOTS at a + scratch fixture dir, restoring the real repo afterward.""" + + def __init__(self, fixture_root: Path) -> None: + self.fixture_root = fixture_root + + def __enter__(self) -> None: + self._real_repo_root = lint.REPO_ROOT + self._real_import_roots = lint.IMPORT_ROOTS + lint.REPO_ROOT = self.fixture_root + lint.IMPORT_ROOTS = [self.fixture_root / "MPCAutofill"] + + def __exit__(self, *exc: object) -> None: + lint.REPO_ROOT = self._real_repo_root + lint.IMPORT_ROOTS = self._real_import_roots + + class TestAgplMarkerDetection(unittest.TestCase): def test_detects_agpl_provenance_marker(self) -> None: self.assertTrue(lint.is_agpl_marked("# PROVENANCE: some/repo, v1.2.3, AGPL-3.0\n")) @@ -35,6 +67,62 @@ def test_mit_marker_is_not_agpl(self) -> None: def test_no_marker_at_all(self) -> None: self.assertFalse(lint.is_agpl_marked("import os\nimport sys\n")) + # --- JS comment leaders. The `#`-only regex could not see any of these, + # which left every .mjs roster entry ungated on the self-marker rule. + def test_detects_agpl_marker_behind_double_slash(self) -> None: + self.assertTrue(lint.is_agpl_marked('// PROVENANCE: some/repo, v1, AGPL-3.0\nimport x from "y";\n')) + + def test_detects_agpl_marker_inside_block_comment(self) -> None: + self.assertTrue(lint.is_agpl_marked("/**\n * PROVENANCE: some/repo, v1, AGPL-3.0\n */\n")) + + def test_mit_marker_behind_double_slash_is_not_agpl(self) -> None: + self.assertFalse(lint.is_agpl_marked("// PROVENANCE: some/repo, v1, MIT\n")) + + +class TestJsImportExtraction(unittest.TestCase): + def test_extracts_every_specifier_form(self) -> None: + src = ( + 'import { a } from "node:crypto";\n' + "import b from './b.mjs';\n" + 'import "./side-effect.mjs";\n' + 'export { c } from "../c.mjs";\n' + 'const d = await import("./d.mjs");\n' + 'const e = require("./e.cjs");\n' + ) + with tempfile.TemporaryDirectory() as tmp: + f = Path(tmp) / "x.mjs" + f.write_text(src) + self.assertEqual( + lint.js_imports(f), + ["node:crypto", "./b.mjs", "./side-effect.mjs", "../c.mjs", "./d.mjs", "./e.cjs"], + ) + + def test_bare_specifier_does_not_resolve(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + f = Path(tmp) / "x.mjs" + f.write_text("") + self.assertIsNone(lint.resolve_js_import("node:crypto", f)) + self.assertIsNone(lint.resolve_js_import("some-npm-package", f)) + + def test_relative_specifier_resolves_to_sibling(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp).resolve() + (root / "dep.mjs").write_text("export const x = 1;\n") + (root / "tests").mkdir() + importer = root / "tests" / "t.mjs" + importer.write_text('import { x } from "../dep.mjs";\n') + with _patched_roots(root): + self.assertEqual(lint.resolve_js_import("../dep.mjs", importer), root / "dep.mjs") + + def test_extensionless_relative_specifier_resolves(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp).resolve() + (root / "dep.mjs").write_text("export const x = 1;\n") + importer = root / "t.mjs" + importer.write_text("") + with _patched_roots(root): + self.assertEqual(lint.resolve_js_import("./dep", importer), root / "dep.mjs") + class TestCheckFileAgainstFixtures(unittest.TestCase): def _fixture_repo(self, tmp: str, clean_module_body: str, protected_body: str) -> None: @@ -85,36 +173,193 @@ def test_missing_protected_core_file_is_a_finding(self) -> None: self.assertEqual(len(findings), 1) self.assertIn("does not exist", findings[0]) + def test_unwalkable_roster_entry_is_a_finding_not_a_silent_pass(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + (root / "tool").mkdir() + (root / "tool" / "thing.rs").write_text("fn main() {}\n") + with _patched_roots(root): + findings = lint.check_file("tool/thing.rs") + self.assertEqual(len(findings), 1) + self.assertIn("unsupported suffix", findings[0]) -class _patched_roots: - """Context manager: temporarily repoint REPO_ROOT/IMPORT_ROOTS at a - scratch fixture dir, restoring the real repo afterward.""" - def __init__(self, fixture_root: Path) -> None: - self.fixture_root = fixture_root +class TestCheckFileJsFixtures(unittest.TestCase): + """The .mjs half of the roster, which had no working rule at all before.""" - def __enter__(self) -> None: - self._real_repo_root = lint.REPO_ROOT - self._real_import_roots = lint.IMPORT_ROOTS - lint.REPO_ROOT = self.fixture_root - lint.IMPORT_ROOTS = [self.fixture_root / "MPCAutofill"] + def _js_fixture(self, tmp: str, dep_body: str, tool_body: str) -> Path: + root = Path(tmp).resolve() + (root / "tool" / "tests").mkdir(parents=True) + (root / "tool" / "dep.mjs").write_text(dep_body) + (root / "tool" / "tests" / "tool.test.mjs").write_text(tool_body) + return root - def __exit__(self, *exc: object) -> None: - lint.REPO_ROOT = self._real_repo_root - lint.IMPORT_ROOTS = self._real_import_roots + def test_clean_js_file_passes(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = self._js_fixture( + tmp, + dep_body="// PROVENANCE: some/repo, v1, MIT\nexport const x = 1;\n", + tool_body='import { x } from "../dep.mjs";\n', + ) + with _patched_roots(root): + self.assertEqual(lint.check_file("tool/tests/tool.test.mjs"), []) + + def test_js_file_importing_agpl_marked_local_module_fails(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = self._js_fixture( + tmp, + dep_body="// PROVENANCE: some/repo, v1, AGPL-3.0\nexport const x = 1;\n", + tool_body='import { x } from "../dep.mjs";\n', + ) + with _patched_roots(root): + findings = lint.check_file("tool/tests/tool.test.mjs") + self.assertEqual(len(findings), 1) + self.assertIn("AGPL", findings[0]) + self.assertIn("tool/dep.mjs", findings[0]) + + def test_js_file_self_marked_agpl_in_block_comment_fails(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = self._js_fixture( + tmp, + dep_body="export const x = 1;\n", + tool_body='/**\n * PROVENANCE: some/repo, v1, AGPL-3.0\n */\nimport { x } from "../dep.mjs";\n', + ) + with _patched_roots(root): + findings = lint.check_file("tool/tests/tool.test.mjs") + self.assertEqual(len(findings), 1) + self.assertIn("itself carries", findings[0]) + + def test_bare_npm_specifier_is_not_walked(self) -> None: + """A bare specifier is out of scope by design (module docstring) — + asserted explicitly so a future 'helpful' resolver change that + starts walking node_modules trips this test.""" + with tempfile.TemporaryDirectory() as tmp: + root = self._js_fixture( + tmp, + dep_body="export const x = 1;\n", + tool_body='import { webcrypto } from "node:crypto";\n', + ) + with _patched_roots(root): + self.assertEqual(lint.check_file("tool/tests/tool.test.mjs"), []) + + +class TestRosterDerivation(unittest.TestCase): + """The roster comes from the doc. These prove the derivation is real — + that it reads what the doc says, and fails loudly rather than quietly + checking nothing when the doc's region is broken.""" + + def _doc(self, tmp: str, body: str) -> Path: + root = Path(tmp) + (root / lint.POLICY_DOC_REL).parent.mkdir(parents=True) + (root / lint.POLICY_DOC_REL).write_text(body) + return root + + def test_derives_paths_from_marker_region(self) -> None: + body = ( + "## 2. Protected core\n\n" + "prose mentioning `MPCAutofill/cardpicker/models.py` OUTSIDE the region\n\n" + f"{lint.ROSTER_BEGIN_MARKER}\n\n" + "- `a/one.py`\n" + "- `b/two.mjs` (+ its test, `b/tests/two.test.mjs`)\n\n" + f"{lint.ROSTER_END_MARKER}\n\n" + "trailing prose with `c/three.py` which must NOT be picked up\n" + ) + with tempfile.TemporaryDirectory() as tmp: + root = self._doc(tmp, body) + with _patched_roots(root): + paths, findings = lint.protected_core_files() + self.assertEqual(findings, []) + self.assertEqual(paths, ["a/one.py", "b/two.mjs", "b/tests/two.test.mjs"]) + + def test_non_path_backticks_in_region_are_ignored(self) -> None: + body = ( + f"{lint.ROSTER_BEGIN_MARKER}\n" + "- `a/one.py` — see `PROTECTED_CORE_FILES` and PR `#242`, dir `a/b/`\n" + f"{lint.ROSTER_END_MARKER}\n" + ) + with tempfile.TemporaryDirectory() as tmp: + root = self._doc(tmp, body) + with _patched_roots(root): + paths, findings = lint.protected_core_files() + self.assertEqual(findings, []) + self.assertEqual(paths, ["a/one.py"]) + + def test_duplicate_path_is_collapsed(self) -> None: + body = f"{lint.ROSTER_BEGIN_MARKER}\n- `a/one.py`\n- `a/one.py`\n{lint.ROSTER_END_MARKER}\n" + with tempfile.TemporaryDirectory() as tmp: + root = self._doc(tmp, body) + with _patched_roots(root): + paths, _ = lint.protected_core_files() + self.assertEqual(paths, ["a/one.py"]) + + def test_missing_markers_is_a_hard_finding(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = self._doc(tmp, "## 2. Protected core\n\n- `a/one.py`\n") + with _patched_roots(root): + paths, findings = lint.protected_core_files() + self.assertEqual(paths, []) + self.assertEqual(len(findings), 1) + self.assertIn("markers", findings[0]) + + def test_markers_out_of_order_is_a_hard_finding(self) -> None: + body = f"{lint.ROSTER_END_MARKER}\n- `a/one.py`\n{lint.ROSTER_BEGIN_MARKER}\n" + with tempfile.TemporaryDirectory() as tmp: + root = self._doc(tmp, body) + with _patched_roots(root): + paths, findings = lint.protected_core_files() + self.assertEqual(paths, []) + self.assertEqual(len(findings), 1) + + def test_empty_region_is_a_hard_finding(self) -> None: + body = f"{lint.ROSTER_BEGIN_MARKER}\n\n(nothing here yet)\n\n{lint.ROSTER_END_MARKER}\n" + with tempfile.TemporaryDirectory() as tmp: + root = self._doc(tmp, body) + with _patched_roots(root): + paths, findings = lint.protected_core_files() + self.assertEqual(paths, []) + self.assertEqual(len(findings), 1) + self.assertIn("checks nothing", findings[0]) + + def test_missing_policy_doc_is_a_hard_finding(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + with _patched_roots(Path(tmp)): + paths, findings = lint.protected_core_files() + self.assertEqual(paths, []) + self.assertEqual(len(findings), 1) + self.assertIn("policy doc", findings[0]) class TestRealRepoIsClean(unittest.TestCase): + def test_real_roster_derives_without_findings(self) -> None: + paths, findings = lint.protected_core_files() + self.assertEqual(findings, [], f"roster derivation failed: {findings}") + self.assertTrue(paths) + def test_real_protected_core_files_are_clean(self) -> None: - all_findings = [] - for rel_path in lint.PROTECTED_CORE_FILES: + paths, all_findings = lint.protected_core_files() + for rel_path in paths: all_findings.extend(lint.check_file(rel_path)) - self.assertEqual(all_findings, [], f"real repo PROTECTED_CORE_FILES has findings: {all_findings}") + self.assertEqual(all_findings, [], f"real repo protected core has findings: {all_findings}") def test_every_protected_core_file_exists(self) -> None: - for rel_path in lint.PROTECTED_CORE_FILES: + paths, _ = lint.protected_core_files() + for rel_path in paths: self.assertTrue((REPO_ROOT / rel_path).is_file(), f"{rel_path} does not exist") + def test_decrypt_tool_is_on_the_real_roster(self) -> None: + """The specific regression this rewrite closed: both decrypt-tool + paths are declared protected core by the policy and were absent + from the CI list for the entire life of PR #242 on master.""" + paths, _ = lint.protected_core_files() + self.assertIn("decrypt-saved-deck-export/decrypt.mjs", paths) + self.assertIn("decrypt-saved-deck-export/tests/decrypt.test.mjs", paths) + + def test_federation_hash_tool_test_is_on_the_real_roster(self) -> None: + """`(+ its test)` used to be a prose parenthetical the machine could + not read; the doc now spells the path out.""" + paths, _ = lint.protected_core_files() + self.assertIn("federation-hash-tool/tests/test_hash_my_cards.py", paths) + if __name__ == "__main__": unittest.main() diff --git a/docs/upstreaming/license-provenance.md b/docs/upstreaming/license-provenance.md index bfa7d8d28..0059743ce 100644 --- a/docs/upstreaming/license-provenance.md +++ b/docs/upstreaming/license-provenance.md @@ -150,7 +150,20 @@ action needed. ## 2. Protected core **Scope — the actual files, not just the concept**, since a policy -nobody can point at doesn't function as one: +nobody can point at doesn't function as one. + +**This list is MACHINE-READ, not just prose.** The markers below bound the +roster, and `.github/scripts/check_protected_core_license.py` PARSES this +region at runtime to build its file list — it holds no list of its own. +There is therefore no second copy to drift: adding a bullet here adds a CI +gate, and the CI script cannot silently disagree with this section because +it has nothing to disagree with. Every backtick-quoted path inside the +markers must resolve to a real file (a typo fails the lint), so paths that +were previously written as a prose parenthetical — `(+ its test)` — are now +spelled out explicitly. Keep prose commentary outside the backticks; only +the backticked spans are read. + + - `MPCAutofill/cardpicker/vote_consensus.py` - `MPCAutofill/cardpicker/printing_consensus.py` @@ -162,24 +175,42 @@ nobody can point at doesn't function as one: - `MPCAutofill/cardpicker/local_fallback.py` — **still protected**; carries one authorised exception, 2026-07-29, logged in §2.1. The exception covers that one change, not the file. -- `federation-hash-tool/hash_my_cards.py` (+ its test) +- `federation-hash-tool/hash_my_cards.py` (+ its test, + `federation-hash-tool/tests/test_hash_my_cards.py`) - `MPCAutofill/cardpicker/tests/test_federation_hash_tool_parity.py` (the parity tether between the previous two) -- `decrypt-saved-deck-export/decrypt.mjs` (+ its test) — the standalone, +- `decrypt-saved-deck-export/decrypt.mjs` (+ its test, + `decrypt-saved-deck-export/tests/decrypt.test.mjs`) — the standalone, zero-import, zero-dependency decrypt tool for a saved-decks export bundle (PR #242); same standalone-trust-anchor risk shape as the federation hash tool above, not itself part of the vote/federation - system. **Not yet in `check_protected_core_license.py`'s - `PROTECTED_CORE_FILES`** — that file only exists on PR #242's branch, - not yet on `master`; add both paths to the CI script's list in the PR - that merges #242 (or immediately after), per this section's own "keep - these in sync in the same PR" convention. -- **Prospectively**: any future verdict schema/signing/export/import/ - keygen module (`federation-v1.md`/`federation/public-export-v1.md` - describe the format; per those docs, "format committed ahead of - code" — none of that code exists yet, so there's nothing to list here - today beyond the commitment that whatever gets built there joins this - list in the same PR). + system. + + + +**Prospectively** (deliberately OUTSIDE the machine-read region — there is +nothing to gate yet, and a marker region containing an unresolvable path +would fail the lint): any future verdict schema/signing/export/import/ +keygen module (`federation-v1.md`/`federation/public-export-v1.md` +describe the format; per those docs, "format committed ahead of code" — +none of that code exists yet, so there's nothing to list here today beyond +the commitment that whatever gets built there joins the roster above in +the same PR). + +**Historical note, kept because the gap it describes was real and lasted**: +this section used to carry a bullet saying the decrypt-tool paths were +"**Not yet in `check_protected_core_license.py`'s `PROTECTED_CORE_FILES`** +— that file only exists on PR #242's branch, not yet on `master`; add both +paths to the CI script's list in the PR that merges #242 (or immediately +after), per this section's own 'keep these in sync in the same PR' +convention." PR #242 merged as `5ddf109c`; both files landed on `master`; +**the CI list was never updated.** Two files this section declares part of +the trust anchor therefore carried NO gate at all from that merge until +2026-07-29. The one-line fix would have been to add them to the script's +list. What actually shipped instead is the derivation above, because "two +hand-maintained lists, kept in sync by a convention written in prose" is +the defect, and adding an entry to the second list would have left the +defect in place for the next entry. **Explicitly NOT file-level protected here, despite being conceptually part of the vote/consensus system**: @@ -218,13 +249,24 @@ genuinely MIT-permissive). The CI check below enforces the real invariant, not the narrower one the directive stated. **The CI check — built, not just designed**: a new -`.github/scripts/check_protected_core_license.py` walks each -protected-core file's local (intra-repo) imports and fails if any -imported local module carries an `AGPL` mention in a `# PROVENANCE:` -header comment (the format §3's absorption protocol requires of any -future external-code intake) — also fails if a protected-core file -carries that marker on itself directly. Wired into `docs-lint.yml` as a -new `protected-core-license` job. **Passes today with zero findings**, +`.github/scripts/check_protected_core_license.py` reads the roster region +above, then walks each protected-core file's local (intra-repo) imports +and fails if any imported local module carries an `AGPL` mention in a +`PROVENANCE:` header comment (the format §3's absorption protocol requires +of any future external-code intake) — also fails if a protected-core file +carries that marker on itself directly, and fails if a path listed above +does not exist. **Both languages on the roster are handled**: Python files +via `ast`, resolving dotted imports against `MPCAutofill/` and +`federation-hash-tool/` as package roots; `.mjs` files via ES-module +`import`/`export ... from` / dynamic `import()` / `require()` extraction, +resolving only RELATIVE specifiers (`./`, `../`) — a bare specifier is an +npm package or a `node:` builtin, out of scope for the same reason the +Python side does not scan PyPI metadata. The comment-marker match accepts +`#`, `//` and `*` comment leaders so a marker in a `.mjs` file is seen; +before 2026-07-29 the regex required `#`, which meant a JS file on the +roster could have carried an AGPL marker in a `// PROVENANCE:` line and +passed. Wired into `docs-lint.yml` as a new `protected-core-license` job. +**Passes today with zero findings**, correctly — nothing in this repo is AGPL-marked; the check's only job is to trip the day that stops being true. Deliberately does NOT attempt to scan transitive PyPI/npm dependency license metadata (a much larger, From b8a05e0fd5e9020257752c47f6355d7f113b4268 Mon Sep 17 00:00:00 2001 From: wilfordgrimley <2397930+WilfordGrimley@users.noreply.github.com> Date: Wed, 29 Jul 2026 18:03:58 +0000 Subject: [PATCH 2/2] docs-lint: trigger the protected-core job on decrypt-tool changes The roster is derived from license-provenance.md (covered by `docs/**`), but the FILES it gates were not all trigger paths. `decrypt-saved-deck-export/**` had none: a PR pasting AGPL-marked code into the decrypt tool touches no doc and no cardpicker module, so the protected-core lint would not have run on it until the weekly cron. `federation-hash-tool/**` was already listed; this is its counterpart. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013NhYmT1PxCcyemA16dFDxN --- .github/workflows/docs-lint.yml | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.github/workflows/docs-lint.yml b/.github/workflows/docs-lint.yml index f4b35a0f8..446349a17 100644 --- a/.github/workflows/docs-lint.yml +++ b/.github/workflows/docs-lint.yml @@ -28,6 +28,14 @@ on: - ".github/workflows/docs-lint.yml" - "readme.md" - "NOTICE" + # PROTECTED CORE roster (check_protected_core_license.py): the roster + # is DERIVED from license-provenance.md's marker region (covered by + # `docs/**` above), but the FILES it gates must trigger too — a PR + # that pastes AGPL-marked code into the decrypt tool touches no doc + # and no cardpicker module, and would otherwise miss this lint + # entirely until the weekly cron. `federation-hash-tool/**` below is + # the same entry for the other standalone trust-anchor tool. + - "decrypt-saved-deck-export/**" - "MPCAutofill/cardpicker/vote_consensus.py" - "MPCAutofill/cardpicker/printing_consensus.py" - "MPCAutofill/cardpicker/tag_consensus.py"