From 62c406edd0a9a9d17899469df4bf2118a47de780 Mon Sep 17 00:00:00 2001 From: Chelsea Kelly-Reif <3114598+ChelseaKR@users.noreply.github.com> Date: Thu, 27 Aug 2026 20:48:44 -0700 Subject: [PATCH] docs: the manifest entry this table said did not exist The Standards Conformance table opened by saying the standards program's applicability manifest had no entry for this repository, so nothing had ever decided which of the fifteen standards bind it, and that the scoping below was this table's reading rather than the registry's. The entry was added on 2026-08-15, in STANDARDS commit 36747be, "fix(applicability): correct the manifest, and register the repo that owns it". The sentence was true when it was written and false the same week. Nothing here noticed for twelve days, and the reason is the shape worth recording: def test_the_table_says_the_applicability_manifest_has_no_entry_for_this_repo(): assert "applicability manifest has no entry" in section with the instruction "Remove this test on the day the manifest entry exists, not before". That test cannot fail while the sentence is present and cannot notice when the sentence stops being true, because the only thing it reads is the sentence. It was a green check over a false claim about this repository's own governance, in the section whose whole job is to state what is true today. The table now transcribes what the registry records: archetype civic-data-tool, publication cleared and public since 2026-08-08, tier B+C, flags html and hosted true, fourteen standards applying and AI Evaluation N/A for "deterministic coverage counting over published CAL FIRE/FRAP datasets; no LLM/model component". It carries the date it was read and says that the registry lives in another repository, so nothing in CI can check the transcription against its source. Same discipline docs/MARKERS.md applies to CAL FIRE's documents: a source that cannot be re-read at build time is transcribed with a date, and the date is part of the claim. One row moved to match the registry. Observability was recorded as Tier C and the registry says B+C, which is also what OBS section 0 asks a repository with two surfaces to state. The B half is a real gap: section 8 wants a Core Web Vitals RUM beacon, which means shipping a script that reports readers of a civic-data page back to somebody, and these pages ship no script and this project takes no telemetry. That refusal is a position rather than an oversight, and the standard has no N/A for it, so it is recorded as not met and says why. The Lighthouse-CI lab gate needs no beacon, is also not met, and is the half that could be built. The replacement tests refuse the specific false claim, hold the transcription's shape and its date, and check the one thing that is mechanically checkable from inside this repository: that the table marks exactly the one standard the registry marks N/A. Measured on this tree. Against the pre-change README, seven of the new tests fail, including: FAILED test_the_table_does_not_claim_the_manifest_has_no_entry FAILED test_the_table_transcribes_the_manifest_entry[civic-data-tool] FAILED test_the_transcription_says_it_cannot_be_checked_from_here 7 failed, 50 passed and 31 positive controls in the same file pass in both states. make verify exits 0: 581 tests, 100% branch coverage, determinism byte-identical. Part of #15. The other half of that issue, applying .github/rulesets/main.json and enabling Dependabot alerts, is a live repository setting and is not something a pull request can do. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 25 +++++++++ README.md | 28 ++++++++--- tests/test_standards_conformance.py | 78 ++++++++++++++++++++++++++--- 3 files changed, 119 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ecdd3bb..92f8975 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,31 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this project a ## [Unreleased] +### Fixed, a caveat that was true when written, false a week later, and pinned by a test + +- **The Standards Conformance table said the applicability manifest had no entry for + this repository. It has had one since 2026-08-15.** The entry records the archetype as + `civic-data-tool`, publication as cleared, the tier as `B+C`, and fourteen of the + fifteen standards as applying with AI Evaluation N/A. The table now transcribes that, + with the date it was read, and says plainly that the registry lives in another + repository so nothing in CI can check the transcription against its source. +- **The test guarding that sentence could only ever pass.** It read + `assert "applicability manifest has no entry" in section` and carried the instruction + "Remove this test on the day the manifest entry exists, not before". It was green for + twelve days over a claim that had become false, because the only thing reading the + sentence was a test asserting the sentence was still there. It is replaced by tests + that refuse the false claim, hold the transcription's shape and date, and check the + one part that is mechanically checkable from inside this repository: that the table + marks exactly the one standard the registry marks N/A. +- **Observability moves from Tier C to Tier B+C**, which is what the registry records and + what OBS section 0 asks a repository with two surfaces to state. The B half is a real + gap and is recorded as one: section 8 wants a Core Web Vitals RUM beacon, which means + shipping a script that reports readers of a civic-data page back to somebody, and + these pages ship no script and this project takes no telemetry. That refusal is a + position rather than an oversight and the standard has no N/A for it. The + Lighthouse-CI lab gate, which needs no beacon, is also not met and is the half that + could be built. + ### Fixed, the three pages could not say which of them a search result was for - **One meta description was shared by all three pages, and none of them had a canonical.** diff --git a/README.md b/README.md index ec5044a..680ed38 100644 --- a/README.md +++ b/README.md @@ -235,21 +235,37 @@ here is how much of each published field is actually filled in, and what the bla ## Standards conformance This repository is held to the portfolio's shared engineering standards, pinned in -`.standards-version` to `v2.0.0`. Every row states what is true on 2026-08-15, not what is +`.standards-version` to `v2.0.0`. Every row states what is true on 2026-08-27, not what is intended. "Applies (not met)" is a recorded gap; a blank state is a defect and `tests/test_standards_conformance.py` fails on one. -Read one caveat first. The standards program's applicability manifest has no entry for -this repository, so nothing has ever decided which of the fifteen standards bind it. The -scoping below was derived here, from each standard's own applicability section, and it is -this table's reading rather than the registry's. A manifest entry supersedes it. +**Which standards bind this repository is the registry's decision, and the registry has +made it.** The applicability manifest carries an entry for `perimeter`, added +2026-08-15. Read 2026-08-27, it records the archetype as `civic-data-tool`, publication +as cleared and public since 2026-08-08, the tier as `B+C` (the published Pages site is +the B surface, the local Python and Node build is the C one), and the flags as +`html: true`, `hosted: true`, `dockerfile: false`, `llm: false`, `bilingual: false`. Of +the fifteen standards it marks fourteen as applying and AI Evaluation as N/A, for +"deterministic coverage counting over published CAL FIRE/FRAP datasets; no LLM/model +component". + +This table used to open by saying the manifest had no entry and that the scoping below +was this table's reading rather than the registry's. That was true when it was written +and stopped being true the same week, and nothing here noticed, because the only test +reading the sentence was one asserting it was still present. The scoping now matches the +registry on all fifteen rows. One row moved to get there: Observability was recorded as +Tier C, and the registry says `B+C`. + +The registry lives in another repository, so this is a transcription with a date, the +same way `docs/MARKERS.md` transcribes CAL FIRE's documents rather than reading them at +build time. Nothing in CI can check it against the source. | Standard | State | |---|---| | Code Quality | Applies: uv, ruff, mypy `--strict`, pytest with branch coverage at 100% over `src/` with nothing omitted, against a 90% floor. `make lock-check` runs `uv lock --check`, because CQ-09's prescribed `uv sync --frozen` exits 0 on lockfile drift (measured 2026-08-15) | | Security & Supply-Chain | Applies: semgrep, gitleaks, pip-audit, `npm audit`, CodeQL over actions/python/javascript, every action SHA-pinned, `permissions: contents: read` at the top of every workflow, `persist-credentials: false` on every checkout. Not met: no SBOM, no OpenSSF Scorecard workflow, no `osv-scanner` alongside pip-audit (SEC-11, SEC-13), no scheduled trufflehog run (SEC-19), and Dependabot alerts are disabled on the repository, so SEC-15 has nothing to read | | CI/CD | Applies (not met). `main` has no ruleset and no branch protection, so the gates report and block nothing. The `protect-main` profile is committed at `.github/rulesets/main.json` and deliberately not applied; applying it is a live repository setting | -| Observability | Applies (Tier C). A library and a CLI writing to stdout, plus static pages with no script. No hosted service, no telemetry, no SLO surface. Not met: no operations runbook | +| Observability | Applies (Tier B+C), which is what the manifest records and what OBS section 0 asks a repo with two surfaces to state. Tier C is the library and CLI writing to stdout: OTel is out of scope with no network surface, which section 10 allows a Tier C surface to declare. Tier B is the published Pages site. Not met, and the B half is the larger gap: section 8 asks a Tier B frontend for a Core Web Vitals RUM beacon, which means shipping a script that reports readers of a civic-data page back to somebody, and these pages ship no script and this project takes no telemetry; that refusal is a position, not an oversight, and the standard has no N/A for it. Also not met: the Lighthouse-CI lab gate on LCP, INP and CLS, which does not need a beacon and is the half that could be built; no `docs/ROADMAP.md` carrying the tier declaration section 0 requires; and no operations runbook | | Accessibility | Applies: html-validate and axe-core over the built pages in CI, in jsdom and again in Chromium, plus SC 1.4.10 Reflow at 320 by 256, contrast measured arithmetically over both palettes, and the same structural floor asserted from Python. An axe rule that comes back undecided fails the jsdom gate unless it is declared with a reason and with where it is checked instead; the browser run declares nothing and fails on any undecided rule. Nothing is suppressed anywhere, so there is no `waivers.yml` and A11Y-06 and A11Y-09 are met rather than waived. Not met: no ACR, and the manual checks README names under "What still needs a person" are unverified. Measured 2026-08-27: the browser run found `scrollable-region-focusable`, serious, on both measurement pages, which the jsdom run cannot see; the scroll containers are now named sections with `tabindex="0"` | | Internationalization | Applies (not met). Civic data presented to the public is in scope per I18N section 1, and these pages are English only with no catalog and no `docs/I18N.md` declaration | | AI Evaluation | N/A (no model, no LLM, no generated text anywhere in the pipeline or the pages) | diff --git a/tests/test_standards_conformance.py b/tests/test_standards_conformance.py index 9929aa0..edc21ad 100644 --- a/tests/test_standards_conformance.py +++ b/tests/test_standards_conformance.py @@ -139,11 +139,77 @@ def test_the_table_carries_no_standard_this_portfolio_does_not_have() -> None: assert not unknown, f"unrecognised standards in the table: {sorted(unknown)}" -def test_the_table_says_the_applicability_manifest_has_no_entry_for_this_repo() -> None: - """FIX-02. Remove this test on the day the manifest entry exists, not before. +# --- FIX-02: the manifest entry, and what happened to the test that pinned its absence - - The scoping above was derived from each standard's own applicability section rather - than read from the registry, and a reader is owed that caveat while it is true. + +def conformance_section() -> str: + return README.split("## Standards conformance", 1)[1].split("\n## ", 1)[0] + + +def test_the_table_does_not_claim_the_manifest_has_no_entry() -> None: + """The entry exists. This asserted the opposite for twelve days. + + The test that stood here read ``assert "applicability manifest has no entry" in + section`` and carried the instruction "Remove this test on the day the manifest + entry exists, not before". The entry was added to the registry on 2026-08-15, the + same week the sentence was written. Nothing here noticed, because the only thing + reading the sentence was a test asserting it was still present: green, permanently, + over a claim that had become false. A test that can only fail when somebody edits + the sentence it is pinning is not a check on the sentence. + + This is the direction that can be checked from inside this repository. The registry + lives in another one, so the table's transcription of it is dated prose and no test + can compare it to the source. What a test can do is refuse the specific false claim + that was made, and hold the transcription's shape. + """ + section = conformance_section() + assert "applicability manifest has no entry" not in section, ( + "the manifest carries an entry for this repository; the table must not say " + "otherwise (FIX-02)" + ) + assert "manifest carries an entry" in section + + +@pytest.mark.parametrize( + "fact", + [ + "civic-data-tool", + "`B+C`", + "cleared", + "added 2026-08-15", + "Read 2026-08-27", + ], +) +def test_the_table_transcribes_the_manifest_entry(fact: str) -> None: + """What the registry records, in the table, with the date it was read. + + Same discipline as ``docs/MARKERS.md``: a document this project cannot re-read at + build time is transcribed with a date, and the date is part of the claim. Matched + against the section with its line breaks flattened, so rewrapping the paragraph + cannot break a check about its content. """ - section = README.split("## Standards conformance", 1)[1].split("\n## ", 1)[0] - assert "applicability manifest has no entry" in section + assert fact in " ".join(conformance_section().split()) + + +def test_the_transcription_says_it_cannot_be_checked_from_here() -> None: + """A transcription that reads like a live check is worse than no transcription.""" + assert "Nothing in CI can check it against the source." in conformance_section() + + +def test_the_table_marks_exactly_the_one_standard_the_manifest_marks_not_applicable() -> ( + None +): + """The registry marks one N/A and fourteen applying. A second N/A here is a divergence. + + This is the one part of the transcription that is mechanically checkable from inside + the repository, because the verdicts are in this table. It does not prove the table + agrees with the registry; it fails when the table stops agreeing with what it says + the registry said. + """ + not_applicable = sorted( + name for name, state in conformance_rows().items() if state.startswith("N/A") + ) + assert not_applicable == ["AI Evaluation"], ( + "the manifest marks AI Evaluation N/A and fourteen standards as applying; " + f"this table marks {not_applicable}" + )