diff --git a/.gitignore b/.gitignore index fbc2c8a4e..d3f808dcf 100644 --- a/.gitignore +++ b/.gitignore @@ -290,6 +290,25 @@ scripts/security/scan-tokens.local.txt # directory itself readable so the negation binds. Do not "simplify" it back. /docs/testing/* !/docs/testing/VERIFY.md +# docs/testing/README.md is the SECOND negation, and it is a marker rather than QA material. +# +# The rule above works, and its success is the problem: from an engine checkout the directory holds +# one file, so a path cited under it is INDISTINGUISHABLE from work that was never done. A reader +# greps a cited path, finds nothing, and concludes the document does not exist -- when it does, in +# the vault. Measured 2026-09-03: BACKLOG #1011 step 6 instructed a reader to mark a disposition +# discharged in a path under this tree and called that file "the only place in the repo that records +# the true state", a sentence that is false from any engine checkout because the file is not in the +# repo at all. A builder acting on it would have claimed a criterion it could never read. +# +# The marker is the checkout-side answer: standing in the directory, or browsing to it on GitHub, +# you are told the material moved and told not to read absence as evidence. It is a POINTER, never +# an index -- it names no vaulted document, because a path-to-document map over a closed set +# discloses what is NOT covered by subtraction. Keep it that way if you edit it. +# +# This negation and the VERIFY.md one are the ONLY two permitted here, and +# tests/test_private_paths_stay_ignored.py pins the pair as an exact set. A third fails there until +# someone writes down why. See BACKLOG #1435. +!/docs/testing/README.md # DO NOT ADD docs/benchmarks/ OR docs/quality-gates/ HERE. Both are DELIBERATELY ABSENT. ADR 0160 D6.2 # considered both and left them: the 21 bench handoffs carry the measurement narrative for the ~133 # data files beside them, so untracking either half strips the rationale from records that stay. diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 987a97699..e85140148 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -4876,7 +4876,7 @@ Retiring the tree costs the engine nothing operationally: **`tests/test_ech_egre 3. **Rewrite the `SECURITY.md` 12.1.5 paragraph** so the residual rests on the true ground: not "infeasible", but *buildable off-stdlib and deliberately not owned, because no partner endpoint publishes an ECHConfig and the engine's own TLS stack cannot originate ECH until OpenSSL 4.0*. Same accepted residual; a premise that survives inspection. 4. **Re-point `rest.py`'s docstring** — `ech_sidecar_url_from_settings` names `tools/ech-sidecar/` by path and asserts it is "proven to hide the SNI". After deletion that path resolves to nothing; the reference becomes the historical commit, and the "proven" claim needs whatever evidence actually backs it or should go. 5. **Decide `samples/ech-sidecar/README.md`** — the operator recipe (the only file under that sample dir) describes running a sidecar the repo no longer contains. Re-aim it at the generic contract (any loopback ECH-terminating proxy) or retire it with the tree. -6. **Mark SEC-71 discharged** in `docs/testing/master-test-plan/16-security-phi-and-supply-chain.md`, which specifies this exact disposition and is currently the only place in the repo that records the true state. +6. **Mark SEC-71 discharged** in `docs/testing/master-test-plan/16-security-phi-and-supply-chain.md`, which specifies this exact disposition. **Corrected 2026-09-03 (BACKLOG #1435): that path is NOT in this repository and this step is not reachable from an engine checkout.** `docs/testing/` is gitignored except `VERIFY.md`, so the document lives in the vault clone and whoever holds it performs this step. The clause this replaces said the file "is currently the only place in the repo that records the true state" — false from any checkout, since the file is not in the repo at all, and a builder acting on it would have claimed a criterion it could never read. See [`testing/README.md`](testing/README.md). **Migration cost: none.** MessageFoundry is a not-yet-deployed beta with zero production instances, and `tools/` has never been in an sdist or a wheel — so no consumer of any published artifact is affected by deleting it. There is no deprecation window to run and nothing to sequence. @@ -4884,7 +4884,7 @@ Retiring the tree costs the engine nothing operationally: **`tests/test_ech_egre **Related:** #1010 (split from the same draft — the licence-header gate; `main.go` is 1 of the 196 headerless sources and gets a header only if this rules *keep*), #272 (ADR 0139's owning item), #353 (an ungated compliance artifact — same "nothing compares it to the record" shape), #1000 (a gate that has never been watched fail; here the failure is a *language* nothing gates at all). -**Source:** master test plan **SEC-71** (`docs/testing/master-test-plan/16-security-phi-and-supply-chain.md`), which specifies this disposition; escalated 2026-08-04 when the SPDX half of the original draft was found to rest on a false claim and was split out as **#1010**. Every fact above was re-executed against `origin/main` at `df9c4d54`. +**Source:** master test plan **SEC-71** (`docs/testing/master-test-plan/16-security-phi-and-supply-chain.md` — vault-only, unreadable from an engine checkout; see [`testing/README.md`](testing/README.md)), which specifies this disposition; escalated 2026-08-04 when the SPDX half of the original draft was found to rest on a false claim and was split out as **#1010**. Every fact above was re-executed against `origin/main` at `df9c4d54`. ## 1002. AG-rig validation: prove the multi-subnet failover reconnect @@ -21928,6 +21928,65 @@ The other four are posture, inherited from the tripwire as it already stood: --- +## 1435. docs/testing has no checkout-side marker, so a vaulted path reads as unstarted work + +> 🚧 **Filed 2026-09-03. The marker is built; the gate is researched and deliberately NOT built.** Value **5/10** · Difficulty **2/10** for what shipped, **6/10** for what is left. A documentation and process defect, not a product exposure: MessageFoundry has zero deployments, nothing here reaches an operator, and no impact claim below is present-tense about a running site. + +**The defect, in one sentence: from an engine checkout, work that is UNREADABLE is indistinguishable from work that is UNSTARTED.** + +`git ls-files docs/testing` returns one file, and `/docs/security/` is blanket-ignored at `.gitignore:177`. The real documents live in the vault clone. So a reader who greps a cited path, finds nothing, and concludes the work was never done is reasoning correctly from everything the tree shows them, and reaching a false conclusion. + +**The measured instance, and it carries its own proof.** #1011 step 6 told a reader to mark SEC-71 discharged in a path under `docs/testing/`, and called that file *"currently the only place in the repo that records the true state"*. That sentence is false from any engine checkout, because the file is not in the repo at all. A builder acting on it would have claimed a criterion it could never read. Corrected in the same change that filed this item; both citations in #1011 now name the unreadability in the same breath as the path. + +Two peer fleets hit the same wall independently on 2026-09-03, on #1152 and on #1193, and a third case put half an availability strategy in a vaulted document under #1191. Three sessions worked it out unaided. The next one may not, and a better brief does not fix it, because the brief author hits the same blind spot. + +### What shipped here + +`docs/testing/README.md`, tracked through a second negation under `/docs/testing/*`. A reader standing in the directory, or browsing to it on GitHub, is told the material moved and told not to read absence as evidence. `docs/README.md` carries the same warning at the front door. + +**It is a POINTER and never an index.** It names no vaulted document. A path-to-document map over a closed set discloses what is NOT covered by subtraction, which is the disclosure the gitignore exists to prevent. Keep it that way. The `docs/security/` half already has its rule in [`SECURITY-DOCS-POLICY.md`](SECURITY-DOCS-POLICY.md) and the marker links there rather than restating it. + +`tests/test_private_paths_stay_ignored.py` pins the pair as an exact set, so a third negation fails until someone writes down why, and the marker cannot be silently dropped from fresh clones. + +### The gate: researched, measured, and NOT built. This is the finding, not a shortfall. + +A test that reds when a ledger body cites a vaulted path without flagging it is the most durable shape and was the first choice. It does not survive contact with the corpus. All figures measured 2026-09-03 against `46ea10a78`, and they are measurements rather than targets. + +| Granularity | Detection | Result on the corpus | +|---|---|---| +| Per line | path pattern | 55 lines cite, 23 carry no flag on the same line | +| Per line, about three lines of context | path pattern plus `THREAT-MODEL` and `asvs-scorecard` | 75 lines cite, 22 carry no signal (measured independently, two readers agreeing to the line) | +| Per item body | path pattern | 27 items cite, 2 unflagged | +| Per item body | path pattern plus `master-test-plan` | 32 items cite, 5 unflagged | +| Per citation, plus or minus 300 to 1200 characters | path plus name | 15 to 21 items carry at least one unflagged citation | + +**Three findings, each independently enough to stop the build.** + +1. **An unscoped gate reds about 22 times on day one**, against a corpus nobody repairs in one pull request. It needs a grandfather baseline or a changed-lines-only scope before it is shippable at all. +2. **The item-granularity gate is quiet and measures the wrong thing.** At 2 unflagged it would ship comfortably, and it MISSES #1152 — the case that motivated this item. #1152's body flags a different vaulted document (the scorecard) while leaving its master-test-plan clause unflagged, so the item passes while the defect stands. A gate that is quiet because it is aimed slightly wrong is worse than no gate. +3. **"Cites a vaulted path" overfires, and the real predicate may not be expressible as a pattern.** Line 22 is prose ABOUT the arrangement, not a citation anyone must follow. A table row giving a cell count is permitted vocabulary. Neither is a defect. The actual predicate is **a citation a reader must FOLLOW in order to act**, which is a judgement about intent, and no regex tried here expresses it. + +**The window parameter is undefended.** Three lines is arbitrary; nobody has a reason for it, and a single-block rule already missed a stale assertion in `docs/SERVICE.md` on a peer's item. Treat 53 against 22 as an order of magnitude, not a threshold. + +**IF SOMEONE BUILDS THIS, THE FAILURE MESSAGE IS ALSO A DISCLOSURE SURFACE.** A message naming WHICH paths are vaulted publishes the map into CI logs and into the workflow file, both public. That is the enumerating-marker defect wearing different clothes, and the helpful-error-message instinct walks straight into it. A message of the shape *"this item body cites a maintainer-internal path; state in the body that it is unreadable from an engine checkout"* tells the author exactly what to do and discloses nothing. Do not echo the offending path into the message, the summary, or the annotation. + +### The two-sided controls, named, so nobody re-derives them + +A gate proven on one side proves nothing: one that fires on everything and one that fires on nothing both look green against a single-sided test. + +**Fires side (must red), and it is on `main` today:** `docs/BACKLOG.md` lines 4763 and 4771 before the correction in this change. Both named a `docs/testing/master-test-plan/` chapter, and 4763 instructed the reader to act in it. The cleanest true positive in the corpus, and independent of #1152. Recover the pre-correction text from this item's own commit. + +**Quiet side (must pass):** the remedy bodies on PR 804 (#1191) and PR 812 (#1193). **Both were still unmerged on 2026-09-03**, so a builder calibrating against `main` alone sees the UNFLAGGED versions and draws the wrong conclusion about what the remedy looks like. Read them with `gh pr diff 804` and `gh pr diff 812`. #1191 is the sharper of the two: it records that the availability inventory is vaulted AND records that its builder deliberately did not add a read-side anchor to the public `_RESOURCE_ANCHORS` registry, because the registry is public and the document is not, so the anchor would red a tree nobody in an engine checkout can see or repair. + +Both are the shape a gate must PERMIT. An unflagged citation is the defect; a flagged one is the remedy the gate exists to produce. + +### What is left + +- Decide whether the follow-to-act predicate is expressible. If it is not, that is a finding and this item closes on the marker. +- If it is, scope the gate to changed lines or grandfather the day-one population, and prove it on both controls above. +- The `docs/security/` half has no marker of its own. `/docs/security/` is a blanket rule, so a placeholder there needs a deliberate gitignore negation. Judged not worth it here: [`SECURITY-DOCS-POLICY.md`](SECURITY-DOCS-POLICY.md) already states that rule in a tracked file and says how to request the material, and a second marker would restate a load-bearing fact instead of linking to it. + +**Related:** #1011 (the corrected instance), #1152, #1191 and #1193 (the three peer cases), #1244 (an engine change breaking a vault anchor produces no attribution — the same public-repo-cannot-see-the-vault shape, from the other direction). ## 1439. webconsole_seam_snapshot.py resolves messagefoundry from sys.path, not from its own repo > 🔢 **Filed 2026-09-03 - BUILT IN THIS COMMIT, not yet landed.** Found while building #1139, by a session that spent its debugging on the gate instead of the generator. The census in LIMB 3 is the reason this is filed as one defect and not a class. diff --git a/docs/README.md b/docs/README.md index db3cd2c3d..3dc2cef5f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,6 +8,12 @@ different repositories for opposite audiences. `docs/ASVS-L2-PHASE0-CHANGES.md` `docs/testing/VERIFY.md` is an operator tool, not a test plan. Where a name misleads, this index says so rather than repeating it. +**A path you cannot open is not proof the work was never done.** Parts of `docs/` are kept in a separate +maintainer repository, so a citation elsewhere in the tree can be real and still not resolve here. +[`testing/README.md`](testing/README.md) covers `docs/testing/`, and +[`SECURITY-DOCS-POLICY.md`](SECURITY-DOCS-POLICY.md) states the separate rule for `docs/security/` and +`docs/reviews/`, including how to request the withheld material. + > **Reporting a security vulnerability?** → **[`.github/SECURITY.md`](../.github/SECURITY.md)**, the > disclosure policy. **Not** `docs/SECURITY.md`, which despite the identical filename is the > authentication and RBAC reference. Two files, same name, different jobs. diff --git a/docs/testing/README.md b/docs/testing/README.md new file mode 100644 index 000000000..2040e1f8e --- /dev/null +++ b/docs/testing/README.md @@ -0,0 +1,63 @@ +# Most of this directory is not in this repository + +Only [`VERIFY.md`](VERIFY.md) is tracked here. Everything else under `docs/testing/` is +maintainer QA planning, and it is kept in a separate maintainer repository. + +**So a path under `docs/testing/` that you cannot open says nothing about whether the work +exists.** Unreadable and unstarted look identical from an engine checkout, and that is the +mistake this page exists to stop. + +## Why you are probably reading this + +You followed a citation. `docs/BACKLOG.md`, an ADR, a test docstring or a source comment names a +path under `docs/testing/`, you went looking for it, and it is not on disk. That citation is real. +It names a document that exists, and it is written for provenance: it records where a decision or +a test row came from, not a file you can open here. + +`git ls-files docs/testing` returns one file. That is the rule working, not a gap. + +## What this means when you are grading work + +Do not treat an unreadable document as an absent one. + +If an item asks you to satisfy a criterion, discharge a row, or check a claim in a document under +`docs/testing/`, you cannot verify that half from an engine checkout. Say so in those words. Record +the limb as unverified, not as satisfied and not as missing. Claiming you met a criterion you could +not read is the failure mode; it happened, and it is why this file is here. + +The same caution applies to the reverse move. A limb that looks undone may simply be graded in a +document you do not have. + +## Why the split + +[ADR 0160](../adr/0160-public-repo-content-policy-operator-and-security-review-material-only.md) +sets the test: a tracked file must be something an operator running MessageFoundry needs, or +something a security reviewer assessing it needs. QA planning for a maintainer build box is +neither, so it moved. `VERIFY.md` stayed because it documents `messagefoundry verify`, the on-box +acceptance check a real deployment runs. + +None of this is a confidentiality control and it must never be described as one. This material was +never secret. It is process noise, and moving it was subject-matter tidying. + +The `.gitignore` block that implements the split carries the full reasoning. Search it for +`/docs/testing/*`. + +## The neighbouring case, which is a different rule + +`docs/security/` and `docs/reviews/` are also absent, and they are held back for their own reasons. +[`SECURITY-DOCS-POLICY.md`](../SECURITY-DOCS-POLICY.md) states that rule, says what is public, and +tells adopters, evaluators and security reviewers how to request the withheld material. Read it +there. Do not assume the reasoning on this page carries over. + +## What this page will not do + +It will not list what is in the other repository. A path-to-document map over a closed set hands out +what is not covered by subtraction, so this is a pointer and never an index. If you need something +named here, ask a maintainer through the route `SECURITY-DOCS-POLICY.md` documents. + +## If you are writing a citation + +Name the unreadability in the same breath as the path. One clause is enough, for example that the +path is vault-only, gitignored, or unreachable from an engine checkout. A reader who meets your +citation months later gets the same warning you would have wanted, without having to find this page +first. diff --git a/tests/test_private_paths_stay_ignored.py b/tests/test_private_paths_stay_ignored.py index a961da860..12c1ab898 100644 --- a/tests/test_private_paths_stay_ignored.py +++ b/tests/test_private_paths_stay_ignored.py @@ -81,7 +81,15 @@ # runs, and docs/README.md lists it as step 6 of "Start here -- a new operator, in order" while # line 8 warns that it "is an operator tool, not a test plan". Its siblings are maintainer QA: # two drafts awaiting owner approval, and a matrix and plan scoped to one specific build box. - "/docs/testing/*": frozenset({"docs/testing/VERIFY.md"}), + # + # docs/testing/README.md is the second, and it is a MARKER rather than QA material -- the + # checkout-side answer to the rule above succeeding too well. With one file tracked here, a path + # cited under this tree is indistinguishable from work that was never done: a reader greps, finds + # nothing, and concludes the document does not exist. It does, in the vault. The marker says so + # in the one place a reader standing in the directory cannot miss it, and it is a POINTER rather + # than an index -- it names no vaulted document, because a path-to-document map over a closed set + # discloses what is NOT covered by subtraction. See BACKLOG #1435. + "/docs/testing/*": frozenset({"docs/testing/VERIFY.md", "docs/testing/README.md"}), } @@ -181,27 +189,41 @@ def test_the_negation_re_includes_exactly_one_file() -> None: ) -def test_the_testing_negation_re_includes_exactly_verify_md() -> None: - """The second negation, asserted in both directions for the same reason as the first. +@pytest.mark.parametrize("negated_path", ["docs/testing/VERIFY.md", "docs/testing/README.md"]) +def test_the_testing_negations_re_include_their_two_files(negated_path: str) -> None: + """Both negations under `/docs/testing/*`, asserted for the same reason as the `.claude/` one. `/docs/testing/*` carries the identical one-character hazard: written `/docs/testing/` the - directory is excluded, git never descends into it, and `!/docs/testing/VERIFY.md` parses fine - while applying to nothing. The failure is silent -- the operator's step 6 link would simply stop - resolving for anyone who cloned, with no error at commit, push or CI. - - The siblings are asserted too, because the risk here is the opposite of the `.claude/` one. There - the danger was publishing session state; here it is publishing maintainer QA that names a specific - build box (`WIN2025-TEST-PLAN.md` carries the host and its service identity), plus two drafts that - say on their face they are awaiting owner approval. + directory is excluded, git never descends into it, and a `!` line under it parses fine while + applying to nothing. The failure is silent -- the file would simply stop reaching anyone who + cloned, with no error at commit, push or CI. + + `VERIFY.md` is the operator tool docs/README.md links as step 6 of the new-operator path. + `README.md` is the MARKER (BACKLOG #1435), and its silent loss is the more interesting one: it is + the only thing in a checkout that tells a reader a cited path under this tree is unreadable rather + than unwritten. Lose it and the tree reverts to the state where absence reads as evidence. """ - negated = _git("check-ignore", "-q", "--no-index", "docs/testing/VERIFY.md") + negated = _git("check-ignore", "-q", "--no-index", negated_path) assert negated.returncode != 0, ( - "`docs/testing/VERIFY.md` is IGNORED — the `!/docs/testing/VERIFY.md` negation is not taking " - "effect. Check that the rule above it is `/docs/testing/*` and not `/docs/testing/`: a " - "negation cannot re-include a file whose parent directory is excluded, and it fails silently. " - "This file is an OPERATOR tool and docs/README.md links it as step 6 of the new-operator path." + f"`{negated_path}` is IGNORED -- its `!` negation is not taking effect. Check that the rule " + "above it is `/docs/testing/*` and not `/docs/testing/`: a negation cannot re-include a file " + "whose parent directory is excluded, and it fails silently." ) + +def test_the_testing_negations_re_include_nothing_else() -> None: + """The other direction: exactly two files travel, and a third needs a written decision. + + The risk here is the opposite of the `.claude/` one. There the danger was publishing session + state; here it is publishing maintainer QA -- two drafts that say on their face they await owner + approval, plus a matrix and plan scoped to one specific build box. + + On that build box, note what the risk is NOT. `WIN2025-TEST-PLAN.md` uses ``, + `` and credential placeholders throughout, so it names no host and no identity; .gitignore + records that correction beside the rule. The removal is right on ADR 0160's subject test -- QA + planning for one build box is neither operator nor security-reviewer material -- and overstating + it as a disclosure risk is how a correct decision acquires a wrong rationale that outlives it. + """ for sibling in ( "docs/testing/MASTER-TEST-PLAN.md", "docs/testing/WIN2025-TEST-PLAN.md", @@ -210,6 +232,7 @@ def test_the_testing_negation_re_includes_exactly_verify_md() -> None: ): res = _git("check-ignore", "-q", "--no-index", sibling) assert res.returncode == 0, ( - f"{sibling!r} is NOT ignored. The negation is meant to cover `VERIFY.md` alone. If this " - "path is now meant to travel, pin it in _TRACKED_EXCEPTIONS and say why in .gitignore." + f"{sibling!r} is NOT ignored. The negations are meant to cover `VERIFY.md` and " + "`README.md` alone. If this path is now meant to travel, pin it in _TRACKED_EXCEPTIONS " + "and say why in .gitignore." )