From cd1e4e278262fcf8bc17f0db66eb07b5f873ade2 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 5 Oct 2026 09:45:51 +0000 Subject: [PATCH 001/105] Acceptance process: make the cohort runnable and markable mechanically Reader isolation and the packet, a marking section (question 2, traceable misses, unanswered questions, the same passage, delegated marking), thresholds for any cohort size, row precedence, the record, the reviewer and agent-surface rebuild in CONTRIBUTING.md, and how the process itself changes. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 65 +++++++++++++++++++++++++++++++++------ CHANGELOG.md | 22 +++++++++++++ CONTRIBUTING.md | 18 +++++++---- acceptance/README.md | 15 ++++----- acceptance/runs/README.md | 8 ++--- 5 files changed, 101 insertions(+), 27 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 31c8df0..f0395b3 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -11,8 +11,15 @@ test is run with a low-strength model so that clarity is measured at the floor, - **At least five independent readers**, each a fresh instance of a low-strength model (the smallest current model in its family, for example Haiku), with no shared context and no access to the change history, the critique findings or each other. -- Each reader receives **only the changed document or documents** and the fixed question set - below. Nothing else is read first. +- **Each reader is a separate process with no tools and no context but its packet**, started + outside any repository so that no project instruction or settings file reaches it: for + example `claude -p` with every tool disabled, run from an empty directory. A reader that + shares the editor's context, such as an agent dispatched from the editor's own session, is not + a cold reader, and its answers do not count towards a run. +- Each reader receives **only its packet**: the changed document, the document's question set + and the fixture the questions name, each in full under its file name, in the reader's prompt. + The question set carries the instruction to the reader, and nothing is added around it. + Every reader in a run receives the same packet. Nothing else is read first. - Readers are run **after** every stronger review has been applied. They are the last gate, not the first draft. @@ -61,23 +68,49 @@ about any of them is the editor's decision. An earlier version of this test aske what would make a clause clearer and for a clarity rating; both were removed on the author's instruction for that reason, and neither is reinstated. +## Marking + +The editor marks each answer sheet against the document's key, or delegates the marking to a +separate marker that has seen neither the drafting nor the review, and receives only the +document at the commit the readers read, its key, this file and the answer sheets. A delegated +marker's verdict stands. Each sheet is marked question by question, as follows. + +- **Questions 1 and 3** are marked as the key states for each item. +- **Question 2.** An answer is correct when its sentence states what the key's answer states, + including anything the key says an answer is wrong to omit, and its quotation is a sentence of + the document that supports it. The quotation need not be the one the key gives. +- **A miss traceable to the text** is one whose reasoning quotes the document. A miss whose + reasoning quotes nothing, or which misreads the fixture, is the reader's. +- **An unanswered question.** A keyed question left unanswered is a miss, and is not traceable + to the text. Question 4 left unanswered counts as no passage quoted. +- **The same passage.** Two quotations under question 4 are of the same passage when they share + a sentence of the document. Quotations of different sentences are different passages, even + within one paragraph or clause. + +Where a mark needs a judgement these rules do not settle, the marker makes it, and the record +states the judgement and why. + ## Pass criteria -The change passes when all of the following hold on a single run: +The change passes when all of the following hold on a single run. With a cohort of five, "every +reader" is five of five and "all but one" is four of five. | Criterion | Threshold | | ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | -| Load-bearing statements (question 2) | **5 of 5** readers correct on every one | -| Restatement and application (questions 1 and 3) | **At least 4 of 5** readers correct on each keyed item | +| Load-bearing statements (question 2) | **Every** reader correct on every one | +| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | | A wrong answer traceable to the text | **None.** A miss whose reasoning quotes the document is a finding, whatever the count | | A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it | -| Anything from one reader only | Recorded; applied at the editor's discretion; does not block | +| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | + +The rows are applied in order, so a miss traceable to the text blocks even when only one reader +made it; the last row covers only what no earlier row has caught. The asymmetry is deliberate. A load-bearing statement is the sentence the whole document exists to make, and one reader in five getting it wrong means one agent in five will act on the wrong reading; unanimity is the only acceptable result there. Everywhere else a single divergent reader -is more likely to be the reader than the text, so four of five is tolerated, but a miss that the -reader can justify from the document is always the document's fault. +is more likely to be the reader than the text, so one such reader is tolerated, but a miss that +the reader can justify from the document is always the document's fault. ## The bar only moves up @@ -107,8 +140,9 @@ A wording change after the run means a fresh run; readers are never reused. Each run has a file under `acceptance/runs/`, named `NNN-.md` with `NNN` the next number in the one sequence shared by the three documents, stating: the document, the pull request and the -commit read, the cohort size and model, the run number in that pull request, the marks per -reader per question, the findings applied, and whether it passed. The changelog line for the +commit read, the cohort size and model, how the readers were isolated, the run number in that +pull request, who marked, the marks per reader per question, any judgement made in marking, the +findings applied, and whether it passed. The changelog line for the change names the file. CI refuses a specification change whose pull request adds no run file; it checks for at least one, and a pull request that changes two documents adds a run for each. @@ -116,3 +150,14 @@ Each published version's changelog entry carries the lines of the changes it con version's cohort record is the set of runs those lines name. A version with no such line has not been accepted and MUST NOT be described as published. The question sets and answer keys live under `acceptance/` in this repository so that a run can be repeated and compared. + +## Changes to this process + +This file, `acceptance/README.md` and the steps in `CONTRIBUTING.md` are not specifications, and +a change to them is not tested by a cohort: the cohort measures whether a low-strength reader can +follow a specification, and the process is followed by the editor. Such a change names the runs +or the case that showed the need for it, has the same independent review as any other, and is +recorded by a line under `Unreleased` in the changelog's entry for the acceptance process. It is +bound by the section on the bar: it may raise a criterion, never relax one. A change to a +question set, key or fixture is not a change to this process; `acceptance/README.md` governs +those. diff --git a/CHANGELOG.md b/CHANGELOG.md index f5f8122..05324bb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -133,3 +133,25 @@ First published version, the draft label dropped on the same day as the method s custom rules, a single-rule harness, identifier resolution, a derived listing of active defences, the project record, self-application, and the conformance definition in section 11. Published before the acceptance test in `ACCEPTANCE.md` existed; 0.2.0 is the first version to pass it. + +## Acceptance process (ACCEPTANCE.md) + +Not a specification and not versioned. Changes are recorded here as `ACCEPTANCE.md` describes, +and need no cohort. + +### Unreleased + +- The cohort can be run and marked without judgement calls that the rules leave open. Each + reader is a separate process with no tools and no context but its packet, and the packet is + the document, its question set and the fixture the questions name, which `ACCEPTANCE.md` and + `acceptance/README.md` previously disagreed on. A new marking section settles what a correct + load-bearing answer needs, what makes a miss traceable to the text, an unanswered question and + "the same passage", and lets the editor delegate marking to a separate marker whose verdict + stands. Thresholds read "every reader" and "all but one", the same as five of five and four of + five for a cohort of five, and the pass criteria rows apply in order. The run record states + how readers were isolated and who marked. `CONTRIBUTING.md` names the reviewer, the + agent-facing rebuild after an Appendix A change, and how a change to this process is made. + Prompted by an agent-driven run of the process that could not follow it mechanically; the + rulings in [run 16](acceptance/runs/016-deferral-clause-run-1.md), + [run 17](acceptance/runs/017-deferral-clause-run-2.md) and + [run 18](acceptance/runs/018-deferral-clause-run-3.md) are written in. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fa1743e..bc249de 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,7 +29,8 @@ review. It is a patch version when released. SHOULD or MAY in a numbered section or appendix. It needs everything a clarity edit needs, plus the five tests in SCOPE.md answered in the pull request, and it is at least a minor version when released. `tools/changes.py` tells the two apart mechanically and CI runs it on every pull -request. +request. A change to the acceptance process itself is neither, and ACCEPTANCE.md says how it is +made. ## What a pull request contains @@ -39,6 +40,9 @@ request. request template lays them out. - The observed case that prompted it, for either kind. A clarity edit still has a reason: a reader who took the passage the wrong way. +- Where a specification's Appendix A changed, the agent-facing files rebuilt with + `python3 build-agent-surfaces.py` and committed with the change; CI runs it with `--check` + and fails when they are stale. - `python3 spec-qa.py` clean and the unit tests passing, which CI checks for you. A pull request is a suggestion, not a demand. The documents have one editor, who keeps ownership @@ -55,9 +59,10 @@ survives, the change is yours. the editor sets the pre-release header. 2. **Scope.** For an obligation, the editor confirms each SCOPE.md test. A failed test closes the pull request with a row in `DECLINED.md`. -3. **Review and rewrite.** An independent review of the diff against the document, then the - editor's rewrite, pushed to the pull request branch where the contributor allows it or to an - editor's branch that supersedes it. +3. **Review and rewrite.** An independent review of the diff against the whole document, by a + reviewer who did not draft the change, then the editor's rewrite, pushed to the pull request + branch where the contributor allows it or to an editor's branch that supersedes it. A review + finding applied after a cohort has run is a wording change, and a fresh run follows. 4. **Keys.** Where the change touches a sentence an acceptance key quotes, or adds a load-bearing obligation, the editor updates the key before the cohort runs, never after. 5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text. Its record is @@ -67,8 +72,9 @@ survives, the change is yours. version. A wording change after the run means a fresh run. 6. **Merge** into `next` by merge commit, and the branch is deleted. -Steps 2 to 5 are done by a person with a Claude Code session, not by CI, and they happen in -batches. A quiet pull request is not a declined one; a declined one says so. +Steps 2 to 5 are the editor's, not CI's, and they happen in batches. The editor may have an +agent draft or review, and may delegate marking as ACCEPTANCE.md allows, in which case the +marker's verdict stands; every other decision in these steps stays the editor's. A quiet pull request is not a declined one; a declined one says so. ## Versions and releases diff --git a/acceptance/README.md b/acceptance/README.md index 4cfb6ec..57a2880 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -14,13 +14,14 @@ the record of every run lives in `runs/`. ## How a run works 1. The editor identifies which documents the change touched. Only those are tested. -2. For each, five or more fresh low-strength readers each receive the document's current text, - its `QUESTIONS.md`, and the fixture the questions name. Nothing else: no key, no changelog, no - critique findings, no other reader's answers. -3. The editor marks each answer sheet against `KEY.md` and applies the pass criteria in - `ACCEPTANCE.md`. -4. The result is recorded as a file under [runs/](runs/), with cohort size, model, run number, - marks and the findings applied, and the change's `Unreleased` changelog line names that file. +2. For each, five or more fresh low-strength readers, each isolated as `ACCEPTANCE.md` describes, + each receive the document's current text, its `QUESTIONS.md`, and the fixture the questions + name. Nothing else: no key, no changelog, no critique findings, no other reader's answers. +3. The editor, or a separate marker the editor delegates to, marks each answer sheet against + `KEY.md` under the marking rules in `ACCEPTANCE.md` and applies its pass criteria. +4. The result is recorded as a file under [runs/](runs/), with cohort size, model, how the + readers were isolated, run number, who marked, marks and the findings applied, and the + change's `Unreleased` changelog line names that file. A run that fails is followed by a change to the text, never to the questions, and a fresh cohort. diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index 4994f74..3435821 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -6,10 +6,10 @@ share. Runs 1 to 15 predate this directory and are summarised in the changelog e the toolchain specification; numbering here continues from 16. A record states, in this order: the document; the pull request and the commit the readers were -given; the cohort size and model; the run number within that pull request; the marks per reader -per question against the key; the passages quoted as confusing and by how many readers; the -findings applied to the text as a result; and the verdict against the criteria in -`ACCEPTANCE.md`. Answer sheets are not pasted in full; the marks are the record. +given; the cohort size and model; how the readers were isolated; the run number within that +pull request; who marked; the marks per reader per question against the key, with any judgement +made in marking; the passages quoted as confusing and by how many readers; the findings applied +to the text as a result; and the verdict against the criteria in `ACCEPTANCE.md`. Answer sheets are not pasted in full; the marks are the record. The `Unreleased` changelog line for the change names the record, and `tools/changes.py` refuses a specification change whose pull request adds no file here. From e7a0c5d6e6c5eb4e9866368f88c7c2c59f56c972 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 5 Oct 2026 09:56:07 +0000 Subject: [PATCH 002/105] Acceptance process: apply the first review One rule for a miss traceable to the text: the reasoning quotes the document, or the key says the miss counts; the key can add, never take away, and a quoting miss is traceable even where it misreads the fixture. Quoting is defined, question 2 answers must assert nothing the key marks wrong, unanswered keyed items count one miss each, and "the same passage" is settled for fragments, whole paragraphs and passages named unquoted. A delegated marker receives the question set and fixture too, and the editor supplies the record's facts. Every pass-criteria row must hold; the last covers only what the others do not. Answer sheets, the reader command and a delegated marker's instruction are kept with the record. The process files are named; review findings that change the document mean a fresh run, the step 5 header does not. The three record lists agree, lines are rewrapped, and the changelog names the case from the repository. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 92 ++++++++++++++++++++++++--------------- CHANGELOG.md | 34 +++++++++------ CONTRIBUTING.md | 10 +++-- acceptance/README.md | 6 +-- acceptance/runs/README.md | 14 +++--- 5 files changed, 95 insertions(+), 61 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index f0395b3..e0ba89a 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -17,9 +17,10 @@ test is run with a low-strength model so that clarity is measured at the floor, shares the editor's context, such as an agent dispatched from the editor's own session, is not a cold reader, and its answers do not count towards a run. - Each reader receives **only its packet**: the changed document, the document's question set - and the fixture the questions name, each in full under its file name, in the reader's prompt. - The question set carries the instruction to the reader, and nothing is added around it. - Every reader in a run receives the same packet. Nothing else is read first. + and the fixture the questions name, each in full in the reader's prompt and headed by its file + name. The question set carries the instruction to the reader; no preamble, wrapper or other + text is added beyond the file names. Every reader in a run receives the same packet. Nothing + else is read first. - Readers are run **after** every stronger review has been applied. They are the last gate, not the first draft. @@ -44,9 +45,9 @@ document, so that marking is mechanical. Each reader answers all of them. from different sources; how many grades a project that ships tooling has. 3. **Application.** A fixture case supplied with the questions, graded clause by clause with - Yes, Partial or No and one line of reasoning. Keyed against the intended verdicts; a divergence - counts against the document only when the reader's reasoning shows the clause was read - differently, not when the fixture was misread. + Yes, Partial or No and one line of reasoning. Keyed against the intended verdicts; whether a + divergence counts against the document is settled by the rule for a miss traceable to the + text, under [Marking](#marking). 4. **Confusion.** Any passage the reader found confusing on its one reading, quoted with its clause number, and nothing else: not what would make it clearer, not a proposed change. @@ -71,21 +72,36 @@ instruction for that reason, and neither is reinstated. ## Marking The editor marks each answer sheet against the document's key, or delegates the marking to a -separate marker that has seen neither the drafting nor the review, and receives only the -document at the commit the readers read, its key, this file and the answer sheets. A delegated -marker's verdict stands. Each sheet is marked question by question, as follows. - +separate marker that has seen neither the drafting nor the review. Whoever marks is the marker +in this file. A delegated marker receives only these: the document at the commit the readers +read, its question set, the fixture the questions name and its key; this file and +`acceptance/runs/README.md`, with earlier run records of other pull requests for the record's +shape; the answer sheets; and, from the editor, the facts the record states that the sheets do +not, which are the pull request and the commit, the cohort size and model, the command the +readers were started with and the run number. The record states those facts as the editor +supplied them. A delegated marker's verdict stands. Each sheet is marked question by question, +as follows. + +- **Quoting.** A reader quotes the document when it gives words of the document verbatim, a + whole sentence or part of one. Naming a clause number, or paraphrasing, is not quoting. - **Questions 1 and 3** are marked as the key states for each item. - **Question 2.** An answer is correct when its sentence states what the key's answer states, - including anything the key says an answer is wrong to omit, and its quotation is a sentence of - the document that supports it. The quotation need not be the one the key gives. -- **A miss traceable to the text** is one whose reasoning quotes the document. A miss whose - reasoning quotes nothing, or which misreads the fixture, is the reader's. -- **An unanswered question.** A keyed question left unanswered is a miss, and is not traceable - to the text. Question 4 left unanswered counts as no passage quoted. -- **The same passage.** Two quotations under question 4 are of the same passage when they share - a sentence of the document. Quotations of different sentences are different passages, even - within one paragraph or clause. + including anything the key says an answer is wrong to omit, asserts nothing the key says makes + an answer wrong, and quotes the document in support. The quotation need not be the one the key + gives. +- **A miss traceable to the text** is a miss whose reasoning quotes the document, or a miss the + key says counts against the document. A miss whose reasoning quotes the document is traceable + even where it also misreads the fixture, and even where the key says that miss is the reader's + or counts only with a particular quotation: the key can add to the misses that count against + the document, never take one away. Every other miss is the reader's. +- **An unanswered question.** Each keyed item left unanswered is one miss, not traceable to the + text, so a sheet that leaves three items of question 3 blank has three misses. Question 4 left + unanswered counts as no passage quoted. +- **The same passage.** A quotation under question 4 covers every sentence of the document it + quotes any part of, and a passage named without a quotation, by clause number or description, + covers every sentence of what it names. Two passages are the same when they cover a sentence in + common. Passages with no sentence in common are different, even within one paragraph or + clause. Where a mark needs a judgement these rules do not settle, the marker makes it, and the record states the judgement and why. @@ -99,12 +115,13 @@ reader" is five of five and "all but one" is four of five. | ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | | Load-bearing statements (question 2) | **Every** reader correct on every one | | Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | -| A wrong answer traceable to the text | **None.** A miss whose reasoning quotes the document is a finding, whatever the count | +| A miss traceable to the text | **None.** Any such miss, as Marking defines it, is a finding, whatever the count | | A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it | | Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | -The rows are applied in order, so a miss traceable to the text blocks even when only one reader -made it; the last row covers only what no earlier row has caught. +Every row must hold. The first four apply whatever the count, and the last covers only what none +of them catches: a miss traceable to the text blocks even when only one reader made it, and the +second row's tolerance of one reader does not excuse it. The asymmetry is deliberate. A load-bearing statement is the sentence the whole document exists to make, and one reader in five getting it wrong means one agent in five will act on the wrong @@ -140,11 +157,15 @@ A wording change after the run means a fresh run; readers are never reused. Each run has a file under `acceptance/runs/`, named `NNN-.md` with `NNN` the next number in the one sequence shared by the three documents, stating: the document, the pull request and the -commit read, the cohort size and model, how the readers were isolated, the run number in that -pull request, who marked, the marks per reader per question, any judgement made in marking, the -findings applied, and whether it passed. The changelog line for the -change names the file. CI refuses a specification change whose pull request adds no run file; -it checks for at least one, and a pull request that changes two documents adds a run for each. +commit read, the cohort size and model, how the readers were isolated and the command they were +started with, the run number in that pull request, who marked and, for a delegated marker, how it +was started, the marks per reader per question, any judgement made in marking, the passages +quoted as confusing and by how many readers, the findings applied, and whether it passed. Each +reader's answer sheet is kept unedited beside the record, one file per reader in a directory +`NNN-/`, with the instruction a delegated marker was given, so that an owner can check the +isolation and the marks. The changelog line for the change names the file. CI refuses a +specification change whose pull request adds no run file; it checks for at least one, and a pull +request that changes two documents adds a run for each. Each published version's changelog entry carries the lines of the changes it contains, so a version's cohort record is the set of runs those lines name. A version with no such line has not @@ -153,11 +174,12 @@ under `acceptance/` in this repository so that a run can be repeated and compare ## Changes to this process -This file, `acceptance/README.md` and the steps in `CONTRIBUTING.md` are not specifications, and -a change to them is not tested by a cohort: the cohort measures whether a low-strength reader can -follow a specification, and the process is followed by the editor. Such a change names the runs -or the case that showed the need for it, has the same independent review as any other, and is -recorded by a line under `Unreleased` in the changelog's entry for the acceptance process. It is -bound by the section on the bar: it may raise a criterion, never relax one. A change to a -question set, key or fixture is not a change to this process; `acceptance/README.md` governs -those. +This file, `acceptance/README.md`, `acceptance/runs/README.md` and `CONTRIBUTING.md` are the +process files, and a change to any part of them is a change to this process. They are not +specifications, and a change to them is not tested by a cohort: the cohort measures whether a +low-strength reader can follow a specification, and the process is followed by the editor. Such +a change names the runs or the case that showed the need for it, has the same independent review +as any other, and is recorded by a line under `Unreleased` in the changelog's entry for the +acceptance process. It is bound by the section on the bar: it may raise a criterion, never relax +one. A change to a question set, key or fixture is not a change to this process; +`acceptance/README.md` governs those. diff --git a/CHANGELOG.md b/CHANGELOG.md index 05324bb..7df15d9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -134,24 +134,30 @@ custom rules, a single-rule harness, identifier resolution, a derived listing of the project record, self-application, and the conformance definition in section 11. Published before the acceptance test in `ACCEPTANCE.md` existed; 0.2.0 is the first version to pass it. -## Acceptance process (ACCEPTANCE.md) +## Acceptance process -Not a specification and not versioned. Changes are recorded here as `ACCEPTANCE.md` describes, -and need no cohort. +Not a specification and not versioned. Changes to the process files `ACCEPTANCE.md` names are +recorded here as it describes, and need no cohort. ### Unreleased - The cohort can be run and marked without judgement calls that the rules leave open. Each reader is a separate process with no tools and no context but its packet, and the packet is - the document, its question set and the fixture the questions name, which `ACCEPTANCE.md` and - `acceptance/README.md` previously disagreed on. A new marking section settles what a correct - load-bearing answer needs, what makes a miss traceable to the text, an unanswered question and - "the same passage", and lets the editor delegate marking to a separate marker whose verdict - stands. Thresholds read "every reader" and "all but one", the same as five of five and four of - five for a cohort of five, and the pass criteria rows apply in order. The run record states - how readers were isolated and who marked. `CONTRIBUTING.md` names the reviewer, the - agent-facing rebuild after an Appendix A change, and how a change to this process is made. - Prompted by an agent-driven run of the process that could not follow it mechanically; the - rulings in [run 16](acceptance/runs/016-deferral-clause-run-1.md), + the document, its question set and the fixture the questions name, with nothing added around + them. A new marking section says what a delegated marker receives and that its verdict stands, + what counts as quoting, what a correct load-bearing answer needs, which misses are traceable + to the text and how the key's statements bear on that, how unanswered items count, and when + two confusing passages are the same. Thresholds read "every reader" and "all but one", the + same as five of five and four of five for a cohort of five; every pass-criteria row must hold, + and the last covers only what the others do not. The run record states how readers were + isolated and who marked, and the answer sheets are kept beside it. `CONTRIBUTING.md` requires + a reviewer who did not draft the change, says which review findings mean a fresh run, adds + the agent-facing rebuild after an Appendix A change, and points to how a change to the process + files is made. Prompted by gaps in the text as it stood: `ACCEPTANCE.md` named no fixture in + what a reader receives whilst `acceptance/README.md` did, the thresholds were written as five + of five and four of five for a cohort of at least five, two pass-criteria rows both covered a + single reader's traceable miss, and nothing settled an unanswered question or "the same + passage". [Run 16](acceptance/runs/016-deferral-clause-run-1.md), [run 17](acceptance/runs/017-deferral-clause-run-2.md) and - [run 18](acceptance/runs/018-deferral-clause-run-3.md) are written in. + [run 18](acceptance/runs/018-deferral-clause-run-3.md) ruled on these case by case, and their + rulings are written in. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bc249de..a555d7f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,8 +29,8 @@ review. It is a patch version when released. SHOULD or MAY in a numbered section or appendix. It needs everything a clarity edit needs, plus the five tests in SCOPE.md answered in the pull request, and it is at least a minor version when released. `tools/changes.py` tells the two apart mechanically and CI runs it on every pull -request. A change to the acceptance process itself is neither, and ACCEPTANCE.md says how it is -made. +request. A change to the process files ACCEPTANCE.md names, this file among them, is neither, +and ACCEPTANCE.md says how it is made. ## What a pull request contains @@ -62,7 +62,8 @@ survives, the change is yours. 3. **Review and rewrite.** An independent review of the diff against the whole document, by a reviewer who did not draft the change, then the editor's rewrite, pushed to the pull request branch where the contributor allows it or to an editor's branch that supersedes it. A review - finding applied after a cohort has run is a wording change, and a fresh run follows. + finding that changes the document's text after a cohort has run is a wording change, and a + fresh run follows; the header the editor sets at step 5 is not a wording change. 4. **Keys.** Where the change touches a sentence an acceptance key quotes, or adds a load-bearing obligation, the editor updates the key before the cohort runs, never after. 5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text. Its record is @@ -74,7 +75,8 @@ survives, the change is yours. Steps 2 to 5 are the editor's, not CI's, and they happen in batches. The editor may have an agent draft or review, and may delegate marking as ACCEPTANCE.md allows, in which case the -marker's verdict stands; every other decision in these steps stays the editor's. A quiet pull request is not a declined one; a declined one says so. +marker's verdict stands; every other decision in these steps stays the editor's. A quiet pull +request is not a declined one; a declined one says so. ## Versions and releases diff --git a/acceptance/README.md b/acceptance/README.md index 57a2880..c7e8d04 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -19,9 +19,9 @@ the record of every run lives in `runs/`. name. Nothing else: no key, no changelog, no critique findings, no other reader's answers. 3. The editor, or a separate marker the editor delegates to, marks each answer sheet against `KEY.md` under the marking rules in `ACCEPTANCE.md` and applies its pass criteria. -4. The result is recorded as a file under [runs/](runs/), with cohort size, model, how the - readers were isolated, run number, who marked, marks and the findings applied, and the - change's `Unreleased` changelog line names that file. +4. The result is recorded as a file under [runs/](runs/), stating what `ACCEPTANCE.md` lists, + with the answer sheets kept beside it, and the change's `Unreleased` changelog line names + that file. A run that fails is followed by a change to the text, never to the questions, and a fresh cohort. diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index 3435821..6f0fddc 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -5,11 +5,15 @@ share. Runs 1 to 15 predate this directory and are summarised in the changelog e 1.0.0 and 1.0.1 of the method specification, 1.0.0 of the detector specification and 0.2.0 of the toolchain specification; numbering here continues from 16. -A record states, in this order: the document; the pull request and the commit the readers were -given; the cohort size and model; how the readers were isolated; the run number within that -pull request; who marked; the marks per reader per question against the key, with any judgement -made in marking; the passages quoted as confusing and by how many readers; the findings applied -to the text as a result; and the verdict against the criteria in `ACCEPTANCE.md`. Answer sheets are not pasted in full; the marks are the record. +A record states what `ACCEPTANCE.md` lists, in this order: the document; the pull request and +the commit the readers were given; the cohort size and model; how the readers were isolated and +the command they were started with; the run number within that pull request; who marked and, for +a delegated marker, how it was started; the marks per reader per question against the key, with +any judgement made in marking; the passages quoted as confusing and by how many readers; the +findings applied to the text as a result; and the verdict against the criteria in +`ACCEPTANCE.md`. Answer sheets are not pasted into the record, and the marks are the record; the +sheets are kept unedited beside it in a directory `NNN-/`, one file per reader, with the +instruction a delegated marker was given. The `Unreleased` changelog line for the change names the record, and `tools/changes.py` refuses a specification change whose pull request adds no file here. From 62c837aeff19d35b9c87c388765c8fcbc3a68dbf Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 5 Oct 2026 10:31:16 +0000 Subject: [PATCH 003/105] Method 3.2: prefer semantic detectors; detector verdicts name the kind SPEC.md defines lexical, syntactic and semantic detectors in section 1. Clause 3.2 says the practitioner SHOULD host a rule in a semantic detector where the language has one that accepts bespoke rules, a syntactic one before a lexical one otherwise, and MUST record why with the remediation docs where a rule sits lower. The section 3 map and Appendix A carry it, and the agent-facing prompt is rebuilt. DETECTOR-SPEC.md section 8: a verdict on a detector MUST say which of the three it is; that is not a level of conformance. Section 2's gloss carries the terms. TOOLING-SPEC.md needs nothing: its 4.1 summary counts the MUSTs of sections 4 to 7, which are unchanged. Unreleased changelog lines carry a run NNN placeholder; the cohort is next. Headers stay on the published versions until the record commit. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 21 ++++++++++++++ DETECTOR-SPEC.md | 29 ++++++++++++++------ SPEC.md | 41 +++++++++++++++++++++++++--- defence-before-fix-project-prompt.md | 6 ++-- 4 files changed, 82 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f5f8122..cf2a935 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,19 @@ version entry with no cohort line has not been accepted and is not published. ## Method specification (SPEC.md) +### Unreleased + +- New obligation, clause [3.2](SPEC.md#32-build-the-net): where the language has a semantic detector, one that reads types + and symbols, that accepts bespoke rules, the practitioner SHOULD host the rule there, and a + syntactic one before a lexical one where it has none; a rule placed lower MUST have the reason + recorded with its remediation docs. Section [1](SPEC.md#1-terminology) defines lexical, syntactic and semantic, and the + section [3](SPEC.md#3-the-method) map and Appendix A carry the preference. Prompted by project rules hosted in + ast-grep, where the language already had a semantic host, that decided by names: they reported + unrelated calls of the same name and missed calls made through an alias, and clause [3.2](SPEC.md#32-build-the-net) said + any detector qualified whatever its category. The method question set gains load-bearing + statement 2d, which `ACCEPTANCE.md` now lists. Accepted by run NNN. Minor version when + released. + ### 1.1.0, 2026-10-02 - New obligation, section [2](SPEC.md#2-when-the-method-applies): a defect whose fix is deferred MUST be recorded where the project's @@ -68,6 +81,14 @@ pass it. ## Detector specification (DETECTOR-SPEC.md) +### Unreleased + +- New obligation, section [8](DETECTOR-SPEC.md#8-conformance): a verdict on a detector MUST say whether it is semantic, syntactic + or lexical, by the most it can give a bespoke rule to read. It is not a level of conformance and + does not change the verdict; it is stated because method clause [3.2](SPEC.md#32-build-the-net) now prefers a semantic + host, and a verdict of conforming alone presents a lexical detector as an equal host. Section [2](DETECTOR-SPEC.md#2-terminology)'s + gloss carries the three terms. Accepted by run NNN. Minor version when released. + ### 1.0.0, 2026-09-08 First version, split out of toolchain specification 0.1.0 on the same day. States what a detector, diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index cdc6c9e..7c46f3d 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -35,15 +35,16 @@ construct the means has moved the obligation rather than met it. Terms from the method specification carry over unchanged. Every capitalised term links to its definition; the ones this document leans on most are, in short: -| Term | In one line | -| -------------- | ------------------------------------------------------------------------------ | -| [Detector] | A tool that reads code without executing it and reports what it finds | -| [Rule] | One check a [Detector] evaluates | -| [Identifier] | The stable name printed with a finding that leads to its documentation | -| [Practitioner] | Whoever is doing the work, a person or an [Agent] | -| [Toolchain] | Everything a project assembles to run its checks through, [Detectors] included | -| [Suppression] | Making a finding go away without fixing it | -| [Baseline] | A recorded set of existing findings a [Rule] is told to ignore | +| Term | In one line | +| ---------------------------------- | ------------------------------------------------------------------------------- | +| [Detector] | A tool that reads code without executing it and reports what it finds | +| [Semantic], [Syntactic], [Lexical] | What a [Detector] reads: types and symbols, a parse tree without types, or text | +| [Rule] | One check a [Detector] evaluates | +| [Identifier] | The stable name printed with a finding that leads to its documentation | +| [Practitioner] | Whoever is doing the work, a person or an [Agent] | +| [Toolchain] | Everything a project assembles to run its checks through, [Detectors] included | +| [Suppression] | Making a finding go away without fixing it | +| [Baseline] | A recorded set of existing findings a [Rule] is told to ignore | These are additional. @@ -306,6 +307,13 @@ summary for an [Agent]'s context, are obligations on a project's assembled [Tool and are stated in the [toolchain specification](TOOLING-SPEC.md), so there is no separate [Agent]-support level for a [Detector]. +**A verdict on a [Detector] MUST also say whether it is [Semantic], [Syntactic] or [Lexical]**, +by the most it can give a bespoke [Rule] to read. That is not a level of [Conformance] and does +not change the verdict: a [Lexical] [Detector] can [Conform], and a [Semantic] one can fail to. It +is stated because the method specification's clause 3.2 asks a [Practitioner] to prefer a +[Semantic] [Detector] where the language has one that accepts bespoke [Rules], and a verdict that +said only [Conforming] would present a [Lexical] [Detector] as an equal host to a [Semantic] one. + ### 8.1 A maintainer MAY declare the version of this document the detector conforms to, and the declaration records known gaps The declaration is machine-readable, in whatever form the [Detector]'s ecosystem uses to @@ -389,6 +397,7 @@ does not have to. [hazards]: SPEC.md#hazard [identifier]: SPEC.md#identifier [identifiers]: SPEC.md#identifier +[lexical]: SPEC.md#lexical [message]: SPEC.md#message [owner]: SPEC.md#owner [practitioner]: SPEC.md#practitioner @@ -396,5 +405,7 @@ does not have to. [rule]: SPEC.md#rule [rule author]: #rule-author [rules]: SPEC.md#rule +[semantic]: SPEC.md#semantic [suppression]: SPEC.md#suppression +[syntactic]: SPEC.md#syntactic [toolchain]: SPEC.md#toolchain diff --git a/SPEC.md b/SPEC.md index 54ddeb2..e27c911 100644 --- a/SPEC.md +++ b/SPEC.md @@ -70,12 +70,24 @@ The harm a [Class] causes, which is **not necessarily a failure**. It may be a f #### Detector -A tool that reads code without executing it and reports occurrences of a pattern. A [Detector] is not a [Runner]. What a [Detector] must offer so that a [Practitioner] can write, prove, run and resolve a [Rule] in it is specified in the [detector specification](DETECTOR-SPEC.md). +A tool that reads code without executing it and reports occurrences of a pattern. A [Detector] is not a [Runner]. By what it reads, a [Detector] is [Lexical], [Syntactic] or [Semantic], and clause 3.2 says which to prefer. What a [Detector] must offer so that a [Practitioner] can write, prove, run and resolve a [Rule] in it is specified in the [detector specification](DETECTOR-SPEC.md). #### Runner A tool that **executes** code and reports what happened: a test [Runner], a compilation, a benchmark, a smoke check. A [Runner] answers questions about behaviour at one moment, on the paths it happened to exercise; a [Detector] answers questions about the text of the code, everywhere it exists. Detection under this method MUST come from a [Detector], for that reason. +#### Lexical + +Of a [Detector]: reading code as text, as grep does. It sees how the code is spelt and nothing of its structure, so it cannot tell a call from a comment that names it. + +#### Syntactic + +Of a [Detector]: reading a parse tree of the code without types, as ast-grep and Semgrep do. It sees the structure, but knows what a name refers to only by the name. + +#### Semantic + +Of a [Detector]: reading types and resolved symbols, as PHPStan, mypy, Clippy and ESLint with typescript-eslint's type information do. It sees what a name refers to and what type a value has, whatever either is called. The same [Detector] can be [Syntactic] in one configuration and [Semantic] in another: ESLint without type information is [Syntactic]. + #### Rule One pattern definition within a [Detector]. @@ -218,7 +230,7 @@ Six clauses, in order. In brief, before the detail: | Clause | The [Practitioner] MUST | The record shows | Goes to the [Owner] under section 4 | | ------ | ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | 3.1 | Name the [Class] the [Defect] belongs to, bounded both ways, after an independent search | The [Class], the [Hazard] sentence, the two search techniques, the next wider [Rule] not built, whether a [Runner] check pins the reported behaviour | Whether the next wider [Rule] is built, and whether the [Rule] stays narrower than the search showed | -| 3.2 | Express the [Class] as a [Rule] in a [Detector], never a test | The [Rule], and any [Toolchain] gap that stopped a bespoke one | Whether a [Class] that could be defended is left undefended | +| 3.2 | Express the [Class] as a [Rule] in a [Detector], never a test | The [Rule], why it sits below a [Semantic] [Detector] where the language has one, and any [Toolchain] gap that stopped a bespoke one | Whether a [Class] that could be defended is left undefended | | 3.3 | Make the [Rule] fire, on the [Instance] or a [Fixture], in a commit of its own | The red run, and the sentence behind every [Narrowing] | Any exclusion whose sentence cannot be written: that is [Suppression] | | 3.4 | [Sweep] the whole codebase, record the count, then fix every [Instance] | The count, corroborated, and what was fixed by hand or by pattern | Any [Instance] left unfixed, and any [Baseline] | | 3.5 | Make the [Rule] permanent and [Blocking] in the project's own checks | The green run through the project's entry point, and the recorded decision behind any [Suppression] | Removing or disabling the [Rule], and any [Suppression] added to it | @@ -396,6 +408,21 @@ Any tool that reads code and reports pattern matches qualifies, whatever its cat walkers, architectural fitness checks, and [Detectors] that inspect a change before it lands. These are examples rather than a permitted list, and a [Detector] is not disqualified for being unfamiliar. +**Qualifying is not the same as being the right host.** Where the language has a [Semantic] +[Detector] that accepts bespoke [Rules], the [Practitioner] SHOULD express the [Rule] in it; where +it has none, a [Syntactic] one is preferred to a [Lexical] one. The language has one when the +project already runs it or the language's own [Detector] ecosystem offers it, the same check +section 2 asks for before a [Toolchain] gap is recorded. Where the [Rule] is placed lower than +that, the [Practitioner] MUST record why with its [Remediation docs], in a sentence an [Owner] can +check: +that the [Class] is defined by spelling alone, as a forbidden comment is, so a [Lexical] [Rule] +sees every [Instance] of it; or that the project runs no [Semantic] [Detector], and why adopting +one was not practical. A [Detector] that cannot see types decides by names and shapes. It reports +every call named `context`, whatever it is called on, which is a [False positive] clause 3.1 does +not tolerate, and it misses the same call made through an alias, which is an [Instance] the +[Rule] never sees; restricting it by name trades the first failure for the second. A [Semantic] +[Detector] decides by what the code is, so the bounds of clause 3.1 can be held there. + Where the affected language has no extensible [Detector] available, the [Practitioner] MAY build a bespoke one: a program of the project's own that reads code and reports matches of the [Class], run through the project's own entry point like any other [Detector]. This is a last resort rather than @@ -1051,8 +1078,10 @@ This appendix restates sections 2 to 4; where the two differ, the sections gover > for it, say so and fix the [Defect] conventionally. > > Write a custom [Rule] that detects the [Class], in a tool that reads code rather than running it. -> Draw it so that it catches more than the single [Instance] you started from, but never so broadly -> that it matches code which does not carry the [Hazard]. +> Put it in a [Semantic] [Detector], one that sees types and symbols, where the language has one +> that takes custom [Rules]; if you put it in one that sees only syntax or text, write down why in +> the [Rule]'s documentation. Draw the [Rule] so that it catches more than the single [Instance] +> you started from, but never so broadly that it matches code which does not carry the [Hazard]. > > Do not trust the [Rule] as your only way of finding [Instances]. Search independently as well, by > text search and by reading the code, and check the [Rule] catches what you found by hand. Make that @@ -1137,6 +1166,7 @@ This appendix restates sections 2 to 4; where the two differ, the sections gover [escalation]: #escalation [exception]: #exception [exceptions]: #exception +[false positive]: #false-positive [false positives]: #false-positive [fixture]: #fixture [fixtures]: #fixture @@ -1146,6 +1176,7 @@ This appendix restates sections 2 to 4; where the two differ, the sections gover [identifiers]: #identifier [instance]: #instance [instances]: #instance +[lexical]: #lexical [message]: #message [narrowing]: #narrowing [owner]: #owner @@ -1155,9 +1186,11 @@ This appendix restates sections 2 to 4; where the two differ, the sections gover [rules]: #rule [runner]: #runner [runners]: #runner +[semantic]: #semantic [suppress]: #suppression [suppression]: #suppression [suppressions]: #suppression [sweep]: #sweep +[syntactic]: #syntactic [toolchain]: #toolchain [warning]: #warning diff --git a/defence-before-fix-project-prompt.md b/defence-before-fix-project-prompt.md index ab7f4ce..a30009d 100644 --- a/defence-before-fix-project-prompt.md +++ b/defence-before-fix-project-prompt.md @@ -13,8 +13,10 @@ it. Do not decide in advance whether that is possible, attempt it. If you cannot for it, say so and fix the Defect conventionally. Write a custom Rule that detects the Class, in a tool that reads code rather than running it. -Draw it so that it catches more than the single Instance you started from, but never so broadly -that it matches code which does not carry the Hazard. +Put it in a Semantic Detector, one that sees types and symbols, where the language has one +that takes custom Rules; if you put it in one that sees only syntax or text, write down why in +the Rule's documentation. Draw the Rule so that it catches more than the single Instance +you started from, but never so broadly that it matches code which does not carry the Hazard. Do not trust the Rule as your only way of finding Instances. Search independently as well, by text search and by reading the code, and check the Rule catches what you found by hand. Make that From 191f05fb0414701026fbb403b8d6c779c4e737eb Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 5 Oct 2026 10:31:16 +0000 Subject: [PATCH 004/105] Acceptance keys: method 2d on the detector preference, before the cohort The method question set gains load-bearing statement 2d, on which kind of detector should host a rule and what is recorded when another is used; ACCEPTANCE.md lists it with the other three. The method key adds 2d, a restatement bullet and a wrong-if for "only a semantic detector", and keeps 3.2 at Yes for the ledger fixture, which does not say what its analyser reads. The detector key accepts a restatement bullet on the verdict naming the kind and says the Lintomatic verdict is not marked on it. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 5 +++-- acceptance/detector/KEY.md | 37 +++++++++++++++++++++------------- acceptance/method/KEY.md | 24 ++++++++++++++++++---- acceptance/method/QUESTIONS.md | 2 ++ 4 files changed, 48 insertions(+), 20 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 31c8df0..7b5485e 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -25,11 +25,12 @@ document, so that marking is mechanical. Each reader answers all of them. bullets. Keyed against the list of MUST clauses: a bullet is correct when it names an obligation the document actually states. -2. **Load-bearing statements.** Three questions, fixed per document, each asking for one +2. **Load-bearing statements.** Three or more questions, fixed per document, each asking for one sentence and a quotation: - `SPEC.md`: what a practitioner does before fixing a defect, in order; who may baseline or - suppress; what "conforms" means for a remediation. + suppress; what "conforms" means for a remediation; which kind of detector should host a + rule, and what is recorded when another is used. - `DETECTOR-SPEC.md`: what "conforms" means and whether a tool that predates the document can conform; what a detector must do about an inline suppression route; where a bundled rule's documentation must live. diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 132e6cb..9466b0b 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -18,7 +18,11 @@ bullet asserting an obligation the document does not state, is a correct restate offline, with the documentation shipped and versioned with the rule ([6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) to [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together)). - Make any inline suppression route detectable or disableable ([7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)). -Wrong if it asserts: the detector must enforce a project's suppression policy; the detector must +A bullet saying that a verdict on a detector names whether it is semantic, syntactic or lexical +(section [8](../../DETECTOR-SPEC.md#8-conformance)) is correct too, though it is an obligation on whoever gives the verdict. + +Wrong if it asserts: the detector must be semantic to conform; the detector must enforce a +project's suppression policy; the detector must declare a version to conform; the detector must list a project's active defences or produce an agent summary (those belong to the toolchain specification). @@ -44,25 +48,30 @@ agent summary (those belong to the toolchain specification). ## 3. Application: Lintomatic 4.2 -| Clause | Intended verdict | Reasoning the fixture supports | -| ------ | ---------------- | -------------------------------------------------------------------------------------------------------------------- | -| [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules via the `Rule` interface, first class in every report | -| [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `check --only ` runs one rule, bundled or project, with a distinguishing exit code | -| [4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `LM-NNNN` fixed at first release; project names taken as given, not rewritten; printed on every finding line | -| [5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | Local, no account, no server, no network | -| [5.2](../../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Yes | `check ` accepts a single file | -| [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | One line per finding on standard output whether or not the HTML report is enabled | -| [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | The hosted dashboard shows nothing the command line does not print | -| [6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Yes | `explain ` resolves a printed identifier; project rules resolve to their declared `doc` string | -| [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Yes | `explain` reads the installed package; there is no website | -| [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | No | Two of forty bundled rules print `No documentation declared`; every identifier a bundled rule can print MUST resolve | -| [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | The ignore comment is disableable with `ignore_comments = false` | +| Clause | Intended verdict | Reasoning the fixture supports | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | -------------------------------------------------------------------------------------------------------------------- | +| [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules via the `Rule` interface, first class in every report | +| [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `check --only ` runs one rule, bundled or project, with a distinguishing exit code | +| [4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `LM-NNNN` fixed at first release; project names taken as given, not rewritten; printed on every finding line | +| [5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | Local, no account, no server, no network | +| [5.2](../../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Yes | `check ` accepts a single file | +| [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | One line per finding on standard output whether or not the HTML report is enabled | +| [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | The hosted dashboard shows nothing the command line does not print | +| [6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Yes | `explain ` resolves a printed identifier; project rules resolve to their declared `doc` string | +| [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Yes | `explain` reads the installed package; there is no website | +| [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | No | Two of forty bundled rules print `No documentation declared`; every identifier a bundled rule can print MUST resolve | +| [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | The ignore comment is disableable with `ignore_comments = false` | Section [8](../../DETECTOR-SPEC.md#8-conformance) verdict: **does not conform**, because [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) fails and partial conformance MUST NOT be described as conformance. A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Partial on the grounds that thirty-eight resolve has read the clause differently and that counts against the document; a reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Yes because they missed the last paragraph of the fixture has misread the fixture, not the document. +The fixture does not say what Lintomatic reads. A reader who names its kind as unknown, or names +none, is not marked on it; a reader who makes the verdict turn on its kind has read section [8](../../DETECTOR-SPEC.md#8-conformance) +differently, which counts against the document: the kind "is not a level of Conformance and does +not change the verdict". + Clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation): **does not change the verdict**. It is a SHOULD, and Lintomatic's failure to check its own identifiers in CI is a missed SHOULD, not a further MUST failure; the MUST that fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 3ca1a80..74618d9 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -11,6 +11,8 @@ bullet asserting an obligation the document does not state, is a correct restate - Attribute the defect to a class, bounded on both sides, with the hazard stated ([3.1](../../SPEC.md#31-attribute-the-defect-to-a-class)). - Express the class as a rule in a detector that reads code, never a test ([3.2](../../SPEC.md#32-build-the-net)). +- Host the rule in a semantic detector where the language has one that takes bespoke rules, + else a syntactic one before a lexical one, and record why where it sits lower ([3.2](../../SPEC.md#32-build-the-net)). - Prove the rule fires, against the instance or a fixture, in a commit of its own ([3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire)). - Sweep the whole codebase, record the count, then fix every instance ([3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance)). - Make the rule permanent and blocking in the project's checks ([3.5](../../SPEC.md#35-enforce-permanently-and-block)). @@ -20,7 +22,8 @@ bullet asserting an obligation the document does not state, is a correct restate than only mentioning it (section [2](../../SPEC.md#2-when-the-method-applies)). Wrong if it asserts: the practitioner fixes the reported instance first and defends afterwards; -a test may serve as the detector; an agent may baseline the remaining instances. +a test may serve as the detector; an agent may baseline the remaining instances; only a semantic +detector may host a rule. ## 2. Load-bearing statements @@ -42,13 +45,23 @@ a test may serve as the detector; an agent may baseline the remaining instances. rest on reproduction, not on the report". An answer that accepts the pull request description as the verdict is wrong, and so is one that states the six clauses and says nothing of reproduction. +- **2d.** A semantic detector, one that reads types and symbols, where the language has one that + accepts bespoke rules; failing that a syntactic one before a lexical one; and where the rule + sits lower than the language allows, the practitioner records why with the rule's remediation + docs. Key quotations, [3.2](../../SPEC.md#32-build-the-net): "Where the language has a Semantic Detector that accepts bespoke + Rules, the Practitioner SHOULD express the Rule in it; where it has none, a Syntactic one is + preferred to a Lexical one."; "Where the Rule is placed lower than that, the Practitioner MUST + record why with its Remediation docs". An answer that says any detector is as good as another, + that a lower one is forbidden, that the owner must approve it (section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) leaves the choice of + detector to the practitioner), or that names the preference and says nothing of the recorded + reason, is wrong. ## 3. Application: the ledger remediation | Clause | Intended verdict | Reasoning the fixture supports | | ----------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | Class, hazard and the next wider rule are named; Partial or No is accepted where the reader notes that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | -| [3.2](../../SPEC.md#32-build-the-net) | Yes | A rule in the project's static analyser, reading code, not a test | +| [3.2](../../SPEC.md#32-build-the-net) | Yes | A rule in the project's static analyser, reading code, not a test; the fixture does not say what the analyser reads, so nothing shows the rule sits lower than the language allows | | [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | Commit B is red on the fixture and the originating instance and survives as its own commit | | [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Five findings recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority; Partial accepted where the reasoning names that suppression | | [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | The rule is permanent and blocking, but the suppression was not a recorded decision; [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | @@ -68,8 +81,11 @@ A reader who grades [3.5](../../SPEC.md#35-enforce-permanently-and-block) Yes be sentence on unrecorded suppression, and that counts against the document. A reader who grades [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) Yes because the URL resolves has read the versioning sentence of [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) differently, which counts against the document; a reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the -fixture. A reader who attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing -passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as covering it, which is a fair reading and is not marked wrong. +fixture. A reader who grades [3.2](../../SPEC.md#32-build-the-net) Partial or No because no reason for the detector's kind +is recorded has assumed a lower kind of detector than the fixture shows: the record is owed only +where the rule sits lower than the language allows. That counts against the document only if the +reasoning quotes [3.2](../../SPEC.md#32-build-the-net) as requiring the reason whatever the detector. A reader who attributes +the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as covering it, which is a fair reading and is not marked wrong. ## 4. Confusion diff --git a/acceptance/method/QUESTIONS.md b/acceptance/method/QUESTIONS.md index 072a3c0..2c772a2 100644 --- a/acceptance/method/QUESTIONS.md +++ b/acceptance/method/QUESTIONS.md @@ -18,6 +18,8 @@ Answer each in one sentence, followed by the sentence of the document you are re authority? - 2c. What does it mean for a remediation to conform, and on what must a reviewer's verdict on it rest? +- 2d. Where more than one kind of detector could host the rule, which kind should the + practitioner write it in, and what must they do if they write it in another? ## 3. Application From ad97dfaff012079ea282188de7a90a96c5433d86 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 5 Oct 2026 12:18:40 +0000 Subject: [PATCH 005/105] Acceptance process: apply the owner's decisions on review 1 - Process changes are recorded in acceptance/CHANGES.md; the fourth section leaves CHANGELOG.md, which is again for the three specifications only, so tools/versions.py no longer reads a phantom toolchain Unreleased entry. - The isolation rule and the kept answer sheets apply from run 019; runs 16 to 18 stand as recorded. - A delegated marker's verdict is final unless the editor shows a rule was misapplied, then re-marked with the reason recorded; any open judgement takes the stricter reading. - The claude -p example carries the flags that isolate a reader: --tools "", --setting-sources "", --strict-mcp-config, --no-session-persistence, from an empty directory, packet inline. - A cohort is an odd number of readers, at least five; the editor may choose a larger one for a more load-bearing change. Fewer than five counting readers voids the run. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 47 +++++++++++++++++++++++++++++-------------- CHANGELOG.md | 28 -------------------------- CONTRIBUTING.md | 8 +++++--- acceptance/CHANGES.md | 35 ++++++++++++++++++++++++++++++++ acceptance/README.md | 12 ++++++----- 5 files changed, 79 insertions(+), 51 deletions(-) create mode 100644 acceptance/CHANGES.md diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index e0ba89a..548d732 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -8,22 +8,36 @@ test is run with a low-strength model so that clarity is measured at the floor, ## The cohort -- **At least five independent readers**, each a fresh instance of a low-strength model (the - smallest current model in its family, for example Haiku), with no shared context and no - access to the change history, the critique findings or each other. +- **An odd number of independent readers, at least five**, each a fresh instance of a + low-strength model (the smallest current model in its family, for example Haiku), with no + shared context and no access to the change history, the critique findings or each other. The + editor chooses the size, and may choose a larger cohort the more load-bearing the change is. + - **Each reader is a separate process with no tools and no context but its packet**, started - outside any repository so that no project instruction or settings file reaches it: for - example `claude -p` with every tool disabled, run from an empty directory. A reader that - shares the editor's context, such as an agent dispatched from the editor's own session, is not - a cold reader, and its answers do not count towards a run. + from an empty directory outside any repository, with no settings files, no MCP servers and no + saved session, so that no project instruction or settings file reaches it. For example, with + the packet given inline as the prompt: + + ``` + claude -p --model haiku --tools "" --setting-sources "" --strict-mcp-config --no-session-persistence "" + ``` + + A reader that shares the editor's context, such as an agent dispatched from the editor's own + session, is not a cold reader, and its answers do not count towards a run. If fewer than five + readers' answers count, the run is void and a fresh cohort runs. + - Each reader receives **only its packet**: the changed document, the document's question set and the fixture the questions name, each in full in the reader's prompt and headed by its file name. The question set carries the instruction to the reader; no preamble, wrapper or other text is added beyond the file names. Every reader in a run receives the same packet. Nothing else is read first. + - Readers are run **after** every stronger review has been applied. They are the last gate, not the first draft. +The isolation rule above and the answer sheets kept under [The record](#the-record) apply to runs +from 019 onwards; runs 16 to 18 stand as recorded. + ## The question set Written before the run, with an **answer key** for every keyed question drawn verbatim from the @@ -79,8 +93,9 @@ read, its question set, the fixture the questions name and its key; this file an shape; the answer sheets; and, from the editor, the facts the record states that the sheets do not, which are the pull request and the commit, the cohort size and model, the command the readers were started with and the run number. The record states those facts as the editor -supplied them. A delegated marker's verdict stands. Each sheet is marked question by question, -as follows. +supplied them. A delegated marker's verdict is final unless the editor shows that a rule in this +file was misapplied; the editor then re-marks what the misapplied rule affected, and the record +says which rule and why. Each sheet is marked question by question, as follows. - **Quoting.** A reader quotes the document when it gives words of the document verbatim, a whole sentence or part of one. Naming a clause number, or paraphrasing, is not quoting. @@ -103,8 +118,9 @@ as follows. common. Passages with no sentence in common are different, even within one paragraph or clause. -Where a mark needs a judgement these rules do not settle, the marker makes it, and the record -states the judgement and why. +Where a mark needs a judgement these rules do not settle, the marker makes it, taking the stricter +reading, the one under which the change is less likely to pass, and the record states the +judgement and why. ## Pass criteria @@ -179,7 +195,8 @@ process files, and a change to any part of them is a change to this process. The specifications, and a change to them is not tested by a cohort: the cohort measures whether a low-strength reader can follow a specification, and the process is followed by the editor. Such a change names the runs or the case that showed the need for it, has the same independent review -as any other, and is recorded by a line under `Unreleased` in the changelog's entry for the -acceptance process. It is bound by the section on the bar: it may raise a criterion, never relax -one. A change to a question set, key or fixture is not a change to this process; -`acceptance/README.md` governs those. +as any other, and is recorded by a line under `Unreleased` in +[acceptance/CHANGES.md](acceptance/CHANGES.md), not in the changelog, which is for the three +specifications. It is bound by the section on the bar: it may raise a criterion, never relax one. +A change to a question set, key or fixture is not a change to this process; `acceptance/README.md` +governs those. diff --git a/CHANGELOG.md b/CHANGELOG.md index 7df15d9..f5f8122 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -133,31 +133,3 @@ First published version, the draft label dropped on the same day as the method s custom rules, a single-rule harness, identifier resolution, a derived listing of active defences, the project record, self-application, and the conformance definition in section 11. Published before the acceptance test in `ACCEPTANCE.md` existed; 0.2.0 is the first version to pass it. - -## Acceptance process - -Not a specification and not versioned. Changes to the process files `ACCEPTANCE.md` names are -recorded here as it describes, and need no cohort. - -### Unreleased - -- The cohort can be run and marked without judgement calls that the rules leave open. Each - reader is a separate process with no tools and no context but its packet, and the packet is - the document, its question set and the fixture the questions name, with nothing added around - them. A new marking section says what a delegated marker receives and that its verdict stands, - what counts as quoting, what a correct load-bearing answer needs, which misses are traceable - to the text and how the key's statements bear on that, how unanswered items count, and when - two confusing passages are the same. Thresholds read "every reader" and "all but one", the - same as five of five and four of five for a cohort of five; every pass-criteria row must hold, - and the last covers only what the others do not. The run record states how readers were - isolated and who marked, and the answer sheets are kept beside it. `CONTRIBUTING.md` requires - a reviewer who did not draft the change, says which review findings mean a fresh run, adds - the agent-facing rebuild after an Appendix A change, and points to how a change to the process - files is made. Prompted by gaps in the text as it stood: `ACCEPTANCE.md` named no fixture in - what a reader receives whilst `acceptance/README.md` did, the thresholds were written as five - of five and four of five for a cohort of at least five, two pass-criteria rows both covered a - single reader's traceable miss, and nothing settled an unanswered question or "the same - passage". [Run 16](acceptance/runs/016-deferral-clause-run-1.md), - [run 17](acceptance/runs/017-deferral-clause-run-2.md) and - [run 18](acceptance/runs/018-deferral-clause-run-3.md) ruled on these case by case, and their - rulings are written in. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a555d7f..b7d8456 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,8 +29,9 @@ review. It is a patch version when released. SHOULD or MAY in a numbered section or appendix. It needs everything a clarity edit needs, plus the five tests in SCOPE.md answered in the pull request, and it is at least a minor version when released. `tools/changes.py` tells the two apart mechanically and CI runs it on every pull -request. A change to the process files ACCEPTANCE.md names, this file among them, is neither, -and ACCEPTANCE.md says how it is made. +request. A change to the process files ACCEPTANCE.md names, this file among them, is neither: +ACCEPTANCE.md says how it is made, and it is recorded in `acceptance/CHANGES.md`, not in +CHANGELOG.md. ## What a pull request contains @@ -75,7 +76,8 @@ survives, the change is yours. Steps 2 to 5 are the editor's, not CI's, and they happen in batches. The editor may have an agent draft or review, and may delegate marking as ACCEPTANCE.md allows, in which case the -marker's verdict stands; every other decision in these steps stays the editor's. A quiet pull +marker's verdict is final unless the editor shows a rule was misapplied, as ACCEPTANCE.md says; +every other decision in these steps stays the editor's. A quiet pull request is not a declined one; a declined one says so. ## Versions and releases diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md new file mode 100644 index 0000000..af466a9 --- /dev/null +++ b/acceptance/CHANGES.md @@ -0,0 +1,35 @@ +# Changes to the acceptance process + +Changes to the process files that [ACCEPTANCE.md](../ACCEPTANCE.md) names, recorded here as it +describes. The process is not a specification and is not versioned, and a change to it is not +tested by a cohort. Changes to the three specifications are recorded in +[CHANGELOG.md](../CHANGELOG.md). + +## Unreleased + +- The cohort can be run and marked without judgement calls that the rules leave open. A cohort + is an odd number of readers, at least five, and the editor may choose a larger one for a more + load-bearing change. Each reader is a separate process with no tools, no settings files, no + MCP servers and no context but its packet, started from an empty directory outside any + repository, and the example command shows the flags that do this; a reader sharing the + editor's context does not count, and a run left with fewer than five counting readers is void. + The packet is the document, its question set and the fixture the questions name, with nothing + added around them. A new marking section says what a delegated marker receives, that its + verdict is final unless the editor shows a rule was misapplied, that the marker takes the + stricter reading wherever it makes an open judgement, what counts as quoting, what a correct + load-bearing answer needs, which misses are traceable to the text and how the key's statements + bear on that, how unanswered items count, and when two confusing passages are the same. + Thresholds read "every reader" and "all but one", the same as five of five and four of five for + a cohort of five; every pass-criteria row must hold, and the last covers only what the others + do not. The run record states how readers were isolated, the command they were started with + and who marked, and the answer sheets are kept beside it; the isolation rule and the kept + sheets apply from run 019, and runs 16 to 18 stand as recorded. Process changes are recorded + in this file, not in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the + change, says which review findings mean a fresh run, adds the agent-facing rebuild after an + Appendix A change, and points to how a change to the process files is made. Prompted by gaps in + the text as it stood: `ACCEPTANCE.md` named no fixture in what a reader receives whilst + `acceptance/README.md` did, the thresholds were written as five of five and four of five for a + cohort of at least five, two pass-criteria rows both covered a single reader's traceable miss, + and nothing settled an unanswered question or "the same passage". [Run + 16](runs/016-deferral-clause-run-1.md), [run 17](runs/017-deferral-clause-run-2.md) and + [run 18](runs/018-deferral-clause-run-3.md) ruled on these case by case. diff --git a/acceptance/README.md b/acceptance/README.md index c7e8d04..b5ae59d 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -2,8 +2,9 @@ The question sets and answer keys that [ACCEPTANCE.md](../ACCEPTANCE.md) requires. One directory per specification document, each holding the fixed questions a cold reader answers and the key an -editor marks against. The fixture cases the application question uses live in `fixtures/`, and -the record of every run lives in `runs/`. +editor marks against. The fixture cases the application question uses live in `fixtures/`, the +record of every run lives in `runs/`, and changes to the acceptance process itself are recorded +in [CHANGES.md](CHANGES.md). | Document | Questions | Key | | ------------------ | ------------------------------------------------ | ------------------------------------ | @@ -14,9 +15,10 @@ the record of every run lives in `runs/`. ## How a run works 1. The editor identifies which documents the change touched. Only those are tested. -2. For each, five or more fresh low-strength readers, each isolated as `ACCEPTANCE.md` describes, - each receive the document's current text, its `QUESTIONS.md`, and the fixture the questions - name. Nothing else: no key, no changelog, no critique findings, no other reader's answers. +2. For each, an odd number of fresh low-strength readers, at least five, each isolated as + `ACCEPTANCE.md` describes, each receive the document's current text, its `QUESTIONS.md`, and + the fixture the questions name. Nothing else: no key, no changelog, no critique findings, no + other reader's answers. 3. The editor, or a separate marker the editor delegates to, marks each answer sheet against `KEY.md` under the marking rules in `ACCEPTANCE.md` and applies its pass criteria. 4. The result is recorded as a file under [runs/](runs/), stating what `ACCEPTANCE.md` lists, From 02aefca20109a60af990e542cb605797c79fd0b4 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 5 Oct 2026 16:13:34 +0000 Subject: [PATCH 006/105] Method 3.2: the strongest kind of detector is a MUST; textual replaces lexical The owner decided: a rule the practitioner writes MUST be written in the strongest kind of detector that can express it, semantic, else syntactic, else textual. A weaker kind is allowed only where the stronger cannot express the rule or the language has no detector of that kind that accepts bespoke rules, and the reason is recorded on the rule's remediation docs and in the description of the commit that adds or moves the rule. The practitioner judges it; a project MAY require the owner's sign-off, which the section 4 line on choosing a detector now notes. Lexical becomes textual, each kind defined by what it can see. Rust examples checked against current sources: Clippy's adding-lints guide is upstream contribution only, so it is semantic but takes no bespoke rules; Dylint runs lints from user-specified dynamic libraries written as LateLintPass, which runs after type checking, so it is the Rust semantic host. Detector section 8: the verdict names the kind by the most a rule in it can see in any configuration it documents. Appendix A and the project prompt rebuilt. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 31 ++++++++------- DETECTOR-SPEC.md | 35 +++++++++-------- SPEC.md | 59 ++++++++++++++++------------ defence-before-fix-project-prompt.md | 10 +++-- 4 files changed, 74 insertions(+), 61 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cf2a935..4d98982 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,16 +12,18 @@ version entry with no cohort line has not been accepted and is not published. ### Unreleased -- New obligation, clause [3.2](SPEC.md#32-build-the-net): where the language has a semantic detector, one that reads types - and symbols, that accepts bespoke rules, the practitioner SHOULD host the rule there, and a - syntactic one before a lexical one where it has none; a rule placed lower MUST have the reason - recorded with its remediation docs. Section [1](SPEC.md#1-terminology) defines lexical, syntactic and semantic, and the - section [3](SPEC.md#3-the-method) map and Appendix A carry the preference. Prompted by project rules hosted in - ast-grep, where the language already had a semantic host, that decided by names: they reported - unrelated calls of the same name and missed calls made through an alias, and clause [3.2](SPEC.md#32-build-the-net) said - any detector qualified whatever its category. The method question set gains load-bearing - statement 2d, which `ACCEPTANCE.md` now lists. Accepted by run NNN. Minor version when - released. +- New obligation, clause [3.2](SPEC.md#32-build-the-net): a rule the practitioner writes MUST be written in the strongest + kind of detector that can express it, semantic, else syntactic, else textual; a weaker kind is + allowed only where the stronger cannot express the rule or the language has no detector of + that kind that accepts bespoke rules, and the reason MUST be recorded on the rule's remediation + docs and in the description of the commit that adds or moves the rule. The practitioner judges it; + a project MAY require the owner's sign-off, which section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) notes. Section [1](SPEC.md#1-terminology) defines textual, + syntactic and semantic by what each can see, and the section [3](SPEC.md#3-the-method) map and Appendix A carry + the requirement. Prompted by project rules hosted in ast-grep, where the language already had a + semantic host, that decided by names: they reported unrelated calls of the same name and + missed calls made through an alias, and clause [3.2](SPEC.md#32-build-the-net) said any detector qualified whatever its + category. The method question set gains load-bearing statement 2d, which `ACCEPTANCE.md` now + lists. Accepted by run NNN. Minor version when released. ### 1.1.0, 2026-10-02 @@ -84,10 +86,11 @@ pass it. ### Unreleased - New obligation, section [8](DETECTOR-SPEC.md#8-conformance): a verdict on a detector MUST say whether it is semantic, syntactic - or lexical, by the most it can give a bespoke rule to read. It is not a level of conformance and - does not change the verdict; it is stated because method clause [3.2](SPEC.md#32-build-the-net) now prefers a semantic - host, and a verdict of conforming alone presents a lexical detector as an equal host. Section [2](DETECTOR-SPEC.md#2-terminology)'s - gloss carries the three terms. Accepted by run NNN. Minor version when released. + or textual, by the most a rule in it can see in any configuration it documents. It is not a + level of conformance and does not change the verdict; it is stated because method clause [3.2](SPEC.md#32-build-the-net) + now requires a rule to be written in the strongest kind of detector that can express it, and a + verdict of conforming alone presents a textual detector as an equal host. Section [2](DETECTOR-SPEC.md#2-terminology)'s gloss + carries the three terms. Accepted by run NNN. Minor version when released. ### 1.0.0, 2026-09-08 diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index 7c46f3d..fb6bbcf 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -35,16 +35,16 @@ construct the means has moved the obligation rather than met it. Terms from the method specification carry over unchanged. Every capitalised term links to its definition; the ones this document leans on most are, in short: -| Term | In one line | -| ---------------------------------- | ------------------------------------------------------------------------------- | -| [Detector] | A tool that reads code without executing it and reports what it finds | -| [Semantic], [Syntactic], [Lexical] | What a [Detector] reads: types and symbols, a parse tree without types, or text | -| [Rule] | One check a [Detector] evaluates | -| [Identifier] | The stable name printed with a finding that leads to its documentation | -| [Practitioner] | Whoever is doing the work, a person or an [Agent] | -| [Toolchain] | Everything a project assembles to run its checks through, [Detectors] included | -| [Suppression] | Making a finding go away without fixing it | -| [Baseline] | A recorded set of existing findings a [Rule] is told to ignore | +| Term | In one line | +| ---------------------------------- | -------------------------------------------------------------------------------------------- | +| [Detector] | A tool that reads code without executing it and reports what it finds | +| [Semantic], [Syntactic], [Textual] | What a [Detector] can see: types, symbols and data flow; a parse tree without types; or text | +| [Rule] | One check a [Detector] evaluates | +| [Identifier] | The stable name printed with a finding that leads to its documentation | +| [Practitioner] | Whoever is doing the work, a person or an [Agent] | +| [Toolchain] | Everything a project assembles to run its checks through, [Detectors] included | +| [Suppression] | Making a finding go away without fixing it | +| [Baseline] | A recorded set of existing findings a [Rule] is told to ignore | These are additional. @@ -307,12 +307,13 @@ summary for an [Agent]'s context, are obligations on a project's assembled [Tool and are stated in the [toolchain specification](TOOLING-SPEC.md), so there is no separate [Agent]-support level for a [Detector]. -**A verdict on a [Detector] MUST also say whether it is [Semantic], [Syntactic] or [Lexical]**, -by the most it can give a bespoke [Rule] to read. That is not a level of [Conformance] and does -not change the verdict: a [Lexical] [Detector] can [Conform], and a [Semantic] one can fail to. It -is stated because the method specification's clause 3.2 asks a [Practitioner] to prefer a -[Semantic] [Detector] where the language has one that accepts bespoke [Rules], and a verdict that -said only [Conforming] would present a [Lexical] [Detector] as an equal host to a [Semantic] one. +**A verdict on a [Detector] MUST also say whether it is [Semantic], [Syntactic] or [Textual]**, +by the most a [Rule] in it can see in any configuration it documents. That is not a level of +[Conformance] and does not change the verdict: a [Textual] [Detector] can [Conform], and a +[Semantic] one can fail to. It is stated because the method specification's clause 3.2 requires a +[Practitioner] to write a [Rule] in the strongest kind of [Detector] that can express it, and a +verdict that said only [Conforming] would present a [Textual] [Detector] as an equal host to a +[Semantic] one. ### 8.1 A maintainer MAY declare the version of this document the detector conforms to, and the declaration records known gaps @@ -397,7 +398,6 @@ does not have to. [hazards]: SPEC.md#hazard [identifier]: SPEC.md#identifier [identifiers]: SPEC.md#identifier -[lexical]: SPEC.md#lexical [message]: SPEC.md#message [owner]: SPEC.md#owner [practitioner]: SPEC.md#practitioner @@ -408,4 +408,5 @@ does not have to. [semantic]: SPEC.md#semantic [suppression]: SPEC.md#suppression [syntactic]: SPEC.md#syntactic +[textual]: SPEC.md#textual [toolchain]: SPEC.md#toolchain diff --git a/SPEC.md b/SPEC.md index e27c911..05b5aaf 100644 --- a/SPEC.md +++ b/SPEC.md @@ -70,23 +70,23 @@ The harm a [Class] causes, which is **not necessarily a failure**. It may be a f #### Detector -A tool that reads code without executing it and reports occurrences of a pattern. A [Detector] is not a [Runner]. By what it reads, a [Detector] is [Lexical], [Syntactic] or [Semantic], and clause 3.2 says which to prefer. What a [Detector] must offer so that a [Practitioner] can write, prove, run and resolve a [Rule] in it is specified in the [detector specification](DETECTOR-SPEC.md). +A tool that reads code without executing it and reports occurrences of a pattern. A [Detector] is not a [Runner]. By what it can see, a [Detector] is [Textual], [Syntactic] or [Semantic], and clause 3.2 says which kind a [Rule] is written in. What a [Detector] must offer so that a [Practitioner] can write, prove, run and resolve a [Rule] in it is specified in the [detector specification](DETECTOR-SPEC.md). #### Runner A tool that **executes** code and reports what happened: a test [Runner], a compilation, a benchmark, a smoke check. A [Runner] answers questions about behaviour at one moment, on the paths it happened to exercise; a [Detector] answers questions about the text of the code, everywhere it exists. Detection under this method MUST come from a [Detector], for that reason. -#### Lexical +#### Textual -Of a [Detector]: reading code as text, as grep does. It sees how the code is spelt and nothing of its structure, so it cannot tell a call from a comment that names it. +Of a [Detector]: seeing the code only as text, as grep does. It sees how the code is spelt and nothing of its structure, so it cannot tell a call from a comment that names it. #### Syntactic -Of a [Detector]: reading a parse tree of the code without types, as ast-grep and Semgrep do. It sees the structure, but knows what a name refers to only by the name. +Of a [Detector]: seeing the parse tree of the code, without types, as ast-grep and Semgrep do. It sees the structure, but knows what a name refers to only by the name. #### Semantic -Of a [Detector]: reading types and resolved symbols, as PHPStan, mypy, Clippy and ESLint with typescript-eslint's type information do. It sees what a name refers to and what type a value has, whatever either is called. The same [Detector] can be [Syntactic] in one configuration and [Semantic] in another: ESLint without type information is [Syntactic]. +Of a [Detector]: seeing types, resolved symbols and data flow, as PHPStan, mypy, Clippy, Dylint and ESLint with typescript-eslint's type information do. It sees what a name refers to and what type a value has, whatever either is called. The same [Detector] can be [Syntactic] in one configuration and [Semantic] in another: ESLint without type information is [Syntactic]. A [Semantic] [Detector] need not accept bespoke [Rules]: Clippy does not, and in Rust a project's own [Semantic] [Rules] are written for Dylint. #### Rule @@ -230,7 +230,7 @@ Six clauses, in order. In brief, before the detail: | Clause | The [Practitioner] MUST | The record shows | Goes to the [Owner] under section 4 | | ------ | ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | 3.1 | Name the [Class] the [Defect] belongs to, bounded both ways, after an independent search | The [Class], the [Hazard] sentence, the two search techniques, the next wider [Rule] not built, whether a [Runner] check pins the reported behaviour | Whether the next wider [Rule] is built, and whether the [Rule] stays narrower than the search showed | -| 3.2 | Express the [Class] as a [Rule] in a [Detector], never a test | The [Rule], why it sits below a [Semantic] [Detector] where the language has one, and any [Toolchain] gap that stopped a bespoke one | Whether a [Class] that could be defended is left undefended | +| 3.2 | Express the [Class] as a [Rule] in the strongest kind of [Detector] that can express it, never a test | The [Rule], why it is in a weaker kind than [Semantic] where it is, and any [Toolchain] gap that stopped a bespoke one | Whether a [Class] that could be defended is left undefended | | 3.3 | Make the [Rule] fire, on the [Instance] or a [Fixture], in a commit of its own | The red run, and the sentence behind every [Narrowing] | Any exclusion whose sentence cannot be written: that is [Suppression] | | 3.4 | [Sweep] the whole codebase, record the count, then fix every [Instance] | The count, corroborated, and what was fixed by hand or by pattern | Any [Instance] left unfixed, and any [Baseline] | | 3.5 | Make the [Rule] permanent and [Blocking] in the project's own checks | The green run through the project's entry point, and the recorded decision behind any [Suppression] | Removing or disabling the [Rule], and any [Suppression] added to it | @@ -408,20 +408,24 @@ Any tool that reads code and reports pattern matches qualifies, whatever its cat walkers, architectural fitness checks, and [Detectors] that inspect a change before it lands. These are examples rather than a permitted list, and a [Detector] is not disqualified for being unfamiliar. -**Qualifying is not the same as being the right host.** Where the language has a [Semantic] -[Detector] that accepts bespoke [Rules], the [Practitioner] SHOULD express the [Rule] in it; where -it has none, a [Syntactic] one is preferred to a [Lexical] one. The language has one when the -project already runs it or the language's own [Detector] ecosystem offers it, the same check -section 2 asks for before a [Toolchain] gap is recorded. Where the [Rule] is placed lower than -that, the [Practitioner] MUST record why with its [Remediation docs], in a sentence an [Owner] can -check: -that the [Class] is defined by spelling alone, as a forbidden comment is, so a [Lexical] [Rule] -sees every [Instance] of it; or that the project runs no [Semantic] [Detector], and why adopting -one was not practical. A [Detector] that cannot see types decides by names and shapes. It reports -every call named `context`, whatever it is called on, which is a [False positive] clause 3.1 does -not tolerate, and it misses the same call made through an alias, which is an [Instance] the -[Rule] never sees; restricting it by name trades the first failure for the second. A [Semantic] -[Detector] decides by what the code is, so the bounds of clause 3.1 can be held there. +**Qualifying is not the same as being the right host.** A [Rule] the [Practitioner] writes MUST +be written in the strongest kind of [Detector] that can express it: a [Semantic] one, else a +[Syntactic] one, else a [Textual] one. A weaker kind is allowed only where the stronger kind +cannot express the [Rule], as where the [Rule] reads a file the [Semantic] [Detector] does not +parse, or where the language has no [Detector] of the stronger kind that accepts bespoke [Rules]. +The language has one when the project already runs it or the language's own [Detector] ecosystem +offers it, the same check section 2 asks for before a [Toolchain] gap is recorded. Where the +[Rule] is written in a weaker kind than [Semantic], the [Practitioner] MUST record which of those +two reasons holds, and why, on the [Rule]'s [Remediation docs] and in the description of the +commit that adds the [Rule] or moves it to another [Detector]. The [Practitioner] judges whether +a reason holds, as section 4 gives them the choice of [Detector]; a project MAY require its +[Owner]'s sign-off on a weaker kind as well, as a [Calibration] of how strict it is. + +A [Detector] that cannot see types decides by names and shapes. It reports every call named +`context`, whatever it is called on, which is a [False positive] clause 3.1 does not tolerate, and +it misses the same call made through an alias, which is an [Instance] the [Rule] never sees; +restricting it by name trades the first failure for the second. A [Semantic] [Detector] decides by +what the code is, so the bounds of clause 3.1 can be held there. Where the affected language has no extensible [Detector] available, the [Practitioner] MAY build a bespoke one: a program of the project's own that reads code and reports matches of the [Class], run @@ -794,7 +798,8 @@ they do not. **Decisions the [Practitioner] takes alone**, without seeking approval: - What [Class] the [Defect] belongs to, and where its bounds sit (clause 3.1). -- Which [Detector] to use and how to express the [Rule] in it (clause 3.2). +- Which [Detector] to use and how to express the [Rule] in it (clause 3.2), unless the project + requires the [Owner]'s sign-off on a weaker kind of [Detector], as clause 3.2 allows. - How to prove the [Rule], and what [Fixture] to prove it against (clause 3.3). - What the correct behaviour is at each [Instance], and therefore what each fix should be (clause 3.4). @@ -1078,10 +1083,12 @@ This appendix restates sections 2 to 4; where the two differ, the sections gover > for it, say so and fix the [Defect] conventionally. > > Write a custom [Rule] that detects the [Class], in a tool that reads code rather than running it. -> Put it in a [Semantic] [Detector], one that sees types and symbols, where the language has one -> that takes custom [Rules]; if you put it in one that sees only syntax or text, write down why in -> the [Rule]'s documentation. Draw the [Rule] so that it catches more than the single [Instance] -> you started from, but never so broadly that it matches code which does not carry the [Hazard]. +> Put it in a [Semantic] [Detector], one that sees types and symbols; failing that, one that sees +> syntax; and only then one that sees text. Use a weaker one only where the stronger one cannot +> express the [Rule], or the language has none that takes custom [Rules], and write down which, +> and why, in the [Rule]'s documentation and in the description of the commit that adds it. Draw +> the [Rule] so that it catches more than the single [Instance] you started from, but never so +> broadly that it matches code which does not carry the [Hazard]. > > Do not trust the [Rule] as your only way of finding [Instances]. Search independently as well, by > text search and by reading the code, and check the [Rule] catches what you found by hand. Make that @@ -1176,7 +1183,6 @@ This appendix restates sections 2 to 4; where the two differ, the sections gover [identifiers]: #identifier [instance]: #instance [instances]: #instance -[lexical]: #lexical [message]: #message [narrowing]: #narrowing [owner]: #owner @@ -1192,5 +1198,6 @@ This appendix restates sections 2 to 4; where the two differ, the sections gover [suppressions]: #suppression [sweep]: #sweep [syntactic]: #syntactic +[textual]: #textual [toolchain]: #toolchain [warning]: #warning diff --git a/defence-before-fix-project-prompt.md b/defence-before-fix-project-prompt.md index a30009d..ec8d353 100644 --- a/defence-before-fix-project-prompt.md +++ b/defence-before-fix-project-prompt.md @@ -13,10 +13,12 @@ it. Do not decide in advance whether that is possible, attempt it. If you cannot for it, say so and fix the Defect conventionally. Write a custom Rule that detects the Class, in a tool that reads code rather than running it. -Put it in a Semantic Detector, one that sees types and symbols, where the language has one -that takes custom Rules; if you put it in one that sees only syntax or text, write down why in -the Rule's documentation. Draw the Rule so that it catches more than the single Instance -you started from, but never so broadly that it matches code which does not carry the Hazard. +Put it in a Semantic Detector, one that sees types and symbols; failing that, one that sees +syntax; and only then one that sees text. Use a weaker one only where the stronger one cannot +express the Rule, or the language has none that takes custom Rules, and write down which, +and why, in the Rule's documentation and in the description of the commit that adds it. Draw +the Rule so that it catches more than the single Instance you started from, but never so +broadly that it matches code which does not carry the Hazard. Do not trust the Rule as your only way of finding Instances. Search independently as well, by text search and by reading the code, and check the Rule catches what you found by hand. Make that From dd1677909b367fb7df070e7760fcc001a0dd2778 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 5 Oct 2026 16:13:34 +0000 Subject: [PATCH 007/105] Acceptance keys: 2d and the restatements follow the MUST and textual Method 2d quotes the two new MUST sentences of clause 3.2; an answer that calls the semantic kind only a preference, or makes the owner's approval always needed, is wrong. The ledger fixture's 3.2 verdict stays Yes, since the fixture does not say what its analyser can see. Detector key: lexical becomes textual. Co-Authored-By: Claude Opus 5.5 --- acceptance/detector/KEY.md | 4 ++-- acceptance/method/KEY.md | 35 ++++++++++++++++++++--------------- 2 files changed, 22 insertions(+), 17 deletions(-) diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 9466b0b..e9724a6 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -18,7 +18,7 @@ bullet asserting an obligation the document does not state, is a correct restate offline, with the documentation shipped and versioned with the rule ([6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) to [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together)). - Make any inline suppression route detectable or disableable ([7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)). -A bullet saying that a verdict on a detector names whether it is semantic, syntactic or lexical +A bullet saying that a verdict on a detector names whether it is semantic, syntactic or textual (section [8](../../DETECTOR-SPEC.md#8-conformance)) is correct too, though it is an obligation on whoever gives the verdict. Wrong if it asserts: the detector must be semantic to conform; the detector must enforce a @@ -67,7 +67,7 @@ described as conformance. A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a- has read the clause differently and that counts against the document; a reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Yes because they missed the last paragraph of the fixture has misread the fixture, not the document. -The fixture does not say what Lintomatic reads. A reader who names its kind as unknown, or names +The fixture does not say what Lintomatic can see. A reader who names its kind as unknown, or names none, is not marked on it; a reader who makes the verdict turn on its kind has read section [8](../../DETECTOR-SPEC.md#8-conformance) differently, which counts against the document: the kind "is not a level of Conformance and does not change the verdict". diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 74618d9..2648051 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -11,8 +11,8 @@ bullet asserting an obligation the document does not state, is a correct restate - Attribute the defect to a class, bounded on both sides, with the hazard stated ([3.1](../../SPEC.md#31-attribute-the-defect-to-a-class)). - Express the class as a rule in a detector that reads code, never a test ([3.2](../../SPEC.md#32-build-the-net)). -- Host the rule in a semantic detector where the language has one that takes bespoke rules, - else a syntactic one before a lexical one, and record why where it sits lower ([3.2](../../SPEC.md#32-build-the-net)). +- Write the rule in the strongest kind of detector that can express it, semantic, else + syntactic, else textual, and record why where it is in a weaker kind ([3.2](../../SPEC.md#32-build-the-net)). - Prove the rule fires, against the instance or a fixture, in a commit of its own ([3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire)). - Sweep the whole codebase, record the count, then fix every instance ([3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance)). - Make the rule permanent and blocking in the project's checks ([3.5](../../SPEC.md#35-enforce-permanently-and-block)). @@ -45,23 +45,28 @@ detector may host a rule. rest on reproduction, not on the report". An answer that accepts the pull request description as the verdict is wrong, and so is one that states the six clauses and says nothing of reproduction. -- **2d.** A semantic detector, one that reads types and symbols, where the language has one that - accepts bespoke rules; failing that a syntactic one before a lexical one; and where the rule - sits lower than the language allows, the practitioner records why with the rule's remediation - docs. Key quotations, [3.2](../../SPEC.md#32-build-the-net): "Where the language has a Semantic Detector that accepts bespoke - Rules, the Practitioner SHOULD express the Rule in it; where it has none, a Syntactic one is - preferred to a Lexical one."; "Where the Rule is placed lower than that, the Practitioner MUST - record why with its Remediation docs". An answer that says any detector is as good as another, - that a lower one is forbidden, that the owner must approve it (section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) leaves the choice of - detector to the practitioner), or that names the preference and says nothing of the recorded - reason, is wrong. +- **2d.** The strongest kind that can express it: a semantic detector, one that sees types and + symbols, else a syntactic one, else a textual one; a weaker kind only where the stronger cannot + express the rule or the language has none of that kind that accepts bespoke rules, and then the + practitioner records which reason holds, and why, on the rule's remediation docs and in the + description of the commit that adds or moves the rule. Key quotations, [3.2](../../SPEC.md#32-build-the-net): "A Rule the + Practitioner writes MUST be written in the strongest kind of Detector that can express it: a + Semantic one, else a Syntactic one, else a Textual one."; "Where the Rule is written in a weaker + kind than Semantic, the Practitioner MUST record which of those two reasons holds, and why, on + the Rule's Remediation docs and in the description of the commit that adds the Rule or moves it + to another Detector." An answer naming the recorded reason in one of the two places is correct. + An answer that says any detector is as good as another, that the semantic kind is only + preferred and may be passed over for another reason, that a weaker kind is never allowed, that + the owner must always approve a weaker kind (the practitioner judges it unless the project + requires the owner's sign-off), or that names the kind and says nothing of the recorded reason, + is wrong. ## 3. Application: the ledger remediation | Clause | Intended verdict | Reasoning the fixture supports | | ----------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | Class, hazard and the next wider rule are named; Partial or No is accepted where the reader notes that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | -| [3.2](../../SPEC.md#32-build-the-net) | Yes | A rule in the project's static analyser, reading code, not a test; the fixture does not say what the analyser reads, so nothing shows the rule sits lower than the language allows | +| [3.2](../../SPEC.md#32-build-the-net) | Yes | A rule in the project's static analyser, reading code, not a test; the fixture does not say what the analyser can see, so nothing shows the rule is in a weaker kind than semantic | | [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | Commit B is red on the fixture and the originating instance and survives as its own commit | | [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Five findings recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority; Partial accepted where the reasoning names that suppression | | [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | The rule is permanent and blocking, but the suppression was not a recorded decision; [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | @@ -82,8 +87,8 @@ sentence on unrecorded suppression, and that counts against the document. A read [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) Yes because the URL resolves has read the versioning sentence of [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) differently, which counts against the document; a reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the fixture. A reader who grades [3.2](../../SPEC.md#32-build-the-net) Partial or No because no reason for the detector's kind -is recorded has assumed a lower kind of detector than the fixture shows: the record is owed only -where the rule sits lower than the language allows. That counts against the document only if the +is recorded has assumed a weaker kind of detector than the fixture shows: the reason is owed only +where the rule is in a weaker kind than semantic. That counts against the document only if the reasoning quotes [3.2](../../SPEC.md#32-build-the-net) as requiring the reason whatever the detector. A reader who attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as covering it, which is a fair reading and is not marked wrong. From d45df88760e4f2d9ef4bbe1666b85df4b4addc1e Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 20:34:04 +0100 Subject: [PATCH 008/105] Acceptance process: answer sheets posted on the pull request, not kept in the repository The owner chose to keep the repository from growing by five or more sheet files a run. The record links one pull request comment holding every sheet unedited and the instruction a delegated marker was given. A process change carries no run placeholder: it is recorded in acceptance/CHANGES.md and is not tested by a cohort, as ACCEPTANCE.md already says. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 13 +++++++------ acceptance/CHANGES.md | 5 +++-- acceptance/README.md | 4 ++-- acceptance/runs/README.md | 6 +++--- 4 files changed, 15 insertions(+), 13 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 548d732..307e00d 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -35,8 +35,8 @@ test is run with a low-strength model so that clarity is measured at the floor, - Readers are run **after** every stronger review has been applied. They are the last gate, not the first draft. -The isolation rule above and the answer sheets kept under [The record](#the-record) apply to runs -from 019 onwards; runs 16 to 18 stand as recorded. +The isolation rule above and the answer sheets posted under [The record](#the-record) apply to +runs from 019 onwards; runs 16 to 18 stand as recorded. ## The question set @@ -176,10 +176,11 @@ the one sequence shared by the three documents, stating: the document, the pull commit read, the cohort size and model, how the readers were isolated and the command they were started with, the run number in that pull request, who marked and, for a delegated marker, how it was started, the marks per reader per question, any judgement made in marking, the passages -quoted as confusing and by how many readers, the findings applied, and whether it passed. Each -reader's answer sheet is kept unedited beside the record, one file per reader in a directory -`NNN-/`, with the instruction a delegated marker was given, so that an owner can check the -isolation and the marks. The changelog line for the change names the file. CI refuses a +quoted as confusing and by how many readers, the findings applied, whether it passed, and a link +to the answer sheets. The answer sheets are not kept in this repository: every reader's sheet, +unedited, and the instruction a delegated marker was given are posted in one comment on the pull +request, so that an owner can check the isolation and the marks. The changelog line for the +change names the file. CI refuses a specification change whose pull request adds no run file; it checks for at least one, and a pull request that changes two documents adds a run for each. diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index af466a9..0cd5100 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -22,8 +22,9 @@ tested by a cohort. Changes to the three specifications are recorded in Thresholds read "every reader" and "all but one", the same as five of five and four of five for a cohort of five; every pass-criteria row must hold, and the last covers only what the others do not. The run record states how readers were isolated, the command they were started with - and who marked, and the answer sheets are kept beside it; the isolation rule and the kept - sheets apply from run 019, and runs 16 to 18 stand as recorded. Process changes are recorded + and who marked, and links the answer sheets, which are posted unedited on the pull request + rather than kept in the repository; the isolation rule and the posted sheets apply from run + 019, and runs 16 to 18 stand as recorded. Process changes are recorded in this file, not in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the change, says which review findings mean a fresh run, adds the agent-facing rebuild after an Appendix A change, and points to how a change to the process files is made. Prompted by gaps in diff --git a/acceptance/README.md b/acceptance/README.md index b5ae59d..bb387a5 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -22,8 +22,8 @@ in [CHANGES.md](CHANGES.md). 3. The editor, or a separate marker the editor delegates to, marks each answer sheet against `KEY.md` under the marking rules in `ACCEPTANCE.md` and applies its pass criteria. 4. The result is recorded as a file under [runs/](runs/), stating what `ACCEPTANCE.md` lists, - with the answer sheets kept beside it, and the change's `Unreleased` changelog line names - that file. + with the answer sheets posted on the pull request and linked from it, and the change's + `Unreleased` changelog line names that file. A run that fails is followed by a change to the text, never to the questions, and a fresh cohort. diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index 6f0fddc..fd7f56c 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -11,9 +11,9 @@ the command they were started with; the run number within that pull request; who a delegated marker, how it was started; the marks per reader per question against the key, with any judgement made in marking; the passages quoted as confusing and by how many readers; the findings applied to the text as a result; and the verdict against the criteria in -`ACCEPTANCE.md`. Answer sheets are not pasted into the record, and the marks are the record; the -sheets are kept unedited beside it in a directory `NNN-/`, one file per reader, with the -instruction a delegated marker was given. +`ACCEPTANCE.md`, with a link to the answer sheets. Answer sheets are not kept in this repository, +and the marks are the record; the sheets, unedited, and the instruction a delegated marker was +given are posted in one comment on the pull request, which the record links. The `Unreleased` changelog line for the change names the record, and `tools/changes.py` refuses a specification change whose pull request adds no file here. From 6cc2fb69b1d90ee0a9c51822847e6414cfa8ec49 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 20:47:13 +0100 Subject: [PATCH 009/105] Acceptance process: apply the editor's rulings on review 2 Defines a wording change once, keyed items, quoting, the same passage and cold readers; makes the reader packet, isolation command and set-aside and void-run rules exact; has the marker write the record; posts the exact prompt and the marker's instruction with the sheets; restores the paraphrase rule for traceable misses; removes the keys' limits on which misses count against the document; adds the enforcing checks to the process files and says how a process change moves through the steps. acceptance/CHANGES.md loses its Unreleased heading and names the pull request. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 189 +++++++++++++++++++++--------------- CONTRIBUTING.md | 16 +-- acceptance/CHANGES.md | 65 +++++++------ acceptance/README.md | 8 +- acceptance/detector/KEY.md | 3 +- acceptance/method/KEY.md | 6 +- acceptance/runs/README.md | 18 ++-- acceptance/toolchain/KEY.md | 4 +- 8 files changed, 176 insertions(+), 133 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 307e00d..83afafb 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -8,35 +8,44 @@ test is run with a low-strength model so that clarity is measured at the floor, ## The cohort -- **An odd number of independent readers, at least five**, each a fresh instance of a +- **An odd number of independent readers started, at least five**, each a fresh instance of a low-strength model (the smallest current model in its family, for example Haiku), with no shared context and no access to the change history, the critique findings or each other. The - editor chooses the size, and may choose a larger cohort the more load-bearing the change is. + editor chooses the size, and may choose a larger cohort the more obligations the change + touches. -- **Each reader is a separate process with no tools and no context but its packet**, started - from an empty directory outside any repository, with no settings files, no MCP servers and no - saved session, so that no project instruction or settings file reaches it. For example, with - the packet given inline as the prompt: +- **Each reader is a separate process with no tools and no project context, only its packet**, + started from an empty directory outside any repository, with no settings files, no `CLAUDE.md`, + skills, plugins, hooks or MCP servers, and no saved session, so that no project instruction or + settings file reaches it. The model tool's own fixed system prompt is not project context. For + example, with the packet on standard input: ``` - claude -p --model haiku --tools "" --setting-sources "" --strict-mcp-config --no-session-persistence "" + claude -p --safe-mode --model haiku --tools "" --setting-sources "" --strict-mcp-config --no-session-persistence --output-format json < packet.md ``` - A reader that shares the editor's context, such as an agent dispatched from the editor's own - session, is not a cold reader, and its answers do not count towards a run. If fewer than five - readers' answers count, the run is void and a fresh cohort runs. - -- Each reader receives **only its packet**: the changed document, the document's question set - and the fixture the questions name, each in full in the reader's prompt and headed by its file - name. The question set carries the instruction to the reader; no preamble, wrapper or other - text is added beyond the file names. Every reader in a run receives the same packet. Nothing - else is read first. + The JSON output gives the model version each reader reported. A reader isolated in this way is + a cold reader. A reader whose process failed or returned nothing, or that shared the editor's + context, such as an agent dispatched from the editor's own session, is set aside and its + answers do not count; no reader is set aside for any other cause, and the record shows the + cause. If fewer than five readers' answers count, the run is void and a fresh cohort runs, + unless the readers whose answers count already fail a criterion, when the run fails. A void run + takes a run number, has a record and posted sheets like any other, and counts towards the + number of runs. + +- Each reader receives **only its packet**: the document's question set, the changed document + and the fixture the questions name, each in full in the reader's prompt, the question set + first. Each file is preceded by exactly one line, `===== FILE: =====`, where `` is + the name the question set gives the file (so `fixtures/remediation-ledger.md`, and + `QUESTIONS.md` for the question set itself). The question set carries the instruction to the + reader, and nothing else is added. Every reader in a run receives the same packet. Nothing else + is read first. - Readers are run **after** every stronger review has been applied. They are the last gate, not the first draft. -The isolation rule above and the answer sheets posted under [The record](#the-record) apply to -runs from 019 onwards; runs 16 to 18 stand as recorded. +The isolation rule above and the answer sheets posted under [The record](#the-record) apply from +run 19; runs 16 to 18 stand as recorded. ## The question set @@ -89,55 +98,65 @@ The editor marks each answer sheet against the document's key, or delegates the separate marker that has seen neither the drafting nor the review. Whoever marks is the marker in this file. A delegated marker receives only these: the document at the commit the readers read, its question set, the fixture the questions name and its key; this file and -`acceptance/runs/README.md`, with earlier run records of other pull requests for the record's -shape; the answer sheets; and, from the editor, the facts the record states that the sheets do -not, which are the pull request and the commit, the cohort size and model, the command the -readers were started with and the run number. The record states those facts as the editor -supplied them. A delegated marker's verdict is final unless the editor shows that a rule in this -file was misapplied; the editor then re-marks what the misapplied rule affected, and the record -says which rule and why. Each sheet is marked question by question, as follows. - -- **Quoting.** A reader quotes the document when it gives words of the document verbatim, a - whole sentence or part of one. Naming a clause number, or paraphrasing, is not quoting. +`acceptance/runs/README.md`, with earlier run records of other pull requests as examples of +style only; the answer sheets; and, from the editor, the facts the record states that the sheets +do not, which are the pull request and the commit, the cohort size and model, how the readers +were isolated and the command they were started with, how the marker was started, and the run +number. The marker writes the record, stating those facts as the editor supplied them, and the +editor adds the findings applied and the link to the posted sheets after marking. A delegated +marker's verdict, its marks and its PASS or FAIL, is final unless the editor shows that a rule +in this file was misapplied; the editor then re-marks what the misapplied rule affected, and the +record says which rule and why. Each sheet is marked question by question, as follows. + +- **Keyed items.** Question 1 as a whole is one keyed item, and so is each item of question 2 + and each clause verdict and each summary verdict of question 3. +- **Quoting.** A reader quotes the document when it gives five or more consecutive words of the + document, verbatim apart from markdown link syntax and emphasis markers, as a quotation, in + quotation marks or a block quote. Naming a clause number, or paraphrasing, is not quoting. - **Questions 1 and 3** are marked as the key states for each item. - **Question 2.** An answer is correct when its sentence states what the key's answer states, including anything the key says an answer is wrong to omit, asserts nothing the key says makes an answer wrong, and quotes the document in support. The quotation need not be the one the key gives. -- **A miss traceable to the text** is a miss whose reasoning quotes the document, or a miss the - key says counts against the document. A miss whose reasoning quotes the document is traceable - even where it also misreads the fixture, and even where the key says that miss is the reader's - or counts only with a particular quotation: the key can add to the misses that count against - the document, never take one away. Every other miss is the reader's. +- **A miss traceable to the text** is a miss whose reasoning quotes the document or states, in + the reader's own words, what the document says or requires, or a miss the key says counts + against the document. Such a miss is traceable even where it also misreads the fixture, and + whatever the key says of it: the key can add to the misses that count against the document, + never take one away. Only a miss whose reasoning states nothing about the document, such as a + misread fixture alone, is the reader's. Question 1 has no reasoning, so a miss there is + traceable only where the key says it counts. - **An unanswered question.** Each keyed item left unanswered is one miss, not traceable to the text, so a sheet that leaves three items of question 3 blank has three misses. Question 4 left - unanswered counts as no passage quoted. + unanswered counts as no passage quoted. A truncated sheet is marked as it stands; a reader + whose process failed or returned nothing is set aside, as [The cohort](#the-cohort) says. - **The same passage.** A quotation under question 4 covers every sentence of the document it - quotes any part of, and a passage named without a quotation, by clause number or description, - covers every sentence of what it names. Two passages are the same when they cover a sentence in - common. Passages with no sentence in common are different, even within one paragraph or - clause. + quotes any part of, in every place the quoted words appear, and a passage named without a + quotation, by clause number or description, covers every sentence of what it names. A heading, + list item or table row counts as one sentence. Two passages are the same when they cover a + sentence in common or lie in the same paragraph, and a list item or table row is its own + paragraph. Where a mark needs a judgement these rules do not settle, the marker makes it, taking the stricter -reading, the one under which the change is less likely to pass, and the record states the -judgement and why. +reading, the one that records more misses or findings, and the record states the judgement and +why. ## Pass criteria -The change passes when all of the following hold on a single run. With a cohort of five, "every -reader" is five of five and "all but one" is four of five. +The change passes when all of the following hold on a single run. The thresholds apply to the +readers whose answers count; with five such readers, "every reader" is five of five and "all but +one" is four of five. -| Criterion | Threshold | -| ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | -| Load-bearing statements (question 2) | **Every** reader correct on every one | -| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | -| A miss traceable to the text | **None.** Any such miss, as Marking defines it, is a finding, whatever the count | -| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it | -| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | +| Criterion | Threshold | +| ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | +| Load-bearing statements (question 2) | **Every** reader correct on every one | +| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | +| A miss traceable to the text | **None.** Any such miss, as [Marking](#marking) defines it, is a finding, whatever the count | +| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort on the changed text passes this row | +| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | -Every row must hold. The first four apply whatever the count, and the last covers only what none -of them catches: a miss traceable to the text blocks even when only one reader made it, and the -second row's tolerance of one reader does not excuse it. +Every row must hold, and the last covers only what none of the others catches. The first, third +and fourth rows block on what a single reader, or for the fourth two readers, did, and the second +row's tolerance of one reader never excuses a miss under the third. The asymmetry is deliberate. A load-bearing statement is the sentence the whole document exists to make, and one reader in five getting it wrong means one agent in five will act on the wrong @@ -149,8 +168,9 @@ the reader can justify from the document is always the document's fault. No criterion in this document is ever relaxed to let a change through. If the cohort is confused, the test is working and the text is wrong; the fix is to the text, never to the threshold, the -question, the fixture or the key. A threshold may be raised, a keyed question may be added, and a -key may be corrected where it misquoted the document; none of those makes a failing run pass. The +question, the fixture or the key. A threshold may be raised, a keyed question may be added, a key +may be corrected where it misquoted the document, and a key's limit on which misses count against +the document may be removed, which raises the bar; none of those makes a failing run pass. The one time a threshold was relaxed, after three runs, the relaxation was reversed on the author's instruction and is recorded here so that it is not tried again. Removing the clarity rating and the request for suggestions was not a relaxation: it changed what the cohort is asked, not how @@ -164,25 +184,36 @@ something. ## When it runs -Once per pull request, on the final text, after the editor's rewrite and any key update. It is -the last gate before the merge to `next`, so that every change on `next` is one the cohort has -accepted, and the release branch needs no cohort of its own unless it changes specification text. -A wording change after the run means a fresh run; readers are never reused. +On the final text of each pull request, after the editor's rewrite and any key update, and again +after every failure, void run or wording change. It is the last gate before the merge to `next`, +so that every change on `next` is one the cohort has accepted, and the release branch needs no +cohort of its own unless it changes specification text. A wording change is any change to the +document's text after the readers read it, from any source (a review finding, the editor, or +`next` merged into the branch), except the lines `PUBLISHING.md` and step 5 of `CONTRIBUTING.md` +have the editor set for a version: the header, the status paragraph and the citation. A wording +change means a fresh run; readers are never reused. ## The record Each run has a file under `acceptance/runs/`, named `NNN-.md` with `NNN` the next number in the one sequence shared by the three documents, stating: the document, the pull request and the -commit read, the cohort size and model, how the readers were isolated and the command they were -started with, the run number in that pull request, who marked and, for a delegated marker, how it -was started, the marks per reader per question, any judgement made in marking, the passages -quoted as confusing and by how many readers, the findings applied, whether it passed, and a link -to the answer sheets. The answer sheets are not kept in this repository: every reader's sheet, -unedited, and the instruction a delegated marker was given are posted in one comment on the pull -request, so that an owner can check the isolation and the marks. The changelog line for the -change names the file. CI refuses a -specification change whose pull request adds no run file; it checks for at least one, and a pull -request that changes two documents adds a run for each. +commit read, the cohort size and model, with the model version each reader reported, how the +readers were isolated and the command they were started with, the run number in that pull +request, who marked and, for a delegated marker, how it was started, the marks per reader per +question, any judgement made in marking, the passages quoted as confusing and by how many +readers, the findings applied, whether it passed, and a link to the answer sheets. The answer +sheets are not kept in this repository: every reader's sheet, unedited, with a set-aside +reader's marked as set aside, and the instruction a delegated marker was given, which is its +task prompt and the agent definition it ran under, are posted in one or more consecutive +comments on the pull request, each linked from the record, so that an owner can check the marks. +The comments also hold the exact prompt every reader received and the command and the form of +the working directory the readers were started in; a sheet cannot show how its process was +started, so the record says that the editor attests the rest of the isolation. A comment is not +edited after posting, and a record whose comment was edited is void; that is checkable only +through GitHub's edit history, which is the limit of keeping the sheets outside the repository. +The changelog line for the change names the file. CI refuses a specification change whose pull +request adds no run file; it checks for at least one, and a pull request that changes two +documents adds a run for each. Each published version's changelog entry carries the lines of the changes it contains, so a version's cohort record is the set of runs those lines name. A version with no such line has not @@ -191,13 +222,17 @@ under `acceptance/` in this repository so that a run can be repeated and compare ## Changes to this process -This file, `acceptance/README.md`, `acceptance/runs/README.md` and `CONTRIBUTING.md` are the -process files, and a change to any part of them is a change to this process. They are not -specifications, and a change to them is not tested by a cohort: the cohort measures whether a -low-strength reader can follow a specification, and the process is followed by the editor. Such -a change names the runs or the case that showed the need for it, has the same independent review -as any other, and is recorded by a line under `Unreleased` in -[acceptance/CHANGES.md](acceptance/CHANGES.md), not in the changelog, which is for the three -specifications. It is bound by the section on the bar: it may raise a criterion, never relax one. -A change to a question set, key or fixture is not a change to this process; `acceptance/README.md` +The process files are this file, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md` and the checks that enforce them: `tools/changes.py`, the key-quotation checks +in `spec-qa.py`, the CI workflow and the pull request template. A change to any part of them is +a change to this process. They are not specifications, and a change to them is not tested by a +cohort: the cohort measures whether a low-strength reader can follow a specification, and the +process is followed by the editor. Such a change names the runs or the case that showed the need +for it, and goes through steps 1, 3 and 6 of `CONTRIBUTING.md`, with no scope, key or cohort +step; its review reads every process file it changes. It applies from its merge into `next`; a +run made before then may follow it, and its record says so. It is recorded by a line added at +the top of [acceptance/CHANGES.md](acceptance/CHANGES.md), naming its pull request, not in the +changelog, which is for the three specifications. It is bound by +[the section on the bar](#the-bar-only-moves-up): it may raise a criterion, never relax one. A +change to a question set, key or fixture is not a change to this process; `acceptance/README.md` governs those. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b7d8456..796be43 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -41,9 +41,9 @@ CHANGELOG.md. request template lays them out. - The observed case that prompted it, for either kind. A clarity edit still has a reason: a reader who took the passage the wrong way. -- Where a specification's Appendix A changed, the agent-facing files rebuilt with - `python3 build-agent-surfaces.py` and committed with the change; CI runs it with `--check` - and fails when they are stale. +- Where a specification's Appendix A changed, the files `python3 build-agent-surfaces.py` + writes, rebuilt with it and committed with the change; CI runs it with `--check` and fails + when they are stale. - `python3 spec-qa.py` clean and the unit tests passing, which CI checks for you. A pull request is a suggestion, not a demand. The documents have one editor, who keeps ownership @@ -62,9 +62,9 @@ survives, the change is yours. pull request with a row in `DECLINED.md`. 3. **Review and rewrite.** An independent review of the diff against the whole document, by a reviewer who did not draft the change, then the editor's rewrite, pushed to the pull request - branch where the contributor allows it or to an editor's branch that supersedes it. A review - finding that changes the document's text after a cohort has run is a wording change, and a - fresh run follows; the header the editor sets at step 5 is not a wording change. + branch where the contributor allows it or to an editor's branch that supersedes it. The pull + request names the reviewer and links or contains the review. A wording change, as + ACCEPTANCE.md defines it under "When it runs", means a fresh run. 4. **Keys.** Where the change touches a sentence an acceptance key quotes, or adds a load-bearing obligation, the editor updates the key before the cohort runs, never after. 5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text. Its record is @@ -77,8 +77,8 @@ survives, the change is yours. Steps 2 to 5 are the editor's, not CI's, and they happen in batches. The editor may have an agent draft or review, and may delegate marking as ACCEPTANCE.md allows, in which case the marker's verdict is final unless the editor shows a rule was misapplied, as ACCEPTANCE.md says; -every other decision in these steps stays the editor's. A quiet pull -request is not a declined one; a declined one says so. +every other decision in these steps stays the editor's. A quiet pull request is not a declined +one; a declined one says so. ## Versions and releases diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 0cd5100..81a9ac7 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -1,34 +1,45 @@ # Changes to the acceptance process Changes to the process files that [ACCEPTANCE.md](../ACCEPTANCE.md) names, recorded here as it -describes. The process is not a specification and is not versioned, and a change to it is not -tested by a cohort. Changes to the three specifications are recorded in -[CHANGELOG.md](../CHANGELOG.md). +describes: a line is added at the top for each change, naming its pull request. The process is +not a specification and is not versioned, and a change to it is not tested by a cohort. Changes +to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). -## Unreleased - -- The cohort can be run and marked without judgement calls that the rules leave open. A cohort - is an odd number of readers, at least five, and the editor may choose a larger one for a more - load-bearing change. Each reader is a separate process with no tools, no settings files, no - MCP servers and no context but its packet, started from an empty directory outside any - repository, and the example command shows the flags that do this; a reader sharing the - editor's context does not count, and a run left with fewer than five counting readers is void. - The packet is the document, its question set and the fixture the questions name, with nothing - added around them. A new marking section says what a delegated marker receives, that its - verdict is final unless the editor shows a rule was misapplied, that the marker takes the - stricter reading wherever it makes an open judgement, what counts as quoting, what a correct - load-bearing answer needs, which misses are traceable to the text and how the key's statements - bear on that, how unanswered items count, and when two confusing passages are the same. - Thresholds read "every reader" and "all but one", the same as five of five and four of five for - a cohort of five; every pass-criteria row must hold, and the last covers only what the others - do not. The run record states how readers were isolated, the command they were started with - and who marked, and links the answer sheets, which are posted unedited on the pull request - rather than kept in the repository; the isolation rule and the posted sheets apply from run - 019, and runs 16 to 18 stand as recorded. Process changes are recorded - in this file, not in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the - change, says which review findings mean a fresh run, adds the agent-facing rebuild after an - Appendix A change, and points to how a change to the process files is made. Prompted by gaps in - the text as it stood: `ACCEPTANCE.md` named no fixture in what a reader receives whilst +- Pull request #PR: the cohort can be run and marked without judgement calls that the rules + leave open. A cohort is an odd number of readers started, at least five, larger at the + editor's choice the more obligations the change touches. Each reader is a separate process + with no tools and no project context, started from an empty directory outside any repository + with no settings files, `CLAUDE.md`, skills, plugins, hooks, MCP servers or saved session; the + example command takes the packet on standard input. A reader is set aside only if its process + failed or returned nothing or it shared the editor's context; a run left with fewer than five + counting readers is void unless those readers already fail it, and a void run has a number, a + record and posted sheets and counts towards the number of runs. The packet is the question + set, the document and the fixture, each after one `===== FILE: =====` line, with + nothing else added. A new marking section says what a delegated marker receives, that it + writes the record and its marks and PASS or FAIL are final unless the editor shows a rule was + misapplied, that open judgements take the reading that records more misses or findings, what + a keyed item is, that quoting is five or more consecutive verbatim words given as a quotation, + what a correct load-bearing answer needs, that a miss is traceable to the text when its + reasoning quotes or states what the document says or the key says it counts, and that a key + can add such misses, never remove one, how unanswered items and truncated sheets count, and + that two confusing passages are the same when they share a sentence or a paragraph. The + thresholds apply to the readers whose answers count, read "every reader" and "all but one", + and block on a single reader for the first, third and fourth rows; the fourth row's block lifts + when a fresh cohort on the changed text passes it. A wording change is any change to the text + after the readers read it, except the lines set for a version, and the cohort runs again after + every failure, void run or wording change. The run record states the model version each reader + reported, how readers were isolated, the command and who marked, and links the answer sheets, + the exact prompt and the marker's instruction, posted unedited in comments on the pull request + rather than kept in the repository, with the rest of the isolation attested by the editor; the + isolation rule and the posted sheets apply from run 19, and runs 16 to 18 stand as recorded. + The three keys no longer limit which misses count against the document, and removing such a + limit is named as raising the bar. The process files include the checks that enforce them, and + a process change goes through steps 1, 3 and 6 of `CONTRIBUTING.md`, applies on its merge into + `next` and is recorded in this file, not in the changelog. `CONTRIBUTING.md` requires a + reviewer who did not draft the change, named in the pull request with the review, points to + the definition of a wording change, and adds the rebuild of the files + `build-agent-surfaces.py` writes after an Appendix A change. Prompted by gaps in the text as it + stood: `ACCEPTANCE.md` named no fixture in what a reader receives whilst `acceptance/README.md` did, the thresholds were written as five of five and four of five for a cohort of at least five, two pass-criteria rows both covered a single reader's traceable miss, and nothing settled an unanswered question or "the same passage". [Run diff --git a/acceptance/README.md b/acceptance/README.md index bb387a5..9af1d2f 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -1,10 +1,10 @@ # Acceptance question sets The question sets and answer keys that [ACCEPTANCE.md](../ACCEPTANCE.md) requires. One directory per -specification document, each holding the fixed questions a cold reader answers and the key an -editor marks against. The fixture cases the application question uses live in `fixtures/`, the -record of every run lives in `runs/`, and changes to the acceptance process itself are recorded -in [CHANGES.md](CHANGES.md). +specification document, each holding the fixed questions a +[cold reader](../ACCEPTANCE.md#the-cohort) answers and the key an editor marks against. The +fixture cases the application question uses live in `fixtures/`, the record of every run lives in +`runs/`, and changes to the acceptance process itself are recorded in [CHANGES.md](CHANGES.md). | Document | Questions | Key | | ------------------ | ------------------------------------------------ | ------------------------------------ | diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 132e6cb..6de0386 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -60,8 +60,7 @@ agent summary (those belong to the toolchain specification). Section [8](../../DETECTOR-SPEC.md#8-conformance) verdict: **does not conform**, because [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) fails and partial conformance MUST NOT be described as conformance. A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Partial on the grounds that thirty-eight resolve -has read the clause differently and that counts against the document; a reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Yes -because they missed the last paragraph of the fixture has misread the fixture, not the document. +has read the clause differently and that counts against the document. Clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation): **does not change the verdict**. It is a SHOULD, and Lintomatic's failure to check its own identifiers in CI is a missed SHOULD, not a further MUST failure; the MUST that fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 3ca1a80..71f3935 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -56,8 +56,7 @@ a test may serve as the detector; an agent may baseline the remaining instances. Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**, because [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) and [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) were not followed. The **defence does not conform** either, because [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) fails; a reader who says the defence conforms -because the sweep is what failed has missed [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) and that counts against the document only if the -reasoning quotes the section [7](../../SPEC.md#7-conformance) definition of a conforming defence. +because the sweep is what failed has missed [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation). Section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): the **inline ignore on `Report::render()`** was the owner's decision, not the practitioner's. An answer naming instead the decision not to build the wider rule is half right: @@ -67,8 +66,7 @@ An answer naming neither is wrong. A reader who grades [3.5](../../SPEC.md#35-enforce-permanently-and-block) Yes because the rule fails the build has not applied the clause's opening sentence on unrecorded suppression, and that counts against the document. A reader who grades [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) Yes because the URL resolves has read the versioning sentence of [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) -differently, which counts against the document; a reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the -fixture. A reader who attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing +differently, which counts against the document. A reader who attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as covering it, which is a fair reading and is not marked wrong. ## 4. Confusion diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index fd7f56c..9a20bad 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -6,14 +6,16 @@ share. Runs 1 to 15 predate this directory and are summarised in the changelog e the toolchain specification; numbering here continues from 16. A record states what `ACCEPTANCE.md` lists, in this order: the document; the pull request and -the commit the readers were given; the cohort size and model; how the readers were isolated and -the command they were started with; the run number within that pull request; who marked and, for -a delegated marker, how it was started; the marks per reader per question against the key, with -any judgement made in marking; the passages quoted as confusing and by how many readers; the -findings applied to the text as a result; and the verdict against the criteria in -`ACCEPTANCE.md`, with a link to the answer sheets. Answer sheets are not kept in this repository, -and the marks are the record; the sheets, unedited, and the instruction a delegated marker was -given are posted in one comment on the pull request, which the record links. +the commit the readers were given; the cohort size and model, with the model version each reader +reported; how the readers were isolated and the command they were started with; the run number +within that pull request; who marked and, for a delegated marker, how it was started; the marks +per reader per question against the key, with any judgement made in marking; the passages quoted +as confusing and by how many readers; the findings applied to the text as a result; and the +verdict against the criteria in `ACCEPTANCE.md`, with a link to the answer sheets. Earlier +records show the style only; where one differs from this order, this file wins. Answer sheets +are not kept in this repository, and the marks are the record; the sheets, unedited, and the +instruction a delegated marker was given are posted in one or more consecutive comments on the +pull request, each linked from the record. The `Unreleased` changelog line for the change names the record, and `tools/changes.py` refuses a specification change whose pull request adds no file here. diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 5a2602b..91112fa 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -73,9 +73,7 @@ Declaration: **does not affect the verdict**. Key quotation, [9.2](../../TOOLING A reader who says Zigpipe cannot be graded without one has read [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) wrongly. A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes on the grounds that Lintomatic "mostly" conforms has read either [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) -or the detector document's section [8](../../DETECTOR-SPEC.md#8-conformance) differently; it counts against this document only if the -reasoning quotes [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification). A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes because they missed the two undocumented rules -has misread the fixture. +or the detector document's section [8](../../DETECTOR-SPEC.md#8-conformance) differently. ## 4. Confusion From d85becaae9c5a48c8f3b388e2ef796504380ff2e Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 20:56:37 +0100 Subject: [PATCH 010/105] Method 3.2 and detector 8: apply the rulings on review 2 The kind of detector is that of the configuration a rule runs in; enabling or configuring an existing rule is not writing one; a semantic host a toolchain may not route a defence through under toolchain clause 4.1 does not count. The remediation docs always name the detector and its kind, and each step down from semantic records its reason with the stronger detectors considered. The owner's sign-off is a recorded project requirement, not a calibration; section 4's dividing line, the section 3 map and section 7's gloss carry it. Syntactic and semantic are defined by whether a detector sees the types of values; Semgrep leaves the examples. The context paragraph is a marked worked example without the alias claim. Appendix A names the kind, the move and the sign-off; agent surfaces rebuilt. Detector section 8 names the kind per graded configuration and credits none reachable only through a hosted service or licence tier; section 9 names that as its one obligation on whoever gives a verdict. README, PRIMER and the mypy and semgrep register pages defer to 3.2. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 40 ++++++++------- DETECTOR-SPEC.md | 29 ++++++----- PRIMER.md | 3 +- README.md | 4 +- SPEC.md | 73 ++++++++++++++++------------ defence-before-fix-project-prompt.md | 6 ++- tools/mypy.md | 2 +- tools/semgrep.md | 2 +- 8 files changed, 91 insertions(+), 68 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4d98982..8e3b1c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,17 +13,22 @@ version entry with no cohort line has not been accepted and is not published. ### Unreleased - New obligation, clause [3.2](SPEC.md#32-build-the-net): a rule the practitioner writes MUST be written in the strongest - kind of detector that can express it, semantic, else syntactic, else textual; a weaker kind is - allowed only where the stronger cannot express the rule or the language has no detector of - that kind that accepts bespoke rules, and the reason MUST be recorded on the rule's remediation - docs and in the description of the commit that adds or moves the rule. The practitioner judges it; - a project MAY require the owner's sign-off, which section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) notes. Section [1](SPEC.md#1-terminology) defines textual, - syntactic and semantic by what each can see, and the section [3](SPEC.md#3-the-method) map and Appendix A carry - the requirement. Prompted by project rules hosted in ast-grep, where the language already had a - semantic host, that decided by names: they reported unrelated calls of the same name and - missed calls made through an alias, and clause [3.2](SPEC.md#32-build-the-net) said any detector qualified whatever its - category. The method question set gains load-bearing statement 2d, which `ACCEPTANCE.md` now - lists. Accepted by run NNN. Minor version when released. + kind of detector that can express it, semantic, else syntactic, else textual, judged by the + configuration it runs in; enabling or configuring an existing rule is not writing one. A weaker + kind is allowed only where the stronger cannot express the rule or the language has no + detector of that kind that accepts bespoke rules, and a detector the toolchain specification's + clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) forbids routing a defence through does not count. The rule's remediation docs MUST + name its detector and kind, and for each step down from semantic the reason, with the stronger + detectors considered, MUST be recorded there and in the description of the commit that adds or + moves the rule. The practitioner judges it; a project MAY require the owner's sign-off, recorded + where it records its calibrations, which section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) and the section [3](SPEC.md#3-the-method) map note. Section [1](SPEC.md#1-terminology) + defines textual, syntactic and semantic by whether a detector sees the types of values, section + [7](SPEC.md#7-conformance)'s conforming defence carries the kind, and Appendix A carries the requirement. Prompted by + project rules hosted in ast-grep, where the language already had a semantic host, that decided + by names: they reported unrelated calls of the same name, and clause [3.2](SPEC.md#32-build-the-net) said any detector + qualified whatever its category. The method question set gains load-bearing statement 2d, which + `ACCEPTANCE.md` now lists, and `README.md`, `PRIMER.md` and two register pages defer to the + clause. Accepted by run NNN. Minor version when released. ### 1.1.0, 2026-10-02 @@ -85,12 +90,13 @@ pass it. ### Unreleased -- New obligation, section [8](DETECTOR-SPEC.md#8-conformance): a verdict on a detector MUST say whether it is semantic, syntactic - or textual, by the most a rule in it can see in any configuration it documents. It is not a - level of conformance and does not change the verdict; it is stated because method clause [3.2](SPEC.md#32-build-the-net) - now requires a rule to be written in the strongest kind of detector that can express it, and a - verdict of conforming alone presents a textual detector as an equal host. Section [2](DETECTOR-SPEC.md#2-terminology)'s gloss - carries the three terms. Accepted by run NNN. Minor version when released. +- New obligation, section [8](DETECTOR-SPEC.md#8-conformance): a verdict on a detector MUST name the kind, semantic, syntactic + or textual, of each configuration it grades, crediting none reachable only through a hosted + service or licence tier. It is not a level of conformance and does not change the verdict; it + is stated because method clause [3.2](SPEC.md#32-build-the-net) now requires a rule to be written in the strongest kind of + detector that can express it, and a verdict of conforming alone presents a textual detector as + an equal host. Section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) names it as the one obligation the document adds for a practitioner, + since it binds whoever gives a verdict, and section [2](DETECTOR-SPEC.md#2-terminology)'s gloss carries the three terms. Accepted by run NNN. Minor version when released. ### 1.0.0, 2026-09-08 diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index fb6bbcf..0fa98df 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -35,16 +35,16 @@ construct the means has moved the obligation rather than met it. Terms from the method specification carry over unchanged. Every capitalised term links to its definition; the ones this document leans on most are, in short: -| Term | In one line | -| ---------------------------------- | -------------------------------------------------------------------------------------------- | -| [Detector] | A tool that reads code without executing it and reports what it finds | -| [Semantic], [Syntactic], [Textual] | What a [Detector] can see: types, symbols and data flow; a parse tree without types; or text | -| [Rule] | One check a [Detector] evaluates | -| [Identifier] | The stable name printed with a finding that leads to its documentation | -| [Practitioner] | Whoever is doing the work, a person or an [Agent] | -| [Toolchain] | Everything a project assembles to run its checks through, [Detectors] included | -| [Suppression] | Making a finding go away without fixing it | -| [Baseline] | A recorded set of existing findings a [Rule] is told to ignore | +| Term | In one line | +| ---------------------------------- | ----------------------------------------------------------------------------------------------------- | +| [Detector] | A tool that reads code without executing it and reports what it finds | +| [Semantic], [Syntactic], [Textual] | What a [Detector] can see in a configuration: the types of values; a parse tree without them; or text | +| [Rule] | One check a [Detector] evaluates | +| [Identifier] | The stable name printed with a finding that leads to its documentation | +| [Practitioner] | Whoever is doing the work, a person or an [Agent] | +| [Toolchain] | Everything a project assembles to run its checks through, [Detectors] included | +| [Suppression] | Making a finding go away without fixing it | +| [Baseline] | A recorded set of existing findings a [Rule] is told to ignore | These are additional. @@ -307,8 +307,10 @@ summary for an [Agent]'s context, are obligations on a project's assembled [Tool and are stated in the [toolchain specification](TOOLING-SPEC.md), so there is no separate [Agent]-support level for a [Detector]. -**A verdict on a [Detector] MUST also say whether it is [Semantic], [Syntactic] or [Textual]**, -by the most a [Rule] in it can see in any configuration it documents. That is not a level of +**A verdict on a [Detector] MUST also name the kind of each configuration it grades, [Semantic], +[Syntactic] or [Textual]**: ESLint, for example, is [Syntactic] without type information and +[Semantic] with it. It credits no configuration reachable only through a hosted service or a +licence tier, as clause 5.4 credits no finding reported only there. The kind is not a level of [Conformance] and does not change the verdict: a [Textual] [Detector] can [Conform], and a [Semantic] one can fail to. It is stated because the method specification's clause 3.2 requires a [Practitioner] to write a [Rule] in the strongest kind of [Detector] that can express it, and a @@ -359,7 +361,8 @@ maintainer's reading rather than the [Detector]. ## 9. Relationship to the other specifications -This document adds no obligations to a [Practitioner] and relaxes none. Every clause here +This document adds no obligations to a [Practitioner] and relaxes none, save one: a +verdict on a [Detector], whoever gives it, names its kind under section 8. Every clause here exists to make a clause of the method specification achievable with a [Detector] in hand. Where this document and the method specification disagree, the method specification governs. It diff --git a/PRIMER.md b/PRIMER.md index edb1f15..e950a3f 100644 --- a/PRIMER.md +++ b/PRIMER.md @@ -28,7 +28,8 @@ it is an instance of coalescing an absent value into a falsy one, which is a pat patterns can be detected mechanically. Then write the rule that detects that pattern. A static analysis rule, a custom lint rule, a -check in whatever tool your project already runs on every commit. Something that reads the code +check in a tool your project already runs on every commit, one that sees types where your +language has one. Something that reads the code rather than running it, so it can be pointed at every file you have rather than only the paths your tests happen to exercise. diff --git a/README.md b/README.md index e43ba91..1f1bbd6 100644 --- a/README.md +++ b/README.md @@ -11,8 +11,8 @@ Six clauses, in order: 1. **Attribute the defect to a class.** Not "what went wrong here" but "what kind of thing is this an instance of", within a lower bound and an upper bound that are both checkable. -2. **Build the net.** A rule that reads code, in whatever detector the toolchain offers, bespoke - where nothing off the shelf will take it. +2. **Build the net.** A rule that reads code, in the strongest kind of detector that can express + it, bespoke where nothing off the shelf will take it. 3. **Prove the net by making the rule fire.** Red before green, on the originating instance, committed before the fix. 4. **Sweep the codebase, then fix every instance.** The count is the finding; the fix addresses diff --git a/SPEC.md b/SPEC.md index 05b5aaf..c14cf61 100644 --- a/SPEC.md +++ b/SPEC.md @@ -70,7 +70,7 @@ The harm a [Class] causes, which is **not necessarily a failure**. It may be a f #### Detector -A tool that reads code without executing it and reports occurrences of a pattern. A [Detector] is not a [Runner]. By what it can see, a [Detector] is [Textual], [Syntactic] or [Semantic], and clause 3.2 says which kind a [Rule] is written in. What a [Detector] must offer so that a [Practitioner] can write, prove, run and resolve a [Rule] in it is specified in the [detector specification](DETECTOR-SPEC.md). +A tool that reads code without executing it and reports occurrences of a pattern. A [Detector] is not a [Runner]. By what it can see in the configuration it runs in, a [Detector] is [Textual], [Syntactic] or [Semantic], and clause 3.2 says which kind a [Rule] is written in. What a [Detector] must offer so that a [Practitioner] can write, prove, run and resolve a [Rule] in it is specified in the [detector specification](DETECTOR-SPEC.md). #### Runner @@ -82,11 +82,11 @@ Of a [Detector]: seeing the code only as text, as grep does. It sees how the cod #### Syntactic -Of a [Detector]: seeing the parse tree of the code, without types, as ast-grep and Semgrep do. It sees the structure, but knows what a name refers to only by the name. +Of a [Detector]: seeing the parse tree of the code but not the types of values, as ast-grep does. It sees the structure, but cannot tell apart two calls of the same name on values of different types. #### Semantic -Of a [Detector]: seeing types, resolved symbols and data flow, as PHPStan, mypy, Clippy, Dylint and ESLint with typescript-eslint's type information do. It sees what a name refers to and what type a value has, whatever either is called. The same [Detector] can be [Syntactic] in one configuration and [Semantic] in another: ESLint without type information is [Syntactic]. A [Semantic] [Detector] need not accept bespoke [Rules]: Clippy does not, and in Rust a project's own [Semantic] [Rules] are written for Dylint. +Of a [Detector]: seeing the types of values as the language's type system resolves them, and so what a name refers to, as PHPStan, mypy, Clippy, Dylint and ESLint with typescript-eslint's type information do. The same [Detector] can be [Syntactic] in one configuration and [Semantic] in another: ESLint without type information is [Syntactic]. A [Semantic] [Detector] need not accept bespoke [Rules]: Clippy does not, and in Rust a project's own [Semantic] [Rules] are written for Dylint. #### Rule @@ -227,14 +227,14 @@ the [Defence] it implies is ever reached. Six clauses, in order. In brief, before the detail: -| Clause | The [Practitioner] MUST | The record shows | Goes to the [Owner] under section 4 | -| ------ | ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | -| 3.1 | Name the [Class] the [Defect] belongs to, bounded both ways, after an independent search | The [Class], the [Hazard] sentence, the two search techniques, the next wider [Rule] not built, whether a [Runner] check pins the reported behaviour | Whether the next wider [Rule] is built, and whether the [Rule] stays narrower than the search showed | -| 3.2 | Express the [Class] as a [Rule] in the strongest kind of [Detector] that can express it, never a test | The [Rule], why it is in a weaker kind than [Semantic] where it is, and any [Toolchain] gap that stopped a bespoke one | Whether a [Class] that could be defended is left undefended | -| 3.3 | Make the [Rule] fire, on the [Instance] or a [Fixture], in a commit of its own | The red run, and the sentence behind every [Narrowing] | Any exclusion whose sentence cannot be written: that is [Suppression] | -| 3.4 | [Sweep] the whole codebase, record the count, then fix every [Instance] | The count, corroborated, and what was fixed by hand or by pattern | Any [Instance] left unfixed, and any [Baseline] | -| 3.5 | Make the [Rule] permanent and [Blocking] in the project's own checks | The green run through the project's entry point, and the recorded decision behind any [Suppression] | Removing or disabling the [Rule], and any [Suppression] added to it | -| 3.6 | Print a terse [Message] with a stable [Identifier] that resolves to documentation versioned with the [Rule] | The [Remediation docs] | Nothing | +| Clause | The [Practitioner] MUST | The record shows | Goes to the [Owner] under section 4 | +| ------ | ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| 3.1 | Name the [Class] the [Defect] belongs to, bounded both ways, after an independent search | The [Class], the [Hazard] sentence, the two search techniques, the next wider [Rule] not built, whether a [Runner] check pins the reported behaviour | Whether the next wider [Rule] is built, and whether the [Rule] stays narrower than the search showed | +| 3.2 | Express the [Class] as a [Rule] in the strongest kind of [Detector] that can express it, never a test | The [Rule], its [Detector] and kind, why it is in a weaker kind than [Semantic] where it is, and any [Toolchain] gap that stopped a bespoke one | Whether a [Class] that could be defended is left undefended, and a weaker kind where the project requires the [Owner]'s sign-off | +| 3.3 | Make the [Rule] fire, on the [Instance] or a [Fixture], in a commit of its own | The red run, and the sentence behind every [Narrowing] | Any exclusion whose sentence cannot be written: that is [Suppression] | +| 3.4 | [Sweep] the whole codebase, record the count, then fix every [Instance] | The count, corroborated, and what was fixed by hand or by pattern | Any [Instance] left unfixed, and any [Baseline] | +| 3.5 | Make the [Rule] permanent and [Blocking] in the project's own checks | The green run through the project's entry point, and the recorded decision behind any [Suppression] | Removing or disabling the [Rule], and any [Suppression] added to it | +| 3.6 | Print a terse [Message] with a stable [Identifier] that resolves to documentation versioned with the [Rule] | The [Remediation docs] | Nothing | The fourth column is what an [Agent] or other [Practitioner] MUST NOT decide alone, clause by clause; section 4 states the full list and governs where the two differ. @@ -410,22 +410,29 @@ are examples rather than a permitted list, and a [Detector] is not disqualified **Qualifying is not the same as being the right host.** A [Rule] the [Practitioner] writes MUST be written in the strongest kind of [Detector] that can express it: a [Semantic] one, else a -[Syntactic] one, else a [Textual] one. A weaker kind is allowed only where the stronger kind -cannot express the [Rule], as where the [Rule] reads a file the [Semantic] [Detector] does not -parse, or where the language has no [Detector] of the stronger kind that accepts bespoke [Rules]. -The language has one when the project already runs it or the language's own [Detector] ecosystem -offers it, the same check section 2 asks for before a [Toolchain] gap is recorded. Where the -[Rule] is written in a weaker kind than [Semantic], the [Practitioner] MUST record which of those -two reasons holds, and why, on the [Rule]'s [Remediation docs] and in the description of the -commit that adds the [Rule] or moves it to another [Detector]. The [Practitioner] judges whether -a reason holds, as section 4 gives them the choice of [Detector]; a project MAY require its -[Owner]'s sign-off on a weaker kind as well, as a [Calibration] of how strict it is. - -A [Detector] that cannot see types decides by names and shapes. It reports every call named -`context`, whatever it is called on, which is a [False positive] clause 3.1 does not tolerate, and -it misses the same call made through an alias, which is an [Instance] the [Rule] never sees; -restricting it by name trades the first failure for the second. A [Semantic] [Detector] decides by -what the code is, so the bounds of clause 3.1 can be held there. +[Syntactic] one, else a [Textual] one. The kind is that of the configuration the [Rule] runs in. +Enabling or configuring an existing [Rule] is not writing one; the paragraph above covers that. A +weaker kind is allowed only where the stronger kind cannot express the [Rule], as where the +[Rule] reads a file the [Semantic] [Detector] does not parse, or where the language has no +[Detector] of the stronger kind that accepts bespoke [Rules]. The language has one when the +project already runs it or the language's own [Detector] ecosystem offers it, the same check +section 2 asks for before a [Toolchain] gap is recorded, but not one through which the +[toolchain specification](TOOLING-SPEC.md)'s clause 4.1 forbids a [Toolchain] to route a +[Defence]. The [Practitioner] MUST name the [Detector] and its kind on the [Rule]'s +[Remediation docs]. Where that kind is weaker than [Semantic], the [Practitioner] MUST also +record, for each step down from [Semantic], which of those two reasons holds and why, naming the +[Detectors] of the stronger kind that were considered, on the [Rule]'s [Remediation docs] and in +the description of the commit that adds the [Rule] or moves it to another [Detector]. The +[Practitioner] judges whether a reason holds, as section 4 gives them the choice of [Detector]; a +project MAY require its [Owner]'s sign-off on a weaker kind as well, recorded where it records +its [Calibrations], and where it has recorded no such requirement none is needed. + +**A worked example.** Take a [Rule] against calling a method named `context` on one type. A +[Detector] that cannot see types reports every call named `context`, whatever type it is called +on, which is a [False positive] clause 3.1 does not tolerate; restricting it by the name of the +variable the call is made on misses the same call on a variable named otherwise, which is an +[Instance] the [Rule] never sees. A [Semantic] [Detector] decides by the type, so the bounds of +clause 3.1 can be held there. Where the affected language has no extensible [Detector] available, the [Practitioner] MAY build a bespoke one: a program of the project's own that reads code and reports matches of the [Class], run @@ -822,7 +829,8 @@ they do not. than that the [Hazard] is absent from what it would add. The [Instances] the search found are fixed regardless; what stays with the [Owner] is the [Class] left partly undefended. -The dividing line is that the [Practitioner] decides **how the [Defence] is built** and the [Owner] +The dividing line is that the [Practitioner] decides **how the [Defence] is built**, save a weaker +kind of [Detector] where the project requires the [Owner]'s sign-off on it, and the [Owner] decides **what the codebase is permitted to keep**. An [Agent] MUST NOT [Baseline], suppress or knowingly leave an [Instance] unfixed on its own authority, whatever the [Instance] count turns out to be. @@ -927,7 +935,8 @@ the verdict that they were rests on reproduction, as this section closes by requ report. **A [Defence] [Conforms]** if it satisfies clauses 3.1, 3.2, 3.3, 3.5 and 3.6: it is drawn to a -[Class] within both bounds and not to the reported [Instance], it is evaluated by reading code, it +[Class] within both bounds and not to the reported [Instance], it is evaluated by reading code, +in the kind of [Detector] clause 3.2 requires, it was proven to fire against either the originating [Instance] or [Fixture] code, it fails rather than warns, and its [Message] is terse and resolves to [Remediation docs] versioned alongside it. A [Defence] can [Conform] whilst the [Sweep] is incomplete; what that describes is a @@ -1085,8 +1094,10 @@ This appendix restates sections 2 to 4; where the two differ, the sections gover > Write a custom [Rule] that detects the [Class], in a tool that reads code rather than running it. > Put it in a [Semantic] [Detector], one that sees types and symbols; failing that, one that sees > syntax; and only then one that sees text. Use a weaker one only where the stronger one cannot -> express the [Rule], or the language has none that takes custom [Rules], and write down which, -> and why, in the [Rule]'s documentation and in the description of the commit that adds it. Draw +> express the [Rule], or the language has none that takes custom [Rules]. Name the [Detector] and its +> kind in the [Rule]'s documentation. Where you use a weaker one, write down which reason holds, +> and why, there and in the description of the commit that adds the [Rule] or moves it to another +> [Detector], and where the project requires its [Owner]'s sign-off on a weaker one, get it. Draw > the [Rule] so that it catches more than the single [Instance] you started from, but never so > broadly that it matches code which does not carry the [Hazard]. > diff --git a/defence-before-fix-project-prompt.md b/defence-before-fix-project-prompt.md index ec8d353..e5d044c 100644 --- a/defence-before-fix-project-prompt.md +++ b/defence-before-fix-project-prompt.md @@ -15,8 +15,10 @@ for it, say so and fix the Defect conventionally. Write a custom Rule that detects the Class, in a tool that reads code rather than running it. Put it in a Semantic Detector, one that sees types and symbols; failing that, one that sees syntax; and only then one that sees text. Use a weaker one only where the stronger one cannot -express the Rule, or the language has none that takes custom Rules, and write down which, -and why, in the Rule's documentation and in the description of the commit that adds it. Draw +express the Rule, or the language has none that takes custom Rules. Name the Detector and its +kind in the Rule's documentation. Where you use a weaker one, write down which reason holds, +and why, there and in the description of the commit that adds the Rule or moves it to another +Detector, and where the project requires its Owner's sign-off on a weaker one, get it. Draw the Rule so that it catches more than the single Instance you started from, but never so broadly that it matches code which does not carry the Hazard. diff --git a/tools/mypy.md b/tools/mypy.md index 0170163..b5bc601 100644 --- a/tools/mypy.md +++ b/tools/mypy.md @@ -43,4 +43,4 @@ Clause [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-writ ## Notes for a practitioner -Treat mypy as a detector for type-shaped classes only: where the class is "this value can be None here", tightening the configuration is a legitimate off-the-shelf rule under method clause [3.2](../SPEC.md#32-build-the-net), proven by watching the count go red. For any other class, host the defence in Pylint or a Flake8 plugin instead, and enable `warn_unused_ignores` plus a check in a second detector on new `# type: ignore` comments so the inline route stays visible. +Treat mypy as a detector for type-shaped classes only: where the class is "this value can be None here", tightening the configuration is a legitimate off-the-shelf rule under method clause [3.2](../SPEC.md#32-build-the-net), proven by watching the count go red. For any other class, choose the host under method clause [3.2](../SPEC.md#32-build-the-net), which asks for the strongest kind of detector that can express the rule and a recorded reason for a weaker one, and enable `warn_unused_ignores` plus a check in a second detector on new `# type: ignore` comments so the inline route stays visible. diff --git a/tools/semgrep.md b/tools/semgrep.md index 30de714..ec54f37 100644 --- a/tools/semgrep.md +++ b/tools/semgrep.md @@ -12,7 +12,7 @@ checked: 2026-09-08, version 1.176.0 {% include tool-grades.html %} -Semgrep is a pattern-based static analyser whose rules are written in YAML using the syntax of the target language itself, across some thirty languages. The open-source command line tool runs locally; the Semgrep AppSec Platform adds a hosted registry, Pro rules, cross-file analysis and triage. In a pipeline it is the usual choice for a project-specific rule that a language linter cannot express cheaply, and its rule-testing harness is one of the best in the field, which makes it a natural detector for the method. +Semgrep is a pattern-based static analyser whose rules are written in YAML using the syntax of the target language itself, across some thirty languages. The open-source command line tool runs locally; the Semgrep AppSec Platform adds a hosted registry, Pro rules, cross-file analysis and triage. In a pipeline it can host a project-specific rule that a language linter cannot express, where method clause [3.2](../SPEC.md#32-build-the-net), which asks for the strongest kind of detector that can express the rule, allows it, and its rule-testing harness is one of the best in the field, which makes it a natural detector for the method. ## How it is conformant From 28c0a0cb86b00d983ef7f287731c6231786273ba Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 20:57:38 +0100 Subject: [PATCH 011/105] Acceptance keys: follow the revised 3.2 and detector 8 Question 2d reworded before any run, as ruling 17 decides, with its ACCEPTANCE.md line. Key 2d quotes the revised 3.2 sentences verbatim, adds the kind on the remediation docs as supporting, asks for the semantic kind first and keeps the either-place leniency (C12). The restatement bullet carries the kind on the remediation docs. The ledger fixture's 3.2 row is now Partial or No, since its wiki page names neither detector nor kind, and section 7's verdicts name 3.2 with 3.6. The detector key's restatement and Lintomatic quotation follow section 8's per-configuration wording and section 9. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 4 +-- acceptance/detector/KEY.md | 7 ++-- acceptance/method/KEY.md | 61 +++++++++++++++++++--------------- acceptance/method/QUESTIONS.md | 4 +-- 4 files changed, 43 insertions(+), 33 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 7b5485e..d0101bb 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -29,8 +29,8 @@ document, so that marking is mechanical. Each reader answers all of them. sentence and a quotation: - `SPEC.md`: what a practitioner does before fixing a defect, in order; who may baseline or - suppress; what "conforms" means for a remediation; which kind of detector should host a - rule, and what is recorded when another is used. + suppress; what "conforms" means for a remediation; which kind of detector must host a + rule the practitioner writes, when a weaker one is allowed, and what is recorded then. - `DETECTOR-SPEC.md`: what "conforms" means and whether a tool that predates the document can conform; what a detector must do about an inline suppression route; where a bundled rule's documentation must live. diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index e9724a6..068dc4e 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -18,8 +18,9 @@ bullet asserting an obligation the document does not state, is a correct restate offline, with the documentation shipped and versioned with the rule ([6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) to [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together)). - Make any inline suppression route detectable or disableable ([7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)). -A bullet saying that a verdict on a detector names whether it is semantic, syntactic or textual -(section [8](../../DETECTOR-SPEC.md#8-conformance)) is correct too, though it is an obligation on whoever gives the verdict. +A bullet saying that a verdict on a detector names the kind, semantic, syntactic or textual, of +each configuration it grades (section [8](../../DETECTOR-SPEC.md#8-conformance)) is correct too, though it is an obligation on whoever +gives the verdict, as section [9](../../DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) says. Wrong if it asserts: the detector must be semantic to conform; the detector must enforce a project's suppression policy; the detector must @@ -69,7 +70,7 @@ because they missed the last paragraph of the fixture has misread the fixture, n The fixture does not say what Lintomatic can see. A reader who names its kind as unknown, or names none, is not marked on it; a reader who makes the verdict turn on its kind has read section [8](../../DETECTOR-SPEC.md#8-conformance) -differently, which counts against the document: the kind "is not a level of Conformance and does +differently, which counts against the document: "The kind is not a level of Conformance and does not change the verdict". Clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation): **does not change the verdict**. It is a SHOULD, and Lintomatic's failure to check its diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 2648051..64bbead 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -12,7 +12,8 @@ bullet asserting an obligation the document does not state, is a correct restate - Attribute the defect to a class, bounded on both sides, with the hazard stated ([3.1](../../SPEC.md#31-attribute-the-defect-to-a-class)). - Express the class as a rule in a detector that reads code, never a test ([3.2](../../SPEC.md#32-build-the-net)). - Write the rule in the strongest kind of detector that can express it, semantic, else - syntactic, else textual, and record why where it is in a weaker kind ([3.2](../../SPEC.md#32-build-the-net)). + syntactic, else textual, name the detector and its kind on the rule's remediation docs, and + record why where it is in a weaker kind ([3.2](../../SPEC.md#32-build-the-net)). - Prove the rule fires, against the instance or a fixture, in a commit of its own ([3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire)). - Sweep the whole codebase, record the count, then fix every instance ([3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance)). - Make the rule permanent and blocking in the project's checks ([3.5](../../SPEC.md#35-enforce-permanently-and-block)). @@ -45,17 +46,22 @@ detector may host a rule. rest on reproduction, not on the report". An answer that accepts the pull request description as the verdict is wrong, and so is one that states the six clauses and says nothing of reproduction. -- **2d.** The strongest kind that can express it: a semantic detector, one that sees types and - symbols, else a syntactic one, else a textual one; a weaker kind only where the stronger cannot - express the rule or the language has none of that kind that accepts bespoke rules, and then the - practitioner records which reason holds, and why, on the rule's remediation docs and in the - description of the commit that adds or moves the rule. Key quotations, [3.2](../../SPEC.md#32-build-the-net): "A Rule the +- **2d.** The strongest kind that can express it: a semantic detector, one that sees the types of + values, else a syntactic one, else a textual one, judged by the configuration the rule runs in; + a weaker kind only where the stronger cannot express the rule or the language has no detector of + that kind that accepts bespoke rules (one the toolchain may not route a defence through does + not count), and then the practitioner records, for each step down from semantic, which reason + holds and why, naming the stronger detectors considered, on the rule's remediation docs and in + the description of the commit that adds or moves the rule. Key quotations, [3.2](../../SPEC.md#32-build-the-net): "A Rule the Practitioner writes MUST be written in the strongest kind of Detector that can express it: a - Semantic one, else a Syntactic one, else a Textual one."; "Where the Rule is written in a weaker - kind than Semantic, the Practitioner MUST record which of those two reasons holds, and why, on + Semantic one, else a Syntactic one, else a Textual one."; "Where that kind is weaker than + Semantic, the Practitioner MUST also record, for each step down from Semantic, which of those + two reasons holds and why, naming the Detectors of the stronger kind that were considered, on the Rule's Remediation docs and in the description of the commit that adds the Rule or moves it - to another Detector." An answer naming the recorded reason in one of the two places is correct. - An answer that says any detector is as good as another, that the semantic kind is only + to another Detector." Supporting: "The Practitioner MUST name the Detector and its kind on the + Rule's Remediation docs." The answer must put the semantic kind first. An answer naming the + recorded reason in one of the two places is correct. An answer that says any detector is as + good as another, that the semantic kind is only preferred and may be passed over for another reason, that a weaker kind is never allowed, that the owner must always approve a weaker kind (the practitioner judges it unless the project requires the owner's sign-off), or that names the kind and says nothing of the recorded reason, @@ -63,19 +69,20 @@ detector may host a rule. ## 3. Application: the ledger remediation -| Clause | Intended verdict | Reasoning the fixture supports | -| ----------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | Class, hazard and the next wider rule are named; Partial or No is accepted where the reader notes that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | -| [3.2](../../SPEC.md#32-build-the-net) | Yes | A rule in the project's static analyser, reading code, not a test; the fixture does not say what the analyser can see, so nothing shows the rule is in a weaker kind than semantic | -| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | Commit B is red on the fixture and the originating instance and survives as its own commit | -| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Five findings recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority; Partial accepted where the reasoning names that suppression | -| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | The rule is permanent and blocking, but the suppression was not a recorded decision; [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | -| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule; Partial accepted where the reasoning quotes the versioning sentence | +| Clause | Intended verdict | Reasoning the fixture supports | +| ----------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | Class, hazard and the next wider rule are named; Partial or No is accepted where the reader notes that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | +| [3.2](../../SPEC.md#32-build-the-net) | Partial or No | A rule in the project's static analyser, reading code, not a test, but the wiki page of commit D names neither the detector nor its kind; [3.2](../../SPEC.md#32-build-the-net): "The Practitioner MUST name the Detector and its kind on the Rule's Remediation docs." The fixture does not say what the analyser can see, so no reason for a weaker kind is shown to be owed | +| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | Commit B is red on the fixture and the originating instance and survives as its own commit | +| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Five findings recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority; Partial accepted where the reasoning names that suppression | +| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | The rule is permanent and blocking, but the suppression was not a recorded decision; [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | +| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule; Partial accepted where the reasoning quotes the versioning sentence | -Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**, because [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) and [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) were not followed. The -**defence does not conform** either, because [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) fails; a reader who says the defence conforms -because the sweep is what failed has missed [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) and that counts against the document only if the -reasoning quotes the section [7](../../SPEC.md#7-conformance) definition of a conforming defence. +Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**, because [3.2](../../SPEC.md#32-build-the-net), [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) and [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) were not followed. The +**defence does not conform** either, because [3.2](../../SPEC.md#32-build-the-net) and [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) fail; a reader who names either is correct. +A reader who says the defence conforms because the sweep is what failed has missed both, and that +counts against the document only if the reasoning quotes the section [7](../../SPEC.md#7-conformance) definition of a +conforming defence. Section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): the **inline ignore on `Report::render()`** was the owner's decision, not the practitioner's. An answer naming instead the decision not to build the wider rule is half right: @@ -86,10 +93,12 @@ A reader who grades [3.5](../../SPEC.md#35-enforce-permanently-and-block) Yes be sentence on unrecorded suppression, and that counts against the document. A reader who grades [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) Yes because the URL resolves has read the versioning sentence of [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) differently, which counts against the document; a reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the -fixture. A reader who grades [3.2](../../SPEC.md#32-build-the-net) Partial or No because no reason for the detector's kind -is recorded has assumed a weaker kind of detector than the fixture shows: the reason is owed only -where the rule is in a weaker kind than semantic. That counts against the document only if the -reasoning quotes [3.2](../../SPEC.md#32-build-the-net) as requiring the reason whatever the detector. A reader who attributes +fixture. A reader who grades [3.2](../../SPEC.md#32-build-the-net) Yes has not applied the sentence requiring the remediation +docs to name the detector and its kind, and that counts against the document. A reader who grades +[3.2](../../SPEC.md#32-build-the-net) Partial or No only because no reason for a weaker kind is recorded has the verdict on a +ground the fixture does not show: the reason is owed only where the rule is in a weaker kind than +semantic. That is not marked wrong, and counts against the document only if the reasoning quotes +[3.2](../../SPEC.md#32-build-the-net) as requiring the reason whatever the detector. A reader who attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as covering it, which is a fair reading and is not marked wrong. ## 4. Confusion diff --git a/acceptance/method/QUESTIONS.md b/acceptance/method/QUESTIONS.md index 2c772a2..721fb9c 100644 --- a/acceptance/method/QUESTIONS.md +++ b/acceptance/method/QUESTIONS.md @@ -18,8 +18,8 @@ Answer each in one sentence, followed by the sentence of the document you are re authority? - 2c. What does it mean for a remediation to conform, and on what must a reviewer's verdict on it rest? -- 2d. Where more than one kind of detector could host the rule, which kind should the - practitioner write it in, and what must they do if they write it in another? +- 2d. Which kind of detector must a rule the practitioner writes be written in, when may a weaker + kind be used, and what must the practitioner record then? ## 3. Application From 23e7d8c140d1ebadbec6330f6ff5a10f218d226c Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 21:02:50 +0100 Subject: [PATCH 012/105] Acceptance process: apply the editor's rulings on review 3 The fourth row's block lifts only when no reader of a fresh cohort names the same passage, as on origin/next, and the keys say "names". Quoting has no length floor and excuses only case, quotation-mark style and markup. A question 2 answer states each thing the key's answer states unless the key says it may be left out. A reader is set aside only for empty answer text or shared context, and any set-aside voids the run unless it already fails. A failed text is never rerun unchanged; only version numbers and dates set for a release are excepted from a wording change. A process change names the run it applies from and is followed whole. The marker receives this pull request's earlier runs and the readers set aside; keys and marking name the commit the readers read. The record lists set-asides, the attestation, re-marks, managed settings, early adoption and void runs; posted sheets are whole outputs, never edited, hidden or deleted. Question 1 misses count only where the key says; half right is a miss; detector 7.1's conditional is gone. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 146 ++++++++++++++++++------------------ acceptance/CHANGES.md | 69 ++++++++--------- acceptance/README.md | 2 + acceptance/detector/KEY.md | 8 +- acceptance/method/KEY.md | 7 +- acceptance/runs/README.md | 17 +++-- acceptance/toolchain/KEY.md | 5 +- 7 files changed, 129 insertions(+), 125 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 83afafb..bab883d 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -15,23 +15,23 @@ test is run with a low-strength model so that clarity is measured at the floor, touches. - **Each reader is a separate process with no tools and no project context, only its packet**, - started from an empty directory outside any repository, with no settings files, no `CLAUDE.md`, - skills, plugins, hooks or MCP servers, and no saved session, so that no project instruction or - settings file reaches it. The model tool's own fixed system prompt is not project context. For - example, with the packet on standard input: + started from an empty directory outside any repository, with no user, project or local settings + files, no `CLAUDE.md`, skills, plugins, hooks or MCP servers, and no saved session, so that no + project instruction reaches it. The model tool's own fixed system prompt is not project + context. For example, with the packet on standard input: ``` claude -p --safe-mode --model haiku --tools "" --setting-sources "" --strict-mcp-config --no-session-persistence --output-format json < packet.md ``` The JSON output gives the model version each reader reported. A reader isolated in this way is - a cold reader. A reader whose process failed or returned nothing, or that shared the editor's - context, such as an agent dispatched from the editor's own session, is set aside and its - answers do not count; no reader is set aside for any other cause, and the record shows the - cause. If fewer than five readers' answers count, the run is void and a fresh cohort runs, - unless the readers whose answers count already fail a criterion, when the run fails. A void run - takes a run number, has a record and posted sheets like any other, and counts towards the - number of runs. + a cold reader. A reader whose answer text is empty, or that shared the editor's context, such + as an agent dispatched from the editor's own session, is set aside and its answers do not + count; no reader is set aside for any other cause, and the record shows the cause. If any + reader is set aside, the run is void and a fresh cohort runs, unless the readers whose answers + count already fail a criterion, when the run fails; a run never passes with a reader set + aside. A void run takes a run number, has a record and posted sheets like any other, and + counts towards the number of runs. - Each reader receives **only its packet**: the document's question set, the changed document and the fixture the questions name, each in full in the reader's prompt, the question set @@ -44,9 +44,6 @@ test is run with a low-strength model so that clarity is measured at the floor, - Readers are run **after** every stronger review has been applied. They are the last gate, not the first draft. -The isolation rule above and the answer sheets posted under [The record](#the-record) apply from -run 19; runs 16 to 18 stand as recorded. - ## The question set Written before the run, with an **answer key** for every keyed question drawn verbatim from the @@ -96,13 +93,13 @@ instruction for that reason, and neither is reinstated. The editor marks each answer sheet against the document's key, or delegates the marking to a separate marker that has seen neither the drafting nor the review. Whoever marks is the marker -in this file. A delegated marker receives only these: the document at the commit the readers -read, its question set, the fixture the questions name and its key; this file and -`acceptance/runs/README.md`, with earlier run records of other pull requests as examples of -style only; the answer sheets; and, from the editor, the facts the record states that the sheets -do not, which are the pull request and the commit, the cohort size and model, how the readers -were isolated and the command they were started with, how the marker was started, and the run -number. The marker writes the record, stating those facts as the editor supplied them, and the +in this file. A delegated marker receives only these: the document, its question set, the +fixture the questions name and its key, each at the commit the readers read; this file and +`acceptance/runs/README.md`; the earlier run records of the same pull request, and those of +other pull requests as examples of style only; the answer sheets; and, from the editor, the +facts the record states that the sheets do not, which are the pull request and the commit, the +cohort size and model, how the readers were isolated and the command they were started with, the +readers set aside and why, how the marker was started, and the run number. The marker writes the record, stating those facts as the editor supplied them, and the editor adds the findings applied and the link to the posted sheets after marking. A delegated marker's verdict, its marks and its PASS or FAIL, is final unless the editor shows that a rule in this file was misapplied; the editor then re-marks what the misapplied rule affected, and the @@ -110,28 +107,30 @@ record says which rule and why. Each sheet is marked question by question, as fo - **Keyed items.** Question 1 as a whole is one keyed item, and so is each item of question 2 and each clause verdict and each summary verdict of question 3. -- **Quoting.** A reader quotes the document when it gives five or more consecutive words of the - document, verbatim apart from markdown link syntax and emphasis markers, as a quotation, in - quotation marks or a block quote. Naming a clause number, or paraphrasing, is not quoting. +- **Quoting.** A reader quotes the document when it gives consecutive words of the document, in + the document's order and of any length, as a quotation, in quotation marks or a block quote. + The words are verbatim apart from letter case, quotation-mark or apostrophe style, and markdown + link or emphasis markup. Naming a clause number, or paraphrasing, is not quoting. - **Questions 1 and 3** are marked as the key states for each item. -- **Question 2.** An answer is correct when its sentence states what the key's answer states, - including anything the key says an answer is wrong to omit, asserts nothing the key says makes - an answer wrong, and quotes the document in support. The quotation need not be the one the key - gives. -- **A miss traceable to the text** is a miss whose reasoning quotes the document or states, in - the reader's own words, what the document says or requires, or a miss the key says counts - against the document. Such a miss is traceable even where it also misreads the fixture, and - whatever the key says of it: the key can add to the misses that count against the document, - never take one away. Only a miss whose reasoning states nothing about the document, such as a - misread fixture alone, is the reader's. Question 1 has no reasoning, so a miss there is - traceable only where the key says it counts. +- **Question 2.** An answer is correct when its sentence states each thing the key's answer + states, unless the key says it may be left out, asserts nothing the key says makes an answer + wrong, and quotes the document in support. The quotation need not be the one the key gives. +- **A miss traceable to the text**, under questions 2 and 3, is a miss whose reasoning quotes the + document or states, in the reader's own words, what the document says or requires. Such a miss + is traceable even where it also misreads the fixture, and whatever the key says of it. Only a + miss whose reasoning states nothing about the document, such as a misread fixture alone, is the + reader's. A miss under any question is also traceable where the key says it counts against the + document: the key can add to these misses, never take one away. Any other miss on question 1 + is the reader's. - **An unanswered question.** Each keyed item left unanswered is one miss, not traceable to the text, so a sheet that leaves three items of question 3 blank has three misses. Question 4 left - unanswered counts as no passage quoted. A truncated sheet is marked as it stands; a reader - whose process failed or returned nothing is set aside, as [The cohort](#the-cohort) says. -- **The same passage.** A quotation under question 4 covers every sentence of the document it - quotes any part of, in every place the quoted words appear, and a passage named without a - quotation, by clause number or description, covers every sentence of what it names. A heading, + unanswered counts as no passage quoted. A sheet with any answer text is marked as it stands, + whatever its process returned; one whose answer text is empty is set aside, as + [The cohort](#the-cohort) says. +- **The same passage.** A quotation under question 4, of any length, covers every sentence + containing any part of the quoted words taken as one run of words, in every place that run + appears, and a passage named without a quotation, by clause number or description, covers + every sentence of what it names. A heading, list item or table row counts as one sentence. Two passages are the same when they cover a sentence in common or lie in the same paragraph, and a list item or table row is its own paragraph. @@ -146,13 +145,13 @@ The change passes when all of the following hold on a single run. The thresholds readers whose answers count; with five such readers, "every reader" is five of five and "all but one" is four of five. -| Criterion | Threshold | -| ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | -| Load-bearing statements (question 2) | **Every** reader correct on every one | -| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | -| A miss traceable to the text | **None.** Any such miss, as [Marking](#marking) defines it, is a finding, whatever the count | -| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort on the changed text passes this row | -| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | +| Criterion | Threshold | +| ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Load-bearing statements (question 2) | **Every** reader correct on every one | +| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | +| A miss traceable to the text | **None.** Any such miss, as [Marking](#marking) defines it, is a finding, whatever the count | +| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it: no reader of the fresh cohort names a passage of the changed text that is the same as it, as [Marking](#marking) defines the same passage | +| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | Every row must hold, and the last covers only what none of the others catches. The first, third and fourth rows block on what a single reader, or for the fourth two readers, did, and the second @@ -184,34 +183,38 @@ something. ## When it runs -On the final text of each pull request, after the editor's rewrite and any key update, and again -after every failure, void run or wording change. It is the last gate before the merge to `next`, -so that every change on `next` is one the cohort has accepted, and the release branch needs no -cohort of its own unless it changes specification text. A wording change is any change to the -document's text after the readers read it, from any source (a review finding, the editor, or -`next` merged into the branch), except the lines `PUBLISHING.md` and step 5 of `CONTRIBUTING.md` -have the editor set for a version: the header, the status paragraph and the citation. A wording -change means a fresh run; readers are never reused. +On the final text of each pull request, after the editor's rewrite and any key update. A fresh +cohort runs after every change to the text that follows a failure, after every void run, and +after every wording change; a text that failed is never run again unchanged. It is the last gate +before the merge to `next`, so that every change on `next` is one the cohort has accepted, and +the release branch needs no cohort of its own unless it changes specification text. A wording +change is any change to the document's text after the readers read it, from any source (a +review finding, the editor, or `next` merged into the branch), except a version number or date +that `PUBLISHING.md` or step 5 of `CONTRIBUTING.md` has the editor set, in the header, the status +section, the citation or the closing table. Any other change to those paragraphs is a wording +change. Readers are never reused. ## The record Each run has a file under `acceptance/runs/`, named `NNN-.md` with `NNN` the next number in the one sequence shared by the three documents, stating: the document, the pull request and the commit read, the cohort size and model, with the model version each reader reported, how the -readers were isolated and the command they were started with, the run number in that pull -request, who marked and, for a delegated marker, how it was started, the marks per reader per -question, any judgement made in marking, the passages quoted as confusing and by how many -readers, the findings applied, whether it passed, and a link to the answer sheets. The answer -sheets are not kept in this repository: every reader's sheet, unedited, with a set-aside -reader's marked as set aside, and the instruction a delegated marker was given, which is its -task prompt and the agent definition it ran under, are posted in one or more consecutive -comments on the pull request, each linked from the record, so that an owner can check the marks. -The comments also hold the exact prompt every reader received and the command and the form of -the working directory the readers were started in; a sheet cannot show how its process was -started, so the record says that the editor attests the rest of the isolation. A comment is not -edited after posting, and a record whose comment was edited is void; that is checkable only -through GitHub's edit history, which is the limit of keeping the sheets outside the repository. -The changelog line for the change names the file. CI refuses a specification change whose pull +readers were isolated and the command they were started with, whether the machine applies +managed (policy) settings to the readers, that the editor attests the rest of the isolation, the +readers set aside and the cause, the run number in that pull request, counting void runs, who +marked and, for a delegated marker, how it was started, the marks per reader per question, any +judgement made in marking, on a re-mark which rule and why, the passages quoted as confusing and +by how many readers, the findings applied, whether it was made under a process change before +that change merged, whether it passed, and a link to the answer sheets. The answer sheets are +not kept in this repository: every reader's sheet, which is its whole output as its process +returned it (the JSON, where the command asks for JSON), with a set-aside reader's marked as set +aside, and the instruction a delegated marker was given, which is its task prompt and the agent +definition it ran under, are posted in one or more comments on the pull request, each linked +from the record, so that an owner can check the marks. The comments also hold the exact prompt +every reader received and the command and the form of the working directory the readers were +started in; a sheet cannot show how its process was started, which is why the editor attests the +rest of the isolation. The comments are never edited, hidden or deleted after posting; if one +is, the run is void and a fresh cohort runs. The changelog line for the change names the file. CI refuses a specification change whose pull request adds no run file; it checks for at least one, and a pull request that changes two documents adds a run for each. @@ -229,8 +232,9 @@ a change to this process. They are not specifications, and a change to them is n cohort: the cohort measures whether a low-strength reader can follow a specification, and the process is followed by the editor. Such a change names the runs or the case that showed the need for it, and goes through steps 1, 3 and 6 of `CONTRIBUTING.md`, with no scope, key or cohort -step; its review reads every process file it changes. It applies from its merge into `next`; a -run made before then may follow it, and its record says so. It is recorded by a line added at +step; its review reads every process file it changes. It names the run number it applies from; +a run under it made before it merges follows all of it, never part, and its record says so. It +is recorded by a line added at the top of [acceptance/CHANGES.md](acceptance/CHANGES.md), naming its pull request, not in the changelog, which is for the three specifications. It is bound by [the section on the bar](#the-bar-only-moves-up): it may raise a criterion, never relax one. A diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 81a9ac7..69550db 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -6,42 +6,37 @@ not a specification and is not versioned, and a change to it is not tested by a to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). - Pull request #PR: the cohort can be run and marked without judgement calls that the rules - leave open. A cohort is an odd number of readers started, at least five, larger at the - editor's choice the more obligations the change touches. Each reader is a separate process - with no tools and no project context, started from an empty directory outside any repository - with no settings files, `CLAUDE.md`, skills, plugins, hooks, MCP servers or saved session; the - example command takes the packet on standard input. A reader is set aside only if its process - failed or returned nothing or it shared the editor's context; a run left with fewer than five - counting readers is void unless those readers already fail it, and a void run has a number, a - record and posted sheets and counts towards the number of runs. The packet is the question - set, the document and the fixture, each after one `===== FILE: =====` line, with - nothing else added. A new marking section says what a delegated marker receives, that it - writes the record and its marks and PASS or FAIL are final unless the editor shows a rule was - misapplied, that open judgements take the reading that records more misses or findings, what - a keyed item is, that quoting is five or more consecutive verbatim words given as a quotation, - what a correct load-bearing answer needs, that a miss is traceable to the text when its - reasoning quotes or states what the document says or the key says it counts, and that a key - can add such misses, never remove one, how unanswered items and truncated sheets count, and - that two confusing passages are the same when they share a sentence or a paragraph. The - thresholds apply to the readers whose answers count, read "every reader" and "all but one", - and block on a single reader for the first, third and fourth rows; the fourth row's block lifts - when a fresh cohort on the changed text passes it. A wording change is any change to the text - after the readers read it, except the lines set for a version, and the cohort runs again after - every failure, void run or wording change. The run record states the model version each reader - reported, how readers were isolated, the command and who marked, and links the answer sheets, - the exact prompt and the marker's instruction, posted unedited in comments on the pull request - rather than kept in the repository, with the rest of the isolation attested by the editor; the - isolation rule and the posted sheets apply from run 19, and runs 16 to 18 stand as recorded. - The three keys no longer limit which misses count against the document, and removing such a - limit is named as raising the bar. The process files include the checks that enforce them, and - a process change goes through steps 1, 3 and 6 of `CONTRIBUTING.md`, applies on its merge into - `next` and is recorded in this file, not in the changelog. `CONTRIBUTING.md` requires a - reviewer who did not draft the change, named in the pull request with the review, points to - the definition of a wording change, and adds the rebuild of the files - `build-agent-surfaces.py` writes after an Appendix A change. Prompted by gaps in the text as it - stood: `ACCEPTANCE.md` named no fixture in what a reader receives whilst - `acceptance/README.md` did, the thresholds were written as five of five and four of five for a - cohort of at least five, two pass-criteria rows both covered a single reader's traceable miss, - and nothing settled an unanswered question or "the same passage". [Run + leave open; this applies from run 19, and runs 16 to 18 stand as recorded. A cohort is an odd + number of readers started, at least five, larger at the editor's choice the more obligations + the change touches, each a separate process with no tools and no project context, started from + an empty directory outside any repository with no user, project or local settings files, + `CLAUDE.md`, skills, plugins, hooks, MCP servers or saved session. A reader is set aside only + if its answer text is empty or it shared the editor's context, and a run with a reader set + aside is void, or fails if the readers who count already fail it; a void run has a number, a + record and posted sheets. The packet is the question set, the document and the fixture, each + after one `===== FILE: =====` line. A new marking section says what a delegated marker + receives and that its verdict is final unless a rule was misapplied, that open judgements take + the stricter reading, and defines a keyed item, quoting (consecutive words of any length, + verbatim apart from case, quotation-mark style and markup), a correct load-bearing answer, a + miss traceable to the text, which a key can add to but never remove, an unanswered item and + the same passage. The thresholds apply to the readers whose answers count and block on a single + reader for the first, third and fourth rows, and the fourth row says what a fresh cohort no + longer naming a passage means. A fresh cohort runs after every change following a failure, + every void run and every wording change, which is any change to the text after the readers + read it except a version number or date the editor sets for a release. The record adds the + model version each reader reported, the isolation, managed settings, readers set aside, void + runs and re-marks, and links each reader's whole output and the marker's instruction, posted on + the pull request and never edited, hidden or deleted, or the run is void. The keys are marked at + the commit the readers read and no longer limit which misses count against the document, + removing such a limit is named as raising the bar, and the method key's "half right" is a miss. + The process files include the checks that enforce them, and a process change goes through + steps 1, 3 and 6 of `CONTRIBUTING.md`, names the run it applies from and is recorded here, not + in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the change, points to + the definition of a wording change, and adds the rebuild of the files `build-agent-surfaces.py` + writes after an Appendix A change. Prompted by gaps in the text as it stood: `ACCEPTANCE.md` + named no fixture in what a reader receives whilst `acceptance/README.md` did, the thresholds + were written as five of five and four of five for a cohort of at least five, two pass-criteria + rows both covered a single reader's traceable miss, and nothing settled an unanswered question + or "the same passage". [Run 16](runs/016-deferral-clause-run-1.md), [run 17](runs/017-deferral-clause-run-2.md) and [run 18](runs/018-deferral-clause-run-3.md) ruled on these case by case. diff --git a/acceptance/README.md b/acceptance/README.md index 9af1d2f..0e43599 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -35,6 +35,8 @@ matches the document is a defect in this directory, not a reason to mark a reade Questions are fixed so that runs can be compared across versions. Adding a keyed question is allowed when a new clause is load-bearing; removing or softening one to make a run pass is not. +A process change may remove a limit a key puts on which misses count against the document, as +[the section on the bar](../ACCEPTANCE.md#the-bar-only-moves-up) allows. The reader may quote a passage it found confusing; it is never asked to rate the document, say what would make a passage clearer, or propose a change: the cohort confirms comprehension and does not design the text. diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 6de0386..ee01416 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -1,6 +1,7 @@ # Answer key for DETECTOR-SPEC.md -Marked against the document at the version named in the changelog entry that records the run. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, update the quotation here in the same commit. @@ -65,12 +66,9 @@ has read the clause differently and that counts against the document. Clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation): **does not change the verdict**. It is a SHOULD, and Lintomatic's failure to check its own identifiers in CI is a missed SHOULD, not a further MUST failure; the MUST that fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). -A reader who marks [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) No because the comment takes no reason has confused [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) with [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason), which is -SHOULD; that counts against the document if the reasoning quotes [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) or [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason). - ## 4. Confusion No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to +finding that blocks until the text is changed and a fresh cohort no longer names it. What to change is the editor's decision; the reader is not asked and any suggestion it volunteers is discarded unread. diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 71f3935..837b819 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -1,6 +1,7 @@ # Answer key for SPEC.md -Marked against the document at the version named in the changelog entry that records the run. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, update the quotation here in the same commit. @@ -59,7 +60,7 @@ Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**, because the sweep is what failed has missed [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation). Section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): the **inline ignore on `Report::render()`** was the owner's decision, not the -practitioner's. An answer naming instead the decision not to build the wider rule is half right: +practitioner's. An answer naming instead the decision not to build the wider rule is half right, which is a miss: that is also listed in section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) as the owner's to take, and an answer naming both is correct. An answer naming neither is wrong. @@ -72,6 +73,6 @@ passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as cove ## 4. Confusion No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to +finding that blocks until the text is changed and a fresh cohort no longer names it. What to change is the editor's decision; the reader is not asked and any suggestion it volunteers is discarded unread. diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index 9a20bad..4af8149 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -7,15 +7,18 @@ the toolchain specification; numbering here continues from 16. A record states what `ACCEPTANCE.md` lists, in this order: the document; the pull request and the commit the readers were given; the cohort size and model, with the model version each reader -reported; how the readers were isolated and the command they were started with; the run number -within that pull request; who marked and, for a delegated marker, how it was started; the marks -per reader per question against the key, with any judgement made in marking; the passages quoted -as confusing and by how many readers; the findings applied to the text as a result; and the +reported; how the readers were isolated and the command they were started with; whether the +machine applies managed (policy) settings to the readers; that the editor attests the rest of the +isolation; the readers set aside and the cause; the run number within that pull request, counting +void runs; who marked and, for a delegated marker, how it was started; the marks per reader per +question against the key, with any judgement made in marking and, on a re-mark, which rule and +why; the passages quoted as confusing and by how many readers; the findings applied to the text +as a result; whether the run was made under a process change before that change merged; and the verdict against the criteria in `ACCEPTANCE.md`, with a link to the answer sheets. Earlier records show the style only; where one differs from this order, this file wins. Answer sheets -are not kept in this repository, and the marks are the record; the sheets, unedited, and the -instruction a delegated marker was given are posted in one or more consecutive comments on the -pull request, each linked from the record. +are not kept in this repository, and the marks are the record; the sheets, each the reader's +whole output, and the instruction a delegated marker was given are posted in one or more +comments on the pull request, each linked from the record, and never edited, hidden or deleted. The `Unreleased` changelog line for the change names the record, and `tools/changes.py` refuses a specification change whose pull request adds no file here. diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 91112fa..1568db2 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -1,6 +1,7 @@ # Answer key for TOOLING-SPEC.md -Marked against the document at the version named in the changelog entry that records the run. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, update the quotation here in the same commit. @@ -78,6 +79,6 @@ or the detector document's section [8](../../DETECTOR-SPEC.md#8-conformance) dif ## 4. Confusion No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to +finding that blocks until the text is changed and a fresh cohort no longer names it. What to change is the editor's decision; the reader is not asked and any suggestion it volunteers is discarded unread. From 1116138d92b0e55723d16871242e754643ee292f Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 21:08:05 +0100 Subject: [PATCH 013/105] Method 3.2 and detector 8: apply the rulings on review 3 Clause 3.2 drops the toolchain 4.1 exclusion, so a partly conforming semantic detector still counts; a stronger configuration of a detector the project runs counts, and one only a hosted service or licence tier provides does not. The duty to name the detector and kind is on a rule the practitioner writes, and the reason may go in the commit that first records it, for rules that predate the clause. Section 1's Semantic is now what a rule can ask: the type of any expression and what any name refers to, across the whole program; partial or single-file inference is Syntactic. Detector 8 names the kind the detector has without any hosted service or licence tier. Appendix A carries the detectors considered and the test for "has one", and the project prompt is rebuilt. index.md and PRIMER.md follow 3.2. The changelog lines follow. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 22 +++++++++++--------- DETECTOR-SPEC.md | 22 ++++++++++---------- PRIMER.md | 4 ++-- SPEC.md | 31 +++++++++++++++------------- defence-before-fix-project-prompt.md | 15 ++++++++------ index.md | 4 ++-- 6 files changed, 53 insertions(+), 45 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8e3b1c5..2bc3902 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,19 +16,21 @@ version entry with no cohort line has not been accepted and is not published. kind of detector that can express it, semantic, else syntactic, else textual, judged by the configuration it runs in; enabling or configuring an existing rule is not writing one. A weaker kind is allowed only where the stronger cannot express the rule or the language has no - detector of that kind that accepts bespoke rules, and a detector the toolchain specification's - clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) forbids routing a defence through does not count. The rule's remediation docs MUST - name its detector and kind, and for each step down from semantic the reason, with the stronger - detectors considered, MUST be recorded there and in the description of the commit that adds or - moves the rule. The practitioner judges it; a project MAY require the owner's sign-off, recorded + detector of that kind that accepts bespoke rules; a stronger configuration of a detector the + project runs counts, and one only a hosted service or licence tier provides does not. The + remediation docs of a rule the practitioner writes MUST name its detector and kind, and for + each step down from semantic the reason, with the stronger detectors considered, MUST be + recorded there and in the description of the commit that adds or moves the rule or first + records the reason. The practitioner judges it; a project MAY require the owner's sign-off, recorded where it records its calibrations, which section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) and the section [3](SPEC.md#3-the-method) map note. Section [1](SPEC.md#1-terminology) - defines textual, syntactic and semantic by whether a detector sees the types of values, section + defines textual, syntactic and semantic by whether a rule can ask the type of any expression + and what any name refers to across the whole program, section [7](SPEC.md#7-conformance)'s conforming defence carries the kind, and Appendix A carries the requirement. Prompted by project rules hosted in ast-grep, where the language already had a semantic host, that decided by names: they reported unrelated calls of the same name, and clause [3.2](SPEC.md#32-build-the-net) said any detector qualified whatever its category. The method question set gains load-bearing statement 2d, which - `ACCEPTANCE.md` now lists, and `README.md`, `PRIMER.md` and two register pages defer to the - clause. Accepted by run NNN. Minor version when released. + `ACCEPTANCE.md` now lists, and `index.md`, `README.md`, `PRIMER.md` and two register pages + defer to the clause. Accepted by run NNN. Minor version when released. ### 1.1.0, 2026-10-02 @@ -91,8 +93,8 @@ pass it. ### Unreleased - New obligation, section [8](DETECTOR-SPEC.md#8-conformance): a verdict on a detector MUST name the kind, semantic, syntactic - or textual, of each configuration it grades, crediting none reachable only through a hosted - service or licence tier. It is not a level of conformance and does not change the verdict; it + or textual, of each configuration it grades, as the detector has it without any hosted service + or licence tier. It is not a level of conformance and does not change the verdict; it is stated because method clause [3.2](SPEC.md#32-build-the-net) now requires a rule to be written in the strongest kind of detector that can express it, and a verdict of conforming alone presents a textual detector as an equal host. Section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) names it as the one obligation the document adds for a practitioner, diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index 0fa98df..e6a9cc2 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -35,16 +35,16 @@ construct the means has moved the obligation rather than met it. Terms from the method specification carry over unchanged. Every capitalised term links to its definition; the ones this document leans on most are, in short: -| Term | In one line | -| ---------------------------------- | ----------------------------------------------------------------------------------------------------- | -| [Detector] | A tool that reads code without executing it and reports what it finds | -| [Semantic], [Syntactic], [Textual] | What a [Detector] can see in a configuration: the types of values; a parse tree without them; or text | -| [Rule] | One check a [Detector] evaluates | -| [Identifier] | The stable name printed with a finding that leads to its documentation | -| [Practitioner] | Whoever is doing the work, a person or an [Agent] | -| [Toolchain] | Everything a project assembles to run its checks through, [Detectors] included | -| [Suppression] | Making a finding go away without fixing it | -| [Baseline] | A recorded set of existing findings a [Rule] is told to ignore | +| Term | In one line | +| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| [Detector] | A tool that reads code without executing it and reports what it finds | +| [Semantic], [Syntactic], [Textual] | What a [Rule] can ask in a configuration: the type of any expression and what any name refers to, program-wide; a parse tree short of that; or text | +| [Rule] | One check a [Detector] evaluates | +| [Identifier] | The stable name printed with a finding that leads to its documentation | +| [Practitioner] | Whoever is doing the work, a person or an [Agent] | +| [Toolchain] | Everything a project assembles to run its checks through, [Detectors] included | +| [Suppression] | Making a finding go away without fixing it | +| [Baseline] | A recorded set of existing findings a [Rule] is told to ignore | These are additional. @@ -309,7 +309,7 @@ for a [Detector]. **A verdict on a [Detector] MUST also name the kind of each configuration it grades, [Semantic], [Syntactic] or [Textual]**: ESLint, for example, is [Syntactic] without type information and -[Semantic] with it. It credits no configuration reachable only through a hosted service or a +[Semantic] with it. The kind named is the one the [Detector] has without any hosted service or licence tier, as clause 5.4 credits no finding reported only there. The kind is not a level of [Conformance] and does not change the verdict: a [Textual] [Detector] can [Conform], and a [Semantic] one can fail to. It is stated because the method specification's clause 3.2 requires a diff --git a/PRIMER.md b/PRIMER.md index e950a3f..732ee0e 100644 --- a/PRIMER.md +++ b/PRIMER.md @@ -28,8 +28,8 @@ it is an instance of coalescing an absent value into a falsy one, which is a pat patterns can be detected mechanically. Then write the rule that detects that pattern. A static analysis rule, a custom lint rule, a -check in a tool your project already runs on every commit, one that sees types where your -language has one. Something that reads the code +check in a tool your project already runs on every commit or your language's ecosystem offers, +one that sees types where your language has one. Something that reads the code rather than running it, so it can be pointed at every file you have rather than only the paths your tests happen to exercise. diff --git a/SPEC.md b/SPEC.md index c14cf61..2ca5647 100644 --- a/SPEC.md +++ b/SPEC.md @@ -82,11 +82,11 @@ Of a [Detector]: seeing the code only as text, as grep does. It sees how the cod #### Syntactic -Of a [Detector]: seeing the parse tree of the code but not the types of values, as ast-grep does. It sees the structure, but cannot tell apart two calls of the same name on values of different types. +Of a [Detector]: seeing the parse tree of the code, and at most some of the types of values, as ast-grep does. It sees the structure, but cannot always tell apart two calls of the same name on values of different types. #### Semantic -Of a [Detector]: seeing the types of values as the language's type system resolves them, and so what a name refers to, as PHPStan, mypy, Clippy, Dylint and ESLint with typescript-eslint's type information do. The same [Detector] can be [Syntactic] in one configuration and [Semantic] in another: ESLint without type information is [Syntactic]. A [Semantic] [Detector] need not accept bespoke [Rules]: Clippy does not, and in Rust a project's own [Semantic] [Rules] are written for Dylint. +Of a [Detector]: one in which a [Rule] can ask the type of any expression and what any name refers to, across the whole program, as the language's compiler or type checker resolves them, as in PHPStan, mypy, Clippy, Dylint and ESLint with typescript-eslint's type information. One that infers types only in part, or only within one file, is [Syntactic]. The same [Detector] can be [Syntactic] in one configuration and [Semantic] in another: ESLint without type information is [Syntactic]. A [Semantic] [Detector] need not accept bespoke [Rules]: Clippy does not, and in Rust a project's own [Semantic] [Rules] are written for Dylint. #### Rule @@ -416,14 +416,14 @@ weaker kind is allowed only where the stronger kind cannot express the [Rule], a [Rule] reads a file the [Semantic] [Detector] does not parse, or where the language has no [Detector] of the stronger kind that accepts bespoke [Rules]. The language has one when the project already runs it or the language's own [Detector] ecosystem offers it, the same check -section 2 asks for before a [Toolchain] gap is recorded, but not one through which the -[toolchain specification](TOOLING-SPEC.md)'s clause 4.1 forbids a [Toolchain] to route a -[Defence]. The [Practitioner] MUST name the [Detector] and its kind on the [Rule]'s -[Remediation docs]. Where that kind is weaker than [Semantic], the [Practitioner] MUST also +section 2 asks for before a [Toolchain] gap is recorded. A stronger configuration of a [Detector] +the project runs is one; a [Detector] or configuration of the stronger kind available only +through a hosted service or a licence tier is not. On the [Remediation docs] of a [Rule] they +write, the [Practitioner] MUST name the [Detector] and its kind. Where that kind is weaker than [Semantic], the [Practitioner] MUST also record, for each step down from [Semantic], which of those two reasons holds and why, naming the [Detectors] of the stronger kind that were considered, on the [Rule]'s [Remediation docs] and in -the description of the commit that adds the [Rule] or moves it to another [Detector]. The -[Practitioner] judges whether a reason holds, as section 4 gives them the choice of [Detector]; a +the description of the commit that adds the [Rule], moves it to another [Detector], or first +records the reason. The [Practitioner] judges whether a reason holds, as section 4 gives them the choice of [Detector]; a project MAY require its [Owner]'s sign-off on a weaker kind as well, recorded where it records its [Calibrations], and where it has recorded no such requirement none is needed. @@ -1092,12 +1092,15 @@ This appendix restates sections 2 to 4; where the two differ, the sections gover > for it, say so and fix the [Defect] conventionally. > > Write a custom [Rule] that detects the [Class], in a tool that reads code rather than running it. -> Put it in a [Semantic] [Detector], one that sees types and symbols; failing that, one that sees -> syntax; and only then one that sees text. Use a weaker one only where the stronger one cannot -> express the [Rule], or the language has none that takes custom [Rules]. Name the [Detector] and its -> kind in the [Rule]'s documentation. Where you use a weaker one, write down which reason holds, -> and why, there and in the description of the commit that adds the [Rule] or moves it to another -> [Detector], and where the project requires its [Owner]'s sign-off on a weaker one, get it. Draw +> Put it in a [Semantic] [Detector], one in which a [Rule] can ask the type of any expression and +> what any name refers to, across the whole program; failing that, one that sees syntax; and only +> then one that sees text. Use a weaker one only where the stronger one cannot express the [Rule], +> or the language has none that takes custom [Rules]: none the project runs, in any configuration, +> and none its ecosystem offers outside a hosted service or a licence tier. Name the [Detector] and +> its kind in the [Rule]'s documentation. Where you use a weaker one, write down which reason +> holds, and why, naming the stronger [Detectors] you considered, there and in the description of +> the commit that adds the [Rule], moves it to another [Detector], or first records the reason, +> and where the project requires its [Owner]'s sign-off on a weaker one, get it. Draw > the [Rule] so that it catches more than the single [Instance] you started from, but never so > broadly that it matches code which does not carry the [Hazard]. > diff --git a/defence-before-fix-project-prompt.md b/defence-before-fix-project-prompt.md index e5d044c..5508266 100644 --- a/defence-before-fix-project-prompt.md +++ b/defence-before-fix-project-prompt.md @@ -13,12 +13,15 @@ it. Do not decide in advance whether that is possible, attempt it. If you cannot for it, say so and fix the Defect conventionally. Write a custom Rule that detects the Class, in a tool that reads code rather than running it. -Put it in a Semantic Detector, one that sees types and symbols; failing that, one that sees -syntax; and only then one that sees text. Use a weaker one only where the stronger one cannot -express the Rule, or the language has none that takes custom Rules. Name the Detector and its -kind in the Rule's documentation. Where you use a weaker one, write down which reason holds, -and why, there and in the description of the commit that adds the Rule or moves it to another -Detector, and where the project requires its Owner's sign-off on a weaker one, get it. Draw +Put it in a Semantic Detector, one in which a Rule can ask the type of any expression and +what any name refers to, across the whole program; failing that, one that sees syntax; and only +then one that sees text. Use a weaker one only where the stronger one cannot express the Rule, +or the language has none that takes custom Rules: none the project runs, in any configuration, +and none its ecosystem offers outside a hosted service or a licence tier. Name the Detector and +its kind in the Rule's documentation. Where you use a weaker one, write down which reason +holds, and why, naming the stronger Detectors you considered, there and in the description of +the commit that adds the Rule, moves it to another Detector, or first records the reason, +and where the project requires its Owner's sign-off on a weaker one, get it. Draw the Rule so that it catches more than the single Instance you started from, but never so broadly that it matches code which does not carry the Hazard. diff --git a/index.md b/index.md index 39ce92d..354eab0 100644 --- a/index.md +++ b/index.md @@ -22,8 +22,8 @@ version. 1. **Attribute the defect to a class.** Not "what went wrong here" but "what kind of thing is this an instance of", within a lower bound and an upper bound that are both checkable. -2. **Build the net.** A rule that reads code, in whatever detector the toolchain offers, bespoke - where nothing off the shelf will take it. +2. **Build the net.** A rule that reads code, in the strongest kind of detector that can express + it, bespoke where nothing off the shelf will take it. 3. **Prove the net by making the rule fire.** Red before green, on the originating instance, committed before the fix. 4. **Sweep the codebase, then fix every instance.** The count is the finding; the fix addresses From 333a5de8a2f6d307d498e09f2c205d7031a9e215 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 21:08:33 +0100 Subject: [PATCH 014/105] Acceptance keys: follow the 3.2 rulings on review 3 The method key quotes the revised 3.2 sentences verbatim, in 2d and the application row. 2d drops the toolchain 4.1 exclusion and says which three elements a correct answer gives; the restatement key accepts a bullet that puts rules in a semantic detector; the application notes no longer count every 3.2 Yes against the document, leaving that to marking. The detector key quotes no changed sentence and is unchanged. Co-Authored-By: Claude Opus 5.5 --- acceptance/method/KEY.md | 57 ++++++++++++++++++++-------------------- 1 file changed, 29 insertions(+), 28 deletions(-) diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 64bbead..2d0a23a 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -24,7 +24,8 @@ bullet asserting an obligation the document does not state, is a correct restate Wrong if it asserts: the practitioner fixes the reported instance first and defends afterwards; a test may serve as the detector; an agent may baseline the remaining instances; only a semantic -detector may host a rule. +detector may host a rule. A bullet saying that rules go in a semantic detector, with no +fallback, names an obligation the document states and is correct. ## 2. Load-bearing statements @@ -46,37 +47,39 @@ detector may host a rule. rest on reproduction, not on the report". An answer that accepts the pull request description as the verdict is wrong, and so is one that states the six clauses and says nothing of reproduction. -- **2d.** The strongest kind that can express it: a semantic detector, one that sees the types of - values, else a syntactic one, else a textual one, judged by the configuration the rule runs in; - a weaker kind only where the stronger cannot express the rule or the language has no detector of - that kind that accepts bespoke rules (one the toolchain may not route a defence through does - not count), and then the practitioner records, for each step down from semantic, which reason - holds and why, naming the stronger detectors considered, on the rule's remediation docs and in - the description of the commit that adds or moves the rule. Key quotations, [3.2](../../SPEC.md#32-build-the-net): "A Rule the +- **2d.** The strongest kind that can express it: a semantic detector, else a syntactic one, else + a textual one, judged by the configuration the rule runs in; a weaker kind only where the + stronger cannot express the rule or the language has no detector of that kind that accepts + bespoke rules, and then the practitioner records, for each step down from semantic, which + reason holds and why, naming the stronger detectors considered, on the rule's remediation docs + and in the description of the commit that adds the rule, moves it, or first records the + reason. Key quotations, [3.2](../../SPEC.md#32-build-the-net): "A Rule the Practitioner writes MUST be written in the strongest kind of Detector that can express it: a Semantic one, else a Syntactic one, else a Textual one."; "Where that kind is weaker than Semantic, the Practitioner MUST also record, for each step down from Semantic, which of those two reasons holds and why, naming the Detectors of the stronger kind that were considered, on - the Rule's Remediation docs and in the description of the commit that adds the Rule or moves it - to another Detector." Supporting: "The Practitioner MUST name the Detector and its kind on the - Rule's Remediation docs." The answer must put the semantic kind first. An answer naming the - recorded reason in one of the two places is correct. An answer that says any detector is as - good as another, that the semantic kind is only - preferred and may be passed over for another reason, that a weaker kind is never allowed, that - the owner must always approve a weaker kind (the practitioner judges it unless the project - requires the owner's sign-off), or that names the kind and says nothing of the recorded reason, - is wrong. + the Rule's Remediation docs and in the description of the commit that adds the Rule, moves it to + another Detector, or first records the reason." Supporting: "On the Remediation docs of a Rule + they write, the Practitioner MUST name the Detector and its kind." A correct answer gives three + elements, and may leave out everything else above: (i) the strongest kind that can express the + rule, in the order semantic, syntactic, textual; (ii) a weaker kind only where the stronger + cannot express the rule or the language has no detector of that kind that takes bespoke rules, + both grounds; (iii) the reason recorded, in either of the two places. An answer that misses one + of the three, that says any detector is as good as another, that the semantic kind is only + preferred and may be passed over for another reason, that a weaker kind is never allowed, or + that the owner must always approve a weaker kind (the practitioner judges it unless the + project requires the owner's sign-off), is wrong. ## 3. Application: the ledger remediation -| Clause | Intended verdict | Reasoning the fixture supports | -| ----------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | Class, hazard and the next wider rule are named; Partial or No is accepted where the reader notes that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | -| [3.2](../../SPEC.md#32-build-the-net) | Partial or No | A rule in the project's static analyser, reading code, not a test, but the wiki page of commit D names neither the detector nor its kind; [3.2](../../SPEC.md#32-build-the-net): "The Practitioner MUST name the Detector and its kind on the Rule's Remediation docs." The fixture does not say what the analyser can see, so no reason for a weaker kind is shown to be owed | -| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | Commit B is red on the fixture and the originating instance and survives as its own commit | -| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Five findings recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority; Partial accepted where the reasoning names that suppression | -| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | The rule is permanent and blocking, but the suppression was not a recorded decision; [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | -| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule; Partial accepted where the reasoning quotes the versioning sentence | +| Clause | Intended verdict | Reasoning the fixture supports | +| ----------------------------------------------------------------------------------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | Class, hazard and the next wider rule are named; Partial or No is accepted where the reader notes that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | +| [3.2](../../SPEC.md#32-build-the-net) | Partial or No | A rule in the project's static analyser, reading code, not a test, but the wiki page of commit D names neither the detector nor its kind; [3.2](../../SPEC.md#32-build-the-net): "On the Remediation docs of a Rule they write, the Practitioner MUST name the Detector and its kind." The fixture does not say what the analyser can see, so no reason for a weaker kind is shown to be owed | +| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | Commit B is red on the fixture and the originating instance and survives as its own commit | +| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Five findings recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority; Partial accepted where the reasoning names that suppression | +| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | The rule is permanent and blocking, but the suppression was not a recorded decision; [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | +| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule; Partial accepted where the reasoning quotes the versioning sentence | Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**, because [3.2](../../SPEC.md#32-build-the-net), [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) and [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) were not followed. The **defence does not conform** either, because [3.2](../../SPEC.md#32-build-the-net) and [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) fail; a reader who names either is correct. @@ -93,9 +96,7 @@ A reader who grades [3.5](../../SPEC.md#35-enforce-permanently-and-block) Yes be sentence on unrecorded suppression, and that counts against the document. A reader who grades [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) Yes because the URL resolves has read the versioning sentence of [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) differently, which counts against the document; a reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the -fixture. A reader who grades [3.2](../../SPEC.md#32-build-the-net) Yes has not applied the sentence requiring the remediation -docs to name the detector and its kind, and that counts against the document. A reader who grades -[3.2](../../SPEC.md#32-build-the-net) Partial or No only because no reason for a weaker kind is recorded has the verdict on a +fixture. A reader who grades [3.2](../../SPEC.md#32-build-the-net) Partial or No only because no reason for a weaker kind is recorded has the verdict on a ground the fixture does not show: the reason is owed only where the rule is in a weaker kind than semantic. That is not marked wrong, and counts against the document only if the reasoning quotes [3.2](../../SPEC.md#32-build-the-net) as requiring the reason whatever the detector. A reader who attributes From dba74615373fbce126b640e38275227d8a64f33a Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 21:13:54 +0100 Subject: [PATCH 015/105] Acceptance process: apply the editor's rulings on review 4 A question 1 bullet that quotes the document makes its miss traceable; a reasoning that names the clause a case fails or meets states what the document requires; a question 2 quotation must itself state part of the key's answer. The fourth row matches a fresh reader's passage against the earlier passage's paragraph as it reads after the change, the record quotes confusing passages verbatim, and a delegated marker gets the earlier runs' documents and the three facts it was missing. An edited comment voids only a run not already failed, before merge. A process change applies to runs after it merges, or to earlier runs made under it at a named commit. A set-aside reader's traceable miss fails the run. The keys count passages named, not only quoted; every keyed answer of question 3 is a keyed item; "the same passage" covers the sentences an occurrence of the whole quotation overlaps. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 81 ++++++++++++++++++++----------------- acceptance/CHANGES.md | 23 ++++++----- acceptance/detector/KEY.md | 8 ++-- acceptance/method/KEY.md | 8 ++-- acceptance/runs/README.md | 5 ++- acceptance/toolchain/KEY.md | 8 ++-- 6 files changed, 73 insertions(+), 60 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index bab883d..7b5663a 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -30,7 +30,8 @@ test is run with a low-strength model so that clarity is measured at the floor, count; no reader is set aside for any other cause, and the record shows the cause. If any reader is set aside, the run is void and a fresh cohort runs, unless the readers whose answers count already fail a criterion, when the run fails; a run never passes with a reader set - aside. A void run takes a run number, has a record and posted sheets like any other, and + aside. A set-aside reader's sheet is still marked, and a miss traceable to the text in it fails + the run. A void run takes a run number, has a record and posted sheets like any other, and counts towards the number of runs. - Each reader receives **only its packet**: the document's question set, the changed document @@ -96,17 +97,21 @@ separate marker that has seen neither the drafting nor the review. Whoever marks in this file. A delegated marker receives only these: the document, its question set, the fixture the questions name and its key, each at the commit the readers read; this file and `acceptance/runs/README.md`; the earlier run records of the same pull request, and those of -other pull requests as examples of style only; the answer sheets; and, from the editor, the -facts the record states that the sheets do not, which are the pull request and the commit, the -cohort size and model, how the readers were isolated and the command they were started with, the -readers set aside and why, how the marker was started, and the run number. The marker writes the record, stating those facts as the editor supplied them, and the -editor adds the findings applied and the link to the posted sheets after marking. A delegated -marker's verdict, its marks and its PASS or FAIL, is final unless the editor shows that a rule -in this file was misapplied; the editor then re-marks what the misapplied rule affected, and the -record says which rule and why. Each sheet is marked question by question, as follows. +other pull requests as examples of style only; the document at the commit each earlier run of +the pull request read; the answer sheets; and, from the editor, the facts the record states that +the sheets do not, which are the pull request and the commit, the cohort size and model, how the +readers were isolated and the command they were started with, whether the machine applies +managed (policy) settings to the readers, that the editor attests the rest of the isolation, the +readers set aside and why, how the marker was started, the run number, and whether the run was +made under a process change before that change merged and, if so, at which commit. The marker +writes the record, stating those facts as the editor supplied them, and the editor adds the +findings applied and the link to the posted sheets after marking. A delegated marker's verdict, +its marks and its PASS or FAIL, is final unless the editor shows that a rule in this file was +misapplied; the editor then re-marks what the misapplied rule affected, and the record says +which rule and why. Each sheet is marked question by question, as follows. - **Keyed items.** Question 1 as a whole is one keyed item, and so is each item of question 2 - and each clause verdict and each summary verdict of question 3. + and each answer of question 3 for which the key gives an intended answer. - **Quoting.** A reader quotes the document when it gives consecutive words of the document, in the document's order and of any length, as a quotation, in quotation marks or a block quote. The words are verbatim apart from letter case, quotation-mark or apostrophe style, and markdown @@ -114,24 +119,27 @@ record says which rule and why. Each sheet is marked question by question, as fo - **Questions 1 and 3** are marked as the key states for each item. - **Question 2.** An answer is correct when its sentence states each thing the key's answer states, unless the key says it may be left out, asserts nothing the key says makes an answer - wrong, and quotes the document in support. The quotation need not be the one the key gives. + wrong, and quotes in support words of the document that by themselves state at least one + thing the key's answer states. The quotation need not be the one the key gives. - **A miss traceable to the text**, under questions 2 and 3, is a miss whose reasoning quotes the document or states, in the reader's own words, what the document says or requires. Such a miss is traceable even where it also misreads the fixture, and whatever the key says of it. Only a miss whose reasoning states nothing about the document, such as a misread fixture alone, is the - reader's. A miss under any question is also traceable where the key says it counts against the - document: the key can add to these misses, never take one away. Any other miss on question 1 - is the reader's. + reader's. A reasoning that only reports facts of the fixture, attributing no requirement, + condition or meaning to the document or a clause of it, states nothing about the document; + naming the clause a case fails or meets attributes a requirement to it. A miss under any + question is also traceable where the key says it counts against the document: the key can add + to these misses, never take one away. A miss on question 1 is also traceable where the bullet + it turns on quotes the document; any other miss on question 1 is the reader's. - **An unanswered question.** Each keyed item left unanswered is one miss, not traceable to the text, so a sheet that leaves three items of question 3 blank has three misses. Question 4 left unanswered counts as no passage quoted. A sheet with any answer text is marked as it stands, whatever its process returned; one whose answer text is empty is set aside, as [The cohort](#the-cohort) says. -- **The same passage.** A quotation under question 4, of any length, covers every sentence - containing any part of the quoted words taken as one run of words, in every place that run - appears, and a passage named without a quotation, by clause number or description, covers - every sentence of what it names. A heading, - list item or table row counts as one sentence. Two passages are the same when they cover a +- **The same passage.** A quotation under question 4, of any length, covers every sentence that + an occurrence of the quoted words, taken as one run, overlaps, at every place that run occurs, + and a passage named without a quotation, by clause number or description, covers every + sentence of what it names. A heading, list item or table row counts as one sentence. Two passages are the same when they cover a sentence in common or lie in the same paragraph, and a list item or table row is its own paragraph. @@ -145,13 +153,13 @@ The change passes when all of the following hold on a single run. The thresholds readers whose answers count; with five such readers, "every reader" is five of five and "all but one" is four of five. -| Criterion | Threshold | -| ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Load-bearing statements (question 2) | **Every** reader correct on every one | -| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | -| A miss traceable to the text | **None.** Any such miss, as [Marking](#marking) defines it, is a finding, whatever the count | -| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it: no reader of the fresh cohort names a passage of the changed text that is the same as it, as [Marking](#marking) defines the same passage | -| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | +| Criterion | Threshold | +| ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Load-bearing statements (question 2) | **Every** reader correct on every one | +| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | +| A miss traceable to the text | **None.** Any such miss, as [Marking](#marking) defines it, is a finding, whatever the count | +| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it: no reader of the fresh cohort names a passage that covers a sentence of the paragraph the earlier passage lay in, as that paragraph reads in the changed document, or a sentence the change kept from it | +| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | Every row must hold, and the last covers only what none of the others catches. The first, third and fourth rows block on what a single reader, or for the fourth two readers, did, and the second @@ -203,10 +211,10 @@ readers were isolated and the command they were started with, whether the machin managed (policy) settings to the readers, that the editor attests the rest of the isolation, the readers set aside and the cause, the run number in that pull request, counting void runs, who marked and, for a delegated marker, how it was started, the marks per reader per question, any -judgement made in marking, on a re-mark which rule and why, the passages quoted as confusing and -by how many readers, the findings applied, whether it was made under a process change before -that change merged, whether it passed, and a link to the answer sheets. The answer sheets are -not kept in this repository: every reader's sheet, which is its whole output as its process +judgement made in marking, on a re-mark which rule and why, the passages quoted as confusing, +each with the words quoted, verbatim, and its clause, and by how many readers, the findings +applied, whether it was made under a process change before that change merged and, if so, at +which commit, whether it passed, and a link to the answer sheets. The answer sheets are not kept in this repository: every reader's sheet, which is its whole output as its process returned it (the JSON, where the command asks for JSON), with a set-aside reader's marked as set aside, and the instruction a delegated marker was given, which is its task prompt and the agent definition it ran under, are posted in one or more comments on the pull request, each linked @@ -214,8 +222,9 @@ from the record, so that an owner can check the marks. The comments also hold th every reader received and the command and the form of the working directory the readers were started in; a sheet cannot show how its process was started, which is why the editor attests the rest of the isolation. The comments are never edited, hidden or deleted after posting; if one -is, the run is void and a fresh cohort runs. The changelog line for the change names the file. CI refuses a specification change whose pull -request adds no run file; it checks for at least one, and a pull request that changes two +is before the change merges, the run is void and a fresh cohort runs, unless the run had already +failed, when it stays failed. The changelog line for the change names the file. CI refuses a +specification change whose pull request adds no run file; it checks for at least one, and a pull request that changes two documents adds a run for each. Each published version's changelog entry carries the lines of the changes it contains, so a @@ -232,10 +241,10 @@ a change to this process. They are not specifications, and a change to them is n cohort: the cohort measures whether a low-strength reader can follow a specification, and the process is followed by the editor. Such a change names the runs or the case that showed the need for it, and goes through steps 1, 3 and 6 of `CONTRIBUTING.md`, with no scope, key or cohort -step; its review reads every process file it changes. It names the run number it applies from; -a run under it made before it merges follows all of it, never part, and its record says so. It -is recorded by a line added at -the top of [acceptance/CHANGES.md](acceptance/CHANGES.md), naming its pull request, not in the +step; its review reads every process file it changes. It applies to every run made after it +merges; a run made before then may be made under it, follows all of it at a named commit, never +part, and its record says so and names that commit. It is recorded by a line added at the top +of [acceptance/CHANGES.md](acceptance/CHANGES.md), naming its pull request, not in the changelog, which is for the three specifications. It is bound by [the section on the bar](#the-bar-only-moves-up): it may raise a criterion, never relax one. A change to a question set, key or fixture is not a change to this process; `acceptance/README.md` diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 69550db..b555ee6 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -6,32 +6,35 @@ not a specification and is not versioned, and a change to it is not tested by a to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). - Pull request #PR: the cohort can be run and marked without judgement calls that the rules - leave open; this applies from run 19, and runs 16 to 18 stand as recorded. A cohort is an odd + leave open; this applies to runs made after it merges, and to run 19 if made under it, and + runs 16 to 18 stand as recorded. A cohort is an odd number of readers started, at least five, larger at the editor's choice the more obligations the change touches, each a separate process with no tools and no project context, started from an empty directory outside any repository with no user, project or local settings files, `CLAUDE.md`, skills, plugins, hooks, MCP servers or saved session. A reader is set aside only if its answer text is empty or it shared the editor's context, and a run with a reader set - aside is void, or fails if the readers who count already fail it; a void run has a number, a - record and posted sheets. The packet is the question set, the document and the fixture, each + aside is void, or fails if the readers who count already fail it or its own sheet has a miss + traceable to the text; a void run has a number, a record and posted sheets. The packet is the question set, the document and the fixture, each after one `===== FILE: =====` line. A new marking section says what a delegated marker receives and that its verdict is final unless a rule was misapplied, that open judgements take the stricter reading, and defines a keyed item, quoting (consecutive words of any length, - verbatim apart from case, quotation-mark style and markup), a correct load-bearing answer, a - miss traceable to the text, which a key can add to but never remove, an unanswered item and - the same passage. The thresholds apply to the readers whose answers count and block on a single + verbatim apart from case, quotation-mark style and markup), a correct load-bearing answer, + whose quotation must itself state part of the key's answer, a miss traceable to the text, + including a question 1 bullet that quotes the document, which a key can add to but never + remove, an unanswered item and the same passage. The thresholds apply to the readers whose answers count and block on a single reader for the first, third and fourth rows, and the fourth row says what a fresh cohort no longer naming a passage means. A fresh cohort runs after every change following a failure, every void run and every wording change, which is any change to the text after the readers read it except a version number or date the editor sets for a release. The record adds the model version each reader reported, the isolation, managed settings, readers set aside, void - runs and re-marks, and links each reader's whole output and the marker's instruction, posted on - the pull request and never edited, hidden or deleted, or the run is void. The keys are marked at + runs, re-marks and each confusing passage's words verbatim, and links each reader's whole + output and the marker's instruction, posted on the pull request and never edited, hidden or + deleted, or a run not already failed is void. The keys are marked at the commit the readers read and no longer limit which misses count against the document, removing such a limit is named as raising the bar, and the method key's "half right" is a miss. The process files include the checks that enforce them, and a process change goes through - steps 1, 3 and 6 of `CONTRIBUTING.md`, names the run it applies from and is recorded here, not - in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the change, points to + steps 1, 3 and 6 of `CONTRIBUTING.md`, applies to runs made after it merges, or earlier + ones made under it at a named commit, and is recorded here, not in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the change, points to the definition of a wording change, and adds the rebuild of the files `build-agent-surfaces.py` writes after an Appendix A change. Prompted by gaps in the text as it stood: `ACCEPTANCE.md` named no fixture in what a reader receives whilst `acceptance/README.md` did, the thresholds diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index ee01416..8608f7d 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -68,7 +68,7 @@ own identifiers in CI is a missed SHOULD, not a further MUST failure; the MUST t ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer names it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage quoted. Two or more readers naming the same passage, as +`ACCEPTANCE.md` defines it, is a finding that blocks until the text is changed and a fresh +cohort no longer names it. What to change is the editor's decision; the reader is not asked +and any suggestion it volunteers is discarded unread. diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 837b819..2e37b07 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -72,7 +72,7 @@ passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as cove ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer names it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage quoted. Two or more readers naming the same passage, as +`ACCEPTANCE.md` defines it, is a finding that blocks until the text is changed and a fresh +cohort no longer names it. What to change is the editor's decision; the reader is not asked +and any suggestion it volunteers is discarded unread. diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index 4af8149..2d0b96a 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -12,8 +12,9 @@ machine applies managed (policy) settings to the readers; that the editor attest isolation; the readers set aside and the cause; the run number within that pull request, counting void runs; who marked and, for a delegated marker, how it was started; the marks per reader per question against the key, with any judgement made in marking and, on a re-mark, which rule and -why; the passages quoted as confusing and by how many readers; the findings applied to the text -as a result; whether the run was made under a process change before that change merged; and the +why; the passages quoted as confusing, each with the words quoted, verbatim, and its clause, and +by how many readers; the findings applied to the text as a result; whether the run was made +under a process change before that change merged and, if so, at which commit; and the verdict against the criteria in `ACCEPTANCE.md`, with a link to the answer sheets. Earlier records show the style only; where one differs from this order, this file wins. Answer sheets are not kept in this repository, and the marks are the record; the sheets, each the reader's diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 1568db2..cae0efc 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -78,7 +78,7 @@ or the detector document's section [8](../../DETECTOR-SPEC.md#8-conformance) dif ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer names it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage quoted. Two or more readers naming the same passage, as +`ACCEPTANCE.md` defines it, is a finding that blocks until the text is changed and a fresh +cohort no longer names it. What to change is the editor's decision; the reader is not asked +and any suggestion it volunteers is discarded unread. From 97d1d53df87bfd7e5003b682e6f378632e512d92 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 21:19:46 +0100 Subject: [PATCH 016/105] Method 3.2 and 3.6: apply the rulings on review 4 Clause 3.2 defines when a detector accepts bespoke rules (a documented extension point, whether or not it meets the rest of the detector specification) and says a rule its extension points cannot reach is one it cannot express. The owner's sign-off, where a project requires it, is required by a record beside the calibrations and is itself recorded with the reason. Clause 3.6's three things defer to what 3.2 adds. Appendix A and the project prompt carry the licence exclusion for a configuration the project runs, and the recording of the sign-off. The mypy register note no longer steers away from a mypy plugin as the semantic host. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 10 +++++++--- SPEC.md | 27 +++++++++++++++++---------- defence-before-fix-project-prompt.md | 13 +++++++------ tools/mypy.md | 2 +- 4 files changed, 32 insertions(+), 20 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2bc3902..d177d25 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,12 +17,16 @@ version entry with no cohort line has not been accepted and is not published. configuration it runs in; enabling or configuring an existing rule is not writing one. A weaker kind is allowed only where the stronger cannot express the rule or the language has no detector of that kind that accepts bespoke rules; a stronger configuration of a detector the - project runs counts, and one only a hosted service or licence tier provides does not. The + project runs counts, and one only a hosted service or licence tier provides does not. A + detector accepts bespoke rules when the project can add its own through a documented extension + point, and a rule its extension points cannot reach is one it cannot express. The remediation docs of a rule the practitioner writes MUST name its detector and kind, and for each step down from semantic the reason, with the stronger detectors considered, MUST be recorded there and in the description of the commit that adds or moves the rule or first - records the reason. The practitioner judges it; a project MAY require the owner's sign-off, recorded - where it records its calibrations, which section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) and the section [3](SPEC.md#3-the-method) map note. Section [1](SPEC.md#1-terminology) + records the reason. The practitioner judges it; a project MAY require the owner's sign-off by + recording that requirement where it records its calibrations, the sign-off then recorded with + the reason, which section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) and the section [3](SPEC.md#3-the-method) map note. Clause [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation)'s list of what the + remediation docs state defers to what clause [3.2](SPEC.md#32-build-the-net) adds. Section [1](SPEC.md#1-terminology) defines textual, syntactic and semantic by whether a rule can ask the type of any expression and what any name refers to across the whole program, section [7](SPEC.md#7-conformance)'s conforming defence carries the kind, and Appendix A carries the requirement. Prompted by diff --git a/SPEC.md b/SPEC.md index 2ca5647..0067657 100644 --- a/SPEC.md +++ b/SPEC.md @@ -418,14 +418,19 @@ weaker kind is allowed only where the stronger kind cannot express the [Rule], a project already runs it or the language's own [Detector] ecosystem offers it, the same check section 2 asks for before a [Toolchain] gap is recorded. A stronger configuration of a [Detector] the project runs is one; a [Detector] or configuration of the stronger kind available only -through a hosted service or a licence tier is not. On the [Remediation docs] of a [Rule] they +through a hosted service or a licence tier is not. A [Detector] accepts bespoke [Rules] when the +project can add a [Rule] of its own to it through a documented extension point, such as a plugin +interface or an API for writing [Rules], whether or not it meets the rest of the [detector specification](DETECTOR-SPEC.md); a [Rule] whose +construct its extension points cannot reach is one that [Detector] cannot express, and one its +kind cannot express where no other [Detector] of that kind reaches it. On the [Remediation docs] of a [Rule] they write, the [Practitioner] MUST name the [Detector] and its kind. Where that kind is weaker than [Semantic], the [Practitioner] MUST also record, for each step down from [Semantic], which of those two reasons holds and why, naming the [Detectors] of the stronger kind that were considered, on the [Rule]'s [Remediation docs] and in the description of the commit that adds the [Rule], moves it to another [Detector], or first records the reason. The [Practitioner] judges whether a reason holds, as section 4 gives them the choice of [Detector]; a -project MAY require its [Owner]'s sign-off on a weaker kind as well, recorded where it records -its [Calibrations], and where it has recorded no such requirement none is needed. +project MAY require its [Owner]'s sign-off on a weaker kind as well, by recording that +requirement where it records its [Calibrations]; the sign-off is then recorded with the reason, +and where the project has recorded no such requirement none is needed. **A worked example.** Take a [Rule] against calling a method named `context` on one type. A [Detector] that cannot see types reports every call named `context`, whatever type it is called @@ -749,7 +754,8 @@ pressure, which is the condition under which the original [Defect] was written. The failure [Message] MUST be terse, and it MUST carry a stable [Identifier] that resolves to [Remediation docs] shipped with the project. -That documentation MUST state three things: what the [Rule] is about, why it exists, and how to +That documentation MUST state three things, besides what clause 3.2 adds for a [Rule] the +[Practitioner] writes: what the [Rule] is about, why it exists, and how to fix a violation correctly using the project's preferred approach. A [Practitioner] who finds that an existing [Rule]'s printed [Identifier] does not resolve records @@ -1095,12 +1101,13 @@ This appendix restates sections 2 to 4; where the two differ, the sections gover > Put it in a [Semantic] [Detector], one in which a [Rule] can ask the type of any expression and > what any name refers to, across the whole program; failing that, one that sees syntax; and only > then one that sees text. Use a weaker one only where the stronger one cannot express the [Rule], -> or the language has none that takes custom [Rules]: none the project runs, in any configuration, -> and none its ecosystem offers outside a hosted service or a licence tier. Name the [Detector] and -> its kind in the [Rule]'s documentation. Where you use a weaker one, write down which reason -> holds, and why, naming the stronger [Detectors] you considered, there and in the description of -> the commit that adds the [Rule], moves it to another [Detector], or first records the reason, -> and where the project requires its [Owner]'s sign-off on a weaker one, get it. Draw +> or the language has none that takes custom [Rules]: none the project runs, in any configuration +> it can use without a hosted service or a licence tier, and none its ecosystem offers outside +> one. Name the [Detector] and its kind in the [Rule]'s documentation. Where you use a weaker one, +> write down which reason holds, and why, naming the stronger [Detectors] you considered, there and +> in the description of the commit that adds the [Rule], moves it to another [Detector], or first +> records the reason, and where the project requires its [Owner]'s sign-off on a weaker one, get +> it and record it with the reason. Draw > the [Rule] so that it catches more than the single [Instance] you started from, but never so > broadly that it matches code which does not carry the [Hazard]. > diff --git a/defence-before-fix-project-prompt.md b/defence-before-fix-project-prompt.md index 5508266..8b55913 100644 --- a/defence-before-fix-project-prompt.md +++ b/defence-before-fix-project-prompt.md @@ -16,12 +16,13 @@ Write a custom Rule that detects the Class, in a tool that reads code rather tha Put it in a Semantic Detector, one in which a Rule can ask the type of any expression and what any name refers to, across the whole program; failing that, one that sees syntax; and only then one that sees text. Use a weaker one only where the stronger one cannot express the Rule, -or the language has none that takes custom Rules: none the project runs, in any configuration, -and none its ecosystem offers outside a hosted service or a licence tier. Name the Detector and -its kind in the Rule's documentation. Where you use a weaker one, write down which reason -holds, and why, naming the stronger Detectors you considered, there and in the description of -the commit that adds the Rule, moves it to another Detector, or first records the reason, -and where the project requires its Owner's sign-off on a weaker one, get it. Draw +or the language has none that takes custom Rules: none the project runs, in any configuration +it can use without a hosted service or a licence tier, and none its ecosystem offers outside +one. Name the Detector and its kind in the Rule's documentation. Where you use a weaker one, +write down which reason holds, and why, naming the stronger Detectors you considered, there and +in the description of the commit that adds the Rule, moves it to another Detector, or first +records the reason, and where the project requires its Owner's sign-off on a weaker one, get +it and record it with the reason. Draw the Rule so that it catches more than the single Instance you started from, but never so broadly that it matches code which does not carry the Hazard. diff --git a/tools/mypy.md b/tools/mypy.md index b5bc601..e9e01d7 100644 --- a/tools/mypy.md +++ b/tools/mypy.md @@ -43,4 +43,4 @@ Clause [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-writ ## Notes for a practitioner -Treat mypy as a detector for type-shaped classes only: where the class is "this value can be None here", tightening the configuration is a legitimate off-the-shelf rule under method clause [3.2](../SPEC.md#32-build-the-net), proven by watching the count go red. For any other class, choose the host under method clause [3.2](../SPEC.md#32-build-the-net), which asks for the strongest kind of detector that can express the rule and a recorded reason for a weaker one, and enable `warn_unused_ignores` plus a check in a second detector on new `# type: ignore` comments so the inline route stays visible. +Treat mypy as a detector for type-shaped classes: where the class is "this value can be None here", tightening the configuration is a legitimate off-the-shelf rule under method clause [3.2](../SPEC.md#32-build-the-net), proven by watching the count go red. For any other class, a mypy plugin can host a rule its hooks reach, and method clause [3.2](../SPEC.md#32-build-the-net) asks for that semantic host before a weaker kind; a rule the hooks cannot reach goes to a weaker detector, with the reason the clause requires recorded. Enable `warn_unused_ignores` plus a check in a second detector on new `# type: ignore` comments so the inline route stays visible. From fc3b0f967a7f5f4d513e2028cee6620b86bb1a1a Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 21:20:00 +0100 Subject: [PATCH 017/105] Acceptance keys: follow the rulings on review 4 Method key, question 1: a bullet naming the semantic kind without the fallback is correct, one saying no weaker kind is ever allowed is wrong. Statement 2d (iii): naming one of the two places is enough; presenting them as alternatives is wrong. Application 3.2: a Yes is marked wrong, including one that leaves the wiki failure to 3.6 alone, and counts against the document where the reasoning quotes it. No quoted sentence changed, and no question, threshold or fixture. Co-Authored-By: Claude Opus 5.5 --- acceptance/method/KEY.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 2d0a23a..c2809e4 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -24,8 +24,8 @@ bullet asserting an obligation the document does not state, is a correct restate Wrong if it asserts: the practitioner fixes the reported instance first and defends afterwards; a test may serve as the detector; an agent may baseline the remaining instances; only a semantic -detector may host a rule. A bullet saying that rules go in a semantic detector, with no -fallback, names an obligation the document states and is correct. +detector may host a rule. A bullet that names the semantic kind and does not mention the fallback +is correct; a bullet that says no weaker kind is ever allowed is wrong. ## 2. Load-bearing statements @@ -64,7 +64,8 @@ fallback, names an obligation the document states and is correct. elements, and may leave out everything else above: (i) the strongest kind that can express the rule, in the order semantic, syntactic, textual; (ii) a weaker kind only where the stronger cannot express the rule or the language has no detector of that kind that takes bespoke rules, - both grounds; (iii) the reason recorded, in either of the two places. An answer that misses one + both grounds; (iii) the reason recorded; naming one of the two places is enough, and an answer + that presents them as alternatives is wrong. An answer that misses one of the three, that says any detector is as good as another, that the semantic kind is only preferred and may be passed over for another reason, that a weaker kind is never allowed, or that the owner must always approve a weaker kind (the practitioner judges it unless the @@ -99,7 +100,10 @@ differently, which counts against the document; a reader who grades [3.4](../../ fixture. A reader who grades [3.2](../../SPEC.md#32-build-the-net) Partial or No only because no reason for a weaker kind is recorded has the verdict on a ground the fixture does not show: the reason is owed only where the rule is in a weaker kind than semantic. That is not marked wrong, and counts against the document only if the reasoning quotes -[3.2](../../SPEC.md#32-build-the-net) as requiring the reason whatever the detector. A reader who attributes +[3.2](../../SPEC.md#32-build-the-net) as requiring the reason whatever the detector. A reader who grades [3.2](../../SPEC.md#32-build-the-net) Yes is marked +wrong, including one who assigns the wiki page's failure to [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) alone: a rule with no +conforming remediation docs has its detector and kind named on none, so [3.2](../../SPEC.md#32-build-the-net) fails either +way, and the miss counts against the document where the reasoning quotes it. A reader who attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as covering it, which is a fair reading and is not marked wrong. ## 4. Confusion From 5a863aeb9d75ec83b27995359d200b1bb8d2b430 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 22:49:13 +0100 Subject: [PATCH 018/105] Acceptance process: apply the editor's rulings on review 5 - Question 2: a correct answer also asserts nothing the document does not state, so a wrong claim the key does not list is no longer correct. - A reader is set aside when its process reports an error, when the command or working directory recorded for it is not one The cohort requires, or when its reported model version is not the cohort's model; the existing void rule applies. The unanswered-question bullet agrees. - CONTRIBUTING.md step 5 sets the status paragraph and citation to the -dev version with the header, so the version exception covers them. - The sentence after the pass-criteria table agrees with the fourth row. - A delegated marker of a run made under an unmerged process change receives ACCEPTANCE.md and acceptance/runs/README.md at the named commit. - acceptance/CHANGES.md line updated to match. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 27 ++++++++++++++++----------- CONTRIBUTING.md | 3 ++- acceptance/CHANGES.md | 11 +++++++---- 3 files changed, 25 insertions(+), 16 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 7b5663a..8afed4b 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -25,9 +25,12 @@ test is run with a low-strength model so that clarity is measured at the floor, ``` The JSON output gives the model version each reader reported. A reader isolated in this way is - a cold reader. A reader whose answer text is empty, or that shared the editor's context, such - as an agent dispatched from the editor's own session, is set aside and its answers do not - count; no reader is set aside for any other cause, and the record shows the cause. If any + a cold reader. A reader is set aside, and its answers do not count, when its answer text is + empty; when its process reports an error, for example `is_error` in its JSON output or an API + error message in place of answers; when it shared the editor's context, such as an agent + dispatched from the editor's own session; when the command or working directory recorded for + it is not one this section requires; or when the model version it reported is not the + cohort's model. No reader is set aside for any other cause, and the record shows the cause. If any reader is set aside, the run is void and a fresh cohort runs, unless the readers whose answers count already fail a criterion, when the run fails; a run never passes with a reader set aside. A set-aside reader's sheet is still marked, and a miss traceable to the text in it fails @@ -96,7 +99,8 @@ The editor marks each answer sheet against the document's key, or delegates the separate marker that has seen neither the drafting nor the review. Whoever marks is the marker in this file. A delegated marker receives only these: the document, its question set, the fixture the questions name and its key, each at the commit the readers read; this file and -`acceptance/runs/README.md`; the earlier run records of the same pull request, and those of +`acceptance/runs/README.md`, at the named commit where the run is made under a process change +before that change merges; the earlier run records of the same pull request, and those of other pull requests as examples of style only; the document at the commit each earlier run of the pull request read; the answer sheets; and, from the editor, the facts the record states that the sheets do not, which are the pull request and the commit, the cohort size and model, how the @@ -118,8 +122,8 @@ which rule and why. Each sheet is marked question by question, as follows. link or emphasis markup. Naming a clause number, or paraphrasing, is not quoting. - **Questions 1 and 3** are marked as the key states for each item. - **Question 2.** An answer is correct when its sentence states each thing the key's answer - states, unless the key says it may be left out, asserts nothing the key says makes an answer - wrong, and quotes in support words of the document that by themselves state at least one + states, unless the key says it may be left out, asserts nothing the document does not state + and nothing the key says makes an answer wrong, and quotes in support words of the document that by themselves state at least one thing the key's answer states. The quotation need not be the one the key gives. - **A miss traceable to the text**, under questions 2 and 3, is a miss whose reasoning quotes the document or states, in the reader's own words, what the document says or requires. Such a miss @@ -133,9 +137,9 @@ which rule and why. Each sheet is marked question by question, as follows. it turns on quotes the document; any other miss on question 1 is the reader's. - **An unanswered question.** Each keyed item left unanswered is one miss, not traceable to the text, so a sheet that leaves three items of question 3 blank has three misses. Question 4 left - unanswered counts as no passage quoted. A sheet with any answer text is marked as it stands, - whatever its process returned; one whose answer text is empty is set aside, as - [The cohort](#the-cohort) says. + unanswered counts as no passage quoted. A sheet with any answer text is marked as it stands; + one whose answer text is empty, or whose process reports an error, is set aside, as + [The cohort](#the-cohort) says, and is still marked. - **The same passage.** A quotation under question 4, of any length, covers every sentence that an occurrence of the quoted words, taken as one run, overlaps, at every place that run occurs, and a passage named without a quotation, by clause number or description, covers every @@ -161,8 +165,9 @@ one" is four of five. | A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it: no reader of the fresh cohort names a passage that covers a sentence of the paragraph the earlier passage lay in, as that paragraph reads in the changed document, or a sentence the change kept from it | | Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | -Every row must hold, and the last covers only what none of the others catches. The first, third -and fourth rows block on what a single reader, or for the fourth two readers, did, and the second +Every row must hold, and the last covers only what none of the others catches. The first and +third rows block on what a single reader did, and the fourth on what two readers did and then +on what any one reader of a fresh cohort names, and the second row's tolerance of one reader never excuses a miss under the third. The asymmetry is deliberate. A load-bearing statement is the sentence the whole document exists diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 796be43..29a3772 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -70,7 +70,8 @@ survives, the change is yours. 5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text. Its record is committed under `acceptance/runs/`, the `Unreleased` changelog line names it, and CI refuses a specification change that lacks one. For an obligation, the editor sets the document's - header to `-dev, unpublished` in the same commit if it still names a published + header to `-dev, unpublished`, and the status paragraph and citation that state + its version to `-dev`, in the same commit if the header still names a published version. A wording change after the run means a fresh run. 6. **Merge** into `next` by merge commit, and the branch is deleted. diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index b555ee6..254c23d 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -12,20 +12,23 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). the change touches, each a separate process with no tools and no project context, started from an empty directory outside any repository with no user, project or local settings files, `CLAUDE.md`, skills, plugins, hooks, MCP servers or saved session. A reader is set aside only - if its answer text is empty or it shared the editor's context, and a run with a reader set + if its answer text is empty, its process reports an error, it shared the editor's context, it + was not started as required or it reported another model, and a run with a reader set aside is void, or fails if the readers who count already fail it or its own sheet has a miss traceable to the text; a void run has a number, a record and posted sheets. The packet is the question set, the document and the fixture, each after one `===== FILE: =====` line. A new marking section says what a delegated marker receives and that its verdict is final unless a rule was misapplied, that open judgements take the stricter reading, and defines a keyed item, quoting (consecutive words of any length, verbatim apart from case, quotation-mark style and markup), a correct load-bearing answer, - whose quotation must itself state part of the key's answer, a miss traceable to the text, + which asserts nothing the document does not state and whose quotation must itself state part + of the key's answer, a miss traceable to the text, including a question 1 bullet that quotes the document, which a key can add to but never remove, an unanswered item and the same passage. The thresholds apply to the readers whose answers count and block on a single - reader for the first, third and fourth rows, and the fourth row says what a fresh cohort no + reader for the first and third rows and on two readers for the fourth, and the fourth row says what a fresh cohort no longer naming a passage means. A fresh cohort runs after every change following a failure, every void run and every wording change, which is any change to the text after the readers - read it except a version number or date the editor sets for a release. The record adds the + read it except a version number or date the editor sets for a release, and step 5 sets the + status paragraph and citation to the `-dev` version with the header. The record adds the model version each reader reported, the isolation, managed settings, readers set aside, void runs, re-marks and each confusing passage's words verbatim, and links each reader's whole output and the marker's instruction, posted on the pull request and never edited, hidden or From 76832dd4c41b9408aa59ef84d9abb97698dc25c9 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 22:49:41 +0100 Subject: [PATCH 019/105] Method 2, 3.2 and section 1, detector 9: apply the rulings on review 5 - Section 2's toolchain-gap exit and clause 3.2's bespoke-detector and gap paragraphs now turn on no extensible detector of a kind that can express the rule within clause 3.1's bounds, so a class only a kind the language lacks could hold is routed to a bespoke detector or a gap. - Clause 3.2 states one reading of writing a rule: enabling an existing rule or tightening its thresholds is not; giving an existing rule a project pattern, or changing the pattern of a project rule, is. The reference now names the off-the-shelf paragraph. - Section 1 names mypy as semantic only with check_untyped_defs (mypy's command-line documentation: by default the bodies of functions without annotations are not type checked); tools/mypy.md says the same. - Detector section 9 no longer calls the verdict obligation one on a practitioner: section 8 binds whoever gives a verdict on a detector. - CHANGELOG lines follow, run NNN placeholders kept. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 14 ++++++++++---- DETECTOR-SPEC.md | 4 ++-- SPEC.md | 15 ++++++++++----- tools/mypy.md | 2 +- 4 files changed, 23 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d177d25..0e181da 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,7 +14,9 @@ version entry with no cohort line has not been accepted and is not published. - New obligation, clause [3.2](SPEC.md#32-build-the-net): a rule the practitioner writes MUST be written in the strongest kind of detector that can express it, semantic, else syntactic, else textual, judged by the - configuration it runs in; enabling or configuring an existing rule is not writing one. A weaker + configuration it runs in; enabling an existing rule or tightening its thresholds is not writing + one, and giving an existing rule a pattern of the project's own, or changing the pattern of one + the project wrote, is. A weaker kind is allowed only where the stronger cannot express the rule or the language has no detector of that kind that accepts bespoke rules; a stronger configuration of a detector the project runs counts, and one only a hosted service or licence tier provides does not. A @@ -28,7 +30,11 @@ version entry with no cohort line has not been accepted and is not published. the reason, which section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) and the section [3](SPEC.md#3-the-method) map note. Clause [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation)'s list of what the remediation docs state defers to what clause [3.2](SPEC.md#32-build-the-net) adds. Section [1](SPEC.md#1-terminology) defines textual, syntactic and semantic by whether a rule can ask the type of any expression - and what any name refers to across the whole program, section + and what any name refers to across the whole program, naming mypy as semantic only with + `check_untyped_defs`, since by default it does not check the bodies of unannotated functions. + Section [2](SPEC.md#2-when-the-method-applies) and clause [3.2](SPEC.md#32-build-the-net) send a class that no extensible detector of a kind the language + has can express within clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s bounds to a bespoke detector or a toolchain gap, not to + "no pattern exists". Section [7](SPEC.md#7-conformance)'s conforming defence carries the kind, and Appendix A carries the requirement. Prompted by project rules hosted in ast-grep, where the language already had a semantic host, that decided by names: they reported unrelated calls of the same name, and clause [3.2](SPEC.md#32-build-the-net) said any detector @@ -101,8 +107,8 @@ pass it. or licence tier. It is not a level of conformance and does not change the verdict; it is stated because method clause [3.2](SPEC.md#32-build-the-net) now requires a rule to be written in the strongest kind of detector that can express it, and a verdict of conforming alone presents a textual detector as - an equal host. Section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) names it as the one obligation the document adds for a practitioner, - since it binds whoever gives a verdict, and section [2](DETECTOR-SPEC.md#2-terminology)'s gloss carries the three terms. Accepted by run NNN. Minor version when released. + an equal host. Section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) still adds no obligation to a practitioner and names this + one as binding whoever gives a verdict on a detector, and section [2](DETECTOR-SPEC.md#2-terminology)'s gloss carries the three terms. Accepted by run NNN. Minor version when released. ### 1.0.0, 2026-09-08 diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index e6a9cc2..655ddc5 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -361,8 +361,8 @@ maintainer's reading rather than the [Detector]. ## 9. Relationship to the other specifications -This document adds no obligations to a [Practitioner] and relaxes none, save one: a -verdict on a [Detector], whoever gives it, names its kind under section 8. Every clause here +This document adds no obligations to a [Practitioner] and relaxes none. Section 8 adds one on +whoever gives a verdict on a [Detector]: to name its kind. Every clause here exists to make a clause of the method specification achievable with a [Detector] in hand. Where this document and the method specification disagree, the method specification governs. It diff --git a/SPEC.md b/SPEC.md index 0067657..39625f0 100644 --- a/SPEC.md +++ b/SPEC.md @@ -86,7 +86,7 @@ Of a [Detector]: seeing the parse tree of the code, and at most some of the type #### Semantic -Of a [Detector]: one in which a [Rule] can ask the type of any expression and what any name refers to, across the whole program, as the language's compiler or type checker resolves them, as in PHPStan, mypy, Clippy, Dylint and ESLint with typescript-eslint's type information. One that infers types only in part, or only within one file, is [Syntactic]. The same [Detector] can be [Syntactic] in one configuration and [Semantic] in another: ESLint without type information is [Syntactic]. A [Semantic] [Detector] need not accept bespoke [Rules]: Clippy does not, and in Rust a project's own [Semantic] [Rules] are written for Dylint. +Of a [Detector]: one in which a [Rule] can ask the type of any expression and what any name refers to, across the whole program, as the language's compiler or type checker resolves them, as in PHPStan, mypy with `check_untyped_defs`, Clippy, Dylint and ESLint with typescript-eslint's type information. One that infers types only in part, or only within one file, is [Syntactic]. The same [Detector] can be [Syntactic] in one configuration and [Semantic] in another: ESLint without type information is [Syntactic]. A [Semantic] [Detector] need not accept bespoke [Rules]: Clippy does not, and in Rust a project's own [Semantic] [Rules] are written for Dylint. #### Rule @@ -201,7 +201,8 @@ them judge whether the attempt was one. There are three ways out of the method and they are recorded differently. No pattern exists: the [Defect] is out of scope under this section, and only the sentence above is recorded. A pattern exists but the -language has no extensible [Detector] and a bespoke one is not practical: a [Toolchain] gap +language has no extensible [Detector] of a kind that can express the [Rule] within clause 3.1's +bounds, and a bespoke one is not practical: a [Toolchain] gap under clause 3.2. A pattern exists, a [Detector] exists, but the [Toolchain] lacks a mechanism the [toolchain specification](TOOLING-SPEC.md) requires: a mechanism gap under clause 3.2, and the [Rule] is still built. The attempt that separates the first @@ -411,7 +412,9 @@ are examples rather than a permitted list, and a [Detector] is not disqualified **Qualifying is not the same as being the right host.** A [Rule] the [Practitioner] writes MUST be written in the strongest kind of [Detector] that can express it: a [Semantic] one, else a [Syntactic] one, else a [Textual] one. The kind is that of the configuration the [Rule] runs in. -Enabling or configuring an existing [Rule] is not writing one; the paragraph above covers that. A +Enabling an existing [Rule], or tightening its thresholds, is not writing one, and the +off-the-shelf paragraph above covers it; giving an existing [Rule] a pattern of the project's own, +or changing the pattern of a [Rule] the project wrote, is writing one. A weaker kind is allowed only where the stronger kind cannot express the [Rule], as where the [Rule] reads a file the [Semantic] [Detector] does not parse, or where the language has no [Detector] of the stronger kind that accepts bespoke [Rules]. The language has one when the @@ -439,7 +442,8 @@ variable the call is made on misses the same call on a variable named otherwise, [Instance] the [Rule] never sees. A [Semantic] [Detector] decides by the type, so the bounds of clause 3.1 can be held there. -Where the affected language has no extensible [Detector] available, the [Practitioner] MAY build a +Where the affected language has no extensible [Detector] available of a kind that can express the +[Rule] within clause 3.1's bounds, the [Practitioner] MAY build a bespoke one: a program of the project's own that reads code and reports matches of the [Class], run through the project's own entry point like any other [Detector]. This is a last resort rather than a default, because a [Detector] the ecosystem maintains is cheaper to keep and easier for the next @@ -458,7 +462,8 @@ then the only requirement is that somebody deciding what to invest in the [Toolc it. A gap recorded where nobody looks has been forgotten with extra steps. There are two kinds of gap and they lead to different work. The language having no extensible -[Detector] and no practical bespoke one is the gap above, and the [Class] waits for one. The project's [Toolchain] +[Detector] of a kind that can express the [Rule] within clause 3.1's bounds, and no practical +bespoke one, is the gap above, and the [Class] waits for one. The project's [Toolchain] lacking a mechanism the [toolchain specification](TOOLING-SPEC.md) requires, such as a proving harness or an [Identifier] resolver, is a mechanism gap, and it does not put the [Class] out of reach: the [Practitioner] builds the [Rule], uses the substitutes clauses 3.3 and 3.6 already allow, and records diff --git a/tools/mypy.md b/tools/mypy.md index e9e01d7..713c2bf 100644 --- a/tools/mypy.md +++ b/tools/mypy.md @@ -43,4 +43,4 @@ Clause [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-writ ## Notes for a practitioner -Treat mypy as a detector for type-shaped classes: where the class is "this value can be None here", tightening the configuration is a legitimate off-the-shelf rule under method clause [3.2](../SPEC.md#32-build-the-net), proven by watching the count go red. For any other class, a mypy plugin can host a rule its hooks reach, and method clause [3.2](../SPEC.md#32-build-the-net) asks for that semantic host before a weaker kind; a rule the hooks cannot reach goes to a weaker detector, with the reason the clause requires recorded. Enable `warn_unused_ignores` plus a check in a second detector on new `# type: ignore` comments so the inline route stays visible. +Treat mypy as a detector for type-shaped classes: where the class is "this value can be None here", tightening the configuration is a legitimate off-the-shelf rule under method clause [3.2](../SPEC.md#32-build-the-net), proven by watching the count go red. For any other class, a mypy plugin, with `check_untyped_defs` enabled, can host a rule its hooks reach, and method clause [3.2](../SPEC.md#32-build-the-net) asks for that semantic host before a weaker kind; a rule the hooks cannot reach goes to a weaker detector, with the reason the clause requires recorded. Enable `warn_unused_ignores` plus a check in a second detector on new `# type: ignore` comments so the inline route stays visible. From 7180b40fea52653fd6306026477a966cbc2553af Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 22:50:00 +0100 Subject: [PATCH 020/105] Acceptance keys: follow the rulings on review 5 The detector key's restatement now says how a bullet that the detector, or a verdict on it, must state its kind is marked: correct, as the section 8 obligation, though the document puts it on the verdict. No key quotes a sentence changed in this revision, so no quotation moves. Co-Authored-By: Claude Opus 5.5 --- acceptance/detector/KEY.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 068dc4e..ac2d083 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -20,7 +20,8 @@ bullet asserting an obligation the document does not state, is a correct restate A bullet saying that a verdict on a detector names the kind, semantic, syntactic or textual, of each configuration it grades (section [8](../../DETECTOR-SPEC.md#8-conformance)) is correct too, though it is an obligation on whoever -gives the verdict, as section [9](../../DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) says. +gives the verdict, as section [9](../../DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) says. A bullet saying that the detector, or a verdict on it, must state +its kind is marked correct as this obligation, although the document puts it on the verdict. Wrong if it asserts: the detector must be semantic to conform; the detector must enforce a project's suppression policy; the detector must From b713023c70a4082da5786545eb407e6e3feaffab Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 22:56:58 +0100 Subject: [PATCH 021/105] Acceptance process: the commit named before readers start; sheet-visible set-asides by the marker Two minor points of review 6: a run made under an unmerged process change names its commit before the readers start, so the choice cannot follow the marks; and a delegated marker sets aside, with the cause, a reader whose sheet shows one, the editor supplying only causes the sheets cannot show. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 8afed4b..a6834a7 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -106,9 +106,10 @@ the pull request read; the answer sheets; and, from the editor, the facts the re the sheets do not, which are the pull request and the commit, the cohort size and model, how the readers were isolated and the command they were started with, whether the machine applies managed (policy) settings to the readers, that the editor attests the rest of the isolation, the -readers set aside and why, how the marker was started, the run number, and whether the run was +readers set aside for a cause the sheets cannot show and why, how the marker was started, the run number, and whether the run was made under a process change before that change merged and, if so, at which commit. The marker -writes the record, stating those facts as the editor supplied them, and the editor adds the +writes the record, stating those facts as the editor supplied them and setting aside, with the +cause, any reader whose sheet shows a cause, and the editor adds the findings applied and the link to the posted sheets after marking. A delegated marker's verdict, its marks and its PASS or FAIL, is final unless the editor shows that a rule in this file was misapplied; the editor then re-marks what the misapplied rule affected, and the record says @@ -247,8 +248,8 @@ cohort: the cohort measures whether a low-strength reader can follow a specifica process is followed by the editor. Such a change names the runs or the case that showed the need for it, and goes through steps 1, 3 and 6 of `CONTRIBUTING.md`, with no scope, key or cohort step; its review reads every process file it changes. It applies to every run made after it -merges; a run made before then may be made under it, follows all of it at a named commit, never -part, and its record says so and names that commit. It is recorded by a line added at the top +merges; a run made before then may be made under it, follows all of it at a commit named before +the readers start, never part, and its record says so and names that commit. It is recorded by a line added at the top of [acceptance/CHANGES.md](acceptance/CHANGES.md), naming its pull request, not in the changelog, which is for the three specifications. It is bound by [the section on the bar](#the-bar-only-moves-up): it may raise a criterion, never relax one. A From a6950109940da0f7dfcf521752a6418cc3ac12db Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 22:57:27 +0100 Subject: [PATCH 022/105] Acceptance process: name pull request 13 in acceptance/CHANGES.md Co-Authored-By: Claude Opus 5.5 --- acceptance/CHANGES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 254c23d..68814cb 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -5,7 +5,7 @@ describes: a line is added at the top for each change, naming its pull request. not a specification and is not versioned, and a change to it is not tested by a cohort. Changes to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). -- Pull request #PR: the cohort can be run and marked without judgement calls that the rules +- [Pull request #13](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/13): the cohort can be run and marked without judgement calls that the rules leave open; this applies to runs made after it merges, and to run 19 if made under it, and runs 16 to 18 stand as recorded. A cohort is an odd number of readers started, at least five, larger at the editor's choice the more obligations From 56558fa9f607442c679b3788e85e46e0c8e35c10 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 22:57:55 +0100 Subject: [PATCH 023/105] Register: Ruff and Pyright advice defers to method clause 3.2 Review 6 found both pages still sent a project's own Python classes straight to Pylint or a Flake8 plugin, skipping the semantic host and the recorded reason clause 3.2 requires. Co-Authored-By: Claude Opus 5.5 --- tools/pyright.md | 2 +- tools/ruff.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/pyright.md b/tools/pyright.md index fed4533..5685db3 100644 --- a/tools/pyright.md +++ b/tools/pyright.md @@ -43,4 +43,4 @@ Clause [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-writ ## Notes for a practitioner -Pyright can carry an off-the-shelf defence where the class is a type hazard one of its named rules already detects, and raising that rule to `error` in `pyrightconfig.json` is a recorded, proven change. It cannot host anything bespoke, so keep a second detector such as Pylint for the project's own classes, set `enableTypeIgnoreComments` to false and `reportUnnecessaryTypeIgnoreComment` to error, and have the second detector flag `# pyright: ignore` since Pyright will not. +Pyright can carry an off-the-shelf defence where the class is a type hazard one of its named rules already detects, and raising that rule to `error` in `pyrightconfig.json` is a recorded, proven change. It cannot host anything bespoke, so keep a second detector for the project's own classes, chosen as method clause [3.2](../SPEC.md#32-build-the-net) requires (a mypy plugin where its hooks can express the rule, a weaker kind such as Pylint only with the reason recorded), set `enableTypeIgnoreComments` to false and `reportUnnecessaryTypeIgnoreComment` to error, and have the second detector flag `# pyright: ignore` since Pyright will not. diff --git a/tools/ruff.md b/tools/ruff.md index f875bbb..b011ac1 100644 --- a/tools/ruff.md +++ b/tools/ruff.md @@ -43,4 +43,4 @@ Clause [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-writ ## Notes for a practitioner -Ruff cannot host a bespoke defence, so use it for the industry's known hazards and pair it with a second detector that accepts project rules, such as Pylint or a Flake8 plugin, for the class that the method turns up. Enable `PGH004` and `RUF100`, and forbid the specific `# noqa: CODE` form in the configuration of that second detector or in review policy, since Ruff can neither switch it off nor see it. +Ruff cannot host a bespoke defence, so use it for the industry's known hazards and pair it with a second detector that accepts project rules for the class that the method turns up, chosen as method clause [3.2](../SPEC.md#32-build-the-net) requires: a mypy plugin where its hooks can express the rule, and a weaker kind such as Pylint or a Flake8 plugin only with the reason recorded. Enable `PGH004` and `RUF100`, and forbid the specific `# noqa: CODE` form in the configuration of that second detector or in review policy, since Ruff can neither switch it off nor see it. From 5b6093f6f0c29ed0d45be1293c8077b07f03b54d Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 22:58:49 +0100 Subject: [PATCH 024/105] Acceptance process: record the 2d question-set line for pull request 14 Co-Authored-By: Claude Opus 5.5 --- acceptance/CHANGES.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 68814cb..a876cc8 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -5,6 +5,11 @@ describes: a line is added at the top for each change, naming its pull request. not a specification and is not versioned, and a change to it is not tested by a cohort. Changes to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). +- [Pull request #14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14): + the `SPEC.md` question set gains a fourth load-bearing question, 2d, on which kind of detector + must host a rule the practitioner writes, so `ACCEPTANCE.md` lists three or more such + questions per document. Adding a keyed question raises the bar, as "The bar only moves up" + allows. Prompted by the change to method clause 3.2 in the same pull request. - [Pull request #13](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/13): the cohort can be run and marked without judgement calls that the rules leave open; this applies to runs made after it merges, and to run 19 if made under it, and runs 16 to 18 stand as recorded. A cohort is an odd From bdfd055069b65d40f51cbeb6e73af74b0d255410 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:12:27 +0100 Subject: [PATCH 025/105] Acceptance process: mark question 2 for its gist, judge open marks as a careful reader would Applies the editor's rulings after runs 19 and 20. Question 2 reads the reader's sentence and quotation together and marks the gist of the key's answer, failing only what the document does not state, what the key says makes an answer wrong and what the key says an answer is wrong to omit. An open judgement is made as a careful reader of the document would, not by the reading that records more misses. A passage under question 4 is one the reader found confusing, not one named only to explain another. Each key names what a question 2 answer is wrong to omit, from what it already marked. None of these rules was on next, so none relaxes it. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 19 +++++++++++-------- acceptance/CHANGES.md | 16 ++++++++++------ acceptance/detector/KEY.md | 3 +++ acceptance/method/KEY.md | 14 +++++++------- acceptance/toolchain/KEY.md | 3 +++ 5 files changed, 34 insertions(+), 21 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index a6834a7..1de652c 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -122,10 +122,13 @@ which rule and why. Each sheet is marked question by question, as follows. The words are verbatim apart from letter case, quotation-mark or apostrophe style, and markdown link or emphasis markup. Naming a clause number, or paraphrasing, is not quoting. - **Questions 1 and 3** are marked as the key states for each item. -- **Question 2.** An answer is correct when its sentence states each thing the key's answer - states, unless the key says it may be left out, asserts nothing the document does not state - and nothing the key says makes an answer wrong, and quotes in support words of the document that by themselves state at least one - thing the key's answer states. The quotation need not be the one the key gives. +- **Question 2.** An answer is the reader's sentence and its quotation read together. It is + correct when, so read, it states what the key's answer states, in the document's words or the + reader's own (a paraphrase, a synonym, or a defined term used in its defined sense states what + it means), asserts nothing the document does not state or the key says makes an answer wrong, + omits nothing the key says an answer is wrong to omit, and quotes the document in support. The + quotation need not be the one the key gives. The key names the elements an answer is wrong to + omit; for the rest, the marker reads for the gist of the key's answer, not a checklist. - **A miss traceable to the text**, under questions 2 and 3, is a miss whose reasoning quotes the document or states, in the reader's own words, what the document says or requires. Such a miss is traceable even where it also misreads the fixture, and whatever the key says of it. Only a @@ -141,16 +144,16 @@ which rule and why. Each sheet is marked question by question, as follows. unanswered counts as no passage quoted. A sheet with any answer text is marked as it stands; one whose answer text is empty, or whose process reports an error, is set aside, as [The cohort](#the-cohort) says, and is still marked. -- **The same passage.** A quotation under question 4, of any length, covers every sentence that +- **The same passage.** A passage under question 4 is one the reader says it found confusing, not + one it names only to explain another confusion. A quotation of one, of any length, covers every sentence that an occurrence of the quoted words, taken as one run, overlaps, at every place that run occurs, and a passage named without a quotation, by clause number or description, covers every sentence of what it names. A heading, list item or table row counts as one sentence. Two passages are the same when they cover a sentence in common or lie in the same paragraph, and a list item or table row is its own paragraph. -Where a mark needs a judgement these rules do not settle, the marker makes it, taking the stricter -reading, the one that records more misses or findings, and the record states the judgement and -why. +Where a mark needs a judgement these rules do not settle, the marker makes it as a careful reader +of the document would, and the record states the judgement and why. ## Pass criteria diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 68814cb..15ea7a2 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -17,11 +17,13 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). aside is void, or fails if the readers who count already fail it or its own sheet has a miss traceable to the text; a void run has a number, a record and posted sheets. The packet is the question set, the document and the fixture, each after one `===== FILE: =====` line. A new marking section says what a delegated marker - receives and that its verdict is final unless a rule was misapplied, that open judgements take - the stricter reading, and defines a keyed item, quoting (consecutive words of any length, - verbatim apart from case, quotation-mark style and markup), a correct load-bearing answer, - which asserts nothing the document does not state and whose quotation must itself state part - of the key's answer, a miss traceable to the text, + receives and that its verdict is final unless a rule was misapplied, that an open judgement is + made as a careful reader of the document would, and defines a keyed item, quoting (consecutive + words of any length, verbatim apart from case, quotation-mark style and markup), a correct + load-bearing answer, read with its quotation for the gist of the key's answer, asserting + nothing the document does not state and omitting nothing the key says it is wrong to omit, a + passage found confusing, which excludes one named only to explain another, a miss traceable + to the text, including a question 1 bullet that quotes the document, which a key can add to but never remove, an unanswered item and the same passage. The thresholds apply to the readers whose answers count and block on a single reader for the first and third rows and on two readers for the fourth, and the fourth row says what a fresh cohort no @@ -34,7 +36,9 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). output and the marker's instruction, posted on the pull request and never edited, hidden or deleted, or a run not already failed is void. The keys are marked at the commit the readers read and no longer limit which misses count against the document, - removing such a limit is named as raising the bar, and the method key's "half right" is a miss. + removing such a limit is named as raising the bar; the method key's "half right" is a miss, and + each key names what a question 2 answer is wrong to omit. No marking rule was on `next` + before, so none relaxes it. The process files include the checks that enforce them, and a process change goes through steps 1, 3 and 6 of `CONTRIBUTING.md`, applies to runs made after it merges, or earlier ones made under it at a named commit, and is recorded here, not in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the change, points to diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 8608f7d..c0749a8 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -25,6 +25,9 @@ agent summary (those belong to the toolchain specification). ## 2. Load-bearing statements +No entry names an element an answer is wrong to omit, so each is marked for the gist of its +answer, as `ACCEPTANCE.md` says. + - **2a.** Conformance is satisfying every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression), judged on evidence; a detector that predates the document can conform, and the declaration is optional. Key quotation, section 8: "A Detector Conforms if it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression)." Supporting, [8.1](../../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps): "A diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 2e37b07..8839bc7 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -29,20 +29,20 @@ a test may serve as the detector; an agent may baseline the remaining instances. then sweep and fix every instance, in that order. Key quotations, section [3](../../SPEC.md#3-the-method): "Six clauses, in order."; [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance): "The Practitioner MUST run the Rule across the entire codebase and record the total Instance count before fixing anything, and MUST then fix every Instance found rather - than only the one". An answer that has the fix before the rule, or the sweep before the proof, - is wrong. + than only the one". An answer that omits the rule, the proof or the sweep, or has the fix + before the rule or the sweep before the proof, is wrong. - **2b.** The owner of the codebase, always a human; an agent may not. Key quotation, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): "An Agent MUST NOT Baseline, suppress or knowingly leave an Instance unfixed on its own - authority, whatever the Instance count turns out to be." An answer that lets the practitioner - decide because the count was small, or because the project recorded no rule, is wrong (the - default owner paragraph in the terminology entry for Owner still requires a recorded decision). + authority, whatever the Instance count turns out to be." An answer that omits the owner, or + lets the practitioner decide because the count was small or because the project recorded no + rule, is wrong (the default owner paragraph in the terminology entry for Owner still requires a + recorded decision). - **2c.** All six clauses of section [3](../../SPEC.md#3-the-method) were followed for that defect, and the verdict rests on the reviewer reproducing the red and green runs, not on the report. Key quotations, section [7](../../SPEC.md#7-conformance): "A remediation Conforms if all six clauses of section [3](../../SPEC.md#3-the-method) were followed for that Defect, and the verdict that they were rests on reproduction"; "A verdict on a remediation or a Defence MUST rest on reproduction, not on the report". An answer that accepts the pull request description - as the verdict is wrong, and so is one that states the six clauses and says nothing of - reproduction. + as the verdict is wrong, and so is one that omits reproduction. ## 3. Application: the ledger remediation diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index cae0efc..32b312c 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -30,6 +30,9 @@ conform; the agent summary (section [7](../../TOOLING-SPEC.md#7-agent-context)) ## 2. Load-bearing statements +No entry names an element an answer is wrong to omit, so each is marked for the gist of its +answer, as `ACCEPTANCE.md` says. + - **2a.** At the project level; the toolchain is whatever the project has assembled from third-party, first-party and project-level parts. Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "A project's Toolchain Conforms if every MUST in sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record) holds across the assembled parts, wherever From 87536c5f1a092c6688db3171b97a1df00fe589d1 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:16:40 +0100 Subject: [PATCH 026/105] Clarity edits after runs 19 and 20 SPEC.md: clause 3.2 sets out the two grounds for a weaker kind as a numbered pair, refers to "which of the two grounds", and says the extension-points rule in two short sentences; section 2 says what independent means instead of pointing at clause 3.1's vocabulary; clause 3.1 says what the level of a class is; clause 3.3 Part B says an instance behind an inline ignore is a suppression for clauses 3.4, 3.5 and section 4, not a narrowing; clause 3.4 says plainly that a baseline is rare. DETECTOR-SPEC.md: clause 5.3 says what the usable summary is and when the location of the rest is owed; clause 6.1 says it asks for the mechanism, missing bundled documentation failing 6.3, and why a path in the message is not enough; clause 7.1 says the project writes the detecting rule and the detector need not remove or forbid the route; clause 8.1 says what a gap is, where it is recorded, and that conformance is judged on evidence with or without a declaration. CHANGELOG.md: both Unreleased lines note the edits; run NNN kept. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 17 +++++++++++++++-- DETECTOR-SPEC.md | 38 +++++++++++++++++++++++++------------- SPEC.md | 45 ++++++++++++++++++++++++++++----------------- 3 files changed, 68 insertions(+), 32 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0e181da..c94f798 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,7 +40,12 @@ version entry with no cohort line has not been accepted and is not published. by names: they reported unrelated calls of the same name, and clause [3.2](SPEC.md#32-build-the-net) said any detector qualified whatever its category. The method question set gains load-bearing statement 2d, which `ACCEPTANCE.md` now lists, and `index.md`, `README.md`, `PRIMER.md` and two register pages - defer to the clause. Accepted by run NNN. Minor version when released. + defer to the clause. After run 19, clarity edits: clause [3.2](SPEC.md#32-build-the-net) sets out the two grounds + for a weaker kind as a numbered pair and says the extension-points rule in two sentences; + section [2](SPEC.md#2-when-the-method-applies) says what independent means instead of pointing at clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s vocabulary; clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class) + says what the level of a class is; clause [3.3](SPEC.md#33-prove-the-net-by-making-the-rule-fire)'s Part B says an instance left behind an inline + ignore is a suppression for clauses [3.4](SPEC.md#34-sweep-the-codebase-then-fix-every-instance) and [3.5](SPEC.md#35-enforce-permanently-and-block) and section [4](SPEC.md#4-authority-which-decisions-belong-to-whom), not a narrowing; and clause [3.4](SPEC.md#34-sweep-the-codebase-then-fix-every-instance) + says plainly that a baseline is rare. Accepted by run NNN. Minor version when released. ### 1.1.0, 2026-10-02 @@ -108,7 +113,15 @@ pass it. is stated because method clause [3.2](SPEC.md#32-build-the-net) now requires a rule to be written in the strongest kind of detector that can express it, and a verdict of conforming alone presents a textual detector as an equal host. Section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) still adds no obligation to a practitioner and names this - one as binding whoever gives a verdict on a detector, and section [2](DETECTOR-SPEC.md#2-terminology)'s gloss carries the three terms. Accepted by run NNN. Minor version when released. + one as binding whoever gives a verdict on a detector, and section [2](DETECTOR-SPEC.md#2-terminology)'s gloss carries the three terms. After run 20, clarity edits: clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) says + the usable summary is every finding with its location and identifier, and the location of the + rest is owed only where a finding is left out; clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) says it asks for the mechanism and + that missing documentation for a bundled rule fails clause [6.3](DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), and says plainly why a path in the + message is not enough; clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) says the project writes the detecting rule, of the kind clause + [4.1](DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) requires, and that the detector need not remove or forbid the route; clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) says + what a gap is, that a detector declaring a version records it there, and that conformance is + judged on evidence with or without a declaration. Accepted by run NNN. Minor version when + released. ### 1.0.0, 2026-09-08 diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index 655ddc5..3ffc1e8 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -158,7 +158,10 @@ any. ### 5.3 The result MUST reach the practitioner in the output of the command they ran Where the [Detector] writes fuller detail elsewhere, the invoked command's own output MUST -carry both a usable summary and the location of the remainder. +still carry a usable summary: every finding, with where it is and its [Identifier]. If that +output leaves any finding out, it MUST also give the location of the rest. Output that prints +every finding meets this clause, even where extra detail, such as a code snippet, is written only +to a file the output does not name. **Why**: method specification clause 8.2. A report the [Practitioner] has to go and find is a report that arrives after the decision it was meant to inform. @@ -184,13 +187,17 @@ Two cases, by who wrote the [Rule]: | The project's own | The project | Print the [Identifier] unaltered; the lookup is the [Toolchain]'s to give | In both, the mechanism is keyed on **the [Identifier] exactly as printed**. A command, an index -file or a URL are all acceptable forms. Each case in turn: +file or a URL are all acceptable forms. This clause asks for the mechanism, not for every page. +Where the mechanism works but a [Bundled rule]'s [Identifier] has no documentation to resolve +to, that fails clause 6.3, not this clause; for a project's own [Rule], the missing +documentation is the project's to write. Each case in turn: **For a [Bundled rule]**, the [Detector] supplies the documentation and the lookup, and clauses 6.2 and 6.3 say where. The mechanism MUST resolve an [Identifier] presented alone, because the reader who needs it most has the [Identifier] from a log, a -ticket or a colleague and not the [Message]. A [Message] that carries its own -documentation path resolves that [Message], not the [Identifier]. +ticket or a colleague and not the [Message]. A documentation path printed inside the +[Message] does not meet this: it helps only a reader who has the [Message], and the +[Identifier] on its own still leads nowhere. **For a project's own [Rule]**, the [Detector] cannot know the documentation, so what it owes is the half it can give: the [Identifier] printed unaltered under clause 4.3, and no @@ -270,9 +277,11 @@ nothing to run. An inline ignore comment, a per-line directive and a generated [Baseline] are all such routes. The [Detector] MAY ship them. For each one it MUST do at least one of two things, and either alone satisfies this clause: make the route disableable by configuration; or make the route -detectable, by a [Rule] the project can write in the [Detector] itself or by a mechanical check -the [Detector] documents. Either lets a project which decides to forbid the route enforce that -decision. A route that can be neither switched off nor seen does not [Conform]. +detectable. Detectable means the project can find every use of the route, either with a [Rule] +the project writes itself, of the kind clause 4.1 requires the [Detector] to host, or with a +mechanical check the [Detector] documents. The [Detector] is not required to remove the route +or to forbid it. Either of the two things lets a project which decides to forbid the route +enforce that decision. A route that can be neither switched off nor seen does not [Conform]. **Why**: the method specification's position is that [Suppression] is an [Owner] decision under its clause 3.4 and section 4, and an [Owner] cannot decide something they are @@ -320,15 +329,18 @@ verdict that said only [Conforming] would present a [Textual] [Detector] as an e ### 8.1 A maintainer MAY declare the version of this document the detector conforms to, and the declaration records known gaps The declaration is machine-readable, in whatever form the [Detector]'s ecosystem uses to -record dependencies. The same declaration is where a gap against this document is recorded once it is -known: a [Detector] that has learnt, from its own checks or from a [Practitioner]'s -report under the method's clause 3.2, that it fails a MUST in sections 4 to 7 MUST record that gap -alongside the version it declares, in the same file or one it names. A declaration with a non-empty +record dependencies. The same declaration is where a known gap is recorded. A gap is a MUST in +sections 4 to 7 that the [Detector] fails. Once its maintainer knows of a gap, from the +[Detector]'s own checks or from a [Practitioner]'s report under the method's clause 3.2, a +[Detector] that declares a version MUST record the gap alongside that version, in the same file +as the declaration or in a file the declaration names. A declaration with a non-empty gap record is a statement of where the [Detector] stands and is not a claim of [Conformance]. -The declaration is optional. It is how a maintainer claims [Conformance]; it is not a -condition of it. A [Detector] that predates this document, or whose maintainer has never read +The declaration is optional: a [Detector] with no declaration can still [Conform]. Whether a +[Detector] [Conforms] is judged on the evidence, with or without a declaration. The +declaration is for the maintainer: it is how they claim [Conformance] and record the gaps they +know of. A [Detector] that predates this document, or whose maintainer has never read it, MAY be graded [Conforming] on evidence by anyone who exercises the clauses above against it, and a verdict on any [Detector], declared or not, rests on that exercise and not on the claim. A declaration tells the reader what the maintainer believes and what they know to be missing; diff --git a/SPEC.md b/SPEC.md index 39625f0..7b41b22 100644 --- a/SPEC.md +++ b/SPEC.md @@ -193,8 +193,9 @@ rather than deciding in advance whether it is expressible. Failing to write a [R clause 3.1's bounds is itself the evidence that the [Defect] is out of scope, and it is cheaper and more reliable than a judgement made before trying. The conclusion that no pattern exists MUST be stated in one sentence, naming at least two independent techniques tried at expressing the -[Rule], in the vocabulary clause 3.1 uses for a search, and recorded where the project's -other decisions are enumerable under clause 8.7, not only with the fix. It is the cheapest way +[Rule], and recorded where the project's other decisions are enumerable under clause 8.7, not +only with the fix. Two techniques are independent, as clause 3.1 defines it for a search, when +they would miss different things. It is the cheapest way out of the method and the only one that would otherwise leave no trace; enumerating it is how an [Owner] sees an [Agent] routing around the method, and naming the techniques is what lets them judge whether the attempt was one. @@ -326,12 +327,12 @@ Two bounds apply, and both are checkable during the work rather than matters of missing [Rule] does. One report on code that does not carry the [Hazard] is enough; there is no tolerated rate. -**Both bounds measure breadth within one level; the level itself is a separate question, answered -against the report.** -Where the [Defect] was reported as a behaviour, a check that reported clean having run nothing, a -process that stopped before its work, the [Class] a [Detector] can read usually sits one level -below that [Hazard]: the mechanism of this [Instance], not the failure the report opens with. -That [Class] is the right one to build, because no [Detector] reads "this check still produces +**The bounds say how broad the [Class] is. Its level is a separate question: does the [Class] +name the failure the report describes, or the mechanism in the code that caused it?** +Where the [Defect] was reported as a behaviour, such as a check that reported clean having run +nothing, or a process that stopped before its work, a [Detector] usually cannot read that failure. +The [Class] it can read sits one level lower: the mechanism of this [Instance], not the failure +the report opens with. That [Class] is the right one to build, because no [Detector] reads "this check still produces its signal". It is not the whole answer. The record MUST state whether the reported behaviour is also pinned by a check a [Runner] executes, under section 6, either one added with this remediation or an existing one named, and where neither is offered, why not. A [Class] drawn at the mechanism with @@ -415,19 +416,24 @@ be written in the strongest kind of [Detector] that can express it: a [Semantic] Enabling an existing [Rule], or tightening its thresholds, is not writing one, and the off-the-shelf paragraph above covers it; giving an existing [Rule] a pattern of the project's own, or changing the pattern of a [Rule] the project wrote, is writing one. A -weaker kind is allowed only where the stronger kind cannot express the [Rule], as where the -[Rule] reads a file the [Semantic] [Detector] does not parse, or where the language has no -[Detector] of the stronger kind that accepts bespoke [Rules]. The language has one when the +weaker kind is allowed on one of two grounds only: + +1. the stronger kind cannot express the [Rule], as where the [Rule] reads a file the [Semantic] + [Detector] does not parse; or +2. the language has no [Detector] of the stronger kind that accepts bespoke [Rules]. + +For the second ground, the language has such a [Detector] when the project already runs it or the language's own [Detector] ecosystem offers it, the same check section 2 asks for before a [Toolchain] gap is recorded. A stronger configuration of a [Detector] the project runs is one; a [Detector] or configuration of the stronger kind available only through a hosted service or a licence tier is not. A [Detector] accepts bespoke [Rules] when the project can add a [Rule] of its own to it through a documented extension point, such as a plugin -interface or an API for writing [Rules], whether or not it meets the rest of the [detector specification](DETECTOR-SPEC.md); a [Rule] whose -construct its extension points cannot reach is one that [Detector] cannot express, and one its -kind cannot express where no other [Detector] of that kind reaches it. On the [Remediation docs] of a [Rule] they +interface or an API for writing [Rules], whether or not it meets the rest of the [detector specification](DETECTOR-SPEC.md). +Where those extension points cannot reach the construct a [Rule] is about, that [Detector] +cannot express the [Rule]. Where no [Detector] of that kind can reach it, the kind cannot express +the [Rule], and the first ground holds. On the [Remediation docs] of a [Rule] they write, the [Practitioner] MUST name the [Detector] and its kind. Where that kind is weaker than [Semantic], the [Practitioner] MUST also -record, for each step down from [Semantic], which of those two reasons holds and why, naming the +record, for each step down from [Semantic], which of the two grounds holds and why, naming the [Detectors] of the stronger kind that were considered, on the [Rule]'s [Remediation docs] and in the description of the commit that adds the [Rule], moves it to another [Detector], or first records the reason. The [Practitioner] judges whether a reason holds, as section 4 gives them the choice of [Detector]; a @@ -523,6 +529,11 @@ what must survive the merge is not. **Part B: [Narrowing].** +A [Narrowing] is a change to the [Rule] itself, so that it stops matching some code. An +[Instance] left in place behind an inline ignore comment, an ignore entry or a [Baseline] is not +a [Narrowing]: the [Rule] is unchanged and still matches it. That is a [Suppression] of the +[Instance], and clauses 3.4 and 3.5 and section 4 govern it, not this clause. + **The decision comes first, and it turns on one sentence.** Can the [Practitioner] write down why the [Hazard] cannot arise in the code being excluded? If they can, the exclusion is a [Narrowing], it is theirs to make, and they record that sentence. If they cannot, or are not sure, the exclusion @@ -687,8 +698,8 @@ been seen. "All examined" with nothing behind it is a claim, not a record. stops only when the next [Instance] needs a decision they lack the authority to make, and section 4 says which decisions those are; a hundred [Instances], or a thousand, is work rather than a decision. A [Baseline] of the existing [Instances], so that the [Rule] blocks only new ones, MAY -be adopted where fixing every [Instance] is genuinely unfeasible, and the scale at which that -becomes true is tremendous. **No threshold is given here deliberately.** Any figure would be +be adopted where fixing every [Instance] is genuinely unfeasible. That is rare: fixing every +[Instance] becomes unfeasible only when the change it needs is very large indeed. **No threshold is given here deliberately.** Any figure would be calibrated to the [Toolchain] of the moment rather than to anything durable, and the honest measure is the size of the change rather than the hours it would take, since the hours depend entirely on what is doing the work. From 85110c146f600a43d6ffaf236ecaa510b95c410c Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:17:05 +0100 Subject: [PATCH 027/105] Keys: quotations follow the clarity edits after runs 19 and 20 acceptance/method/KEY.md, 2d: the clause 3.2 quotation now reads "which of the two grounds holds". acceptance/detector/KEY.md, 2b: the clause 7.1 quotation now carries the reworded detectable sentence verbatim. No question, threshold or answer changed. Co-Authored-By: Claude Opus 5.5 --- acceptance/detector/KEY.md | 5 +++-- acceptance/method/KEY.md | 4 ++-- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index f18ed23..e76ad3c 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -40,8 +40,9 @@ agent summary (those belong to the toolchain specification). project can write or a mechanical check it documents; it is not required to remove or forbid it. Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each one it MUST do at least one of two things, and either alone satisfies this clause: make the route disableable by configuration; or make the route - detectable, by a Rule the project can write in the Detector itself or by a mechanical check the - Detector documents". An answer that says the detector must forbid the route, or must require a + detectable. Detectable means the project can find every use of the route, either with a Rule + the project writes itself, of the kind clause [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) requires the Detector to host, or with a + mechanical check the Detector documents." An answer that says the detector must forbid the route, or must require a reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD), is wrong. - **2c.** With the rule, in the installed copy, at a version tracked together, and yes, offline. Key quotations, [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access): "Resolution of a bundled rule's identifier MUST work from the installed diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index bb02010..3e812db 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -57,8 +57,8 @@ is correct; a bullet that says no weaker kind is ever allowed is wrong. reason. Key quotations, [3.2](../../SPEC.md#32-build-the-net): "A Rule the Practitioner writes MUST be written in the strongest kind of Detector that can express it: a Semantic one, else a Syntactic one, else a Textual one."; "Where that kind is weaker than - Semantic, the Practitioner MUST also record, for each step down from Semantic, which of those - two reasons holds and why, naming the Detectors of the stronger kind that were considered, on + Semantic, the Practitioner MUST also record, for each step down from Semantic, which of the + two grounds holds and why, naming the Detectors of the stronger kind that were considered, on the Rule's Remediation docs and in the description of the commit that adds the Rule, moves it to another Detector, or first records the reason." Supporting: "On the Remediation docs of a Rule they write, the Practitioner MUST name the Detector and its kind." A correct answer gives three From 7f6a196657812b2e0e82dde4ce3973a1d8eb85c3 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:18:30 +0100 Subject: [PATCH 028/105] Acceptance process: review 7, every element of a question 2 answer counts, read for its gist A question 2 answer is wrong to omit any element of the key's answer that answers a part of the question, each read for its gist, as run 16 marked; the keys' "no entry names" lines go. A passage quoted under question 4 counts whatever the reader quoted it for, as on next. A delegated marker takes the process files from next where no process change is pending, and the record gives a description where a confusing passage was named without a quotation. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 13 ++++++------- acceptance/CHANGES.md | 12 +++++------- acceptance/detector/KEY.md | 3 --- acceptance/runs/README.md | 2 +- acceptance/toolchain/KEY.md | 3 --- 5 files changed, 12 insertions(+), 21 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 1de652c..43f23e1 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -100,7 +100,7 @@ separate marker that has seen neither the drafting nor the review. Whoever marks in this file. A delegated marker receives only these: the document, its question set, the fixture the questions name and its key, each at the commit the readers read; this file and `acceptance/runs/README.md`, at the named commit where the run is made under a process change -before that change merges; the earlier run records of the same pull request, and those of +before that change merges, and otherwise as they stand on `next` when the readers start; the earlier run records of the same pull request, and those of other pull requests as examples of style only; the document at the commit each earlier run of the pull request read; the answer sheets; and, from the editor, the facts the record states that the sheets do not, which are the pull request and the commit, the cohort size and model, how the @@ -126,9 +126,9 @@ which rule and why. Each sheet is marked question by question, as follows. correct when, so read, it states what the key's answer states, in the document's words or the reader's own (a paraphrase, a synonym, or a defined term used in its defined sense states what it means), asserts nothing the document does not state or the key says makes an answer wrong, - omits nothing the key says an answer is wrong to omit, and quotes the document in support. The - quotation need not be the one the key gives. The key names the elements an answer is wrong to - omit; for the rest, the marker reads for the gist of the key's answer, not a checklist. + omits no element of the key's answer that answers a part of the question, nor anything the key + says an answer is wrong to omit, and quotes the document in support. Each element is read for + its gist, not its exact words. The quotation need not be the one the key gives. - **A miss traceable to the text**, under questions 2 and 3, is a miss whose reasoning quotes the document or states, in the reader's own words, what the document says or requires. Such a miss is traceable even where it also misreads the fixture, and whatever the key says of it. Only a @@ -144,8 +144,7 @@ which rule and why. Each sheet is marked question by question, as follows. unanswered counts as no passage quoted. A sheet with any answer text is marked as it stands; one whose answer text is empty, or whose process reports an error, is set aside, as [The cohort](#the-cohort) says, and is still marked. -- **The same passage.** A passage under question 4 is one the reader says it found confusing, not - one it names only to explain another confusion. A quotation of one, of any length, covers every sentence that +- **The same passage.** A quotation under question 4, of any length, covers every sentence that an occurrence of the quoted words, taken as one run, overlaps, at every place that run occurs, and a passage named without a quotation, by clause number or description, covers every sentence of what it names. A heading, list item or table row counts as one sentence. Two passages are the same when they cover a @@ -221,7 +220,7 @@ managed (policy) settings to the readers, that the editor attests the rest of th readers set aside and the cause, the run number in that pull request, counting void runs, who marked and, for a delegated marker, how it was started, the marks per reader per question, any judgement made in marking, on a re-mark which rule and why, the passages quoted as confusing, -each with the words quoted, verbatim, and its clause, and by how many readers, the findings +each with the words quoted, verbatim, or the description used where none were quoted, and its clause, and by how many readers, the findings applied, whether it was made under a process change before that change merged and, if so, at which commit, whether it passed, and a link to the answer sheets. The answer sheets are not kept in this repository: every reader's sheet, which is its whole output as its process returned it (the JSON, where the command asks for JSON), with a set-aside reader's marked as set diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 15ea7a2..56e6350 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -20,10 +20,9 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). receives and that its verdict is final unless a rule was misapplied, that an open judgement is made as a careful reader of the document would, and defines a keyed item, quoting (consecutive words of any length, verbatim apart from case, quotation-mark style and markup), a correct - load-bearing answer, read with its quotation for the gist of the key's answer, asserting - nothing the document does not state and omitting nothing the key says it is wrong to omit, a - passage found confusing, which excludes one named only to explain another, a miss traceable - to the text, + load-bearing answer, read with its quotation, asserting nothing the document does not state + and omitting no element of the key's answer that answers the question, each read for its + gist, a miss traceable to the text, including a question 1 bullet that quotes the document, which a key can add to but never remove, an unanswered item and the same passage. The thresholds apply to the readers whose answers count and block on a single reader for the first and third rows and on two readers for the fourth, and the fourth row says what a fresh cohort no @@ -36,9 +35,8 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). output and the marker's instruction, posted on the pull request and never edited, hidden or deleted, or a run not already failed is void. The keys are marked at the commit the readers read and no longer limit which misses count against the document, - removing such a limit is named as raising the bar; the method key's "half right" is a miss, and - each key names what a question 2 answer is wrong to omit. No marking rule was on `next` - before, so none relaxes it. + removing such a limit is named as raising the bar; the method key's "half right" is a miss. + Each marking rule writes down how runs 16 to 18 marked or is stricter, so none relaxes `next`. The process files include the checks that enforce them, and a process change goes through steps 1, 3 and 6 of `CONTRIBUTING.md`, applies to runs made after it merges, or earlier ones made under it at a named commit, and is recorded here, not in the changelog. `CONTRIBUTING.md` requires a reviewer who did not draft the change, points to diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index c0749a8..8608f7d 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -25,9 +25,6 @@ agent summary (those belong to the toolchain specification). ## 2. Load-bearing statements -No entry names an element an answer is wrong to omit, so each is marked for the gist of its -answer, as `ACCEPTANCE.md` says. - - **2a.** Conformance is satisfying every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression), judged on evidence; a detector that predates the document can conform, and the declaration is optional. Key quotation, section 8: "A Detector Conforms if it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression)." Supporting, [8.1](../../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps): "A diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index 2d0b96a..633d90d 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -12,7 +12,7 @@ machine applies managed (policy) settings to the readers; that the editor attest isolation; the readers set aside and the cause; the run number within that pull request, counting void runs; who marked and, for a delegated marker, how it was started; the marks per reader per question against the key, with any judgement made in marking and, on a re-mark, which rule and -why; the passages quoted as confusing, each with the words quoted, verbatim, and its clause, and +why; the passages quoted as confusing, each with the words quoted, verbatim, or the description used where none were quoted, and its clause, and by how many readers; the findings applied to the text as a result; whether the run was made under a process change before that change merged and, if so, at which commit; and the verdict against the criteria in `ACCEPTANCE.md`, with a link to the answer sheets. Earlier diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 32b312c..cae0efc 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -30,9 +30,6 @@ conform; the agent summary (section [7](../../TOOLING-SPEC.md#7-agent-context)) ## 2. Load-bearing statements -No entry names an element an answer is wrong to omit, so each is marked for the gist of its -answer, as `ACCEPTANCE.md` says. - - **2a.** At the project level; the toolchain is whatever the project has assembled from third-party, first-party and project-level parts. Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "A project's Toolchain Conforms if every MUST in sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record) holds across the assembled parts, wherever From 66301b9a863eda71da45280adc126b36db5f612d Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:24:50 +0100 Subject: [PATCH 029/105] Review rulings on the edits after runs 19 and 20 DETECTOR-SPEC.md: clause 5.3's usable summary gives each finding in it with its location and identifier, and the location of the rest is owed if the summary leaves any finding out, so the two sentences no longer read two ways; clause 6.4's Why names clause 6.3, matching clause 6.1; clause 7.1's detecting rule is written in the detector, of the kind clause 4.1 requires it to host. SPEC.md: section 2's sentence after the independence definition opens "That conclusion"; clause 3.2 and Appendix A say "grounds" throughout; clause 3.3 Part B's opening is keyed on the hazard, not the mechanism. CHANGELOG.md: the detector line records clause 5.3 as a changed obligation, with its reason from run 20; the method line follows the edits; run NNN kept. defence-before-fix-project-prompt.md rebuilt from Appendix A. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 25 +++++++++++++++---------- DETECTOR-SPEC.md | 8 ++++---- SPEC.md | 22 +++++++++++----------- defence-before-fix-project-prompt.md | 8 ++++---- 4 files changed, 34 insertions(+), 29 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c94f798..7b5049b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -41,11 +41,12 @@ version entry with no cohort line has not been accepted and is not published. qualified whatever its category. The method question set gains load-bearing statement 2d, which `ACCEPTANCE.md` now lists, and `index.md`, `README.md`, `PRIMER.md` and two register pages defer to the clause. After run 19, clarity edits: clause [3.2](SPEC.md#32-build-the-net) sets out the two grounds - for a weaker kind as a numbered pair and says the extension-points rule in two sentences; - section [2](SPEC.md#2-when-the-method-applies) says what independent means instead of pointing at clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s vocabulary; clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class) - says what the level of a class is; clause [3.3](SPEC.md#33-prove-the-net-by-making-the-rule-fire)'s Part B says an instance left behind an inline - ignore is a suppression for clauses [3.4](SPEC.md#34-sweep-the-codebase-then-fix-every-instance) and [3.5](SPEC.md#35-enforce-permanently-and-block) and section [4](SPEC.md#4-authority-which-decisions-belong-to-whom), not a narrowing; and clause [3.4](SPEC.md#34-sweep-the-codebase-then-fix-every-instance) - says plainly that a baseline is rare. Accepted by run NNN. Minor version when released. + for a weaker kind as a numbered pair, calls them grounds throughout, as Appendix A now does, and + says the extension-points rule in two sentences; section [2](SPEC.md#2-when-the-method-applies) says what independent means instead + of pointing at clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s vocabulary; clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class) says what the level of a class is; clause [3.3](SPEC.md#33-prove-the-net-by-making-the-rule-fire)'s + Part B says an instance left behind an inline ignore comment, an ignore entry or a baseline is + not a narrowing, since the code still carries the hazard, but a suppression for clauses [3.4](SPEC.md#34-sweep-the-codebase-then-fix-every-instance) and + [3.5](SPEC.md#35-enforce-permanently-and-block) and section [4](SPEC.md#4-authority-which-decisions-belong-to-whom); and clause [3.4](SPEC.md#34-sweep-the-codebase-then-fix-every-instance) says plainly that a baseline is rare. Accepted by run NNN. Minor version when released. ### 1.1.0, 2026-10-02 @@ -113,11 +114,15 @@ pass it. is stated because method clause [3.2](SPEC.md#32-build-the-net) now requires a rule to be written in the strongest kind of detector that can express it, and a verdict of conforming alone presents a textual detector as an equal host. Section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) still adds no obligation to a practitioner and names this - one as binding whoever gives a verdict on a detector, and section [2](DETECTOR-SPEC.md#2-terminology)'s gloss carries the three terms. After run 20, clarity edits: clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) says - the usable summary is every finding with its location and identifier, and the location of the - rest is owed only where a finding is left out; clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) says it asks for the mechanism and - that missing documentation for a bundled rule fails clause [6.3](DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), and says plainly why a path in the - message is not enough; clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) says the project writes the detecting rule, of the kind clause + one as binding whoever gives a verdict on a detector, and section [2](DETECTOR-SPEC.md#2-terminology)'s gloss carries the three terms. After run 20, changed obligation, clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran): the + usable summary gives each finding in it with its location and identifier, and the location of + the rest is owed only where the summary leaves a finding out, so output that prints every + finding conforms though extra detail goes to a file it does not name; the old text owed that + location unconditionally, run 20's five readers graded the fixture's [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) Partial on it, and the + key's intended Yes already assumed the new reading. Clarity edits: clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) says it asks + for the mechanism and that missing documentation for a bundled rule fails clause [6.3](DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), as + clause [6.4](DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation)'s Why now says too, and says plainly why a path in the message is not enough; + clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) says the project writes the detecting rule in the detector, of the kind clause [4.1](DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) requires, and that the detector need not remove or forbid the route; clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) says what a gap is, that a detector declaring a version records it there, and that conformance is judged on evidence with or without a declaration. Accepted by run NNN. Minor version when diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index 3ffc1e8..e9d5f23 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -158,8 +158,8 @@ any. ### 5.3 The result MUST reach the practitioner in the output of the command they ran Where the [Detector] writes fuller detail elsewhere, the invoked command's own output MUST -still carry a usable summary: every finding, with where it is and its [Identifier]. If that -output leaves any finding out, it MUST also give the location of the rest. Output that prints +still carry a usable summary, each finding in it with where it is and its [Identifier]. If the +summary leaves any finding out, the output MUST also give the location of the rest. Output that prints every finding meets this clause, even where extra detail, such as a code snippet, is written only to a file the output does not name. @@ -244,7 +244,7 @@ An automated check over every [Identifier] a [Bundled rule] can print, applying family list or pattern of clause 6.5 where one is used, that blocks the release when any of them lands on no page. -**Why**: clause 6.1 is the clause most easily believed to be satisfied whilst being broken, because +**Why**: clause 6.3 is the clause most easily believed to be satisfied whilst being broken, because the documentation is written by the same person who wrote the [Rule] and its absence is invisible from the inside. Where the [Detector] is shipped inside a [Toolchain], the [toolchain specification](TOOLING-SPEC.md)'s self-audit makes this check a MUST for the [Toolchain]; a @@ -278,7 +278,7 @@ An inline ignore comment, a per-line directive and a generated [Baseline] are al routes. The [Detector] MAY ship them. For each one it MUST do at least one of two things, and either alone satisfies this clause: make the route disableable by configuration; or make the route detectable. Detectable means the project can find every use of the route, either with a [Rule] -the project writes itself, of the kind clause 4.1 requires the [Detector] to host, or with a +the project writes itself in the [Detector], of the kind clause 4.1 requires it to host, or with a mechanical check the [Detector] documents. The [Detector] is not required to remove the route or to forbid it. Either of the two things lets a project which decides to forbid the route enforce that decision. A route that can be neither switched off nor seen does not [Conform]. diff --git a/SPEC.md b/SPEC.md index 7b41b22..6a71dea 100644 --- a/SPEC.md +++ b/SPEC.md @@ -194,8 +194,8 @@ clause 3.1's bounds is itself the evidence that the [Defect] is out of scope, an more reliable than a judgement made before trying. The conclusion that no pattern exists MUST be stated in one sentence, naming at least two independent techniques tried at expressing the [Rule], and recorded where the project's other decisions are enumerable under clause 8.7, not -only with the fix. Two techniques are independent, as clause 3.1 defines it for a search, when -they would miss different things. It is the cheapest way +only with the fix. Two techniques are independent when they would miss different things, as +clause 3.1 says of a search. That conclusion is the cheapest way out of the method and the only one that would otherwise leave no trace; enumerating it is how an [Owner] sees an [Agent] routing around the method, and naming the techniques is what lets them judge whether the attempt was one. @@ -436,9 +436,9 @@ write, the [Practitioner] MUST name the [Detector] and its kind. Where that kind record, for each step down from [Semantic], which of the two grounds holds and why, naming the [Detectors] of the stronger kind that were considered, on the [Rule]'s [Remediation docs] and in the description of the commit that adds the [Rule], moves it to another [Detector], or first -records the reason. The [Practitioner] judges whether a reason holds, as section 4 gives them the choice of [Detector]; a +records the grounds. The [Practitioner] judges whether the grounds hold, as section 4 gives them the choice of [Detector]; a project MAY require its [Owner]'s sign-off on a weaker kind as well, by recording that -requirement where it records its [Calibrations]; the sign-off is then recorded with the reason, +requirement where it records its [Calibrations]; the sign-off is then recorded with the grounds, and where the project has recorded no such requirement none is needed. **A worked example.** Take a [Rule] against calling a method named `context` on one type. A @@ -529,9 +529,9 @@ what must survive the merge is not. **Part B: [Narrowing].** -A [Narrowing] is a change to the [Rule] itself, so that it stops matching some code. An -[Instance] left in place behind an inline ignore comment, an ignore entry or a [Baseline] is not -a [Narrowing]: the [Rule] is unchanged and still matches it. That is a [Suppression] of the +Leaving an [Instance] in place behind an inline ignore comment, an ignore entry or a [Baseline] +is not a [Narrowing], whatever note accompanies it: the code still carries the [Hazard]. That is +a [Suppression] of the [Instance], and clauses 3.4 and 3.5 and section 4 govern it, not this clause. **The decision comes first, and it turns on one sentence.** Can the [Practitioner] write down why @@ -1120,10 +1120,10 @@ This appendix restates sections 2 to 4; where the two differ, the sections gover > or the language has none that takes custom [Rules]: none the project runs, in any configuration > it can use without a hosted service or a licence tier, and none its ecosystem offers outside > one. Name the [Detector] and its kind in the [Rule]'s documentation. Where you use a weaker one, -> write down which reason holds, and why, naming the stronger [Detectors] you considered, there and -> in the description of the commit that adds the [Rule], moves it to another [Detector], or first -> records the reason, and where the project requires its [Owner]'s sign-off on a weaker one, get -> it and record it with the reason. Draw +> write down which of those two grounds holds, and why, naming the stronger [Detectors] you +> considered, there and in the description of the commit that adds the [Rule], moves it to another +> [Detector], or first records the grounds, and where the project requires its [Owner]'s sign-off +> on a weaker one, get it and record it with the grounds. Draw > the [Rule] so that it catches more than the single [Instance] you started from, but never so > broadly that it matches code which does not carry the [Hazard]. > diff --git a/defence-before-fix-project-prompt.md b/defence-before-fix-project-prompt.md index 8b55913..84b5a88 100644 --- a/defence-before-fix-project-prompt.md +++ b/defence-before-fix-project-prompt.md @@ -19,10 +19,10 @@ then one that sees text. Use a weaker one only where the stronger one cannot exp or the language has none that takes custom Rules: none the project runs, in any configuration it can use without a hosted service or a licence tier, and none its ecosystem offers outside one. Name the Detector and its kind in the Rule's documentation. Where you use a weaker one, -write down which reason holds, and why, naming the stronger Detectors you considered, there and -in the description of the commit that adds the Rule, moves it to another Detector, or first -records the reason, and where the project requires its Owner's sign-off on a weaker one, get -it and record it with the reason. Draw +write down which of those two grounds holds, and why, naming the stronger Detectors you +considered, there and in the description of the commit that adds the Rule, moves it to another +Detector, or first records the grounds, and where the project requires its Owner's sign-off +on a weaker one, get it and record it with the grounds. Draw the Rule so that it catches more than the single Instance you started from, but never so broadly that it matches code which does not carry the Hazard. From ba8a0bd2d4f5cf62d7663ec89edffd7721754db9 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:25:01 +0100 Subject: [PATCH 030/105] Keys: quotations follow the review rulings acceptance/method/KEY.md, 2d: the clause 3.2 quotation now ends "or first records the grounds". acceptance/detector/KEY.md, 2b: the clause 7.1 quotation now reads "a Rule the project writes itself in the Detector, of the kind clause 4.1 requires it to host". No question, threshold, fixture or answer changed. Co-Authored-By: Claude Opus 5.5 --- acceptance/detector/KEY.md | 2 +- acceptance/method/KEY.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index e76ad3c..7a0cb42 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -41,7 +41,7 @@ agent summary (those belong to the toolchain specification). it. Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each one it MUST do at least one of two things, and either alone satisfies this clause: make the route disableable by configuration; or make the route detectable. Detectable means the project can find every use of the route, either with a Rule - the project writes itself, of the kind clause [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) requires the Detector to host, or with a + the project writes itself in the Detector, of the kind clause [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) requires it to host, or with a mechanical check the Detector documents." An answer that says the detector must forbid the route, or must require a reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD), is wrong. - **2c.** With the rule, in the installed copy, at a version tracked together, and yes, offline. diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 3e812db..865f259 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -60,7 +60,7 @@ is correct; a bullet that says no weaker kind is ever allowed is wrong. Semantic, the Practitioner MUST also record, for each step down from Semantic, which of the two grounds holds and why, naming the Detectors of the stronger kind that were considered, on the Rule's Remediation docs and in the description of the commit that adds the Rule, moves it to - another Detector, or first records the reason." Supporting: "On the Remediation docs of a Rule + another Detector, or first records the grounds." Supporting: "On the Remediation docs of a Rule they write, the Practitioner MUST name the Detector and its kind." A correct answer gives three elements, and may leave out everything else above: (i) the strongest kind that can express the rule, in the order semantic, syntactic, textual; (ii) a weaker kind only where the stronger From 7a4bb2dd82c74d5ac206db5a3afa355413106e94 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:25:45 +0100 Subject: [PATCH 031/105] Acceptance process: review 8, a set-aside sheet's answers still count towards a failure A run fails if any criterion fails with a set-aside reader's answered items counted alongside the others, so setting a reader aside can never turn a failing run into a void one. The fourth row's clearing test covers every paragraph the earlier passage lay in. acceptance/CHANGES.md names runs 19 and 20 as made under this change. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 6 +++--- acceptance/CHANGES.md | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 43f23e1..cb0f011 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -33,8 +33,8 @@ test is run with a low-strength model so that clarity is measured at the floor, cohort's model. No reader is set aside for any other cause, and the record shows the cause. If any reader is set aside, the run is void and a fresh cohort runs, unless the readers whose answers count already fail a criterion, when the run fails; a run never passes with a reader set - aside. A set-aside reader's sheet is still marked, and a miss traceable to the text in it fails - the run. A void run takes a run number, has a record and posted sheets like any other, and + aside. A set-aside reader's sheet is still marked, and the run fails if any criterion fails when its + answered items are counted with the other readers', or if it has a miss traceable to the text. A void run takes a run number, has a record and posted sheets like any other, and counts towards the number of runs. - Each reader receives **only its packet**: the document's question set, the changed document @@ -165,7 +165,7 @@ one" is four of five. | Load-bearing statements (question 2) | **Every** reader correct on every one | | Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | | A miss traceable to the text | **None.** Any such miss, as [Marking](#marking) defines it, is a finding, whatever the count | -| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it: no reader of the fresh cohort names a passage that covers a sentence of the paragraph the earlier passage lay in, as that paragraph reads in the changed document, or a sentence the change kept from it | +| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it: no reader of the fresh cohort names a passage that covers a sentence of each paragraph the earlier passage lay in, as that paragraph reads in the changed document, or a sentence the change kept from it | | Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | Every row must hold, and the last covers only what none of the others catches. The first and diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 56e6350..eb74588 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -6,7 +6,7 @@ not a specification and is not versioned, and a change to it is not tested by a to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). - [Pull request #13](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/13): the cohort can be run and marked without judgement calls that the rules - leave open; this applies to runs made after it merges, and to run 19 if made under it, and + leave open; this applies to runs made after it merges, and to runs 19 and 20, which were made under it at a695010, and runs 16 to 18 stand as recorded. A cohort is an odd number of readers started, at least five, larger at the editor's choice the more obligations the change touches, each a separate process with no tools and no project context, started from @@ -14,8 +14,8 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). `CLAUDE.md`, skills, plugins, hooks, MCP servers or saved session. A reader is set aside only if its answer text is empty, its process reports an error, it shared the editor's context, it was not started as required or it reported another model, and a run with a reader set - aside is void, or fails if the readers who count already fail it or its own sheet has a miss - traceable to the text; a void run has a number, a record and posted sheets. The packet is the question set, the document and the fixture, each + aside is void, or fails if the readers who count already fail it, if it fails with the set-aside + sheet's answered items counted too, or if that sheet has a miss traceable to the text; a void run has a number, a record and posted sheets. The packet is the question set, the document and the fixture, each after one `===== FILE: =====` line. A new marking section says what a delegated marker receives and that its verdict is final unless a rule was misapplied, that an open judgement is made as a careful reader of the document would, and defines a keyed item, quoting (consecutive From 204f95b0957f20fb869e20d42fb5257b7c59b97e Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:30:15 +0100 Subject: [PATCH 032/105] Acceptance process: review 9, the fourth row's clearing test covers any earlier passage A fresh reader naming a sentence of any paragraph an earlier confusing passage lay in keeps the finding open, as the keys say. "Status paragraph" is used as in CONTRIBUTING.md step 5, and the method key's 2a lists every element an answer is wrong to omit. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 4 ++-- acceptance/method/KEY.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index cb0f011..8c396dd 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -165,7 +165,7 @@ one" is four of five. | Load-bearing statements (question 2) | **Every** reader correct on every one | | Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | | A miss traceable to the text | **None.** Any such miss, as [Marking](#marking) defines it, is a finding, whatever the count | -| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it: no reader of the fresh cohort names a passage that covers a sentence of each paragraph the earlier passage lay in, as that paragraph reads in the changed document, or a sentence the change kept from it | +| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it: no reader of the fresh cohort names a passage that covers a sentence of any paragraph an earlier passage lay in, as that paragraph reads in the changed document, or a sentence the change kept from one | | Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | Every row must hold, and the last covers only what none of the others catches. The first and @@ -207,7 +207,7 @@ the release branch needs no cohort of its own unless it changes specification te change is any change to the document's text after the readers read it, from any source (a review finding, the editor, or `next` merged into the branch), except a version number or date that `PUBLISHING.md` or step 5 of `CONTRIBUTING.md` has the editor set, in the header, the status -section, the citation or the closing table. Any other change to those paragraphs is a wording +paragraph, the citation or the closing table. Any other change to those paragraphs is a wording change. Readers are never reused. ## The record diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 8839bc7..de03f0d 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -29,7 +29,7 @@ a test may serve as the detector; an agent may baseline the remaining instances. then sweep and fix every instance, in that order. Key quotations, section [3](../../SPEC.md#3-the-method): "Six clauses, in order."; [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance): "The Practitioner MUST run the Rule across the entire codebase and record the total Instance count before fixing anything, and MUST then fix every Instance found rather - than only the one". An answer that omits the rule, the proof or the sweep, or has the fix + than only the one". An answer that omits the class, the rule, the proof, the sweep or the fix of every instance, or has the fix before the rule or the sweep before the proof, is wrong. - **2b.** The owner of the codebase, always a human; an agent may not. Key quotation, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): "An Agent MUST NOT Baseline, suppress or knowingly leave an Instance unfixed on its own From 3b848c7731f5bf12450fd7cacf4f050f37fa3249 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:30:17 +0100 Subject: [PATCH 033/105] Detector 5.3 and method 3.3 Part B: the review's last readings settled Detector 5.3: where the summary leaves a finding out, the output says where the findings it leaves out can be read, and only output that prints every finding so meets the clause outright. Method 3.3 Part B: the instance left behind an ignore is one that carries the hazard, so a rule's match on its own source, which Part C narrows, is not read as a suppression. Co-Authored-By: Claude Opus 5.5 --- DETECTOR-SPEC.md | 4 ++-- SPEC.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index e9d5f23..c0f7036 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -159,8 +159,8 @@ any. Where the [Detector] writes fuller detail elsewhere, the invoked command's own output MUST still carry a usable summary, each finding in it with where it is and its [Identifier]. If the -summary leaves any finding out, the output MUST also give the location of the rest. Output that prints -every finding meets this clause, even where extra detail, such as a code snippet, is written only +summary leaves any finding out, the output MUST also say where the findings it leaves out can be read. Output that prints +every finding so meets this clause, even where extra detail, such as a code snippet, is written only to a file the output does not name. **Why**: method specification clause 8.2. A report the [Practitioner] has to go and find is a diff --git a/SPEC.md b/SPEC.md index 6a71dea..22848af 100644 --- a/SPEC.md +++ b/SPEC.md @@ -529,8 +529,8 @@ what must survive the merge is not. **Part B: [Narrowing].** -Leaving an [Instance] in place behind an inline ignore comment, an ignore entry or a [Baseline] -is not a [Narrowing], whatever note accompanies it: the code still carries the [Hazard]. That is +Leaving an [Instance] that carries the [Hazard] in place behind an inline ignore comment, an +ignore entry or a [Baseline] is not a [Narrowing], whatever note accompanies it. That is a [Suppression] of the [Instance], and clauses 3.4 and 3.5 and section 4 govern it, not this clause. From 27b4ce6e9b9eedd2a15a64590ea04dec5fd2a9ca Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:30:40 +0100 Subject: [PATCH 034/105] Acceptance runs 19 and 20: SPEC.md and DETECTOR-SPEC.md, first cohort, both FAIL Five cold Haiku readers each, made under pull request 13 at a695010 and marked by a delegated marker. Answer sheets, prompt, command and the marker's instruction are posted on pull request 14 and linked from each record. The findings applied to the text are listed in each record. Co-Authored-By: Claude Opus 5.5 --- .../runs/019-semantic-detectors-run-1.md | 190 +++++++++++++++++ .../runs/020-semantic-detectors-run-2.md | 191 ++++++++++++++++++ 2 files changed, 381 insertions(+) create mode 100644 acceptance/runs/019-semantic-detectors-run-1.md create mode 100644 acceptance/runs/020-semantic-detectors-run-2.md diff --git a/acceptance/runs/019-semantic-detectors-run-1.md b/acceptance/runs/019-semantic-detectors-run-1.md new file mode 100644 index 0000000..d2ff6e7 --- /dev/null +++ b/acceptance/runs/019-semantic-detectors-run-1.md @@ -0,0 +1,190 @@ +# Run 19: SPEC.md, semantic detectors, first run + +- **Document**: SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + branch `editor/semantic-detectors` at 5b6093f, the commit the readers were given +- **Cohort**: five readers, model alias `haiku`; each reader reported `claude-haiku-5-5` +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input; no tools, no settings, no project context. + Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt` +- **Managed settings**: the machine applies no managed (policy) settings to the readers; no + managed settings file is present +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. Every sheet has answer text, no process reports an error + (`is_error` false, subtype `success`), and every reader reported the cohort's model. The editor + reports no reader set aside for a cause the sheets cannot show. +- **Run within the pull request**: 1, counting void runs; runs in the pull request so far: 1 +- **Marker**: delegated, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with the + marking instruction and its agent definition, having seen neither the drafting nor the review. + It read `ACCEPTANCE.md` and `acceptance/runs/README.md` in this worktree, which differ from + commit a695010 only by this pull request's question-set line for 2d, and the document, question + set, key and fixture at 5b6093f. The packet's three files match that commit, apart from one + blank line between the end of each file and the next file's marker line. + +## Marks + +`ok` is correct, `miss` is a miss; `T` marks a miss traceable to the text; `J1` and `J2` mark a +miss that rests on the judgement of that name below. + +| Reader | Q1 | 2a | 2b | 2c | 2d | 3.1 | 3.2 | 3.3 | 3.4 | 3.5 | 3.6 | Remediation | Defence | Section 4 | Q4 passages | +| ------ | ------- | --------- | --------- | --------- | ------ | ------- | ------- | -------------- | --- | --- | --- | ----------- | ------- | --------- | ----------- | +| 1 | correct | miss T J2 | miss T J1 | miss T J2 | miss T | Partial | Partial | Partial miss T | No | No | No | ok | ok | ok | two | +| 2 | correct | miss T | miss T J1 | miss T J2 | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 3 | correct | miss T | ok | miss T | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 4 | correct | miss T J2 | miss T J2 | miss T | miss T | No | Partial | Partial miss T | No | No | No | ok | ok | ok | three | +| 5 | correct | miss T J2 | ok | miss T J2 | miss T | Partial | Partial | Partial miss T | No | No | No | ok | ok | ok | three | + +Every grade under 3.1, 3.2 and 3.4 to 3.6 is one the key accepts, and every 3.1 grade of Partial +or No notes that no independent search technique is recorded. + +### Question 1 + +Five of five correct. Every sheet has at least four bullets naming obligations the key lists, and +no bullet asserts an obligation the document does not state. Reader 1's "neither a single +instance nor every use of a language feature" is supported by section 7 ("drawn to a [Class] +within both bounds and not to the reported [Instance]"); reader 4's search "chosen and run before +the rule exists" is supported by clause 3.1, which has the search recorded before clause 3.2 +begins. + +### Question 2 + +Every reader quoted words of the document for every item, so every miss here is traceable to the +text. + +- **2a**, none of five. The sentence states the order only partly or not at all: + - Reader 1 states class, defence built, rule proven, sweep, every instance fixed, in order; it + says the rule "is proven", not that it fires, and "the defence" rather than a detector rule + (J2). + - Reader 2: "attribute the defect to a class and build and prove the defence"; no sweep and no + fixing of every instance. + - Reader 3: "The Defence is built and proven first, then the Defect is fixed"; no class + attribution, no sweep, no fixing of every instance. + - Reader 4: "then sweep, then fix"; the fix is not stated as every instance (J2). + - Reader 5: "sweeps for every instance before fixing anything, then fixes the defect"; the fix + is stated as the defect, not every instance (J2). Its clause 3.4 quotation after the first + quotation is not part of the sentence (J1). +- **2b**, two of five (readers 3 and 5, who say the owner "is always a human" in the sentence). + Readers 1 and 2 say the owner is "Always a human" only after the quotation (J1); reader 4 does + not say it (J2: "the owner" is not read as carrying its definition). All five quote the key's + section 4 sentence and say an agent may not. +- **2c**, none of five. No sentence says the reviewer reproduces the red and green runs; each says + only that the verdict rests on "reproduction" (J2). Readers 3 and 4 also leave "not on the + report" out of the sentence: reader 3 has it only in its quotation, and reader 4's sentence + ends "The verdict rests on reproduction:" before the quotation. No reader accepts the pull + request description as the verdict. +- **2d**, none of five. No sentence states the second ground for a weaker kind, that the language + has no detector of the stronger kind that accepts bespoke rules: + - Reader 1: "a weaker kind allowed only where the stronger cannot"; one ground; no order of + kinds and no place for the record in the sentence (J2 for the order). + - Reader 2: "A weaker kind is allowed only where the stronger cannot express it"; one ground, + though its quotation gives both; no order and no place (J2 for the order). + - Reader 3: "only where the stronger one cannot express the Rule"; one ground; no place in the + sentence, only in the quotation. + - Reader 4: the sentence before the quotation states only the strongest kind; it states no + ground at all, and the record and its places come after the quotation (J1). + - Reader 5: "only when a stronger one cannot express the rule", then "which of the two + reasons holds and why" without stating the second; places only after the quotation (J1). + Whether (iii) needs a place named is a judgement (J3); it changes no mark, since every sheet + already misses (ii). + +### Question 3 + +- **3.3**, two of five (readers 2 and 3, Yes). Readers 1, 4 and 5 grade Partial because the + inline ignore on `Report::render()` is a Suppression and not a Narrowing under clause 3.3's + Part B, and also fail 3.4 on it. The key excuses a reader who attributes the suppression to 3.3 + "rather than" 3.4; these readers attribute it to both, which that sentence does not cover, so + each is a miss. Each reasoning states what the document requires (reader 1: "a suppression, + with no searched sentence, and it was not referred"; reader 4: "That makes it a Suppression + under 3.3, not a Narrowing"; reader 5: "its sentence does not address the hazard"), so each is + traceable. +- **Remediation**: five of five say it does not conform, each naming at least one failing clause + the key accepts. +- **Defence**: five of five say it does not conform, each naming 3.6. +- **Section 4**: five of five name the inline ignore on `Report::render()`; readers 1, 2, 4 and 5 also + mention the unbuilt wider rule as a second or arguable decision, which the key accepts alongside + it. + +### Judgements made in marking + +Each takes the stricter reading, the one that records more misses, as `ACCEPTANCE.md` requires +where its rules leave a judgement open. + +- **J1. The answer's sentence is what precedes its quotation.** Question 2 asks for one sentence + followed by a quotation, and a mark turns on "its sentence". Text the reader gives before the + quotation is read as the sentence however it is punctuated, since the questions ask two or + three things and nothing picks out one full stop; text after the quotation is outside the form + the question asks for and is not read as part of the sentence. Affects 2b (readers 1, 2), 2d + (readers 4, 5) and 2a (reader 5). +- **J2. An element of the key's answer is stated only when the sentence says it.** A less + specific word does not state the more specific element ("fix" is not "fix every instance", + "reproduction" is not "reproducing the red and green runs", "proven" is not "fires", "the + strongest kind" is not the order semantic, syntactic, textual), and a defined term is not read + as carrying its definition into the sentence ("the owner" is not "always a human", "the + defence" is not "a detector rule"). Affects 2a (readers 1, 4, 5), 2b (reader 4), 2c (readers + 1, 2, 5) and the order in 2d (readers 1, 2, 4). Runs 17 and 18 accepted "reproduction" for 2c; + those records are examples of style, not of marking. +- **J3. 2d's element (iii) needs a place named.** The key says naming one of the two places "is + enough", read as requiring at least one. No mark turns on it. +- **The 3.3 excuse is read as written**, covering a reader who attributes the suppression to 3.3 + rather than 3.4, not one who attributes it to both. +- **Not treated as open: the remediation verdict.** The key's section 7 line gives "does not + conform" as the intended answer, and its "because 3.2, 3.4 and 3.6" is reasoning, not a list a + reader must reproduce: it leaves out 3.5, which the key also has failing, and where the key + wants a clause named, for the defence, it says so. + +## Passages quoted as confusing + +Same passage as `ACCEPTANCE.md` defines it: sentences in common, or the same paragraph. + +| Passage | Clause | Readers | Words quoted, verbatim | +| ------------------------------------------------------------------- | -------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| The level paragraph, "Both bounds measure breadth within one level" | 3.1 | three: 1, 2, 3 | 1 and 3: "Both bounds measure breadth within one level; the level itself is a separate question, answered against the report." 2: "Where the [Defect] was reported as a behaviour, ... the [Class] a [Detector] can read usually sits one level below that \[Hazard\]: the mechanism of this [Instance], not the failure the report opens with." The bold sentence and the "Where the [Defect]" sentence are one paragraph. | +| The extension-points sentence | 3.2 | three: 3, 4, 5 | "a [Rule] whose construct its extension points cannot reach is one that [Detector] cannot express, and one its kind cannot express where no other [Detector] of that kind reaches it." | +| The search vocabulary in "Attempt rather than pre-judge" | 2 | two: 2, 5 | "naming at least two independent techniques tried at expressing the [Rule], in the vocabulary clause 3.1 uses for a search" | +| The scale at which a baseline is allowed | 3.4 | two: 4, 5 | "and the scale at which that becomes true is tremendous." | +| The default owner | 1, Owner | one: 1 | "Where the project has recorded nobody, the [Owner] is whoever instructed the [Practitioner], and [Escalation] goes there." | +| The limit on the wider-rule bullet | 4 | one: 4 | "This does not reopen a [Class] whose bounds a [Hazard] sentence has settled, and it reaches no further than the one wider [Rule] already on the record." | + +Of the four passages named by two or more, only the clause 3.2 sentence is text this pull request +adds; the other three predate it. The criterion does not distinguish, so all four are findings. +Reader 2's remarks on what it could not tell are not suggestions and are not recorded as +passages. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in 87536c5, 66301b9 and 3b848c7 +(key quotations kept verbatim in 85110c1 and ba8a0bd), after an independent review of each round of +edits. SPEC.md: clause 3.2 sets out the two grounds for a weaker kind as a numbered pair and says +the extension-points rule in two plain sentences; clause 3.3 Part B says an instance left behind an +inline ignore is a Suppression, governed by 3.4, 3.5 and section 4, not a Narrowing; clause 3.1 says +what a class's level is; section 2 says what independent techniques are; clause 3.4 says plainly +that a baseline is rare. Separately, pull request 13 changed how question 2 is marked (sentence and +quotation read together, each element for its gist) after this run, so the misses that rested on +judgements J1 and J2 needed no text change. A fresh cohort follows. + +## Process change + +Made under a process change before it merged: yes, pull request 13 at commit a695010, named +before the readers started. + +## Verdict + +**FAIL.** Against the criteria in `ACCEPTANCE.md`: + +- Load-bearing statements: not met. 2a none of five, 2b two of five, 2c none of five, 2d none of + five. Without J1 and J2 the result is still a fail: 2d none of five on the second ground, 2a + three of five (readers 2 and 3 miss), 2c three of five (readers 3 and 4 miss). +- Restatement and application: not met on 3.3, two of five; every other keyed item five of five. +- Misses traceable to the text: every miss on question 2, and 3.3 for readers 1, 4 and 5; each is + a finding. +- Passages found confusing by two or more: four, each a finding that blocks until the text is + changed and a fresh cohort no longer names it: clause 3.1's level paragraph, clause 3.2's + extension-points sentence, section 2's search-vocabulary sentence and clause 3.4's + "tremendous" sentence. +- Single-reader notes, recorded: section 1's default owner (reader 1); section 4's limit on the + wider-rule bullet (reader 4). + +Answer sheets, the reader prompt, the command and the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090071898), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090072435), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090072816), [comment 4](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090073231). diff --git a/acceptance/runs/020-semantic-detectors-run-2.md b/acceptance/runs/020-semantic-detectors-run-2.md new file mode 100644 index 0000000..83e9795 --- /dev/null +++ b/acceptance/runs/020-semantic-detectors-run-2.md @@ -0,0 +1,191 @@ +# Run 20: DETECTOR-SPEC.md, semantic detectors, second run in pull request 14 + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + editor's branch `editor/semantic-detectors` at 5b6093f; the question set, key and fixture read + at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input, with no tools, no settings, no project context + and no saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is the question set, the document and the fixture at + 5b6093f, in that order, each preceded by one `===== FILE: =====` line and nothing else + added. +- **Managed (policy) settings**: none; the machine applies no managed settings and no managed + settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. The editor reports none set aside for a cause the sheets cannot + show; every sheet has answer text, no process reports an error (`is_error` false, subtype + `success`), and every reported model is the cohort's model +- **Run within the pull request**: 2, counting void runs; run 19, on SPEC.md, was made at the same + time and does not bear on this document +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review +- **Verdict**: FAIL + +## Marks + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | Section 8 | Kind | 6.4 | Q4 passages | +| ------ | ------- | ---- | ---- | ---- | --- | --- | --- | --- | --- | ---- | --- | ---- | --- | --- | --- | --------- | ---------- | --- | ----------- | +| 1 | correct | miss | miss | miss | ok | ok | ok | ok | ok | miss | ok | miss | ok | ok | ok | ok | not marked | ok | 2 | +| 2 | correct | miss | miss | ok | ok | ok | ok | ok | ok | miss | ok | miss | ok | ok | ok | ok | not marked | ok | 1 | +| 3 | correct | ok | miss | miss | ok | ok | ok | ok | ok | miss | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 4 | correct | miss | miss | miss | ok | ok | ok | ok | ok | miss | ok | miss | ok | ok | ok | ok | not marked | ok | 4 | +| 5 | correct | miss | miss | miss | ok | ok | ok | ok | ok | miss | ok | miss | ok | ok | ok | ok | not marked | ok | 2 | + +Every miss above is traceable to the text: each one's reasoning quotes the document or states +what a clause requires. No question was left unanswered. + +### Question 1 + +All five restatements name four or more obligations from the key's list (4.1, 4.2, 4.3, 5.1 to +5.4, 6.1 to 6.3, 7.1), and none asserts an obligation the document does not state. Correct, five +of five. + +### Question 2 + +The key's answers state these things, and the key says none of them may be left out: + +- 2a: (a) Conformance is satisfying every MUST in sections 4 to 7; (b) judged on evidence; (c) a + detector that predates the document can conform; (d) the declaration is optional. +- 2b: (a) make the route disableable by configuration, or (b) detectable by a Rule the project can + write or a mechanical check the Detector documents; (c) the detector is not required to remove or + forbid it. +- 2c: (a) with the rule; (b) in the installed copy; (c) at a version tracked together; (d) yes, + reachable offline. + +| Reader | 2a | 2b | 2c | +| ------ | -------------------------------------------- | -------------------------------------------------------------------- | -------------------------------------- | +| 1 | miss: (b) only in the quotation | miss: (c) not stated; the means of detection only in the quotation | miss: (a) not stated (judgement 3) | +| 2 | miss: (b) and (d) only in the quotation | miss: (c) not stated; the means of detection only in the quotation | ok: all four, supported by 6.2 and 6.3 | +| 3 | ok: all four, supported by the 8.1 quotation | miss: (c) not stated; the means of detection only in the quotation | miss: (c) not stated | +| 4 | miss: (d) not stated (judgement 2) | miss: (c) not stated; the quotation states no key item (judgement 4) | miss: (b) not stated (judgement 3) | +| 5 | miss: (d) not stated (judgement 2) | miss: (c) not stated; the quotation states no key item (judgement 4) | miss: (c) not stated | + +Each miss is traceable: every reader's answer quotes the document in support. + +### Question 3 + +The key's intended verdicts are Yes for every clause but 6.3, which is No; the section 8 verdict +is "does not conform", because 6.3 fails; 6.4 does not change the verdict. + +- **5.3, all five readers Partial against Yes.** Each grades 5.3 Partial because the fixture does + not say the command's output gives the location of `.lintomatic/report.html`. Readers 2, 4 and 5 + state the requirement outright ("the clause requires both halves"; "which the clause requires + alongside it"; "5.3 requires it to"); reader 1 uses the clause's words "usable summary"; reader + 3 reports the fixture's silence in the row graded against 5.3 (judgement 5). All five are + traceable to the text. +- **6.1, readers 1, 2, 4 and 5 Partial against Yes.** Each holds that the two bundled identifiers + printing `No documentation declared` do not resolve to documentation, which each attributes to + 6.1 ("their identifiers resolve to nothing"; "which is not documentation"; "which resolves to no + documentation"). Traceable. Reader 3 graded Yes. +- **6.3**: No from all five, each citing that one unresolved identifier fails the clause. No reader + graded it Partial on the grounds that thirty-eight resolve. +- **Section 8**: all five say Lintomatic does not conform and name 6.3 as a failure; readers 1, 2, + 4 and 5 also count their 5.3 and 6.1 Partials, and reader 3 its 5.3 Partial, which repeats those + misses but does not change the verdict marked here. +- **Kind**: all five say the fixture does not let them name the kind and name none; reader 4 says + outright that this does not change the verdict. No reader made the verdict turn on the kind, so + none is marked on it. +- **6.4**: all five say it does not change the verdict, because it is a SHOULD. + +### Judgements made in marking + +Each takes the stricter reading, the one recording more misses or findings, as `ACCEPTANCE.md` +requires where its rules leave a mark open. + +1. **What counts as "its sentence" under question 2.** The rule asks that the answer's sentence + state each thing the key's answer states and that a quotation support it, so a key item that + appears only inside the reader's quotation is not stated by the sentence. Where a reader wrote + two sentences of answer before or between its quotations, both are taken as the answer. +2. **2a (d), readers 4 and 5.** Reader 4 says a detector "can be graded on evidence without its + maintainer knowing the document", and reader 5 that it is "judged on evidence rather than a + declaration". Each implies that no declaration is needed; neither says the declaration is + optional. Marked not stated. +3. **2c (a) and (b), readers 1 and 4.** Reader 1 says the documentation "must ship in the + installed package, at a version tracked with the rule", which does not say it ships with the + rule; reader 4 says it "must ship on disk with the rule", which does not say the installed + copy. Each marked not stated. +4. **2b quotations, readers 4 and 5.** Both quote only "For each one it MUST do at least one of + two things, and either alone satisfies this clause", which names neither thing and does not say + that forbidding is not required, so it does not by itself state a key item. A further reason + for the miss, already a miss on (c). For readers 1 to 4, "make it detectable" without its + means is also taken as short of (b). +5. **5.3, reader 3.** Its reasoning reports only the fixture's silence on the report's location, + but it is a grade against clause 5.3 that the clause is partly unmet, and naming the clause a + case fails attributes a requirement to it. Traceable. +6. **Question 4, a clause named as the context of a confusion.** Readers 1, 3 and 4 explain their + 7.1 confusion by naming clause 4.1 ("Clause 4.1 has the project writing its own rules"; "does + not fit 4.1's authoring route"; "given 4.1 describes project rules as separate"). Each is taken + as naming clause 4.1, which covers every sentence of it. Likewise reader 3's "given that the + declaration is optional" is taken as naming the 8.1 sentence "The declaration is optional.", + and reader 4's "the table in section 2" as naming that table. +7. **Question 4, single words quoted.** Reader 5's "optional" and "not a condition of" are + quotations under question 4, and each covers the 8.1 sentence "The declaration is optional." + or "It is how a maintainer claims Conformance; it is not a condition of it.", the only places + those words occur. + +No re-mark. + +## Passages quoted as confusing + +Grouped by paragraph, as `ACCEPTANCE.md` defines the same passage. + +| Clause and paragraph | Readers | Words quoted, verbatim | Blocks | +| --------------------------------------------------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------ | +| 7.1, first paragraph, the sentence beginning "For each one it MUST" | 1, 2, 3, 4, 5 | 1, 2: "make the route detectable, by a [Rule] the project can write in the [Detector] itself or by a mechanical check the [Detector] documents." 3, 4, 5: "by a [Rule] the project can write in the [Detector] itself." / "make the route detectable, by a [Rule] the project can write in the [Detector] itself" / "make the route detectable, by a [Rule] the project can write in the [Detector] itself." | Yes | +| 8.1, first paragraph, the sentence beginning "The same declaration" | 3, 4, 5 | 3, 4: "a [Detector] that has learnt, from its own checks or from a [Practitioner]'s report under the method's clause 3.2, that it fails a MUST in sections 4 to 7 MUST record that gap alongside the version it declares" (reader 3 with a closing full stop). 5: "a [Detector] that has learnt … that it fails a MUST in sections 4 to 7 MUST record that gap alongside the version it declares." | Yes | +| 8.1, second paragraph, "The declaration is optional." and the next sentence | 3, 4, 5 | 4: "The declaration is optional." 5: "optional"; "not a condition of". 3, by description: "given that the declaration is optional" (judgements 6 and 7) | Yes | +| 6.1, the paragraph beginning "For a [Bundled rule]", its last sentence | 1, 4 | 1, 4: "A [Message] that carries its own documentation path resolves that [Message], not the [Identifier]." | Yes | +| 4.1, named by clause number | 1, 3, 4 | No quotation; named as the context of the 7.1 confusion (judgement 6) | Yes | +| 6.1, the paragraph beginning "For a project's own [Rule]" | 4 | "Where the [Identifier] is itself a URL, as method clause 3.6 allows" | No | +| 1, second paragraph, the sentence beginning "The two are separable" | 2 | "installed into projects its maintainer will never see" | No | +| 2, the terminology table | 4 | No quotation; "the table in section 2", named as not defining "Message" (judgement 6) | No | + +Of these, only the section 2 table lies in text this pull request changed; the passages that block +all lie in text it kept. + +## Against the criteria + +- **Load-bearing statements (every reader on every one)**: not met. 2a one of five, 2b none of + five, 2c one of five. +- **Restatement and application (all but one on each keyed item)**: not met. Question 1 five of + five; 5.3 none of five; 6.1 one of five; every other clause, the section 8 verdict and 6.4 five + of five. +- **A miss traceable to the text (none)**: not met. Twenty-two, listed per reader above: 2a by + readers 1, 2, 4 and 5; 2b by all five; 2c by readers 1, 3, 4 and 5; 5.3 by all five; 6.1 by + readers 1, 2, 4 and 5. Each is a finding. +- **A passage found confusing by two or more (none outstanding)**: not met. Five passages are + named by two or more readers, as the table above shows. +- **From one reader only**: the 6.1 paragraph on a project's own rule, the section 1 sentence and + the section 2 table, recorded. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in 87536c5, 66301b9 and 3b848c7 +(key quotations kept verbatim in 85110c1 and ba8a0bd), after an independent review of each round of +edits. DETECTOR-SPEC.md: 5.3 says what the command's own output must carry, and that where its +summary leaves a finding out it must say where those findings can be read (an obligation change, +recorded in the changelog, so the text says what the key's intended Yes assumed); 6.1 asks for the +resolution mechanism, and missing bundled documentation fails 6.3, with 6.4's Why aligned; 7.1 says +the project writes the detecting rule in the detector itself and the detector need not remove or +forbid the route; 8.1 says what a gap is and that conformance is judged on evidence, with or without +a declaration. Separately, pull request 13 changed how question 2 is marked after this run, so the +misses that rested on judgements 1 to 4 needed no text change. A fresh cohort follows. + +## Process change + +Made under the process change of pull request 13 before it merged, at commit a695010, named +before the readers started; `ACCEPTANCE.md` and `acceptance/runs/README.md` in this worktree +differ from a695010 only by this pull request's question-set line adding question 2d for SPEC.md, +which does not bear on this document. + +## Verdict + +FAIL on every blocking criterion. Answer sheets, the readers' prompt and command, and the +marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090073628), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090074094), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090074397). From 72f1679351dbcf70a269c581806c7784b852031b Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:45:23 +0100 Subject: [PATCH 035/105] Method 2, 3.2, 3.3 and 4, detector 5.3, 6.1, 7.1 and 8: runs 21 and 22 findings applied SPEC.md: section 2 names the order of the steps; clause 3.2 states both grounds for a weaker kind in the sentence that allows one, and says the extension-points rule in short sentences; clause 3.3 Part B says an inline ignore is not judged under it, Part C says which run reports a fixture; section 4's lead-in names the Owner, and its wider-rule bullet, with the section 3 map's row, says plainly who decides. DETECTOR-SPEC.md: section 8 says conformance is judged on evidence and a declaration is optional; clause 8.1 says which declaration claims conformance; clause 7.1 opens by saying the route need not be removed or forbidden; clause 5.3 says the summary need not list every finding; clause 6.1's sentence on a path in the message is plain, and section 2's gloss defines Message. Text only; no question, threshold or fixture changed. Changelog lines extended, run NNN kept. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 20 +++++++++-- DETECTOR-SPEC.md | 44 ++++++++++++------------- SPEC.md | 86 +++++++++++++++++++++++++----------------------- 3 files changed, 84 insertions(+), 66 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7b5049b..4f6c998 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -46,7 +46,15 @@ version entry with no cohort line has not been accepted and is not published. of pointing at clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s vocabulary; clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class) says what the level of a class is; clause [3.3](SPEC.md#33-prove-the-net-by-making-the-rule-fire)'s Part B says an instance left behind an inline ignore comment, an ignore entry or a baseline is not a narrowing, since the code still carries the hazard, but a suppression for clauses [3.4](SPEC.md#34-sweep-the-codebase-then-fix-every-instance) and - [3.5](SPEC.md#35-enforce-permanently-and-block) and section [4](SPEC.md#4-authority-which-decisions-belong-to-whom); and clause [3.4](SPEC.md#34-sweep-the-codebase-then-fix-every-instance) says plainly that a baseline is rare. Accepted by run NNN. Minor version when released. + [3.5](SPEC.md#35-enforce-permanently-and-block) and section [4](SPEC.md#4-authority-which-decisions-belong-to-whom); and clause [3.4](SPEC.md#34-sweep-the-codebase-then-fix-every-instance) says plainly that a baseline is rare. After run 21, clarity + edits: section [2](SPEC.md#2-when-the-method-applies) names the order of the steps, since three readers stopped at its MUST + sentence; clause [3.2](SPEC.md#32-build-the-net) states both grounds for a weaker kind in the sentence that allows one, since + no reader named them, and says the extension-points rule in short sentences; clause [3.3](SPEC.md#33-prove-the-net-by-making-the-rule-fire)'s Part + B says an inline ignore is not judged under it, and Part C says the red run may report a + fixture and the green run does not; section [4](SPEC.md#4-authority-which-decisions-belong-to-whom)'s lead-in names the owner, and its bullet on the + next wider rule, with the section [3](SPEC.md#3-the-method) map's row, says the owner decides unless the practitioner + has written and confirmed why the hazard cannot arise there. Accepted by run NNN. Minor version + when released. ### 1.1.0, 2026-10-02 @@ -125,8 +133,14 @@ pass it. clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) says the project writes the detecting rule in the detector, of the kind clause [4.1](DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) requires, and that the detector need not remove or forbid the route; clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) says what a gap is, that a detector declaring a version records it there, and that conformance is - judged on evidence with or without a declaration. Accepted by run NNN. Minor version when - released. + judged on evidence with or without a declaration. After run 22, clarity edits: section [8](DETECTOR-SPEC.md#8-conformance) + says beside the definition of conformance that it is judged on evidence and a declaration is + optional, and clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) says plainly which declaration claims conformance; clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) opens by + saying the detector need not remove or forbid the route and says what detectable means in short + sentences; clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) says the summary need not list every finding; clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) says + plainly why a path in the message is not enough, and section [2](DETECTOR-SPEC.md#2-terminology)'s gloss defines the message. + Each after readers of run 22 missed or named the passage. Accepted by run NNN. Minor version + when released. ### 1.0.0, 2026-09-08 diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index c0f7036..ced457c 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -40,6 +40,7 @@ definition; the ones this document leans on most are, in short: | [Detector] | A tool that reads code without executing it and reports what it finds | | [Semantic], [Syntactic], [Textual] | What a [Rule] can ask in a configuration: the type of any expression and what any name refers to, program-wide; a parse tree short of that; or text | | [Rule] | One check a [Detector] evaluates | +| [Message] | The text a [Rule] prints when it fires | | [Identifier] | The stable name printed with a finding that leads to its documentation | | [Practitioner] | Whoever is doing the work, a person or an [Agent] | | [Toolchain] | Everything a project assembles to run its checks through, [Detectors] included | @@ -158,10 +159,10 @@ any. ### 5.3 The result MUST reach the practitioner in the output of the command they ran Where the [Detector] writes fuller detail elsewhere, the invoked command's own output MUST -still carry a usable summary, each finding in it with where it is and its [Identifier]. If the -summary leaves any finding out, the output MUST also say where the findings it leaves out can be read. Output that prints -every finding so meets this clause, even where extra detail, such as a code snippet, is written only -to a file the output does not name. +still carry a usable summary, each finding in it with where it is and its [Identifier]. The +summary need not list every finding. If it leaves any finding out, the output MUST also say where +the rest can be read. Output that lists every finding meets this clause, even where extra detail, +such as a code snippet, is written only to a file the output does not name. **Why**: method specification clause 8.2. A report the [Practitioner] has to go and find is a report that arrives after the decision it was meant to inform. @@ -195,9 +196,9 @@ documentation is the project's to write. Each case in turn: **For a [Bundled rule]**, the [Detector] supplies the documentation and the lookup, and clauses 6.2 and 6.3 say where. The mechanism MUST resolve an [Identifier] presented alone, because the reader who needs it most has the [Identifier] from a log, a -ticket or a colleague and not the [Message]. A documentation path printed inside the -[Message] does not meet this: it helps only a reader who has the [Message], and the -[Identifier] on its own still leads nowhere. +ticket or a colleague and not the [Message]. Printing a documentation path in the [Message] is +not enough. It helps only a reader who has the [Message]; the [Identifier] alone still leads +nowhere. **For a project's own [Rule]**, the [Detector] cannot know the documentation, so what it owes is the half it can give: the [Identifier] printed unaltered under clause 4.3, and no @@ -274,14 +275,15 @@ nothing to run. ### 7.1 A detector MAY offer an inline suppression route, but MUST make it detectable or disableable -An inline ignore comment, a per-line directive and a generated [Baseline] are all such -routes. The [Detector] MAY ship them. For each one it MUST do at least one of two things, and either -alone satisfies this clause: make the route disableable by configuration; or make the route -detectable. Detectable means the project can find every use of the route, either with a [Rule] -the project writes itself in the [Detector], of the kind clause 4.1 requires it to host, or with a -mechanical check the [Detector] documents. The [Detector] is not required to remove the route -or to forbid it. Either of the two things lets a project which decides to forbid the route -enforce that decision. A route that can be neither switched off nor seen does not [Conform]. +The [Detector] does not have to remove or forbid an inline [Suppression] route. An inline ignore +comment, a per-line directive and a generated [Baseline] are all such routes. The [Detector] MAY +ship them. For each one it MUST do at least one of two things, and either alone satisfies this +clause: make the route disableable by configuration; or make the route detectable. Detectable +means the project can find every use of the route. One way is a [Rule] the project writes and +registers with the [Detector], as clause 4.1 lets it; the [Detector] need not ship that [Rule]. +The other way is a mechanical check the [Detector] documents. Either of the two things lets a +project which decides to forbid the route enforce that decision. A route that can be neither +switched off nor seen does not [Conform]. **Why**: the method specification's position is that [Suppression] is an [Owner] decision under its clause 3.4 and section 4, and an [Owner] cannot decide something they are @@ -304,7 +306,8 @@ minute at the moment they have the reason in mind, and it is the only thing that ## 8. Conformance -**A [Detector] [Conforms]** if it satisfies every MUST in sections 4 to 7. +**A [Detector] [Conforms]** if it satisfies every MUST in sections 4 to 7. It is judged on the +evidence, and a declaration under clause 8.1 is optional. Partial [Conformance] MUST NOT be described as [Conformance]. A [Detector] that satisfies most of this document is in a normal and respectable condition; it is not @@ -333,14 +336,11 @@ record dependencies. The same declaration is where a known gap is recorded. A ga sections 4 to 7 that the [Detector] fails. Once its maintainer knows of a gap, from the [Detector]'s own checks or from a [Practitioner]'s report under the method's clause 3.2, a [Detector] that declares a version MUST record the gap alongside that version, in the same file -as the declaration or in a file the declaration names. A declaration with a non-empty -gap record is a statement of where the [Detector] stands and is not a claim of -[Conformance]. +as the declaration or in a file the declaration names. A declaration that records no gap claims +[Conformance]. A declaration that records a gap does not; it says where the [Detector] stands. The declaration is optional: a [Detector] with no declaration can still [Conform]. Whether a -[Detector] [Conforms] is judged on the evidence, with or without a declaration. The -declaration is for the maintainer: it is how they claim [Conformance] and record the gaps they -know of. A [Detector] that predates this document, or whose maintainer has never read +[Detector] [Conforms] is judged on the evidence, with or without a declaration. A [Detector] that predates this document, or whose maintainer has never read it, MAY be graded [Conforming] on evidence by anyone who exercises the clauses above against it, and a verdict on any [Detector], declared or not, rests on that exercise and not on the claim. A declaration tells the reader what the maintainer believes and what they know to be missing; diff --git a/SPEC.md b/SPEC.md index 22848af..b660e7e 100644 --- a/SPEC.md +++ b/SPEC.md @@ -182,7 +182,9 @@ Defence Before Fix applies to a [Defect] when that [Defect] can be attributed to idiom or configuration that a [Detector] can be made to recognise. When it can, building the [Defence] is not optional and MUST be attempted before the [Defect] is -fixed. This is the substance of the method rather than a preliminary to it. +fixed. The order is section 3's: attribute the [Defect] to a [Class], build the [Rule], prove +that it fires, run the [Sweep], then fix every [Instance]. This is the substance of the method +rather than a preliminary to it. When it cannot, the [Defect] is outside the method's scope and is fixed conventionally. Not every [Defect] is an [Instance] of a detectable pattern, and forcing a [Rule] where no pattern exists @@ -229,14 +231,14 @@ the [Defence] it implies is ever reached. Six clauses, in order. In brief, before the detail: -| Clause | The [Practitioner] MUST | The record shows | Goes to the [Owner] under section 4 | -| ------ | ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | -| 3.1 | Name the [Class] the [Defect] belongs to, bounded both ways, after an independent search | The [Class], the [Hazard] sentence, the two search techniques, the next wider [Rule] not built, whether a [Runner] check pins the reported behaviour | Whether the next wider [Rule] is built, and whether the [Rule] stays narrower than the search showed | -| 3.2 | Express the [Class] as a [Rule] in the strongest kind of [Detector] that can express it, never a test | The [Rule], its [Detector] and kind, why it is in a weaker kind than [Semantic] where it is, and any [Toolchain] gap that stopped a bespoke one | Whether a [Class] that could be defended is left undefended, and a weaker kind where the project requires the [Owner]'s sign-off | -| 3.3 | Make the [Rule] fire, on the [Instance] or a [Fixture], in a commit of its own | The red run, and the sentence behind every [Narrowing] | Any exclusion whose sentence cannot be written: that is [Suppression] | -| 3.4 | [Sweep] the whole codebase, record the count, then fix every [Instance] | The count, corroborated, and what was fixed by hand or by pattern | Any [Instance] left unfixed, and any [Baseline] | -| 3.5 | Make the [Rule] permanent and [Blocking] in the project's own checks | The green run through the project's entry point, and the recorded decision behind any [Suppression] | Removing or disabling the [Rule], and any [Suppression] added to it | -| 3.6 | Print a terse [Message] with a stable [Identifier] that resolves to documentation versioned with the [Rule] | The [Remediation docs] | Nothing | +| Clause | The [Practitioner] MUST | The record shows | Goes to the [Owner] under section 4 | +| ------ | ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | +| 3.1 | Name the [Class] the [Defect] belongs to, bounded both ways, after an independent search | The [Class], the [Hazard] sentence, the two search techniques, the next wider [Rule] not built, whether a [Runner] check pins the reported behaviour | Leaving the next wider [Rule] unbuilt, unless the [Hazard] cannot arise in the code it would add; leaving the [Rule] narrower than the search showed | +| 3.2 | Express the [Class] as a [Rule] in the strongest kind of [Detector] that can express it, never a test | The [Rule], its [Detector] and kind, why it is in a weaker kind than [Semantic] where it is, and any [Toolchain] gap that stopped a bespoke one | Whether a [Class] that could be defended is left undefended, and a weaker kind where the project requires the [Owner]'s sign-off | +| 3.3 | Make the [Rule] fire, on the [Instance] or a [Fixture], in a commit of its own | The red run, and the sentence behind every [Narrowing] | Any exclusion whose sentence cannot be written: that is [Suppression] | +| 3.4 | [Sweep] the whole codebase, record the count, then fix every [Instance] | The count, corroborated, and what was fixed by hand or by pattern | Any [Instance] left unfixed, and any [Baseline] | +| 3.5 | Make the [Rule] permanent and [Blocking] in the project's own checks | The green run through the project's entry point, and the recorded decision behind any [Suppression] | Removing or disabling the [Rule], and any [Suppression] added to it | +| 3.6 | Print a terse [Message] with a stable [Identifier] that resolves to documentation versioned with the [Rule] | The [Remediation docs] | Nothing | The fourth column is what an [Agent] or other [Practitioner] MUST NOT decide alone, clause by clause; section 4 states the full list and governs where the two differ. @@ -415,25 +417,25 @@ be written in the strongest kind of [Detector] that can express it: a [Semantic] [Syntactic] one, else a [Textual] one. The kind is that of the configuration the [Rule] runs in. Enabling an existing [Rule], or tightening its thresholds, is not writing one, and the off-the-shelf paragraph above covers it; giving an existing [Rule] a pattern of the project's own, -or changing the pattern of a [Rule] the project wrote, is writing one. A -weaker kind is allowed on one of two grounds only: - -1. the stronger kind cannot express the [Rule], as where the [Rule] reads a file the [Semantic] - [Detector] does not parse; or -2. the language has no [Detector] of the stronger kind that accepts bespoke [Rules]. - -For the second ground, the language has such a [Detector] when the -project already runs it or the language's own [Detector] ecosystem offers it, the same check -section 2 asks for before a [Toolchain] gap is recorded. A stronger configuration of a [Detector] -the project runs is one; a [Detector] or configuration of the stronger kind available only -through a hosted service or a licence tier is not. A [Detector] accepts bespoke [Rules] when the -project can add a [Rule] of its own to it through a documented extension point, such as a plugin -interface or an API for writing [Rules], whether or not it meets the rest of the [detector specification](DETECTOR-SPEC.md). -Where those extension points cannot reach the construct a [Rule] is about, that [Detector] -cannot express the [Rule]. Where no [Detector] of that kind can reach it, the kind cannot express -the [Rule], and the first ground holds. On the [Remediation docs] of a [Rule] they +or changing the pattern of a [Rule] the project wrote, is writing one. A weaker kind is allowed +on two grounds only: where the stronger kind cannot express the [Rule], or where the language has +no [Detector] of the stronger kind that accepts bespoke [Rules]. For example, the stronger kind +cannot express a [Rule] that reads a file the [Semantic] [Detector] does not parse. + +The language has a [Detector] of the stronger kind when the project already runs one, or the +language's own [Detector] ecosystem offers one. That is the same check section 2 asks for before +a [Toolchain] gap is recorded. A stronger configuration of a [Detector] the project runs counts. +A [Detector] or configuration that only a hosted service or a licence tier provides does not +count. A [Detector] accepts bespoke [Rules] when the project can add its own [Rule] to it through +a documented extension point, such as a plugin interface or an API for writing [Rules]. It need +not meet the rest of the [detector specification](DETECTOR-SPEC.md). If its extension points +cannot reach the construct a [Rule] is about, that [Detector] cannot express the [Rule]. If no +[Detector] of the stronger kind can reach it, the stronger kind cannot express the [Rule]: that +is the first ground, not the second. + +On the [Remediation docs] of a [Rule] they write, the [Practitioner] MUST name the [Detector] and its kind. Where that kind is weaker than [Semantic], the [Practitioner] MUST also -record, for each step down from [Semantic], which of the two grounds holds and why, naming the +record, for each step down from [Semantic], which of the two grounds above holds and why, naming the [Detectors] of the stronger kind that were considered, on the [Rule]'s [Remediation docs] and in the description of the commit that adds the [Rule], moves it to another [Detector], or first records the grounds. The [Practitioner] judges whether the grounds hold, as section 4 gives them the choice of [Detector]; a @@ -529,10 +531,10 @@ what must survive the merge is not. **Part B: [Narrowing].** -Leaving an [Instance] that carries the [Hazard] in place behind an inline ignore comment, an -ignore entry or a [Baseline] is not a [Narrowing], whatever note accompanies it. That is -a [Suppression] of the -[Instance], and clauses 3.4 and 3.5 and section 4 govern it, not this clause. +**An inline ignore comment, an ignore entry or a [Baseline] is not judged under this clause.** +Leaving an [Instance] that carries the [Hazard] in place behind one is not a [Narrowing], whatever +note accompanies it. It is a [Suppression] of the [Instance]. Clauses 3.4 and 3.5 and section 4 +govern it, and it does not fail this clause. **The decision comes first, and it turns on one sentence.** Can the [Practitioner] write down why the [Hazard] cannot arise in the code being excluded? If they can, the exclusion is a [Narrowing], @@ -622,9 +624,10 @@ Where a [Detector] makes [Fixture]-based testing impractical, an acceptance or s demonstrates the [Rule] firing is an acceptable substitute. [Fixtures] are part of the [Rule] and are versioned with it, under the co-location requirement in -clause 3.6. A [Fixture] is input to the [Rule]'s test, not part of the code the [Defence] enforces: -the [Sweep] excludes it under clause 3.4, and the enforced run does not report it, so there is no -circularity in a [Fixture] that deliberately fails the [Rule] it proves. Writing the [Fixture] is the [Practitioner]'s own decision under section 4, not something +clause 3.6. A [Fixture] is input to the [Rule]'s test, not part of the code the [Defence] enforces. +The red run of Part A may report it; that is the proof. The [Sweep] excludes it under clause 3.4, +and the green run under clause 3.5 does not report it. So there is no circularity in a [Fixture] +that deliberately fails the [Rule] it proves. Writing the [Fixture] is the [Practitioner]'s own decision under section 4, not something to refer upwards. Where they sit within the repository is the project's business. **Absent a convention, put them @@ -834,18 +837,19 @@ they do not. 3.4). - The wording of the failure [Message] and the [Remediation docs] (clause 3.6). -**Decisions that MUST go to whoever owns the codebase:** +**Decisions that MUST go to the [Owner]**, which the [Practitioner] never takes alone: - Adopting a [Baseline], in whole or in part (clause 3.4). - Suppressing an [Instance], or removing or disabling an existing [Rule] (clauses 3.4 and 3.5). - Accepting a known [Instance] as unfixed for any reason. - Deciding that a [Class] will not be defended at all, where a [Rule] for it is achievable. -- Leaving unbuilt the next wider [Rule] that clause 3.1 required the [Practitioner] to name, where - it was rejected because no [Instance] of it exists today rather than because the [Hazard] cannot - arise there. The absence of an [Instance] is the [Practitioner]'s reason not to widen unasked; it - is not authority to exclude the extension, and recording it as a known gap does not change whose - decision it is. This does not reopen a [Class] whose bounds a [Hazard] sentence has settled, and - it reaches no further than the one wider [Rule] already on the record. +- Leaving unbuilt the next wider [Rule] that clause 3.1 requires the [Practitioner] to name, + unless they have written why the [Hazard] cannot arise in the code it would add and confirmed it + by search, as clause 3.1 requires. That no [Instance] of it exists today is not enough. It is a + reason not to widen unasked, not authority to leave the wider [Rule] out, and recording it as a + known gap does not make it the [Practitioner]'s decision. This bullet covers only that one wider + [Rule], already named on the record. It does not reopen a [Class] whose bounds a [Hazard] + sentence has settled. - Leaving a [Rule] narrower than an independent search under clause 3.1 showed it should be, where the reason is that the wider check is harder to build without [False positives] rather than that the [Hazard] is absent from what it would add. The [Instances] the search found are From c5bfd10ab328e84e688ea42a2cef023eddf99e3c Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:45:47 +0100 Subject: [PATCH 036/105] Keys: method 2d and detector 2b quotations follow the reworded 3.2 and 7.1 Quotations only, verbatim to the text of the previous commit; no answer, element or condition changed. Co-Authored-By: Claude Opus 5.5 --- acceptance/detector/KEY.md | 6 +++--- acceptance/method/KEY.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 7a0cb42..423a7a9 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -40,9 +40,9 @@ agent summary (those belong to the toolchain specification). project can write or a mechanical check it documents; it is not required to remove or forbid it. Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each one it MUST do at least one of two things, and either alone satisfies this clause: make the route disableable by configuration; or make the route - detectable. Detectable means the project can find every use of the route, either with a Rule - the project writes itself in the Detector, of the kind clause [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) requires it to host, or with a - mechanical check the Detector documents." An answer that says the detector must forbid the route, or must require a + detectable. Detectable means the project can find every use of the route. One way is a Rule + the project writes and registers with the Detector, as clause [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) lets it; the Detector need + not ship that Rule. The other way is a mechanical check the Detector documents." An answer that says the detector must forbid the route, or must require a reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD), is wrong. - **2c.** With the rule, in the installed copy, at a version tracked together, and yes, offline. Key quotations, [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access): "Resolution of a bundled rule's identifier MUST work from the installed diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index f3b6211..db04c55 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -58,7 +58,7 @@ is correct; a bullet that says no weaker kind is ever allowed is wrong. Practitioner writes MUST be written in the strongest kind of Detector that can express it: a Semantic one, else a Syntactic one, else a Textual one."; "Where that kind is weaker than Semantic, the Practitioner MUST also record, for each step down from Semantic, which of the - two grounds holds and why, naming the Detectors of the stronger kind that were considered, on + two grounds above holds and why, naming the Detectors of the stronger kind that were considered, on the Rule's Remediation docs and in the description of the commit that adds the Rule, moves it to another Detector, or first records the grounds." Supporting: "On the Remediation docs of a Rule they write, the Practitioner MUST name the Detector and its kind." A correct answer gives three From 43923ca789b85c98e5ef31e94ef1972bcf460b2d Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:51:00 +0100 Subject: [PATCH 037/105] Method 3.3 Parts B and C: the review's two readings settled Part B's opening sentence covers only an ignore left over an instance that carries the hazard, so an ignore entry over hazard-free code stays a narrowing; Part C's green run does not report the match as an instance. Co-Authored-By: Claude Opus 5.5 --- SPEC.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/SPEC.md b/SPEC.md index b660e7e..9a2a579 100644 --- a/SPEC.md +++ b/SPEC.md @@ -531,7 +531,7 @@ what must survive the merge is not. **Part B: [Narrowing].** -**An inline ignore comment, an ignore entry or a [Baseline] is not judged under this clause.** +**An inline ignore comment, an ignore entry or a [Baseline] left over an [Instance] that carries the [Hazard] is not judged under this clause.** Leaving an [Instance] that carries the [Hazard] in place behind one is not a [Narrowing], whatever note accompanies it. It is a [Suppression] of the [Instance]. Clauses 3.4 and 3.5 and section 4 govern it, and it does not fail this clause. @@ -626,7 +626,7 @@ demonstrates the [Rule] firing is an acceptable substitute. [Fixtures] are part of the [Rule] and are versioned with it, under the co-location requirement in clause 3.6. A [Fixture] is input to the [Rule]'s test, not part of the code the [Defence] enforces. The red run of Part A may report it; that is the proof. The [Sweep] excludes it under clause 3.4, -and the green run under clause 3.5 does not report it. So there is no circularity in a [Fixture] +and the green run under clause 3.5 does not report it as an [Instance]. So there is no circularity in a [Fixture] that deliberately fails the [Rule] it proves. Writing the [Fixture] is the [Practitioner]'s own decision under section 4, not something to refer upwards. From 0b8d69901eed099361ebd0ff162358c6ac107f2a Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:51:00 +0100 Subject: [PATCH 038/105] Acceptance runs 21 and 22: SPEC.md and DETECTOR-SPEC.md, second cohort, both FAIL Five cold Haiku readers each, made under pull request 13 at 204f95b and marked by a delegated marker. Evidence is posted on pull request 14 and linked from each record, with the findings applied to the text. Co-Authored-By: Claude Opus 5.5 --- .../runs/021-semantic-detectors-run-3.md | 241 ++++++++++++++++++ .../runs/022-semantic-detectors-run-4.md | 217 ++++++++++++++++ 2 files changed, 458 insertions(+) create mode 100644 acceptance/runs/021-semantic-detectors-run-3.md create mode 100644 acceptance/runs/022-semantic-detectors-run-4.md diff --git a/acceptance/runs/021-semantic-detectors-run-3.md b/acceptance/runs/021-semantic-detectors-run-3.md new file mode 100644 index 0000000..7535f10 --- /dev/null +++ b/acceptance/runs/021-semantic-detectors-run-3.md @@ -0,0 +1,241 @@ +# Run 21: SPEC.md, semantic detectors, third run in pull request 14 + +- **Document**: SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + branch `editor/semantic-detectors` at 3aefdd1, the commit the readers were given; the question + set, key and fixture read at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input; no tools, no settings, no project context and no + saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is the question set, the document and the fixture at + 3aefdd1, in that order, each preceded by one `===== FILE: =====` line, with nothing + else added apart from one blank line between the end of each file and the next file's marker + line. +- **Managed (policy) settings**: none; the machine applies no managed settings to the readers and + no managed settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. Every sheet has answer text, no process reports an error + (`is_error` false, subtype `success`), and every reported model is the cohort's model. The + editor reports no reader set aside for a cause the sheets cannot show. +- **Run within the pull request**: 3, counting void runs; runs in the pull request so far: 3 (19, + 20 and this one). Run 22, on DETECTOR-SPEC.md, is made at the same time and does not bear on + this document. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review. It read `ACCEPTANCE.md` and + `acceptance/runs/README.md` in this worktree, which differ from commit 204f95b only by this pull + request's question-set line adding 2d for SPEC.md; the document, question set, key and fixture + at 3aefdd1; runs 19 and 20; and, for the fourth pass-criteria row, SPEC.md at 5b6093f, the + document run 19 read. + +## Marks + +`ok` is correct, `miss` is a miss; `T` marks a miss traceable to the text. Judgements J1 to J8 +are set out below. + +| Reader | Q1 | 2a | 2b | 2c | 2d | 3.1 | 3.2 | 3.3 | 3.4 | 3.5 | 3.6 | Remediation | Defence | Section 4 | Q4 passages | +| ------ | ------- | ------ | --- | --- | ------ | ------- | ------- | -------------- | --- | --- | --- | ----------- | ------- | --------- | ----------- | +| 1 | correct | miss T | ok | ok | miss T | Partial | Partial | Yes ok | No | No | No | ok | ok | ok | four | +| 2 | correct | ok | ok | ok | miss T | Partial | Partial | Partial miss T | No | No | No | ok | ok | ok | four | +| 3 | correct | miss T | ok | ok | miss T | Partial | Partial | Partial miss T | No | No | No | ok | ok | ok | two | +| 4 | correct | miss T | ok | ok | miss T | Partial | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 5 | correct | ok | ok | ok | miss T | Partial | Partial | Yes ok | No | No | No | ok | ok | ok | three | + +Every grade under 3.1, 3.2 and 3.4 to 3.6 is one the key accepts, and every 3.1 grade of Partial +notes that no independent search technique is recorded. + +### Question 1 + +Five of five correct. Every sheet has at least four bullets naming obligations the key lists, and +no bullet asserts an obligation the document does not state. Reader 3's "Write a custom rule" is +Appendix A's own wording ("Write a custom [Rule] that detects the [Class]"), and the "only then" +of readers 1 and 4, that the original defect is fixed last with a reproducing test, is Appendix +A's closing sentence. + +### Question 2 + +Every reader quoted words of the document for every item, so every miss here is traceable to the +text. + +- **2a**, two of five (readers 2 and 5). Every reader quotes the section 2 sentence "When it can, + building the [Defence] is not optional and MUST be attempted before the [Defect] is fixed.", which + states no order and no sweep, so each answer stands or falls on its sentence. + - Reader 1: "attributes it to a class, builds and proves a rule for that class, then sweeps and + counts before fixing (clauses 3.1 to 3.4). The original defect is fixed last"; the fix of + every instance is not stated, and naming clause 3.4 does not state it. + - Reader 2: class, "builds and proves the net", "sweeps and fixes every instance", in order; + correct (J2). + - Reader 3: "builds the defence before fixing the defect"; no class, no proof, no sweep, no fix + of every instance. + - Reader 4: "attributes the defect to a class and attempts the rule before fixing the defect, in + the order of section 3"; no proof, no sweep, no fix of every instance. + - Reader 5: class, rule, proof, "sweep and fix every instance", in order; correct. +- **2b**, five of five. Readers 1, 2 and 3 say the owner is a human; readers 4 and 5 say "the + owner of the codebase" and "the owner" (J4). All five quote the key's section 4 sentence and say + an agent may not. +- **2c**, five of five. Readers 1, 4 and 5 state the reviewer's rerun of the red and green runs, + reader 4 in its quotation of section 7's closing paragraph; readers 2 and 3 state reproduction, + not the report (J3). No reader accepts the pull request description as the verdict. +- **2d**, none of five. No answer, sentence and quotation read together, states the two grounds + on which a weaker kind is allowed: each says only that there are two. + - Reader 1: "A weaker kind is allowed only on one of two grounds"; its quotations give the order + and the record with both places, not the grounds. + - Reader 2: "A weaker kind is allowed only on one of two grounds"; no place for the record of + the grounds, only for naming the detector and kind (J5). + - Reader 3: "A weaker kind is allowed only on two grounds"; its second quotation stops before + the places (J5). + - Reader 4: "only on one of two grounds", quoting "A weaker kind is allowed on one of two + grounds only:"; nor does its answer state the order semantic, syntactic, textual, and it names + no place (J5). + - Reader 5: "only on one of two grounds"; no place (J5). + +### Question 3 + +- **3.3**, three of five (readers 1, 4 and 5, Yes). Reader 1's caveat, that the description does + not show how the fixture was excluded from the enforced run, leaves its grade at Yes. + - Reader 2 grades Partial because "Part B is breached: the `Report::render()` inline ignore is + a suppression, not a narrowing", and also grades 3.4 No on it. Its reasoning states what the + document requires of Part B, so the miss is traceable. Part B now opens by saying such an + ignore is a [Suppression] "and clauses 3.4 and 3.5 and section 4 govern it, not this clause"; + the key's excuse does not cover a reader who attributes it to both (J6). + - Reader 3 grades Partial because "the enforced run reports the fixture as one of its five + findings, and Part C says the enforced run 'does not report it'". It quotes clause 3.3's + sentence on a [Rule]'s [Fixtures], so the miss is traceable. +- **Remediation**: five of five say it does not conform, each naming at least one failing clause + the key accepts. +- **Defence**: five of five say it does not conform, each naming 3.6. +- **Section 4**: five of five name the inline ignore on `Report::render()`. Readers 1, 2, 4 and 5 + also raise the unbuilt wider rule as a second or less clear decision, which the key accepts + alongside it. + +### Judgements made in marking + +Where the rules leave a judgement open, the stricter reading, the one that records more misses or +more passages, is taken. Where a rule settles the point, the record says which. + +- **J1. The sentence.** Where a reader wrote more than one sentence before its quotation, all of + that text is read with the quotation as the answer. No mark turns on it: in each such case the + quotation states what the later sentences add, or the answer misses an element no sentence + states. +- **J2. "The net" (reader 2, 2a).** Settled by the rule that an element may be stated in the + document's words: "Build the net" and "Prove the net by making the rule fire" are the headings + of clauses 3.2 and 3.3, and the net is the [Rule] in a [Detector]. Read otherwise, reader 2 + would add one traceable miss on 2a; the verdict would not change. +- **J3. "Reproduction" (readers 2 and 3, 2c).** Settled by the rule that each element is read + for its gist, with sentence and quotation together: both quotations are section 7's sentence + that the verdict "rests on reproduction, as this section closes by requiring, not on the + report", and the closing paragraph it points to is the red and green reruns. The key's own + condition for a wrong answer is one that omits reproduction. +- **J4. "The owner" (readers 4 and 5, 2b).** Settled by the rule that a defined term used in its + defined sense states what it means: [Owner] is "Always a human", and section 4 and the key + both call it the owner of the codebase. Reader 4's question 4 doubt whether "whoever owns the + codebase" is the defined term is not read into its 2b answer, which is its sentence and + quotation; that doubt is recorded as a passage below. +- **J5. 2d's element (iii) needs a place named.** The key says naming one of the two places "is + enough", read as requiring at least one, as run 19 read it. No mark turns on it: every sheet + already misses (ii). +- **J6. The 3.3 excuse is read as written**, covering a reader who attributes the suppression to + 3.3 rather than 3.4, not one who attributes it to both, as run 19 read it. +- **J7. Reader 1's "its owner list".** Reader 1's first passage quotes table row 3.1 and section + 4's first practitioner bullet, and describes section 4's owner list as covering "only the wider + rule left unbuilt because no instance exists". The description is read as naming the whole + owner list, its lead-in included, so it covers the wider-rule bullet and the line "Decisions + that MUST go to whoever owns the codebase:". The narrower reading, the wider-rule bullet alone, + would leave the lead-in to reader 4 alone and remove one finding below. Reader 1's other + descriptions are read the same way: "the two grounds" covers both items of clause 3.2's list, + and "Clause 3.4, first paragraph against the Fixture paragraph" covers 3.4's opening paragraph + as well as the scope paragraph it quotes. Reader 3's "on the second ground" covers the second + item of that list. +- **J8. Clause labels.** A clause number given with a quotation, as question 4 asks, is read as + the quotation's label, not as naming the whole clause. Settled by the rule on quoting: naming a + clause number is not quoting, and question 4 asks for a quotation "with its clause number". + +## Passages quoted as confusing + +Same passage as `ACCEPTANCE.md` defines it: sentences in common, or the same paragraph; a list +item or table row is its own paragraph. + +| Passage | Clause | Readers | Words quoted, verbatim, or the description used | +| --------------------------------------------------------------------- | ------------------------ | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| The two grounds and the paragraph after them, "For the second ground" | 3.2 | four: 1, 2, 3, 5 | 1: "Where no [Detector] of that kind can reach it, the kind cannot express the [Rule], and the first ground holds." and "the two grounds" (J7). 2: "naming the [Detectors] of the stronger kind that were considered, on the [Rule]'s [Remediation docs] and in the description of the commit that adds the [Rule], moves it to another [Detector], or first records the grounds." 3: "The language has such a [Detector] when the project already runs it or the language's own [Detector] ecosystem offers it, the same check section 2 asks for before a [Toolchain] gap is recorded." and "the second ground" (J7). 5: "Where those extension points cannot reach the construct a [Rule] is about, that [Detector] cannot express the [Rule]."; "the stronger kind cannot express the [Rule]" (the first ground's list item); "A stronger configuration of a [Detector] the project runs is one; a [Detector] or configuration of the stronger kind available only through a hosted service or a licence tier is not." Every sentence quoted lies in the one paragraph that begins "For the second ground"; the first list item is named by readers 1 and 5, the second by readers 1 and 3 | +| The bullet on leaving the next wider rule unbuilt | 4 | three: 1, 2, 5 | 2: "Leaving unbuilt the next wider [Rule] that clause 3.1 required the [Practitioner] to name, where it was rejected because no [Instance] of it exists today rather than because the [Hazard] cannot arise there." 5: "it reaches no further than the one wider [Rule] already on the record." 1: by description, "its owner list covers only the wider rule left unbuilt because no instance exists" (J7) | +| Table row 3.1, the fourth column | 3 (given as 3.1 by both) | two: 1, 2 | 1: "Whether the next wider [Rule] is built". 2: "Whether the next wider [Rule] is built, and whether the [Rule] stays narrower than the search showed" | +| The lead-in to the owner's decisions | 4 | two: 1, 4 | 4: "**Decisions that MUST go to whoever owns the codebase:**", set beside the Owner entry (below). 1: by description, "its owner list" (J7) | +| The Owner entry | 1, Owner | one: 4 | "Whoever holds authority over a project's [Exceptions] and [Calibrations]. Always a human." (quoted after "**Owner**: ") | +| The practitioner's first decision | 4 | one: 1 | "What [Class] the [Defect] belongs to, and where its bounds sit (clause 3.1)" | +| The lower bound, the exact value or name | 3.1 | one: 1 | "A [Rule] drawn to the exact value or name found in the [Defect] is the same [Narrowing] however it is spelt." | +| The clean run of the rule and saturation | 3.1 | one: 3 | "A clean run of the [Rule] is therefore not evidence of saturation either: "the [Rule] found nothing the reading had missed" is one technique and a [Rule] run, not two techniques, and the stopping criterion cannot be applied to it." | +| What is in scope of the sweep, and a rule's own fixtures | 3.4 | one: 1 | "Whether vendored dependencies, generated code and test [Fixtures] are in scope is the project's decision" and "A [Rule]'s own [Fixtures] under clause 3.3 are never [Instances]."; by description, also 3.4's first paragraph (J7) | +| The three ways out of the method | 2 | one: 4 | "There are three ways out of the method and they are recorded differently." and "a mechanism gap under clause 3.2, and the [Rule] is still built" | +| What a conforming remediation is | 7 | one: 2 | "A remediation [Conforms] if all six clauses of section 3 were followed for that [Defect], and the verdict that they were rests on reproduction, as this section closes by requiring, not on the report." | + +Readers' accounts of what they could not tell are recorded with the passage they attach to and +are not suggestions; none of the sheets proposes a change. + +**Earlier passages (the fourth row).** Run 19 found four passages named by two or more readers. +As the paragraphs read in the changed document: + +- Clause 3.2's extension-points sentence lay in the paragraph that now begins "For the second + ground". Readers 1, 2, 3 and 5 of this cohort name sentences of that paragraph, so that + finding is not cleared. +- Clause 3.1's level paragraph, now "The bounds say how broad the [Class] is", is named by no + reader. Cleared. +- Section 2's "Attempt rather than pre-judge" paragraph is named by no reader; reader 4's section + 2 passage lies in the next paragraph, "There are three ways out". Cleared. +- Clause 3.4's "Fix them all" paragraph is named by no reader; reader 1's clause 3.4 passage lies + in the scope paragraph and, by description, the opening paragraph. Cleared. + +No sentence kept from one of those paragraphs is named elsewhere in the document. + +**Editor's note on the marker's instruction.** The instruction for this run asked for the stricter +reading wherever the rules leave a judgement open, which the process at 204f95b had replaced with +the judgement of a careful reader. The verdict does not turn on it: the findings that fail the run +rest on rules, not on those judgements. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in 72f1679 and the commit after +it that settles a review of those edits (key quotations kept verbatim in c5bfd10). SPEC.md: clause +3.2 states both grounds for a weaker kind in the one sentence that allows it, and its paragraph on +the second ground is short and plain; section 2 names the order (attribute, build, prove, sweep, +then fix every instance) where it says the defence comes before the fix; clause 3.3 Part B says an +ignore left over an instance that carries the hazard is not this clause's matter, and Part C says +which run it means; section 4's bullet on the next wider rule, the map's row 3.1 and section 4's +lead-in say plainly what the decision is and who takes it. A fresh cohort follows. + +## Process change + +Made under a process change before it merged: yes, pull request 13 at commit 204f95b, named +before the readers started. + +## Verdict + +**FAIL.** Against the criteria in `ACCEPTANCE.md`: + +- Load-bearing statements: not met. 2a two of five (readers 1, 3 and 4 miss), 2b five of five, + 2c five of five, 2d none of five: no reader states the two grounds for a weaker kind. +- Restatement and application: not met on 3.3, three of five (readers 2 and 3 miss); question 1 + and every other keyed item five of five. +- Misses traceable to the text, each a finding: 2a for readers 1, 3 and 4, each relying on the + section 2 sentence "When it can, building the [Defence] is not optional and MUST be attempted + before the [Defect] is fixed."; 2d for all five readers, on clause 3.2's grounds for a weaker + kind; 3.3 for reader 2, on Part B and the inline ignore; 3.3 for reader 3, on Part C's + "the enforced run does not report it". +- Passages found confusing by two or more, each a finding that blocks until the text is changed + and a fresh cohort no longer names it: clause 3.2's two grounds and the "For the second ground" + paragraph (four readers), which is also run 19's extension-points finding, not cleared; + section 4's bullet on leaving the next wider rule unbuilt (three); the fourth column of table + row 3.1 in section 3 (two); section 4's lead-in "Decisions that MUST go to whoever owns the + codebase:" (two, on J7). +- Single-reader notes, recorded: the Owner entry (reader 4); section 4's first practitioner + bullet (reader 1); clause 3.1's lower-bound sentence on the exact value or name (reader 1); + clause 3.1's clean-run sentence (reader 3); clause 3.4's scope paragraph and opening paragraph + (reader 1); section 2's three ways out (reader 4); section 7's conforming remediation (reader + 2). + +Answer sheets, the reader prompt, the command and the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090464782), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090465417), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090465825), [comment 4](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090466147). diff --git a/acceptance/runs/022-semantic-detectors-run-4.md b/acceptance/runs/022-semantic-detectors-run-4.md new file mode 100644 index 0000000..e8262c1 --- /dev/null +++ b/acceptance/runs/022-semantic-detectors-run-4.md @@ -0,0 +1,217 @@ +# Run 22: DETECTOR-SPEC.md, semantic detectors, fourth run in pull request 14 + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + editor's branch `editor/semantic-detectors` at 3aefdd1; the question set, key and fixture read + at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input, with no tools, no settings, no project context + and no saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is the question set, the document and the fixture at + 3aefdd1, in that order, each preceded by one `===== FILE: =====` line, with the files + separated by one blank line and nothing else added (judgement 11) +- **Managed (policy) settings**: none; the machine applies no managed settings and no managed + settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. The editor reports none set aside for a cause the sheets cannot + show; every sheet has answer text, no process reports an error (`is_error` false, subtype + `success`), and every reported model is the cohort's model +- **Run within the pull request**: 4, counting void runs; run 21, on SPEC.md, was made at the same + time and does not bear on this document +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review + +## Marks + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | Section 8 | Kind | 6.4 | Q4 passages | +| ------ | ------- | ---- | ---- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --------- | ---------- | --- | ----------- | +| 1 | miss | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 3 | +| 2 | correct | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 3 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 3 | +| 4 | miss | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 5 | miss | miss | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 3 | + +Every miss under question 2 is traceable to the text: each answer quotes the document in support. +The three misses under question 1 are the readers' own: the bullet each turns on quotes nothing. +No question was left unanswered. + +### Question 1 + +Every restatement names four or more obligations from the key's list. Readers 2 and 3 assert no +obligation the document does not state: correct. Readers 1, 4 and 5 each have a bullet that +requires every finding in the command's own output: reader 1, "the command's own output carries +every finding"; reader 4, "put every finding in the command's own output"; reader 5, "put each +finding and its identifier in the command's own output". Clause 5.3 requires a usable summary +and, where the summary leaves a finding out, a statement of where the rest can be read; printing +every finding is one way to meet it, not the obligation. Each is marked a bullet asserting an +obligation the document does not state, so incorrect (judgement 2). Correct, two of five. + +### Question 2 + +The key's answers state these elements: + +- 2a: (a) Conformance is satisfying every MUST in sections 4 to 7; (b) judged on evidence; (c) a + detector that predates the document can conform; (d) the declaration is optional. +- 2b: (a) make the route disableable by configuration, or (b) detectable by a Rule the project can + write or a mechanical check the Detector documents; (c) the detector is not required to remove or + forbid it. +- 2c: (a) with the rule; (b) in the installed copy; (c) at a version tracked together; (d) yes, + reachable offline. + +Each answer is read as its sentence and its quotation together, for the gist of each element. + +| Reader | 2a | 2b | 2c | +| ------ | ------------------------------------------------------------------------------------------- | ---------------------------------- | ------------------------------------------------------------------- | +| 1 | miss: (d) not stated (judgement 3) | miss: (c) not stated (judgement 4) | ok: all four, (c) from the 6.3 quotation | +| 2 | miss: (d) not stated (judgement 3); (a) as "the evidence against the MUSTs", not every MUST | miss: (c) not stated (judgement 4) | ok: all four, (c) from the 6.3 quotation | +| 3 | ok: all four; (d) as "rests on evidence rather than on the detector's claim" | miss: (c) not stated (judgement 4) | ok: all four, from the sentence and the 6.2 and 6.3 headings quoted | +| 4 | ok: all four; (d) as "judged on evidence rather than on any claim it makes" | miss: (c) not stated (judgement 4) | ok: all four | +| 5 | miss: (d) not stated (judgement 3) | miss: (c) not stated (judgement 4) | miss: (a) and (c) not stated; only 6.2 quoted (judgement 6) | + +No answer asserts anything the document does not state or that the key says makes an answer +wrong: none says a declaration is required, that partial satisfaction is conformance, that the +detector must forbid the route or require a reason, or that a website will do. + +### Question 3 + +The key's intended verdicts are Yes for every clause but 6.3, which is No; the section 8 verdict +is "does not conform", because 6.3 fails; 6.4 does not change the verdict. + +- **Every clause**: all five readers give the intended verdict on every clause, 4.1 to 7.1. On + 6.3 each cites that one unresolved identifier fails the clause; no reader grades it Partial on + the grounds that thirty-eight resolve. On 5.3 and 6.1, the clauses run 20 failed on, all five + grade Yes. +- **Section 8**: all five say Lintomatic does not conform, because 6.3 fails. +- **Kind**: all five say the fixture does not let them name the kind and name none, so none is + marked on it. Reader 1 calls its verdict "incomplete on that point" but does not make the + verdict turn on the kind; readers 3 and 5 say outright that it does not change the verdict. +- **6.4**: all five say it does not change the verdict, because it is a SHOULD. Reader 1 adds that + whether 6.4 applies as a MUST depends on whether Lintomatic ships inside a Toolchain, which does + not alter its answer. + +### Judgements made in marking + +01. **How an open judgement is made.** `ACCEPTANCE.md` at 204f95b says an open judgement is made + as a careful reader of the document would; the editor's instruction asks for the stricter + reading wherever the rules leave a judgement open. Each judgement below is made as a careful + reader would, and where a careful reader could take either reading, the stricter one, the one + recording more misses or findings, is taken and the judgement says so. +02. **Question 1, "every finding" in the output, readers 1, 4 and 5.** The heading of 5.3 speaks + of "the result", which a reader may take as every finding, but its body requires only a + summary and a pointer to what the summary leaves out. A careful reader could read these bullets + as a compression of 5.3 or as an obligation it does not state; the stricter reading is taken, + and each is marked incorrect. +03. **2a (d), the declaration optional.** The question's second part asks whether a detector that + has never heard of the document can conform; the key answers it with both (c) and (d), so (d) + is an element that answers a part of the question. Its gist is that conformance does not depend + on a declaration or claim. Readers 3 and 4 say the grade rests on evidence rather than on the + detector's claim, which the document calls the declaration's purpose ("it is how they claim + [Conformance]"), and so state it. Readers 2 and 5 say only that the detector is judged on + evidence; reader 1 says it is judged "not on whether its maintainer has heard of the document". + Each of the three, with the 8.1 quotation all five give, implies that no declaration is needed + without saying so; a careful reader could go either way, and the stricter reading is taken. +04. **2b (c), not required to remove or forbid.** The question asks what the detector must do + about the route, and the key's answer includes what it need not do, so (c) answers the + question. All five say either of the two things alone is enough. That entails that nothing + further is required but does not say that the route may stay; a careful reader could go either + way, and the stricter reading is taken. Marked not stated, as run 20 marked it. +05. **2b (b), the means of detection.** Clause 7.1 defines "Detectable" in its own text, and each + reader uses "detectable", or reader 4 "findable", in that sense, so (b) is stated: a defined + term used in its defined sense states what it means. It decides no mark, since every 2b answer + misses (c). +06. **2c, reader 5.** Its sentence says the documentation "lives in the installed package" and its + only quotation is 6.2. Neither says the documentation ships with the rule nor that it is + versioned with it. Element (c) answers where the documentation must live, as part of the key's + answer to that part; not stated. Whether "in the installed package" carries (a) is open, and + the stricter reading is taken. +07. **Question 4, a clause number inside a quotation.** Readers 1 to 4 quote the 7.1 sentence that + contains "of the kind clause 4.1 requires it to host". The clause number is part of the quoted + run, so it covers the 7.1 sentence; it is not a passage named by clause number and does not + cover clause 4.1. +08. **Question 4, reader 4's "in the [Detector]".** Taken as one run, these words occur in the 7.1 + sentence and, on the stricter reading, in the 6.3 Why sentence "If its documentation lives only + in the [Detector]'s repository or on its website, ...", where the following word is + "[Detector]'s". The quotation is taken to cover both. The second blocks nothing. +09. **Question 4, a passage quoted only to set another against it.** Reader 1 quotes the 8.1 + sentence "A declaration with a non-empty gap record ..." as what it read the other 8.1 sentence + against. A quotation under question 4 counts whatever it is quoted for, so it is recorded. +10. **The fourth row's "earlier passage".** Read as a passage named by two or more readers in an + earlier run of this document in this pull request, which is run 20 at 5b6093f. Run 20's + single-reader passages, the 6.1 paragraph on a project's own rule, the section 1 sentence and + the section 2 table, are covered by no passage this cohort names, so the stricter reading would + change nothing. +11. **The packet's blank lines.** The prompt joins the three files with one blank line between + each file's end and the next file's header. A blank line adds nothing for a reader to read, + every reader received the same packet, and a packet's form is not a cause in `ACCEPTANCE.md` + for setting a reader aside, so no reader is set aside for it. + +No re-mark. + +## Passages quoted as confusing + +Grouped by paragraph, as `ACCEPTANCE.md` defines the same passage. A passage blocks when two or +more readers name it, or when it covers a sentence of a paragraph in which one of run 20's +blocking passages lay, as that paragraph reads at 3aefdd1. + +| Clause and paragraph | Readers | Words quoted, verbatim | Blocks | +| ------------------------------------------------------------------------------------------ | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | +| 7.1, first paragraph, the sentence beginning "Detectable means" | 1, 2, 3, 4 | 1, 2, 4: "Detectable means the project can find every use of the route, either with a [Rule] the project writes itself in the [Detector], of the kind clause 4.1 requires it to host, or with a mechanical check the [Detector] documents." 3: "either with a [Rule] the project writes itself in the [Detector], of the kind clause 4.1 requires it to host, or with a mechanical check the [Detector] documents." 4 also: "in the [Detector]" (judgement 8) | Yes: four readers, and run 20's 7.1 paragraph | +| 8.1, first paragraph, its last sentence | 1 | "A declaration with a non-empty gap record is a statement of where the [Detector] stands and is not a claim of [Conformance]." (judgement 9) | Yes: run 20's 8.1 first paragraph | +| 8.1, second paragraph, the sentence beginning "The declaration is for" | 1 | "The declaration is for the maintainer: it is how they claim [Conformance] and record the gaps they know of." | Yes: run 20's 8.1 second paragraph | +| 6.1, the paragraph beginning "For a [Bundled rule]" | 3 | "A documentation path printed inside the [Message] does not meet this." | Yes: run 20's 6.1 paragraph on a bundled rule | +| 6.1, the paragraph beginning "In both", the sentence beginning "Where the mechanism works" | 5 | "Where the mechanism works but a [Bundled rule]'s [Identifier] has no documentation to resolve to, that fails clause 6.3, not this clause." | No | +| 5.3, its paragraph, the sentence beginning "Output that prints" | 3 | "Output that prints every finding so meets this clause." | No | +| 6.4, the Why paragraph | 5 | "the [toolchain specification]'s self-audit makes this check a MUST for the [Toolchain]." | No | +| 8, the paragraph beginning "A verdict on a [Detector] MUST also name" | 5 | "the kind of each configuration it grades"; "the one the [Detector] has without any hosted service or licence tier" | No | +| 6.3, the Why paragraph, the sentence beginning "If its documentation lives" | 4 | "in the [Detector]" (judgement 8) | No | + +No reader names clause 4.1, the fifth of run 20's blocking passages (judgement 7). + +## Against the criteria + +- **Load-bearing statements (every reader on every one)**: not met. 2a three of five correct + (readers 3 and 4), 2b none of five, 2c four of five (all but reader 5). +- **Restatement and application (all but one on each keyed item)**: not met. Question 1 two of + five; every clause of question 3, the section 8 verdict and 6.4 five of five. +- **A miss traceable to the text (none)**: not met. Nine, each a finding: 2a by readers 1, 2 and + 5; 2b by all five; 2c by reader 5. +- **A passage found confusing by two or more (none outstanding)**: not met. The 7.1 sentence + defining "Detectable" is named by four readers; and run 20's blocking passages in 7.1's first + paragraph, 8.1's first and second paragraphs and 6.1's paragraph on a bundled rule are still + outstanding, since this cohort names passages that cover sentences of each of those paragraphs. +- **From one reader only**: the 6.1 sentence on a mechanism that works without documentation, the + 5.3 sentence on output that prints every finding, the 6.4 Why sentence, the section 8 paragraph + on kind and the 6.3 Why sentence, recorded. + +**Editor's note on the marker's instruction.** The instruction for this run asked for the stricter +reading wherever the rules leave a judgement open, which the process at 204f95b had replaced with +the judgement of a careful reader. The verdict does not turn on it: the findings that fail the run +rest on rules, not on those judgements. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in 72f1679 and the commit after +it that settles a review of those edits (key quotations kept verbatim in c5bfd10). DETECTOR-SPEC.md: +7.1 says in its first sentence that the detector need not remove or forbid the route, and defines +detectable in short sentences; 8.1's two paragraphs and 6.1's sentence on a documentation path +inside a Message are shorter and plainer; section 8 says conformance is judged on the evidence and a +declaration is optional; 5.3 says the output need not print every finding if it says where the rest +can be read. A fresh cohort follows. + +## Process change + +Made under the process change of pull request 13 before it merged, at commit 204f95b, named +before the readers started; `ACCEPTANCE.md` and `acceptance/runs/README.md` in this worktree +differ from 204f95b only by this pull request's question-set line adding question 2d for SPEC.md, +which does not bear on this document. + +## Verdict + +FAIL on every blocking criterion. Answer sheets, the readers' prompt and command, and the +marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090466450), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090466829), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090467103). From 598908ba9e02b83683c622baa1d9a84e9107f027 Mon Sep 17 00:00:00 2001 From: joseph Date: Fri, 9 Oct 2026 23:57:42 +0100 Subject: [PATCH 039/105] Acceptance process: a short quotation under question 4 covers the place the reader names Run 24 showed one-word quotations ("MAY", "configuration") covering every paragraph where the word occurs. A quotation now covers the sentences it overlaps at the clause or place the reader names it by, and every occurrence only where the reader names no place. On next the same passage was undefined and run 16 read it narrowly, so this relaxes nothing there. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 8c396dd..43adb33 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -145,8 +145,9 @@ which rule and why. Each sheet is marked question by question, as follows. one whose answer text is empty, or whose process reports an error, is set aside, as [The cohort](#the-cohort) says, and is still marked. - **The same passage.** A quotation under question 4, of any length, covers every sentence that - an occurrence of the quoted words, taken as one run, overlaps, at every place that run occurs, - and a passage named without a quotation, by clause number or description, covers every + an occurrence of the quoted words, taken as one run, overlaps: at the place the reader names + it, by clause number or description, and where the reader names no place, at every place that + run occurs; a passage named without a quotation, by clause number or description, covers every sentence of what it names. A heading, list item or table row counts as one sentence. Two passages are the same when they cover a sentence in common or lie in the same paragraph, and a list item or table row is its own paragraph. From b172fd60c74be26b9e447829d3200022a308a25d Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 00:01:56 +0100 Subject: [PATCH 040/105] Method 1, 3.1 and 3.2, detector 7.1, 8 and 8.1: runs 23 and 24 findings applied Clarity edits, no obligation changed in meaning: - Method 3.2: the record sentence names the order of kinds, both grounds and the two places, places first; the second ground is asked as two questions of each stronger detector. - Method 3.1: the lower bound says an exact-value rule matches one thing and states the test of the wider rule's rejection in short sentences, without calling it a narrowing. - Method 1: Narrowing and Suppression entries say an exclusion of code that carries the hazard is a suppression, not a narrowing; the Owner entry says the default owner is the human who instructed the practitioner, or the human who instructed the instructing agent, and takes section 4's decisions. - Detector 7.1: either thing alone satisfies the clause, and neither requires the detector to remove or forbid the route; shorter sentences. - Detector 8.1: the MAY sentence says a declaration is optional; the paragraph in short sentences. - Detector 8: one detector can be graded in more than one configuration, and one needing a hosted service or licence tier does not count. Key quotations follow in a separate commit. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 23 +++++++++++++++++++---- DETECTOR-SPEC.md | 34 ++++++++++++++++++---------------- SPEC.md | 46 +++++++++++++++++++++++++--------------------- 3 files changed, 62 insertions(+), 41 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4f6c998..24410f4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -53,8 +53,17 @@ version entry with no cohort line has not been accepted and is not published. B says an inline ignore is not judged under it, and Part C says the red run may report a fixture and the green run does not; section [4](SPEC.md#4-authority-which-decisions-belong-to-whom)'s lead-in names the owner, and its bullet on the next wider rule, with the section [3](SPEC.md#3-the-method) map's row, says the owner decides unless the practitioner - has written and confirmed why the hazard cannot arise there. Accepted by run NNN. Minor version - when released. + has written and confirmed why the hazard cannot arise there. After run 23, clarity edits: clause + [3.2](SPEC.md#32-build-the-net)'s record sentence names the order of kinds, both grounds and the two places first, since + no reader stated all three; clause [3.2](SPEC.md#32-build-the-net) asks the second ground as two questions of each + stronger detector; clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s lower bound says an exact-value rule matches one thing and + states the test of the wider rule's rejection in short sentences, without calling it a + narrowing; section [1](SPEC.md#1-terminology)'s Narrowing and Suppression entries say an exclusion of code that carries + the hazard is a suppression, not a narrowing; and the Owner entry says the default owner is the + human who instructed the practitioner, or, where an agent gave the instruction, the human who + instructed that agent, and takes section [4](SPEC.md#4-authority-which-decisions-belong-to-whom)'s + decisions. Each after readers of run 23 missed or named the passage. Accepted by run NNN. Minor + version when released. ### 1.1.0, 2026-10-02 @@ -139,8 +148,14 @@ pass it. saying the detector need not remove or forbid the route and says what detectable means in short sentences; clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) says the summary need not list every finding; clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) says plainly why a path in the message is not enough, and section [2](DETECTOR-SPEC.md#2-terminology)'s gloss defines the message. - Each after readers of run 22 missed or named the passage. Accepted by run NNN. Minor version - when released. + Each after readers of run 22 missed or named the passage. After run 24, clarity edits: clause + [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)'s sentence that either thing alone satisfies the clause adds that neither requires the + detector to remove or forbid the route, and the clause is in shorter sentences; clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps)'s + sentence that a detector may be graded conforming on evidence says a declaration is optional, + and its paragraph is in short sentences; section [8](DETECTOR-SPEC.md#8-conformance)'s paragraph on the kind says one + detector can be graded in more than one configuration and that one needing a hosted service or + licence tier does not count. Each after readers of run 24 missed or named the passage. Accepted + by run NNN. Minor version when released. ### 1.0.0, 2026-09-08 diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index ced457c..2139361 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -276,14 +276,15 @@ nothing to run. ### 7.1 A detector MAY offer an inline suppression route, but MUST make it detectable or disableable The [Detector] does not have to remove or forbid an inline [Suppression] route. An inline ignore -comment, a per-line directive and a generated [Baseline] are all such routes. The [Detector] MAY -ship them. For each one it MUST do at least one of two things, and either alone satisfies this -clause: make the route disableable by configuration; or make the route detectable. Detectable -means the project can find every use of the route. One way is a [Rule] the project writes and -registers with the [Detector], as clause 4.1 lets it; the [Detector] need not ship that [Rule]. -The other way is a mechanical check the [Detector] documents. Either of the two things lets a -project which decides to forbid the route enforce that decision. A route that can be neither -switched off nor seen does not [Conform]. +comment, a per-line directive and a generated [Baseline] are all such routes, and the [Detector] +MAY ship them. For each route it ships, it MUST do at least one of two things: let the project +switch the route off in the [Detector]'s configuration, or make the route detectable. Either alone +satisfies this clause, and neither requires the [Detector] to remove or forbid the route. +Detectable means the project can find every use of the route. There are two ways. One is a +[Rule] the project writes and registers with the [Detector], as clause 4.1 lets it; the +[Detector] need not ship that [Rule]. The other is a mechanical check the [Detector] documents. +Either of the two things lets a project that decides to forbid the route enforce that decision. +A route that can be neither switched off nor seen does not [Conform]. **Why**: the method specification's position is that [Suppression] is an [Owner] decision under its clause 3.4 and section 4, and an [Owner] cannot decide something they are @@ -320,9 +321,11 @@ and are stated in the [toolchain specification](TOOLING-SPEC.md), so there is no for a [Detector]. **A verdict on a [Detector] MUST also name the kind of each configuration it grades, [Semantic], -[Syntactic] or [Textual]**: ESLint, for example, is [Syntactic] without type information and -[Semantic] with it. The kind named is the one the [Detector] has without any hosted service or -licence tier, as clause 5.4 credits no finding reported only there. The kind is not a level of +[Syntactic] or [Textual]**. One [Detector] can be graded in more than one configuration: ESLint, +for example, is [Syntactic] without type information and [Semantic] with it. A configuration that +needs a hosted service or a licence tier does not count: the kind named is the one the [Detector] +has without either. Clause 5.4 is the reason: a finding reported only there does +not [Conform]. The kind is not a level of [Conformance] and does not change the verdict: a [Textual] [Detector] can [Conform], and a [Semantic] one can fail to. It is stated because the method specification's clause 3.2 requires a [Practitioner] to write a [Rule] in the strongest kind of [Detector] that can express it, and a @@ -339,11 +342,10 @@ sections 4 to 7 that the [Detector] fails. Once its maintainer knows of a gap, f as the declaration or in a file the declaration names. A declaration that records no gap claims [Conformance]. A declaration that records a gap does not; it says where the [Detector] stands. -The declaration is optional: a [Detector] with no declaration can still [Conform]. Whether a -[Detector] [Conforms] is judged on the evidence, with or without a declaration. A [Detector] that predates this document, or whose maintainer has never read -it, MAY be graded [Conforming] on evidence by anyone who exercises the clauses above against -it, and a verdict on any [Detector], declared or not, rests on that exercise and not on the -claim. A declaration tells the reader what the maintainer believes and what they know to be missing; +A [Detector] that predates this document, or whose maintainer has never read it, MAY be graded +[Conforming] on evidence: a declaration is optional. Anyone may grade a [Detector], by checking it +against the clauses above. The verdict rests on that check, not on what a declaration claims, +whether or not the [Detector] has one. A declaration tells the reader what the maintainer believes and what they know to be missing; the reader still runs the [Harness]. The shape is illustrative rather than prescribed. In a Composer manifest: diff --git a/SPEC.md b/SPEC.md index 9a2a579..1e9d7de 100644 --- a/SPEC.md +++ b/SPEC.md @@ -126,11 +126,11 @@ A report on code that does not carry the [Hazard]. #### Narrowing -Reducing a [Rule]'s scope so it stops reporting code that does not carry the [Hazard]. [Narrowing] that excludes code which does carry the [Hazard] is a [Suppression]. +Reducing a [Rule]'s scope so it stops reporting code that does not carry the [Hazard]. If the code left out does carry the [Hazard], the reduction is not a [Narrowing] but a [Suppression]. #### Suppression -Any means of preventing a [Rule] from reporting an [Instance] that carries the [Hazard]: an ignore comment, an ignore entry, a [Baseline], or a [Narrowing] that excludes it. +Any means of preventing a [Rule] from reporting an [Instance] that carries the [Hazard]: an ignore comment, an ignore entry, a [Baseline], or a reduction of the [Rule]'s scope that excludes it. #### Baseline @@ -162,7 +162,7 @@ A [Practitioner] that is an automated system. Everything said of a [Practitioner #### Owner -Whoever holds authority over a project's [Exceptions] and [Calibrations]. Always a human. The project decides who, and clause 8.7 requires the answer to be findable. Where the project has recorded nobody, the [Owner] is whoever instructed the [Practitioner], and [Escalation] goes there. One limit applies to that default. The default [Owner] is usually the person under the same deadline as the [Practitioner], so a decision they take under section 4 to keep an [Instance] unfixed, [Suppress], [Baseline] or remove a [Rule] MUST be recorded in the project record with its justification. That record MUST mark the decision as taken by a default [Owner] and state why nobody better placed was reachable, where the next reader can see who decided and why. Such a decision SHOULD be revisited by someone not under that deadline before the next release. A project that wants the separation section 4 describes names an [Owner]; the default exists so work is not blocked, not to certify that a second judgement was applied. +Whoever holds authority over a project's [Exceptions] and [Calibrations]. The [Owner] is always a human. The project decides who, and clause 8.7 requires the answer to be findable. Where the project has recorded nobody, the [Owner] is the human who instructed the [Practitioner]. If an [Agent] gave that instruction, the [Owner] is the human who instructed that [Agent]. [Escalation] goes to the [Owner], who takes the decisions section 4 gives the [Owner]. One limit applies to that default. The default [Owner] is usually the person under the same deadline as the [Practitioner], so a decision they take under section 4 to keep an [Instance] unfixed, [Suppress], [Baseline] or remove a [Rule] MUST be recorded in the project record with its justification. That record MUST mark the decision as taken by a default [Owner] and state why nobody better placed was reachable, where the next reader can see who decided and why. Such a decision SHOULD be revisited by someone not under that deadline before the next release. A project that wants the separation section 4 describes names an [Owner]; the default exists so work is not blocked, not to certify that a second judgement was applied. #### Escalation @@ -318,12 +318,12 @@ Two bounds apply, and both are checkable during the work rather than matters of too narrowly and MUST be widened, unless the independent search below has found no other [Instance] and the [Practitioner] records that the [Class] is genuinely singular today. A [Rule] that matches exactly one thing is an [Instance] [Detector] wearing a [Rule]'s clothes. - A [Rule] drawn to the exact value or name found in the [Defect] is the same [Narrowing] however - it is spelt, and where it stays because the search found nothing wider, the record MUST also name the next - wider [Rule], the one with the [Instance]'s distinguishing detail dropped, and state why it was - rejected. That rejection is tested as a [Narrowing] is under clause 3.3: a sentence stating why the - [Hazard] cannot arise in the code the wider [Rule] would add, confirmed by search, with uncertainty - routed upwards under section 4. + A [Rule] drawn to the exact value or name found in the [Defect] also matches exactly one thing, + however it is written. Where it stays because the search found nothing wider, the record MUST also name the + next wider [Rule], the one with the [Instance]'s distinguishing detail dropped, and state why it + was rejected. The reason is one sentence: why the [Hazard] cannot arise in the code the wider + [Rule] would add. A search confirms it, as clause 3.3 requires for the sentence behind a + [Narrowing]. Where the [Practitioner] is not sure, the decision goes to the [Owner] under section 4. - **Upper bound.** If the [Rule] matches code that does not carry the [Hazard], the [Class] is drawn too broadly and MUST be narrowed. [False positives] destroy a [Defence]'s credibility faster than a missing [Rule] does. One report on code that does not carry the [Hazard] is enough; there is no @@ -422,23 +422,27 @@ on two grounds only: where the stronger kind cannot express the [Rule], or where no [Detector] of the stronger kind that accepts bespoke [Rules]. For example, the stronger kind cannot express a [Rule] that reads a file the [Semantic] [Detector] does not parse. -The language has a [Detector] of the stronger kind when the project already runs one, or the -language's own [Detector] ecosystem offers one. That is the same check section 2 asks for before -a [Toolchain] gap is recorded. A stronger configuration of a [Detector] the project runs counts. -A [Detector] or configuration that only a hosted service or a licence tier provides does not -count. A [Detector] accepts bespoke [Rules] when the project can add its own [Rule] to it through -a documented extension point, such as a plugin interface or an API for writing [Rules]. It need -not meet the rest of the [detector specification](DETECTOR-SPEC.md). If its extension points -cannot reach the construct a [Rule] is about, that [Detector] cannot express the [Rule]. If no +The second ground asks two questions of each [Detector] of the stronger kind. First, is it +available? It is if the project already runs it, or the language's own [Detector] ecosystem +offers it. That is the same check section 2 asks for before a [Toolchain] gap is recorded. A +stronger configuration of a [Detector] the project runs counts. A [Detector] or configuration +that only a hosted service or a licence tier provides does not count. Second, does it accept +bespoke [Rules]? It does if the project can add its own [Rule] to it through a documented +extension point, such as a plugin interface or an API for writing [Rules]. It need not meet the +rest of the [detector specification](DETECTOR-SPEC.md). The second ground holds only when no +[Detector] of the stronger kind passes both questions. If a [Detector] passes both but its +extension points cannot reach the construct a [Rule] is about, that [Detector] cannot express the +[Rule]. If no [Detector] of the stronger kind can reach it, the stronger kind cannot express the [Rule]: that is the first ground, not the second. On the [Remediation docs] of a [Rule] they write, the [Practitioner] MUST name the [Detector] and its kind. Where that kind is weaker than [Semantic], the [Practitioner] MUST also -record, for each step down from [Semantic], which of the two grounds above holds and why, naming the -[Detectors] of the stronger kind that were considered, on the [Rule]'s [Remediation docs] and in -the description of the commit that adds the [Rule], moves it to another [Detector], or first -records the grounds. The [Practitioner] judges whether the grounds hold, as section 4 gives them the choice of [Detector]; a +record, on the [Rule]'s [Remediation docs] and in the description of the commit that adds the +[Rule], moves it to another [Detector], or first records the grounds, for each step down the +order [Semantic], [Syntactic], [Textual], which of the two grounds holds (the stronger kind cannot +express the [Rule], or the language has no [Detector] of the stronger kind that accepts bespoke +[Rules]) and why, naming the [Detectors] of the stronger kind that were considered. The [Practitioner] judges whether the grounds hold, as section 4 gives them the choice of [Detector]; a project MAY require its [Owner]'s sign-off on a weaker kind as well, by recording that requirement where it records its [Calibrations]; the sign-off is then recorded with the grounds, and where the project has recorded no such requirement none is needed. From 296aad5fe4722ba196ada4c76dda23638d018c29 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 00:02:13 +0100 Subject: [PATCH 041/105] Keys: method 2d and detector 2b quotations follow the reworded 3.2 and 7.1 Quotations updated verbatim to the new wording; no question, element or threshold changed. Co-Authored-By: Claude Opus 5.5 --- acceptance/detector/KEY.md | 11 ++++++----- acceptance/method/KEY.md | 10 ++++++---- 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 423a7a9..0712b1f 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -38,11 +38,12 @@ agent summary (those belong to the toolchain specification). says a declaration is required, or that partial satisfaction is conformance, is wrong. - **2b.** It must make the route either disableable by configuration or detectable by a rule the project can write or a mechanical check it documents; it is not required to remove or forbid - it. Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each one it MUST do at least one of two things, and either alone - satisfies this clause: make the route disableable by configuration; or make the route - detectable. Detectable means the project can find every use of the route. One way is a Rule - the project writes and registers with the Detector, as clause [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) lets it; the Detector need - not ship that Rule. The other way is a mechanical check the Detector documents." An answer that says the detector must forbid the route, or must require a + it. Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each route it ships, it MUST do at least one of two things: let + the project switch the route off in the Detector's configuration, or make the route detectable. + Either alone satisfies this clause, and neither requires the Detector to remove or forbid the + route. Detectable means the project can find every use of the route. There are two ways. One is + a Rule the project writes and registers with the Detector, as clause [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) lets it; the + Detector need not ship that Rule. The other is a mechanical check the Detector documents." An answer that says the detector must forbid the route, or must require a reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD), is wrong. - **2c.** With the rule, in the installed copy, at a version tracked together, and yes, offline. Key quotations, [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access): "Resolution of a bundled rule's identifier MUST work from the installed diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index db04c55..cbafcea 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -57,10 +57,12 @@ is correct; a bullet that says no weaker kind is ever allowed is wrong. reason. Key quotations, [3.2](../../SPEC.md#32-build-the-net): "A Rule the Practitioner writes MUST be written in the strongest kind of Detector that can express it: a Semantic one, else a Syntactic one, else a Textual one."; "Where that kind is weaker than - Semantic, the Practitioner MUST also record, for each step down from Semantic, which of the - two grounds above holds and why, naming the Detectors of the stronger kind that were considered, on - the Rule's Remediation docs and in the description of the commit that adds the Rule, moves it to - another Detector, or first records the grounds." Supporting: "On the Remediation docs of a Rule + Semantic, the Practitioner MUST also record, on the Rule's Remediation docs and in the + description of the commit that adds the Rule, moves it to another Detector, or first records the + grounds, for each step down the order Semantic, Syntactic, Textual, which of the two grounds + holds (the stronger kind cannot express the Rule, or the language has no Detector of the + stronger kind that accepts bespoke Rules) and why, naming the Detectors of the stronger kind + that were considered." Supporting: "On the Remediation docs of a Rule they write, the Practitioner MUST name the Detector and its kind." A correct answer gives three elements, and may leave out everything else above: (i) the strongest kind that can express the rule, in the order semantic, syntactic, textual; (ii) a weaker kind only where the stronger From 431b60a99d2ae9a8cc1f08a25e2238683321ac2c Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 00:05:59 +0100 Subject: [PATCH 042/105] Method section 1, Owner: the first human up a chain of agents Where an agent gave the instruction, the default Owner is the first human up the chain of instructions, so a practitioner several agents deep still finds one. Co-Authored-By: Claude Opus 5.5 --- SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index 1e9d7de..8f0199f 100644 --- a/SPEC.md +++ b/SPEC.md @@ -162,7 +162,7 @@ A [Practitioner] that is an automated system. Everything said of a [Practitioner #### Owner -Whoever holds authority over a project's [Exceptions] and [Calibrations]. The [Owner] is always a human. The project decides who, and clause 8.7 requires the answer to be findable. Where the project has recorded nobody, the [Owner] is the human who instructed the [Practitioner]. If an [Agent] gave that instruction, the [Owner] is the human who instructed that [Agent]. [Escalation] goes to the [Owner], who takes the decisions section 4 gives the [Owner]. One limit applies to that default. The default [Owner] is usually the person under the same deadline as the [Practitioner], so a decision they take under section 4 to keep an [Instance] unfixed, [Suppress], [Baseline] or remove a [Rule] MUST be recorded in the project record with its justification. That record MUST mark the decision as taken by a default [Owner] and state why nobody better placed was reachable, where the next reader can see who decided and why. Such a decision SHOULD be revisited by someone not under that deadline before the next release. A project that wants the separation section 4 describes names an [Owner]; the default exists so work is not blocked, not to certify that a second judgement was applied. +Whoever holds authority over a project's [Exceptions] and [Calibrations]. The [Owner] is always a human. The project decides who, and clause 8.7 requires the answer to be findable. Where the project has recorded nobody, the [Owner] is the human who instructed the [Practitioner]. If an [Agent] gave that instruction, the [Owner] is the first human up the chain of instructions that led to it. [Escalation] goes to the [Owner], who takes the decisions section 4 gives the [Owner]. One limit applies to that default. The default [Owner] is usually the person under the same deadline as the [Practitioner], so a decision they take under section 4 to keep an [Instance] unfixed, [Suppress], [Baseline] or remove a [Rule] MUST be recorded in the project record with its justification. That record MUST mark the decision as taken by a default [Owner] and state why nobody better placed was reachable, where the next reader can see who decided and why. Such a decision SHOULD be revisited by someone not under that deadline before the next release. A project that wants the separation section 4 describes names an [Owner]; the default exists so work is not blocked, not to certify that a second judgement was applied. #### Escalation From dfb9b68ed60cf85964f418714d80435a3e251b24 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 00:05:59 +0100 Subject: [PATCH 043/105] Acceptance runs 23 and 24: SPEC.md and DETECTOR-SPEC.md, third cohort, both FAIL Five cold Haiku readers each, made under pull request 13 at 204f95b and marked by a delegated marker. Evidence is posted on pull request 14 and linked from each record, with the findings applied to the text. Co-Authored-By: Claude Opus 5.5 --- .../runs/023-semantic-detectors-run-5.md | 238 ++++++++++++++++++ .../runs/024-semantic-detectors-run-6.md | 226 +++++++++++++++++ 2 files changed, 464 insertions(+) create mode 100644 acceptance/runs/023-semantic-detectors-run-5.md create mode 100644 acceptance/runs/024-semantic-detectors-run-6.md diff --git a/acceptance/runs/023-semantic-detectors-run-5.md b/acceptance/runs/023-semantic-detectors-run-5.md new file mode 100644 index 0000000..1c9ddd6 --- /dev/null +++ b/acceptance/runs/023-semantic-detectors-run-5.md @@ -0,0 +1,238 @@ +# Run 23: SPEC.md, semantic detectors, fifth run in pull request 14 + +- **Document**: SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + branch `editor/semantic-detectors` at 0b8d699, the commit the readers were given; the question + set, key and fixture read at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input; no tools, no settings, no project context and no + saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is byte for byte the question set, the document and the + fixture at 0b8d699, in that order, each preceded by one `===== FILE: =====` line, with + nothing else added apart from one blank line between the end of each file and the next file's + marker line. +- **Managed (policy) settings**: none; the machine applies no managed settings to the readers and + no managed settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. Every sheet has answer text, no process reports an error + (`is_error` false, subtype `success`), and every reported model is the cohort's model. The + editor reports no reader set aside for a cause the sheets cannot show. +- **Run within the pull request**: 5, counting void runs; runs in the pull request so far: 5 (19, + 20, 21, 22 and this one). Run 24, on DETECTOR-SPEC.md, is made at the same time and does not + bear on this document. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review. It read `ACCEPTANCE.md` and + `acceptance/runs/README.md` in this worktree, which differ from commit 204f95b only by this pull + request's question-set line adding 2d for SPEC.md; the document, question set, key and fixture + at 0b8d699; run 21 in full and run 19 in part; and, for the fourth pass-criteria row, the change + to SPEC.md between 3aefdd1, the document run 21 read, and 0b8d699. It also looked at the key's + commit history, which a delegated marker is not given; no mark rests on it (J2 is reasoned from + the key's text at 0b8d699). + +## Marks + +`ok` is correct, `miss` is a miss; `T` marks a miss traceable to the text. Judgements J1 to J8 +are set out below. + +| Reader | Q1 | 2a | 2b | 2c | 2d | 3.1 | 3.2 | 3.3 | 3.4 | 3.5 | 3.6 | Remediation | Defence | Section 4 | Q4 passages | +| ------ | ------- | --- | --- | --- | ------ | ------- | ------- | ------ | --- | --- | --- | ----------- | ------- | --------- | ----------- | +| 1 | correct | ok | ok | ok | miss T | No | No | Yes ok | No | No | No | ok | ok | ok | six | +| 2 | correct | ok | ok | ok | miss T | Partial | Partial | Yes ok | No | No | No | ok | ok | ok | three | +| 3 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | three | +| 4 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 5 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | three | + +Every grade under 3.1, 3.2 and 3.4 to 3.6 is one the key accepts. No reader grades 3.2, 3.5 or +3.6 Yes, so none of the key's cases that count against the document arises. + +### Question 1 + +Five of five correct. Every sheet has at least four bullets naming obligations the key lists +(class and hazard, the strongest kind of detector, the proof in a commit of its own, the sweep and +the fix of every instance, permanent and blocking, the terse message with an identifier resolving +to versioned documentation), and no bullet asserts an obligation the document does not state. +Reader 5's second bullet, recording one sentence naming two techniques tried where no rule can be +written, is section 2's "Attempt rather than pre-judge" paragraph; the closing "fix the original +defect with a reproducing test" of readers 2, 3 and 5 is section 6 and Appendix A's last sentence. + +### Question 2 + +Every reader quoted words of the document for every item, so every miss here is traceable to the +text. + +- **2a**, five of five. Each states class, rule, proof, sweep and the fix of every instance, in + that order, and quotes section 2's sentence "The order is section 3's: attribute the [Defect] to + a [Class], build the [Rule], prove that it fires, run the [Sweep], then fix every [Instance]." +- **2b**, five of five. Readers 1, 2, 3 and 4 say the owner is a human; reader 5 says "Only the + Owner" (J4). All five quote the key's section 4 sentence and say an agent may not. +- **2c**, five of five. Each states all six clauses and a verdict resting on reproduction, not on + the report, in its sentence or its quotation of section 7; reader 2's second quotation gives the + red and green reruns. No reader accepts the pull request description as the verdict. +- **2d**, none of five. The key's three elements are (i) the strongest kind, in the order + semantic, syntactic, textual; (ii) both grounds for a weaker kind; (iii) the reason recorded, + with at least one place named and the places not given as alternatives (J2). + - Reader 1: (i) in its first quotation. (ii) missed: "A weaker kind is allowed 'on two grounds + only'" says there are two and states neither, and its second quotation, "which of the two + grounds above holds and why, naming the [Detectors] of the stronger kind that were + considered", does not state them either. (iii) missed: that quotation stops before the places. + - Reader 2: (i) and (ii) stated, the second quotation giving both grounds. (iii) missed: "the + practitioner must record which ground holds and why", with a quotation ending at "were + considered...", names no place (J2). + - Reader 3: (i) and (iii) stated, its second quotation naming the remediation docs "and" the + commit description. (ii) missed: "A weaker kind is allowed only on one of two grounds", and no + quotation states them. + - Reader 4: (i) and (ii) stated. (iii) missed: "The record must include 'for each step down from + [Semantic], which of the two grounds above holds and why, naming the [Detectors] of the + stronger kind that were considered'" names no place (J2). + - Reader 5: (ii) and (iii) stated, the places given with "and". (i) missed: "A Rule must be + written in the strongest detector kind that can express it", and neither quotation names + semantic, syntactic or textual (J3). + +### Question 3 + +Five of five on every keyed item. + +- **3.3**: five of five Yes. Each says the inline ignore is a [Suppression] not judged under this + clause, as Part B's opening now says; none attributes it to 3.3. Reader 5's caveat that the + fixture does not show a fresh-checkout reproduction leaves its grade at Yes. +- **Remediation**: five of five say it does not conform, each naming at least one failing clause + the key accepts. +- **Defence**: five of five say it does not conform, each naming 3.6, and readers 1, 2, 3 and 4 + naming 3.2 as well. +- **Section 4**: five of five name the inline ignore on `Report::render()`, and each also names + leaving the next wider rule unbuilt, which the key accepts alongside it. + +### Judgements made in marking + +Made as a careful reader of the document would, as `ACCEPTANCE.md` requires. Where a rule settles +the point, the record says which. + +- **J1. The sentence.** Where a reader wrote more than one sentence before or around its + quotations, all of that text is read with the quotations as the answer. No mark turns on it. +- **J2. 2d's element (iii) needs a place named.** The key says of (iii) that "naming one of the + two places is enough" and that presenting them as alternatives is wrong; "enough" sets a + minimum, and a key that let an answer name no place would have no minimum to state and no + alternatives to forbid. Runs 19 and 21 read it the same way. Readers 2 and 4 miss on this + alone; read otherwise, both would be correct on 2d and the 2d miss would rest on readers 1, 3 + and 5. The verdict does not turn on it. +- **J3. 2d's element (i) needs the order (reader 5).** The key's element is "the strongest kind + that can express the rule, in the order semantic, syntactic, textual". "The strongest detector + kind that can express it" names no kind, and the question asks which kind. Read otherwise, + reader 5 would be correct on 2d; the verdict would not change. +- **J4. "The Owner" (reader 5, 2b).** Settled by the rule that a defined term used in its defined + sense states what it means: [Owner] is "Always a human". +- **J5. Every passage under question 4 counts, including one set beside the main passage.** + Settled in part by the rule that "a quotation under question 4, of any length" covers what its + words overlap, and that a passage named "by clause number or description" covers what it names. + Where a reader says it was confused by two passages read together, a careful reader takes both + as the confusing passage, since the confusion lies in neither alone. So reader 1's description + of the Narrowing entry, its "Fix them all" and its section 4 sentence, and reader 3's + description of the Semantic entry's "need not accept bespoke [Rules]" sentence, are recorded as + passages. The finding on the Narrowing entry turns on this: read only as context, reader 1's + description would leave the entry to reader 4 alone. No other finding turns on it. +- **J6. Elided quotations.** A quotation with an ellipsis is read as one quotation per unbroken + run of words, each in the document's order (reader 5's 3.3 passage, reader 2's 3.5 passage). + Settled by the rule on quoting: consecutive words of any length. +- **J7. Clause labels.** A clause number given with a quotation, as question 4 asks, is read as + the quotation's label, not as naming the whole clause, as run 21 read it. Reader 5's "Clause + 3.3, Part B" label on "the [Practitioner]'s to decide", which lies in the paragraph introducing + the three parts, does not extend the passage to all of Part B. +- **J8. The earlier 3.2 passage, as it reads in the changed document.** Run 21's passage lay in + clause 3.2's two numbered grounds and the paragraph that began "For the second ground". In the + changed document the grounds are one sentence at the end of the paragraph beginning + "**Qualifying is not the same as being the right host.**", and the "For the second ground" + paragraph is split: its sentences on when the language has a detector of the stronger kind and + when one accepts bespoke rules, reworded, form the paragraph beginning "The language has a + [Detector] of the stronger kind", and its record sentences, kept, form the next paragraph, + beginning "On the [Remediation docs] of a [Rule] they write". All three are read as that + earlier paragraph and list as they now read. + +## Passages quoted as confusing + +Same passage as `ACCEPTANCE.md` defines it: sentences in common, or the same paragraph; a list +item, table row or terminology entry is its own paragraph. + +| Passage | Clause | Readers | Words quoted, verbatim, or the description used | +| ---------------------------------------------------------------------------- | ------------ | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| The lower bound, the exact value or name and the rejection of the wider rule | 3.1 | three: 1, 2, 4 | 1 and 4: "A [Rule] drawn to the exact value or name found in the [Defect] is the same [Narrowing] however it is spelt." 2: "That rejection is tested as a [Narrowing] is under clause 3.3: a sentence stating why the [Hazard] cannot arise in the code the wider [Rule] would add, confirmed by search, with uncertainty routed upwards under section 4." All in the lower-bound list item | +| The Owner entry | 1, Owner | two: 1, 5 | 1: "Where the project has recorded nobody, the [Owner] is whoever instructed the [Practitioner]." 5: "Always a human." and "Where the project has recorded nobody, the [Owner] is whoever instructed the [Practitioner]." | +| The Narrowing entry | 1, Narrowing | two: 1, 4 | 4: "[Narrowing] that excludes code which does carry the [Hazard] is a [Suppression]." 1: by description, "The terminology section defines [Narrowing] as reducing a rule's scope to stop reporting code without the hazard" (J5) | +| When the language has a detector of the stronger kind | 3.2 | one: 3 | "The language has a [Detector] of the stronger kind when the project already runs one, or the language's own [Detector] ecosystem offers one." Run 21's open passage (J8) | +| The two grounds for a weaker kind | 3.2 | one: 3 | "no [Detector] of the stronger kind that accepts bespoke [Rules]", in the sentence that allows a weaker kind on two grounds. Run 21's open passage (J8) | +| A semantic detector need not accept bespoke rules | 1, Semantic | one: 3 | By description, "which the terminology says may not accept bespoke rules" (J5) | +| Where a gap is recorded | 3.2 | one: 2 | "A gap is recorded in the project record where the [Toolchain] defines one, as the [toolchain specification](TOOLING-SPEC.md) requires it to; only where it does not is the location the project's choice, and then the only requirement is that somebody deciding what to invest in the [Toolchain] would find it." | +| The entry point for a bundled defence | 3.5 | one: 2 | "For a [Bundled defence], the project whose entry point demonstrates it is any project the [Toolchain] is genuinely installed into, the [Consuming project] included", and the terms "Bundled defence" and "Consuming project", which occur only in that sentence | +| Narrowings enumerable and reported to the owner | 3.3, Part B | one: 5 | "Every [Narrowing] MUST be enumerable by the same means as the project's [Exceptions]" and "a [Narrowing] that excludes more code than the [Rule] still covers MUST be reported to the [Owner] as if it were a [Suppression]" (J6) | +| The three parts of clause 3.3 | 3.3 | one: 5 | "the [Practitioner]'s to decide" (J5, J7) | +| A defence conforming whilst the sweep is incomplete | 7 | one: 1 | "A [Defence] can [Conform] whilst the [Sweep] is incomplete." | +| Fix them all | 3.4 | one: 1 | "Fix them all" (J5) | +| The dividing line, an agent's authority | 4 | one: 1 | "An [Agent] MUST NOT [Baseline], suppress or knowingly leave an [Instance] unfixed" (J5) | + +Readers' accounts of what they could not tell are recorded with the passage they attach to and are +not suggestions; none of the sheets proposes a change. + +**Earlier passages (the fourth row).** Run 21 found four passages named by two or more readers, +the open ones. As the paragraphs read in the changed document: + +- Clause 3.2's two grounds and the "For the second ground" paragraph (J8): reader 3 names a + sentence of the paragraph beginning "The language has a [Detector] of the stronger kind", and + words of the sentence that now states the two grounds. Not cleared. +- Section 4's bullet on leaving the next wider rule unbuilt: named by no reader. Reader 2's 3.1 + passage is in clause 3.1's lower-bound list item, not the bullet. Cleared. +- The fourth column of table row 3.1 in section 3: named by no reader. Cleared. +- Section 4's lead-in, now "**Decisions that MUST go to the [Owner]**, which the [Practitioner] + never takes alone:": named by no reader. Reader 1's section 4 sentence lies in the paragraph + beginning "The dividing line", not the lead-in. Cleared. + +No sentence kept from one of the three cleared paragraphs is named elsewhere in the document. +Reader 1's "Fix them all" lies in the clause 3.4 paragraph run 19 named and run 21 cleared; it is +not an open passage and is recorded above as one reader's. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in b172fd6 and the Owner fix +after its review (key quotations kept verbatim in 296aad5). SPEC.md: clause 3.2's record sentence +names both grounds itself, and its sentence on when the language has a detector of the stronger kind +is plain; clause 3.1's lower-bound item is in short, plain sentences; section 1's Owner entry says +plainly who the default Owner is, the first human up the chain of instructions, and its Narrowing +entry says plainly when a narrowing is a suppression. A fresh cohort follows. + +## Process change + +Made under a process change before it merged: yes, pull request 13 at commit 204f95b, named +before the readers started. + +## Verdict + +**FAIL.** Against the criteria in `ACCEPTANCE.md`: + +- Load-bearing statements: not met. 2a, 2b and 2c five of five; 2d none of five. Readers 1 and 3 + state neither ground for a weaker kind, only that there are two; readers 1, 2 and 4 name no + place for the record (J2); reader 5 does not give the order semantic, syntactic, textual (J3). +- Restatement and application: met. Question 1 and every keyed item of question 3 five of five. +- Misses traceable to the text, each a finding: 2d for all five readers, on clause 3.2's + paragraph beginning "**Qualifying is not the same as being the right host.**" and the record + sentence "Where that kind is weaker than [Semantic], the [Practitioner] MUST also record, for + each step down from [Semantic], which of the two grounds above holds and why, naming the + [Detectors] of the stronger kind that were considered, on the [Rule]'s [Remediation docs] and in + the description of the commit that adds the [Rule], moves it to another [Detector], or first + records the grounds.", which every reader quotes. +- Passages found confusing by two or more, each a finding that blocks until the text is changed + and a fresh cohort no longer names it: clause 3.1's lower-bound list item (three readers); the + Owner entry in section 1 (two); the Narrowing entry in section 1 (two, on J5). Run 21's clause + 3.2 passage is not cleared: reader 3 names the paragraph beginning "The language has a + [Detector] of the stronger kind" and the sentence stating the two grounds (J8). Run 21's other + three passages are cleared. +- Single-reader notes, recorded: the Semantic entry (reader 3); clause 3.2's paragraph on where a + gap is recorded (reader 2); clause 3.5's bundled defence sentence (reader 2); clause 3.3's + paragraph on narrowings enumerable and reported to the owner, and its three-parts paragraph + (reader 5); section 7's conforming defence over an incomplete sweep (reader 1); clause 3.4's + "Fix them all" (reader 1); section 4's "An [Agent] MUST NOT" sentence (reader 1). + +Answer sheets, the reader prompt, the command and the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090684297), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090684613), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090684941), [comment 4](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090685201). diff --git a/acceptance/runs/024-semantic-detectors-run-6.md b/acceptance/runs/024-semantic-detectors-run-6.md new file mode 100644 index 0000000..96f845e --- /dev/null +++ b/acceptance/runs/024-semantic-detectors-run-6.md @@ -0,0 +1,226 @@ +# Run 24: DETECTOR-SPEC.md, semantic detectors, sixth run in pull request 14 + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + editor's branch `editor/semantic-detectors` at 0b8d699; the question set, key and fixture read + at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input, with no tools, no settings, no project context + and no saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is the question set, the document and the fixture at + 0b8d699, in that order, each preceded by one `===== FILE: =====` line, with the files + separated by one blank line and nothing else added, the same form as run 22's packet +- **Managed (policy) settings**: none; the machine applies no managed settings and no managed + settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. The editor reports none set aside for a cause the sheets cannot + show; every sheet has answer text, no process reports an error (`is_error` false, subtype + `success`), and every reported model is the cohort's model +- **Run within the pull request**: 6, counting void runs; run 23, on SPEC.md, was made at the same + time and does not bear on this document +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review + +## Marks + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | Section 8 | Kind | 6.4 | Q4 entries | +| ------ | ------- | ---- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --------- | ---------- | --- | ---------- | +| 1 | correct | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 2 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 3 | miss | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 4 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 5 | correct | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | + +"Q4 entries" counts the passages each reader lists under question 4; the words each quotes are in +the table of passages below. Every miss under question 2 is traceable to the text: each answer +quotes the document in support. The miss under question 1 is the reader's own: the bullet it turns +on quotes nothing. No question was left unanswered. + +### Question 1 + +Every restatement names four or more obligations from the key's list. Readers 1, 2, 4 and 5 assert +no obligation the document does not state: correct. Reader 2's "A reason is SHOULD-required" +states 7.2 as the SHOULD it is, and reader 4's "A detector conforms when it meets every MUST in +sections 4 to 7" states section 8's definition, not an obligation; neither is wrong. Reader 5's +"the command's own output must carry a usable summary with each finding's location and +identifier" follows 5.3's own words (judgement 2). Reader 3's fourth bullet says "each finding must +appear in the output of the command the practitioner ran"; clause 5.3 says "The summary need not +list every finding", so the bullet asserts an obligation the document does not state: incorrect +(judgement 2). Correct, four of five. + +### Question 2 + +The key's answers state these elements: + +- 2a: (a) Conformance is satisfying every MUST in sections 4 to 7; (b) judged on evidence; (c) a + detector that predates the document can conform; (d) the declaration is optional. +- 2b: (a) make the route disableable by configuration, or (b) detectable by a Rule the project can + write or a mechanical check the Detector documents; (c) the detector is not required to remove or + forbid it. +- 2c: (a) with the rule; (b) in the installed copy; (c) at a version tracked together; (d) yes, + reachable offline. + +Each answer is read as its sentence and its quotation together, for the gist of each element. + +| Reader | 2a | 2b | 2c | +| ------ | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------- | +| 1 | miss: (d) not stated; (b) from the 8.1 quotation (judgement 3) | miss: (c) not stated (judgement 4) | ok: all four, (c) from the 6.3 quotation | +| 2 | ok: all four; (d) from the section 8 quotation, "a declaration under clause 8.1 is optional" | miss: (c) not stated (judgement 4) | ok: all four, (c) from the 6.3 quotation | +| 3 | miss: (d) not stated; (b) from the 8.1 quotation (judgement 3) | miss: (c) not stated (judgement 4); (b) by the defined term (judgement 5) | ok: all four, (c) from the 6.3 quotation | +| 4 | ok: all four; (d) from the section 8 quotation | miss: (c) not stated (judgement 4); (b) as "findable by the project" | ok: all four | +| 5 | miss: (d) not stated (judgement 3) | miss: (c) not stated (judgement 4) | ok: all four, from the sentence and the 6.3 and 6.2 quotations | + +No answer asserts anything the document does not state or that the key says makes an answer +wrong: none says a declaration is required, that partial satisfaction is conformance, that the +detector must forbid the route or require a reason, or that a website will do. + +### Question 3 + +The key's intended verdicts are Yes for every clause but 6.3, which is No; the section 8 verdict +is "does not conform", because 6.3 fails; 6.4 does not change the verdict. + +- **Every clause**: all five readers give the intended verdict on every clause, 4.1 to 7.1. On + 6.3 each cites that one unresolved identifier fails the clause; no reader grades it Partial on + the grounds that thirty-eight resolve. Reader 2 calls the fixture ambiguous on whether the two + rules are bundled, and grades No either way. +- **Section 8**: all five say Lintomatic does not conform, because 6.3 fails. +- **Kind**: all five say the fixture does not let them name the kind and name none, so none is + marked on it. Readers 2 and 5 call that part of the verdict incomplete or undetermined but do not + make the verdict turn on it; readers 3 and 4 say outright that the kind does not change the + verdict. +- **6.4**: all five say it does not change the verdict, because it is a SHOULD. Reader 1 adds that a + maintainer declaring a version would record 6.3 and 6.4 under `known-gaps`, although 8.1 defines a + gap as a failed MUST; reader 3 adds that 6.4 becomes a MUST only inside a Toolchain. Neither aside + alters the answer marked (judgement 6). + +### Judgements made in marking + +01. **How an open judgement is made.** As `ACCEPTANCE.md` at 204f95b says, and as the editor's + instruction asks: as a careful reader of the document would, with the reason stated. Where a + rule settles a mark, the rule is applied as written, however the result looks. +02. **Question 1, the output bullets of readers 3 and 5.** Clause 5.3 at 0b8d699 says "The summary + need not list every finding". Reader 3's "each finding must appear in the output" contradicts + that sentence, and a careful reader takes it as an obligation the document does not state. + Reader 5's "a usable summary with each finding's location and identifier" repeats the clause's + "a usable summary, each finding in it with where it is and its [Identifier]", and a careful + reader takes "each finding" there as each finding in the summary, as the clause does. +03. **2a (d), the declaration optional, readers 1, 3 and 5.** As run 22 held, (d) answers the + question's second part, whether a detector that has never heard of the document can conform. + Each of the three quotes the 8.1 sentence that a detector whose maintainer has never read the + document "MAY be graded [Conforming] on evidence"; none says that a declaration is optional or + not needed. That no declaration is needed follows from the quotation but is not said by it. + `ACCEPTANCE.md` names the ways an element is stated: the document's words, a paraphrase, a + synonym, a defined term in its defined sense. An element reached only by inference from another + is not among them, and a careful reader of that rule does not add it; this keeps marking + mechanical, as the question set section asks. Marked not stated. Readers 2 and 4 quote section + 8's "a declaration under clause 8.1 is optional", and so state it. +04. **2b (c), not required to remove or forbid, all five.** As run 22 held, (c) answers what the + detector must do about the route, from the side of what it need not do. Every reader says, or + quotes, that either of the two things alone satisfies the clause. That entails that removal or + forbidding is not required, but does not say it, and 7.1's first sentence, "The [Detector] does + not have to remove or forbid an inline [Suppression] route", which says it, is quoted by none. + By the same reasoning as judgement 3, marked not stated. +05. **2b (b), the means of detection.** As in run 22: clause 7.1 defines "Detectable" in its own + text, and each reader uses "detectable", or reader 4 "findable by the project", in that sense, + so (b) is stated. It decides no mark, since every 2b answer misses (c). +06. **Asides outside the keyed answer.** Reader 1's statement that 6.4 would be recorded as a known + gap, and reader 3's on when 6.4 is a MUST, are not keyed items; each sheet's answer to whether + 6.4 changes the verdict is the key's, and is marked so. +07. **Question 4, single words and short runs quoted.** Reader 3 quotes "configuration", "in + configuration" and "configuration file"; reader 4 quotes "credits"; reader 5 quotes "MAY". + `ACCEPTANCE.md` says a quotation "of any length" covers every sentence an occurrence of the + quoted words overlaps, "at every place that run occurs", with letter case ignored. The rule + settles it: "configuration" covers eight sentences, among them 4.1's first and the 7.1 sentence + "For each one it MUST do at least one of two things ..."; "MAY" covers nine, among them the 7.1 + sentence "The [Detector] MAY ship them." and the 8.1 sentence beginning "A [Detector] that + predates". Each place is listed in the table below. +08. **Question 4, a heading.** A heading counts as one sentence; it is taken as its own paragraph, + not part of the paragraph below it, since it is a separate block of the document. Reader 5's + "MAY" covers the headings of 6.5, 7.1 and 8.1; none of those decides anything, since the + paragraphs below 7.1 and 8.1 that block are covered by other sentences. +09. **Question 4, reader 4's closing remark.** "Several clauses also cite method-specification + clauses (3.2, 3.4, 3.6, 8.1–8.3) and the toolchain specification, which were not in the packet." + It names clauses of the method specification, not of this document, and says nothing was + confusing; a careful reader takes it as a remark about the packet, not a passage named. Taken + otherwise it would add nothing that changes the verdict. +10. **The open passages, and clause 4.1.** The passages runs 20 and 22 found confusing by two or + more, and not since cleared, are taken as: 7.1's first paragraph; 8.1's first paragraph; 8.1's + second paragraph; and 6.1's paragraph beginning "For a [Bundled rule]". Clause 4.1, named by + three readers of run 20, is taken as cleared by run 22: the document's text changed after run + 20 and no reader of run 22 named a passage covering a sentence of 4.1. A careful reader takes + "until the text is changed" as the document's text, since the same row speaks of "the changed + document". Read the other way, 4.1 would still be open and outstanding, since reader 3's + "configuration" covers its first sentence; the verdict does not turn on it. + +No re-mark. + +## Passages quoted as confusing + +Grouped by paragraph, as `ACCEPTANCE.md` defines the same passage. A passage blocks when two or +more readers name it, or when it covers a sentence of a paragraph in which one of the open +passages lay (judgement 10), as that paragraph reads at 0b8d699. + +| Clause and paragraph | Readers | Words quoted, verbatim | Blocks | +| ----------------------------------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------- | +| 8, the paragraph beginning "A verdict on a [Detector] MUST also name", its first and second sentences | 1, 3, 4, 5 | 1, 4, 5: "The kind named is the one the [Detector] has without any hosted service or licence tier, as clause 5.4 credits no finding reported only there." 4 also: "credits". 3: "A verdict on a [Detector] MUST also name the kind of each configuration it grades"; and "configuration" (judgement 7) | Yes: four readers | +| 7.1, first paragraph, the sentences "The [Detector] MAY ship them." and "For each one it MUST ..." | 3, 5 | 3: "configuration", covering "make the route disableable by configuration". 5: "MAY", covering "The [Detector] MAY ship them." (judgement 7) | Yes: two readers, and the open 7.1 paragraph of runs 20 and 22 | +| 8.1, second paragraph, the sentence beginning "A [Detector] that predates" | 5 | "A [Detector] that predates this document, or whose maintainer has never read it, MAY be graded [Conforming] on evidence by anyone who exercises the clauses above against it, and a verdict on any [Detector], declared or not, rests on that exercise and not on the claim." Also "MAY" | Yes: the open 8.1 second paragraph of runs 20 and 22 | +| 4.1, first paragraph, "Configuration of an existing [Rule] set is not sufficient." | 3 | "configuration" (judgement 7) | No (judgement 10) | +| 8, second paragraph, its first sentence | 2 | "Partial [Conformance] MUST NOT be described as [Conformance]." | No | +| 8, third paragraph, its first sentence | 2 | "This document defines one level," | No | +| 4.2, first paragraph, its second sentence | 3 | "a [Detector] failing 4.1 is graded here on its [Bundled rules]"; "every [Rule] the [Detector] runs, bundled or bespoke" | No | +| 4.3, first paragraph, the sentences ending "configuration file." and "in configuration is not." | 3 | "configuration"; "in configuration"; "configuration file" (judgement 7) | No | +| 4.3, the table rows for `proj.no-raw-sql` and `rule 7` | 3 | "configuration"; "in configuration"; "configuration file" (judgement 7) | No | +| 2, the terminology table, the row for [Semantic], [Syntactic], [Textual] | 3 | "configuration" (judgement 7) | No | +| 6.4, the Why paragraph | 4 | "the [toolchain specification](TOOLING-SPEC.md)'s self-audit makes this check a MUST for the [Toolchain]" | No | +| 1, first paragraph; 2, the definition of [Rule author]; 6.3, first paragraph; 6.5, first paragraph | 5 | "MAY", each by an occurrence of "MAY" or "May" (judgement 7) | No | +| The headings of 6.5, 7.1 and 8.1 | 5 | "MAY" (judgements 7 and 8) | No | + +Of the open passages, 8.1's first paragraph and 6.1's paragraph beginning "For a [Bundled rule]" +are cleared: the text of each changed after run 22, and no reader of this cohort names a passage +covering a sentence of either. 7.1's first paragraph and 8.1's second paragraph are still +outstanding. The section 8 paragraph on kind was also named by one reader of run 22. + +## Against the criteria + +- **Load-bearing statements (every reader on every one)**: not met. 2a two of five correct + (readers 2 and 4), 2b none of five, 2c five of five. +- **Restatement and application (all but one on each keyed item)**: met. Question 1 four of five; + every clause of question 3, the section 8 verdict and 6.4 five of five. +- **A miss traceable to the text (none)**: not met. Eight, each a finding: 2a by readers 1, 3 and + 5; 2b by all five. +- **A passage found confusing by two or more (none outstanding)**: not met. The section 8 + paragraph on kind is named by four readers, and 7.1's first paragraph by two; and the open + passages in 7.1's first paragraph and 8.1's second paragraph are still outstanding, since this + cohort names passages covering sentences of each. +- **From one reader only**: the section 8 sentences on partial conformance and on one level, the + 4.1, 4.2 and 4.3 passages, the section 2 table row, the 6.4 Why paragraph and the other places + reader 5's "MAY" covers, recorded. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in b172fd6 (key quotations kept +verbatim in 296aad5). DETECTOR-SPEC.md: 8.1's sentence that a detector may be graded on evidence +says the declaration is optional; 7.1's sentence that either means alone satisfies the clause says +neither requires the detector to remove or forbid the route; section 8's paragraph on naming the +kind, 7.1's first paragraph and 8.1's second paragraph are in short, plain sentences. Pull request +13 now limits a question 4 quotation to the place the reader names, so this run's one-word +quotations would no longer reach other passages. A fresh cohort follows. + +## Process change + +Made under the process change of pull request 13 before it merged, at commit 204f95b, named +before the readers started; `ACCEPTANCE.md` and `acceptance/runs/README.md` in this worktree +differ from 204f95b only by this pull request's question-set line adding question 2d for SPEC.md, +which does not bear on this document. + +## Verdict + +FAIL on the first, third and fourth criteria. Answer sheets, the readers' prompt and command, and +the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090685442), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090685662), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090685883). From 0b2d190c66dcec2bf19d1242fbe86d6b9d2cf99a Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 00:17:22 +0100 Subject: [PATCH 044/105] Method 3.1, 3.2, 3.5 and 3.6, detector 7.1 and 8: runs 25 and 26 findings applied Clarity edits, no obligation changed in meaning: - SPEC.md 3.2: the record sentence states both grounds in its main clause; the second-ground paragraph cut to what "has a detector of the stronger kind" and "accepts bespoke rules" mean, with the unreachable-construct case as the first ground. - SPEC.md 3.5: bundled defence glossed in place; the undefined term consuming project dropped. - SPEC.md 3.1: the independence paragraph shorter; a run of the rule is never a technique. - SPEC.md 3.6: "A Rule's documentation MUST be versioned with the Rule." - DETECTOR-SPEC.md 7.1: "need not remove or forbid the route" in the MUST sentence; detectable defined in one sentence. - DETECTOR-SPEC.md 8: the paragraph on the kind shorter, the duty placed on whoever gives the verdict. Changelog Unreleased lines extended, run NNN placeholders kept. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 19 ++++++++++--- DETECTOR-SPEC.md | 39 ++++++++++++-------------- SPEC.md | 73 +++++++++++++++++++++--------------------------- 3 files changed, 65 insertions(+), 66 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 24410f4..39d84ff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -62,8 +62,15 @@ version entry with no cohort line has not been accepted and is not published. the hazard is a suppression, not a narrowing; and the Owner entry says the default owner is the human who instructed the practitioner, or, where an agent gave the instruction, the human who instructed that agent, and takes section [4](SPEC.md#4-authority-which-decisions-belong-to-whom)'s - decisions. Each after readers of run 23 missed or named the passage. Accepted by run NNN. Minor - version when released. + decisions. Each after readers of run 23 missed or named the passage. After run 25, clarity + edits: clause [3.2](SPEC.md#32-build-the-net)'s record sentence states both grounds in its main clause, not in brackets, + and its paragraph on the second ground says in fewer sentences when the language has a + detector of the stronger kind and when it accepts bespoke rules; clause [3.5](SPEC.md#35-enforce-permanently-and-block) says what a + bundled defence is and that a project installing somebody else's toolchain counts, without the + undefined term consuming project; clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s paragraph on independent techniques is shorter + and says plainly that a run of the rule is never one; and clause [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation)'s versioning sentence + says a rule's documentation is versioned with the rule. Each after readers of run 25 missed or + named the passage. Accepted by run NNN. Minor version when released. ### 1.1.0, 2026-10-02 @@ -154,8 +161,12 @@ pass it. sentence that a detector may be graded conforming on evidence says a declaration is optional, and its paragraph is in short sentences; section [8](DETECTOR-SPEC.md#8-conformance)'s paragraph on the kind says one detector can be graded in more than one configuration and that one needing a hosted service or - licence tier does not count. Each after readers of run 24 missed or named the passage. Accepted - by run NNN. Minor version when released. + licence tier does not count. Each after readers of run 24 missed or named the passage. After + run 26, clarity edits: clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)'s obligation sentence itself says the detector need not + remove or forbid the route, and its definition of detectable is one sentence; section [8](DETECTOR-SPEC.md#8-conformance)'s + paragraph on the kind is shorter, says what the kind is, and says the duty is on whoever gives + the verdict, not on the detector. Each after readers of run 26 missed or named the passage. + Accepted by run NNN. Minor version when released. ### 1.0.0, 2026-09-08 diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index 2139361..1681991 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -275,16 +275,15 @@ nothing to run. ### 7.1 A detector MAY offer an inline suppression route, but MUST make it detectable or disableable -The [Detector] does not have to remove or forbid an inline [Suppression] route. An inline ignore -comment, a per-line directive and a generated [Baseline] are all such routes, and the [Detector] -MAY ship them. For each route it ships, it MUST do at least one of two things: let the project -switch the route off in the [Detector]'s configuration, or make the route detectable. Either alone -satisfies this clause, and neither requires the [Detector] to remove or forbid the route. -Detectable means the project can find every use of the route. There are two ways. One is a -[Rule] the project writes and registers with the [Detector], as clause 4.1 lets it; the -[Detector] need not ship that [Rule]. The other is a mechanical check the [Detector] documents. -Either of the two things lets a project that decides to forbid the route enforce that decision. -A route that can be neither switched off nor seen does not [Conform]. +An inline ignore comment, a per-line directive and a generated [Baseline] are all inline +[Suppression] routes, and the [Detector] MAY ship them. For each route it ships, it MUST do at +least one of two things, and need not remove or forbid the route: let the project switch the +route off in the [Detector]'s configuration, or make the route detectable. Either alone satisfies +this clause. A route is detectable when the project can find every use of it, either with a +[Rule] of the project's own that the [Detector] runs, or with a mechanical check the [Detector] +documents. The [Detector] need not ship that [Rule]. Switching the route off or +detecting it lets a project that decides to forbid the route enforce that decision. A route that +can be neither switched off nor detected does not [Conform]. **Why**: the method specification's position is that [Suppression] is an [Owner] decision under its clause 3.4 and section 4, and an [Owner] cannot decide something they are @@ -320,17 +319,15 @@ summary for an [Agent]'s context, are obligations on a project's assembled [Tool and are stated in the [toolchain specification](TOOLING-SPEC.md), so there is no separate [Agent]-support level for a [Detector]. -**A verdict on a [Detector] MUST also name the kind of each configuration it grades, [Semantic], -[Syntactic] or [Textual]**. One [Detector] can be graded in more than one configuration: ESLint, -for example, is [Syntactic] without type information and [Semantic] with it. A configuration that -needs a hosted service or a licence tier does not count: the kind named is the one the [Detector] -has without either. Clause 5.4 is the reason: a finding reported only there does -not [Conform]. The kind is not a level of -[Conformance] and does not change the verdict: a [Textual] [Detector] can [Conform], and a -[Semantic] one can fail to. It is stated because the method specification's clause 3.2 requires a -[Practitioner] to write a [Rule] in the strongest kind of [Detector] that can express it, and a -verdict that said only [Conforming] would present a [Textual] [Detector] as an equal host to a -[Semantic] one. +**A verdict on a [Detector] MUST name its kind, [Semantic], [Syntactic] or [Textual], in each +configuration the verdict grades.** The kind is what a [Rule] can ask in that configuration, as +section 2 glosses it. ESLint, for example, is [Syntactic] without type information and [Semantic] +with it. The kind named is the one the [Detector] has without a hosted service or a licence tier, +since under clause 5.4 a finding reported only there does not [Conform]. This duty is on whoever +gives the verdict, not on the [Detector]: the kind is not a level of [Conformance] and does not +change the verdict, so a [Textual] [Detector] can [Conform], and a [Semantic] one can fail to. It +is named because the method specification's clause 3.2 requires a [Rule] to be written in the +strongest kind of [Detector] that can express it. ### 8.1 A maintainer MAY declare the version of this document the detector conforms to, and the declaration records known gaps diff --git a/SPEC.md b/SPEC.md index 8f0199f..0c373d8 100644 --- a/SPEC.md +++ b/SPEC.md @@ -359,21 +359,16 @@ new means the search is done. That is a test the [Practitioner] can apply and re threshold, and it is the standard clause 3.4 refers back to. Two techniques are independent when they would miss different things. A text search for the token -and a reading of the code paths that consume the value are independent; two text searches for two +and a reading of the code that consumes the value are independent; two text searches for two spellings of the same token are one technique; asking someone who knows the system is a third. For a -[Class] defined by how code is spelt, writing the idiom the other ways it is commonly spelt and searching -the codebase for each of them is a technique, because a search for one spelling misses every other; -whether the [Rule] catches those spellings is the confirmation step, not the technique. The -[Rule] itself is never one of the techniques, because the search exists to check the [Rule]. -A clean run of the [Rule] is therefore not evidence of saturation either: "the [Rule] found nothing -the reading had missed" is one technique and a [Rule] run, not two techniques, and the stopping -criterion cannot be applied to it. -The record MUST name both techniques, chosen before the [Rule] is run against the codebase, and -MUST state what each found that the other could not have checked. A technique counts only if it does -not invoke the [Detector] under proof in any configuration and would exist unchanged had the -[Rule] never been written; a [Sweep]-shaped run of the [Rule], however comprehensive it looks, is -never one of the two. Where a later technique shows that two earlier checks shared a blind spot, they -were one technique, and a third is owed. +[Class] defined by how code is spelt, searching for each other common spelling of the idiom is a +technique. A technique counts only if it does not run the [Detector] under proof in any +configuration and would exist unchanged had the [Rule] never been written. So a run of the +[Rule], however wide, is never a technique, and a clean run of it shows nothing about saturation: +this search exists to check the [Rule]. The record MUST name both techniques, chosen before the +[Rule] is run against the codebase, and MUST state what each found that the other could not have +checked. Where a later technique shows that two earlier checks shared a blind spot, they were one +technique, and a third is owed. That turns the single-[Instance] case into something checkable rather than a matter of taste. If an independent search finds [Instances] the [Rule] missed, the [Class] was drawn too narrowly and the [Rule] @@ -422,27 +417,23 @@ on two grounds only: where the stronger kind cannot express the [Rule], or where no [Detector] of the stronger kind that accepts bespoke [Rules]. For example, the stronger kind cannot express a [Rule] that reads a file the [Semantic] [Detector] does not parse. -The second ground asks two questions of each [Detector] of the stronger kind. First, is it -available? It is if the project already runs it, or the language's own [Detector] ecosystem -offers it. That is the same check section 2 asks for before a [Toolchain] gap is recorded. A -stronger configuration of a [Detector] the project runs counts. A [Detector] or configuration -that only a hosted service or a licence tier provides does not count. Second, does it accept -bespoke [Rules]? It does if the project can add its own [Rule] to it through a documented -extension point, such as a plugin interface or an API for writing [Rules]. It need not meet the -rest of the [detector specification](DETECTOR-SPEC.md). The second ground holds only when no -[Detector] of the stronger kind passes both questions. If a [Detector] passes both but its -extension points cannot reach the construct a [Rule] is about, that [Detector] cannot express the -[Rule]. If no -[Detector] of the stronger kind can reach it, the stronger kind cannot express the [Rule]: that -is the first ground, not the second. +For the second ground, the language has a [Detector] of the stronger kind when the project +already runs one or the language's own [Detector] ecosystem offers one. A stronger configuration +of a [Detector] the project runs counts; one that only a hosted service or a licence tier +provides does not. A [Detector] accepts bespoke [Rules] when the project can add its own [Rule] +to it through a documented extension point, such as a plugin interface; it need not meet the rest +of the [detector specification](DETECTOR-SPEC.md). If no [Detector] of the stronger kind can reach +the construct the [Rule] is about, that is the first ground. On the [Remediation docs] of a [Rule] they -write, the [Practitioner] MUST name the [Detector] and its kind. Where that kind is weaker than [Semantic], the [Practitioner] MUST also -record, on the [Rule]'s [Remediation docs] and in the description of the commit that adds the -[Rule], moves it to another [Detector], or first records the grounds, for each step down the -order [Semantic], [Syntactic], [Textual], which of the two grounds holds (the stronger kind cannot -express the [Rule], or the language has no [Detector] of the stronger kind that accepts bespoke -[Rules]) and why, naming the [Detectors] of the stronger kind that were considered. The [Practitioner] judges whether the grounds hold, as section 4 gives them the choice of [Detector]; a +write, the [Practitioner] MUST name the [Detector] and its kind. Where that kind is weaker than +[Semantic], the [Practitioner] MUST also record, for each step down the order [Semantic], +[Syntactic], [Textual], which ground holds, that the stronger kind cannot express the [Rule] or +that the language has no [Detector] of the stronger kind that accepts bespoke [Rules], and why, +naming the [Detectors] of the stronger kind that were considered, on the [Rule]'s +[Remediation docs] and in the description of the commit that adds the [Rule], moves it to another +[Detector], or first records the grounds. The [Practitioner] judges whether the grounds hold, as +section 4 gives them the choice of [Detector]; a project MAY require its [Owner]'s sign-off on a weaker kind as well, by recording that requirement where it records its [Calibrations]; the sign-off is then recorded with the grounds, and where the project has recorded no such requirement none is needed. @@ -759,10 +750,11 @@ that invocation covers, and see the [Rule] reported there. A [Toolchain] that wr [Detectors] in its own command is asking for that command to be used, and a [Practitioner] who bypasses it has proven the [Rule] and not the [Defence]. -For a [Bundled defence], the project whose entry point demonstrates it is any project the -[Toolchain] is genuinely installed into, the [Consuming project] included, since that is -where the [Rule] will be enforced. A mechanism gap that stops the [Toolchain] running its own entry -point on its own source moves the demonstration to such a project; it does not weaken it. The +A [Bundled defence] is a [Defence] that a [Toolchain] ships and enables by default in every +project that installs it. The project whose entry point demonstrates it is any project the +[Toolchain] is genuinely installed into, whether the [Toolchain]'s own or somebody else's, since +that is where the [Rule] will be enforced. A mechanism gap that stops the [Toolchain] running its +own entry point on its own source moves the demonstration to such a project; it does not weaken it. The substitutes clause 3.2 allows, for proving under clause 3.3 and for resolution under clause 3.6, do not extend to this clause: the demonstration here is through an entry point or it is not made. A project created for the purpose, with the [Toolchain] installed into it from the source under @@ -795,9 +787,9 @@ something else. The documentation carries the reasoning and the remedy; it is re somebody who has decided to understand the [Rule], and it is maintained as the project's thinking develops. Anything that would grow over time belongs in the documentation. -**A [Rule] and its documentation are one artefact and MUST be versioned as one.** They live in the -same repository and are committed together, or in a library the project depends on at a clearly -tracked version. A [Rule] carries knowledge specific to the project it defends, and its +**A [Rule]'s documentation MUST be versioned with the [Rule].** The two are one artefact: they +live in the same repository and are committed together, or in a library the project depends on at +a clearly tracked version. A [Rule] carries knowledge specific to the project it defends, and its documentation is where most of that knowledge actually sits, so anything that lets the two drift apart destroys the value of both. The same applies to a [Rule]'s [Fixtures] under clause 3.3. @@ -1206,7 +1198,6 @@ This appendix restates sections 2 to 4; where the two differ, the sections gover [conformance]: #conform [conforming]: #conform [conforms]: #conform -[consuming project]: TOOLING-SPEC.md#consuming-project [coverage]: #coverage [dbf]: #dbf [defect]: #defect From d07c177d8b6f0eb63b4de498a3fad11063509872 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 00:17:43 +0100 Subject: [PATCH 045/105] Keys: method 2d and detector 2b and kind quotations follow the reworded 3.2, 7.1 and 8 Quotations only, verbatim to the text at 0b2d190; no question, threshold, fixture or marking rule changed. Co-Authored-By: Claude Opus 5.5 --- acceptance/detector/KEY.md | 14 +++++++------- acceptance/method/KEY.md | 11 +++++------ 2 files changed, 12 insertions(+), 13 deletions(-) diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 0712b1f..a9ee653 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -38,12 +38,12 @@ agent summary (those belong to the toolchain specification). says a declaration is required, or that partial satisfaction is conformance, is wrong. - **2b.** It must make the route either disableable by configuration or detectable by a rule the project can write or a mechanical check it documents; it is not required to remove or forbid - it. Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each route it ships, it MUST do at least one of two things: let - the project switch the route off in the Detector's configuration, or make the route detectable. - Either alone satisfies this clause, and neither requires the Detector to remove or forbid the - route. Detectable means the project can find every use of the route. There are two ways. One is - a Rule the project writes and registers with the Detector, as clause [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) lets it; the - Detector need not ship that Rule. The other is a mechanical check the Detector documents." An answer that says the detector must forbid the route, or must require a + it. Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each route it ships, it MUST do at least one of two things, + and need not remove or forbid the route: let the project switch the route off in the Detector's + configuration, or make the route detectable. Either alone satisfies this clause. A route is + detectable when the project can find every use of it, either with a Rule of the project's own + that the Detector runs, or with a mechanical check the Detector documents. The Detector need not + ship that Rule." An answer that says the detector must forbid the route, or must require a reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD), is wrong. - **2c.** With the rule, in the installed copy, at a version tracked together, and yes, offline. Key quotations, [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access): "Resolution of a bundled rule's identifier MUST work from the installed @@ -73,7 +73,7 @@ has read the clause differently and that counts against the document. The fixture does not say what Lintomatic can see. A reader who names its kind as unknown, or names none, is not marked on it; a reader who makes the verdict turn on its kind has read section [8](../../DETECTOR-SPEC.md#8-conformance) -differently, which counts against the document: "The kind is not a level of Conformance and does +differently, which counts against the document: "the kind is not a level of Conformance and does not change the verdict". Clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation): **does not change the verdict**. It is a SHOULD, and Lintomatic's failure to check its diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index cbafcea..a9dba07 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -57,12 +57,11 @@ is correct; a bullet that says no weaker kind is ever allowed is wrong. reason. Key quotations, [3.2](../../SPEC.md#32-build-the-net): "A Rule the Practitioner writes MUST be written in the strongest kind of Detector that can express it: a Semantic one, else a Syntactic one, else a Textual one."; "Where that kind is weaker than - Semantic, the Practitioner MUST also record, on the Rule's Remediation docs and in the - description of the commit that adds the Rule, moves it to another Detector, or first records the - grounds, for each step down the order Semantic, Syntactic, Textual, which of the two grounds - holds (the stronger kind cannot express the Rule, or the language has no Detector of the - stronger kind that accepts bespoke Rules) and why, naming the Detectors of the stronger kind - that were considered." Supporting: "On the Remediation docs of a Rule + Semantic, the Practitioner MUST also record, for each step down the order Semantic, Syntactic, + Textual, which ground holds, that the stronger kind cannot express the Rule or that the language + has no Detector of the stronger kind that accepts bespoke Rules, and why, naming the Detectors + of the stronger kind that were considered, on the Rule's Remediation docs and in the description + of the commit that adds the Rule, moves it to another Detector, or first records the grounds." Supporting: "On the Remediation docs of a Rule they write, the Practitioner MUST name the Detector and its kind." A correct answer gives three elements, and may leave out everything else above: (i) the strongest kind that can express the rule, in the order semantic, syntactic, textual; (ii) a weaker kind only where the stronger From e783b647943de776019a16afb1c97aac4b3bdb36 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 00:21:45 +0100 Subject: [PATCH 046/105] Method 3.1, 3.2, 3.5 and detector 7.1: the review's readings settled 3.2 again excludes any detector or configuration only a hosted service or licence tier provides. 3.5's bundled defence is one a shipped toolchain enables in projects that install it from somebody else, as the toolchain specification defines it. 3.1 keeps the reason two spellings are one technique. 7.1 says the detector need not remove or forbid the route before the two things it must do, and the detector key's quotation follows. Co-Authored-By: Claude Opus 5.5 --- DETECTOR-SPEC.md | 4 ++-- SPEC.md | 8 ++++---- acceptance/detector/KEY.md | 4 ++-- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index 1681991..a07c271 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -276,8 +276,8 @@ nothing to run. ### 7.1 A detector MAY offer an inline suppression route, but MUST make it detectable or disableable An inline ignore comment, a per-line directive and a generated [Baseline] are all inline -[Suppression] routes, and the [Detector] MAY ship them. For each route it ships, it MUST do at -least one of two things, and need not remove or forbid the route: let the project switch the +[Suppression] routes, and the [Detector] MAY ship them. For each route it ships, it need not remove or forbid the route, but it MUST do at +least one of two things: let the project switch the route off in the [Detector]'s configuration, or make the route detectable. Either alone satisfies this clause. A route is detectable when the project can find every use of it, either with a [Rule] of the project's own that the [Detector] runs, or with a mechanical check the [Detector] diff --git a/SPEC.md b/SPEC.md index 0c373d8..e00a87b 100644 --- a/SPEC.md +++ b/SPEC.md @@ -360,7 +360,7 @@ threshold, and it is the standard clause 3.4 refers back to. Two techniques are independent when they would miss different things. A text search for the token and a reading of the code that consumes the value are independent; two text searches for two -spellings of the same token are one technique; asking someone who knows the system is a third. For a +spellings of the same token are one technique, because they miss the same things; asking someone who knows the system is a third. For a [Class] defined by how code is spelt, searching for each other common spelling of the idiom is a technique. A technique counts only if it does not run the [Detector] under proof in any configuration and would exist unchanged had the [Rule] never been written. So a run of the @@ -419,7 +419,7 @@ cannot express a [Rule] that reads a file the [Semantic] [Detector] does not par For the second ground, the language has a [Detector] of the stronger kind when the project already runs one or the language's own [Detector] ecosystem offers one. A stronger configuration -of a [Detector] the project runs counts; one that only a hosted service or a licence tier +of a [Detector] the project runs counts; a [Detector] or configuration that only a hosted service or a licence tier provides does not. A [Detector] accepts bespoke [Rules] when the project can add its own [Rule] to it through a documented extension point, such as a plugin interface; it need not meet the rest of the [detector specification](DETECTOR-SPEC.md). If no [Detector] of the stronger kind can reach @@ -750,8 +750,8 @@ that invocation covers, and see the [Rule] reported there. A [Toolchain] that wr [Detectors] in its own command is asking for that command to be used, and a [Practitioner] who bypasses it has proven the [Rule] and not the [Defence]. -A [Bundled defence] is a [Defence] that a [Toolchain] ships and enables by default in every -project that installs it. The project whose entry point demonstrates it is any project the +A [Bundled defence] is a [Defence] that a shipped [Toolchain] enables by default in every +project that installs it from somebody else. The project whose entry point demonstrates it is any project the [Toolchain] is genuinely installed into, whether the [Toolchain]'s own or somebody else's, since that is where the [Rule] will be enforced. A mechanism gap that stops the [Toolchain] running its own entry point on its own source moves the demonstration to such a project; it does not weaken it. The diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index a9ee653..43e91f7 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -38,8 +38,8 @@ agent summary (those belong to the toolchain specification). says a declaration is required, or that partial satisfaction is conformance, is wrong. - **2b.** It must make the route either disableable by configuration or detectable by a rule the project can write or a mechanical check it documents; it is not required to remove or forbid - it. Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each route it ships, it MUST do at least one of two things, - and need not remove or forbid the route: let the project switch the route off in the Detector's + it. Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each route it ships, it need not remove or forbid the route, + but it MUST do at least one of two things: let the project switch the route off in the Detector's configuration, or make the route detectable. Either alone satisfies this clause. A route is detectable when the project can find every use of it, either with a Rule of the project's own that the Detector runs, or with a mechanical check the Detector documents. The Detector need not From 1db0c670f27fe19fe39c6d4a4be03263551ad90f Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 00:21:45 +0100 Subject: [PATCH 047/105] Acceptance runs 25 and 26: SPEC.md and DETECTOR-SPEC.md, fourth cohort, both FAIL Five cold Haiku readers each, made under pull request 13 at 598908b and marked by a delegated marker. Evidence is posted on pull request 14 and linked from each record, with the findings applied to the text. Co-Authored-By: Claude Opus 5.5 --- .../runs/025-semantic-detectors-run-7.md | 269 ++++++++++++++++++ .../runs/026-semantic-detectors-run-8.md | 243 ++++++++++++++++ 2 files changed, 512 insertions(+) create mode 100644 acceptance/runs/025-semantic-detectors-run-7.md create mode 100644 acceptance/runs/026-semantic-detectors-run-8.md diff --git a/acceptance/runs/025-semantic-detectors-run-7.md b/acceptance/runs/025-semantic-detectors-run-7.md new file mode 100644 index 0000000..95ac89f --- /dev/null +++ b/acceptance/runs/025-semantic-detectors-run-7.md @@ -0,0 +1,269 @@ +# Run 25: SPEC.md, semantic detectors, seventh run in pull request 14 + +- **Document**: SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + branch `editor/semantic-detectors` at dfb9b68, the commit the readers were given; the question + set, key and fixture read at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input; no tools, no settings, no project context and no + saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is byte for byte the question set, the document and the + fixture at dfb9b68, in that order, each preceded by one `===== FILE: =====` line, with + nothing else added apart from one blank line between the end of each file and the next file's + marker line. +- **Managed (policy) settings**: none; the machine applies no managed settings to the readers and + no managed settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. Every sheet has answer text, no process reports an error + (`is_error` false, subtype `success`), and every reported model is the cohort's model. The + editor reports no reader set aside for a cause the sheets cannot show. +- **Run within the pull request**: 7, counting void runs; runs in the pull request so far, + counting void runs: 7, as the editor supplied. Run 26 is made at the same time and does not bear + on this document. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review. It read `ACCEPTANCE.md` and + `acceptance/runs/README.md` in this worktree, which differ from commit 598908b only by this pull + request's question-set line adding 2d for SPEC.md; the document, question set, key and fixture + at dfb9b68; runs 21 and 23 in full, run 19's verdict, and the headers of runs 20, 22 and 24, + which are on DETECTOR-SPEC.md; and, for the fourth pass-criteria row, the change to SPEC.md + between 0b8d699, the document run 23 read, and dfb9b68. + +## Marks + +`ok` is correct, `miss` is a miss; `T` marks a miss traceable to the text. Judgements J1 to J8 +are set out below. + +| Reader | Q1 | 2a | 2b | 2c | 2d | 3.1 | 3.2 | 3.3 | 3.4 | 3.5 | 3.6 | Remediation | Defence | Section 4 | Q4 passages | +| ------ | ------- | --- | --- | --- | ------ | --- | ------- | ------ | --- | --- | -------------- | ----------- | ------- | --------- | ----------- | +| 1 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 2 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | Partial miss T | ok | ok | ok | three | +| 3 | correct | ok | ok | ok | ok | No | Partial | Yes ok | No | No | Partial miss T | ok | ok | ok | two | +| 4 | correct | ok | ok | ok | ok | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 5 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | four | + +Every grade under 3.1, 3.2, 3.4 and 3.5, and every 3.6 grade of No, is one the key accepts; every +3.1 grade of No notes that no independent search technique is recorded. No reader grades 3.2, 3.5 +or 3.6 Yes, so none of the key's cases that count against the document arises. The two 3.6 grades +of Partial are marked under J5. + +### Question 1 + +Five of five correct. Every sheet has at least four bullets naming obligations the key lists (the +class bounded both ways with its hazard, the strongest kind of detector and never a test, the +proof in a commit of its own, the sweep with its count and the fix of every instance, permanent +and blocking, the terse message with an identifier resolving to versioned documentation, the +record of a deferred defect), and no bullet asserts an obligation the document does not state. +Reader 5's "strongest kind of detector", naming no fallback, is correct as the key says of a +bullet naming the strongest kind. The closing "fix the original defect with a reproduction test" +of readers 2 and 5 is section 2 and section 6. + +### Question 2 + +Every reader quoted words of the document for every item, so every miss here is traceable to the +text. + +- **2a**, five of five. Each states class, rule, proof, sweep and the fix of every instance, in + that order, and quotes section 2's sentence "The order is section 3's: attribute the [Defect] to + a [Class], build the [Rule], prove that it fires, run the [Sweep], then fix every [Instance]." +- **2b**, five of five. Each says only the owner may and an agent may not, and quotes the key's + section 4 sentence. Readers 1, 3, 4 and 5 say the owner is a human; reader 2 says "Only the + Owner" (J4). +- **2c**, five of five. Each states all six clauses and a verdict resting on reproduction, not on + the report, in its sentence or its quotation of section 7; readers 1, 4 and 5 quote the red and + green reruns. No reader accepts the pull request description as the verdict. +- **2d**, two of five (readers 3 and 4). The key's three elements are (i) the strongest kind, in + the order semantic, syntactic, textual; (ii) both grounds for a weaker kind; (iii) the reason + recorded, with at least one place named and the places not given as alternatives (J2). + - Reader 1: (i) in its first quotation; (iii) stated, its second quotation giving the + remediation docs "and" the commit description. (ii) missed: "A weaker kind is allowed only on + two grounds" says there are two and states neither, and its quotation of the record sentence, + "... which of the two grounds holds ... and why", elides the parenthesis that names them (J6). + - Reader 2: as reader 1. (i) and (iii) stated; (ii) missed: "only on one of two grounds", with + the same record quotation eliding the grounds (J6). + - Reader 3: (i) in its sentence and quotation, (ii) in its quotation of the "two grounds only" + sentence, (iii) with both places joined by "and". Correct. + - Reader 4: (i), (ii) and (iii) in its three quotations, the places joined by "and". Correct. + - Reader 5: (ii) stated in its quotation of the "two grounds only" sentence. (i) missed: "the + strongest kind of detector that can express it", and no quotation names the order semantic, + syntactic, textual (J3). (iii) missed: its record quotation, "MUST also record, ... which of + the two grounds holds ... and why", elides the places, and its sentence names none (J2, J6). + +### Question 3 + +- **3.1**: five of five No, each noting that no two independent search techniques are recorded. +- **3.2**: five of five Partial, each saying the rule reads code and is not a test, and each that + the detector or its kind is not named. Readers 2, 3 and 5 add that no grounds for a weaker kind + are recorded, which the key does not mark wrong. +- **3.3**: five of five Yes. Readers 2, 3, 4 and 5 say the inline ignore is a [Suppression] judged + under 3.4 and 3.5, not here; none attributes it to 3.3. +- **3.4** and **3.5**: five of five No, each naming the inline ignore on `Report::render()`, under + 3.5 as uncovered by any recorded decision. +- **3.6**, three of five (readers 1, 4 and 5, No). Readers 2 and 3 grade Partial, which the key + accepts only "where the reasoning quotes the versioning sentence" (J5). + - Reader 2: "But it sits on a company wiki, not shipped or versioned with the rule." This + paraphrases the requirement and quotes nothing. The reasoning states what the document + requires, so the miss is traceable; reader 2 also names clause 3.6's external URL sentence + under question 4. + - Reader 3: 'they are not "shipped with the project" or "versioned alongside the [Rule]" as the + clause requires.' The first is verbatim from 3.6's opening sentence, which requires shipping, + not versioning; the second is not the document's words, which are "versions it alongside the + [Rule]" in 3.6 and "versioned alongside it" in section 7. The reasoning quotes the document, + so the miss is traceable. +- **Remediation**: five of five say it does not conform, each naming failing clauses the key + accepts. +- **Defence**: five of five say it does not conform, each naming 3.6 as not met and readers 2, 3 + and 4 naming 3.2 as well. Readers 1, 3, 4 and 5 also count 3.5 against the defence, on the + uncovered suppression; the key does not mark that wrong. +- **Section 4**: five of five name the inline ignore on `Report::render()`, and each also names + leaving the next wider rule unbuilt, which the key accepts alongside it. + +### Judgements made in marking + +Made as a careful reader of the document would, as `ACCEPTANCE.md` requires. Where a rule settles +the point, the record says which. + +- **J1. The sentence.** Where a reader wrote more than one sentence before or around its + quotations, all of that text is read with the quotations as the answer. No mark turns on it. +- **J2. 2d's element (iii) needs a place named.** The key says of (iii) that "naming one of the + two places is enough" and that presenting them as alternatives is wrong; "enough" sets a + minimum, and a key that let an answer name no place would have no minimum to state and no + alternatives to forbid. Runs 19, 21 and 23 read it the same way. Reader 5's (iii) turns on it; + reader 5 misses (i) as well (J3). +- **J3. 2d's element (i) needs the order (reader 5).** The key's element is "the strongest kind + that can express the rule, in the order semantic, syntactic, textual", and the question asks + which kind. "The strongest kind of detector that can express it" names no kind, and "weaker than + [Semantic]" in its record quotation names the top of the order, not the order. Run 23 read it + the same way. Read otherwise under both J2 and J3, reader 5 would be correct on 2d; 2d would + still fail on readers 1 and 2, whose misses on (ii) rest on no judgement. +- **J4. "The Owner" (reader 2, 2b).** Settled by the rule that a defined term used in its defined + sense states what it means: the [Owner] "is always a human". +- **J5. 3.6 Partial and "the versioning sentence" (readers 2 and 3).** The key accepts Partial + "where the reasoning quotes the versioning sentence", and `ACCEPTANCE.md` defines quoting: the + document's consecutive words, verbatim apart from letter case, quotation-mark style and markup; + a paraphrase is not quoting. Which sentence is the versioning sentence the key leaves open. It + is taken in the wider sense, either of 3.6's sentences requiring the documentation to be + versioned with the rule: "A [Rule] and its documentation are one artefact and MUST be versioned + as one." and "An external URL [Conforms] only where the project controls it and versions it + alongside the [Rule].", the second because the key's case of a reader who grades Yes "because + the URL resolves" is a misreading of it. Neither reader quotes either sentence, so both miss on + 3.6, and both reasonings state or quote what the document requires, so both misses are + traceable. Read leniently, taking reader 3's "versioned alongside the [Rule]" as a quotation of + the URL sentence despite the changed word, reader 3 would be correct, 3.6 would be four of five + and the second criterion met; reader 2's traceable miss would stand, and the verdict would not + change. +- **J6. Elided quotations.** A quotation with an ellipsis is read as one quotation per unbroken + run of words, each in the document's order, and states only what those runs state (readers 1, + 2 and 5 under 2d; reader 4's section 8 passage). Settled by the rule on quoting: consecutive + words of any length. +- **J7. Clause labels.** A clause number given with a quotation, as question 4 asks, is read as + the quotation's label, not as naming the whole clause, as runs 21 and 23 read it. Reader 1's + "clause 3.3, Part B" label does not extend its passage to all of Part B. +- **J8. Passages set beside the main one count.** Where a reader says it was confused by two + passages read together, a careful reader takes both as the confusing passage, since the + confusion lies in neither alone, as run 23 read it. So reader 2's first two items each cover + both passages they set side by side, reader 4's first item covers all three of its quotations, + and reader 5's fourth item covers both its quotations. Reader 5's first item quotes section 2 + and describes clause 3.1 as defining "independence for searches", not knowing whether that test + applies to techniques for writing a rule; the confusion lies in how the two read together, so + the description is read as naming clause 3.1's definition, the sentence "Two techniques are + independent when they would miss different things." A finding turns on this: read as context + only, clause 3.1's paragraph beginning with that sentence would be reader 2's alone. The verdict + would not change. + +## Passages quoted as confusing + +Same passage as `ACCEPTANCE.md` defines it: sentences in common, or the same paragraph; a list +item, table row or terminology entry is its own paragraph. + +| Passage | Clause | Readers | Words quoted, verbatim, or the description used | +| ----------------------------------------------------------------- | ----------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Which ground holds when extension points cannot reach a construct | 3.2 | four: 1, 3, 4, 5 | All four: "If a [Detector] passes both but its extension points cannot reach the construct a [Rule] is about, that [Detector] cannot express the [Rule]. If no [Detector] of the stronger kind can reach it, the stronger kind cannot express the \[Rule\]: that is the first ground, not the second." The last two sentences of the paragraph beginning "The second ground asks two questions". Run 23's open passage (below) | +| The entry point for a bundled defence | 3.5 | two: 3, 5 | Both: "For a [Bundled defence], the project whose entry point demonstrates it is any project the [Toolchain] is genuinely installed into, the [Consuming project] included", reader 3 to "since that is where the [Rule] will be enforced.", reader 5 ending in an ellipsis; both add that "Bundled defence" and "Consuming project" are not defined in SPEC.md | +| Two techniques are independent; a rule run is not one of them | 3.1 | two: 2, 5 | 2: "a clean run of the [Rule]", "not evidence of saturation" and "is never one of the two", set beside clause 3.3's "A single execution of the [Rule] answers both" (J8). 5: by description, "Clause 3.1 defines independence for searches", set beside section 2 (J8). All in the paragraph beginning "Two techniques are independent when they would miss different things." | +| Narrowings reported to the owner as if suppressions | 3.3, Part B | one: 1 | "a [Narrowing] that excludes more code than the [Rule] still covers MUST be reported to the [Owner] as if it were a [Suppression]." (J7) | +| What a defence must satisfy of clause 3.5 | 3.5 and 7 | one: 2 | Section 7's "it fails rather than warns", in the paragraph beginning "A [Defence] [Conforms]", and clause 3.5's "does not [Conform]", in its opening paragraph (J8) | +| Proving and sweeping in one run | 3.3, Part A | one: 2 | "A single execution of the [Rule] answers both", set beside clause 3.1 (above, J8) | +| An external URL | 3.6 | one: 2 | "An external URL [Conforms] only where the project controls it and versions it alongside the [Rule]." | +| Check order and conformance | 5, 7 and 8, Scope | one: 4 | 5: "a project whose checks run in another order is a project that does not [Conform]". 7: "Nothing here constrains how the project runs its checks or what it does when they fail". 8: "How a project chooses to run its quality checks" and "is the project's own business and no part of this specification." (J6, J8) | +| Independent techniques for expressing a rule | 2 | one: 5 | "at least two independent techniques tried at expressing the [Rule]" and "Two techniques are independent when they would miss different things, as clause 3.1 says of a search.", in the "Attempt rather than pre-judge" paragraph; set beside clause 3.1 (above, J8) | +| A toolchain gap and the toolchain's claim | 3.6 and 3.2 | one: 5 | 3.6: "it is a [Toolchain] gap and blocks the [Toolchain]'s claim under section 7 until fixed." 3.2: "is recorded against the [Toolchain]'s claim under section 7 as well" (J8) | + +Readers' accounts of what they could not tell are recorded with the passage they attach to and are +not suggestions; none of the sheets proposes a change. + +**Earlier passages (the fourth row).** The open passages are those earlier runs of this document +found confusing by two or more and did not clear: run 23's three, and run 21's clause 3.2 passage, +which run 23 did not clear (run 19's other three and run 21's other three were cleared in runs 21 +and 23). As the paragraphs read in the changed document: + +- Clause 3.2's two grounds, run 21's passage, which run 23 read as the grounds sentence of the + paragraph beginning "**Qualifying is not the same as being the right host.**", the paragraph + then beginning "The language has a [Detector] of the stronger kind", and the record paragraph + beginning "On the [Remediation docs] of a [Rule] they write". The second of those now begins + "The second ground asks two questions"; readers 1, 3, 4 and 5 name its last two sentences. Not + cleared. +- Clause 3.1's lower-bound list item: named by no reader. Reader 2's and reader 5's clause 3.1 + passage lies in the paragraph beginning "Two techniques are independent", not the list item. + Cleared. +- The Owner entry in section 1: named by no reader. Cleared. +- The Narrowing entry in section 1: named by no reader. Reader 1's narrowing passage is clause + 3.3 Part B's sentence, which does not occur in the entry. Cleared. + +No sentence kept from one of the three cleared paragraphs is named elsewhere in the document. +Reader 5's section 2 passage lies in the "Attempt rather than pre-judge" paragraph that run 19 +named and run 21 cleared; it is not an open passage and is recorded above as one reader's. Reader +3 and reader 5's clause 3.5 passage was named by one reader in run 23 and is a new finding here. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in 0b2d190 and the commit after +it that settles its review (key quotations kept verbatim in d07c177 and that commit). SPEC.md: +clause 3.2's record sentence states both grounds in its main clause, and its paragraph on the second +ground is four short sentences; clause 3.5 says what a bundled defence is in plain words; clause +3.1's paragraph on independent techniques is shorter; clause 3.6's versioning requirement is one +short sentence. A fresh cohort follows. + +## Process change + +Made under a process change before it merged: yes, pull request 13 at commit 598908b, named +before the readers started. + +## Verdict + +**FAIL.** Against the criteria in `ACCEPTANCE.md`: + +- Load-bearing statements: not met. 2a, 2b and 2c five of five; 2d two of five. Readers 1 and 2 + state neither ground for a weaker kind, only that there are two; reader 5 does not give the + order semantic, syntactic, textual (J3) and names no place for the record (J2). +- Restatement and application: not met on 3.6, three of five (readers 2 and 3 grade Partial + without quoting the versioning sentence, J5); question 1 and every other keyed item five of + five. +- Misses traceable to the text, each a finding: 2d for readers 1, 2 and 5, each quoting clause + 3.2's record sentence "Where that kind is weaker than [Semantic], the [Practitioner] MUST also + record, on the [Rule]'s [Remediation docs] and in the description of the commit that adds the + [Rule], moves it to another [Detector], or first records the grounds, for each step down the + order [Semantic], [Syntactic], [Textual], which of the two grounds holds (the stronger kind + cannot express the [Rule], or the language has no [Detector] of the stronger kind that accepts + bespoke [Rules]) and why, naming the [Detectors] of the stronger kind that were considered." + with the grounds (readers 1 and 2) or the places (reader 5) elided; 3.6 for readers 2 and 3, + on clause 3.6's requirement that the documentation be shipped and versioned with the rule. +- Passages found confusing by two or more, each a finding that blocks until the text is changed + and a fresh cohort no longer names it: clause 3.2's paragraph beginning "The second ground asks + two questions", its last two sentences (four readers), which is also run 21's open passage, not + cleared; clause 3.5's sentence on the entry point for a [Bundled defence] (two); clause 3.1's + paragraph beginning "Two techniques are independent when they would miss different things." + (two, on J8). Run 23's three passages are cleared. +- Single-reader notes, recorded: clause 3.3 Part B's sentence on narrowings reported to the owner + (reader 1); clause 3.5's opening paragraph against section 7's defence paragraph, clause 3.3 + Part A's proving-and-sweeping paragraph, and clause 3.6's external URL sentence (reader 2); + sections 5, 7 and 8 on check order and conformance (reader 4); section 2's "Attempt rather than + pre-judge" paragraph, and clause 3.6's unresolved identifier paragraph against clause 3.2's + paragraph on the two kinds of gap (reader 5). + +Answer sheets, the reader prompt, the command and the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090845841), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090846386), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090846766), [comment 4](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090847033). diff --git a/acceptance/runs/026-semantic-detectors-run-8.md b/acceptance/runs/026-semantic-detectors-run-8.md new file mode 100644 index 0000000..85b7515 --- /dev/null +++ b/acceptance/runs/026-semantic-detectors-run-8.md @@ -0,0 +1,243 @@ +# Run 26: DETECTOR-SPEC.md, semantic detectors, eighth run in pull request 14 + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + editor's branch `editor/semantic-detectors` at dfb9b68; the question set, key and fixture read + at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input, with no tools, no settings, no project context + and no saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is byte for byte the question set, the document and the + fixture at dfb9b68, in that order, each preceded by one `===== FILE: =====` line, with the + files separated by one blank line and nothing else added, the same form as the packets of runs + 22 and 24 +- **Managed (policy) settings**: none; the machine applies no managed settings and no managed + settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. The editor reports none set aside for a cause the sheets cannot + show; every sheet has answer text, no process reports an error (`is_error` false, subtype + `success`), and every reported model is the cohort's model +- **Run within the pull request**: 8, counting void runs; run 25, made at the same time, does not + bear on this document +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review + +## Marks + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | Section 8 | Kind | 6.4 | Q4 entries | +| ------ | ------- | --- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --------- | ---------- | --- | ---------- | +| 1 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 2 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 3 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 4 | miss | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 5 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | + +"Q4 entries" counts the entries each reader lists under question 4; the words each quotes are in +the table of passages below. Every miss under question 2 is traceable to the text: each answer +quotes the document in support. The miss under question 1 is the reader's own: the bullet it turns +on quotes nothing. No question was left unanswered. + +### Question 1 + +Every restatement names four or more obligations from the key's list. Readers 1, 2, 3 and 5 assert +no obligation the document does not state: correct. Reader 3's last bullet, that "a verdict must +name the rule kind (Semantic, Syntactic, or Textual)", is the obligation the key marks correct on +a verdict (judgement 3). Readers 2 and 5 say findings must appear in, or be put in, the command's +own output, without "each" or "every" (judgement 2). Reader 4's sixth bullet says "The SHOULDs add +a reason requirement for suppressions, a release check for undocumented bundled rules, and a +conformance declaration that records known gaps"; the declaration is a MAY (8.1's heading, and "a +declaration is optional"), so the bullet asserts a recommendation the document does not make: +incorrect (judgement 4). Correct, four of five. + +### Question 2 + +The key's answers state these elements: + +- 2a: (a) Conformance is satisfying every MUST in sections 4 to 7; (b) judged on evidence; (c) a + detector that predates the document can conform; (d) the declaration is optional. +- 2b: (a) make the route disableable by configuration, or (b) detectable by a Rule the project can + write or a mechanical check the Detector documents; (c) the detector is not required to remove or + forbid it. +- 2c: (a) with the rule; (b) in the installed copy; (c) at a version tracked together; (d) yes, + reachable offline. + +Each answer is read as its sentence and its quotation together, for the gist of each element. + +| Reader | 2a | 2b | 2c | +| ------ | --------------------------------------------------------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------- | +| 1 | ok: all four; (b) and (d) from the 8.1 quotation | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6) | ok: all four; (a) and (c) from the 6.3 quotation | +| 2 | ok: all four; (b) and (d) from the 8.1 quotation | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6) | ok: all four; (b) from the 6.2 quotation, (c) from the 6.3 quotation | +| 3 | ok: all four; (d) from the 8.1 quotation | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6) | ok: all four; (a) and (c) from the 6.3 quotation | +| 4 | ok: all four; (d) from the section 8 and 8.1 quotations | miss: (c) not stated (judgement 5) | ok: all four | +| 5 | ok: all four, in the sentence, which says "a declaration is optional" | miss: (c) not stated (judgement 5) | ok: all four; (b) from the 6.2 quotation | + +No answer asserts anything the document does not state or that the key says makes an answer +wrong: none says a declaration is required, that partial satisfaction is conformance, that the +detector must forbid the route or require a reason, or that a website will do. Every quotation is +verbatim apart from letter case, link and emphasis markup and a closing punctuation mark; the 2c +quotations of readers 1, 2, 3, 4 and 5 are the headings of 6.2 and 6.3. + +### Question 3 + +The key's intended verdicts are Yes for every clause but 6.3, which is No; the section 8 verdict +is "does not conform", because 6.3 fails; 6.4 does not change the verdict. + +- **Every clause**: all five readers give the intended verdict on every clause, 4.1 to 7.1. On + 6.3 each cites that one unresolved identifier fails the clause, or quotes 6.3's "One + [Identifier] that does not resolve fails this clause, however many others do."; no reader grades + it Partial on the grounds that thirty-eight resolve. Readers 1, 2, 4 and 5 add caveats on 4.2 or + 4.3 about what the fixture does not say, and grade Yes. +- **Section 8**: all five say Lintomatic does not conform, because 6.3 fails. +- **Kind**: all five say the fixture does not let them name the kind and name none, so none is + marked on it. Readers 1, 3 and 5 call the verdict incomplete on that point and reader 2 says a + complete verdict would need it; none makes the verdict turn on it, and readers 3 and 5 say + outright that it does not change the verdict. +- **6.4**: all five say it does not change the verdict, because it is a SHOULD. Readers 3, 4 and 5 + add asides on what a declaration under 8.1 would record (judgement 7). + +### Judgements made in marking + +01. **How an open judgement is made.** As `ACCEPTANCE.md` at 598908b says: as a careful reader of + the document would, with the reason stated. Where a rule settles a mark, it is applied as + written. +02. **Question 1, "Findings" in the output, readers 2 and 5.** Reader 2 says "Findings must appear + in the output of the command the practitioner ran"; reader 5 that the detector must "put + findings in the command's own output". Run 24 marked incorrect a bullet saying "each finding + must appear", because 5.3 says "The summary need not list every finding". A plain plural + without "each" or "every" says what 5.3's heading says, "The result MUST reach the + practitioner in the output of the command they ran", and a careful reader takes it as that + heading, not as a claim about every finding. Correct. +03. **Question 1, reader 3's "rule kind".** Section 8 puts the naming of the kind on the verdict, + for each configuration graded; the terminology table defines the three kinds as "What a [Rule] + can ask in a configuration". The key marks correct a bullet saying a verdict must state the + detector's kind. "Rule kind" names the same three kinds by what a rule can ask, which the + document's own definition does, and a careful reader takes the bullet as the key's obligation + on the verdict. Correct. +04. **Question 1, reader 4's "SHOULDs".** The document defines its key words under RFC 2119, where + SHOULD recommends and MAY permits, and it says in 8.1 and section 8 that a declaration is + optional. A careful reader of a document that draws that line takes "The SHOULDs add ... a + conformance declaration" as stating a recommendation to declare that the document does not + make, and so an obligation the document does not state. Read the other way, as a loose list of + the non-MUST provisions, the bullet would be correct; Question 1 would then be five of five, and + neither reading changes whether the second criterion holds or adds a traceable miss, since the + bullet quotes nothing. +05. **2b (c), not required to remove or forbid, all five.** The question asks what the detector + must do about the route; the key's answer includes what it need not do, so (c) answers a part + of the question. Readers 1, 2 and 3 say the detector must do one of the two things; readers 4 + and 5 add "Either one is enough". That entails that nothing more is required, but it does not + say that removing or forbidding the route is not required: "Either one is enough" paraphrases + the first half of 7.1's sentence "Either alone satisfies this clause, and neither requires the + [Detector] to remove or forbid the route", not its second half, and no reader quotes that half + or 7.1's first sentence, "The [Detector] does not have to remove or forbid an inline + [Suppression] route." `ACCEPTANCE.md` names the ways an element is stated: the document's words, + a paraphrase, a synonym, a defined term in its defined sense. An element reached only by + inference from another is not among them, and a careful reader of that rule does not add it. + Marked not stated, as runs 22 and 24 marked the same element. The verdict does not turn on this + judgement alone: the fourth criterion fails whichever way it is made. +06. **2b (b), the means of detection.** Clause 7.1 defines "Detectable" in its own text. Readers 1, + 2 and 3 use "detectable" in that sense, and so state (b), as runs 22 and 24 held; reader 5's + "make every use of the route detectable" uses the definition's own words. It decides no mark, + since every 2b answer misses (c). +07. **Asides outside the keyed answer.** Reader 3's statement that 6.4 would not appear in + `known-gaps`, and readers 4's and 5's that a declaration would record the 6.3 gap, are not keyed + items; each sheet's answer to whether 6.4 changes the verdict is the key's, and is marked so. +08. **Question 4, a clause number inside a quotation, and context in the reader's explanation.** + Reader 4's quotation contains "as clause 4.1 lets it", and its explanation speaks of "a rule + written under 4.1"; reader 1's explanation of its section 8 entry speaks of "section 8's + conformance test"; reader 3's quotation contains "a [Detector] failing 4.1". Each reader heads + its entry with the clause it found confusing (7.1, 8 and 4.2) and quotes the words; a careful + reader takes the other clause as the context of the confusion, not a further passage named. + Read the other way, each would add a passage named by one reader only, in a paragraph no + earlier passage of this document left open (clause 4.1 was cleared by run 22, as run 24 + records), and nothing that blocks. +09. **Question 4, reader 5's quoted terms.** Reader 5 heads its entry "Section 2, the terminology + paragraph" and quotes "Every capitalised term links to its definition," and then the terms + "[Semantic], [Syntactic] or [Textual]" and "[Conform]" as the ones it found defined only by + links. `ACCEPTANCE.md` says a quotation covers what an occurrence of its words overlaps "at the + place the reader names it", and reaches every place the run occurs only "where the reader names + no place". Reader 5 names section 2's terminology paragraph for the whole entry, and neither + term's run occurs there (the terminology table, a separate paragraph, reads "[Semantic], + [Syntactic], [Textual]"). A careful reader takes the entry as the section 2 sentence, with the + terms as the examples the reader gives of it, so the term quotations cover no further sentence. + Read the other way, "[Semantic], [Syntactic] or [Textual]" would cover the first sentence of + section 8's paragraph on kind, and "[Conform]" the sentences of 5.4, 7.1's first paragraph, + section 8's paragraph on kind, 8.1's Why paragraph and section 9 where that word occurs; that + would make reader 5 a second reader of section 8's paragraph on kind and of 7.1's first + paragraph, both of which block already, and the verdict does not turn on it. +10. **Question 4, a reader that resolved its confusion.** Reader 2 says of its 6.1 and 6.3 passage + "I resolved this with 6.1's own pointer". Question 4 asks for any passage found confusing on + the one reading, and the reader names it as such "On my first read", so it is recorded. +11. **Question 4, closing punctuation.** Reader 1's "... it grades." ends with a full stop where + the document has a comma, and reader 5's "... to its definition," and reader 2's "... not for + every page," end with a comma where the document has a semicolon or a full stop. The words are + consecutive and verbatim; the punctuation mark that follows them is not a word, so each is a + quotation. +12. **The open passages.** The passages earlier runs of this document found confusing by two or + more, and not since cleared, are those run 24 left outstanding or added: 7.1's first paragraph + (runs 20, 22 and 24), 8.1's second paragraph (runs 20, 22 and 24) and section 8's paragraph + beginning "A verdict on a [Detector] MUST also name" (run 24, four readers). The text of each + changed after run 24, in b172fd6, and each is taken as it reads at dfb9b68. + +No re-mark. + +## Passages quoted as confusing + +Grouped by paragraph, as `ACCEPTANCE.md` defines the same passage. A passage blocks when two or +more readers name it, or when it covers a sentence of a paragraph in which one of the open +passages lay (judgement 12), as that paragraph reads at dfb9b68. + +| Clause and paragraph | Readers | Words quoted, verbatim | Blocks | +| ---------------------------------------------------------------------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | +| 8, the paragraph beginning "A verdict on a [Detector] MUST also name", its first sentence | 1 | "A verdict on a [Detector] MUST also name the kind of each configuration it grades." (judgement 11) | Yes: the open section 8 paragraph of run 24 | +| 7.1, first paragraph, the sentences "Detectable means ...", "There are two ways." and "One is a [Rule] ..." | 4 | "Detectable means the project can find every use of the route. There are two ways. One is a [Rule] the project writes and registers with the [Detector], as clause 4.1 lets it." (judgement 8) | Yes: the open 7.1 paragraph of runs 20, 22 and 24 | +| 4.3, first paragraph, the sentence beginning "The [Detector] MUST print it, unaltered" | 1 | "in its default output and in every machine-readable format it offers." | No | +| 4.2, first paragraph, its second sentence | 3 | "This applies to every [Rule] the [Detector] runs, bundled or bespoke; a [Detector] failing 4.1 is graded here on its [Bundled rules]." (judgement 8) | No; one reader of run 24 also named it | +| 6.1, the paragraph beginning "In both", the sentences "This clause asks for the mechanism ..." and "Where the mechanism works ..." | 2 | "This clause asks for the mechanism, not for every page," and "Where the mechanism works but a [Bundled rule]'s [Identifier] has no documentation to resolve to, that fails clause 6.3, not this clause." (judgements 10 and 11) | No; one reader of run 22 also named the second sentence | +| 6.3, first paragraph, its first sentence | 2 | "Every [Identifier] a [Bundled rule] can print MUST resolve, under clause 6.1, to a page in that shipped documentation." (judgement 10) | No | +| 2, first paragraph, its second sentence | 5 | "Every capitalised term links to its definition," and the terms "[Semantic], [Syntactic] or [Textual]" and "[Conform]" (judgements 9 and 11) | No | + +No passage is named by two or more readers of this cohort, on the reading of judgement 9. Of the +open passages, 8.1's second paragraph is cleared: its text changed after run 24 and no reader of +this cohort names a passage covering a sentence of it, on either reading of judgement 9. 7.1's +first paragraph and section 8's paragraph on kind are still outstanding. + +## Against the criteria + +- **Load-bearing statements (every reader on every one)**: not met. 2a five of five, 2b none of + five, 2c five of five. +- **Restatement and application (all but one on each keyed item)**: met. Question 1 four of five; + every clause of question 3, the section 8 verdict and 6.4 five of five. +- **A miss traceable to the text (none)**: not met. Five, each a finding: 2b by all five readers, + each quoting 7.1's sentence "For each route it ships, it MUST do at least one of two things: ...", + none stating that the detector need not remove or forbid the route. +- **A passage found confusing by two or more (none outstanding)**: not met. The open passages in + 7.1's first paragraph (reader 4) and section 8's paragraph on kind (reader 1) are still named by + this cohort. +- **From one reader only**: the 4.3 sentence on machine-readable formats, the 4.2 sentence on a + detector failing 4.1, the 6.1 sentences on the mechanism and a missing page, the 6.3 sentence on + every identifier resolving, and the section 2 sentence on capitalised terms, recorded. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in 0b2d190 and the commit after +it that settles its review (key quotations kept verbatim in d07c177 and that commit). DETECTOR- +SPEC.md: clause 7.1's MUST sentence says the detector need not remove or forbid the route, and its +first paragraph is plainer; section 8's paragraph on the kind is shorter. A fresh cohort follows. + +## Process change + +Made under the process change of pull request 13 before it merged, at commit 598908b, named +before the readers started; `ACCEPTANCE.md` and `acceptance/runs/README.md` in this worktree +differ from 598908b only by this pull request's question-set line adding question 2d for SPEC.md, +which does not bear on this document. + +## Verdict + +FAIL on the first, third and fourth criteria. Answer sheets, the readers' prompt and command, and +the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090847649), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090848005), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090848260). From c6ee1a860087ec197ea112d7330d468258a23bd7 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 00:33:36 +0100 Subject: [PATCH 048/105] Method 2, 3.1, 3.2 and 7, detector 6.1, 7.1 and 8: runs 27 and 28 findings applied Clarity edits, no obligation changed in meaning. Clause 3.2 states both grounds for a weaker kind straight after the strongest-kind sentence and names the two places for the record, both required, before what is recorded. Section 7's conforming defence satisfies each clause in full and its summary of 3.5 carries the recorded decision behind every suppression. Section 2 and clause 3.1 each say what a technique is there; 3.1's independence paragraph is split in three. Detector 7.1 says after the two things that neither requires removing or forbidding the route and says in its own paragraph what detectable means; 6.1 says printing the identifier unaltered is all it asks for a project's own rule; section 8 says what a configuration is and what a verdict leaving out the kind is left with. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 20 +++++++++++-- DETECTOR-SPEC.md | 56 ++++++++++++++++++++--------------- SPEC.md | 76 ++++++++++++++++++++++++++---------------------- 3 files changed, 92 insertions(+), 60 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 39d84ff..c6f0876 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -70,7 +70,14 @@ version entry with no cohort line has not been accepted and is not published. undefined term consuming project; clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s paragraph on independent techniques is shorter and says plainly that a run of the rule is never one; and clause [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation)'s versioning sentence says a rule's documentation is versioned with the rule. Each after readers of run 25 missed or - named the passage. Accepted by run NNN. Minor version when released. + named the passage. After run 27, clarity edits: clause [3.2](SPEC.md#32-build-the-net) states both grounds for a weaker + kind in the sentence straight after the strongest-kind sentence, and its record paragraph names + the two places, both required, before what is recorded and which commit carries it; section + [7](SPEC.md#7-conformance)'s conforming defence satisfies each clause in full, and its summary of clause [3.5](SPEC.md#35-enforce-permanently-and-block) includes + permanence and the recorded decision behind every suppression; section [2](SPEC.md#2-when-the-method-applies) and clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class) each say + what a technique is there, and clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s paragraph on independent techniques is three short + paragraphs, saying that a run of the rule belongs to the proof or the sweep. Each after readers + of run 27 missed or named the passage. Accepted by run NNN. Minor version when released. ### 1.1.0, 2026-10-02 @@ -166,7 +173,16 @@ pass it. remove or forbid the route, and its definition of detectable is one sentence; section [8](DETECTOR-SPEC.md#8-conformance)'s paragraph on the kind is shorter, says what the kind is, and says the duty is on whoever gives the verdict, not on the detector. Each after readers of run 26 missed or named the passage. - Accepted by run NNN. Minor version when released. + After run 28, clarity edits: clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) says after the two things that neither requires removing + or forbidding the route, and says in a paragraph of its own what detectable means, that the + project writes the detecting rule, which works only where the detector lets it see every use, and what a + documented check is; clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) says that for a project's own rule printing the identifier + unaltered is everything it asks of the detector and the resolving is the toolchain's, without + needing the other specifications; section [8](DETECTOR-SPEC.md#8-conformance)'s paragraph on the kind says what a + configuration is and, in a paragraph of its own, that the duty is not one of the MUSTs + conformance is judged on and that a verdict leaving the kind out still stands on conformance. + Each after readers of run 28 missed or named the passage. Accepted by run NNN. Minor version + when released. ### 1.0.0, 2026-09-08 diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index a07c271..18bfd50 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -200,15 +200,15 @@ ticket or a colleague and not the [Message]. Printing a documentation path in th not enough. It helps only a reader who has the [Message]; the [Identifier] alone still leads nowhere. -**For a project's own [Rule]**, the [Detector] cannot know the documentation, so what -it owes is the half it can give: the [Identifier] printed unaltered under clause 4.3, and no -transformation of it. Where the [Identifier] is itself a URL, as method clause 3.6 allows, -the [Detector] prints that URL and has done what this clause asks. What this clause forbids is -a documentation path printed alongside a shorter [Identifier] that cannot be looked up on its -own. The lookup for such -a [Rule] is an obligation on the project's assembled [Toolchain], under clause 4.2 -of the [toolchain specification](TOOLING-SPEC.md), and a [Detector] that offers it as well has -gone further than this clause asks. +**For a project's own [Rule]**, the [Detector] cannot know the documentation, so this clause +asks of it only the half it can give: the [Identifier] printed unaltered, as clause 4.3 already +requires, with no transformation of it. For such a [Rule], that is everything this clause asks of +the [Detector]; the resolving itself is done by the project's [Toolchain], not by the [Detector]. +Where the project has chosen a URL as the [Identifier], the [Detector] prints that URL unaltered +like any other [Identifier]. What this clause forbids is a documentation path printed alongside a +shorter [Identifier] that cannot be looked up on its own. A [Detector] that offers the lookup as +well has gone further than this clause asks. The [toolchain specification](TOOLING-SPEC.md) states +the [Toolchain]'s side, in its clause 4.2; it is not needed to judge a [Detector] here. **Why**: method specification clause 8.3 requires the [Identifier] to resolve without a human. An index keyed on anything else does not resolve it. This is the most commonly failed clause in this @@ -276,14 +276,18 @@ nothing to run. ### 7.1 A detector MAY offer an inline suppression route, but MUST make it detectable or disableable An inline ignore comment, a per-line directive and a generated [Baseline] are all inline -[Suppression] routes, and the [Detector] MAY ship them. For each route it ships, it need not remove or forbid the route, but it MUST do at -least one of two things: let the project switch the +[Suppression] routes, and the [Detector] MAY ship them. For each route it ships, it need not +remove or forbid the route, but it MUST do at least one of two things: let the project switch the route off in the [Detector]'s configuration, or make the route detectable. Either alone satisfies -this clause. A route is detectable when the project can find every use of it, either with a -[Rule] of the project's own that the [Detector] runs, or with a mechanical check the [Detector] -documents. The [Detector] need not ship that [Rule]. Switching the route off or -detecting it lets a project that decides to forbid the route enforce that decision. A route that -can be neither switched off nor detected does not [Conform]. +this clause; neither requires the [Detector] to remove or forbid the route. A route that can be +neither switched off nor detected does not [Conform]. + +A route is detectable when the project can find every use of it, in one of two ways. The first is +a [Rule] the project writes for itself, under clause 4.1, and the [Detector] runs. That way works +only where the [Detector] lets such a [Rule] see every use of the route, and the [Detector] need +not ship the [Rule] itself. The second is a mechanical check the [Detector] documents, such as a +text search for the route's documented spelling. Switching the route off or detecting it is what +lets a project that decides to forbid the route enforce that decision. **Why**: the method specification's position is that [Suppression] is an [Owner] decision under its clause 3.4 and section 4, and an [Owner] cannot decide something they are @@ -320,14 +324,18 @@ and are stated in the [toolchain specification](TOOLING-SPEC.md), so there is no for a [Detector]. **A verdict on a [Detector] MUST name its kind, [Semantic], [Syntactic] or [Textual], in each -configuration the verdict grades.** The kind is what a [Rule] can ask in that configuration, as -section 2 glosses it. ESLint, for example, is [Syntactic] without type information and [Semantic] -with it. The kind named is the one the [Detector] has without a hosted service or a licence tier, -since under clause 5.4 a finding reported only there does not [Conform]. This duty is on whoever -gives the verdict, not on the [Detector]: the kind is not a level of [Conformance] and does not -change the verdict, so a [Textual] [Detector] can [Conform], and a [Semantic] one can fail to. It -is named because the method specification's clause 3.2 requires a [Rule] to be written in the -strongest kind of [Detector] that can express it. +configuration the verdict grades.** A configuration here is one way of setting the [Detector] up +to run, and one [Detector] can have several: ESLint, for example, is [Syntactic] run without type +information and [Semantic] run with it. The kind is what a [Rule] can ask in that configuration, +as section 2 glosses it. The kind named is the one the [Detector] has without a hosted service or +a licence tier, since under clause 5.4 a finding reported only there does not [Conform]. + +This MUST is on whoever gives the verdict, not on the [Detector], so it is not one of the MUSTs +in sections 4 to 7 that [Conformance] is judged on: the kind is not a level of [Conformance] and +does not change the verdict. A [Textual] [Detector] can [Conform], and a [Semantic] one can fail +to. A verdict that leaves the kind out has not met this duty, but what it finds about +[Conformance] still stands. The kind is named because the method specification's clause 3.2 +requires a [Rule] to be written in the strongest kind of [Detector] that can express it. ### 8.1 A maintainer MAY declare the version of this document the detector conforms to, and the declaration records known gaps diff --git a/SPEC.md b/SPEC.md index e00a87b..ba81052 100644 --- a/SPEC.md +++ b/SPEC.md @@ -196,8 +196,9 @@ clause 3.1's bounds is itself the evidence that the [Defect] is out of scope, an more reliable than a judgement made before trying. The conclusion that no pattern exists MUST be stated in one sentence, naming at least two independent techniques tried at expressing the [Rule], and recorded where the project's other decisions are enumerable under clause 8.7, not -only with the fix. Two techniques are independent when they would miss different things, as -clause 3.1 says of a search. That conclusion is the cheapest way +only with the fix. A technique here is a way of trying to write the [Rule]. Two such techniques +are independent when they would miss different things, which is the same test clause 3.1 applies +to ways of searching for [Instances]. That conclusion is the cheapest way out of the method and the only one that would otherwise leave no trace; enumerating it is how an [Owner] sees an [Agent] routing around the method, and naming the techniques is what lets them judge whether the attempt was one. @@ -358,17 +359,23 @@ second technique that turns up new [Instances] means a third is owed; a second t new means the search is done. That is a test the [Practitioner] can apply and record without a threshold, and it is the standard clause 3.4 refers back to. -Two techniques are independent when they would miss different things. A text search for the token -and a reading of the code that consumes the value are independent; two text searches for two -spellings of the same token are one technique, because they miss the same things; asking someone who knows the system is a third. For a -[Class] defined by how code is spelt, searching for each other common spelling of the idiom is a -technique. A technique counts only if it does not run the [Detector] under proof in any -configuration and would exist unchanged had the [Rule] never been written. So a run of the -[Rule], however wide, is never a technique, and a clean run of it shows nothing about saturation: -this search exists to check the [Rule]. The record MUST name both techniques, chosen before the -[Rule] is run against the codebase, and MUST state what each found that the other could not have -checked. Where a later technique shows that two earlier checks shared a blind spot, they were one -technique, and a third is owed. +In this clause a technique is a way of searching the codebase for [Instances]. Two techniques are +independent when they would miss different things. A text search for the token and a reading of +the code that consumes the value are independent, and asking someone who knows the system is a +third. Two text searches for two spellings of the same token are one technique, because they miss +the same things. For a [Class] defined by how code is spelt, though, searching for each other +common spelling of the idiom is a technique of its own, because there a search for one spelling +misses every other. + +A technique counts only if it does not run the [Detector] under proof in any configuration and +would exist unchanged had the [Rule] never been written, because this search exists to check the +[Rule]. A run of the [Rule], however wide, is therefore never a technique. Such a run belongs to +the proof of clause 3.3 or the [Sweep] of clause 3.4, and a clean one shows nothing about whether +this search is saturated. + +The record MUST name both techniques, chosen before the [Rule] is run against the codebase, and +MUST state what each found that the other could not have checked. Where a later technique shows +that two earlier checks shared a blind spot, they were one technique, and a third is owed. That turns the single-[Instance] case into something checkable rather than a matter of taste. If an independent search finds [Instances] the [Rule] missed, the [Class] was drawn too narrowly and the [Rule] @@ -409,13 +416,14 @@ are examples rather than a permitted list, and a [Detector] is not disqualified **Qualifying is not the same as being the right host.** A [Rule] the [Practitioner] writes MUST be written in the strongest kind of [Detector] that can express it: a [Semantic] one, else a -[Syntactic] one, else a [Textual] one. The kind is that of the configuration the [Rule] runs in. -Enabling an existing [Rule], or tightening its thresholds, is not writing one, and the -off-the-shelf paragraph above covers it; giving an existing [Rule] a pattern of the project's own, -or changing the pattern of a [Rule] the project wrote, is writing one. A weaker kind is allowed -on two grounds only: where the stronger kind cannot express the [Rule], or where the language has -no [Detector] of the stronger kind that accepts bespoke [Rules]. For example, the stronger kind -cannot express a [Rule] that reads a file the [Semantic] [Detector] does not parse. +[Syntactic] one, else a [Textual] one. A weaker kind is allowed only where the stronger kind +cannot express the [Rule], or where the language has no [Detector] of the stronger kind that +accepts bespoke [Rules]; those are the two grounds. For example, the stronger kind cannot express +a [Rule] that reads a file the [Semantic] [Detector] does not parse. The kind is that of the +configuration the [Rule] runs in. Enabling an existing [Rule], or tightening its thresholds, is +not writing one, and the off-the-shelf paragraph above covers it; giving an existing [Rule] a +pattern of the project's own, or changing the pattern of a [Rule] the project wrote, is writing +one. For the second ground, the language has a [Detector] of the stronger kind when the project already runs one or the language's own [Detector] ecosystem offers one. A stronger configuration @@ -425,14 +433,13 @@ to it through a documented extension point, such as a plugin interface; it need of the [detector specification](DETECTOR-SPEC.md). If no [Detector] of the stronger kind can reach the construct the [Rule] is about, that is the first ground. -On the [Remediation docs] of a [Rule] they -write, the [Practitioner] MUST name the [Detector] and its kind. Where that kind is weaker than -[Semantic], the [Practitioner] MUST also record, for each step down the order [Semantic], -[Syntactic], [Textual], which ground holds, that the stronger kind cannot express the [Rule] or -that the language has no [Detector] of the stronger kind that accepts bespoke [Rules], and why, -naming the [Detectors] of the stronger kind that were considered, on the [Rule]'s -[Remediation docs] and in the description of the commit that adds the [Rule], moves it to another -[Detector], or first records the grounds. The [Practitioner] judges whether the grounds hold, as +On the [Remediation docs] of a [Rule] they write, the [Practitioner] MUST name the [Detector] and +its kind. Where that kind is weaker than [Semantic], the [Practitioner] MUST also record the +grounds in two places, both of them: on the [Rule]'s [Remediation docs], and in the description +of a commit. That commit is the one that adds the [Rule], moves it to another [Detector], or first +records the grounds. What is recorded, in both places, is which of the two grounds holds at each +step down the order [Semantic], [Syntactic], [Textual], and why, naming the [Detectors] of the +stronger kind that were considered. The [Practitioner] judges whether the grounds hold, as section 4 gives them the choice of [Detector]; a project MAY require its [Owner]'s sign-off on a weaker kind as well, by recording that requirement where it records its [Calibrations]; the sign-off is then recorded with the grounds, @@ -956,12 +963,13 @@ evidence is gone and the opportunity closes with it. the verdict that they were rests on reproduction, as this section closes by requiring, not on the report. -**A [Defence] [Conforms]** if it satisfies clauses 3.1, 3.2, 3.3, 3.5 and 3.6: it is drawn to a -[Class] within both bounds and not to the reported [Instance], it is evaluated by reading code, -in the kind of [Detector] clause 3.2 requires, it -was proven to fire against either the originating [Instance] or [Fixture] code, it fails rather -than warns, and its [Message] is terse and resolves to [Remediation docs] versioned alongside -it. A [Defence] can [Conform] whilst the [Sweep] is incomplete; what that describes is a +**A [Defence] [Conforms]** if it satisfies each of clauses 3.1, 3.2, 3.3, 3.5 and 3.6 in full. +In brief: it is drawn to a [Class] within both bounds and not to the reported [Instance], it is +evaluated by reading code, in the kind of [Detector] clause 3.2 requires, it was proven to fire +against either the originating [Instance] or [Fixture] code, it is a permanent part of the +project's checks and fails rather than warns, every [Suppression] it carries is covered by a +recorded decision, and its [Message] is terse and resolves to [Remediation docs] versioned +alongside it. A [Defence] can [Conform] whilst the [Sweep] is incomplete; what that describes is a [Conforming] [Defence] over a codebase with known [Instances], which section 4 requires to be recorded, not a [Conforming] remediation. From 224579fbaa6da64cb952fe2b7928f67b8f2f054b Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 00:34:02 +0100 Subject: [PATCH 049/105] Keys: method 2d and detector 2b quotations follow the reworded 3.2 and 7.1 Quotations only, verbatim from c6ee1a8: 2d quotes the grounds sentence and the record sentences as they now read, since the grounds left the record sentence for the one after the strongest-kind sentence; 2b quotes 7.1's obligation sentences and its paragraph on what detectable means. No question, element, threshold or fixture changed. Co-Authored-By: Claude Opus 5.5 --- acceptance/detector/KEY.md | 10 ++++++---- acceptance/method/KEY.md | 14 ++++++++------ 2 files changed, 14 insertions(+), 10 deletions(-) diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 43e91f7..f4271fd 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -40,10 +40,12 @@ agent summary (those belong to the toolchain specification). project can write or a mechanical check it documents; it is not required to remove or forbid it. Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each route it ships, it need not remove or forbid the route, but it MUST do at least one of two things: let the project switch the route off in the Detector's - configuration, or make the route detectable. Either alone satisfies this clause. A route is - detectable when the project can find every use of it, either with a Rule of the project's own - that the Detector runs, or with a mechanical check the Detector documents. The Detector need not - ship that Rule." An answer that says the detector must forbid the route, or must require a + configuration, or make the route detectable. Either alone satisfies this clause; neither requires + the Detector to remove or forbid the route."; "A route is detectable when the project can find + every use of it, in one of two ways. The first is a Rule the project writes for itself, under + clause [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it), and the Detector runs. That way works only where the Detector lets such a Rule see + every use of the route, and the Detector need not ship the Rule itself. The second is a + mechanical check the Detector documents". An answer that says the detector must forbid the route, or must require a reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD), is wrong. - **2c.** With the rule, in the installed copy, at a version tracked together, and yes, offline. Key quotations, [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access): "Resolution of a bundled rule's identifier MUST work from the installed diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index a9dba07..dfe8eea 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -56,12 +56,14 @@ is correct; a bullet that says no weaker kind is ever allowed is wrong. and in the description of the commit that adds the rule, moves it, or first records the reason. Key quotations, [3.2](../../SPEC.md#32-build-the-net): "A Rule the Practitioner writes MUST be written in the strongest kind of Detector that can express it: a - Semantic one, else a Syntactic one, else a Textual one."; "Where that kind is weaker than - Semantic, the Practitioner MUST also record, for each step down the order Semantic, Syntactic, - Textual, which ground holds, that the stronger kind cannot express the Rule or that the language - has no Detector of the stronger kind that accepts bespoke Rules, and why, naming the Detectors - of the stronger kind that were considered, on the Rule's Remediation docs and in the description - of the commit that adds the Rule, moves it to another Detector, or first records the grounds." Supporting: "On the Remediation docs of a Rule + Semantic one, else a Syntactic one, else a Textual one."; "A weaker kind is allowed only where + the stronger kind cannot express the Rule, or where the language has no Detector of the stronger + kind that accepts bespoke Rules; those are the two grounds."; "Where that kind is weaker than + Semantic, the Practitioner MUST also record the grounds in two places, both of them: on the + Rule's Remediation docs, and in the description of a commit. That commit is the one that adds + the Rule, moves it to another Detector, or first records the grounds. What is recorded, in both + places, is which of the two grounds holds at each step down the order Semantic, Syntactic, + Textual, and why, naming the Detectors of the stronger kind that were considered." Supporting: "On the Remediation docs of a Rule they write, the Practitioner MUST name the Detector and its kind." A correct answer gives three elements, and may leave out everything else above: (i) the strongest kind that can express the rule, in the order semantic, syntactic, textual; (ii) a weaker kind only where the stronger From 6ddba40abdbff7a5acdd689edbe41a5f523f2a93 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 00:39:12 +0100 Subject: [PATCH 050/105] Method 3.1 and detector 7.1: two readings from the review settled 3.1 says a search per spelling counts on its own where the spelling is the class itself. 7.1's second way is a check the detector itself documents, so documenting the route's spelling alone is not enough; the detector key's quotation follows. Co-Authored-By: Claude Opus 5.5 --- DETECTOR-SPEC.md | 4 ++-- SPEC.md | 4 ++-- acceptance/detector/KEY.md | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index 18bfd50..10e9fc6 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -285,8 +285,8 @@ neither switched off nor detected does not [Conform]. A route is detectable when the project can find every use of it, in one of two ways. The first is a [Rule] the project writes for itself, under clause 4.1, and the [Detector] runs. That way works only where the [Detector] lets such a [Rule] see every use of the route, and the [Detector] need -not ship the [Rule] itself. The second is a mechanical check the [Detector] documents, such as a -text search for the route's documented spelling. Switching the route off or detecting it is what +not ship the [Rule] itself. The second is a mechanical check that the [Detector] itself documents, such as a +text search it documents for the route's spelling. Switching the route off or detecting it is what lets a project that decides to forbid the route enforce that decision. **Why**: the method specification's position is that [Suppression] is an [Owner] diff --git a/SPEC.md b/SPEC.md index ba81052..926140f 100644 --- a/SPEC.md +++ b/SPEC.md @@ -364,8 +364,8 @@ independent when they would miss different things. A text search for the token a the code that consumes the value are independent, and asking someone who knows the system is a third. Two text searches for two spellings of the same token are one technique, because they miss the same things. For a [Class] defined by how code is spelt, though, searching for each other -common spelling of the idiom is a technique of its own, because there a search for one spelling -misses every other. +common spelling of the idiom is a technique of its own, because there the spelling is the [Class] +itself. A technique counts only if it does not run the [Detector] under proof in any configuration and would exist unchanged had the [Rule] never been written, because this search exists to check the diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index f4271fd..d51400a 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -45,7 +45,7 @@ agent summary (those belong to the toolchain specification). every use of it, in one of two ways. The first is a Rule the project writes for itself, under clause [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it), and the Detector runs. That way works only where the Detector lets such a Rule see every use of the route, and the Detector need not ship the Rule itself. The second is a - mechanical check the Detector documents". An answer that says the detector must forbid the route, or must require a + mechanical check that the Detector itself documents". An answer that says the detector must forbid the route, or must require a reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD), is wrong. - **2c.** With the rule, in the installed copy, at a version tracked together, and yes, offline. Key quotations, [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access): "Resolution of a bundled rule's identifier MUST work from the installed From db47301c41b4cff58e9fd55cb8b7b68e66e39c99 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 00:39:12 +0100 Subject: [PATCH 051/105] Acceptance runs 27 and 28: SPEC.md and DETECTOR-SPEC.md, fifth cohort, both FAIL Five cold Haiku readers each, made under pull request 13 at 598908b and marked by a delegated marker. Evidence is posted on pull request 14 and linked from each record, with the findings applied to the text. Co-Authored-By: Claude Opus 5.5 --- .../runs/027-semantic-detectors-run-9.md | 266 ++++++++++++++++++ .../runs/028-semantic-detectors-run-10.md | 253 +++++++++++++++++ 2 files changed, 519 insertions(+) create mode 100644 acceptance/runs/027-semantic-detectors-run-9.md create mode 100644 acceptance/runs/028-semantic-detectors-run-10.md diff --git a/acceptance/runs/027-semantic-detectors-run-9.md b/acceptance/runs/027-semantic-detectors-run-9.md new file mode 100644 index 0000000..54eb638 --- /dev/null +++ b/acceptance/runs/027-semantic-detectors-run-9.md @@ -0,0 +1,266 @@ +# Run 27: SPEC.md, semantic detectors, ninth run in pull request 14 + +- **Document**: SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + branch `editor/semantic-detectors` at 1db0c67, the commit the readers were given; the question + set, key and fixture read at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input; no tools, no settings, no project context and no + saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is byte for byte the question set, the document and the + fixture at 1db0c67, in that order, each preceded by one `===== FILE: =====` line, with + nothing else added apart from one blank line between the end of each file and the next file's + marker line. +- **Managed (policy) settings**: none; the machine applies no managed settings to the readers and + no managed settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. Every sheet has answer text, no process reports an error + (`is_error` false, subtype `success`), and every reported model is the cohort's model. The + editor reports no reader set aside for a cause the sheets cannot show. +- **Run within the pull request**: 9, counting void runs; runs in the pull request so far, + counting void runs: 9, as the editor supplied. Run 28 is made at the same time and does not bear + on this document. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review. It read `ACCEPTANCE.md` and + `acceptance/runs/README.md` in this worktree, which differ from commit 598908b only by this pull + request's question-set line adding 2d for SPEC.md; the document, question set, key and fixture + at 1db0c67; run 25 in full, the passages and verdicts of runs 19, 21 and 23, and the headers of + runs 20, 22, 24 and 26, which are on DETECTOR-SPEC.md; and, for the fourth pass-criteria row, + the change to SPEC.md between dfb9b68, the document run 25 read, and 1db0c67. + +## Marks + +`ok` is correct, `miss` is a miss; `T` marks a miss traceable to the text. Judgements J1 to J8 +are set out below. + +| Reader | Q1 | 2a | 2b | 2c | 2d | 3.1 | 3.2 | 3.3 | 3.4 | 3.5 | 3.6 | Remediation | Defence | Section 4 | Q4 passages | +| ------ | ------- | --- | --- | --- | ------ | ------- | ------- | ------ | --- | --- | ------- | ----------- | ------- | --------- | ----------- | +| 1 | correct | ok | ok | ok | miss T | Partial | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 2 | correct | ok | ok | ok | ok | No | No | Yes ok | No | No | No | ok | ok | ok | two | +| 3 | correct | ok | ok | ok | miss T | No | No | Yes ok | No | No | No | ok | ok | ok | two | +| 4 | correct | ok | ok | ok | ok | Partial | Partial | Yes ok | No | No | Partial | ok | ok | ok | two | +| 5 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | + +Every grade under 3.1, 3.2, 3.4, 3.5 and 3.6 is one the key accepts; reader 4's 3.6 Partial +quotes the versioning sentence (J7). No reader grades 3.2, 3.5 or 3.6 Yes, so none of the key's +cases that count against the document arises. + +### Question 1 + +Five of five correct. Every sheet has at least four bullets naming obligations the key lists (the +class bounded both ways with its hazard, the strongest kind of detector and never a test, the +proof in a commit of its own, the sweep with its count and the fix of every instance, permanent +and blocking, the terse message with an identifier resolving to versioned documentation, the +record of a deferred defect), and no bullet asserts an obligation the document does not state. +The section 2 bullets of readers 1, 2 and 4, on building the defence first or attempting rather than +pre-judging, state section 2's obligations (J6 for reader 4). The closing "fix the original defect with a reproduction test" +of readers 1, 3 and 5 is section 2 and section 6. Reader 3's line after its six bullets, on what +belongs to the Owner, states section 4 and is not a seventh bullet; no mark turns on it. + +### Question 2 + +Every reader quoted words of the document for every item, so every miss here is traceable to the +text. + +- **2a**, five of five. Each states class, rule, proof, sweep and the fix of every instance, in + that order, and quotes section 2's sentence "The order is section 3's: attribute the [Defect] to + a [Class], build the [Rule], prove that it fires, run the [Sweep], then fix every [Instance]." +- **2b**, five of five. Each says only the owner may and an agent may not, and quotes the key's + section 4 sentence. Every reader says the owner is a human, readers 4 and 5 quoting "The [Owner] + is always a human." +- **2c**, five of five. Each states all six clauses and a verdict resting on reproduction, and + each quotes section 7's "MUST rest on reproduction, not on the report"; no reader accepts the + pull request description as the verdict. +- **2d**, two of five (readers 2 and 4). The key's three elements are (i) the strongest kind, in + the order semantic, syntactic, textual; (ii) both grounds for a weaker kind; (iii) the reason + recorded, with at least one place named and the places not given as alternatives (J2). + - Reader 1: (i) and (ii) in its two quotations, the strongest-kind sentence and the sentence + stating the two grounds. (iii) missed: "the practitioner must record which ground holds and + why" names no place, and neither quotation names one (J2). + - Reader 2: (i) and (ii) in its quotations, (iii) in its sentence, "in the rule's remediation + docs and in the commit description", with its quotation of "naming the [Detectors] of the + stronger kind that were considered". Correct. + - Reader 3: (i) in its quotation. (ii) missed: "A weaker kind is allowed only on two grounds" + says there are two and states neither, and its only quotation is the strongest-kind sentence. + (iii) missed: "record which ground holds and why" names no place (J2). + - Reader 4: (i), (ii) and (iii) in its three quotations, the third giving the places joined by + "and" (J3). Correct. + - Reader 5: (i) in its quotation. (ii) missed: "only on one of two grounds" states neither. + (iii) missed: "record which ground holds at each step down, with the stronger detectors + considered" names no place (J2). + +### Question 3 + +- **3.1**: readers 2, 3 and 5 No, readers 1 and 4 Partial, each noting that no two independent + search techniques are recorded. +- **3.2**: readers 1, 4 and 5 Partial, readers 2 and 3 No, each saying the detector or its kind is + not named; readers 1, 4 and 5 say the rule is not a test, and every reader adds that no grounds + for a weaker kind are recorded, which the key does not mark wrong. +- **3.3**: five of five Yes. Readers 1, 3, 4 and 5 say the inline ignore is not judged here; none + attributes it to 3.3. +- **3.4** and **3.5**: five of five No, each naming the inline ignore on `Report::render()`, under + 3.5 as uncovered by any recorded decision. +- **3.6**: five of five. Readers 1, 2, 3 and 5 grade No; reader 4 grades Partial and quotes "A + [Rule]'s documentation MUST be versioned with the [Rule]." (J7). +- **Remediation**: five of five say it does not conform, each naming 3.4 among the failing clauses + (J8). +- **Defence**: five of five say it does not conform, each naming 3.6 as not met and 3.2 as well, + readers 3 and 4 saying 3.2 cannot be shown or confirmed and reader 5 that it is only partial. Every reader also counts 3.1 or 3.5 against the defence; the key does + not mark that wrong. +- **Section 4**: five of five name the inline ignore on `Report::render()`, and each also names + leaving the next wider rule unbuilt, which the key accepts alongside it. + +### Judgements made in marking + +Made as a careful reader of the document would, as `ACCEPTANCE.md` requires. Where a rule settles +the point, the record says which. + +- **J1. The sentence.** Where a reader wrote more than one sentence before or around its + quotations, all of that text is read with the quotations as the answer. No mark turns on it. +- **J2. 2d's element (iii) needs a place named.** The key says of (iii) that "naming one of the + two places is enough" and that an answer presenting them as alternatives is wrong; "enough" sets + a minimum, and a key that let an answer name no place would have no minimum to state and no + alternatives to forbid. Runs 19, 21, 23 and 25 read it the same way. Reader 1's miss on 2d turns + on it; readers 3 and 5 miss (ii) as well. Read otherwise, reader 1 would be correct and 2d three + of five; 2d would still fail on readers 3 and 5, whose misses on (ii) rest on no judgement, and + the verdict would not change. +- **J3. Elided quotations.** A quotation with an ellipsis is read as one quotation per unbroken + run of words, each in the document's order, and states only what those runs state (reader 4's + third 2d quotation; reader 5's clause 3.2 passage). Settled by the rule on quoting: consecutive + words of any length. Every run in reader 4's quotation is verbatim. +- **J4. Clause labels.** A clause number given with a quotation, as question 4 asks, is read as + the quotation's label, not as naming the whole clause, as runs 21, 23 and 25 read it. Reader 2's + "Section 3.3, Part B" and reader 3's "Clause 3.3, Part B" labels do not extend their passages to + all of Part B, so the two are not the same passage: reader 2's lies in the paragraph beginning + "A [Narrowing] reduces the [Practitioner]'s own work" and the table's first row, reader 3's in + the paragraph beginning "**An inline ignore comment, an ignore entry or a [Baseline]**". +- **J5. Passages set beside the main one count.** Where a reader says it was confused by two + passages read together, a careful reader takes both as the confusing passage, since the + confusion lies in neither alone, as runs 23 and 25 read it. So reader 3's section 7 item covers, + by its description "the recorded-decision condition that clause 3.5 itself sets", the sentences + of clause 3.5's opening paragraph that set it; reader 4's clause 3.5 item covers section 7's + "it fails rather than warns"; reader 2's Part B item covers the table's first row, which gives + the [Narrowing] to the [Practitioner]; reader 4's clause 3.1 item covers both step 4 and the + paragraph beginning "**What this clause leaves on the record**". Reader 2's section 2 item says + that "In 3.1 the same phrase refers to ways of searching for instances" and that it could not + tell "whether the two uses were meant to share the same test of independence"; the test of + independence is clause 3.1's sentence "Two techniques are independent when they would miss + different things.", so the description is read as naming that sentence, as run 25 read reader + 5's like description. A finding turns on this: clause 3.1's paragraph beginning with that + sentence, run 25's open passage, is not cleared on it. Read as context only, that paragraph is + named by no reader and is cleared; the verdict would not change. +- **J6. "Two independent search techniques" (reader 4, question 1).** Reader 4's second bullet + says the out-of-scope conclusion is stated "in one sentence naming two independent search + techniques"; section 2's sentence asks for "at least two independent techniques tried at + expressing the [Rule]". Read for its gist, the bullet names section 2's obligation to state that + conclusion in one sentence naming two independent techniques and to record it, an obligation + the document states, with the techniques mislabelled; it is marked correct. Read as asserting a + different obligation, it would be the reader's miss, since the bullet quotes nothing, question 1 + would be four of five and the second criterion still met. Reader 2 names the same sentence under + question 4. +- **J7. 3.6 Partial and "the versioning sentence" (reader 4).** The key accepts Partial "where the + reasoning quotes the versioning sentence". Clause 3.6's versioning requirement is now the one + sentence "**A [Rule]'s documentation MUST be versioned with the [Rule].**", which reader 4 quotes + verbatim; it is the versioning sentence on either sense run 25 considered, so no mark turns on + which. +- **J8. The remediation item.** The key's answer is that the remediation does not conform, giving + 3.2, 3.4 and 3.6 as the reason. A reader saying it does not conform and naming among the failing + clauses one the key names is correct, as runs 23 and 25 marked it; every reader names 3.4. Reader + 4 names 3.1, 3.4 and 3.5, having graded 3.2 and 3.6 Partial; it asserts nothing the key marks + wrong. + +## Passages quoted as confusing + +Same passage as `ACCEPTANCE.md` defines it: sentences in common, or the same paragraph; a list +item, table row or terminology entry is its own paragraph. + +| Passage | Clause | Readers | Words quoted, verbatim, or the description used | +| ------------------------------------------------------------- | ----------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Where the grounds for a weaker kind are recorded | 3.2 | two: 1, 5 | 1: "on the [Rule]'s [Remediation docs] and in the description of the commit that adds the [Rule], moves it to another [Detector], or first records the grounds." 5: "...naming the [Detectors] of the stronger kind that were considered, on the [Rule]'s [Remediation docs] and in the description of the commit that adds the [Rule], moves it to another [Detector], or first records the grounds." Both in the paragraph beginning "On the [Remediation docs] of a [Rule] they write". Run 21's open passage (below) | +| What a defence must satisfy of clause 3.5 | 3.5 and 7 | two: 3, 4 | 3: "A Defence Conforms if it satisfies clauses 3.1, 3.2, 3.3, 3.5 and 3.6: … it fails rather than warns, …", in section 7's paragraph beginning "**A [Defence] [Conforms]**", set beside, by description, "the recorded-decision condition that clause 3.5 itself sets" (J3, J5). 4: "neither does a [Rule] whose green run depends on a [Suppression] that no recorded decision covers", in clause 3.5's opening paragraph, set beside section 7's "it fails rather than warns" (J5) | +| Independent techniques for expressing a rule and for a search | 2 and 3.1 | one: 2 | "stated in one sentence, naming at least two independent techniques tried at expressing the [Rule]", in section 2's "Attempt rather than pre-judge" paragraph, set beside, by description, clause 3.1's "same test of independence" (J5). Run 25's open clause 3.1 passage (below) | +| Narrowings enumerated and reported as if suppressions | 3.3, Part B | one: 2 | "Every [Narrowing] MUST be enumerable by the same means as the project's [Exceptions]..." and "a [Narrowing] that excludes more code than the [Rule] still covers MUST be reported to the [Owner] as if it were a [Suppression].", in the paragraph beginning "A [Narrowing] reduces the [Practitioner]'s own work", set beside, by description, the table's [Narrowing] row (J3, J4, J5) | +| An inline ignore is not judged under clause 3.3 | 3.3, Part B | one: 3 | "An inline ignore comment, an ignore entry or a Baseline left over an Instance that carries the Hazard is not judged under this clause. Leaving an Instance that carries the Hazard in place behind one is not a Narrowing, whatever note accompanies it. It is a Suppression of the Instance. Clauses 3.4 and 3.5 and section 4 govern it, and it does not fail this clause." (J4) | +| Leaving the next wider rule unbuilt | 4 | one: 1 | "It is a reason not to widen unasked, not authority to leave the wider [Rule] out, and recording it as a known gap does not make it the [Practitioner]'s decision.", in section 4's list item beginning "Leaving unbuilt the next wider [Rule]" | +| Step 4 and the record of the next wider rule | 3.1 | one: 4 | "Where the [Class] stays at the reported [Instance], name the next wider [Rule] and why it was not built.", step 4 of the five steps, set beside "the next wider [Rule] that was not built, with the reason", in the paragraph beginning "**What this clause leaves on the record**" (J5) | +| The default owner's decisions recorded | 1, Owner | one: 5 | "The default [Owner] is usually the person under the same deadline as the [Practitioner], so a decision they take under section 4 to keep an [Instance] unfixed, [Suppress], [Baseline] or remove a [Rule] MUST be recorded in the project record with its justification." | + +Readers' accounts of what they could not tell are recorded with the passage they attach to and are +not suggestions; none of the sheets proposes a change. + +**Earlier passages (the fourth row).** The open passages are those earlier runs of this document +found confusing by two or more and did not clear: run 25's three, of which one is run 21's clause +3.2 passage, not cleared by runs 23 or 25. Every other passage of runs 19, 21 and 23 was cleared +in runs 21, 23 and 25. As the paragraphs read in the changed document: + +- Clause 3.2's grounds and record, run 21's passage, which run 23 read as the grounds sentence of + the paragraph beginning "**Qualifying is not the same as being the right host.**", the paragraph + now beginning "For the second ground", and the record paragraph beginning "On the \[Remediation + docs\] of a [Rule] they write", whose record sentences run 23 found kept from run 21's paragraph. + Readers 1 and 5 name the last part of the record paragraph's second sentence. Not cleared. +- Clause 3.5's sentence on the entry point for a [Bundled defence], now the paragraph beginning "A + [Bundled defence] is a [Defence] that a shipped [Toolchain] enables by default": named by no + reader. Readers 3 and 4 name clause 3.5's opening paragraph, not this one. Cleared. +- Clause 3.1's paragraph beginning "Two techniques are independent when they would miss different + things.": named by reader 2 by description (J5). Not cleared. + +No sentence kept from the cleared paragraph is named elsewhere in the document. Three passages +named here by one reader were once open and have since been cleared: section 4's list item on +leaving the next wider rule unbuilt (run 21, three readers; cleared in run 23), named by reader +1; the Owner entry in section 1 (run 23, two readers; cleared in run 25), named by reader 5; and +section 2's "Attempt rather than pre-judge" paragraph (run 19; cleared in run 21), named by reader +2\. Each is recorded as one reader's. The section 7 and clause 3.5 passage named here by readers 3 +and 4 was named by one reader, reader 2, in run 25, and is a new finding here. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in c6ee1a8 and the commit after +it that settles its review (key quotations kept verbatim in 224579f and that commit). The editor +first read every sheet, in runs 19 to 27, that named each passage, and fixed the cause the sheets +show. SPEC.md: clause 3.2 states the two grounds in their own sentence after the strongest-kind +sentence and names the two places early in the record sentence; section 7 states that a defence +conforms only where clause 3.5 is met in full, including its recorded-decision condition; clause 3.1 +uses "technique" in one sense only and says when a search per spelling counts. A fresh cohort +follows. + +## Process change + +Made under a process change before it merged: yes, pull request 13 at commit 598908b, named +before the readers started. + +## Verdict + +**FAIL.** Against the criteria in `ACCEPTANCE.md`: + +- Load-bearing statements: not met. 2a, 2b and 2c five of five; 2d two of five. Readers 3 and 5 + state neither ground for a weaker kind, only that there are two; readers 1, 3 and 5 name no + place for the record (J2). +- Restatement and application: met. Question 1 and every keyed item of question 3 five of five. +- Misses traceable to the text, each a finding: 2d for readers 1, 3 and 5, each quoting clause + 3.2's sentence "A [Rule] the [Practitioner] writes MUST be written in the strongest kind of + [Detector] that can express it: a [Semantic] one, else a [Syntactic] one, else a [Textual] + one."; readers 3 and 5 on the grounds for a weaker kind, which they say are two without stating + them, and readers 1, 3 and 5 on the record of those grounds, which they give without the place + it is recorded (reader 1's on J2). +- Passages found confusing by two or more, each a finding that blocks until the text is changed + and a fresh cohort no longer names it: clause 3.2's record paragraph beginning "On the + [Remediation docs] of a [Rule] they write", the places named at the end of its second sentence + (two readers), which is also run 21's open passage, not cleared; section 7's paragraph beginning + "**A [Defence] [Conforms]**", its "it fails rather than warns", against clause 3.5's opening + paragraph on a [Suppression] no recorded decision covers (two). Run 25's clause 3.1 paragraph + beginning "Two techniques are independent when they would miss different things." is not + cleared (one reader, on J5). Run 25's clause 3.5 passage on the [Bundled defence] is cleared. +- Single-reader notes, recorded: section 4's list item on leaving the next wider rule unbuilt + (reader 1); section 2's "Attempt rather than pre-judge" paragraph, and clause 3.3 Part B's + paragraph on narrowings enumerated and reported as if suppressions with the table's [Narrowing] + row (reader 2); clause 3.3 Part B's paragraph on an inline ignore (reader 3); clause 3.1's step + 4 against its record paragraph (reader 4); the Owner entry's sentence on the default owner's + decisions (reader 5). + +Answer sheets, the reader prompt, the command and the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090998641), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090998842), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090999356), [comment 4](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090999753). diff --git a/acceptance/runs/028-semantic-detectors-run-10.md b/acceptance/runs/028-semantic-detectors-run-10.md new file mode 100644 index 0000000..6404294 --- /dev/null +++ b/acceptance/runs/028-semantic-detectors-run-10.md @@ -0,0 +1,253 @@ +# Run 28: DETECTOR-SPEC.md, semantic detectors, tenth run in pull request 14 + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + editor's branch `editor/semantic-detectors` at 1db0c67; the question set, key and fixture read + at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input, with no tools, no settings, no project context + and no saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is byte for byte the question set, the document and the + fixture at 1db0c67, in that order, each preceded by one `===== FILE: =====` line, with the + files separated by one blank line and nothing else added, the same form as the packets of runs + 22, 24 and 26 +- **Managed (policy) settings**: none; the machine applies no managed settings and no managed + settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. The editor reports none set aside for a cause the sheets cannot + show; every sheet has answer text, no process reports an error (`is_error` false, subtype + `success`), and every reported model is the cohort's model +- **Run within the pull request**: 10, counting void runs; run 27, made at the same time, does not + bear on this document +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review + +## Marks + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | Section 8 | Kind | 6.4 | Q4 entries | +| ------ | ------- | ---- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --------- | ---------- | --- | ---------- | +| 1 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 2 | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 3 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 4 | correct | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | + +"Q4 entries" counts the entries each reader lists under question 4; the words each quotes are in +the table of passages below. Every miss under question 2 is traceable to the text: each answer +quotes the document in support. The miss under question 1 is the reader's own: the bullet it turns +on quotes nothing. No question was left unanswered. + +### Question 1 + +Every restatement names four or more obligations from the key's list. Readers 1, 3, 4 and 5 assert +no obligation the document does not state: correct. Reader 3's bullet that "a verdict must name the +detector's kind, Semantic, Syntactic or Textual (section 8)" is the obligation the key marks correct +on a verdict, and its bullet that "Inline suppressions should require a written reason (7.2, +SHOULD)" states a recommendation the document makes, as a recommendation (judgement 3). Reader 4's +first bullet and the last sentence of reader 2's sixth state what conformance is; they name no +obligation and assert none the document does not state. Readers 1 and 5 say a printed identifier +must resolve to its documentation, in the generality of 6.1's heading, as the key's own bullet does +(judgement 4). Reader 2's fourth bullet says "every finding must appear in the output of the command +the practitioner ran"; 5.3 says "The summary need not list every finding", so the bullet asserts an +obligation the document does not state: incorrect (judgement 2). Correct, four of five. + +### Question 2 + +The key's answers state these elements: + +- 2a: (a) Conformance is satisfying every MUST in sections 4 to 7; (b) judged on evidence; (c) a + detector that predates the document can conform; (d) the declaration is optional. +- 2b: (a) make the route disableable by configuration, or (b) detectable by a Rule the project can + write or a mechanical check the Detector documents; (c) the detector is not required to remove or + forbid it. +- 2c: (a) with the rule; (b) in the installed copy; (c) at a version tracked together; (d) yes, + reachable offline. + +Each answer is read as its sentence and its quotation together, for the gist of each element. + +| Reader | 2a | 2b | 2c | +| ------ | -------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | -------------------------------------------------------------------- | +| 1 | ok: all four; (d) from the sentence, "not on a declaration", and the 8.1 quotation | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6) | ok: all four; (a) and (c) from the 6.3 quotation | +| 2 | ok: all four; (a) from the section 8 quotation, (b) from the 8.1 quotation | ok: all three; (c) from the 7.1 quotation, "it need not remove or forbid the route" | ok: all four; (b) from the 6.2 quotation, (c) from the 6.3 quotation | +| 3 | ok: all four | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6) | ok: all four; (c) from the 6.3 quotation | +| 4 | miss: (a) not stated, "judged on evidence against the MUSTs" (judgement 7); (d) stated | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6) | ok: all four | +| 5 | ok: all four | ok: all three; (c) from the 7.1 quotation; (b) as "make every use of it detectable" | ok: all four; (b) from the 6.2 quotation | + +No answer asserts anything the document does not state or that the key says makes an answer +wrong: none says a declaration is required, that partial satisfaction is conformance, that the +detector must forbid the route or require a reason, or that a website will do. Every quotation is +verbatim apart from letter case, link and emphasis markup and a closing punctuation mark; reader 4's +8.1 quotation stops before the colon of its sentence, and reader 1's 2a quotation is the whole of +8.1's sentence. + +### Question 3 + +The key's intended verdicts are Yes for every clause but 6.3, which is No; the section 8 verdict +is "does not conform", because 6.3 fails; 6.4 does not change the verdict. + +- **Every clause**: all five readers give the intended verdict on every clause, 4.1 to 7.1. On + 6.3 each cites that one unresolved identifier fails the clause, readers 1, 2 and 3 quoting 6.3's + "One [Identifier] that does not resolve fails this clause"; no reader grades it Partial on the + grounds that thirty-eight resolve. Readers 1, 2, 3 and 4 add caveats on 4.2 or 4.3 about what the + fixture does not say, and grade Yes. +- **Section 8**: all five say Lintomatic does not conform, because 6.3 fails. +- **Kind**: all five say the fixture does not let them name the kind and name none, so none is + marked on it. Reader 3 calls the verdict incomplete on that duty and reader 4 says it cannot + complete the verdict from the fixture; none makes the verdict turn on the kind, and readers 1, 2, + 3 and 5 say outright that it does not change the verdict or the outcome. +- **6.4**: all five say it does not change the verdict, because it is a SHOULD. Readers 2 and 3 add + that 7.2 is an unmet SHOULD, reader 2 that a declaration would record the 6.3 gap, and reader 1 + that the toolchain MUST of 6.4's Why paragraph does not apply (judgement 8). + +### Judgements made in marking + +01. **How an open judgement is made.** As `ACCEPTANCE.md` at 598908b says: as a careful reader of + the document would, with the reason stated. Where a rule settles a mark, it is applied as + written. +02. **Question 1, reader 2's "every finding must appear".** Runs 22 and 24 marked incorrect bullets + saying every finding, or each finding, must appear in the output, because 5.3 says "The summary + need not list every finding" and "If it leaves any finding out, the output MUST also say where + the rest can be read". Run 26 marked correct a plain plural, "findings", as 5.3's heading. + Reader 2's "every finding" is the first case, not the second: a careful reader takes it as a + claim about each finding, which 5.3 contradicts. Incorrect. The bullet quotes nothing, so the + miss is the reader's. +03. **Question 1, reader 3's two further statements.** The kind bullet is the obligation the key + marks correct on a verdict, as run 26 held for the same words. The 7.2 statement is labelled a + SHOULD and is what 7.2 says, "The [Detector] SHOULD reject, or be configurable to reject, an + inline [Suppression] that carries no reason"; unlike run 26's reader 4, who called a MAY a + SHOULD, it asserts nothing the document does not state. Correct. +04. **Question 1, "a printed identifier must resolve", readers 1 and 5.** For a project's own rule + 6.1 asks only that the identifier be printed unaltered. The bullets speak at the level of 6.1's + heading, "The detector MUST provide a mechanism that resolves a printed identifier to its + documentation", and the key's bullet is in the same words, so a careful reader takes them as + that obligation. Correct. +05. **2b (c), not required to remove or forbid, readers 1, 3 and 4.** As runs 22, 24 and 26 held, + the key's answer includes what the detector need not do, so (c) answers a part of the question. + Readers 1, 3 and 4 say the detector must do one of the two things. Each quotes 7.1 from "it MUST + do at least one of two things", which follows the words "it need not remove or forbid the + route"; reader 1 adds in its sentence that either alone is enough, and readers 3 and 4 quote on + to "Either alone satisfies this clause." That either alone suffices entails that nothing more is required, but + does not say that removing or forbidding the route is not required, and an element reached only + by inference from another is not among the ways `ACCEPTANCE.md` names of stating it. Marked not + stated, as the earlier runs marked the same element. Readers 2 and 5 quote the clause from "For + each route it ships, it need not remove or forbid the route", and so state it. +06. **2b (b), the means of detection.** Clause 7.1 defines when a route is detectable in its own + text; readers 1, 3 and 4 use "detectable" in that sense and so state (b), as runs 22, 24 and 26 + held. It decides no mark, since each of those answers misses (c). +07. **2a (a), reader 4's "judged on evidence against the MUSTs".** The question asks what it means to + conform. The key's element is that conformance is satisfying every MUST in sections 4 to 7, and + it says an answer that partial satisfaction is conformance is wrong, so the element's gist is the + threshold, every MUST, not only what is measured. Reader 4's sentence says what conformance is + judged against, which is as true of a detector that fails one MUST as of one that meets them all, + and its quotations, section 8's second sentence and 8.1's sentence on a detector that predates + the document, do not carry "every MUST in sections 4 to 7" either. A careful reader does not + take "judged against the MUSTs" as saying all of them must be met. Not stated: a miss, and since + the answer quotes the document, traceable. Read the other way, the answer would be correct and + the third criterion would lose one finding; the verdict does not turn on this judgement, since + 2b fails the first and third criteria by three readers whichever way it is made. +08. **Asides outside the keyed answer.** Readers 2's and 3's notes on 7.2, reader 2's on what a + declaration would record, and reader 1's on the toolchain MUST of 6.4's Why paragraph are not + keyed items; each sheet's answer to whether 6.4 changes the verdict is the key's, and is marked + so. +09. **Question 4, reader 4's second entry, named by description.** Reader 4 writes "Clauses 6.1, 6.4, + 7.1 and section 9 refer to clauses in `TOOLING-SPEC.md` and `SPEC.md`, which I was not given", + with no quotation. `ACCEPTANCE.md` says a passage named without a quotation, by clause number or + description, covers every sentence of what it names. A careful reader takes what this entry + names as the references in those clauses to the other two specifications, not every sentence of + each clause: the clause numbers say where the references are, and the description says which + sentences confused the reader. So taken, it covers, in 6.1, the sentences of the paragraph + beginning "For a project's own [Rule]" that cite method clause 3.6 and clause 4.2 of the + toolchain specification, and the Why paragraph's first sentence; in 6.4, the Why paragraph's + sentence on the toolchain specification's self-audit; in 7.1, the Why paragraph's sentences + citing method clause 3.4 and section 4 and clause 4.3 of the toolchain specification; and in + section 9, the sentences that refer to the method or toolchain specification. Read the other + way, as every sentence of each clause, it would also cover 7.1's first paragraph, which blocks + already, and 6.1's heading, which reader 2 also names, making that heading a further passage + named by two readers; the fourth criterion fails either way, and the verdict does not turn on it. +10. **Question 4, a passage quoted only to set another against it.** Reader 1 quotes 8.1's "A + [Detector] that predates this document ... MAY be graded [Conforming] on evidence," as what it + read the 8.1 sentence on a declaration with no gap against. As run 22 held, a quotation under + question 4 counts whatever it is quoted for; each run of words either side of the ellipsis lies + in 8.1's second paragraph, at the place the reader names, so it is recorded there. +11. **Question 4, closing punctuation.** Reader 1's "... (TOOLING-SPEC.md)." ends with a full stop + where the document has a comma; reader 2's "... MUST name its kind," and reader 1's "... on + evidence," end with a comma where the document has a comma or a colon. The words are consecutive + and verbatim; the punctuation mark that follows them is not a word, so each is a quotation, as + run 26 held. +12. **The open passages.** The passages earlier runs of this document found confusing by two or + more, and not since cleared, are those run 26 left outstanding: 7.1's first paragraph (runs 20, + 22, 24 and 26) and section 8's paragraph on the kind, which at 1db0c67 begins "A verdict on a + [Detector] MUST name its kind" (runs 24 and 26). The text of each changed after run 26, in + 0b2d190 and e783b64, and each is taken as it reads at 1db0c67. 8.1's first and second paragraphs, + 6.1's paragraph beginning "For a [Bundled rule]" and clause 4.1 were cleared by runs 22, 24 and + 26, and do not reopen when one reader of a later cohort names them. + +No re-mark. + +## Passages quoted as confusing + +Grouped by paragraph, as `ACCEPTANCE.md` defines the same passage. A passage blocks when two or +more readers name it, or when it covers a sentence of a paragraph in which one of the open +passages lay (judgement 12), as that paragraph reads at 1db0c67. + +| Clause and paragraph | Readers | Words quoted, verbatim | Blocks | +| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | +| 8, the paragraph beginning "A verdict on a [Detector] MUST name its kind", its first, second and fifth sentences | 2, 3, 4, 5 | 2: "A verdict on a [Detector] MUST name its kind," and "the kind is not a level of [Conformance] and does not change the verdict." 3: "each configuration the verdict grades" and "configuration". 4: "A verdict on a [Detector] MUST name its kind" and "The kind is not a level of [Conformance] and does not change the verdict." 5: "A verdict on a [Detector] MUST name its kind", "This duty is on whoever gives the verdict, not on the [Detector]" and "the kind is not a level of [Conformance] and does not change the verdict." (judgement 11) | Yes: four readers, and the open section 8 paragraph of runs 24 and 26 | +| 7.1, first paragraph, the sentence beginning "A route is detectable when" | 3 | "A route is detectable when the project can find every use of it, either with a [Rule] of the project's own that the [Detector] runs, or with a mechanical check the [Detector] documents." | Yes: the open 7.1 paragraph of runs 20, 22, 24 and 26 | +| 6.1, the paragraph beginning "For a project's own [Rule]", its first sentence, its sentence beginning "Where the [Identifier] is itself a URL" and its last sentence | 1, 2; 4 by description | 1: "The lookup for such a Rule is an obligation on the project's assembled Toolchain, under clause 4.2 of the [toolchain specification](TOOLING-SPEC.md)." 2: "what it owes is the half it can give: the [Identifier] printed unaltered under clause 4.3, and no transformation of it." 4: no quotation, the references to the other specifications (judgements 9 and 11) | Yes: two readers, three on judgement 9 | +| 6.1, the heading | 2 | "The detector MUST provide a mechanism that resolves a printed identifier to its documentation," (judgement 11) | No; on the other reading of judgement 9, reader 4 too | +| 8.1, first paragraph, the sentence "A declaration that records no gap claims [Conformance]." | 1 | "A declaration that records no gap claims [Conformance]." | No; cleared by run 24 (judgement 12) | +| 8.1, second paragraph, its first sentence | 1 | "A [Detector] that predates this document" and "MAY be graded [Conforming] on evidence," (judgements 10 and 11) | No; cleared by run 26 (judgement 12) | +| 3, the table row "Resolution of a project's own [Rule]" | 5 | "The documentation and, with the [Toolchain], the lookup." | No | +| 6.1, the Why paragraph, its first sentence; 6.4, the Why paragraph, its sentence on the toolchain specification's self-audit; 7.1, the Why paragraph, its first and last sentences; 9, the sentences that refer to the method or toolchain specification | 4 | No quotation; "Clauses 6.1, 6.4, 7.1 and section 9 refer to clauses in `TOOLING-SPEC.md` and `SPEC.md`" (judgement 9) | No | + +Section 8's paragraph on the kind is named by four readers of this cohort, and 6.1's paragraph on a +project's own rule by two, three on judgement 9. Of the open passages, neither is cleared: this +cohort names passages covering sentences of 7.1's first paragraph and of section 8's paragraph on +the kind, as each reads at 1db0c67. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in c6ee1a8 and the commit after +it that settles its review (key quotations kept verbatim in 224579f and that commit). The editor +first read every sheet, in runs 20 to 28, that named each passage, and fixed the cause the sheets +show. DETECTOR-SPEC.md: clause 7.1 says after the two things too that neither requires removing or +forbidding the route, and says what the detector does when the project detects a route with its own +rule; section 8's paragraph on the kind uses "configuration" in one sense and says the kind does not +change the conformance verdict; clause 6.1's paragraph on a project's own rule no longer leans on +cross-references a reader of this document alone cannot follow. A fresh cohort follows. + +## Process change + +Made under the process change of pull request 13 before it merged, at commit 598908b, named +before the readers started; `ACCEPTANCE.md` and `acceptance/runs/README.md` in this worktree +differ from 598908b only by this pull request's question-set line adding question 2d for SPEC.md, +which does not bear on this document. + +## Verdict + +- **Load-bearing statements (every reader on every one)**: not met. 2a four of five (all but + reader 4), 2b two of five (readers 2 and 5), 2c five of five. +- **Restatement and application (all but one on each keyed item)**: met. Question 1 four of five; + every clause of question 3, the section 8 verdict and 6.4 five of five. +- **A miss traceable to the text (none)**: not met. Four, each a finding: 2b by readers 1, 3 and 4, + each quoting 7.1 from "it MUST do at least one of two things", none stating that the detector + need not remove or forbid the route; and 2a by reader 4, quoting section 8's "It is judged on the + evidence, and a declaration under clause 8.1 is optional." and not stating that conformance is + satisfying every MUST in sections 4 to 7 (judgement 7). +- **A passage found confusing by two or more (none outstanding)**: not met. Section 8's paragraph on + the kind is named by readers 2, 3, 4 and 5, and 6.1's paragraph beginning "For a project's own + [Rule]" by readers 1 and 2 (and 4, judgement 9); the open passages in 7.1's first paragraph + (reader 3) and section 8's paragraph on the kind are still outstanding. +- **From one reader only**: 6.1's heading, 8.1's first and second paragraphs, the section 3 table + row on a project's own rule, and the references reader 4 names in 6.1, 6.4, 7.1 and section 9, + recorded. + +FAIL on the first, third and fourth criteria. Answer sheets, the readers' prompt and command, and +the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091000028), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091000433), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091000975). From 00f54d9309d7e6082ef4f49ac59181644c173776 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 00:45:29 +0100 Subject: [PATCH 052/105] Acceptance runs 29 and 30: SPEC.md and DETECTOR-SPEC.md, sixth cohort, both FAIL Five cold Haiku readers each, made under pull request 13 at 598908b and marked by a delegated marker. Evidence is posted on pull request 14 and linked from each record. No findings are applied yet: the recurring misses turn on key elements and the fourth row, which are the owner's to decide. Co-Authored-By: Claude Opus 5.5 --- .../runs/029-semantic-detectors-run-11.md | 267 +++++++++++++++++ .../runs/030-semantic-detectors-run-12.md | 268 ++++++++++++++++++ 2 files changed, 535 insertions(+) create mode 100644 acceptance/runs/029-semantic-detectors-run-11.md create mode 100644 acceptance/runs/030-semantic-detectors-run-12.md diff --git a/acceptance/runs/029-semantic-detectors-run-11.md b/acceptance/runs/029-semantic-detectors-run-11.md new file mode 100644 index 0000000..3917fc6 --- /dev/null +++ b/acceptance/runs/029-semantic-detectors-run-11.md @@ -0,0 +1,267 @@ +# Run 29: SPEC.md, semantic detectors, eleventh run in pull request 14 + +- **Document**: SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + branch `editor/semantic-detectors` at db47301, the commit the readers were given; the question + set, key and fixture read at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input; no tools, no settings, no project context and no + saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is byte for byte the question set, the document and the + fixture at db47301, in that order, each preceded by one `===== FILE: =====` line, with + nothing else added apart from one blank line between the end of each file and the next file's + marker line. +- **Managed (policy) settings**: none; the machine applies no managed settings to the readers and + no managed settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. Every sheet has answer text, no process reports an error + (`is_error` false, subtype `success`), and every reported model is the cohort's model. The + editor reports no reader set aside for a cause the sheets cannot show. +- **Run within the pull request**: 11, counting void runs; runs in the pull request so far, + counting void runs: 11, as the editor supplied. Run 30 is made at the same time and does not + bear on this document. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review. It read `ACCEPTANCE.md` and + `acceptance/runs/README.md` in this worktree, which differ from commit 598908b only by this pull + request's question-set line adding 2d for SPEC.md; the document, question set, key and fixture + at db47301; run 27 in full, the passages, judgements and verdict of run 25, the passage history + of runs 19, 21 and 23, and the headers of runs 20, 22, 24, 26 and 28, which are on + DETECTOR-SPEC.md; and, for the fourth pass-criteria row, the change to SPEC.md between 1db0c67, + the document run 27 read, and db47301. + +## Marks + +`ok` is correct, `miss` is a miss; `T` marks a miss traceable to the text. Judgements J1 to J8 +are set out below. + +| Reader | Q1 | 2a | 2b | 2c | 2d | 3.1 | 3.2 | 3.3 | 3.4 | 3.5 | 3.6 | Remediation | Defence | Section 4 | Q4 passages | +| ------ | ------- | --- | ------ | --- | ------ | ------- | ------- | ------ | --- | --- | --- | ----------- | ------- | --------- | ----------- | +| 1 | correct | ok | ok | ok | ok | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 2 | correct | ok | ok | ok | miss T | Partial | Partial | Yes ok | No | No | No | ok | ok | ok | three | +| 3 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | four | +| 4 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 5 | correct | ok | miss T | ok | ok | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | + +Every grade under 3.1, 3.2, 3.4, 3.5 and 3.6 is one the key accepts. No reader grades 3.2, 3.5 or +3.6 Yes, so none of the key's cases that count against the document arises. + +### Question 1 + +Five of five correct. Every sheet has at least four bullets naming obligations the key lists (the +class bounded both ways with its hazard, the strongest kind of detector and never a test, the +proof in a commit of its own, the sweep with its count and the fix of every instance, permanent +and blocking, the terse message with an identifier resolving to versioned documentation, the +record of a deferred defect), and no bullet asserts an obligation the document does not state. +The bullets of readers 1, 2, 4 and 5 on attempting rather than pre-judging, or on recording a +no-pattern conclusion in one sentence naming two techniques, state section 2's obligations; reader +5's closing "fix the original defect conventionally with a reproduction test" is section 2 and +section 6. + +### Question 2 + +Every reader quoted words of the document for every item, so every miss here is traceable to the +text. + +- **2a**, five of five. Each states class, rule, proof, sweep and the fix of every instance, in + that order, and quotes section 2's sentence "The order is section 3's: attribute the [Defect] to + a [Class], build the [Rule], prove that it fires, run the [Sweep], then fix every [Instance]." +- **2b**, four of five. Readers 1 to 4 say only the owner may and an agent may not, and quote the + key's section 4 sentence; readers 1, 3 and 4 also quote "The [Owner] is always a human." Reader + 5 says "Only a human may adopt a baseline or suppress an instance", quoting the section 4 + sentence and, as an addition, "The [Owner] is always a human."; it does not say the owner is + who may (J4). Miss, traceable. +- **2c**, five of five. Each states all six clauses and a verdict resting on reproduction, and + each quotes section 7's definition of a conforming remediation or its "MUST rest on + reproduction, not on the report"; no reader accepts the pull request description as the + verdict. Reader 5's withdrawn fragment before its quotation is read under J1. +- **2d**, two of five (readers 1 and 5). The key's three elements are (i) the strongest kind, in + the order semantic, syntactic, textual; (ii) both grounds for a weaker kind; (iii) the reason + recorded, with at least one place named and the places not given as alternatives (J2). + - Reader 1: (i) in its quotation of the strongest-kind sentence; (ii) in its sentence, both + grounds; (iii) in its sentence and its quotation of the record sentence, both places joined + by "and". Correct. + - Reader 2: (i) in its sentence and quotation; (iii) in its quotation of the record sentence. + (ii) missed: "A weaker kind is allowed only on one of two grounds" states neither ground, and + neither quotation states one. Miss, traceable. + - Reader 3: (ii) in its quotation of "A weaker kind is allowed only where ... those are the two + grounds."; (iii) in its quotation of the record sentence. (i) missed: "the strongest kind of + detector that can express it" names no kind, and "weaker than [Semantic]" names the top of the + order, not the order (J3). Miss, traceable. + - Reader 4: (ii) in its quotation of the grounds sentence; (iii) in its sentence, "in the rule's + documentation and in the commit". (i) missed as for reader 3: no kind and no order in its + sentence or quotation (J3). Miss, traceable. + - Reader 5: (i), (ii) and (iii) in its three quotations, the strongest-kind sentence, the + grounds sentence and the record sentence. Correct. + +### Question 3 + +- **3.1**: readers 1, 3, 4 and 5 No, reader 2 Partial, each noting that no two independent search + techniques are recorded. +- **3.2**: five of five Partial, each saying the detector or its kind is not named on the + documentation and that the rule is not a test; every reader adds that no grounds for a weaker + kind are recorded, which the key does not mark wrong. +- **3.3**: five of five Yes. Readers 1, 2, 4 and 5 say the inline ignore is not judged here; none + attributes it to 3.3. +- **3.4** and **3.5**: five of five No, each naming the inline ignore on `Report::render()`, under + 3.5 as uncovered by any recorded decision. +- **3.6**: five of five No, each on the wiki documentation not versioned with the rule. +- **Remediation**: five of five say it does not conform, each naming 3.4 among the failing clauses + (J7). +- **Defence**: five of five say it does not conform, each naming 3.6 as failing, reader 3 by its + ground, "its documentation be versioned alongside it" (J8); readers 1, 2 and 5 also name 3.2 as + not met in full. Every reader also counts 3.5, and most 3.1, against the defence; the key does + not mark that wrong. +- **Section 4**: five of five name the inline ignore on `Report::render()`, and each also names + leaving the next wider rule unbuilt, which the key accepts alongside it. + +### Judgements made in marking + +Made as a careful reader of the document would, as `ACCEPTANCE.md` requires. Where a rule settles +the point, the record says which. + +- **J1. The sentence.** Where a reader wrote more than one sentence before or around its + quotations, all of that text is read with the quotations as the answer. Reader 5's 2c opens with + a fragment it withdraws in the same line ("... is not the quote here"); the withdrawn fragment + is not read as part of the answer. No mark turns on it. +- **J2. 2d's element (iii) needs a place named**, as runs 19 to 27 read the key's "naming one of + the two places is enough". Every reader names both places this run, so no mark turns on it. +- **J3. 2d's element (i) needs the order (readers 3 and 4).** The key's element is "the strongest + kind that can express the rule, in the order semantic, syntactic, textual", and the question + asks which kind. "The strongest kind of detector that can express it" names no kind, and + "weaker than [Semantic]" in a record quotation names the top of the order, not the order. Runs + 23 and 25 read it the same way. Both readers give the order under question 1, but each keyed + item is marked on its own answer. Read otherwise, readers 3 and 4 would be correct on 2d; 2d + would still fail on reader 2, whose miss on (ii) rests on no judgement, and the verdict would not + change. +- **J4. "Only a human" (reader 5, 2b).** The key's answer is the owner of the codebase, always a + human, and an answer that omits the owner is wrong. Reader 5's sentence names no one but "a + human", which on its face includes a human practitioner, to whom section 4 gives none of these + decisions; its added quotation "The [Owner] is always a human." says the owner is human, not + that the owner is who may. Read together, the answer does not state that the decision is the + owner's, so it omits the owner. Read otherwise, as naming the owner through the quotation it + gives in support, 2b would be five of five and the verdict would not change, since 2d fails on + reader 2 without a judgement. +- **J5. Passages a reader quotes or sets beside its main one count.** Where a reader's question 4 + item quotes words of the document at a named place, or says it was confused by two passages read + together, a careful reader takes each as a passage it names, since the confusion lies in neither + alone, as runs 23, 25 and 27 read it. So reader 1's section 2 item also covers clause 3.2's + paragraph beginning "There are two kinds of gap", where its quoted "mechanism gap" and "the gap + above" occur in clause 3.2, the first in the same sentence reader 3 quotes; reader 2's section 2 + item, which says "clause 3.1 uses it for searching for instances", covers clause 3.1's sentence + "In this clause a technique is a way of searching the codebase for [Instances].", as run 27 read + that reader's like description; reader 3's and reader 4's items each cover both passages they + quote. Two findings turn on this. Read as context only, reader 1's clause 3.2 passage is one + reader's, not two, and run 25's clause 3.1 passage is cleared; the verdict would not change. +- **J6. Run 25's clause 3.1 paragraph as it reads in the changed document.** The paragraph that + began "Two techniques are independent when they would miss different things." is now three + paragraphs, beginning "In this clause a technique is a way of searching", "A technique counts + only if" and "The record MUST name both techniques", the last kept verbatim from it. All three + are taken as that paragraph as it reads now. No mark turns on it: reader 2's description names a + sentence of the first. +- **J7. The remediation item.** The key's answer is that the remediation does not conform, giving + 3.2, 3.4 and 3.6 as the reason. A reader saying it does not conform and naming among the failing + clauses one the key names is correct, as runs 23, 25 and 27 marked it; every reader names 3.4, + and every reader 3.6. +- **J8. The defence item (reader 3).** The key marks correct a reader who names 3.2 or 3.6. Reader + 3 names 3.5 by number and 3.6 by its requirement, that the documentation "be versioned alongside + it", having graded 3.6 No on that ground; it names 3.6's failure, and is correct. + +## Passages quoted as confusing + +Same passage as `ACCEPTANCE.md` defines it: sentences in common, or the same paragraph; a list +item, table row or terminology entry is its own paragraph. + +| Passage | Clause | Readers | Words quoted, verbatim, or the description used | +| ---------------------------------------------------------- | --------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Independent techniques for expressing a rule | 2 | two: 2, 5 | 2: "A technique here is a way of trying to write the [Rule]. Two such techniques are independent when they would miss different things, which is the same test clause 3.1 applies to ways of searching for [Instances]." 5: "The conclusion that no pattern exists MUST be stated in one sentence, naming at least two independent techniques tried at expressing the [Rule], and recorded where the project's other decisions are enumerable under clause 8.7, not only with the fix." Both in the "Attempt rather than pre-judge" paragraph | +| The entry point and substitutes for a bundled defence | 3.5 | two: 2, 3 | 2: "A mechanism gap that stops the [Toolchain] running its own entry point on its own source moves the demonstration to such a project; it does not weaken it." 3: "The substitutes clause 3.2 allows, for proving under clause 3.3 and for resolution under clause 3.6, do not extend to this clause." Both in the paragraph beginning "A [Bundled defence] is a [Defence] that a shipped [Toolchain] enables by default"; reader 3 sets it beside the clause 3.2 passage below | +| The two kinds of gap | 3.2 | two: 1, 3 | 3: "uses the substitutes clauses 3.3 and 3.6 already allow". 1: "mechanism gap" and "the gap above", attributed to clause 3.2 (J5). Both in the paragraph beginning "There are two kinds of gap and they lead to different work" | +| The three ways out, a toolchain gap | 2 | one: 1 | "A pattern exists but the language has no extensible [Detector] of a kind that can express the [Rule] within clause 3.1's bounds, and a bespoke one is not practical: a [Toolchain] gap under clause 3.2.", in the paragraph beginning "There are three ways out of the method", set beside clause 3.2's paragraph above (J5) | +| A technique in clause 3.1 | 3.1 | one: 2 | By description, "clause 3.1 uses it for searching for instances", set beside its section 2 passage above: the sentence "In this clause a technique is a way of searching the codebase for [Instances]." (J5, J6). Run 25's open passage (below) | +| An assumed calibration against escalating doubt | 3 and 3.1 | one: 3 | "**Where the project has recorded nothing, the [Practitioner] is not blocked.** Proceed on the examples below, **state the [Calibration] assumed**", in section 3's paragraph of that name, set beside "Where the [Practitioner] is not sure, the decision goes to the [Owner] under section 4.", in clause 3.1's lower-bound list item | +| Merging at full width against not blocking until every fix | 4 | one: 4 | "the [Rule] MAY merge at full width with that case recorded as a known [Instance] awaiting the [Owner]", in the paragraph beginning "**What to do whilst waiting.**", set beside "provided the [Rule] does not become [Blocking] until every [Instance] is fixed", in the paragraph beginning "A remediation MAY be delivered across several changes" | +| Leaving the next wider rule unbuilt | 4 | one: 5 | "It is a reason not to widen unasked, not authority to leave the wider [Rule] out, and recording it as a known gap does not make it the [Practitioner]'s decision.", in section 4's list item beginning "Leaving unbuilt the next wider [Rule]" | + +Readers' accounts of what they could not tell are recorded with the passage they attach to and are +not suggestions; none of the sheets proposes a change. + +**Earlier passages (the fourth row).** The open passages are those earlier runs of this document +found confusing by two or more and did not clear: run 27's two new passages and run 25's clause +3.1 passage, which run 27 did not clear; one of run 27's two is run 21's clause 3.2 passage, +which runs 23, 25 and 27 did not clear. Every other passage of runs 19, 21, 23 and 25 was cleared +in runs 21, 23, 25 and 27. As the paragraphs read in the changed document: + +- Clause 3.2's grounds and record, run 21's passage: the paragraph beginning "**Qualifying is not + the same as being the right host.**", the paragraph beginning "For the second ground", and the + record paragraph beginning "On the [Remediation docs] of a [Rule] they write". Named by no + reader. Readers 1 and 3 name clause 3.2's paragraph beginning "There are two kinds of gap", + which is not one of these. Cleared. +- Section 7's paragraph beginning "**A [Defence] [Conforms]**", with clause 3.5's opening + paragraph, run 27's passage: named by no reader. Readers 2 and 3 name clause 3.5's \[Bundled + defence\] paragraph, not its opening paragraph. Cleared. +- Clause 3.1's paragraph that began "Two techniques are independent when they would miss + different things.", run 25's passage, now the three paragraphs beginning "In this clause a + technique is a way of searching", "A technique counts only if" and "The record MUST name both + techniques" (J6): named by reader 2 by description (J5). Not cleared. + +No sentence kept from a cleared paragraph is named elsewhere in the document. Four passages named +here were once open and have since been cleared: section 2's "Attempt rather than pre-judge" +paragraph (run 19, two readers; cleared in run 21), named here by readers 2 and 5; clause 3.5's +[Bundled defence] paragraph (run 25, two readers; cleared in run 27), named here by readers 2 and +3; clause 3.1's lower-bound list item (run 23, three readers; cleared in run 25), named here by +reader 3; and section 4's list item on leaving the next wider rule unbuilt (run 21, three readers; +cleared in run 23), named here by reader 5. The first two are named by two readers and are new +findings here; the last two are recorded as one reader's. Clause 3.2's paragraph on the two kinds +of gap, named by one reader in run 25, is named by two here and is a new finding. + +## Findings applied + +None yet. After six cohorts on this pull request the editor has stopped to put a decision to the +owner before changing the text again: the misses that recur in every run (question 2d's order and +grounds, and on the detector document question 2b's "need not remove or forbid") turn on elements of +the keys that the questions do not ask for directly, and on the fourth row keeping a passage open +while any single reader of a fresh cohort names it. Neither can be changed by the editor, since a +question, a key and a threshold may not be changed to let a run pass. + +## Process change + +Made under a process change before it merged: yes, pull request 13 at commit 598908b, named +before the readers started. + +## Verdict + +**FAIL.** Against the criteria in `ACCEPTANCE.md`: + +- Load-bearing statements: not met. 2a and 2c five of five; 2b four of five (reader 5, on J4); 2d + two of five. Reader 2 states neither ground for a weaker kind, only that there are two; readers + 3 and 4 do not give the order semantic, syntactic, textual (J3). +- Restatement and application: met. Question 1 and every keyed item of question 3 five of five. +- Misses traceable to the text, each a finding: 2d for reader 2, quoting clause 3.2's + strongest-kind and record sentences and saying the grounds are two without stating them; 2d for + readers 3 and 4, quoting clause 3.2's sentence "A weaker kind is allowed only where the stronger + kind cannot express the [Rule], or where the language has no [Detector] of the stronger kind + that accepts bespoke [Rules]; those are the two grounds." and giving the strongest kind without + the order (on J3); 2b for reader 5, quoting section 4's sentence on an agent's authority and + saying only "a human" may, without the owner (on J4). +- Passages found confusing by two or more, each a finding that blocks until the text is changed + and a fresh cohort no longer names it: section 2's "Attempt rather than pre-judge" paragraph, on + techniques tried at expressing a rule (two readers); clause 3.5's paragraph beginning "A + [Bundled defence] is a [Defence] that a shipped [Toolchain] enables by default", on the moved + demonstration and the substitutes that do not extend to it (two); clause 3.2's paragraph + beginning "There are two kinds of gap and they lead to different work" (two, one on J5). Run + 25's clause 3.1 paragraph on independent techniques is not cleared (one reader, on J5). Run 21's + clause 3.2 passage on the grounds and their record, and run 27's section 7 and clause 3.5 + passage, are cleared. +- Single-reader notes, recorded: section 2's "There are three ways out of the method" paragraph + (reader 1); section 3's "Where the project has recorded nothing" paragraph against clause 3.1's + lower-bound list item (reader 3); section 4's "What to do whilst waiting" paragraph against its + paragraph on delivering a remediation across several changes (reader 4); section 4's list item + on leaving the next wider rule unbuilt (reader 5). + +Answer sheets, the reader prompt, the command and the marker's instruction: posted on pull request 14 in posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091186417), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091186696), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091187011), [comment 4](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091187459). diff --git a/acceptance/runs/030-semantic-detectors-run-12.md b/acceptance/runs/030-semantic-detectors-run-12.md new file mode 100644 index 0000000..7005da5 --- /dev/null +++ b/acceptance/runs/030-semantic-detectors-run-12.md @@ -0,0 +1,268 @@ +# Run 30: DETECTOR-SPEC.md, semantic detectors, twelfth run in pull request 14 + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + editor's branch `editor/semantic-detectors` at db47301; the question set, key and fixture read + at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input, with no tools, no settings, no project context + and no saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is byte for byte the question set, the document and the + fixture at db47301, in that order, each preceded by one `===== FILE: =====` line, with the + files separated by one blank line and nothing else added, the same form as the packets of runs + 22, 24, 26 and 28 +- **Managed (policy) settings**: none; the machine applies no managed settings and no managed + settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. The editor reports none set aside for a cause the sheets cannot + show; every sheet has answer text, no process reports an error (`is_error` false, subtype + `success`), and every reported model is the cohort's model +- **Run within the pull request**: 12, counting void runs; run 29, made at the same time, does not + bear on this document +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review + +## Marks + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | Section 8 | Kind | 6.4 | Q4 entries | +| ------ | ------- | --- | ---- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --------- | ---------- | --- | ---------- | +| 1 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 2 | miss | ok | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 3 | correct | ok | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 4 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 4 | +| 5 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | + +"Q4 entries" counts the entries each reader lists under question 4; the words each quotes are in +the table of passages below. Every miss under question 2 is traceable to the text: each answer's +sentence states what the document requires, and each quotes the document in support. The miss +under question 1 is the reader's own: the bullet it turns on quotes nothing. No question was left +unanswered. + +### Question 1 + +Every restatement names four or more obligations from the key's list. Readers 1, 3, 4 and 5 assert +no obligation the document does not state: correct. Reader 1's sixth bullet and reader 5's sixth +name the duty to name the kind on a verdict, which the key marks correct; reader 3's sixth and +reader 5's sixth also say what conformance means, which names no obligation and asserts none the +document does not state. Readers 1, 3 and 5 say a printed identifier must resolve to its +documentation, in the generality of 6.1's heading, as run 28 held (judgement 4). Readers 1, 2, 3, 4 +and 5 say that each rule carries, needs or has a stable identifier (judgement 3). Reader 3's "put +the results in the output of the command the user ran" and reader 5's "findings must appear in the +output of the command the practitioner ran" are plain plurals, as run 26 held. Reader 2's fifth +bullet says "Each finding, with its location and identifier, must appear in the output of the +command the practitioner ran (5.3)"; 5.3 says "The summary need not list every finding", so the +bullet asserts an obligation the document does not state: incorrect (judgement 2). Correct, four of +five. + +### Question 2 + +The key's answers state these elements: + +- 2a: (a) Conformance is satisfying every MUST in sections 4 to 7; (b) judged on evidence; (c) a + detector that predates the document can conform; (d) the declaration is optional. +- 2b: (a) make the route disableable by configuration, or (b) detectable by a Rule the project can + write or a mechanical check the Detector documents; (c) the detector is not required to remove or + forbid it. +- 2c: (a) with the rule; (b) in the installed copy; (c) at a version tracked together; (d) yes, + reachable offline. + +Each answer is read as its sentence and its quotation together, for the gist of each element. + +| Reader | 2a | 2b | 2c | +| ------ | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | +| 1 | ok: all four; (b) and (d) from the 8.1 quotation | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6) | ok: all four; (a) and (c) from the 6.3 quotation | +| 2 | ok: all four; (b) and (d) from the 8.1 quotation | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6); its quotation drops a word (judgement 8) | miss: (c) not stated; only 6.2 quoted (judgement 9) | +| 3 | ok: all four; (b) and (d) from the section 8 quotation | miss: (c) not stated (judgement 5); (b) as "make every use of it detectable" | miss: (c) not stated; only 6.2 quoted (judgement 9) | +| 4 | ok: all four; (d) as "without a declaration", (b) as "the verdict rests on checking the detector" | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6) | ok: all four; (a) and (c) from the 6.3 quotation | +| 5 | ok: all four; (b) and (d) from the 8.1 quotation | miss: (c) stated only as "does not have to remove the route" (judgement 7); (b) by the defined term | ok: all four; (a) and (c) from the 6.3 quotation, (b) from the 6.2 quotation | + +No answer asserts anything the document does not state or that the key says makes an answer +wrong: none says a declaration is required, that partial satisfaction is conformance, that the +detector must forbid the route or require a reason, or that a website will do. Every quotation but +reader 2's under 2b is verbatim apart from letter case, link and emphasis markup and a closing +punctuation mark. + +### Question 3 + +The key's intended verdicts are Yes for every clause but 6.3, which is No; the section 8 verdict +is "does not conform", because 6.3 fails; 6.4 does not change the verdict. + +- **Every clause**: all five readers give the intended verdict on every clause, 4.1 to 7.1. On + 6.3 each cites that one unresolved identifier fails the clause, all five quoting 6.3's "One + [Identifier] that does not resolve fails this clause, however many others do"; no reader grades + it Partial on the grounds that thirty-eight resolve. Readers 1 to 5 add caveats on 4.2 or 4.3 + about what the fixture does not say, and grade Yes. +- **Section 8**: all five say Lintomatic does not conform, because 6.3 fails. +- **Kind**: all five say the fixture does not let them name the kind and name none, so none is + marked on it. Readers 2, 3 and 5 call the verdict short of that duty; none makes the verdict turn + on the kind, and all five say it does not change the verdict or the conformance finding. +- **6.4**: all five say it does not change the verdict, because it is a SHOULD. Reader 3 adds + that 7.2 is an unmet SHOULD and that a declaration would record the 6.3 gap (judgement 10). + +### Judgements made in marking + +01. **How an open judgement is made.** As `ACCEPTANCE.md` at 598908b says: as a careful reader of + the document would, with the reason stated. Where a rule settles a mark, it is applied as + written. +02. **Question 1, reader 2's "Each finding ... must appear".** Runs 22, 24 and 28 marked incorrect + bullets saying every finding, or each finding, must appear in the output, because 5.3 says "The + summary need not list every finding" and "If it leaves any finding out, the output MUST also + say where the rest can be read". Reader 2's bullet is the same claim about each finding. + Incorrect. The bullet quotes nothing, so the miss is the reader's. Reader 2 also gives seven + bullets where the question asks for at most six; the key does not make that wrong, and the mark + does not turn on it. +03. **Question 1, "each rule carries a stable identifier".** Clause 4.3's heading asks the detector + to "allow a rule to carry a stable identifier", and its body says "The [Detector] MUST print it, + unaltered, alongside every finding the [Rule] reports". A detector can print an identifier with + every finding of every rule only if each rule carries one, and the key's own bullet joins the + two. A careful reader takes these bullets as 4.3, not as a further obligation. Correct. Read the + other way, readers 1, 3, 4 and 5 would all miss question 1 and the second criterion would fail + too; the verdict is FAIL either way. +04. **Question 1, "a printed identifier must resolve".** As run 28 held: the bullets speak at the + level of 6.1's heading, "The detector MUST provide a mechanism that resolves a printed + identifier to its documentation", and the key's bullet is in the same words. Correct. +05. **2b (c), not required to remove or forbid, readers 1 to 4.** As runs 22, 24, 26 and 28 held, + the key's answer includes what the detector need not do, so (c) answers a part of the question. + Readers 1, 2, 3 and 4 say the detector must do one of the two things, readers 1, 2 and 4 adding + that one alone is enough. Each quotes 7.1 from "it MUST do at least one of two things" or, in + reader 2's case, from "but MUST do"; none quotes the words before it, "it need not remove or + forbid the route", nor the words after "Either alone satisfies this clause", "neither requires + the [Detector] to remove or forbid the route". That one alone suffices entails that nothing more + is required, but an element reached only by inference from another is not among the ways + `ACCEPTANCE.md` names of stating it. Not stated. +06. **2b (b), the means of detection.** Clause 7.1 defines when a route is detectable in its own + text; readers 1, 2, 4 and 5 use "detectable" in that sense and so state (b), as runs 22 to 28 + held. It decides no mark. +07. **2b (c), reader 5's "It does not have to remove the route."** The element is that the + detector is not required to remove or forbid the route, and 7.1 says so twice in those words. + Removing a route and forbidding it are different things: the first takes the route out of the + detector, the second stops its use whilst it stays; and the answer the key names as wrong is + that the detector "must forbid the route". Reader 5 states the first half and leaves the second + open, and its quotation, from "but it MUST do at least one of two things", does not carry it. + A careful reader does not take half of the element as its gist. Not stated: a miss, traceable, + since the sentence states what the document requires. Read the other way, reader 5 would be + correct on 2b and the third criterion would lose one finding; the verdict does not turn on this + judgement, since 2b fails the first and third criteria by four readers whichever way it is made. +08. **2b, reader 2's quotation.** Reader 2 quotes "but MUST do at least one of two things: ...", + where the document has "but it MUST do". The words from "MUST do" to "detectable" are a run of + the document's consecutive words, verbatim; the quotation as given is not, since it drops a + word. The mark does not turn on it: the answer misses (c) either way, and the miss is traceable + because its sentence states, in the reader's own words, what 7.1 requires. +09. **2c (c), at a version tracked together, readers 2 and 3.** As run 22 held for a reader whose + only quotation was 6.2, element (c) answers where the documentation must live, as part of the + key's answer to that part. Reader 2's sentence says the documentation "must ship with the + installed rule and be reachable offline", reader 3's that it "must ship with the installed rule, + and it must be reachable without network access", and each quotes only 6.2. Each states (a), + (b) and (d); neither says the documentation is versioned with the rule. Not stated: a miss, and + traceable, since each sentence states what the document requires and each quotes 6.2. +10. **Asides outside the keyed answer.** Reader 3's notes on 7.2 and on what a declaration would + record, and the remarks of readers 1, 2, 4 and 5 on Lintomatic missing 6.4, are not keyed + items; each sheet's answer to whether 6.4 changes the verdict is the key's, and is marked so. +11. **Question 4, reader 4's entry on documents not given.** Reader 4 writes "Several terms and + clauses are defined in documents I was not given, for example" and quotes "Terms from the method + specification carry over unchanged" and "the toolchain specification's self-audit" in 6.4. The + entry quotes, and its description names no place in the document: it says what the reader + lacked, and its quotations are the places it names. A careful reader takes the passages as the + two quoted, in section 2's first paragraph and in 6.4's Why paragraph. Read the other way, as + every sentence that uses a term or cites a clause of the other specifications, it would cover + sentences of nearly every paragraph, among them 6.1's paragraph beginning "For a project's own + [Rule]", which would then not be cleared; the fourth criterion fails either way, and the + verdict does not turn on it. +12. **Question 4, closing punctuation.** Reader 4's "... that [Conformance] is judged on." ends with + a full stop where the document has a colon; reader 5's "... in sections 4 to 7." ends with a + full stop where the sentence goes on. The words are consecutive and verbatim; the punctuation + mark that follows them is not a word, so each is a quotation, as runs 26 and 28 held. +13. **Question 4, a passage quoted only to set another against it.** Reader 5 quotes "a declaration + is optional" as what it read 8.1's sentence on a declaration with no gap beside. As runs 22 and + 28 held, a quotation under question 4 counts whatever it is quoted for. The run occurs only in + 8.1's second paragraph, at the place the reader names, and is recorded there. +14. **The open passages, and paragraphs the change split.** The passages earlier runs of this + document found confusing by two or more, and not since cleared, are those run 28 left + outstanding or added: 7.1's first paragraph (runs 20 to 28), section 8's paragraph beginning + "A verdict on a [Detector] MUST name its kind" (runs 24 to 28), and 6.1's paragraph beginning + "For a project's own [Rule]" (run 28, two readers). At 1db0c67, which run 28 read, 7.1's first + paragraph and section 8's paragraph on the kind were each one paragraph; between 1db0c67 and + db47301 the change split each in two, 7.1 at "A route is detectable when", section 8 at "This + MUST is on whoever gives the verdict". `ACCEPTANCE.md` asks whether a reader names a sentence of + the paragraph an earlier passage lay in "as that paragraph reads in the changed document"; a + careful reader takes a split paragraph to read, in the changed document, as the two paragraphs + the change made of it, not as the first of them alone. So taken, a sentence of 7.1's second + paragraph, or of section 8's paragraph beginning "This MUST", covers the open passage. Read the + other way, the open 7.1 passage would be cleared, but 7.1's second paragraph and section 8's + paragraph beginning "This MUST" are each named by two or more readers of this cohort, and + reader 5 names the kept first sentence of section 8's paragraph on the kind, so the fourth + criterion fails either way. 8.1's first and second paragraphs, 6.1's paragraph beginning "For a + [Bundled rule]" and clause 4.1 were cleared by runs 22, 24 and 26, and do not reopen when one + reader of a later cohort names them; 8.1's first paragraph is named by two readers of this + cohort, which is a new finding of this run, not a reopening. + +No re-mark. + +## Passages quoted as confusing + +Grouped by paragraph, as `ACCEPTANCE.md` defines the same passage. A passage blocks when two or +more readers name it, or when it covers a sentence of a paragraph in which one of the open +passages lay (judgement 14), as that paragraph reads at db47301. + +| Clause and paragraph | Readers | Words quoted, verbatim | Blocks | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | +| 7.1, second paragraph, its second and third sentences, "The first is a [Rule] ..." and "That way works only where ..." | 1, 2, 4 | 1: "That way works only where the [Detector] lets such a [Rule] see every use of the route" 2: "The first is a [Rule] the project writes for itself, under clause 4.1, and the [Detector] runs. That way works only where the [Detector] lets such a [Rule] see every use of the route, and the [Detector] need not ship the [Rule] itself." 4: "That way works only where the [Detector] lets such a [Rule] see every use of the route, and the [Detector] need not ship the [Rule] itself." | Yes: three readers, and the open 7.1 paragraph of runs 20 to 28 (judgement 14) | +| 8, the paragraph beginning "This MUST is on whoever gives the verdict", its first sentence; and the paragraph beginning "A verdict on a [Detector] MUST name its kind", its first sentence | 4, 5 | 4: "This MUST is on whoever gives the verdict, not on the [Detector], so it is not one of the MUSTs in sections 4 to 7 that [Conformance] is judged on." 5: "A verdict on a [Detector] MUST name its kind, [Semantic], [Syntactic] or [Textual], in each configuration the verdict grades." and "This MUST is on whoever gives the verdict, not on the [Detector], so it is not one of the MUSTs in sections 4 to 7." (judgement 12) | Yes: two readers, and the open section 8 paragraph of runs 24 to 28 (judgement 14) | +| 8.1, first paragraph, its third sentence, "A gap is a MUST ...", and its last but one, "A declaration that records no gap claims [Conformance]." | 4, 5 | 4: "A gap is a MUST in sections 4 to 7 that the [Detector] fails." 5: "A declaration that records no gap claims [Conformance]." | Yes: two readers, in the same paragraph | +| 8.1, second paragraph, its first sentence | 5 | "a declaration is optional" (judgement 13) | No; cleared by run 26 (judgement 14) | +| 4.3, first paragraph, the sentence beginning "The [Detector] MUST print it, unaltered" | 3 | "The [Detector] MUST print it, unaltered, alongside every finding the [Rule] reports, in its default output and in every machine-readable format it offers." | No; one reader of run 26 also named it | +| 2, first paragraph, its first sentence | 4 | "Terms from the method specification carry over unchanged" (judgement 11) | No | +| 6.4, the Why paragraph, its sentence beginning "Where the [Detector] is shipped inside a [Toolchain]" | 4 | "the toolchain specification's self-audit" (judgement 11) | No | + +7.1's second paragraph is named by three readers of this cohort, section 8's paragraph beginning +"This MUST is on whoever gives the verdict" by two, and 8.1's first paragraph by two. Of the open +passages, 6.1's paragraph beginning "For a project's own [Rule]" is cleared: its text changed after +run 28 and no reader of this cohort names a passage covering a sentence of it, on the reading of +judgement 11. 7.1's first paragraph and section 8's paragraph on the kind, as each reads at +db47301, are still outstanding (judgement 14). + +## Findings applied + +None yet. After six cohorts on this pull request the editor has stopped to put a decision to the +owner before changing the text again: the misses that recur in every run (question 2d's order and +grounds, and on the detector document question 2b's "need not remove or forbid") turn on elements of +the keys that the questions do not ask for directly, and on the fourth row keeping a passage open +while any single reader of a fresh cohort names it. Neither can be changed by the editor, since a +question, a key and a threshold may not be changed to let a run pass. + +## Process change + +Made under the process change of pull request 13 before it merged, at commit 598908b, named +before the readers started; `ACCEPTANCE.md` and `acceptance/runs/README.md` in this worktree +differ from 598908b only by this pull request's question-set line adding question 2d for SPEC.md, +which does not bear on this document. + +## Verdict + +- **Load-bearing statements (every reader on every one)**: not met. 2a five of five, 2b none of + five, 2c three of five (readers 1, 4 and 5). +- **Restatement and application (all but one on each keyed item)**: met. Question 1 four of five; + every clause of question 3, the section 8 verdict and 6.4 five of five. +- **A miss traceable to the text (none)**: not met. Seven, each a finding: 2b by readers 1, 2, 3 + and 4, each quoting 7.1 from "it MUST do at least one of two things" (reader 2 from "but MUST + do"), none stating that the detector need not remove or forbid the route; 2b by reader 5, saying + only that the detector "does not have to remove the route" (judgement 7); and 2c by readers 2 and + 3, each quoting only 6.2 and not stating that the documentation ships at a version tracked + together with the rule (judgement 9). +- **A passage found confusing by two or more (none outstanding)**: not met. 7.1's second paragraph + is named by readers 1, 2 and 4; section 8's paragraph beginning "This MUST is on whoever gives the + verdict" by readers 4 and 5; 8.1's first paragraph by readers 4 and 5. The open passages in 7.1's + first paragraph and section 8's paragraph on the kind are still outstanding (judgement 14); the + open passage in 6.1's paragraph on a project's own rule is cleared. +- **From one reader only**: 8.1's second paragraph, the 4.3 sentence on machine-readable formats, + section 2's first sentence and 6.4's Why sentence on the toolchain specification's self-audit, + recorded. + +FAIL on the first, third and fourth criteria. Answer sheets, the readers' prompt and command, and +the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091188050), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091188377), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091188758). From d88ab30271a9bee188302d19a691780717926e57 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 00:47:46 +0100 Subject: [PATCH 053/105] Acceptance process: name runs 19 to 30 as made under this change Co-Authored-By: Claude Opus 5.5 --- acceptance/CHANGES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index eb74588..18d028b 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -6,7 +6,7 @@ not a specification and is not versioned, and a change to it is not tested by a to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). - [Pull request #13](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/13): the cohort can be run and marked without judgement calls that the rules - leave open; this applies to runs made after it merges, and to runs 19 and 20, which were made under it at a695010, and + leave open; this applies to runs made after it merges, and to runs 19 to 30, which were made under it at the commits their records name, and runs 16 to 18 stand as recorded. A cohort is an odd number of readers started, at least five, larger at the editor's choice the more obligations the change touches, each a separate process with no tools and no project context, started from From 814567b9735140e02434372843d3eb2f063d0385 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 10:51:03 +0100 Subject: [PATCH 054/105] Acceptance process: an earlier run's judgement does not bind where its rule has changed Run 22 made a marking judgement by the stricter reading, a tie-break bdfd055 had already replaced with a careful reader's, and runs 24 to 30 followed it as precedent. The marking section now says a judgement an earlier run recorded does not bind a later run where a rule it rested on has since changed: the marker makes it afresh and the record names the rule. This corrects a misapplied rule and relaxes none. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 4 +++- acceptance/CHANGES.md | 7 +++++-- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 43adb33..eacf840 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -153,7 +153,9 @@ which rule and why. Each sheet is marked question by question, as follows. paragraph. Where a mark needs a judgement these rules do not settle, the marker makes it as a careful reader -of the document would, and the record states the judgement and why. +of the document would, and the record states the judgement and why. A judgement an earlier run +recorded does not bind a later run where a rule it rested on has since changed: the marker makes +it afresh under the rules as they now stand, and the record names the rule that changed. ## Pass criteria diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 18d028b..8735c63 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -18,7 +18,8 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). sheet's answered items counted too, or if that sheet has a miss traceable to the text; a void run has a number, a record and posted sheets. The packet is the question set, the document and the fixture, each after one `===== FILE: =====` line. A new marking section says what a delegated marker receives and that its verdict is final unless a rule was misapplied, that an open judgement is - made as a careful reader of the document would, and defines a keyed item, quoting (consecutive + made as a careful reader of the document would, that an earlier run's judgement does not bind + a later run where a rule it rested on has since changed, and defines a keyed item, quoting (consecutive words of any length, verbatim apart from case, quotation-mark style and markup), a correct load-bearing answer, read with its quotation, asserting nothing the document does not state and omitting no element of the key's answer that answers the question, each read for its @@ -45,6 +46,8 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). named no fixture in what a reader receives whilst `acceptance/README.md` did, the thresholds were written as five of five and four of five for a cohort of at least five, two pass-criteria rows both covered a single reader's traceable miss, and nothing settled an unanswered question - or "the same passage". [Run + or "the same passage". The rule on an earlier run's judgement was prompted by run 22, which + made a judgement by the stricter reading, a tie-break this change had already replaced with a + careful reader's, and which later runs of pull request 14 then followed as precedent. [Run 16](runs/016-deferral-clause-run-1.md), [run 17](runs/017-deferral-clause-run-2.md) and [run 18](runs/018-deferral-clause-run-3.md) ruled on these case by case. From b9e5a3fb4e77f29f8c5ba08fef8a3c6a58f8cf24 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 10:54:56 +0100 Subject: [PATCH 055/105] Method 2, 3.1, 3.2, 3.5 and 4, detector 7.1, 8, 8.1 and 9: runs 29 and 30 findings applied Method: section 2's no-pattern sentence names two independent ways of writing the rule that were tried, not techniques, which clause 3.1 now reserves for ways of searching; section 4 stops using the word otherwise. Clause 3.2's two kinds of gap each get a sentence and the substitutes for a missing mechanism are named; clause 3.5's bundled-defence paragraph says in shorter sentences where the demonstration is made and that those substitutes do not reach it. Detector: clause 7.1 says what a project's rule must be able to match for a route to be detectable that way. The duty to name a detector's kind moves, unchanged, to a clause 8.2 of its own, out of the conformance test, binding whoever gives the verdict; clause 8.1 says a declaration is the maintainer's claim and a detector with none claims nothing. The detector key's 2b quotation follows 7.1, and its references to the kind follow it to 8.2. No obligation is weakened. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 23 +++++++++++++-- DETECTOR-SPEC.md | 51 ++++++++++++++++++-------------- SPEC.md | 59 +++++++++++++++++++++----------------- acceptance/detector/KEY.md | 10 ++++--- 4 files changed, 88 insertions(+), 55 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c6f0876..f25d648 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -77,7 +77,16 @@ version entry with no cohort line has not been accepted and is not published. permanence and the recorded decision behind every suppression; section [2](SPEC.md#2-when-the-method-applies) and clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class) each say what a technique is there, and clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s paragraph on independent techniques is three short paragraphs, saying that a run of the rule belongs to the proof or the sweep. Each after readers - of run 27 missed or named the passage. Accepted by run NNN. Minor version when released. + of run 27 missed or named the passage. After run 29, clarity edits: section [2](SPEC.md#2-when-the-method-applies)'s sentence that + no pattern exists names two independent ways of writing the rule that were tried, in place of + "techniques", and says these are not clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s techniques; the MUST it carries is reworded, not + changed. Clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class) says a technique is a way of searching for instances wherever the + specification uses the word, and section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) no longer uses it otherwise; clause [3.2](SPEC.md#32-build-the-net)'s + paragraph on the two kinds of gap gives each kind a sentence of its own and names the + substitutes clauses [3.3](SPEC.md#33-prove-the-net-by-making-the-rule-fire) and [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) accept for a missing mechanism; and clause [3.5](SPEC.md#35-enforce-permanently-and-block)'s paragraph on a + bundled defence says in shorter sentences where its demonstration is made and that those + substitutes give no way round it. Each after two readers of run 29 named the passage. Accepted + by run NNN. Minor version when released. ### 1.1.0, 2026-10-02 @@ -139,7 +148,7 @@ pass it. ### Unreleased -- New obligation, section [8](DETECTOR-SPEC.md#8-conformance): a verdict on a detector MUST name the kind, semantic, syntactic +- New obligation, clause [8.2](DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades): a verdict on a detector MUST name the kind, semantic, syntactic or textual, of each configuration it grades, as the detector has it without any hosted service or licence tier. It is not a level of conformance and does not change the verdict; it is stated because method clause [3.2](SPEC.md#32-build-the-net) now requires a rule to be written in the strongest kind of @@ -181,7 +190,15 @@ pass it. needing the other specifications; section [8](DETECTOR-SPEC.md#8-conformance)'s paragraph on the kind says what a configuration is and, in a paragraph of its own, that the duty is not one of the MUSTs conformance is judged on and that a verdict leaving the kind out still stands on conformance. - Each after readers of run 28 missed or named the passage. Accepted by run NNN. Minor version + Each after readers of run 28 missed or named the passage. After run 30, the duty to name the + kind moves, unchanged, from section [8](DETECTOR-SPEC.md#8-conformance)'s paragraphs to a clause [8.2](DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades) of its own, with a Why, + out of the section that states the conformance test, and says it binds whoever gives the + verdict, anyone grading under clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) included; section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) names the clause. Clarity edits: + clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) says what a project's own rule must be able to match for the route to be + detectable that way, each ignore comment or directive where it is written and each entry of a + generated baseline; clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) says a gap is a MUST the detector does not satisfy, that a + declaration is the maintainer's claim, and that a detector with no declaration claims nothing. + Each after two or more readers of run 30 named the passage. Accepted by run NNN. Minor version when released. ### 1.0.0, 2026-09-08 diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index 10e9fc6..2c79665 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -283,9 +283,11 @@ this clause; neither requires the [Detector] to remove or forbid the route. A ro neither switched off nor detected does not [Conform]. A route is detectable when the project can find every use of it, in one of two ways. The first is -a [Rule] the project writes for itself, under clause 4.1, and the [Detector] runs. That way works -only where the [Detector] lets such a [Rule] see every use of the route, and the [Detector] need -not ship the [Rule] itself. The second is a mechanical check that the [Detector] itself documents, such as a +a [Rule] the project writes for itself, under clause 4.1, and the [Detector] runs; the [Detector] +need not ship the [Rule] itself. That way works only where such a [Rule] can match every use of +the route: each ignore comment or directive where it is written in the code, and each entry of a +generated [Baseline]. A [Detector] whose [Rules] never see comments, for example, cannot make an +ignore comment detectable this way. The second is a mechanical check that the [Detector] itself documents, such as a text search it documents for the route's spelling. Switching the route off or detecting it is what lets a project that decides to forbid the route enforce that decision. @@ -323,29 +325,16 @@ summary for an [Agent]'s context, are obligations on a project's assembled [Tool and are stated in the [toolchain specification](TOOLING-SPEC.md), so there is no separate [Agent]-support level for a [Detector]. -**A verdict on a [Detector] MUST name its kind, [Semantic], [Syntactic] or [Textual], in each -configuration the verdict grades.** A configuration here is one way of setting the [Detector] up -to run, and one [Detector] can have several: ESLint, for example, is [Syntactic] run without type -information and [Semantic] run with it. The kind is what a [Rule] can ask in that configuration, -as section 2 glosses it. The kind named is the one the [Detector] has without a hosted service or -a licence tier, since under clause 5.4 a finding reported only there does not [Conform]. - -This MUST is on whoever gives the verdict, not on the [Detector], so it is not one of the MUSTs -in sections 4 to 7 that [Conformance] is judged on: the kind is not a level of [Conformance] and -does not change the verdict. A [Textual] [Detector] can [Conform], and a [Semantic] one can fail -to. A verdict that leaves the kind out has not met this duty, but what it finds about -[Conformance] still stands. The kind is named because the method specification's clause 3.2 -requires a [Rule] to be written in the strongest kind of [Detector] that can express it. - ### 8.1 A maintainer MAY declare the version of this document the detector conforms to, and the declaration records known gaps The declaration is machine-readable, in whatever form the [Detector]'s ecosystem uses to record dependencies. The same declaration is where a known gap is recorded. A gap is a MUST in -sections 4 to 7 that the [Detector] fails. Once its maintainer knows of a gap, from the +sections 4 to 7 that the [Detector] does not satisfy. Once its maintainer knows of a gap, from the [Detector]'s own checks or from a [Practitioner]'s report under the method's clause 3.2, a [Detector] that declares a version MUST record the gap alongside that version, in the same file -as the declaration or in a file the declaration names. A declaration that records no gap claims -[Conformance]. A declaration that records a gap does not; it says where the [Detector] stands. +as the declaration or in a file the declaration names. A declaration is the maintainer's claim. +One that records no gap claims [Conformance]; one that records a gap does not, and says where the +[Detector] stands. A [Detector] with no declaration claims nothing, either way. A [Detector] that predates this document, or whose maintainer has never read it, MAY be graded [Conforming] on evidence: a declaration is optional. Anyone may grade a [Detector], by checking it @@ -378,9 +367,29 @@ a specific installed artefact, checkable by anyone, including mechanically, and never [Conform] until its maintainer had heard of this document, which grades the maintainer's reading rather than the [Detector]. +### 8.2 A verdict on a detector MUST name its kind in each configuration it grades + +A verdict on a [Detector] MUST name its kind, [Semantic], [Syntactic] or [Textual], in each +configuration the verdict grades. A configuration here is one way of setting the [Detector] up to +run, and one [Detector] can have several: ESLint, for example, is [Syntactic] run without type +information and [Semantic] run with it. The kind is what a [Rule] can ask in that configuration, +as section 2 glosses it. The kind named is the one the [Detector] has without a hosted service or +a licence tier, since under clause 5.4 a finding reported only there does not [Conform]. + +This clause binds whoever gives the verdict, anyone grading a [Detector] under clause 8.1 +included, and asks nothing of the [Detector] or its maintainer. A [Detector] is judged on +sections 4 to 7, as section 8 says, and the kind is not a level of [Conformance] and does not +change the verdict: a [Textual] [Detector] can [Conform], and a [Semantic] one can fail to. A +verdict that leaves the kind out has not met this clause, but what it finds about [Conformance] +still stands. + +**Why**: the method specification's clause 3.2 requires a [Rule] to be written in the strongest +kind of [Detector] that can express it, and a verdict of [Conforming] alone presents a [Textual] +[Detector] as an equal host. + ## 9. Relationship to the other specifications -This document adds no obligations to a [Practitioner] and relaxes none. Section 8 adds one on +This document adds no obligations to a [Practitioner] and relaxes none. Clause 8.2 adds one on whoever gives a verdict on a [Detector]: to name its kind. Every clause here exists to make a clause of the method specification achievable with a [Detector] in hand. diff --git a/SPEC.md b/SPEC.md index 926140f..6e2885f 100644 --- a/SPEC.md +++ b/SPEC.md @@ -194,14 +194,14 @@ produces [Detectors] that fire on innocent code, which is worse than having no [ rather than deciding in advance whether it is expressible. Failing to write a [Rule] that satisfies clause 3.1's bounds is itself the evidence that the [Defect] is out of scope, and it is cheaper and more reliable than a judgement made before trying. The conclusion that no pattern exists MUST be -stated in one sentence, naming at least two independent techniques tried at expressing the -[Rule], and recorded where the project's other decisions are enumerable under clause 8.7, not -only with the fix. A technique here is a way of trying to write the [Rule]. Two such techniques -are independent when they would miss different things, which is the same test clause 3.1 applies -to ways of searching for [Instances]. That conclusion is the cheapest way -out of the method and the only one that would otherwise leave no trace; enumerating it is how an -[Owner] sees an [Agent] routing around the method, and naming the techniques is what lets -them judge whether the attempt was one. +stated in one sentence, naming at least two independent ways of writing the [Rule] that were +tried, and recorded where the project's other decisions are enumerable under clause 8.7, not +only with the fix. Two ways of writing the [Rule] are independent when they would miss different +things. That is the test clause 3.1 applies to its search techniques, but a way of writing the +[Rule] is not a technique in clause 3.1's sense, which is a way of searching for [Instances]. +That conclusion is the cheapest way out of the method and the only one that would otherwise +leave no trace; enumerating it is how an [Owner] sees an [Agent] routing around the method, and +naming the ways tried is what lets them judge whether the attempt was one. There are three ways out of the method and they are recorded differently. No pattern exists: the [Defect] is out of scope under this section, and only the sentence above is recorded. A pattern exists but the @@ -222,8 +222,8 @@ puts it in front of them. Reporting it in conversation does not satisfy this: th ends, and with it the only trace that the [Defect] was ever seen. **Why**: each of the three ways out costs the [Practitioner] something to take and leaves -something an [Owner] can read: a recorded sentence naming two techniques tried, or a [Toolchain] -gap. Deferral costs nothing, and without this record it would leave nothing, which makes it +something an [Owner] can read: a recorded sentence naming two ways of writing the [Rule] that +were tried, or a [Toolchain] gap. Deferral costs nothing, and without this record it would leave nothing, which makes it cheaper still than the sentence above and unseen from outside. A [Defect] mentioned once and never written down cannot be told apart from a [Defect] nobody found, so neither the [Class] nor the [Defence] it implies is ever reached. @@ -359,7 +359,8 @@ second technique that turns up new [Instances] means a third is owed; a second t new means the search is done. That is a test the [Practitioner] can apply and record without a threshold, and it is the standard clause 3.4 refers back to. -In this clause a technique is a way of searching the codebase for [Instances]. Two techniques are +A technique, here and wherever this specification uses the word, is a way of searching the +codebase for [Instances]. Two techniques are independent when they would miss different things. A text search for the token and a reading of the code that consumes the value are independent, and asking someone who knows the system is a third. Two text searches for two spellings of the same token are one technique, because they miss @@ -471,13 +472,16 @@ A gap is recorded in the project record where the [Toolchain] defines one, as th then the only requirement is that somebody deciding what to invest in the [Toolchain] would find it. A gap recorded where nobody looks has been forgotten with extra steps. -There are two kinds of gap and they lead to different work. The language having no extensible -[Detector] of a kind that can express the [Rule] within clause 3.1's bounds, and no practical -bespoke one, is the gap above, and the [Class] waits for one. The project's [Toolchain] -lacking a mechanism the [toolchain specification](TOOLING-SPEC.md) requires, such as a proving harness or an -[Identifier] resolver, is a mechanism gap, and it does not put the [Class] out of reach: the -[Practitioner] builds the [Rule], uses the substitutes clauses 3.3 and 3.6 already allow, and records -the mechanism gap alongside it. A mechanism gap that is also a breach of the [toolchain specification](TOOLING-SPEC.md)'s +There are two kinds of gap and they lead to different work. The first is the [Toolchain] gap +of the paragraphs above: the language has no extensible [Detector] of a kind that can express the +[Rule] within clause 3.1's bounds, and no bespoke one is practical, so the [Class] waits for one. +The second is a mechanism gap: the project's [Toolchain] lacks a mechanism the +[toolchain specification](TOOLING-SPEC.md) requires, such as a proving harness or an [Identifier] +resolver. A mechanism gap does not put the [Class] out of reach. The [Practitioner] still builds +the [Rule], records the mechanism gap alongside it, and meets clauses 3.3 and 3.6 by the +substitutes those clauses accept: for testing the [Rule], the acceptance or smoke check clause 3.3 +accepts in place of a [Fixture]-based test; for resolution, an [Identifier] in a form clause 3.6 +allows that needs no resolver, such as an anchor in a documentation file. A mechanism gap that is also a breach of the [toolchain specification](TOOLING-SPEC.md)'s own obligations, such as a [Toolchain] that cannot run its [Defences] on its own source, is recorded against the [Toolchain]'s claim under section 7 as well, because that is the fact its [Owner] needs. Where that record lives is the [Toolchain]'s own [Conformance] declaration, under the @@ -758,14 +762,15 @@ that invocation covers, and see the [Rule] reported there. A [Toolchain] that wr who bypasses it has proven the [Rule] and not the [Defence]. A [Bundled defence] is a [Defence] that a shipped [Toolchain] enables by default in every -project that installs it from somebody else. The project whose entry point demonstrates it is any project the -[Toolchain] is genuinely installed into, whether the [Toolchain]'s own or somebody else's, since -that is where the [Rule] will be enforced. A mechanism gap that stops the [Toolchain] running its -own entry point on its own source moves the demonstration to such a project; it does not weaken it. The -substitutes clause 3.2 allows, for proving under clause 3.3 and for resolution under clause 3.6, do -not extend to this clause: the demonstration here is through an entry point or it is not made. A -project created for the purpose, with the [Toolchain] installed into it from the source under -test, is such a project, so a [Toolchain] with no consumer yet is not excused. +project that installs it from somebody else. Its enforcement can be demonstrated through the +entry point of any project the [Toolchain] is genuinely installed into, whether the [Toolchain]'s +own or somebody else's, since that is where the [Rule] will be enforced. A project created for the +purpose, with the [Toolchain] installed into it from the source under test, is such a project, so +a [Toolchain] with no consumer yet is not excused. Where a mechanism gap, as clause 3.2 defines +it, stops the [Toolchain] running its own entry point on its own source, the demonstration is made +in such a project instead, and is owed there in full. Clause 3.2 lets the [Practitioner] meet +clauses 3.3 and 3.6 by substitutes when a mechanism is missing; it allows none for this clause. +The demonstration here is through an entry point or it is not made. **Why**: the purpose is that the mistakes of the past become structurally impossible to repeat, and a [Warning] is not structure. A [Warning] is a suggestion, and suggestions decay under deadline @@ -865,7 +870,7 @@ knowingly leave an [Instance] unfixed on its own authority, whatever the [Instan be. **The default position is that none of these are permitted at all.** [Suppressions], [Baselines] and -every other means of evading a [Defence] are not techniques with a threshold governing their use; they are +every other means of evading a [Defence] are not practices with a threshold governing their use; they are [Exceptions] to the method, and the standing answer is no. An [Exception] exists only where a human has discussed it, agreed it and documented it for that project. Nothing an [Agent] concludes on its own creates one. diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index d51400a..2923b43 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -20,7 +20,7 @@ bullet asserting an obligation the document does not state, is a correct restate - Make any inline suppression route detectable or disableable ([7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)). A bullet saying that a verdict on a detector names the kind, semantic, syntactic or textual, of -each configuration it grades (section [8](../../DETECTOR-SPEC.md#8-conformance)) is correct too, though it is an obligation on whoever +each configuration it grades (clause [8.2](../../DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades)) is correct too, though it is an obligation on whoever gives the verdict, as section [9](../../DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) says. A bullet saying that the detector, or a verdict on it, must state its kind is marked correct as this obligation, although the document puts it on the verdict. @@ -43,8 +43,10 @@ agent summary (those belong to the toolchain specification). configuration, or make the route detectable. Either alone satisfies this clause; neither requires the Detector to remove or forbid the route."; "A route is detectable when the project can find every use of it, in one of two ways. The first is a Rule the project writes for itself, under - clause [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it), and the Detector runs. That way works only where the Detector lets such a Rule see - every use of the route, and the Detector need not ship the Rule itself. The second is a + clause [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it), and the Detector runs; the Detector need not ship the Rule itself. That way works + only where such a Rule can match every use of the route: each ignore comment or directive where + it is written in the code, and each entry of a generated Baseline. A Detector whose Rules never + see comments, for example, cannot make an ignore comment detectable this way. The second is a mechanical check that the Detector itself documents". An answer that says the detector must forbid the route, or must require a reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD), is wrong. - **2c.** With the rule, in the installed copy, at a version tracked together, and yes, offline. @@ -74,7 +76,7 @@ described as conformance. A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a- has read the clause differently and that counts against the document. The fixture does not say what Lintomatic can see. A reader who names its kind as unknown, or names -none, is not marked on it; a reader who makes the verdict turn on its kind has read section [8](../../DETECTOR-SPEC.md#8-conformance) +none, is not marked on it; a reader who makes the verdict turn on its kind has read clause [8.2](../../DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades) differently, which counts against the document: "the kind is not a level of Conformance and does not change the verdict". From 7257121901257bbc6194b6c942c4243a57fb1d98 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 10:55:05 +0100 Subject: [PATCH 056/105] Acceptance process: a question reworded only where it asks the same thing On the author's instruction, "The bar only moves up" allows a question to be reworded only where the new wording asks the same thing with the same certainty, marked against its key unchanged; "After a failure" and acceptance/README.md point to it. Method question 2d now asks for the order of the kinds and the grounds, which its key already required, and each question set's question 2 asks for the sentence or sentences relied on, since the keys for method 2d and detector 2c quote more than one. No key, threshold or fixture changes. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 11 ++++++++--- acceptance/CHANGES.md | 10 +++++++++- acceptance/README.md | 8 +++++--- acceptance/detector/QUESTIONS.md | 2 +- acceptance/method/QUESTIONS.md | 7 ++++--- acceptance/toolchain/QUESTIONS.md | 2 +- 6 files changed, 28 insertions(+), 12 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 9b41021..927c3d1 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -62,7 +62,8 @@ document, so that marking is mechanical. Each reader answers all of them. - `SPEC.md`: what a practitioner does before fixing a defect, in order; who may baseline or suppress; what "conforms" means for a remediation; which kind of detector must host a - rule the practitioner writes, when a weaker one is allowed, and what is recorded then. + rule the practitioner writes and the order of the kinds, on what grounds a weaker one is + allowed, and what is recorded then. - `DETECTOR-SPEC.md`: what "conforms" means and whether a tool that predates the document can conform; what a detector must do about an inline suppression route; where a bundled rule's documentation must live. @@ -189,7 +190,10 @@ No criterion in this document is ever relaxed to let a change through. If the co the test is working and the text is wrong; the fix is to the text, never to the threshold, the question, the fixture or the key. A threshold may be raised, a keyed question may be added, a key may be corrected where it misquoted the document, and a key's limit on which misses count against -the document may be removed, which raises the bar; none of those makes a failing run pass. The +the document may be removed, which raises the bar; none of those makes a failing run pass. A +question may be reworded, on the author's instruction, only where the new wording asks the same +thing with the same certainty, so that it is marked against its key unchanged and no less +strictly; that is not a relaxation, and it makes no failing run pass either. The one time a threshold was relaxed, after three runs, the relaxation was reversed on the author's instruction and is recorded here so that it is not tried again. Removing the clarity rating and the request for suggestions was not a relaxation: it changed what the cohort is asked, not how @@ -197,7 +201,8 @@ strictly an answer is marked. ## After a failure -Fix the text, never the questions, and run a fresh cohort. Readers are not reused across runs. +Fix the text, never the questions beyond the rewording +[the section on the bar](#the-bar-only-moves-up) allows, and run a fresh cohort. Readers are not reused across runs. The number of runs it took is recorded; a document that needed three runs is telling its editor something. diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index c0f3acb..6f1df8f 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -9,7 +9,15 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). the `SPEC.md` question set gains a fourth load-bearing question, 2d, on which kind of detector must host a rule the practitioner writes, so `ACCEPTANCE.md` lists three or more such questions per document. Adding a keyed question raises the bar, as "The bar only moves up" - allows. Prompted by the change to method clause 3.2 in the same pull request. + allows. Prompted by the change to method clause 3.2 in the same pull request. After run 30, on + the author's instruction, "The bar only moves up" allows a question to be reworded only where + it asks the same thing with the same certainty, marked against its key unchanged, and + `acceptance/README.md` and "After a failure" point to it; question 2d asks for the order of the + kinds and the grounds, which its key already required, and each question set's question 2 asks + for the sentence or sentences relied on, since the keys for method 2d and detector 2c quote + more than one. Prompted by runs 29 and 30, where readers who answered 2d as asked were marked + wrong for leaving out the order, and two readers of run 30 quoted only one of the two sentences + detector 2c turns on. - [Pull request #13](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/13): the cohort can be run and marked without judgement calls that the rules leave open; this applies to runs made after it merges, and to runs 19 to 30, which were made under it at the commits their records name, and runs 16 to 18 stand as recorded. A cohort is an odd diff --git a/acceptance/README.md b/acceptance/README.md index 0e43599..088ccd1 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -24,8 +24,9 @@ fixture cases the application question uses live in `fixtures/`, the record of e 4. The result is recorded as a file under [runs/](runs/), stating what `ACCEPTANCE.md` lists, with the answer sheets posted on the pull request and linked from it, and the change's `Unreleased` changelog line names that file. - A run that fails is followed by a change to the text, never to the questions, and a fresh - cohort. + A run that fails is followed by a change to the text, never to the questions beyond the + rewording [the section on the bar](../ACCEPTANCE.md#the-bar-only-moves-up) allows, and a + fresh cohort. ## Keeping the keys honest @@ -34,7 +35,8 @@ the same commit, and the change is one the acceptance run itself then tests. A k matches the document is a defect in this directory, not a reason to mark a reader wrong. Questions are fixed so that runs can be compared across versions. Adding a keyed question is -allowed when a new clause is load-bearing; removing or softening one to make a run pass is not. +allowed when a new clause is load-bearing; removing or softening one to make a run pass is not, +and rewording one is allowed only where it asks the same thing with the same certainty. A process change may remove a limit a key puts on which misses count against the document, as [the section on the bar](../ACCEPTANCE.md#the-bar-only-moves-up) allows. The reader may quote a passage it found confusing; it is never asked to rate the document, say diff --git a/acceptance/detector/QUESTIONS.md b/acceptance/detector/QUESTIONS.md index 58d06d1..d3880db 100644 --- a/acceptance/detector/QUESTIONS.md +++ b/acceptance/detector/QUESTIONS.md @@ -10,7 +10,7 @@ In your own words and in at most six bullets, what does this document require of ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. +Answer each in one sentence, followed by the sentence or sentences of the document you are relying on, quoted. - 2a. What does it mean for a detector to conform, and can a detector that was written before this document existed, and has never heard of it, conform? diff --git a/acceptance/method/QUESTIONS.md b/acceptance/method/QUESTIONS.md index 721fb9c..875c849 100644 --- a/acceptance/method/QUESTIONS.md +++ b/acceptance/method/QUESTIONS.md @@ -11,15 +11,16 @@ do when they are handed a defect? ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. +Answer each in one sentence, followed by the sentence or sentences of the document you are relying on, quoted. - 2a. What does a practitioner do before fixing the defect they were handed, and in what order? - 2b. Who may adopt a baseline or suppress an instance, and may an agent do so on its own authority? - 2c. What does it mean for a remediation to conform, and on what must a reviewer's verdict on it rest? -- 2d. Which kind of detector must a rule the practitioner writes be written in, when may a weaker - kind be used, and what must the practitioner record then? +- 2d. Which kind of detector must a rule the practitioner writes be written in, and in what order + do the kinds go? On what grounds may a weaker kind be used, and what must the practitioner + record then? ## 3. Application diff --git a/acceptance/toolchain/QUESTIONS.md b/acceptance/toolchain/QUESTIONS.md index a4da266..39556de 100644 --- a/acceptance/toolchain/QUESTIONS.md +++ b/acceptance/toolchain/QUESTIONS.md @@ -11,7 +11,7 @@ toolchain? ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. +Answer each in one sentence, followed by the sentence or sentences of the document you are relying on, quoted. - 2a. At what level is toolchain conformance measured, and what is the toolchain? - 2b. A project's tooling is assembled from a third-party detector, a first-party package and From 5cec5573f06a2f82e2890104deb75b18e8144d9c Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 11:17:55 +0100 Subject: [PATCH 057/105] Acceptance process: an earlier run's judgement binds no later run Review 1 of this change, finding 1: the sentence freed a later run only where a rule the judgement rested on had since changed. Run 22's judgement applied a tie-break bdfd055 had replaced before run 22 was made, so no rule changed after it and the sentence missed its own case; it also implied that a judgement binds where its rule has not changed, which no process file said. A judgement an earlier run recorded now binds no later run, one that applied a rule not in force when it was made is never followed, and a record says where its marker departs from an earlier judgement and why. acceptance/CHANGES.md follows. No criterion changes. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 5 +++-- acceptance/CHANGES.md | 8 +++++--- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index eacf840..42e28f3 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -154,8 +154,9 @@ which rule and why. Each sheet is marked question by question, as follows. Where a mark needs a judgement these rules do not settle, the marker makes it as a careful reader of the document would, and the record states the judgement and why. A judgement an earlier run -recorded does not bind a later run where a rule it rested on has since changed: the marker makes -it afresh under the rules as they now stand, and the record names the rule that changed. +recorded binds no later run, and one that applied a rule not in force when it was made is never +followed: the marker makes each judgement afresh under this file as it applies to the run being +marked, and where it departs from an earlier run's judgement, the record says so and why. ## Pass criteria diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 8735c63..433677b 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -18,8 +18,9 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). sheet's answered items counted too, or if that sheet has a miss traceable to the text; a void run has a number, a record and posted sheets. The packet is the question set, the document and the fixture, each after one `===== FILE: =====` line. A new marking section says what a delegated marker receives and that its verdict is final unless a rule was misapplied, that an open judgement is - made as a careful reader of the document would, that an earlier run's judgement does not bind - a later run where a rule it rested on has since changed, and defines a keyed item, quoting (consecutive + made as a careful reader of the document would, that an earlier run's judgement binds no + later run, one that applied a rule not in force when it was made is never followed, and a + record says where its marker departs from an earlier judgement and why, and defines a keyed item, quoting (consecutive words of any length, verbatim apart from case, quotation-mark style and markup), a correct load-bearing answer, read with its quotation, asserting nothing the document does not state and omitting no element of the key's answer that answers the question, each read for its @@ -48,6 +49,7 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). rows both covered a single reader's traceable miss, and nothing settled an unanswered question or "the same passage". The rule on an earlier run's judgement was prompted by run 22, which made a judgement by the stricter reading, a tie-break this change had already replaced with a - careful reader's, and which later runs of pull request 14 then followed as precedent. [Run + careful reader's before run 22 was made, so that the rule it applied was not in force, and which + later runs of pull request 14 then followed as precedent. [Run 16](runs/016-deferral-clause-run-1.md), [run 17](runs/017-deferral-clause-run-2.md) and [run 18](runs/018-deferral-clause-run-3.md) ruled on these case by case. From 8f5b7ee1661aa50869a8703eee3f615bbc9e80e8 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 11:19:51 +0100 Subject: [PATCH 058/105] Acceptance process: a rewording is the editor's decision, question by question Review 1, findings 2 to 5. The rule allowing a question to be reworded where it asks the same thing with the same certainty stays, and now says who and when: whether a rewording asks the same thing is the editor's decision for each rewording, not delegated; a rewording is used from the next run whose readers start after it is committed and changes no earlier marks; and its line in acceptance/CHANGES.md gives the old and new wording and the element of the unchanged key each added part asks for, so an owner can check it. The sentence saying the fix is never to a question names the rewording the section allows, so the paragraph no longer contradicts itself, and acceptance/README.md states the same conditions. No question is reworded by this change: method 2d and the "sentence or sentences" instruction in the three question sets return to their wording at 638093a, since the editor has not decided either, and ACCEPTANCE.md's summary of 2d follows. acceptance/CHANGES.md no longer says readers who answered 2d as asked were marked wrong. No key, threshold or fixture changes. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 20 ++++++++++++-------- acceptance/CHANGES.md | 16 +++++++++------- acceptance/README.md | 6 +++++- acceptance/detector/QUESTIONS.md | 2 +- acceptance/method/QUESTIONS.md | 7 +++---- acceptance/toolchain/QUESTIONS.md | 2 +- 6 files changed, 31 insertions(+), 22 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index ad52a38..ce9e439 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -62,8 +62,7 @@ document, so that marking is mechanical. Each reader answers all of them. - `SPEC.md`: what a practitioner does before fixing a defect, in order; who may baseline or suppress; what "conforms" means for a remediation; which kind of detector must host a - rule the practitioner writes and the order of the kinds, on what grounds a weaker one is - allowed, and what is recorded then. + rule the practitioner writes, when a weaker one is allowed, and what is recorded then. - `DETECTOR-SPEC.md`: what "conforms" means and whether a tool that predates the document can conform; what a detector must do about an inline suppression route; where a bundled rule's documentation must live. @@ -189,12 +188,17 @@ the reader can justify from the document is always the document's fault. No criterion in this document is ever relaxed to let a change through. If the cohort is confused, the test is working and the text is wrong; the fix is to the text, never to the threshold, the -question, the fixture or the key. A threshold may be raised, a keyed question may be added, a key -may be corrected where it misquoted the document, and a key's limit on which misses count against -the document may be removed, which raises the bar; none of those makes a failing run pass. A -question may be reworded, on the author's instruction, only where the new wording asks the same -thing with the same certainty, so that it is marked against its key unchanged and no less -strictly; that is not a relaxation, and it makes no failing run pass either. The +fixture or the key, nor to a question beyond the rewording this section allows. A threshold may be +raised, a keyed question may be added, a key may be corrected where it misquoted the document, and +a key's limit on which misses count against the document may be removed, which raises the bar; +none of those makes a failing run pass. A question may be reworded only where the new wording asks +the same thing with the same certainty, so that it is marked against its key unchanged and no less +strictly. Whether a rewording asks the same thing is the editor's decision, taken for each +rewording and not delegated, and a rewording is made only once the editor has taken it. It is used from the next run whose readers start after it is committed, and never +changes the marks of a run already made. The line recording it in +[acceptance/CHANGES.md](acceptance/CHANGES.md) gives the old and the new wording and names, for +each part the new wording adds, the element of the unchanged key it asks for, or says that it asks +for none. Such a rewording is not a relaxation, and it makes no failing run pass either. The one time a threshold was relaxed, after three runs, the relaxation was reversed on the author's instruction and is recorded here so that it is not tried again. Removing the clarity rating and the request for suggestions was not a relaxation: it changed what the cohort is asked, not how diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 9441fca..8563500 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -11,13 +11,15 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). questions per document. Adding a keyed question raises the bar, as "The bar only moves up" allows. Prompted by the change to method clause 3.2 in the same pull request. After run 30, on the author's instruction, "The bar only moves up" allows a question to be reworded only where - it asks the same thing with the same certainty, marked against its key unchanged, and - `acceptance/README.md` and "After a failure" point to it; question 2d asks for the order of the - kinds and the grounds, which its key already required, and each question set's question 2 asks - for the sentence or sentences relied on, since the keys for method 2d and detector 2c quote - more than one. Prompted by runs 29 and 30, where readers who answered 2d as asked were marked - wrong for leaving out the order, and two readers of run 30 quoted only one of the two sentences - detector 2c turns on. + the new wording asks the same thing with the same certainty, marked against its key unchanged + and no less strictly; whether it does is the editor's decision for each rewording, not + delegated, the rewording is used from the next run whose readers start after it is committed + and changes no earlier marks, and its line here gives the old and new wording and the element + of the unchanged key each added part asks for. `acceptance/README.md` and "After a failure" + point to it, and the sentence saying the fix is never to a question names it. No question is + reworded by this change. Prompted by runs 29 and 30, in which method 2d's key requires the + order of the kinds and readers left it out, and two readers of run 30 quoted only one of the + two sentences detector 2c turns on, whilst question 2 asks for one sentence. - [Pull request #13](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/13): the cohort can be run and marked without judgement calls that the rules leave open; this applies to runs made after it merges, and to runs 19 to 30, which were made under it at the commits their records name, and runs 16 to 18 stand as recorded. A cohort is an odd diff --git a/acceptance/README.md b/acceptance/README.md index 088ccd1..6a1162e 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -36,7 +36,11 @@ matches the document is a defect in this directory, not a reason to mark a reade Questions are fixed so that runs can be compared across versions. Adding a keyed question is allowed when a new clause is load-bearing; removing or softening one to make a run pass is not, -and rewording one is allowed only where it asks the same thing with the same certainty. +and rewording one is allowed only as [the section on the bar](../ACCEPTANCE.md#the-bar-only-moves-up) +allows: where the new wording asks the same thing with the same certainty, on the editor's +decision for that rewording, from the next run whose readers start after it is committed, and +recorded in [CHANGES.md](CHANGES.md) with the element of the unchanged key each added part asks +for. A process change may remove a limit a key puts on which misses count against the document, as [the section on the bar](../ACCEPTANCE.md#the-bar-only-moves-up) allows. The reader may quote a passage it found confusing; it is never asked to rate the document, say diff --git a/acceptance/detector/QUESTIONS.md b/acceptance/detector/QUESTIONS.md index d3880db..58d06d1 100644 --- a/acceptance/detector/QUESTIONS.md +++ b/acceptance/detector/QUESTIONS.md @@ -10,7 +10,7 @@ In your own words and in at most six bullets, what does this document require of ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence or sentences of the document you are relying on, quoted. +Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. - 2a. What does it mean for a detector to conform, and can a detector that was written before this document existed, and has never heard of it, conform? diff --git a/acceptance/method/QUESTIONS.md b/acceptance/method/QUESTIONS.md index 875c849..721fb9c 100644 --- a/acceptance/method/QUESTIONS.md +++ b/acceptance/method/QUESTIONS.md @@ -11,16 +11,15 @@ do when they are handed a defect? ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence or sentences of the document you are relying on, quoted. +Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. - 2a. What does a practitioner do before fixing the defect they were handed, and in what order? - 2b. Who may adopt a baseline or suppress an instance, and may an agent do so on its own authority? - 2c. What does it mean for a remediation to conform, and on what must a reviewer's verdict on it rest? -- 2d. Which kind of detector must a rule the practitioner writes be written in, and in what order - do the kinds go? On what grounds may a weaker kind be used, and what must the practitioner - record then? +- 2d. Which kind of detector must a rule the practitioner writes be written in, when may a weaker + kind be used, and what must the practitioner record then? ## 3. Application diff --git a/acceptance/toolchain/QUESTIONS.md b/acceptance/toolchain/QUESTIONS.md index 39556de..a4da266 100644 --- a/acceptance/toolchain/QUESTIONS.md +++ b/acceptance/toolchain/QUESTIONS.md @@ -11,7 +11,7 @@ toolchain? ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence or sentences of the document you are relying on, quoted. +Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. - 2a. At what level is toolchain conformance measured, and what is the toolchain? - 2b. A project's tooling is assembled from a third-party detector, a first-party package and From 8abd44ffb41818a4ed1604cb5beeb1eb4d0286f0 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 11:21:56 +0100 Subject: [PATCH 059/105] Method 2, 3.2, 3.5 and 4, detector 6.2, 7.1, 8.2 and 9: review 1 findings applied Review 1 of the runs 29 and 30 text changes, findings 7 to 16. No obligation changes; tools/changes.py gives the same findings as before. Method: clause 3.2 no longer says the practitioner meets clauses 3.3 and 3.6 by substitutes. It says the practitioner uses what those clauses allow in place of a missing mechanism, each under its own condition, the smoke check where a detector makes fixture-based testing impractical, and that every other obligation of those clauses still applies; clause 3.5 echoes it, and says the demonstration is made through the entry point of another project the toolchain is installed into, so "instead" has no unclear antecedent. Section 2 says a way of writing the rule is not a technique, which clause 3.1 defines, without implying a second sense. Section 4 says an exception exists only where the owner has discussed, agreed and documented it, as the owner's terminology entry and the section's list already say, after run 29's reader 5 answered 2b with "a human". Detector: clause 7.1 says what a rule must match route by route, so the list cannot be read as cumulative. Clause 8.2 asks nothing of the detector itself, and a maintainer grading their own detector is bound like anyone else. Section 9 says the document adds no obligation to the method a practitioner follows, and that clause 8.2 binds a practitioner who gives a verdict. Clause 6.2 gains a sentence saying the page resolved to is in the documentation clause 6.3 ships with the rule at a version tracked together, after two readers of run 30 answered 2c from clause 6.2 alone. The detector key's 2b quotation of 7.1 follows the text. The changelog entries are corrected: the 8.2 move is described accurately, with its new second paragraph, and the run 29 sentence says which passages two readers named and which followed from run 25's open passage. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 49 ++++++++++++++++++++++++-------------- DETECTOR-SPEC.md | 16 +++++++++---- SPEC.md | 19 ++++++++------- acceptance/detector/KEY.md | 5 ++-- 4 files changed, 56 insertions(+), 33 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f25d648..3d7ae8a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -79,14 +79,21 @@ version entry with no cohort line has not been accepted and is not published. paragraphs, saying that a run of the rule belongs to the proof or the sweep. Each after readers of run 27 missed or named the passage. After run 29, clarity edits: section [2](SPEC.md#2-when-the-method-applies)'s sentence that no pattern exists names two independent ways of writing the rule that were tried, in place of - "techniques", and says these are not clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s techniques; the MUST it carries is reworded, not - changed. Clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class) says a technique is a way of searching for instances wherever the - specification uses the word, and section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) no longer uses it otherwise; clause [3.2](SPEC.md#32-build-the-net)'s - paragraph on the two kinds of gap gives each kind a sentence of its own and names the - substitutes clauses [3.3](SPEC.md#33-prove-the-net-by-making-the-rule-fire) and [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) accept for a missing mechanism; and clause [3.5](SPEC.md#35-enforce-permanently-and-block)'s paragraph on a - bundled defence says in shorter sentences where its demonstration is made and that those - substitutes give no way round it. Each after two readers of run 29 named the passage. Accepted - by run NNN. Minor version when released. + "techniques", and says a way of writing the rule is not a technique, which clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class) defines; + the MUST it carries is reworded, not changed. Clause [3.2](SPEC.md#32-build-the-net)'s paragraph on the two kinds of gap + gives each kind a sentence of its own and names, for a missing mechanism, the substitutes + clauses [3.3](SPEC.md#33-prove-the-net-by-making-the-rule-fire) and [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) already allow, each under the condition its clause sets, and says every + other obligation of those clauses still applies; and clause [3.5](SPEC.md#35-enforce-permanently-and-block)'s paragraph on a bundled + defence says in shorter sentences that, where a mechanism gap stops the toolchain running its + own entry point, the demonstration is made through another project's entry point and owed + there in full, and that clause [3.2](SPEC.md#32-build-the-net)'s substitutes give no way round it. Each after two readers of + run 29 named the passage. Clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class) says a technique is a way of searching for instances + wherever the specification uses the word, after one reader of run 29 named run 25's open + passage there, and section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) calls suppressions and baselines practices, not techniques, to + follow it. Section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) says an exception exists only where the owner, not "a human", has + discussed, agreed and documented it, as the owner's terminology entry and the section's own + list already say, after a reader of run 29 answered 2b with "a human" and not the owner. Accepted by run + NNN. Minor version when released. ### 1.1.0, 2026-10-02 @@ -153,8 +160,9 @@ pass it. or licence tier. It is not a level of conformance and does not change the verdict; it is stated because method clause [3.2](SPEC.md#32-build-the-net) now requires a rule to be written in the strongest kind of detector that can express it, and a verdict of conforming alone presents a textual detector as - an equal host. Section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) still adds no obligation to a practitioner and names this - one as binding whoever gives a verdict on a detector, and section [2](DETECTOR-SPEC.md#2-terminology)'s gloss carries the three terms. After run 20, changed obligation, clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran): the + an equal host. Section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) still adds no obligation to the method a practitioner + follows and names this one as binding whoever gives a verdict on a detector, a practitioner + included when they give one, and section [2](DETECTOR-SPEC.md#2-terminology)'s gloss carries the three terms. After run 20, changed obligation, clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran): the usable summary gives each finding in it with its location and identifier, and the location of the rest is owed only where the summary leaves a finding out, so output that prints every finding conforms though extra detail goes to a file it does not name; the old text owed that @@ -190,15 +198,20 @@ pass it. needing the other specifications; section [8](DETECTOR-SPEC.md#8-conformance)'s paragraph on the kind says what a configuration is and, in a paragraph of its own, that the duty is not one of the MUSTs conformance is judged on and that a verdict leaving the kind out still stands on conformance. - Each after readers of run 28 missed or named the passage. After run 30, the duty to name the - kind moves, unchanged, from section [8](DETECTOR-SPEC.md#8-conformance)'s paragraphs to a clause [8.2](DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades) of its own, with a Why, - out of the section that states the conformance test, and says it binds whoever gives the - verdict, anyone grading under clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) included; section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) names the clause. Clarity edits: - clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) says what a project's own rule must be able to match for the route to be - detectable that way, each ignore comment or directive where it is written and each entry of a - generated baseline; clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) says a gap is a MUST the detector does not satisfy, that a + Each after readers of run 28 missed or named the passage. After run 30, the MUST to name the + kind moves, unchanged, from the body of section [8](DETECTOR-SPEC.md#8-conformance), beside the conformance test, to a clause + [8.2](DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades) of its own, with a Why; the paragraph after it says, in new words, that it binds whoever + gives the verdict, anyone grading under clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) and a maintainer grading their own detector + included, and asks nothing of the detector itself; section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) names the clause and says it + binds a practitioner who gives a verdict. Clarity edits: clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) says, route by route, what + a project's own rule must be able to match for the route to be detectable that way: for an + ignore comment or a per-line directive each place it is written, for a generated baseline each + of its entries; clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) says a gap is a MUST the detector does not satisfy, that a declaration is the maintainer's claim, and that a detector with no declaration claims nothing. - Each after two or more readers of run 30 named the passage. Accepted by run NNN. Minor version + Each after two or more readers of run 30 named the passage. Clause [6.2](DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) gains a sentence + saying that the page an identifier resolves to is in the documentation clause [6.3](DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) requires to + ship with the rule, at a version tracked together with it, after two readers of run 30 answered + 2c from clause [6.2](DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) alone and left out the version. Accepted by run NNN. Minor version when released. ### 1.0.0, 2026-09-08 diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index 2c79665..58892c3 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -218,6 +218,10 @@ given. The lookup they can actually perform is the only one that counts. ### 6.2 Resolution of a bundled rule's identifier MUST work from the installed copy, without network access +The page an [Identifier] of a [Bundled rule] resolves to is in the documentation that clause 6.3 +requires to ship with the [Rule], at a version tracked together with it, so the installed copy +already holds it. + **Why**: an [Agent] working offline, behind a proxy, or against a URL that has since moved needs the answer to be on disk. A dependency the project already installed is on disk by definition. A catalogue on the [Detector]'s website, however complete, is the right documentation in the wrong @@ -285,8 +289,8 @@ neither switched off nor detected does not [Conform]. A route is detectable when the project can find every use of it, in one of two ways. The first is a [Rule] the project writes for itself, under clause 4.1, and the [Detector] runs; the [Detector] need not ship the [Rule] itself. That way works only where such a [Rule] can match every use of -the route: each ignore comment or directive where it is written in the code, and each entry of a -generated [Baseline]. A [Detector] whose [Rules] never see comments, for example, cannot make an +the route: for an ignore comment or a per-line directive, each place it is written in the code; +for a generated [Baseline], each of its entries. A [Detector] whose [Rules] never see comments, for example, cannot make an ignore comment detectable this way. The second is a mechanical check that the [Detector] itself documents, such as a text search it documents for the route's spelling. Switching the route off or detecting it is what lets a project that decides to forbid the route enforce that decision. @@ -377,7 +381,8 @@ as section 2 glosses it. The kind named is the one the [Detector] has without a a licence tier, since under clause 5.4 a finding reported only there does not [Conform]. This clause binds whoever gives the verdict, anyone grading a [Detector] under clause 8.1 -included, and asks nothing of the [Detector] or its maintainer. A [Detector] is judged on +included, and asks nothing of the [Detector] itself; a maintainer who gives a verdict on their +own [Detector] is bound by it like anyone else. A [Detector] is judged on sections 4 to 7, as section 8 says, and the kind is not a level of [Conformance] and does not change the verdict: a [Textual] [Detector] can [Conform], and a [Semantic] one can fail to. A verdict that leaves the kind out has not met this clause, but what it finds about [Conformance] @@ -389,8 +394,9 @@ kind of [Detector] that can express it, and a verdict of [Conforming] alone pres ## 9. Relationship to the other specifications -This document adds no obligations to a [Practitioner] and relaxes none. Clause 8.2 adds one on -whoever gives a verdict on a [Detector]: to name its kind. Every clause here +This document adds no obligation to the method a [Practitioner] follows, and relaxes none. +Clause 8.2 adds one on whoever gives a verdict on a [Detector], a [Practitioner] included when +they give one: to name its kind. Every clause here exists to make a clause of the method specification achievable with a [Detector] in hand. Where this document and the method specification disagree, the method specification governs. It diff --git a/SPEC.md b/SPEC.md index 6e2885f..1e9c48e 100644 --- a/SPEC.md +++ b/SPEC.md @@ -198,7 +198,7 @@ stated in one sentence, naming at least two independent ways of writing the [Rul tried, and recorded where the project's other decisions are enumerable under clause 8.7, not only with the fix. Two ways of writing the [Rule] are independent when they would miss different things. That is the test clause 3.1 applies to its search techniques, but a way of writing the -[Rule] is not a technique in clause 3.1's sense, which is a way of searching for [Instances]. +[Rule] is not a technique, which clause 3.1 defines as a way of searching for [Instances]. That conclusion is the cheapest way out of the method and the only one that would otherwise leave no trace; enumerating it is how an [Owner] sees an [Agent] routing around the method, and naming the ways tried is what lets them judge whether the attempt was one. @@ -478,10 +478,12 @@ of the paragraphs above: the language has no extensible [Detector] of a kind tha The second is a mechanism gap: the project's [Toolchain] lacks a mechanism the [toolchain specification](TOOLING-SPEC.md) requires, such as a proving harness or an [Identifier] resolver. A mechanism gap does not put the [Class] out of reach. The [Practitioner] still builds -the [Rule], records the mechanism gap alongside it, and meets clauses 3.3 and 3.6 by the -substitutes those clauses accept: for testing the [Rule], the acceptance or smoke check clause 3.3 -accepts in place of a [Fixture]-based test; for resolution, an [Identifier] in a form clause 3.6 -allows that needs no resolver, such as an anchor in a documentation file. A mechanism gap that is also a breach of the [toolchain specification](TOOLING-SPEC.md)'s +the [Rule] and records the mechanism gap alongside it. Where the missing mechanism would have +served clause 3.3 or 3.6, the [Practitioner] uses what those clauses allow in its place: the +acceptance or smoke check that clause 3.3 accepts where a [Detector] makes [Fixture]-based testing +impractical, and an [Identifier] in a form clause 3.6 allows that can be followed without a +resolver, such as an anchor in a documentation file. Every obligation of those clauses that the substitute +does not stand in for still applies in full. A mechanism gap that is also a breach of the [toolchain specification](TOOLING-SPEC.md)'s own obligations, such as a [Toolchain] that cannot run its [Defences] on its own source, is recorded against the [Toolchain]'s claim under section 7 as well, because that is the fact its [Owner] needs. Where that record lives is the [Toolchain]'s own [Conformance] declaration, under the @@ -768,8 +770,9 @@ own or somebody else's, since that is where the [Rule] will be enforced. A proje purpose, with the [Toolchain] installed into it from the source under test, is such a project, so a [Toolchain] with no consumer yet is not excused. Where a mechanism gap, as clause 3.2 defines it, stops the [Toolchain] running its own entry point on its own source, the demonstration is made -in such a project instead, and is owed there in full. Clause 3.2 lets the [Practitioner] meet -clauses 3.3 and 3.6 by substitutes when a mechanism is missing; it allows none for this clause. +through the entry point of another project the [Toolchain] is genuinely installed into, and is +owed there in full. Clause 3.2 lets the [Practitioner] use the substitutes it names for clauses +3.3 and 3.6 when a mechanism is missing; it allows none for this clause. The demonstration here is through an entry point or it is not made. **Why**: the purpose is that the mistakes of the past become structurally impossible to repeat, and @@ -871,7 +874,7 @@ be. **The default position is that none of these are permitted at all.** [Suppressions], [Baselines] and every other means of evading a [Defence] are not practices with a threshold governing their use; they are -[Exceptions] to the method, and the standing answer is no. An [Exception] exists only where a human has +[Exceptions] to the method, and the standing answer is no. An [Exception] exists only where the [Owner] has discussed it, agreed it and documented it for that project. Nothing an [Agent] concludes on its own creates one. diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 2923b43..4fe4d28 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -44,8 +44,9 @@ agent summary (those belong to the toolchain specification). the Detector to remove or forbid the route."; "A route is detectable when the project can find every use of it, in one of two ways. The first is a Rule the project writes for itself, under clause [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it), and the Detector runs; the Detector need not ship the Rule itself. That way works - only where such a Rule can match every use of the route: each ignore comment or directive where - it is written in the code, and each entry of a generated Baseline. A Detector whose Rules never + only where such a Rule can match every use of the route: for an ignore comment or a per-line + directive, each place it is written in the code; for a generated Baseline, each of its entries. + A Detector whose Rules never see comments, for example, cannot make an ignore comment detectable this way. The second is a mechanical check that the Detector itself documents". An answer that says the detector must forbid the route, or must require a reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD), is wrong. From 5f01f4cbdc9dfa437b0b705f95a578fcb9cd3aad Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 14:49:06 +0100 Subject: [PATCH 060/105] Acceptance process: a departure is recorded against the same pull request's runs Review 2, finding N4. The clause that a judgement applying a rule not in force is "never followed" is dropped: no earlier judgement binds, and the words could be read as forbidding a mark a careful reader reaches afresh. The duty to record a departure now names the earlier runs of the same pull request, since other pull requests' runs reach a marker as examples of style only. Run 22's case, within one pull request, is still covered. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 6 +++--- acceptance/CHANGES.md | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 42e28f3..107a490 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -154,9 +154,9 @@ which rule and why. Each sheet is marked question by question, as follows. Where a mark needs a judgement these rules do not settle, the marker makes it as a careful reader of the document would, and the record states the judgement and why. A judgement an earlier run -recorded binds no later run, and one that applied a rule not in force when it was made is never -followed: the marker makes each judgement afresh under this file as it applies to the run being -marked, and where it departs from an earlier run's judgement, the record says so and why. +recorded binds no later run: the marker makes each judgement afresh under this file as it +applies to the run being marked, and where it departs from a judgement that an earlier run of the +same pull request recorded, the record says so and why. ## Pass criteria diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 433677b..5fe9e20 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -19,8 +19,8 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). after one `===== FILE: =====` line. A new marking section says what a delegated marker receives and that its verdict is final unless a rule was misapplied, that an open judgement is made as a careful reader of the document would, that an earlier run's judgement binds no - later run, one that applied a rule not in force when it was made is never followed, and a - record says where its marker departs from an earlier judgement and why, and defines a keyed item, quoting (consecutive + later run, and that a record says where its marker departs from a judgement an earlier run of + the same pull request recorded, and why, and defines a keyed item, quoting (consecutive words of any length, verbatim apart from case, quotation-mark style and markup), a correct load-bearing answer, read with its quotation, asserting nothing the document does not state and omitting no element of the key's answer that answers the question, each read for its From 87a1c64f588c5a4ebb48105fc9875c165273a304 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 14:52:18 +0100 Subject: [PATCH 061/105] Method 3.2, detector 6.2 and 8.2, acceptance process: review 2 findings applied Method 3.2 (finding 7, N2): the substitutes for a missing mechanism are named with their scope. Clause 3.3's smoke check is allowed only where a detector makes fixture-based testing impractical, and stands in only for the fixture-based test its test-first paragraph asks for, so Part A's red run and Part C's proof on fixture code are still owed; the anchor stands in only for a resolver. Detector 8.2 (finding 11): a declaration under clause 8.1 is the maintainer's claim, not a verdict, and the clause asks nothing of it. Detector 6.2 (N1): the body says where resolution looks and what the clause asks, and that a page missing from the shipped documentation fails clause 6.3, not 6.2, in place of asserting the page is there. Acceptance process (N3, N5, N6): "The rewording is used from" names its subject; "the author's instruction" becomes "the editor's" in the two ACCEPTANCE.md sentences and the CHANGES.md line; acceptance/README.md gives the whole record rule for a rewording. Co-Authored-By: Claude Opus 5.5 --- ACCEPTANCE.md | 9 +++++---- CHANGELOG.md | 18 ++++++++++++------ DETECTOR-SPEC.md | 10 ++++++---- SPEC.md | 13 ++++++++----- acceptance/CHANGES.md | 6 ++++-- acceptance/README.md | 4 ++-- 6 files changed, 37 insertions(+), 23 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index ecdd2aa..e1d9430 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -91,7 +91,7 @@ not there to design the text, and a document shaped by what its weakest reader s a document designed by that reader. The signals this test takes from the cohort are a keyed answer missed, a wrong verdict reasoned from the document, and a passage pointed at; what to do about any of them is the editor's decision. An earlier version of this test asked the reader -what would make a clause clearer and for a clarity rating; both were removed on the author's +what would make a clause clearer and for a clarity rating; both were removed on the editor's instruction for that reason, and neither is reinstated. ## Marking @@ -194,12 +194,13 @@ a key's limit on which misses count against the document may be removed, which r none of those makes a failing run pass. A question may be reworded only where the new wording asks the same thing with the same certainty, so that it is marked against its key unchanged and no less strictly. Whether a rewording asks the same thing is the editor's decision, taken for each -rewording and not delegated, and a rewording is made only once the editor has taken it. It is used from the next run whose readers start after it is committed, and never -changes the marks of a run already made. The line recording it in +rewording and not delegated, and a rewording is made only once the editor has taken it. The +rewording is used from the next run whose readers start after it is committed, and never changes +the marks of a run already made. The line recording it in [acceptance/CHANGES.md](acceptance/CHANGES.md) gives the old and the new wording and names, for each part the new wording adds, the element of the unchanged key it asks for, or says that it asks for none. Such a rewording is not a relaxation, and it makes no failing run pass either. The -one time a threshold was relaxed, after three runs, the relaxation was reversed on the author's +one time a threshold was relaxed, after three runs, the relaxation was reversed on the editor's instruction and is recorded here so that it is not tried again. Removing the clarity rating and the request for suggestions was not a relaxation: it changed what the cohort is asked, not how strictly an answer is marked. diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d7ae8a..1394744 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -82,8 +82,11 @@ version entry with no cohort line has not been accepted and is not published. "techniques", and says a way of writing the rule is not a technique, which clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class) defines; the MUST it carries is reworded, not changed. Clause [3.2](SPEC.md#32-build-the-net)'s paragraph on the two kinds of gap gives each kind a sentence of its own and names, for a missing mechanism, the substitutes - clauses [3.3](SPEC.md#33-prove-the-net-by-making-the-rule-fire) and [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) already allow, each under the condition its clause sets, and says every - other obligation of those clauses still applies; and clause [3.5](SPEC.md#35-enforce-permanently-and-block)'s paragraph on a bundled + clauses [3.3](SPEC.md#33-prove-the-net-by-making-the-rule-fire) and [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) already allow, each only under the condition its clause sets and only for + what it stands in for: the smoke check for the fixture-based test clause [3.3](SPEC.md#33-prove-the-net-by-making-the-rule-fire)'s test-first + paragraph asks for, so that Part A's red run and Part C's proof on fixture code are still owed, + and the anchor for a resolver; it says every other obligation of those clauses still applies, + so nothing is relaxed; and clause [3.5](SPEC.md#35-enforce-permanently-and-block)'s paragraph on a bundled defence says in shorter sentences that, where a mechanism gap stops the toolchain running its own entry point, the demonstration is made through another project's entry point and owed there in full, and that clause [3.2](SPEC.md#32-build-the-net)'s substitutes give no way round it. Each after two readers of @@ -202,15 +205,18 @@ pass it. kind moves, unchanged, from the body of section [8](DETECTOR-SPEC.md#8-conformance), beside the conformance test, to a clause [8.2](DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades) of its own, with a Why; the paragraph after it says, in new words, that it binds whoever gives the verdict, anyone grading under clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) and a maintainer grading their own detector - included, and asks nothing of the detector itself; section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) names the clause and says it + included, and asks nothing of the detector itself, nor of a declaration under clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps), which + is the maintainer's claim and not a verdict; section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) names the clause and says it binds a practitioner who gives a verdict. Clarity edits: clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) says, route by route, what a project's own rule must be able to match for the route to be detectable that way: for an ignore comment or a per-line directive each place it is written, for a generated baseline each of its entries; clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) says a gap is a MUST the detector does not satisfy, that a declaration is the maintainer's claim, and that a detector with no declaration claims nothing. - Each after two or more readers of run 30 named the passage. Clause [6.2](DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) gains a sentence - saying that the page an identifier resolves to is in the documentation clause [6.3](DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) requires to - ship with the rule, at a version tracked together with it, after two readers of run 30 answered + Each after two or more readers of run 30 named the passage. Clause [6.2](DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) gains a paragraph + saying that resolution reads the documentation clause [6.3](DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) requires to ship with the rule, at a + version tracked together with it, in the installed copy, that clause [6.2](DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) asks that the + lookup reach it without network access, and that a missing page fails clause [6.3](DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), not + clause [6.2](DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access), after two readers of run 30 answered 2c from clause [6.2](DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) alone and left out the version. Accepted by run NNN. Minor version when released. diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index 58892c3..40d0a1a 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -218,9 +218,10 @@ given. The lookup they can actually perform is the only one that counts. ### 6.2 Resolution of a bundled rule's identifier MUST work from the installed copy, without network access -The page an [Identifier] of a [Bundled rule] resolves to is in the documentation that clause 6.3 -requires to ship with the [Rule], at a version tracked together with it, so the installed copy -already holds it. +Resolution reads the documentation that clause 6.3 requires to ship with the [Rule], at a version +tracked together with it, and the installed copy is where that documentation is. This clause asks +that the lookup reach what is there without network access. A page missing from that +documentation fails clause 6.3, not this clause. **Why**: an [Agent] working offline, behind a proxy, or against a URL that has since moved needs the answer to be on disk. A dependency the project already installed is on disk by definition. A @@ -382,7 +383,8 @@ a licence tier, since under clause 5.4 a finding reported only there does not [C This clause binds whoever gives the verdict, anyone grading a [Detector] under clause 8.1 included, and asks nothing of the [Detector] itself; a maintainer who gives a verdict on their -own [Detector] is bound by it like anyone else. A [Detector] is judged on +own [Detector] is bound by it like anyone else. A declaration under clause 8.1 is the +maintainer's claim, not a verdict, and this clause asks nothing of it. A [Detector] is judged on sections 4 to 7, as section 8 says, and the kind is not a level of [Conformance] and does not change the verdict: a [Textual] [Detector] can [Conform], and a [Semantic] one can fail to. A verdict that leaves the kind out has not met this clause, but what it finds about [Conformance] diff --git a/SPEC.md b/SPEC.md index 1e9c48e..590b782 100644 --- a/SPEC.md +++ b/SPEC.md @@ -479,11 +479,14 @@ The second is a mechanism gap: the project's [Toolchain] lacks a mechanism the [toolchain specification](TOOLING-SPEC.md) requires, such as a proving harness or an [Identifier] resolver. A mechanism gap does not put the [Class] out of reach. The [Practitioner] still builds the [Rule] and records the mechanism gap alongside it. Where the missing mechanism would have -served clause 3.3 or 3.6, the [Practitioner] uses what those clauses allow in its place: the -acceptance or smoke check that clause 3.3 accepts where a [Detector] makes [Fixture]-based testing -impractical, and an [Identifier] in a form clause 3.6 allows that can be followed without a -resolver, such as an anchor in a documentation file. Every obligation of those clauses that the substitute -does not stand in for still applies in full. A mechanism gap that is also a breach of the [toolchain specification](TOOLING-SPEC.md)'s +served clause 3.3 or 3.6, the [Practitioner] uses a substitute only where that clause allows one, +and only for what it stands in for. Clause 3.3's acceptance or smoke check is allowed only where +a [Detector] makes [Fixture]-based testing impractical, and it stands in only for the +[Fixture]-based test that clause's test-first paragraph asks for: Part A's red run, and Part C's +proof using [Fixture] code where the pattern is absent, are still owed. Clause 3.6's [Identifier] +in a form that can be followed without a resolver, such as an anchor in a documentation file, +stands in only for the resolver. Every other obligation of those clauses still applies in full. +A mechanism gap that is also a breach of the [toolchain specification](TOOLING-SPEC.md)'s own obligations, such as a [Toolchain] that cannot run its [Defences] on its own source, is recorded against the [Toolchain]'s claim under section 7 as well, because that is the fact its [Owner] needs. Where that record lives is the [Toolchain]'s own [Conformance] declaration, under the diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index d6f7869..6e5e15c 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -10,12 +10,14 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). must host a rule the practitioner writes, so `ACCEPTANCE.md` lists three or more such questions per document. Adding a keyed question raises the bar, as "The bar only moves up" allows. Prompted by the change to method clause 3.2 in the same pull request. After run 30, on - the author's instruction, "The bar only moves up" allows a question to be reworded only where + the editor's instruction, "The bar only moves up" allows a question to be reworded only where the new wording asks the same thing with the same certainty, marked against its key unchanged and no less strictly; whether it does is the editor's decision for each rewording, not delegated, the rewording is used from the next run whose readers start after it is committed and changes no earlier marks, and its line here gives the old and new wording and the element - of the unchanged key each added part asks for. `acceptance/README.md` and "After a failure" + of the unchanged key each added part asks for, or says that it asks for none; the two + sentences that said "the author's instruction" now say "the editor's", the one party + `CONTRIBUTING.md` names. `acceptance/README.md` and "After a failure" point to it, and the sentence saying the fix is never to a question names it. No question is reworded by this change. Prompted by runs 29 and 30, in which method 2d's key requires the order of the kinds and readers left it out, and two readers of run 30 quoted only one of the diff --git a/acceptance/README.md b/acceptance/README.md index 6a1162e..ffa6d04 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -39,8 +39,8 @@ allowed when a new clause is load-bearing; removing or softening one to make a r and rewording one is allowed only as [the section on the bar](../ACCEPTANCE.md#the-bar-only-moves-up) allows: where the new wording asks the same thing with the same certainty, on the editor's decision for that rewording, from the next run whose readers start after it is committed, and -recorded in [CHANGES.md](CHANGES.md) with the element of the unchanged key each added part asks -for. +recorded in [CHANGES.md](CHANGES.md) with the old and the new wording and, for each part the new +wording adds, the element of the unchanged key it asks for, or a statement that it asks for none. A process change may remove a limit a key puts on which misses count against the document, as [the section on the bar](../ACCEPTANCE.md#the-bar-only-moves-up) allows. The reader may quote a passage it found confusing; it is never asked to rate the document, say From ea5cc4c2d80b3356d3c62e1753578470ea489b1f Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 15:03:40 +0100 Subject: [PATCH 062/105] Method 3.2 and 3.5: review 3 findings R3-1 and R3-2 applied Clause 3.2 says the smoke check may be the run that makes Part A's red run, on the originating instance, and Part C's proof, on fixture code, since clause 3.3 sets neither proof's means; clause 3.5's summary of 3.2 gives its substitutes under the conditions 3.2 sets. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 8 ++++++-- SPEC.md | 8 +++++--- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1394744..99dba54 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -95,8 +95,12 @@ version entry with no cohort line has not been accepted and is not published. passage there, and section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) calls suppressions and baselines practices, not techniques, to follow it. Section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) says an exception exists only where the owner, not "a human", has discussed, agreed and documented it, as the owner's terminology entry and the section's own - list already say, after a reader of run 29 answered 2b with "a human" and not the owner. Accepted by run - NNN. Minor version when released. + list already say, after a reader of run 29 answered 2b with "a human" and not the owner. Clause [3.2](SPEC.md#32-build-the-net) then says that the + smoke check may be the run that makes Part A's red run, on the originating instance, and Part + C's proof, on fixture code, as clause [3.3](SPEC.md#33-prove-the-net-by-making-the-rule-fire) sets neither proof's means, and clause [3.5](SPEC.md#35-enforce-permanently-and-block)'s summary + of clause [3.2](SPEC.md#32-build-the-net) gives its substitutes under the conditions clause [3.2](SPEC.md#32-build-the-net) sets, after an + independent review found "still owed" could be read as leaving a rule with no pattern unprovable. + Accepted by run NNN. Minor version when released. ### 1.1.0, 2026-10-02 diff --git a/SPEC.md b/SPEC.md index 590b782..0e5c387 100644 --- a/SPEC.md +++ b/SPEC.md @@ -482,8 +482,10 @@ the [Rule] and records the mechanism gap alongside it. Where the missing mechani served clause 3.3 or 3.6, the [Practitioner] uses a substitute only where that clause allows one, and only for what it stands in for. Clause 3.3's acceptance or smoke check is allowed only where a [Detector] makes [Fixture]-based testing impractical, and it stands in only for the -[Fixture]-based test that clause's test-first paragraph asks for: Part A's red run, and Part C's -proof using [Fixture] code where the pattern is absent, are still owed. Clause 3.6's [Identifier] +[Fixture]-based test that clause's test-first paragraph asks for. Part A's red run, and Part C's +proof using [Fixture] code where the pattern is absent, are still owed, and the smoke check may +be the run that makes either, run on the code that proof names: the originating [Instance] for +Part A, [Fixture] code for Part C. Clause 3.6's [Identifier] in a form that can be followed without a resolver, such as an anchor in a documentation file, stands in only for the resolver. Every other obligation of those clauses still applies in full. A mechanism gap that is also a breach of the [toolchain specification](TOOLING-SPEC.md)'s @@ -775,7 +777,7 @@ a [Toolchain] with no consumer yet is not excused. Where a mechanism gap, as cla it, stops the [Toolchain] running its own entry point on its own source, the demonstration is made through the entry point of another project the [Toolchain] is genuinely installed into, and is owed there in full. Clause 3.2 lets the [Practitioner] use the substitutes it names for clauses -3.3 and 3.6 when a mechanism is missing; it allows none for this clause. +3.3 and 3.6, under the conditions it sets; it allows none for this clause. The demonstration here is through an entry point or it is not made. **Why**: the purpose is that the mistakes of the past become structurally impossible to repeat, and From d20c57145a3330d549bd367ee138c95f9d011930 Mon Sep 17 00:00:00 2001 From: joseph Date: Sat, 10 Oct 2026 15:22:35 +0100 Subject: [PATCH 063/105] Method 3.2: the smoke check named as clause 3.3 names it, one that demonstrates the Rule firing Section 1 lists a smoke check as a Runner, so without the qualifier it could be read as any behavioural check. The words are clause 3.3's own. --- SPEC.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index 0e5c387..0255dba 100644 --- a/SPEC.md +++ b/SPEC.md @@ -480,7 +480,8 @@ The second is a mechanism gap: the project's [Toolchain] lacks a mechanism the resolver. A mechanism gap does not put the [Class] out of reach. The [Practitioner] still builds the [Rule] and records the mechanism gap alongside it. Where the missing mechanism would have served clause 3.3 or 3.6, the [Practitioner] uses a substitute only where that clause allows one, -and only for what it stands in for. Clause 3.3's acceptance or smoke check is allowed only where +and only for what it stands in for. Clause 3.3's acceptance or smoke check that demonstrates the +[Rule] firing is allowed only where a [Detector] makes [Fixture]-based testing impractical, and it stands in only for the [Fixture]-based test that clause's test-first paragraph asks for. Part A's red run, and Part C's proof using [Fixture] code where the pattern is absent, are still owed, and the smoke check may From 4383fb8a45812f01f141dc695445ce74f8bead6d Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 11:34:10 +0100 Subject: [PATCH 064/105] Acceptance process: the test proves the clarity of the change, with keys that ask what the question asks Keys list the required elements per part of each question, questions ask for exactly those, a confusion finding is settled run by run and blocks only in text the change wrote or altered (others go to acceptance/ROUTED.md), single misses count against the document only where the key names the reading, and open marking points are settled in the rules before the next run rather than invented per run. Co-Authored-By: Claude Fable 5.1 --- ACCEPTANCE.md | 252 ++++++++++++++++++++++-------- CONTRIBUTING.md | 32 ++-- acceptance/CHANGES.md | 36 +++++ acceptance/README.md | 51 ++++-- acceptance/ROUTED.md | 32 ++++ acceptance/detector/KEY.md | 129 +++++++++------ acceptance/detector/QUESTIONS.md | 9 +- acceptance/method/KEY.md | 107 +++++++------ acceptance/method/QUESTIONS.md | 3 +- acceptance/runs/README.md | 25 +-- acceptance/toolchain/KEY.md | 116 ++++++++------ acceptance/toolchain/QUESTIONS.md | 3 +- 12 files changed, 546 insertions(+), 249 deletions(-) create mode 100644 acceptance/CHANGES.md create mode 100644 acceptance/ROUTED.md diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 31c8df0..84ab39b 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -6,49 +6,76 @@ document that only a strong reader can follow is not a specification, and the re matter most, coding agents meeting the method in a failing pipeline, are often the weakest. The test is run with a low-strength model so that clarity is measured at the floor, not the ceiling. +The test proves one thing: that a capable reader, reading the document cold and once, takes from +it what the document exists to say, and is not confused by the text the change put there. The +keyed questions prove the document; the confusion question proves the change. Everything below +serves that, and a rule that stops it proving that is a defect in this file, fixed as +[Changes to this process](#changes-to-this-process) describes. + ## The cohort -- **At least five independent readers**, each a fresh instance of a low-strength model (the - smallest current model in its family, for example Haiku), with no shared context and no - access to the change history, the critique findings or each other. -- Each reader receives **only the changed document or documents** and the fixed question set - below. Nothing else is read first. +- **An odd number of independent readers, at least five**, each a fresh instance of a + low-strength model (the smallest current model in its family, for example Haiku), with no + shared context and no access to the change history, the critique findings or each other. + +- **Each reader is a separate process with no tools and no project context, only its packet**: + started from an empty directory outside any repository, with no user, project or local settings + files, no `CLAUDE.md`, skills, plugins, hooks or MCP servers, and no saved session, so that no + project instruction reaches it. The model tool's own fixed system prompt is not project + context. For example, with the packet on standard input: + + ``` + claude -p --safe-mode --model haiku --tools "" --setting-sources "" --strict-mcp-config --no-session-persistence --output-format json < packet.md + ``` + + A reader is set aside, and its answers do not count, when its answer text is empty, when its + process reports an error, or when the model version it reports is not the cohort's model. A run + with a reader set aside is void: it takes a run number and has a record, and a fresh cohort + runs. No reader is set aside for any other cause. + +- Each reader receives **only its packet**: the document's question set, the changed document + and the fixture the questions name, each in full in the reader's prompt, the question set + first. Each file is preceded by exactly one line, `===== FILE: =====`, where `` is + the name the question set gives the file (so `fixtures/remediation-ledger.md`, and + `QUESTIONS.md` for the question set itself). The question set carries the instruction to the + reader, and nothing else is added. Every reader in a run receives the same packet. + - Readers are run **after** every stronger review has been applied. They are the last gate, not the first draft. ## The question set -Written before the run, with an **answer key** for every keyed question drawn verbatim from the -document, so that marking is mechanical. Each reader answers all of them. +Written before the run, with an **answer key** for every keyed question, so that marking is +mechanical. The key states, for each keyed item, exactly what an answer must contain and what +makes it wrong; the question asks for exactly that and nothing the key does not expect. A key +asks for nothing the question does not. Each reader answers all of them. 1. **Restatement.** In their own words, what the document asks of its subject, in at most six bullets. Keyed against the list of MUST clauses: a bullet is correct when it names an obligation the document actually states. -2. **Load-bearing statements.** Three questions, fixed per document, each asking for one - sentence and a quotation: +2. **Load-bearing statements.** Three or more questions, fixed per document, each asking for a + short answer and the sentence or sentences of the document it rests on, quoted. Each has one + or more parts, and the key lists the element an answer must state for each part: - `SPEC.md`: what a practitioner does before fixing a defect, in order; who may baseline or - suppress; what "conforms" means for a remediation. + suppress, and whether an agent may; what "conforms" means for a remediation and what a + verdict rests on. - `DETECTOR-SPEC.md`: what "conforms" means and whether a tool that predates the document can - conform; what a detector must do about an inline suppression route; where a bundled rule's - documentation must live. - - `TOOLING-SPEC.md`: where conformance is measured; how a grade is assigned when parts come - from different sources; how many grades a project that ships tooling has. + conform; what a detector must do about an inline suppression route and whether one of those + things is enough; where a bundled rule's documentation must live, at what version, and + whether it must be reachable offline. + - `TOOLING-SPEC.md`: where conformance is measured and what the toolchain is; how a grade is + assigned when parts come from different sources; how many grades a project that ships + tooling has and whether one implies the other. 3. **Application.** A fixture case supplied with the questions, graded clause by clause with - Yes, Partial or No and one line of reasoning. Keyed against the intended verdicts; a divergence - counts against the document only when the reader's reasoning shows the clause was read - differently, not when the fixture was misread. + Yes, Partial or No and one line of reasoning. Keyed against the intended verdicts, with the + readings the key names as ones the text invites counting against the document. 4. **Confusion.** Any passage the reader found confusing on its one reading, quoted with its clause number, and nothing else: not what would make it clearer, not a proposed change. -The cohort settles one thing: is the document clear, coherent and comprehensive. A reader getting -everything right after one reading proves it; a reader getting confused fails it. The document -is designed by its editor to resolve the comprehension failures the cohort shows; the cohort is -the acceptance layer, not the designer. - ## What the cohort is not asked The cohort may point at what it found confusing. It is never asked to rate the document, to say @@ -58,61 +85,162 @@ not there to design the text, and a document shaped by what its weakest reader s a document designed by that reader. The signals this test takes from the cohort are a keyed answer missed, a wrong verdict reasoned from the document, and a passage pointed at; what to do about any of them is the editor's decision. An earlier version of this test asked the reader -what would make a clause clearer and for a clarity rating; both were removed on the author's +what would make a clause clearer and for a clarity rating; both were removed on the editor's instruction for that reason, and neither is reinstated. +## Marking + +The editor marks each answer sheet against the document's key, or delegates the marking to a +separate marker that has seen neither the drafting nor the review. A delegated marker receives +the document, its question set, fixture and key at the commit the readers read, this file, the +answer sheets, and the facts the record needs that the sheets cannot show. Its marks are final +unless the editor shows that a rule here was misapplied, and a re-mark says which rule and why. + +Every mark is given by a rule in this file or in the key, as follows. Where neither settles a +mark, the marker gives the reader the benefit of the doubt, records the point in the run record, +and the editor settles it by a change to this file or to the key before the next run. A run +record states no judgement that is not written here or in the key; a judgement an earlier run +recorded is not a rule. + +- **Keyed items.** Question 1 as a whole is one keyed item, and so is each item of question 2 + and each answer of question 3 for which the key gives an intended answer. +- **Reading an answer.** An answer is read for its sense, not its wording: a paraphrase, a + synonym, or a defined term used in its defined sense states what it means. A reader quotes the + document when it gives consecutive words of the document, in the document's order and of any + length, in quotation marks or a block quote, verbatim apart from letter case, quotation-mark or + apostrophe style, and markdown link or emphasis markup. +- **Question 1** is correct when at least four bullets each name an obligation the key lists and + no bullet asserts one the key marks wrong. +- **Question 2.** An answer is the reader's sentences and its quotation read together. It is + correct when it states every element the key lists as required for that item, in the + document's words or the reader's own, asserts nothing the key marks wrong, and quotes the + document. An element stated only in the quotation is stated. Anything in the key beyond the + required elements and the wrong assertions is explanation for the marker, and an answer is + never marked down for leaving it out. +- **Question 3.** Each item is correct when its grade is one the key accepts and its reasoning + mentions what the key says it must mention for that grade. A grade the key accepts without a + condition is correct whatever the reasoning. +- **A miss that counts against the document.** The key names, for question 1 and for each item + of question 3, the wrong readings the text invites. A miss of that kind blocks whatever the + count, since one reader in five taking that reading means one agent in five acting on it. Every + other single miss under questions 1 and 3 is the reader's and is tolerated by the threshold + below; a second reader making the same miss is a failed threshold, not a judgement. +- **An unanswered question.** Each keyed item left unanswered is a miss. Question 4 left + unanswered counts as no passage named. +- **Question 4.** A reader names a passage when it quotes words of the document or gives a clause + number with a description that identifies a sentence or paragraph. A remark that is not about + a passage of the document, such as that another document was not supplied, is recorded as a + note and is not a passage. Two readers name the same passage when the words or sentences they + name lie in the same paragraph; a heading, a list item, a table row and a terminology entry are + each a paragraph of their own. A passage named by two or more readers of the run is a finding. +- **Where a finding lies.** The record says, for each finding, whether it lies in text the change + wrote or altered: a sentence or paragraph the pull request adds, rewords or moves, read against + the branch it targets. A finding there blocks the change. A finding anywhere else is a finding + against the document, not against the change: it is recorded in + [acceptance/ROUTED.md](acceptance/ROUTED.md), routed to a change of its own, and does not + block. The change is judged on what it changed; the document is proved by the keyed questions, + which every run answers over the whole of it. + ## Pass criteria -The change passes when all of the following hold on a single run: +The change passes when all of the following hold on a single run. The thresholds apply to the +readers whose answers count; with five readers, "every reader" is five of five and "all but one" +is four of five. -| Criterion | Threshold | -| ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | -| Load-bearing statements (question 2) | **5 of 5** readers correct on every one | -| Restatement and application (questions 1 and 3) | **At least 4 of 5** readers correct on each keyed item | -| A wrong answer traceable to the text | **None.** A miss whose reasoning quotes the document is a finding, whatever the count | -| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it | -| Anything from one reader only | Recorded; applied at the editor's discretion; does not block | +| Criterion | Threshold | +| ------------------------------------------------------------------------------------- | -------------------------------------------------------------- | +| Load-bearing statements (question 2) | **Every** reader correct on every item | +| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | +| A miss the key says counts against the document | **None**, whatever the count | +| A passage found confusing by two or more readers, in text the change wrote or altered | **None.** It is a finding and blocks until the text is changed | +| A passage found confusing by two or more readers, elsewhere | Recorded in `acceptance/ROUTED.md` and routed; does not block | +| Anything from one reader only | Recorded; applied at the editor's discretion; does not block | The asymmetry is deliberate. A load-bearing statement is the sentence the whole document exists to make, and one reader in five getting it wrong means one agent in five will act on the wrong reading; unanimity is the only acceptable result there. Everywhere else a single divergent reader -is more likely to be the reader than the text, so four of five is tolerated, but a miss that the -reader can justify from the document is always the document's fault. - -## The bar only moves up - -No criterion in this document is ever relaxed to let a change through. If the cohort is confused, -the test is working and the text is wrong; the fix is to the text, never to the threshold, the -question, the fixture or the key. A threshold may be raised, a keyed question may be added, and a -key may be corrected where it misquoted the document; none of those makes a failing run pass. The -one time a threshold was relaxed, after three runs, the relaxation was reversed on the author's -instruction and is recorded here so that it is not tried again. Removing the clarity rating and -the request for suggestions was not a relaxation: it changed what the cohort is asked, not how -strictly an answer is marked. +is more likely to be the reader than the text, so one such reader is tolerated, unless the key +says the text invites that reading. A finding is settled run by run: a passage two readers named +in one run has no standing in the next, which is judged on its own readers, and a passage the +fresh cohort names by fewer than two is clear. + +## Changing the test + +Nothing in this test is changed to let a run through. A threshold, a question, a key or a +fixture is changed only before a run's readers start, never changes a mark already given, and +is recorded with its reason: a threshold, a question or the rules here by a change to this +process, a key by step 4 of `CONTRIBUTING.md` when the text it quotes is reworded or when a +load-bearing obligation is added, and each in [acceptance/CHANGES.md](acceptance/CHANGES.md). A +question is reworded only so that it asks for what its key requires, and a key's required +elements are never more than its question asks for. A run made after such a change is a fresh +run under the new rules, not a re-mark of an old one. The one time a threshold was relaxed to +let a change through, after three runs, the relaxation was reversed on the editor's instruction +and is recorded here so that it is not tried again. Removing the clarity rating and the request +for suggestions was not that: it changed what the cohort is asked, not how strictly an answer is +marked. ## After a failure -Fix the text, never the questions, and run a fresh cohort. Readers are not reused across runs. -The number of runs it took is recorded; a document that needed three runs is telling its editor -something. +Fix the text the findings name and run a fresh cohort; readers are never reused. A keyed miss is +fixed in the text that carries the obligation, wherever the change left it. A finding on text the +change wrote or altered is fixed there. A finding elsewhere goes to `acceptance/ROUTED.md` and +is not fixed in this change. The number of runs it took is recorded; a document that needed three +runs is telling its editor something. ## When it runs -Once per pull request, on the final text, after the editor's rewrite and any key update. It is -the last gate before the merge to `next`, so that every change on `next` is one the cohort has -accepted, and the release branch needs no cohort of its own unless it changes specification text. -A wording change after the run means a fresh run; readers are never reused. +On the final text of each pull request, after the editor's rewrite and any key update, one cohort +per changed document. It is the last gate before the merge to `next`, so that every change on +`next` is one the cohort has accepted, and the release branch needs no cohort of its own unless it +changes specification text. A fresh cohort runs after every change to the pull request's own +text that follows a run, and after every void run; a text that failed is never run again +unchanged. A version number or date that `PUBLISHING.md` or step 5 of `CONTRIBUTING.md` has the +editor set is not such a change, and neither is a merge of `next` into the branch that brings in +text another cohort accepted and alters no sentence this change wrote. ## The record -Each run has a file under `acceptance/runs/`, named `NNN-.md` with `NNN` the next number in -the one sequence shared by the three documents, stating: the document, the pull request and the -commit read, the cohort size and model, the run number in that pull request, the marks per -reader per question, the findings applied, and whether it passed. The changelog line for the -change names the file. CI refuses a specification change whose pull request adds no run file; -it checks for at least one, and a pull request that changes two documents adds a run for each. - -Each published version's changelog entry carries the lines of the changes it contains, so a -version's cohort record is the set of runs those lines name. A version with no such line has not -been accepted and MUST NOT be described as published. The question sets and answer keys live -under `acceptance/` in this repository so that a run can be repeated and compared. +Each run has a file under `acceptance/runs/`, named `NNN--run-.md` with `NNN` the next +number in the one sequence shared by the three documents and `` the run number within the pull +request, stating: the document; the pull request and the commit read; the cohort size and model, +with the model version each reader reported; how the readers were isolated and the command they +were started with; any reader set aside and why; who marked and, for a delegated marker, how it +was started; the marks per reader per keyed item, with each miss and the reader's reasoning in +brief; every passage named under question 4, with the words quoted or the description given, its +clause and the readers naming it; each finding and whether it lies in text the change wrote or +altered; any point the rules did not settle; whether the run was made under a process change +before it merged and, if so, at which commit; the findings applied afterwards; the verdict; and +a link to the answer sheets. + +The answer sheets are not kept in this repository. Every reader's whole output as its process +returned it, the exact prompt every reader received, the command the readers were started with, +and the instruction a delegated marker was given are posted in one or more comments on the pull +request, linked from the record and never edited, hidden or deleted after posting. + +The changelog line for the change names the record. CI refuses a specification change whose pull +request adds no run file; it checks for at least one, and a pull request that changes two +documents adds a run for each. Each published version's changelog entry carries the lines of the +changes it contains, so a version's cohort record is the set of runs those lines name. A version +with no such line has not been accepted and MUST NOT be described as published. The question sets +and answer keys live under `acceptance/` in this repository so that a run can be repeated and +compared. + +`acceptance/ROUTED.md` lists every finding routed from a run, open until a change that fixes +its passage passes a cohort, or until the editor closes it with a recorded reason. The editor +reads that list before cutting a release. + +## Changes to this process + +The process files are this file, `acceptance/README.md`, `acceptance/runs/README.md`, +`acceptance/ROUTED.md`, `CONTRIBUTING.md` and the checks that enforce them: `tools/changes.py`, +the key-quotation checks in `spec-qa.py`, the CI workflow and the pull request template. A change +to any of them, or to a question set, is a change to this process. They are not specifications, +and a change to them is not tested by a cohort: the cohort measures whether a low-strength reader +can follow a specification, and the process is followed by the editor. Such a change names the +runs or the case that showed the need for it, and goes through steps 1, 3 and 6 of +`CONTRIBUTING.md`, with no scope, key or cohort step; its review reads every process file it +changes. It applies to every run whose readers start after it merges; a run made before then may +be made under it, following all of it at a commit named in the record before the readers start. +It is recorded by a line added at the top of [acceptance/CHANGES.md](acceptance/CHANGES.md), +naming its pull request, not in the changelog, which is for the three specifications. A change to +a key or a fixture follows `acceptance/README.md`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fa1743e..013bb9d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,7 +29,9 @@ review. It is a patch version when released. SHOULD or MAY in a numbered section or appendix. It needs everything a clarity edit needs, plus the five tests in SCOPE.md answered in the pull request, and it is at least a minor version when released. `tools/changes.py` tells the two apart mechanically and CI runs it on every pull -request. +request. A change to the process files ACCEPTANCE.md names, this file among them, is neither: +ACCEPTANCE.md says how it is made, and it is recorded in `acceptance/CHANGES.md`, not in +CHANGELOG.md. ## What a pull request contains @@ -55,20 +57,26 @@ survives, the change is yours. the editor sets the pre-release header. 2. **Scope.** For an obligation, the editor confirms each SCOPE.md test. A failed test closes the pull request with a row in `DECLINED.md`. -3. **Review and rewrite.** An independent review of the diff against the document, then the - editor's rewrite, pushed to the pull request branch where the contributor allows it or to an - editor's branch that supersedes it. +3. **Review and rewrite.** An independent review of the diff against the whole document, by a + reviewer who did not draft the change, then the editor's rewrite, pushed to the pull request + branch where the contributor allows it or to an editor's branch that supersedes it. The pull + request names the reviewer and links or contains the review. 4. **Keys.** Where the change touches a sentence an acceptance key quotes, or adds a load-bearing - obligation, the editor updates the key before the cohort runs, never after. -5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text. Its record is - committed under `acceptance/runs/`, the `Unreleased` changelog line names it, and CI refuses - a specification change that lacks one. For an obligation, the editor sets the document's - header to `-dev, unpublished` in the same commit if it still names a published - version. A wording change after the run means a fresh run. + obligation, the editor updates the key before the cohort runs, never after. The key's + required elements stay what its question asks for; `acceptance/README.md` says what a key + update may and may not do. +5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text, one cohort per + changed document. Its record is committed under `acceptance/runs/`, the `Unreleased` + changelog line names it, and CI refuses a specification change that lacks one. For an + obligation, the editor sets the document's header to `-dev, unpublished` in the + same commit if it still names a published version. A change to the pull request's own text + after the run means a fresh run, as ACCEPTANCE.md says under "When it runs". A finding the + run routes to `acceptance/ROUTED.md` is not fixed in this pull request. 6. **Merge** into `next` by merge commit, and the branch is deleted. -Steps 2 to 5 are done by a person with a Claude Code session, not by CI, and they happen in -batches. A quiet pull request is not a declined one; a declined one says so. +Steps 2 to 5 are the editor's, not CI's, and they happen in batches. The editor may have an +agent draft, review or mark, as ACCEPTANCE.md allows for marking; every decision in these steps +stays the editor's. A quiet pull request is not a declined one; a declined one says so. ## Versions and releases diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md new file mode 100644 index 0000000..a076ff1 --- /dev/null +++ b/acceptance/CHANGES.md @@ -0,0 +1,36 @@ +# Changes to the acceptance process + +Changes to the process files that [ACCEPTANCE.md](../ACCEPTANCE.md) names, recorded here as it +describes: a line is added at the top for each change, naming its pull request. The process is +not a specification and is not versioned, and a change to it is not tested by a cohort. Changes +to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). + +- Pull request NNN: the test is made to prove what it exists to prove, the clarity of the + document to a cold reader, after thirty-four runs on two changes (19 to 30, 31 to 33 and 35 + to 52) in which keyed answers reached five of five and the runs still failed. Four defects in + the process were found. Keys required elements the question never asked for (detector 2b's + "need not remove or forbid", detector 2c's "at a version tracked together", and on pull + request 14 method 2d's order of the kinds), so readers who answered the question as put were + marked wrong: now each key lists, per item, the elements an answer must state, one for + each part of the question, and the questions are reworded to ask for exactly those; anything + else in the key only catches a wrong assertion. A passage two readers found confusing stayed + open whilst any single reader of a later cohort named any part of its paragraph, so cleared + passages kept coming back: now a finding is settled run by run, raised by two readers of one + run and clear when the next names it by fewer than two. A finding landed on text the change + never touched, so a change's cohort acted as a review of the whole document and could not + converge: now a finding in text the change wrote or altered blocks the change, and a finding + elsewhere is routed to `acceptance/ROUTED.md` and to a change of its own, whilst the keyed + questions still prove the whole document on every run. Markers invented judgements the rules + left open and later runs cited them as precedent: now an open point is marked for the reader + and settled in this file or the key before the next run, and a run record states no judgement + the rules do not. The rule that any miss whose reasoning quoted the document was a finding + whatever the count is replaced by the key naming the wrong readings the text invites, which + count against the document on one reader, every other single miss being tolerated by the + all-but-one threshold. Question 2's instruction asks for the sentence or sentences relied on, + since three keys rest on two. The cohort, the packet, set-aside readers, the posted sheets, + the fuller record and the route for a process change are kept from the earlier proposal in + pull request 13, which this change supersedes. Relative to the rules on `next`, the + traceable-miss row is narrower and the confusion row is applied to the change rather than + the document; both are recorded here as the reason the earlier rules could not converge, and + neither changes a mark already given. Prompted by runs 19 to 30 on pull request 14 and runs + 31 to 52 on the defined-terms change. diff --git a/acceptance/README.md b/acceptance/README.md index 4cfb6ec..49a097b 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -2,8 +2,10 @@ The question sets and answer keys that [ACCEPTANCE.md](../ACCEPTANCE.md) requires. One directory per specification document, each holding the fixed questions a cold reader answers and the key an -editor marks against. The fixture cases the application question uses live in `fixtures/`, and -the record of every run lives in `runs/`. +editor marks against. The fixture cases the application question uses live in `fixtures/`, the +record of every run lives in `runs/`, findings routed out of a run to a change of their own are +listed in [ROUTED.md](ROUTED.md), and changes to the acceptance process itself are recorded +in [CHANGES.md](CHANGES.md). | Document | Questions | Key | | ------------------ | ------------------------------------------------ | ------------------------------------ | @@ -14,24 +16,47 @@ the record of every run lives in `runs/`. ## How a run works 1. The editor identifies which documents the change touched. Only those are tested. -2. For each, five or more fresh low-strength readers each receive the document's current text, - its `QUESTIONS.md`, and the fixture the questions name. Nothing else: no key, no changelog, no - critique findings, no other reader's answers. -3. The editor marks each answer sheet against `KEY.md` and applies the pass criteria in - `ACCEPTANCE.md`. -4. The result is recorded as a file under [runs/](runs/), with cohort size, model, run number, - marks and the findings applied, and the change's `Unreleased` changelog line names that file. - A run that fails is followed by a change to the text, never to the questions, and a fresh - cohort. +2. For each, an odd number of fresh low-strength readers, at least five, each isolated as + `ACCEPTANCE.md` describes, receive the document's current text, its `QUESTIONS.md`, and the + fixture the questions name. Nothing else: no key, no changelog, no critique findings, no other + reader's answers. +3. The editor, or a separate marker the editor delegates to, marks each answer sheet against + `KEY.md` under the marking rules in `ACCEPTANCE.md` and applies its pass criteria. +4. The result is recorded as a file under [runs/](runs/), stating what `ACCEPTANCE.md` lists, + with the answer sheets posted on the pull request and linked from it, and the change's + `Unreleased` changelog line names that file. A run that fails is followed by a change to the + text the findings name and a fresh cohort; a finding on text the change did not write or + alter is added to [ROUTED.md](ROUTED.md) instead. + +## How a key is written + +A key says, for each keyed item, exactly what is marked and nothing a marker has to guess: + +- **Question 1**: the obligations a bullet may name, and the assertions that make a bullet wrong. +- **Question 2**: for each item, the elements an answer is required to state, numbered, one for + each part of the question; the assertions that make an answer wrong; and the sentence or + sentences of the document the answer rests on, quoted verbatim. The required elements are + what the question asks for, no more: an element the question does not ask for is not + required, however load-bearing, until the question is reworded to ask for it. +- **Question 3**: for each item, the grade or grades accepted, any condition on the reasoning, + and the wrong readings the text invites, which count against the document when one reader + takes them. + +Everything else in a key is explanation for the marker and never marks an answer down. ## Keeping the keys honest A key quotes the document. When a clause is reworded, the key line that quotes it is updated in the same commit, and the change is one the acceptance run itself then tests. A key that no longer -matches the document is a defect in this directory, not a reason to mark a reader wrong. +matches the document is a defect in this directory, not a reason to mark a reader wrong. A key +update under step 4 of `CONTRIBUTING.md` changes a quotation to follow the text, or adds a keyed +item for a new load-bearing obligation; it does not change what an existing item requires. A +change to what an item requires, or to a question, is a change to the process and is recorded in +[CHANGES.md](CHANGES.md) with its reason, before the readers of the next run start. Questions are fixed so that runs can be compared across versions. Adding a keyed question is -allowed when a new clause is load-bearing; removing or softening one to make a run pass is not. +allowed when a new clause is load-bearing; removing or softening one to make a run pass is not; +rewording one is allowed so that it asks for what its key requires, and for no other reason. The reader may quote a passage it found confusing; it is never asked to rate the document, say what would make a passage clearer, or propose a change: the cohort confirms comprehension and does not design the text. diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md new file mode 100644 index 0000000..bb82718 --- /dev/null +++ b/acceptance/ROUTED.md @@ -0,0 +1,32 @@ +# Passages routed out of a cohort + +A finding is a passage two or more readers of one cohort named as confusing, under question 4, +that lies outside the text the change being tested wrote or altered. `ACCEPTANCE.md` routes such +a finding here rather than blocking the change whose cohort raised it, so that each change is +judged on what it changed and the document is still made clearer, one change at a time. The +editor reads this list before cutting a release. + +A finding is open until a change that reworks its passage passes a cohort, when the line names +that run and moves below the rule, or until the editor closes it with a reason recorded here. A +finding raised again by a later run is noted on its line, not listed twice. + +## Open + +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | -------------------------------------- | +| `TOOLING-SPEC.md` | 9 and 9.1 | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | 2, 4.2, 8.1 | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | 9.2 | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | 4.3 | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | run 33 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | 4.1 | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | 5.3 | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | open | +| `DETECTOR-SPEC.md` | 7.1 | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | 8.1 | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | 5.3 | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | +| `SPEC.md` | 1 (Owner) | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50 | open | +| `SPEC.md` | 4 | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 (one reader, 29) | open; a method question for the editor | +| `SPEC.md` | 2 and 3.1 | "technique" used for a way of writing the Rule in section 2 and for a way of searching in clause 3.1 | runs 29, 38 | pull request 14 rewords both | + +## Closed + +None yet. diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 132e6cb..b0a3f97 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -1,8 +1,9 @@ # Answer key for DETECTOR-SPEC.md -Marked against the document at the version named in the changelog entry that records the run. -Quotations below are from the document; when a clause is reworded, update the quotation here in -the same commit. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, +update the quotation here in the same commit. `ACCEPTANCE.md` says how each item is marked; +`acceptance/README.md` says what each part of this key is for. ## 1. Restatement @@ -11,67 +12,93 @@ bullet asserting an obligation the document does not state, is a correct restate - Support bespoke rules written by the project ([4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it)). - Provide a harness that runs a single rule against supplied code ([4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code)). -- Allow a stable identifier per rule and print it, unaltered, with every finding ([4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding)). +- Allow a stable identifier per rule and print it, unaltered, with every finding ([4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding)); a + bullet that says each rule carries a stable identifier names this obligation. - Be invocable locally by the practitioner, over a subset down to one file, with the result in - the command's own output and never only through a hosted service or CI-only mode ([5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally)). + the command's own output and never only through a hosted service or CI-only mode ([5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally)); + a bullet that says every finding, or each finding, must appear in that output names [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran), + whose summary may leave findings out, and is not marked wrong for the plural. - Resolve a printed identifier to documentation; for bundled rules from the installed copy, offline, with the documentation shipped and versioned with the rule ([6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) to [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together)). - Make any inline suppression route detectable or disableable ([7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)). Wrong if it asserts: the detector must enforce a project's suppression policy; the detector must declare a version to conform; the detector must list a project's active defences or produce an -agent summary (those belong to the toolchain specification). +agent summary (those belong to the toolchain specification). A bullet that states what +conformance means, or restates a SHOULD as advice, asserts no obligation and is neither correct +nor wrong. + +Wrong readings the text invites, which count against the document on one reader: none named. ## 2. Load-bearing statements -- **2a.** Conformance is satisfying every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression), judged on evidence; a detector - that predates the document can conform, and the declaration is optional. Key quotation, section - 8: "A Detector Conforms if it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression)." Supporting, [8.1](../../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps): "A - maintainer MAY declare the version of this document the detector conforms to". An answer that - says a declaration is required, or that partial satisfaction is conformance, is wrong. -- **2b.** It must make the route either disableable by configuration or detectable by a rule the - project can write or a mechanical check it documents; it is not required to remove or forbid - it. Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each one it MUST do at least one of two things, and either alone - satisfies this clause: make the route disableable by configuration; or make the route - detectable, by a Rule the project can write in the Detector itself or by a mechanical check the - Detector documents". An answer that says the detector must forbid the route, or must require a - reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD), is wrong. -- **2c.** With the rule, in the installed copy, at a version tracked together, and yes, offline. - Key quotations, [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access): "Resolution of a bundled rule's identifier MUST work from the installed - copy, without network access"; [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together): "A bundled rule's documentation MUST ship with the rule, at - a version tracked together". An answer naming the detector's website as an acceptable location - is wrong. +- **2a.** Required: (1) a detector conforms when it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression); + (2) yes, a detector that predates the document can conform, because conformance is judged on + the evidence and a declaration is not a condition of it. Wrong if it says a declaration is + required, or that partial satisfaction is conformance. Key quotation, section [8](../../DETECTOR-SPEC.md#8-conformance): "A Detector + Conforms if it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression)." Supporting, [8.1](../../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps): "A maintainer + MAY declare the version of this document the detector conforms to". "Judged on evidence", "the + declaration is optional" or "graded on evidence by anyone" each state the reason in element + (2); one of them is enough. +- **2b.** Required: (1) it must let the project switch the route off in its configuration, or + make the route detectable (by a rule the project can write, or by a mechanical check the + detector documents; naming either means of detection, or just "detectable", states this); + (2) yes, either one alone is enough. Wrong if it says the detector must remove or forbid the + route, or must require a reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD). Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each one it + MUST do at least one of two things, and either alone satisfies this clause: make the route + disableable by configuration; or make the route detectable, by a Rule the project can write in + the Detector itself or by a mechanical check the Detector documents". An answer need not say + that the detector is not required to remove or forbid the route; that follows from element (2) + and the question does not ask it. +- **2c.** Required: (1) with the rule, in the installed copy; (2) at a version tracked together + with the rule; (3) yes, it must be reachable without network access. Wrong if it names the + detector's website, or any location reached over the network, as acceptable. Key quotations, + [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access): "Resolution of a bundled rule's identifier MUST work from the installed copy, without + network access"; [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together): "A bundled rule's documentation MUST ship with the rule, at a version + tracked together". "Shipped with the rule", "in the installed package" or "on disk with the + detector" each state element (1); "versioned with the rule" or "at the same version" states + element (2). ## 3. Application: Lintomatic 4.2 -| Clause | Intended verdict | Reasoning the fixture supports | -| ------ | ---------------- | -------------------------------------------------------------------------------------------------------------------- | -| [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules via the `Rule` interface, first class in every report | -| [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `check --only ` runs one rule, bundled or project, with a distinguishing exit code | -| [4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `LM-NNNN` fixed at first release; project names taken as given, not rewritten; printed on every finding line | -| [5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | Local, no account, no server, no network | -| [5.2](../../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Yes | `check ` accepts a single file | -| [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | One line per finding on standard output whether or not the HTML report is enabled | -| [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | The hosted dashboard shows nothing the command line does not print | -| [6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Yes | `explain ` resolves a printed identifier; project rules resolve to their declared `doc` string | -| [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Yes | `explain` reads the installed package; there is no website | -| [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | No | Two of forty bundled rules print `No documentation declared`; every identifier a bundled rule can print MUST resolve | -| [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | The ignore comment is disableable with `ignore_comments = false` | - -Section [8](../../DETECTOR-SPEC.md#8-conformance) verdict: **does not conform**, because [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) fails and partial conformance MUST NOT be -described as conformance. A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Partial on the grounds that thirty-eight resolve -has read the clause differently and that counts against the document; a reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Yes -because they missed the last paragraph of the fixture has misread the fixture, not the document. - -Clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation): **does not change the verdict**. It is a SHOULD, and Lintomatic's failure to check its -own identifiers in CI is a missed SHOULD, not a further MUST failure; the MUST that fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). - -A reader who marks [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) No because the comment takes no reason has confused [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) with [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason), which is -SHOULD; that counts against the document if the reasoning quotes [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) or [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason). +| Clause | Accepted grade | Condition on the reasoning, and what the fixture supports | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules via the `Rule` interface, first class in every report | +| [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `check --only ` runs one rule, bundled or project, with a distinguishing exit code | +| [4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `LM-NNNN` fixed at first release; project names taken as given, not rewritten; printed on every finding line | +| [5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | Local, no account, no server, no network | +| [5.2](../../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Yes | `check ` accepts a single file | +| [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | One line per finding on standard output whether or not the HTML report is enabled; a reader who notes that the HTML report need not be named for that reason is right | +| [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | The hosted dashboard shows nothing the command line does not print | +| [6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Yes | `explain ` resolves a printed identifier; project rules resolve to their declared `doc` string; the two undocumented bundled rules are [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together)'s failure | +| [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Yes | `explain` reads the installed package; there is no website | +| [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | No | Two of forty bundled rules print `No documentation declared`; every identifier a bundled rule can print MUST resolve | +| [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | The ignore comment is disableable with `ignore_comments = false` | + +No grade above carries a condition on the reasoning: the accepted grade is correct whatever the +reasoning, and a reader who adds a caveat about what the fixture does not say and still gives +the accepted grade is correct. + +Section [8](../../DETECTOR-SPEC.md#8-conformance) verdict: **does not conform**; a reader is correct who says so and names [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), the +two undocumented bundled rules, or partial conformance not counting, as the reason. + +Clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation): **does not change the verdict**; a reader is correct who says so and gives as the +reason that it is a SHOULD, or that the MUST which fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). + +Wrong readings the text invites, which count against the document on one reader: + +- [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) graded Partial on the grounds that thirty-eight of forty resolve, which has read the + clause as admitting a proportion. +- [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) graded No because the comment takes no reason, where the reasoning quotes [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) or [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason), + which has read [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason)'s SHOULD into [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable). + +A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Yes because they missed the last paragraph of the fixture has misread the +fixture, which is the reader's miss. ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage named, as `ACCEPTANCE.md` defines naming and the same +passage. A passage named by two or more readers of the run is a finding: it blocks the change +where it lies in text the change wrote or altered, and is routed to `acceptance/ROUTED.md` +otherwise. What to change is the editor's decision; the reader is not asked and any suggestion +it volunteers is discarded unread. diff --git a/acceptance/detector/QUESTIONS.md b/acceptance/detector/QUESTIONS.md index 58d06d1..2666b2f 100644 --- a/acceptance/detector/QUESTIONS.md +++ b/acceptance/detector/QUESTIONS.md @@ -10,14 +10,15 @@ In your own words and in at most six bullets, what does this document require of ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. +Answer each part of each question in a sentence or two, then quote the sentence or sentences of +the document you are relying on. - 2a. What does it mean for a detector to conform, and can a detector that was written before this document existed, and has never heard of it, conform? - 2b. A detector ships an inline ignore comment. What must the detector do about that route for - it to conform? -- 2c. Where must the documentation for a bundled rule's identifier live, and must it be - reachable without network access? + it to conform, and is doing one of those things enough? +- 2c. Where must the documentation for a bundled rule's identifier live, at what version, and + must it be reachable without network access? ## 3. Application diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 3ca1a80..182d6b2 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -1,8 +1,9 @@ # Answer key for SPEC.md -Marked against the document at the version named in the changelog entry that records the run. -Quotations below are from the document; when a clause is reworded, update the quotation here in -the same commit. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, +update the quotation here in the same commit. `ACCEPTANCE.md` says how each item is marked; +`acceptance/README.md` says what each part of this key is for. ## 1. Restatement @@ -18,62 +19,80 @@ bullet asserting an obligation the document does not state, is a correct restate with the rule ([3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation)). - Record a defect whose fix is deferred where the project's other decisions are enumerable, rather than only mentioning it (section [2](../../SPEC.md#2-when-the-method-applies)). +- Take a baseline, a suppression or an instance left unfixed to the owner, never on the + practitioner's own authority (section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom)). Wrong if it asserts: the practitioner fixes the reported instance first and defends afterwards; a test may serve as the detector; an agent may baseline the remaining instances. +Wrong readings the text invites, which count against the document on one reader: none named. + ## 2. Load-bearing statements -- **2a.** Attribute the defect to a class, express it as a detector rule, prove the rule fires, - then sweep and fix every instance, in that order. Key quotations, section [3](../../SPEC.md#3-the-method): "Six clauses, in +- **2a.** Required: (1) the four steps before the fix: attribute the defect to a class, express + the class as a rule in a detector, prove the rule fires, and sweep the codebase; (2) in that + order, with the fix after the sweep. Wrong if it has the fix before the rule, or the sweep + before the proof, or a test as the detector. Key quotations, section [3](../../SPEC.md#3-the-method): "Six clauses, in order."; [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance): "The Practitioner MUST run the Rule across the entire codebase and record the total Instance count before fixing anything, and MUST then fix every Instance found rather - than only the one". An answer that has the fix before the rule, or the sweep before the proof, - is wrong. -- **2b.** The owner of the codebase, always a human; an agent may not. Key quotation, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): - "An Agent MUST NOT Baseline, suppress or knowingly leave an Instance unfixed on its own - authority, whatever the Instance count turns out to be." An answer that lets the practitioner - decide because the count was small, or because the project recorded no rule, is wrong (the - default owner paragraph in the terminology entry for Owner still requires a recorded decision). -- **2c.** All six clauses of section [3](../../SPEC.md#3-the-method) were followed for that defect, and the verdict rests on the - reviewer reproducing the red and green runs, not on the report. Key quotations, section [7](../../SPEC.md#7-conformance): "A - remediation Conforms if all six clauses of section [3](../../SPEC.md#3-the-method) were followed for that Defect, and the - verdict that they were rests on reproduction"; "A verdict on a remediation or a Defence MUST - rest on reproduction, not on the report". An answer that accepts the pull request description - as the verdict is wrong, and so is one that states the six clauses and says nothing of - reproduction. + than only the one". An answer that names the steps by their clause numbers in order states + both elements. +- **2b.** Required: (1) the owner of the codebase, the human who holds that authority; (2) no, + an agent may not do so on its own authority. Wrong if it lets the practitioner or an agent + decide because the count was small, because the project recorded no owner, or at all. Key + quotation, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): "An Agent MUST NOT Baseline, suppress or knowingly leave an Instance + unfixed on its own authority, whatever the Instance count turns out to be." "The owner" names + the defined term and states that the decision is a human's; an answer need not add "a human" + to be correct. The default owner paragraph in the terminology entry for Owner still requires a + recorded decision, so an answer naming whoever instructed the practitioner, for a project that + has recorded nobody, names the owner and is correct. +- **2c.** Required: (1) all six clauses of section [3](../../SPEC.md#3-the-method) were followed for that defect; (2) the verdict + rests on reproduction, the reviewer rerunning the defence, not on the report. Wrong if it + accepts the pull request description or the practitioner's report as the verdict, or says + that most clauses suffice. Key quotations, section [7](../../SPEC.md#7-conformance): "A remediation Conforms if all six clauses + of section [3](../../SPEC.md#3-the-method) were followed for that Defect, and the verdict that they were rests on + reproduction"; "A verdict on a remediation or a Defence MUST rest on reproduction, not on the + report". "Reproduction" or "reproducing the runs" states element (2); the red and green runs + need not be named. ## 3. Application: the ledger remediation -| Clause | Intended verdict | Reasoning the fixture supports | -| ----------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | Class, hazard and the next wider rule are named; Partial or No is accepted where the reader notes that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | -| [3.2](../../SPEC.md#32-build-the-net) | Yes | A rule in the project's static analyser, reading code, not a test | -| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | Commit B is red on the fixture and the originating instance and survives as its own commit | -| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Five findings recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority; Partial accepted where the reasoning names that suppression | -| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | The rule is permanent and blocking, but the suppression was not a recorded decision; [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | -| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule; Partial accepted where the reasoning quotes the versioning sentence | +| Clause | Accepted grades | Condition on the reasoning, and what the fixture supports | +| ----------------------------------------------------------------------------------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | No condition. Class, hazard and the next wider rule are named; Partial or No is the reading that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | +| [3.2](../../SPEC.md#32-build-the-net) | Yes | A rule in the project's static analyser, reading code, not a test | +| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | Commit B is red on the fixture and the originating instance and survives as its own commit | +| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | The reasoning names the instance left unfixed behind an inline ignore on the agent's own authority; five findings were recorded | +| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | The reasoning names the suppression no recorded decision covers; the rule is otherwise permanent and blocking. [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | +| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | The reasoning names the documentation on a wiki, not versioned with the rule; the message is terse and carries an identifier | -Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**, because [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) and [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) were not followed. The -**defence does not conform** either, because [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) fails; a reader who says the defence conforms -because the sweep is what failed has missed [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) and that counts against the document only if the -reasoning quotes the section [7](../../SPEC.md#7-conformance) definition of a conforming defence. +Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**; a reader is correct who says so and names +[3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) or [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the clauses not followed. The **defence does not conform** either; a reader is +correct who says so and names [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the reasons, or [3.5](../../SPEC.md#35-enforce-permanently-and-block)'s unrecorded suppression, which +section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence also requires to be absent. Section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): the **inline ignore on `Report::render()`** was the owner's decision, not the -practitioner's. An answer naming instead the decision not to build the wider rule is half right: -that is also listed in section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) as the owner's to take, and an answer naming both is correct. -An answer naming neither is wrong. +practitioner's. A reader who names it is correct, with or without the decision to leave the +wider rule unbuilt, which section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) also gives to the owner. A reader who names only the wider +rule, or neither, is wrong. + +Wrong readings the text invites, which count against the document on one reader: + +- [3.5](../../SPEC.md#35-enforce-permanently-and-block) graded Yes because the rule fails the build, which has not applied the clause's opening + sentence on unrecorded suppression. +- [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) graded Yes because the URL resolves, which has read the versioning sentence of [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) + differently. +- The defence graded as conforming because the sweep is what failed, where the reasoning quotes + section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence. -A reader who grades [3.5](../../SPEC.md#35-enforce-permanently-and-block) Yes because the rule fails the build has not applied the clause's opening -sentence on unrecorded suppression, and that counts against the document. A reader who grades -[3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) Yes because the URL resolves has read the versioning sentence of [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) -differently, which counts against the document; a reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the -fixture. A reader who attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing -passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as covering it, which is a fair reading and is not marked wrong. +A reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the fixture, which is the reader's miss. A reader who +attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as +covering it, which is a fair reading and is not marked wrong. ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage named, as `ACCEPTANCE.md` defines naming and the same +passage. A passage named by two or more readers of the run is a finding: it blocks the change +where it lies in text the change wrote or altered, and is routed to `acceptance/ROUTED.md` +otherwise. What to change is the editor's decision; the reader is not asked and any suggestion +it volunteers is discarded unread. diff --git a/acceptance/method/QUESTIONS.md b/acceptance/method/QUESTIONS.md index 072a3c0..34fa82d 100644 --- a/acceptance/method/QUESTIONS.md +++ b/acceptance/method/QUESTIONS.md @@ -11,7 +11,8 @@ do when they are handed a defect? ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. +Answer each part of each question in a sentence or two, then quote the sentence or sentences of +the document you are relying on. - 2a. What does a practitioner do before fixing the defect they were handed, and in what order? - 2b. Who may adopt a baseline or suppress an instance, and may an agent do so on its own diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index 4994f74..3f7ad64 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -1,15 +1,22 @@ # Acceptance run records -One file per cohort run, `NNN-.md`, numbered in the one sequence the three documents -share. Runs 1 to 15 predate this directory and are summarised in the changelog entries for -1.0.0 and 1.0.1 of the method specification, 1.0.0 of the detector specification and 0.2.0 of -the toolchain specification; numbering here continues from 16. +One file per cohort run, `NNN--run-.md`, numbered in the one sequence the three +documents share, with `` the run number within its pull request. Runs 1 to 15 predate this +directory and are summarised in the changelog entries for 1.0.0 and 1.0.1 of the method +specification, 1.0.0 of the detector specification and 0.2.0 of the toolchain specification; +numbering here continues from 16. -A record states, in this order: the document; the pull request and the commit the readers were -given; the cohort size and model; the run number within that pull request; the marks per reader -per question against the key; the passages quoted as confusing and by how many readers; the -findings applied to the text as a result; and the verdict against the criteria in -`ACCEPTANCE.md`. Answer sheets are not pasted in full; the marks are the record. +A record states what `ACCEPTANCE.md` lists under "The record", in this order: the document; the +pull request and the commit the readers were given; the cohort size and model, with the model +version each reader reported; how the readers were isolated and the command they were started +with; any reader set aside and why; who marked and, for a delegated marker, how it was started; +the marks per reader per keyed item, with each miss and the reader's reasoning in brief; every +passage named under question 4, with the words quoted or the description given, its clause and +the readers naming it; each finding and whether it lies in text the change wrote or altered; any +point the rules did not settle; whether the run was made under a process change before it merged +and, if so, at which commit; the findings applied afterwards; the verdict; and a link to the +answer sheets posted on the pull request. Earlier records show the style only; where one differs +from this order, this file wins. Answer sheets are not kept here; the marks are the record. The `Unreleased` changelog line for the change names the record, and `tools/changes.py` refuses a specification change whose pull request adds no file here. diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 5a2602b..c11ae32 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -1,8 +1,9 @@ # Answer key for TOOLING-SPEC.md -Marked against the document at the version named in the changelog entry that records the run. -Quotations below are from the document; when a clause is reworded, update the quotation here in -the same commit. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, +update the quotation here in the same commit. `ACCEPTANCE.md` says how each item is marked; +`acceptance/README.md` says what each part of this key is for. ## 1. Restatement @@ -25,61 +26,72 @@ bullet asserting an obligation the document does not state, is a correct restate defences on its own source ([8.1](../../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation), [8.2](../../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source)). Wrong if it asserts: the toolchain must be a single product; a declaration is required to -conform; the agent summary (section [7](../../TOOLING-SPEC.md#7-agent-context)) is a MUST. +conform; the agent summary (section [7](../../TOOLING-SPEC.md#7-agent-context)) is a MUST. A bullet that calls agent support or the +self-audit a further condition "where it applies", or says a declaration is not a condition, +asserts no obligation and is neither correct nor wrong. + +Wrong readings the text invites, which count against the document on one reader: none named. ## 2. Load-bearing statements -- **2a.** At the project level; the toolchain is whatever the project has assembled from - third-party, first-party and project-level parts. Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "A project's - Toolchain Conforms if every MUST in sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record) holds across the assembled parts, wherever - each part came from". An answer that says conformance is graded per tool, or that only a - packaged product can conform, is wrong. -- **2b.** One grade; which part satisfied each clause is evidence, not a grade of its own. Key - quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "The project-level verdict is a single grade; which part of the Toolchain - satisfied each clause is evidence for that grade, not a second grade." -- **2c.** Two, graded and declared separately, and neither implies the other. Key quotation, - [9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately): "The two verdicts MUST be graded and declared separately, and neither implies the other." - An answer giving one grade, or saying the artefact grade follows from the project grade, is - wrong. +- **2a.** Required: (1) at the project level; (2) the toolchain is whatever the project has + assembled to run its checks through, from third-party, first-party and project-level parts. + Wrong if it says conformance is graded per tool, or that only a packaged product can conform. + Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "A project's Toolchain Conforms if every MUST in sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record) + holds across the assembled parts, wherever each part came from". An answer that says everything the + project runs its checks through, or the assembled parts from wherever they came, or that lists + the three kinds of part, states element (2). +- **2b.** Required: (1) one grade; (2) which part satisfied each clause is evidence for that + grade, not a grade of its own. Wrong if it gives a grade per part. Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): + "The project-level verdict is a single grade; which part of the Toolchain satisfied each + clause is evidence for that grade, not a second grade." +- **2c.** Required: (1) two, graded and declared separately; (2) neither implies the other. + Wrong if it gives one grade, or says the artefact grade follows from the project grade or the + reverse. Key quotation, [9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately): "The two verdicts MUST be graded and declared separately, and + neither implies the other." ## 3. Application: the Zigpipe project -| Clause | Intended verdict | Reasoning the fixture supports | -| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | Lintomatic fails detector [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) on two undocumented bundled rules and nothing in the wrapping closes that gap | -| [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)` | -| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | Ignore comments disabled project-wide; exceptions only through the record the toolchain reads | -| [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | `zig-qa run` is local, prints Lintomatic's output unchanged, one line per finding | -| [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | Lintomatic, then the tests, stopping on a Lintomatic failure | -| [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes, Partial or No | `zig-qa list` prints identifier, summary and documentation path without running the rules; Partial or No is accepted where the reasoning counts the two `(none)` routes, since 5.1 says the listing MUST provide the route and [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) point 2 says a gap may be named again by a later clause | -| [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | Derived from `lintomatic.toml`, not hand-maintained | -| [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | Yes | The `proj.*` rules appear in the same listing | -| [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes | `quality-exceptions.toml` is a defined path that `zig-qa run` loads | -| [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Partial or No | A justification is required and an empty one rejected, but nothing rejects a justification that could be pasted onto any entry | -| [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | `zig-qa list --exceptions` uses the same format and operation as the rule listing | - -Either Partial or No is accepted for [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one); the reasoning must mention the generic-justification -requirement. A Yes on [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) with reasoning that quotes only the first sentence of the clause is a -finding against the document. - -Section [9](../../TOOLING-SPEC.md#9-conformance) verdict: **does not conform**, because [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) and [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) fail; partial conformance MUST NOT be -described as conformance. **Not with agent support**, because section [7](../../TOOLING-SPEC.md#7-agent-context) is not met (no summary). -**Section [8](../../TOOLING-SPEC.md#8-self-audit) does not apply**, because Zigpipe ships nothing to other projects; a reader who applies -[8.1](../../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) to Zigpipe on the basis that Zigpipe's tooling includes a shipped package (zig-qa) has read -"where the Toolchain is one the project ships" (section [9](../../TOOLING-SPEC.md#9-conformance)) as reaching the consuming project, and that counts against the document. The -correct reading is that section [8](../../TOOLING-SPEC.md#8-self-audit) bears on zig-qa's own artefact grade, not on Zigpipe. - -Declaration: **does not affect the verdict**. Key quotation, [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance): "The declaration is optional." -A reader who says Zigpipe cannot be graded without one has read [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) wrongly. - -A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes on the grounds that Lintomatic "mostly" conforms has read either [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) -or the detector document's section [8](../../DETECTOR-SPEC.md#8-conformance) differently; it counts against this document only if the -reasoning quotes [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification). A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes because they missed the two undocumented rules -has misread the fixture. +| Clause | Accepted grades | Condition on the reasoning, and what the fixture supports | +| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | Lintomatic fails detector [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) on two undocumented bundled rules and nothing in the wrapping closes that gap; a reader who notes that the detector specification was not supplied and grades from this document's summary of it is correct | +| [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)`. The two are blocking, since their firing stops `zig-qa run`; a reader who grades No on that condition is correct | +| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | Ignore comments disabled project-wide; exceptions only through the record the toolchain reads | +| [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | `zig-qa run` is local, prints Lintomatic's output unchanged, one line per finding | +| [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | Lintomatic, then the tests, stopping on a Lintomatic failure | +| [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes, Partial or No | No condition. `zig-qa list` prints identifier, summary and documentation path without running the rules; Partial or No is the reading that counts the two `(none)` routes, since 5.1 says the listing MUST provide the route and [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) point 2 says a gap may be named again by a later clause | +| [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | Derived from `lintomatic.toml`, not hand-maintained | +| [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | Yes | The `proj.*` rules appear in the same listing | +| [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes or Partial | `quality-exceptions.toml` is a defined path that `zig-qa run` loads. Partial is accepted only where the reasoning says the fixture names a place for exceptions and none for calibrations or conventions, which the project record entry also covers | +| [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Partial or No | The reasoning mentions that nothing rejects a justification that could be pasted onto any entry; a justification is required and an empty one rejected | +| [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | `zig-qa list --exceptions` uses the same format and operation as the rule listing | + +Section [9](../../TOOLING-SPEC.md#9-conformance) verdict: **does not conform**; a reader is correct who says so and names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) or +[4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) among the failing clauses. **Not with agent support**; a reader is correct who says so because +section [7](../../TOOLING-SPEC.md#7-agent-context) is not met, there being no summary. **Section [8](../../TOOLING-SPEC.md#8-self-audit) does not apply**; a reader is correct +who says so because Zigpipe ships nothing to other projects, or because section [8](../../TOOLING-SPEC.md#8-self-audit) bears on +zig-qa's own artefact grade and not on Zigpipe. **The absence of a declaration does not affect +the verdict**; a reader is correct who says so. Key quotation, [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance): "The declaration is +optional." + +Wrong readings the text invites, which count against the document on one reader: + +- [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) graded Yes with reasoning that quotes only the first sentence of the clause, which has + not read the sentence on a generic justification. +- [8.1](../../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) applied to Zigpipe on the basis that its tooling includes a shipped package, which has + read "where the Toolchain is one the project ships" (section [9](../../TOOLING-SPEC.md#9-conformance)) as reaching the consuming + project. +- [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) graded Yes on the grounds that Lintomatic "mostly" conforms, where the reasoning quotes + [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), which has read it as admitting a proportion. +- Zigpipe held ungradeable for want of a declaration, which has read [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) as a condition. + +A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes because they missed the two undocumented rules has misread the +fixture, which is the reader's miss. ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage named, as `ACCEPTANCE.md` defines naming and the same +passage. A passage named by two or more readers of the run is a finding: it blocks the change +where it lies in text the change wrote or altered, and is routed to `acceptance/ROUTED.md` +otherwise. What to change is the editor's decision; the reader is not asked and any suggestion +it volunteers is discarded unread. diff --git a/acceptance/toolchain/QUESTIONS.md b/acceptance/toolchain/QUESTIONS.md index a4da266..35a2b91 100644 --- a/acceptance/toolchain/QUESTIONS.md +++ b/acceptance/toolchain/QUESTIONS.md @@ -11,7 +11,8 @@ toolchain? ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. +Answer each part of each question in a sentence or two, then quote the sentence or sentences of +the document you are relying on. - 2a. At what level is toolchain conformance measured, and what is the toolchain? - 2b. A project's tooling is assembled from a third-party detector, a first-party package and From 36d87d18fe3b2aa586472a45b5888812a6831aae Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 11:43:12 +0100 Subject: [PATCH 065/105] Acceptance process: review 1 findings applied One route for each kind of change to the test (quotation and new item in the specification pull request, everything else a process change); ROUTED.md and runs/ are records, not process files; question 1 defined once; open points marked in a stated direction; quoting needs a six-word run; a finding lies in the change's text when any sentence of its paragraph changed; wrong readings stated as grade plus reasoning; keys ask only what their questions ask (detector 2a, toolchain 2c); the question 3 tables separate the condition from the explanation. Co-Authored-By: Claude Fable 5.1 --- ACCEPTANCE.md | 123 ++++++++++++++++++------------ acceptance/CHANGES.md | 4 +- acceptance/README.md | 17 +++-- acceptance/ROUTED.md | 7 +- acceptance/detector/KEY.md | 17 ++--- acceptance/method/KEY.md | 37 ++++----- acceptance/toolchain/KEY.md | 49 ++++++------ acceptance/toolchain/QUESTIONS.md | 2 +- 8 files changed, 144 insertions(+), 112 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 84ab39b..c5bd8cd 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -51,8 +51,8 @@ makes it wrong; the question asks for exactly that and nothing the key does not asks for nothing the question does not. Each reader answers all of them. 1. **Restatement.** In their own words, what the document asks of its subject, in at most six - bullets. Keyed against the list of MUST clauses: a bullet is correct when it names an - obligation the document actually states. + bullets. Keyed against the key's list of the document's obligations: a bullet is correct when + it names one of them. 2. **Load-bearing statements.** Three or more questions, fixed per document, each asking for a short answer and the sentence or sentences of the document it rests on, quoted. Each has one @@ -97,34 +97,41 @@ answer sheets, and the facts the record needs that the sheets cannot show. Its m unless the editor shows that a rule here was misapplied, and a re-mark says which rule and why. Every mark is given by a rule in this file or in the key, as follows. Where neither settles a -mark, the marker gives the reader the benefit of the doubt, records the point in the run record, -and the editor settles it by a change to this file or to the key before the next run. A run -record states no judgement that is not written here or in the key; a judgement an earlier run -recorded is not a rule. +mark, the marker records the point in the run record and marks it as the open-point rule below +says, and the editor settles it in this file or in the key, as [Changing the test](#changing-the-test) +allows, before the next run. A run record states no judgement that is not written here or in +the key; a judgement an earlier run recorded is not a rule. - **Keyed items.** Question 1 as a whole is one keyed item, and so is each item of question 2 - and each answer of question 3 for which the key gives an intended answer. + and each answer of question 3 for which the key gives an accepted grade or answer. - **Reading an answer.** An answer is read for its sense, not its wording: a paraphrase, a synonym, or a defined term used in its defined sense states what it means. A reader quotes the - document when it gives consecutive words of the document, in the document's order and of any - length, in quotation marks or a block quote, verbatim apart from letter case, quotation-mark or - apostrophe style, and markdown link or emphasis markup. + document when it gives, in quotation marks or a block quote, a run of six or more consecutive + words of the document in the document's order, verbatim apart from letter case, quotation-mark + or apostrophe style, and markdown link or emphasis markup; a quotation that drops or changes a + word quotes the document by its longest such run. - **Question 1** is correct when at least four bullets each name an obligation the key lists and - no bullet asserts one the key marks wrong. + no bullet asserts one the key marks wrong. Several bullets that name different clauses of one + line of the key's list each count. A bullet asserting an obligation that is in neither list is + recorded as an open point for the key and is not counted either way. - **Question 2.** An answer is the reader's sentences and its quotation read together. It is correct when it states every element the key lists as required for that item, in the document's words or the reader's own, asserts nothing the key marks wrong, and quotes the document. An element stated only in the quotation is stated. Anything in the key beyond the required elements and the wrong assertions is explanation for the marker, and an answer is never marked down for leaving it out. -- **Question 3.** Each item is correct when its grade is one the key accepts and its reasoning - mentions what the key says it must mention for that grade. A grade the key accepts without a - condition is correct whatever the reasoning. +- **Question 3.** Each item is correct when its grade is one the key accepts and, where the key + sets a condition on the reasoning for that grade, the reasoning meets it. A grade the key + accepts with no condition is correct whatever the reasoning. A grade given on a stated + condition is read as the grade for the case the fixture presents, where the key says which + case that is; otherwise the item is unanswered. - **A miss that counts against the document.** The key names, for question 1 and for each item - of question 3, the wrong readings the text invites. A miss of that kind blocks whatever the - count, since one reader in five taking that reading means one agent in five acting on it. Every + of question 3, the wrong readings the text invites, each as a grade or answer together with + what the reasoning states or leaves out. A miss that matches one blocks whatever the count, + since one reader in five taking that reading means one agent in five acting on it, and it + blocks wherever the text lies, because the keyed questions are what prove the document. Every other single miss under questions 1 and 3 is the reader's and is tolerated by the threshold - below; a second reader making the same miss is a failed threshold, not a judgement. + below; a second miss on the same item fails the threshold, and no judgement is needed. - **An unanswered question.** Each keyed item left unanswered is a miss. Question 4 left unanswered counts as no passage named. - **Question 4.** A reader names a passage when it quotes words of the document or gives a clause @@ -134,12 +141,15 @@ recorded is not a rule. name lie in the same paragraph; a heading, a list item, a table row and a terminology entry are each a paragraph of their own. A passage named by two or more readers of the run is a finding. - **Where a finding lies.** The record says, for each finding, whether it lies in text the change - wrote or altered: a sentence or paragraph the pull request adds, rewords or moves, read against - the branch it targets. A finding there blocks the change. A finding anywhere else is a finding - against the document, not against the change: it is recorded in - [acceptance/ROUTED.md](acceptance/ROUTED.md), routed to a change of its own, and does not - block. The change is judged on what it changed; the document is proved by the keyed questions, - which every run answers over the whole of it. + wrote or altered: a paragraph in which the pull request, read against the branch it targets, + adds, rewords, moves or deletes a sentence, or changes a word, a term or a link. A finding + there blocks the change. A finding anywhere else is a finding against the document, not + against the change: it is recorded in [acceptance/ROUTED.md](acceptance/ROUTED.md), routed to a + change of its own, and does not block. The change is judged on what it changed; the document + is proved by the keyed questions, which every run answers over the whole of it. +- **Open points.** An open point in marking an answer is marked for the reader. An open point on + whether a reader named a passage, or whether two readers named the same one, counts the + passage. An open point on where a finding lies treats it as the change's own text. ## Pass criteria @@ -166,26 +176,37 @@ fresh cohort names by fewer than two is clear. ## Changing the test -Nothing in this test is changed to let a run through. A threshold, a question, a key or a -fixture is changed only before a run's readers start, never changes a mark already given, and -is recorded with its reason: a threshold, a question or the rules here by a change to this -process, a key by step 4 of `CONTRIBUTING.md` when the text it quotes is reworded or when a -load-bearing obligation is added, and each in [acceptance/CHANGES.md](acceptance/CHANGES.md). A -question is reworded only so that it asks for what its key requires, and a key's required +Nothing in this test is changed to let a particular run through. A threshold, a question, a +key, a fixture or a rule here is changed only before a run's readers start, never changes a mark +already given, and is changed in one of three ways, each recorded with its reason: + +- A key's quotation follows the text it quotes, and a key gains an item for a new load-bearing + obligation, with the question that asks for it, under step 4 of `CONTRIBUTING.md`, in the + specification pull request that makes the change; `acceptance/README.md` says what that may + and may not do. +- A threshold, a fixture, a rule here, what an existing key item requires, or the wording of a + question changes by a change to this process, with its own pull request. +- An open point a run recorded is settled by either route, whichever it needs. + +A question is reworded only so that it asks for what its key requires, and a key's required elements are never more than its question asks for. A run made after such a change is a fresh -run under the new rules, not a re-mark of an old one. The one time a threshold was relaxed to -let a change through, after three runs, the relaxation was reversed on the editor's instruction -and is recorded here so that it is not tried again. Removing the clarity rating and the request -for suggestions was not that: it changed what the cohort is asked, not how strictly an answer is -marked. +run under the new rules, not a re-mark of an old one. The test for a change to a rule is what +the rule measured: a rule that measured something other than whether a cold reader could follow +the text is a defect in this file and is corrected, and the correction says what the rule +measured instead; a rule that measured that, and found the text wanting, is not touched, and the +text is fixed. The one time a threshold was relaxed to let a change through, after three runs, +the relaxation was reversed on the editor's instruction and is recorded here so that it is not +tried again. Removing the clarity rating and the request for suggestions was not that: it +changed what the cohort is asked, not how strictly an answer is marked. ## After a failure -Fix the text the findings name and run a fresh cohort; readers are never reused. A keyed miss is -fixed in the text that carries the obligation, wherever the change left it. A finding on text the -change wrote or altered is fixed there. A finding elsewhere goes to `acceptance/ROUTED.md` and -is not fixed in this change. The number of runs it took is recorded; a document that needed three -runs is telling its editor something. +Fix the text the run's misses and findings name and run a fresh cohort; readers are never reused. +A keyed miss is fixed in the text that carries the obligation, wherever it lies, since the keyed +questions prove the document and a document one reader in five misreads is not merged to, whoever +wrote the sentence. A finding on text the change wrote or altered is fixed there. A finding +elsewhere goes to `acceptance/ROUTED.md` and is not fixed in this change. The number of runs it +took is recorded; a document that needed three runs is telling its editor something. ## When it runs @@ -217,8 +238,8 @@ returned it, the exact prompt every reader received, the command the readers wer and the instruction a delegated marker was given are posted in one or more comments on the pull request, linked from the record and never edited, hidden or deleted after posting. -The changelog line for the change names the record. CI refuses a specification change whose pull -request adds no run file; it checks for at least one, and a pull request that changes two +The changelog line for the change names every run made on it, the passing run last. CI refuses a +specification change whose pull request adds no run file; it checks for at least one, and a pull request that changes two documents adds a run for each. Each published version's changelog entry carries the lines of the changes it contains, so a version's cohort record is the set of runs those lines name. A version with no such line has not been accepted and MUST NOT be described as published. The question sets @@ -232,15 +253,17 @@ reads that list before cutting a release. ## Changes to this process The process files are this file, `acceptance/README.md`, `acceptance/runs/README.md`, -`acceptance/ROUTED.md`, `CONTRIBUTING.md` and the checks that enforce them: `tools/changes.py`, -the key-quotation checks in `spec-qa.py`, the CI workflow and the pull request template. A change -to any of them, or to a question set, is a change to this process. They are not specifications, -and a change to them is not tested by a cohort: the cohort measures whether a low-strength reader -can follow a specification, and the process is followed by the editor. Such a change names the -runs or the case that showed the need for it, and goes through steps 1, 3 and 6 of -`CONTRIBUTING.md`, with no scope, key or cohort step; its review reads every process file it +`CONTRIBUTING.md` and the checks that enforce them: `tools/changes.py`, the key-quotation checks +in `spec-qa.py`, the CI workflow and the pull request template. A change to any of them is a +change to this process, and so is a change to a threshold, a fixture, the wording of a question +or what a key item requires, as [Changing the test](#changing-the-test) says. They are not +specifications, and a change to them is not tested by a cohort: the cohort measures whether a +low-strength reader can follow a specification, and the process is followed by the editor. Such +a change names the runs or the case that showed the need for it, and goes through steps 1, 3 and +6 of `CONTRIBUTING.md`, with no scope, key or cohort step; its review reads every process file it changes. It applies to every run whose readers start after it merges; a run made before then may be made under it, following all of it at a commit named in the record before the readers start. It is recorded by a line added at the top of [acceptance/CHANGES.md](acceptance/CHANGES.md), -naming its pull request, not in the changelog, which is for the three specifications. A change to -a key or a fixture follows `acceptance/README.md`. +naming its pull request, not in the changelog, which is for the three specifications. +`acceptance/ROUTED.md` and `acceptance/runs/` are records, not process files: a specification +pull request writes to them. diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index a076ff1..cc01848 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -6,7 +6,7 @@ not a specification and is not versioned, and a change to it is not tested by a to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). - Pull request NNN: the test is made to prove what it exists to prove, the clarity of the - document to a cold reader, after thirty-four runs on two changes (19 to 30, 31 to 33 and 35 + document to a cold reader, after thirty-three runs on two changes (19 to 30, 31 to 33 and 35 to 52) in which keyed answers reached five of five and the runs still failed. Four defects in the process were found. Keys required elements the question never asked for (detector 2b's "need not remove or forbid", detector 2c's "at a version tracked together", and on pull @@ -33,4 +33,4 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). traceable-miss row is narrower and the confusion row is applied to the change rather than the document; both are recorded here as the reason the earlier rules could not converge, and neither changes a mark already given. Prompted by runs 19 to 30 on pull request 14 and runs - 31 to 52 on the defined-terms change. + 31 to 33 and 35 to 52 on the defined-terms change. diff --git a/acceptance/README.md b/acceptance/README.md index 49a097b..dcf39c0 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -48,11 +48,18 @@ Everything else in a key is explanation for the marker and never marks an answer A key quotes the document. When a clause is reworded, the key line that quotes it is updated in the same commit, and the change is one the acceptance run itself then tests. A key that no longer -matches the document is a defect in this directory, not a reason to mark a reader wrong. A key -update under step 4 of `CONTRIBUTING.md` changes a quotation to follow the text, or adds a keyed -item for a new load-bearing obligation; it does not change what an existing item requires. A -change to what an item requires, or to a question, is a change to the process and is recorded in -[CHANGES.md](CHANGES.md) with its reason, before the readers of the next run start. +matches the document is a defect in this directory, not a reason to mark a reader wrong. + +A key, a question set or a fixture changes in one of two ways, as `ACCEPTANCE.md` says under +"Changing the test", and always before the readers of the next run start: + +- **In a specification pull request, under step 4 of `CONTRIBUTING.md`**: a quotation follows + the text it quotes, or a keyed item is added for a new load-bearing obligation together with + the question that asks for it. The pull request's own changelog line records it. Such an + update does not change what an existing item requires, nor an existing question's wording. +- **In a process change, with its own pull request**: what an existing item requires, the + wording of a question, a fixture, or the rules of marking. It is recorded in + [CHANGES.md](CHANGES.md) with its reason. Questions are fixed so that runs can be compared across versions. Adding a keyed question is allowed when a new clause is load-bearing; removing or softening one to make a run pass is not; diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index bb82718..eb35815 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -8,7 +8,9 @@ editor reads this list before cutting a release. A finding is open until a change that reworks its passage passes a cohort, when the line names that run and moves below the rule, or until the editor closes it with a reason recorded here. A -finding raised again by a later run is noted on its line, not listed twice. +finding raised again by a later run is noted on its line, not listed twice. The rows below were +raised by runs 31 to 52 under the earlier rules, on untouched text of the change those runs +tested, and are listed here so that they are not lost. ## Open @@ -24,8 +26,7 @@ finding raised again by a later run is noted on its line, not listed twice. | `DETECTOR-SPEC.md` | 8.1 | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | | `DETECTOR-SPEC.md` | 5.3 | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | | `SPEC.md` | 1 (Owner) | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50 | open | -| `SPEC.md` | 4 | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 (one reader, 29) | open; a method question for the editor | -| `SPEC.md` | 2 and 3.1 | "technique" used for a way of writing the Rule in section 2 and for a way of searching in clause 3.1 | runs 29, 38 | pull request 14 rewords both | +| `SPEC.md` | 4 | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | open; a method question for the editor | ## Closed diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index b0a3f97..2f6fd96 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -33,13 +33,12 @@ Wrong readings the text invites, which count against the document on one reader: ## 2. Load-bearing statements - **2a.** Required: (1) a detector conforms when it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression); - (2) yes, a detector that predates the document can conform, because conformance is judged on - the evidence and a declaration is not a condition of it. Wrong if it says a declaration is + (2) yes, a detector that predates the document can conform. Wrong if it says a declaration is required, or that partial satisfaction is conformance. Key quotation, section [8](../../DETECTOR-SPEC.md#8-conformance): "A Detector Conforms if it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression)." Supporting, [8.1](../../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps): "A maintainer - MAY declare the version of this document the detector conforms to". "Judged on evidence", "the - declaration is optional" or "graded on evidence by anyone" each state the reason in element - (2); one of them is enough. + MAY declare the version of this document the detector conforms to". The reason, that + conformance is judged on evidence and a declaration is optional, is explanation: an answer + that gives it, or gives it instead of a plain yes, states element (2). - **2b.** Required: (1) it must let the project switch the route off in its configuration, or make the route detectable (by a rule the project can write, or by a mechanical check the detector documents; naming either means of detection, or just "detectable", states this); @@ -61,7 +60,7 @@ Wrong readings the text invites, which count against the document on one reader: ## 3. Application: Lintomatic 4.2 -| Clause | Accepted grade | Condition on the reasoning, and what the fixture supports | +| Clause | Accepted grade | What the fixture supports (no row carries a condition on the reasoning) | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules via the `Rule` interface, first class in every report | | [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `check --only ` runs one rule, bundled or project, with a distinguishing exit code | @@ -87,9 +86,9 @@ reason that it is a SHOULD, or that the MUST which fails is [6.3](../../DETECTOR Wrong readings the text invites, which count against the document on one reader: -- [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) graded Partial on the grounds that thirty-eight of forty resolve, which has read the - clause as admitting a proportion. -- [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) graded No because the comment takes no reason, where the reasoning quotes [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) or [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason), +- [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) graded Partial with reasoning that says most, or thirty-eight of forty, bundled rules + resolve, which has read the clause as admitting a proportion. +- [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) graded No with reasoning that says the comment takes no reason and quotes [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) or [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason), which has read [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason)'s SHOULD into [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable). A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Yes because they missed the last paragraph of the fixture has misread the diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 182d6b2..36bfafd 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -57,33 +57,34 @@ Wrong readings the text invites, which count against the document on one reader: ## 3. Application: the ledger remediation -| Clause | Accepted grades | Condition on the reasoning, and what the fixture supports | -| ----------------------------------------------------------------------------------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | No condition. Class, hazard and the next wider rule are named; Partial or No is the reading that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | -| [3.2](../../SPEC.md#32-build-the-net) | Yes | A rule in the project's static analyser, reading code, not a test | -| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | Commit B is red on the fixture and the originating instance and survives as its own commit | -| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | The reasoning names the instance left unfixed behind an inline ignore on the agent's own authority; five findings were recorded | -| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | The reasoning names the suppression no recorded decision covers; the rule is otherwise permanent and blocking. [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | -| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | The reasoning names the documentation on a wiki, not versioned with the rule; the message is terse and carries an identifier | +| Clause | Accepted grades | Condition on the reasoning | What the fixture supports | +| ----------------------------------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | None | Class, hazard and the next wider rule are named; Partial or No is the reading that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | +| [3.2](../../SPEC.md#32-build-the-net) | Yes | None | A rule in the project's static analyser, reading code, not a test | +| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | None | Commit B is red on the fixture and the originating instance and survives as its own commit | +| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Mentions the instance left unfixed, the inline ignore, or `Report::render()` | Five findings were recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority | +| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | Mentions the suppression, the inline ignore, or that no recorded decision covers it | The rule is permanent and blocking, but the suppression was not a recorded decision. [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | +| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | Mentions the wiki, or that the documentation is not versioned or co-located with the rule | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule | Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**; a reader is correct who says so and names [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) or [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the clauses not followed. The **defence does not conform** either; a reader is correct who says so and names [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the reasons, or [3.5](../../SPEC.md#35-enforce-permanently-and-block)'s unrecorded suppression, which -section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence also requires to be absent. +section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence also requires to be absent. Other clauses named +alongside are not marked. Section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): the **inline ignore on `Report::render()`** was the owner's decision, not the -practitioner's. A reader who names it is correct, with or without the decision to leave the -wider rule unbuilt, which section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) also gives to the owner. A reader who names only the wider -rule, or neither, is wrong. +practitioner's. A reader who names it is correct; any other decision named alongside it is not +marked. A reader who names only the decision to leave the wider rule unbuilt, or neither, is +wrong. Wrong readings the text invites, which count against the document on one reader: -- [3.5](../../SPEC.md#35-enforce-permanently-and-block) graded Yes because the rule fails the build, which has not applied the clause's opening - sentence on unrecorded suppression. -- [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) graded Yes because the URL resolves, which has read the versioning sentence of [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) - differently. -- The defence graded as conforming because the sweep is what failed, where the reasoning quotes - section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence. +- [3.5](../../SPEC.md#35-enforce-permanently-and-block) graded Yes with reasoning that says the rule blocks, or is permanent, and does not mention + the suppression. +- [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) graded Yes with reasoning that says the identifier resolves, or the URL works, and does not + mention versioning or co-location. +- The defence graded as conforming with reasoning that says only the sweep failed and that + quotes section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence. A reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the fixture, which is the reader's miss. A reader who attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index c11ae32..af28752 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -45,26 +45,26 @@ Wrong readings the text invites, which count against the document on one reader: grade, not a grade of its own. Wrong if it gives a grade per part. Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "The project-level verdict is a single grade; which part of the Toolchain satisfied each clause is evidence for that grade, not a second grade." -- **2c.** Required: (1) two, graded and declared separately; (2) neither implies the other. - Wrong if it gives one grade, or says the artefact grade follows from the project grade or the - reverse. Key quotation, [9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately): "The two verdicts MUST be graded and declared separately, and +- **2c.** Required: (1) two; (2) neither implies the other. Wrong if it gives one grade, or says + the artefact grade follows from the project grade or the reverse. That the two are graded and + declared separately is explanation; an answer that says it has stated element (1). Key quotation, [9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately): "The two verdicts MUST be graded and declared separately, and neither implies the other." ## 3. Application: the Zigpipe project -| Clause | Accepted grades | Condition on the reasoning, and what the fixture supports | -| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | Lintomatic fails detector [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) on two undocumented bundled rules and nothing in the wrapping closes that gap; a reader who notes that the detector specification was not supplied and grades from this document's summary of it is correct | -| [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)`. The two are blocking, since their firing stops `zig-qa run`; a reader who grades No on that condition is correct | -| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | Ignore comments disabled project-wide; exceptions only through the record the toolchain reads | -| [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | `zig-qa run` is local, prints Lintomatic's output unchanged, one line per finding | -| [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | Lintomatic, then the tests, stopping on a Lintomatic failure | -| [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes, Partial or No | No condition. `zig-qa list` prints identifier, summary and documentation path without running the rules; Partial or No is the reading that counts the two `(none)` routes, since 5.1 says the listing MUST provide the route and [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) point 2 says a gap may be named again by a later clause | -| [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | Derived from `lintomatic.toml`, not hand-maintained | -| [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | Yes | The `proj.*` rules appear in the same listing | -| [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes or Partial | `quality-exceptions.toml` is a defined path that `zig-qa run` loads. Partial is accepted only where the reasoning says the fixture names a place for exceptions and none for calibrations or conventions, which the project record entry also covers | -| [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Partial or No | The reasoning mentions that nothing rejects a justification that could be pasted onto any entry; a justification is required and an empty one rejected | -| [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | `zig-qa list --exceptions` uses the same format and operation as the rule listing | +| Clause | Accepted grades | Condition on the reasoning | What the fixture supports | +| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | None | Lintomatic fails detector [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) on two undocumented bundled rules and nothing in the wrapping closes that gap; a reader who notes that the detector specification was not supplied and grades from this document's summary of it is correct | +| [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | None. A No given on the condition that the two bundled rules are blocking is a No: their firing stops `zig-qa run`, so they are | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)` | +| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | None | Ignore comments disabled project-wide; exceptions only through the record the toolchain reads | +| [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | None | `zig-qa run` is local, prints Lintomatic's output unchanged, one line per finding | +| [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | None | Lintomatic, then the tests, stopping on a Lintomatic failure | +| [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes, Partial or No | None | `zig-qa list` prints identifier, summary and documentation path without running the rules; Partial or No is the reading that counts the two `(none)` routes, since [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) says the listing MUST provide the route and [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) point 2 says a gap may be named again by a later clause | +| [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | None | Derived from `lintomatic.toml`, not hand-maintained | +| [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | Yes | None | The `proj.*` rules appear in the same listing | +| [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes or Partial | For Partial: mentions that the fixture names a place for exceptions and none for calibrations or conventions | `quality-exceptions.toml` is a defined path that `zig-qa run` loads; the project record entry also covers calibrations and conventions | +| [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Partial or No | Mentions that nothing rejects a generic justification, one that could be pasted onto any entry | A justification is required and an empty one rejected | +| [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | None | `zig-qa list --exceptions` uses the same format and operation as the rule listing | Section [9](../../TOOLING-SPEC.md#9-conformance) verdict: **does not conform**; a reader is correct who says so and names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) or [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) among the failing clauses. **Not with agent support**; a reader is correct who says so because @@ -76,14 +76,15 @@ optional." Wrong readings the text invites, which count against the document on one reader: -- [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) graded Yes with reasoning that quotes only the first sentence of the clause, which has - not read the sentence on a generic justification. -- [8.1](../../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) applied to Zigpipe on the basis that its tooling includes a shipped package, which has - read "where the Toolchain is one the project ships" (section [9](../../TOOLING-SPEC.md#9-conformance)) as reaching the consuming - project. -- [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) graded Yes on the grounds that Lintomatic "mostly" conforms, where the reasoning quotes - [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), which has read it as admitting a proportion. -- Zigpipe held ungradeable for want of a declaration, which has read [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) as a condition. +- [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) graded Yes with reasoning that says a justification is required or an empty one rejected + and does not mention a generic justification, which has not read the sentence on one. +- Section [8](../../TOOLING-SPEC.md#8-self-audit) said to apply to Zigpipe with reasoning that says its tooling includes a shipped + package, which has read "where the Toolchain is one the project ships" (section [9](../../TOOLING-SPEC.md#9-conformance)) as + reaching the consuming project. +- [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) graded Yes with reasoning that says Lintomatic mostly or largely conforms and quotes [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), + which has read it as admitting a proportion. +- The declaration said to affect the verdict, with reasoning that Zigpipe cannot be graded without + one, which has read [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) as a condition. A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes because they missed the two undocumented rules has misread the fixture, which is the reader's miss. diff --git a/acceptance/toolchain/QUESTIONS.md b/acceptance/toolchain/QUESTIONS.md index 35a2b91..86de411 100644 --- a/acceptance/toolchain/QUESTIONS.md +++ b/acceptance/toolchain/QUESTIONS.md @@ -26,7 +26,7 @@ the document you are relying on. Grade the Zigpipe project's toolchain, as described in the fixture, against every MUST clause in sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record): [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record), [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure), [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration), [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones), [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences). For each give Yes, Partial or No and one line of reasoning. Then state whether the toolchain conforms under section -9, whether it conforms with agent support, and whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies to it. Finally, state +[9](../../TOOLING-SPEC.md#9-conformance), whether it conforms with agent support, and whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies to it. Finally, state whether the absence of a declaration affects the verdict. ## 4. Confusion From 8a63448820785166faeef81263fc1143b3028710 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 11:52:20 +0100 Subject: [PATCH 066/105] Acceptance process: review 2 findings applied Question 1 defined once (first six bullets, same clause counts once, a bullet on neither list not counted); the verdict items of question 3 ask for their reason and the keys state each as an accepted answer with a condition; a finding lies in the change's text also where the readers set it against that text, read against the merge base; the cohort's model is the version the editor names; a keyed miss whose fix would change an obligation is routed and waited for; the question set is the one list of load-bearing questions; ROUTED.md defines its status column and links its clauses. Co-Authored-By: Claude Fable 5.1 --- ACCEPTANCE.md | 102 +++++++++++++++--------------- CONTRIBUTING.md | 6 +- acceptance/CHANGES.md | 11 ++-- acceptance/README.md | 9 +-- acceptance/ROUTED.md | 33 +++++----- acceptance/detector/KEY.md | 11 ++-- acceptance/detector/QUESTIONS.md | 2 +- acceptance/method/KEY.md | 30 ++++----- acceptance/method/QUESTIONS.md | 4 +- acceptance/toolchain/KEY.md | 22 ++++--- acceptance/toolchain/QUESTIONS.md | 4 +- 11 files changed, 123 insertions(+), 111 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index c5bd8cd..39110a7 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -29,7 +29,8 @@ serves that, and a rule that stops it proving that is a defect in this file, fix ``` A reader is set aside, and its answers do not count, when its answer text is empty, when its - process reports an error, or when the model version it reports is not the cohort's model. A run + process reports an error, or when the model version it reports is not the cohort's model, the + one version the editor names for the run before its readers start. A run with a reader set aside is void: it takes a run number and has a record, and a fresh cohort runs. No reader is set aside for any other cause. @@ -54,24 +55,17 @@ asks for nothing the question does not. Each reader answers all of them. bullets. Keyed against the key's list of the document's obligations: a bullet is correct when it names one of them. -2. **Load-bearing statements.** Three or more questions, fixed per document, each asking for a - short answer and the sentence or sentences of the document it rests on, quoted. Each has one - or more parts, and the key lists the element an answer must state for each part: - - - `SPEC.md`: what a practitioner does before fixing a defect, in order; who may baseline or - suppress, and whether an agent may; what "conforms" means for a remediation and what a - verdict rests on. - - `DETECTOR-SPEC.md`: what "conforms" means and whether a tool that predates the document can - conform; what a detector must do about an inline suppression route and whether one of those - things is enough; where a bundled rule's documentation must live, at what version, and - whether it must be reachable offline. - - `TOOLING-SPEC.md`: where conformance is measured and what the toolchain is; how a grade is - assigned when parts come from different sources; how many grades a project that ships - tooling has and whether one implies the other. +2. **Load-bearing statements.** Three or more questions, fixed per document and listed in its + question set under `acceptance/`, each asking for a short answer and the sentence or + sentences of the document it rests on, quoted. Each has one or more parts, and the key lists + the element an answer must state for each part. An obligation is load-bearing, and gets a + question, when a reader who misreads it would act against the method; the editor decides, and + the pull request that adds the question says why. 3. **Application.** A fixture case supplied with the questions, graded clause by clause with - Yes, Partial or No and one line of reasoning. Keyed against the intended verdicts, with the - readings the key names as ones the text invites counting against the document. + Yes, Partial or No and one line of reasoning, then the verdicts the question set asks for, + each with its reason. Keyed against the accepted grades and answers, with the readings the + key names as ones the text invites counting against the document. 4. **Confusion.** Any passage the reader found confusing on its one reading, quoted with its clause number, and nothing else: not what would make it clearer, not a proposed change. @@ -98,9 +92,10 @@ unless the editor shows that a rule here was misapplied, and a re-mark says whic Every mark is given by a rule in this file or in the key, as follows. Where neither settles a mark, the marker records the point in the run record and marks it as the open-point rule below -says, and the editor settles it in this file or in the key, as [Changing the test](#changing-the-test) -allows, before the next run. A run record states no judgement that is not written here or in -the key; a judgement an earlier run recorded is not a rule. +says, and the editor settles it in this file or in the key, as +[Changing the test](#changing-the-test) allows, before the next run. A run record states no +judgement that is not written here or in the key; a judgement an earlier run recorded is not a +rule. - **Keyed items.** Question 1 as a whole is one keyed item, and so is each item of question 2 and each answer of question 3 for which the key gives an accepted grade or answer. @@ -111,20 +106,21 @@ the key; a judgement an earlier run recorded is not a rule. or apostrophe style, and markdown link or emphasis markup; a quotation that drops or changes a word quotes the document by its longest such run. - **Question 1** is correct when at least four bullets each name an obligation the key lists and - no bullet asserts one the key marks wrong. Several bullets that name different clauses of one - line of the key's list each count. A bullet asserting an obligation that is in neither list is - recorded as an open point for the key and is not counted either way. + no bullet asserts one the key marks wrong. Only the first six bullets are marked. Several + bullets that name different clauses of one line of the key's list each count; two that name + the same clause count once. A bullet asserting an obligation that is in neither list is not + counted either way, and is recorded as an open point for the key. - **Question 2.** An answer is the reader's sentences and its quotation read together. It is correct when it states every element the key lists as required for that item, in the document's words or the reader's own, asserts nothing the key marks wrong, and quotes the document. An element stated only in the quotation is stated. Anything in the key beyond the required elements and the wrong assertions is explanation for the marker, and an answer is never marked down for leaving it out. -- **Question 3.** Each item is correct when its grade is one the key accepts and, where the key - sets a condition on the reasoning for that grade, the reasoning meets it. A grade the key - accepts with no condition is correct whatever the reasoning. A grade given on a stated - condition is read as the grade for the case the fixture presents, where the key says which - case that is; otherwise the item is unanswered. +- **Question 3.** Each item is correct when its grade or answer is one the key accepts and, + where the key sets a condition on the reasoning for it, the reasoning meets it. A grade or + answer the key accepts with no condition is correct whatever the reasoning. A grade given on a + stated condition is read as the grade for the case the fixture presents, where the key says + which case that is; otherwise the item is unanswered. - **A miss that counts against the document.** The key names, for question 1 and for each item of question 3, the wrong readings the text invites, each as a grade or answer together with what the reasoning states or leaves out. A miss that matches one blocks whatever the count, @@ -134,22 +130,26 @@ the key; a judgement an earlier run recorded is not a rule. below; a second miss on the same item fails the threshold, and no judgement is needed. - **An unanswered question.** Each keyed item left unanswered is a miss. Question 4 left unanswered counts as no passage named. -- **Question 4.** A reader names a passage when it quotes words of the document or gives a clause - number with a description that identifies a sentence or paragraph. A remark that is not about - a passage of the document, such as that another document was not supplied, is recorded as a - note and is not a passage. Two readers name the same passage when the words or sentences they - name lie in the same paragraph; a heading, a list item, a table row and a terminology entry are - each a paragraph of their own. A passage named by two or more readers of the run is a finding. +- **Question 4.** A reader names a passage when it quotes the document, as defined above, or + gives a clause number with a description or a shorter quotation that identifies a sentence or + paragraph. A quotation whose words occur in more than one paragraph, given with no clause + number, names the paragraph where they first occur. A remark that is not about a passage of + the document, such as that another document was not supplied, is recorded as a note and is + not a passage. Two readers name the same passage when the words or sentences they name lie in + the same paragraph; a heading, a list item, a table row and a terminology entry are each a + paragraph of their own. A passage named by two or more readers of the run is a finding. - **Where a finding lies.** The record says, for each finding, whether it lies in text the change - wrote or altered: a paragraph in which the pull request, read against the branch it targets, - adds, rewords, moves or deletes a sentence, or changes a word, a term or a link. A finding + wrote or altered: a paragraph in which the pull request, read against its merge base with the + branch it targets, adds, rewords, moves or deletes a sentence, or changes a word, a term or a + link; or a paragraph the readers naming the finding set against such a paragraph. A finding there blocks the change. A finding anywhere else is a finding against the document, not against the change: it is recorded in [acceptance/ROUTED.md](acceptance/ROUTED.md), routed to a change of its own, and does not block. The change is judged on what it changed; the document is proved by the keyed questions, which every run answers over the whole of it. -- **Open points.** An open point in marking an answer is marked for the reader. An open point on - whether a reader named a passage, or whether two readers named the same one, counts the - passage. An open point on where a finding lies treats it as the change's own text. +- **Open points.** An open point in marking an answer is marked for the reader, except that a + question 1 bullet on neither list is not counted, as above. An open point on whether a reader + named a passage, or whether two readers named the same one, counts the passage. An open point + on where a finding lies treats it as the change's own text. ## Pass criteria @@ -164,7 +164,7 @@ is four of five. | A miss the key says counts against the document | **None**, whatever the count | | A passage found confusing by two or more readers, in text the change wrote or altered | **None.** It is a finding and blocks until the text is changed | | A passage found confusing by two or more readers, elsewhere | Recorded in `acceptance/ROUTED.md` and routed; does not block | -| Anything from one reader only | Recorded; applied at the editor's discretion; does not block | +| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | The asymmetry is deliberate. A load-bearing statement is the sentence the whole document exists to make, and one reader in five getting it wrong means one agent in five will act on the wrong @@ -202,11 +202,12 @@ changed what the cohort is asked, not how strictly an answer is marked. ## After a failure Fix the text the run's misses and findings name and run a fresh cohort; readers are never reused. -A keyed miss is fixed in the text that carries the obligation, wherever it lies, since the keyed -questions prove the document and a document one reader in five misreads is not merged to, whoever -wrote the sentence. A finding on text the change wrote or altered is fixed there. A finding -elsewhere goes to `acceptance/ROUTED.md` and is not fixed in this change. The number of runs it -took is recorded; a document that needed three runs is telling its editor something. +A failed keyed item is fixed in the text that carries the obligation, wherever it lies, since the +keyed questions prove the document as a whole, whoever wrote the sentence; where that fix would +change an obligation, it is routed to a change of its own, and this change waits for it. A +finding on text the change wrote or altered is fixed there. A finding elsewhere goes to +`acceptance/ROUTED.md` and is not fixed in this change. The number of runs it took is recorded; +a document that needed three runs is telling its editor something. ## When it runs @@ -239,11 +240,12 @@ and the instruction a delegated marker was given are posted in one or more comme request, linked from the record and never edited, hidden or deleted after posting. The changelog line for the change names every run made on it, the passing run last. CI refuses a -specification change whose pull request adds no run file; it checks for at least one, and a pull request that changes two -documents adds a run for each. Each published version's changelog entry carries the lines of the -changes it contains, so a version's cohort record is the set of runs those lines name. A version -with no such line has not been accepted and MUST NOT be described as published. The question sets -and answer keys live under `acceptance/` in this repository so that a run can be repeated and +specification change whose pull request adds no run file; it checks for at least one, and a pull +request that changes two documents adds a run for each. Each published version's changelog entry +carries the lines of the changes it contains, so a version's cohort record is the set of runs +those lines name. A version with no such line has not been accepted and MUST NOT be described as +published. The question sets and answer keys live under `acceptance/` in this repository so that +a run can be repeated and compared. `acceptance/ROUTED.md` lists every finding routed from a run, open until a change that fixes diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 013bb9d..733e05b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,9 +29,9 @@ review. It is a patch version when released. SHOULD or MAY in a numbered section or appendix. It needs everything a clarity edit needs, plus the five tests in SCOPE.md answered in the pull request, and it is at least a minor version when released. `tools/changes.py` tells the two apart mechanically and CI runs it on every pull -request. A change to the process files ACCEPTANCE.md names, this file among them, is neither: -ACCEPTANCE.md says how it is made, and it is recorded in `acceptance/CHANGES.md`, not in -CHANGELOG.md. +request. A change to the acceptance process, as ACCEPTANCE.md defines one under "Changes to +this process", is neither: ACCEPTANCE.md says how it is made, and it is recorded in +`acceptance/CHANGES.md`, not in CHANGELOG.md. ## What a pull request contains diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index cc01848..29a20db 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -1,13 +1,14 @@ # Changes to the acceptance process -Changes to the process files that [ACCEPTANCE.md](../ACCEPTANCE.md) names, recorded here as it -describes: a line is added at the top for each change, naming its pull request. The process is -not a specification and is not versioned, and a change to it is not tested by a cohort. Changes -to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). +Changes to the acceptance process, as [ACCEPTANCE.md](../ACCEPTANCE.md) defines one under +"Changes to this process", recorded here as it describes: a line is added at the top for each +change, naming its pull request. The process is not a specification and is not versioned, and a +change to it is not tested by a cohort. Changes to the three specifications are recorded in +[CHANGELOG.md](../CHANGELOG.md). - Pull request NNN: the test is made to prove what it exists to prove, the clarity of the document to a cold reader, after thirty-three runs on two changes (19 to 30, 31 to 33 and 35 - to 52) in which keyed answers reached five of five and the runs still failed. Four defects in + to 52) in which keyed answers reached five of five and the runs still failed. Five defects in the process were found. Keys required elements the question never asked for (detector 2b's "need not remove or forbid", detector 2c's "at a version tracked together", and on pull request 14 method 2d's order of the kinds), so readers who answered the question as put were diff --git a/acceptance/README.md b/acceptance/README.md index dcf39c0..aaeba1f 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -25,8 +25,8 @@ in [CHANGES.md](CHANGES.md). 4. The result is recorded as a file under [runs/](runs/), stating what `ACCEPTANCE.md` lists, with the answer sheets posted on the pull request and linked from it, and the change's `Unreleased` changelog line names that file. A run that fails is followed by a change to the - text the findings name and a fresh cohort; a finding on text the change did not write or - alter is added to [ROUTED.md](ROUTED.md) instead. + text its misses and findings name and a fresh cohort; a finding on text the change did not + write or alter is added to [ROUTED.md](ROUTED.md) instead. ## How a key is written @@ -50,8 +50,9 @@ A key quotes the document. When a clause is reworded, the key line that quotes i the same commit, and the change is one the acceptance run itself then tests. A key that no longer matches the document is a defect in this directory, not a reason to mark a reader wrong. -A key, a question set or a fixture changes in one of two ways, as `ACCEPTANCE.md` says under -"Changing the test", and always before the readers of the next run start: +A key, a question set or a fixture changes by one of two routes, as `ACCEPTANCE.md` says under +"Changing the test", always before the readers of the next run start, and an open point a run +recorded is settled by whichever route it needs: - **In a specification pull request, under step 4 of `CONTRIBUTING.md`**: a quotation follows the text it quotes, or a keyed item is added for a new load-bearing obligation together with diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index eb35815..60fc616 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -8,25 +8,26 @@ editor reads this list before cutting a release. A finding is open until a change that reworks its passage passes a cohort, when the line names that run and moves below the rule, or until the editor closes it with a reason recorded here. A -finding raised again by a later run is noted on its line, not listed twice. The rows below were -raised by runs 31 to 52 under the earlier rules, on untouched text of the change those runs -tested, and are listed here so that they are not lost. +finding raised again by a later run is noted on its line, not listed twice. The status column +holds `open`, or the `editor/` branch or pull request number of the change that is +reworking the passage. The rows below were raised by runs 31 to 52 under the earlier rules, on +untouched text of the change those runs tested, and are listed here so that they are not lost. ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | -------------------------------------- | -| `TOOLING-SPEC.md` | 9 and 9.1 | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | 2, 4.2, 8.1 | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | 9.2 | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | 4.3 | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | run 33 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | 4.1 | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | 5.3 | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | open | -| `DETECTOR-SPEC.md` | 7.1 | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | 8.1 | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | 5.3 | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | -| `SPEC.md` | 1 (Owner) | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50 | open | -| `SPEC.md` | 4 | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | open; a method question for the editor | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | ---------------------------------------- | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | run 33 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | ## Closed diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 2f6fd96..aaddb69 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -8,7 +8,7 @@ update the quotation here in the same commit. `ACCEPTANCE.md` says how each item ## 1. Restatement A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting an obligation the document does not state, is a correct restatement. +bullet asserting one the list of wrong assertions below marks wrong, is a correct restatement. - Support bespoke rules written by the project ([4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it)). - Provide a harness that runs a single rule against supplied code ([4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code)). @@ -78,11 +78,12 @@ No grade above carries a condition on the reasoning: the accepted grade is corre reasoning, and a reader who adds a caveat about what the fixture does not say and still gives the accepted grade is correct. -Section [8](../../DETECTOR-SPEC.md#8-conformance) verdict: **does not conform**; a reader is correct who says so and names [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), the -two undocumented bundled rules, or partial conformance not counting, as the reason. +The two answers the question then asks for, each with the condition its reason must meet: -Clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation): **does not change the verdict**; a reader is correct who says so and gives as the -reason that it is a SHOULD, or that the MUST which fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). +- **Conformance under section [8](../../DETECTOR-SPEC.md#8-conformance).** Accepted answer: does not conform. Condition: the reason + names [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), the two undocumented bundled rules, or partial conformance not counting. +- **Whether clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation) changes the verdict.** Accepted answer: it does not. Condition: the reason + says that it is a SHOULD, or that the MUST which fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). Wrong readings the text invites, which count against the document on one reader: diff --git a/acceptance/detector/QUESTIONS.md b/acceptance/detector/QUESTIONS.md index 2666b2f..5c288f8 100644 --- a/acceptance/detector/QUESTIONS.md +++ b/acceptance/detector/QUESTIONS.md @@ -25,7 +25,7 @@ the document you are relying on. Grade Lintomatic 4.2, as described in the fixture, against every MUST clause in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression): [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it), [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), [4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding), [5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure), [5.2](../../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file), [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran), [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally), [6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation), [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access), [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable). For each give Yes, Partial or No and one line of reasoning. Then state whether Lintomatic conforms under section [8](../../DETECTOR-SPEC.md#8-conformance), and why. Finally, -state whether clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation) changes the verdict. +state whether clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation) changes the verdict, and why. ## 4. Confusion diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 36bfafd..b756f47 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -8,7 +8,7 @@ update the quotation here in the same commit. `ACCEPTANCE.md` says how each item ## 1. Restatement A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting an obligation the document does not state, is a correct restatement. +bullet asserting one the list of wrong assertions below marks wrong, is a correct restatement. - Attribute the defect to a class, bounded on both sides, with the hazard stated ([3.1](../../SPEC.md#31-attribute-the-defect-to-a-class)). - Express the class as a rule in a detector that reads code, never a test ([3.2](../../SPEC.md#32-build-the-net)). @@ -43,7 +43,7 @@ Wrong readings the text invites, which count against the document on one reader: quotation, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): "An Agent MUST NOT Baseline, suppress or knowingly leave an Instance unfixed on its own authority, whatever the Instance count turns out to be." "The owner" names the defined term and states that the decision is a human's; an answer need not add "a human" - to be correct. The default owner paragraph in the terminology entry for Owner still requires a + to be correct. The default-owner sentences of the terminology entry for Owner still require a recorded decision, so an answer naming whoever instructed the practitioner, for a project that has recorded nobody, names the owner and is correct. - **2c.** Required: (1) all six clauses of section [3](../../SPEC.md#3-the-method) were followed for that defect; (2) the verdict @@ -61,21 +61,22 @@ Wrong readings the text invites, which count against the document on one reader: | ----------------------------------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | None | Class, hazard and the next wider rule are named; Partial or No is the reading that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | | [3.2](../../SPEC.md#32-build-the-net) | Yes | None | A rule in the project's static analyser, reading code, not a test | -| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | None | Commit B is red on the fixture and the originating instance and survives as its own commit | +| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes, Partial or No | For Partial or No: attributes the inline ignore to this clause, a fair reading of Part B | Commit B is red on the fixture and the originating instance and survives as its own commit | | [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Mentions the instance left unfixed, the inline ignore, or `Report::render()` | Five findings were recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority | | [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | Mentions the suppression, the inline ignore, or that no recorded decision covers it | The rule is permanent and blocking, but the suppression was not a recorded decision. [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | | [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | Mentions the wiki, or that the documentation is not versioned or co-located with the rule | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule | -Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**; a reader is correct who says so and names -[3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) or [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the clauses not followed. The **defence does not conform** either; a reader is -correct who says so and names [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the reasons, or [3.5](../../SPEC.md#35-enforce-permanently-and-block)'s unrecorded suppression, which -section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence also requires to be absent. Other clauses named -alongside are not marked. +The three answers the question then asks for, each with the condition its reason must meet: -Section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): the **inline ignore on `Report::render()`** was the owner's decision, not the -practitioner's. A reader who names it is correct; any other decision named alongside it is not -marked. A reader who names only the decision to leave the wider rule unbuilt, or neither, is -wrong. +- **The remediation under section [7](../../SPEC.md#7-conformance).** Accepted answer: does not conform. Condition: the reason + names [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) or [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the clauses not followed; other clauses named alongside are not marked. +- **The defence alone.** Accepted answer: does not conform. Condition: the reason names [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation), or + [3.5](../../SPEC.md#35-enforce-permanently-and-block)'s unrecorded suppression, which section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence also + requires to be absent; other clauses named alongside are not marked. +- **The decision section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) says was not the practitioner's.** Accepted answer: the inline ignore on + `Report::render()`, which was the owner's decision. Condition: none; any other decision named + alongside it is not marked. An answer naming only the decision to leave the wider rule unbuilt, + or naming none, is wrong. Wrong readings the text invites, which count against the document on one reader: @@ -86,9 +87,8 @@ Wrong readings the text invites, which count against the document on one reader: - The defence graded as conforming with reasoning that says only the sweep failed and that quotes section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence. -A reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the fixture, which is the reader's miss. A reader who -attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as -covering it, which is a fair reading and is not marked wrong. +A reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the fixture, which is the reader's miss, whether or not +the inline ignore was attributed to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) instead. ## 4. Confusion diff --git a/acceptance/method/QUESTIONS.md b/acceptance/method/QUESTIONS.md index 34fa82d..9a2f393 100644 --- a/acceptance/method/QUESTIONS.md +++ b/acceptance/method/QUESTIONS.md @@ -24,8 +24,8 @@ the document you are relying on. Grade the ledger remediation, as described in the fixture, against each of the six clauses of section [3](../../SPEC.md#3-the-method): [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class), [3.2](../../SPEC.md#32-build-the-net), [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire), [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), [3.5](../../SPEC.md#35-enforce-permanently-and-block), [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation). For each give Yes, Partial or No and one line of -reasoning. Then state whether the remediation conforms under section [7](../../SPEC.md#7-conformance), whether the defence -alone conforms, and which decision in the fixture, if any, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) says was not the +reasoning. Then state whether the remediation conforms under section [7](../../SPEC.md#7-conformance) and why, whether the defence +alone conforms and why, and which decision in the fixture, if any, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) says was not the practitioner's to take. ## 4. Confusion diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index af28752..7353852 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -8,7 +8,7 @@ update the quotation here in the same commit. `ACCEPTANCE.md` says how each item ## 1. Restatement A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting an obligation the document does not state, is a correct restatement. +bullet asserting one the list of wrong assertions below marks wrong, is a correct restatement. - Route defences only through detectors that conform to the detector specification, wrapping permitted ([4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification)). @@ -66,13 +66,19 @@ Wrong readings the text invites, which count against the document on one reader: | [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Partial or No | Mentions that nothing rejects a generic justification, one that could be pasted onto any entry | A justification is required and an empty one rejected | | [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | None | `zig-qa list --exceptions` uses the same format and operation as the rule listing | -Section [9](../../TOOLING-SPEC.md#9-conformance) verdict: **does not conform**; a reader is correct who says so and names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) or -[4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) among the failing clauses. **Not with agent support**; a reader is correct who says so because -section [7](../../TOOLING-SPEC.md#7-agent-context) is not met, there being no summary. **Section [8](../../TOOLING-SPEC.md#8-self-audit) does not apply**; a reader is correct -who says so because Zigpipe ships nothing to other projects, or because section [8](../../TOOLING-SPEC.md#8-self-audit) bears on -zig-qa's own artefact grade and not on Zigpipe. **The absence of a declaration does not affect -the verdict**; a reader is correct who says so. Key quotation, [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance): "The declaration is -optional." +The four answers the question then asks for, each with the condition its reason must meet: + +- **Conformance under section [9](../../TOOLING-SPEC.md#9-conformance).** Accepted answer: does not conform. Condition: the reason + names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) or [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) among the failing clauses; other clauses named alongside are not marked. +- **Conformance with agent support.** Accepted answer: no. Condition: the reason says that + section [7](../../TOOLING-SPEC.md#7-agent-context) is not met, or that there is no summary. +- **Whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies.** Accepted answer: it does not. Condition: the reason says that + Zigpipe ships nothing to other projects, or that section [8](../../TOOLING-SPEC.md#8-self-audit) bears on zig-qa's own artefact + grade and not on Zigpipe. +- **Whether the absence of a declaration affects the verdict.** Accepted answer: it does not. + Condition: the reason says that the declaration is optional, or not a condition of + conformance, or that a verdict rests on the evidence. Key quotation, [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance): "The declaration is + optional." Wrong readings the text invites, which count against the document on one reader: diff --git a/acceptance/toolchain/QUESTIONS.md b/acceptance/toolchain/QUESTIONS.md index 86de411..d4e5cd7 100644 --- a/acceptance/toolchain/QUESTIONS.md +++ b/acceptance/toolchain/QUESTIONS.md @@ -26,8 +26,8 @@ the document you are relying on. Grade the Zigpipe project's toolchain, as described in the fixture, against every MUST clause in sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record): [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record), [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure), [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration), [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones), [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences). For each give Yes, Partial or No and one line of reasoning. Then state whether the toolchain conforms under section -[9](../../TOOLING-SPEC.md#9-conformance), whether it conforms with agent support, and whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies to it. Finally, state -whether the absence of a declaration affects the verdict. +[9](../../TOOLING-SPEC.md#9-conformance) and why, whether it conforms with agent support and why, and whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies to it +and why. Finally, state whether the absence of a declaration affects the verdict, and why. ## 4. Confusion From 4cc568884c13ed12c7b31a56f41459e26519de30 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 11:58:31 +0100 Subject: [PATCH 067/105] Acceptance process: review 3 findings applied Two routes for changing the test, with a changed or removed obligation's key item following the text in the specification pull request; an open point holds up only its own pull request; a fresh run follows a change to the document, not to the record; detector 2b names both things; the section 7 and section 9 reasons accept every failing clause; the restatement rule reads plainly; ROUTED.md defines a finding as ACCEPTANCE.md does. Co-Authored-By: Claude Fable 5.1 --- ACCEPTANCE.md | 30 ++++++++++++++++-------------- CONTRIBUTING.md | 4 ++-- acceptance/README.md | 20 +++++++++++++------- acceptance/ROUTED.md | 6 +++--- acceptance/detector/KEY.md | 13 +++++++------ acceptance/method/KEY.md | 7 ++++--- acceptance/toolchain/KEY.md | 10 ++++++---- 7 files changed, 51 insertions(+), 39 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 39110a7..d1bd6af 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -141,8 +141,8 @@ rule. - **Where a finding lies.** The record says, for each finding, whether it lies in text the change wrote or altered: a paragraph in which the pull request, read against its merge base with the branch it targets, adds, rewords, moves or deletes a sentence, or changes a word, a term or a - link; or a paragraph the readers naming the finding set against such a paragraph. A finding - there blocks the change. A finding anywhere else is a finding against the document, not + link; or a paragraph that any reader naming the finding sets against such a paragraph. A + finding there blocks the change. A finding anywhere else is a finding against the document, not against the change: it is recorded in [acceptance/ROUTED.md](acceptance/ROUTED.md), routed to a change of its own, and does not block. The change is judged on what it changed; the document is proved by the keyed questions, which every run answers over the whole of it. @@ -178,15 +178,18 @@ fresh cohort names by fewer than two is clear. Nothing in this test is changed to let a particular run through. A threshold, a question, a key, a fixture or a rule here is changed only before a run's readers start, never changes a mark -already given, and is changed in one of three ways, each recorded with its reason: +already given, and is changed by one of two routes, each recorded with its reason: -- A key's quotation follows the text it quotes, and a key gains an item for a new load-bearing - obligation, with the question that asks for it, under step 4 of `CONTRIBUTING.md`, in the - specification pull request that makes the change; `acceptance/README.md` says what that may - and may not do. -- A threshold, a fixture, a rule here, what an existing key item requires, or the wording of a - question changes by a change to this process, with its own pull request. -- An open point a run recorded is settled by either route, whichever it needs. +- In the specification pull request that changes the text, under step 4 of `CONTRIBUTING.md`: + a key's quotation follows the text it quotes; a key gains an item, with the question that asks + for it, for a new load-bearing obligation; and where the change alters or removes the + obligation an existing item tests, the item and its question follow the text, since that is + the text changing and not the test. `acceptance/README.md` says what that may and may not do. +- By a change to this process, with its own pull request: a threshold, a fixture, a rule here, + what an existing key item requires of an unchanged obligation, or the wording of a question. + +An open point a run recorded is settled by whichever route it needs, before the next run of the +pull request whose run recorded it; a run of another pull request is not held up by it. A question is reworded only so that it asks for what its key requires, and a key's required elements are never more than its question asks for. A run made after such a change is a fresh @@ -214,9 +217,9 @@ a document that needed three runs is telling its editor something. On the final text of each pull request, after the editor's rewrite and any key update, one cohort per changed document. It is the last gate before the merge to `next`, so that every change on `next` is one the cohort has accepted, and the release branch needs no cohort of its own unless it -changes specification text. A fresh cohort runs after every change to the pull request's own +changes specification text. A fresh cohort runs after every change to the changed document's text that follows a run, and after every void run; a text that failed is never run again -unchanged. A version number or date that `PUBLISHING.md` or step 5 of `CONTRIBUTING.md` has the +unchanged, and a commit that adds the run record or the changelog line changes no document. A version number or date that `PUBLISHING.md` or step 5 of `CONTRIBUTING.md` has the editor set is not such a change, and neither is a merge of `next` into the branch that brings in text another cohort accepted and alters no sentence this change wrote. @@ -245,8 +248,7 @@ request that changes two documents adds a run for each. Each published version's carries the lines of the changes it contains, so a version's cohort record is the set of runs those lines name. A version with no such line has not been accepted and MUST NOT be described as published. The question sets and answer keys live under `acceptance/` in this repository so that -a run can be repeated and -compared. +a run can be repeated and compared. `acceptance/ROUTED.md` lists every finding routed from a run, open until a change that fixes its passage passes a cohort, or until the editor closes it with a recorded reason. The editor diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 733e05b..ea5a525 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -69,8 +69,8 @@ survives, the change is yours. changed document. Its record is committed under `acceptance/runs/`, the `Unreleased` changelog line names it, and CI refuses a specification change that lacks one. For an obligation, the editor sets the document's header to `-dev, unpublished` in the - same commit if it still names a published version. A change to the pull request's own text - after the run means a fresh run, as ACCEPTANCE.md says under "When it runs". A finding the + same commit if it still names a published version. A change to the document's text after the + run means a fresh run, as ACCEPTANCE.md says under "When it runs". A finding the run routes to `acceptance/ROUTED.md` is not fixed in this pull request. 6. **Merge** into `next` by merge commit, and the branch is deleted. diff --git a/acceptance/README.md b/acceptance/README.md index aaeba1f..ec5c3b2 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -32,12 +32,15 @@ in [CHANGES.md](CHANGES.md). A key says, for each keyed item, exactly what is marked and nothing a marker has to guess: -- **Question 1**: the obligations a bullet may name, and the assertions that make a bullet wrong. +- **Question 1**: the obligations a bullet may name, the assertions that make a bullet wrong, + and the wrong readings the text invites, if any. - **Question 2**: for each item, the elements an answer is required to state, numbered, one for each part of the question; the assertions that make an answer wrong; and the sentence or - sentences of the document the answer rests on, quoted verbatim. The required elements are - what the question asks for, no more: an element the question does not ask for is not - required, however load-bearing, until the question is reworded to ask for it. + sentences of the document the answer rests on, quoted verbatim, which show the marker where + the elements come from and which `spec-qa.py` checks against the document; a reader's + quotation need not be that one. The required elements are what the question asks for, no + more: an element the question does not ask for is not required, however load-bearing, until + the question is reworded to ask for it. - **Question 3**: for each item, the grade or grades accepted, any condition on the reasoning, and the wrong readings the text invites, which count against the document when one reader takes them. @@ -55,9 +58,12 @@ A key, a question set or a fixture changes by one of two routes, as `ACCEPTANCE. recorded is settled by whichever route it needs: - **In a specification pull request, under step 4 of `CONTRIBUTING.md`**: a quotation follows - the text it quotes, or a keyed item is added for a new load-bearing obligation together with - the question that asks for it. The pull request's own changelog line records it. Such an - update does not change what an existing item requires, nor an existing question's wording. + the text it quotes; a keyed item is added for a new load-bearing obligation together with the + question that asks for it; or, where the change alters or removes the obligation an existing + item tests, that item and its question follow the text, which is the text changing and not + the test. The pull request's own changelog line records it. Such an update does not change + what an item requires of an obligation the change leaves as it was, nor the wording of a + question whose obligation is unchanged. - **In a process change, with its own pull request**: what an existing item requires, the wording of a question, a fixture, or the rules of marking. It is recorded in [CHANGES.md](CHANGES.md) with its reason. diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 60fc616..24beef9 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -1,8 +1,8 @@ # Passages routed out of a cohort -A finding is a passage two or more readers of one cohort named as confusing, under question 4, -that lies outside the text the change being tested wrote or altered. `ACCEPTANCE.md` routes such -a finding here rather than blocking the change whose cohort raised it, so that each change is +A finding is a passage two or more readers of one cohort named as confusing under question 4. +`ACCEPTANCE.md` routes here a finding that lies outside the text the change being tested wrote +or altered, rather than blocking the change whose cohort raised it, so that each change is judged on what it changed and the document is still made clearer, one change at a time. The editor reads this list before cutting a release. diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index aaddb69..5b4d4aa 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -7,8 +7,8 @@ update the quotation here in the same commit. `ACCEPTANCE.md` says how each item ## 1. Restatement -A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting one the list of wrong assertions below marks wrong, is a correct restatement. +A bullet is correct when it names one of these obligations. Four or more correct bullets, with +no bullet making an assertion listed below as wrong, is a correct restatement. - Support bespoke rules written by the project ([4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it)). - Provide a harness that runs a single rule against supplied code ([4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code)). @@ -39,10 +39,11 @@ Wrong readings the text invites, which count against the document on one reader: MAY declare the version of this document the detector conforms to". The reason, that conformance is judged on evidence and a declaration is optional, is explanation: an answer that gives it, or gives it instead of a plain yes, states element (2). -- **2b.** Required: (1) it must let the project switch the route off in its configuration, or - make the route detectable (by a rule the project can write, or by a mechanical check the - detector documents; naming either means of detection, or just "detectable", states this); - (2) yes, either one alone is enough. Wrong if it says the detector must remove or forbid the +- **2b.** Required: (1) the two things, both named: let the project switch the route off in its + configuration, or make the route detectable (by a rule the project can write, or by a + mechanical check the detector documents; naming either means of detection, or just + "detectable", states that alternative); an answer naming only one of the two things misses + this element; (2) yes, either one alone is enough. Wrong if it says the detector must remove or forbid the route, or must require a reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD). Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each one it MUST do at least one of two things, and either alone satisfies this clause: make the route disableable by configuration; or make the route detectable, by a Rule the project can write in diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index b756f47..a990c7b 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -7,8 +7,8 @@ update the quotation here in the same commit. `ACCEPTANCE.md` says how each item ## 1. Restatement -A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting one the list of wrong assertions below marks wrong, is a correct restatement. +A bullet is correct when it names one of these obligations. Four or more correct bullets, with +no bullet making an assertion listed below as wrong, is a correct restatement. - Attribute the defect to a class, bounded on both sides, with the hazard stated ([3.1](../../SPEC.md#31-attribute-the-defect-to-a-class)). - Express the class as a rule in a detector that reads code, never a test ([3.2](../../SPEC.md#32-build-the-net)). @@ -69,7 +69,8 @@ Wrong readings the text invites, which count against the document on one reader: The three answers the question then asks for, each with the condition its reason must meet: - **The remediation under section [7](../../SPEC.md#7-conformance).** Accepted answer: does not conform. Condition: the reason - names [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) or [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the clauses not followed; other clauses named alongside are not marked. + names [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), [3.5](../../SPEC.md#35-enforce-permanently-and-block) or [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the clauses not followed; other clauses named alongside are not + marked. - **The defence alone.** Accepted answer: does not conform. Condition: the reason names [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation), or [3.5](../../SPEC.md#35-enforce-permanently-and-block)'s unrecorded suppression, which section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence also requires to be absent; other clauses named alongside are not marked. diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 7353852..2d2bea3 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -7,8 +7,8 @@ update the quotation here in the same commit. `ACCEPTANCE.md` says how each item ## 1. Restatement -A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting one the list of wrong assertions below marks wrong, is a correct restatement. +A bullet is correct when it names one of these obligations. Four or more correct bullets, with +no bullet making an assertion listed below as wrong, is a correct restatement. - Route defences only through detectors that conform to the detector specification, wrapping permitted ([4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification)). @@ -69,9 +69,11 @@ Wrong readings the text invites, which count against the document on one reader: The four answers the question then asks for, each with the condition its reason must meet: - **Conformance under section [9](../../TOOLING-SPEC.md#9-conformance).** Accepted answer: does not conform. Condition: the reason - names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) or [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) among the failing clauses; other clauses named alongside are not marked. + names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) or [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) among the failing clauses; other clauses named alongside are not + marked. - **Conformance with agent support.** Accepted answer: no. Condition: the reason says that - section [7](../../TOOLING-SPEC.md#7-agent-context) is not met, or that there is no summary. + section [7](../../TOOLING-SPEC.md#7-agent-context) is not met, or that there is no summary, or that the toolchain does not conform + under section [9](../../TOOLING-SPEC.md#9-conformance) at all. - **Whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies.** Accepted answer: it does not. Condition: the reason says that Zigpipe ships nothing to other projects, or that section [8](../../TOOLING-SPEC.md#8-self-audit) bears on zig-qa's own artefact grade and not on Zigpipe. From c31d49a0ac8e66931784ac09ef342d78347b0957 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 12:00:29 +0100 Subject: [PATCH 068/105] Acceptance process: CHANGES.md names pull request 22 Co-Authored-By: Claude Fable 5.1 --- acceptance/CHANGES.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 29a20db..daa3d44 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -6,7 +6,8 @@ change, naming its pull request. The process is not a specification and is not v change to it is not tested by a cohort. Changes to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). -- Pull request NNN: the test is made to prove what it exists to prove, the clarity of the +- [Pull request #22](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/22): + the test is made to prove what it exists to prove, the clarity of the document to a cold reader, after thirty-three runs on two changes (19 to 30, 31 to 33 and 35 to 52) in which keyed answers reached five of five and the runs still failed. Five defects in the process were found. Keys required elements the question never asked for (detector 2b's From e6bdf402e051155cf4b7752f62ad4dd8a5ef41a1 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 12:57:49 +0100 Subject: [PATCH 069/105] Keys and routed findings: open points from runs 53 to 59 The method key lists fixing the original defect with a reproducing test among the obligations, says a bullet on the verdict resting on reproduction describes the reviewer, and says that an answer to 2a naming no step does not state the steps. The toolchain key lists the governing principle, says a bullet on how conformance is measured is neither correct nor wrong, and accepts a section 9 reason that points to the grades given. acceptance/ROUTED.md gains the findings runs 53 to 59 routed, and the runs that raised existing rows again. Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 33 +++++++++++++++++++-------------- acceptance/method/KEY.md | 9 +++++++-- acceptance/toolchain/KEY.md | 12 +++++++++--- 3 files changed, 35 insertions(+), 19 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 24beef9..901caed 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -10,24 +10,29 @@ A finding is open until a change that reworks its passage passes a cohort, when that run and moves below the rule, or until the editor closes it with a reason recorded here. A finding raised again by a later run is noted on its line, not listed twice. The status column holds `open`, or the `editor/` branch or pull request number of the change that is -reworking the passage. The rows below were raised by runs 31 to 52 under the earlier rules, on +reworking the passage. The rows raised by runs 31 to 52 were raised under the earlier rules, on untouched text of the change those runs tested, and are listed here so that they are not lost. ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | ---------------------------------------- | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | run 33 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------- | ---------------------------------------- | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | run 54 | `editor/detector-route-rule` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59 | `open` | ## Closed diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index a990c7b..86cf8ff 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -21,9 +21,13 @@ no bullet making an assertion listed below as wrong, is a correct restatement. than only mentioning it (section [2](../../SPEC.md#2-when-the-method-applies)). - Take a baseline, a suppression or an instance left unfixed to the owner, never on the practitioner's own authority (section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom)). +- Fix the original defect conventionally, with a test that reproduces it, once the defence is in + place (sections [2](../../SPEC.md#2-when-the-method-applies) and [6](../../SPEC.md#6-relationship-to-test-driven-development)). Wrong if it asserts: the practitioner fixes the reported instance first and defends afterwards; -a test may serve as the detector; an agent may baseline the remaining instances. +a test may serve as the detector; an agent may baseline the remaining instances. A bullet that +says the verdict on a remediation rests on reproduction (section [7](../../SPEC.md#7-conformance)) describes the reviewer, +not the practitioner, and is neither correct nor wrong. Wrong readings the text invites, which count against the document on one reader: none named. @@ -36,7 +40,8 @@ Wrong readings the text invites, which count against the document on one reader: order."; [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance): "The Practitioner MUST run the Rule across the entire codebase and record the total Instance count before fixing anything, and MUST then fix every Instance found rather than only the one". An answer that names the steps by their clause numbers in order states - both elements. + both elements; one that says only that section [3](../../SPEC.md#3-the-method)'s six clauses are followed in order, + naming no step, does not state element (1). - **2b.** Required: (1) the owner of the codebase, the human who holds that authority; (2) no, an agent may not do so on its own authority. Wrong if it lets the practitioner or an agent decide because the count was small, because the project recorded no owner, or at all. Key diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 2d2bea3..e0388d4 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -24,11 +24,16 @@ no bullet making an assertion listed below as wrong, is a correct restatement. same way as the defences ([6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) to [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences)). - A shipped toolchain also fails its release on undocumented bundled defences and runs its own defences on its own source ([8.1](../../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation), [8.2](../../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source)). +- Make everything the method requires of a practitioner possible without the project building + the mechanism first (section [1](../../TOOLING-SPEC.md#1-what-this-document-is-for), the governing principle). Wrong if it asserts: the toolchain must be a single product; a declaration is required to conform; the agent summary (section [7](../../TOOLING-SPEC.md#7-agent-context)) is a MUST. A bullet that calls agent support or the self-audit a further condition "where it applies", or says a declaration is not a condition, -asserts no obligation and is neither correct nor wrong. +asserts no obligation and is neither correct nor wrong. So is a bullet that states how +conformance is measured rather than what is required: the toolchain judged as one assembled +whole at the project level (sections [1](../../TOOLING-SPEC.md#1-what-this-document-is-for) and [9](../../TOOLING-SPEC.md#9-conformance)), the project-level verdict a single grade +(section [9](../../TOOLING-SPEC.md#9-conformance)), or a shipped artefact graded separately ([9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately)). Wrong readings the text invites, which count against the document on one reader: none named. @@ -69,8 +74,9 @@ Wrong readings the text invites, which count against the document on one reader: The four answers the question then asks for, each with the condition its reason must meet: - **Conformance under section [9](../../TOOLING-SPEC.md#9-conformance).** Accepted answer: does not conform. Condition: the reason - names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) or [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) among the failing clauses; other clauses named alongside are not - marked. + names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) or [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) among the failing clauses, or points to the grades the reader gave above, as "the + clauses graded No", where those grades have one of the three at No or Partial; other clauses + named alongside are not marked. - **Conformance with agent support.** Accepted answer: no. Condition: the reason says that section [7](../../TOOLING-SPEC.md#7-agent-context) is not met, or that there is no summary, or that the toolchain does not conform under section [9](../../TOOLING-SPEC.md#9-conformance) at all. From ed30e694e1d80e342beef8910228f01602d01557 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:02:16 +0100 Subject: [PATCH 070/105] Detector: the detectable route, the gap duty, the remainder and the mechanism Four passages acceptance/ROUTED.md lists, each quoted by two or more cold readers of the defined-terms change. Clause 7.1's detectable route is a bespoke rule the project writes for the detector under clause 4.1, with a paragraph on what detectable means and the two ways; clause 8.1's gap duty falls only on a detector that declares a version; clause 5.3 says what the remainder is and when there is none; clause 6.1 says it asks for the mechanism, a missing page failing 6.3, and that a path in the message is not enough on its own. The detector key's 7.1 quotation follows the text. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 16 ++++++++++++++ DETECTOR-SPEC.md | 44 ++++++++++++++++++++++++++------------ acceptance/detector/KEY.md | 17 ++++++++------- 3 files changed, 55 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f5f8122..bfeb0df 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -68,6 +68,22 @@ pass it. ## Detector specification (DETECTOR-SPEC.md) +### Unreleased + +- Clarity, no obligation changed: four passages that cold readers of an unrelated change read as + the document pulling against itself or could not follow, each listed in `acceptance/ROUTED.md`. + Clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)'s detectable route is now a bespoke rule the project writes for the detector under + clause [4.1](DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it), after six runs read "a Rule the project can write in the Detector itself" as the + project editing the detector's own source, and a new paragraph says what detectable means and + what each of the two ways is. Clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps)'s duty to record a known gap falls only on a detector + that declares a version, and says so beside "The declaration is optional", with the method's + clause [3.2](SPEC.md#32-build-the-net) record applying either way. Clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) says what the remainder is, and that + there is none where the output prints every finding. Clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) says it asks for the + mechanism and not for every page, a missing page failing clause [6.3](DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), and that a + documentation path printed in the message is not enough on its own, after three readers of + run 54 set that sentence against the mechanism. Prompted by runs 32 to 54 on the defined-terms + change, in which two or more readers quoted each passage. Accepted by run NNN. + ### 1.0.0, 2026-09-08 First version, split out of toolchain specification 0.1.0 on the same day. States what a detector, diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index cdc6c9e..7583801 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -157,7 +157,10 @@ any. ### 5.3 The result MUST reach the practitioner in the output of the command they ran Where the [Detector] writes fuller detail elsewhere, the invoked command's own output MUST -carry both a usable summary and the location of the remainder. +carry both a usable summary and the location of the remainder. The remainder is the findings +the summary leaves out. Where the output prints every finding, each with where it is and its +[Identifier], there is no remainder, and a report that only adds detail to those same findings, +such as a code snippet, need not be named in the output. **Why**: method specification clause 8.2. A report the [Practitioner] has to go and find is a report that arrives after the decision it was meant to inform. @@ -183,13 +186,16 @@ Two cases, by who wrote the [Rule]: | The project's own | The project | Print the [Identifier] unaltered; the lookup is the [Toolchain]'s to give | In both, the mechanism is keyed on **the [Identifier] exactly as printed**. A command, an index -file or a URL are all acceptable forms. Each case in turn: +file or a URL are all acceptable forms. This clause asks for the mechanism, not for every page: +where the mechanism works but a [Bundled rule]'s [Identifier] has no documentation to resolve +to, that fails clause 6.3, not this clause. Each case in turn: **For a [Bundled rule]**, the [Detector] supplies the documentation and the lookup, and clauses 6.2 and 6.3 say where. The mechanism MUST resolve an [Identifier] presented alone, because the reader who needs it most has the [Identifier] from a log, a -ticket or a colleague and not the [Message]. A [Message] that carries its own -documentation path resolves that [Message], not the [Identifier]. +ticket or a colleague and not the [Message]. A documentation path printed in the [Message] is +therefore not enough on its own: it helps only a reader who has the [Message], and the +[Identifier] alone still leads nowhere. **For a project's own [Rule]**, the [Detector] cannot know the documentation, so what it owes is the half it can give: the [Identifier] printed unaltered under clause 4.3, and no @@ -236,7 +242,7 @@ An automated check over every [Identifier] a [Bundled rule] can print, applying family list or pattern of clause 6.5 where one is used, that blocks the release when any of them lands on no page. -**Why**: clause 6.1 is the clause most easily believed to be satisfied whilst being broken, because +**Why**: clause 6.3 is the clause most easily believed to be satisfied whilst being broken, because the documentation is written by the same person who wrote the [Rule] and its absence is invisible from the inside. Where the [Detector] is shipped inside a [Toolchain], the [toolchain specification](TOOLING-SPEC.md)'s self-audit makes this check a MUST for the [Toolchain]; a @@ -269,9 +275,18 @@ nothing to run. An inline ignore comment, a per-line directive and a generated [Baseline] are all such routes. The [Detector] MAY ship them. For each one it MUST do at least one of two things, and either alone satisfies this clause: make the route disableable by configuration; or make the route -detectable, by a [Rule] the project can write in the [Detector] itself or by a mechanical check -the [Detector] documents. Either lets a project which decides to forbid the route enforce that -decision. A route that can be neither switched off nor seen does not [Conform]. +detectable, by a bespoke [Rule] the project writes for the [Detector] under clause 4.1 or by a +mechanical check the [Detector] documents. Either lets a project which decides to forbid the +route enforce that decision. A route that can be neither switched off nor seen does not [Conform]. + +A route is detectable when the project can find every use of it. The first way is a bespoke +[Rule]: the project registers a [Rule] of its own, as clause 4.1 provides for, that matches the +route wherever it is used, and the [Detector] runs it like any other bespoke [Rule]; the +[Detector] ships no such [Rule] itself, and need not. That way is open only where the +[Detector]'s [Rules] can see the route: a [Detector] whose [Rules] never see comments cannot make +an ignore comment detectable this way. The second way is a mechanical check the [Detector] +documents, such as a text search for the route's spelling that the [Detector]'s documentation +gives. **Why**: the method specification's position is that [Suppression] is an [Owner] decision under its clause 3.4 and section 4, and an [Owner] cannot decide something they are @@ -310,14 +325,15 @@ for a [Detector]. The declaration is machine-readable, in whatever form the [Detector]'s ecosystem uses to record dependencies. The same declaration is where a gap against this document is recorded once it is -known: a [Detector] that has learnt, from its own checks or from a [Practitioner]'s -report under the method's clause 3.2, that it fails a MUST in sections 4 to 7 MUST record that gap -alongside the version it declares, in the same file or one it names. A declaration with a non-empty -gap record is a statement of where the [Detector] stands and is not a claim of +known: a [Detector] that declares a version and has learnt, from its own checks or from a +[Practitioner]'s report under the method's clause 3.2, that it fails a MUST in sections 4 to 7 +MUST record that gap alongside that version, in the same file or one it names. A declaration +with a non-empty gap record is a statement of where the [Detector] stands and is not a claim of [Conformance]. -The declaration is optional. It is how a maintainer claims [Conformance]; it is not a -condition of it. A [Detector] that predates this document, or whose maintainer has never read +The declaration is optional, and so the duty above falls only on a [Detector] that has one; a +gap a [Practitioner] finds is recorded under the method's clause 3.2 in any case. It is how a +maintainer claims [Conformance]; it is not a condition of it. A [Detector] that predates this document, or whose maintainer has never read it, MAY be graded [Conforming] on evidence by anyone who exercises the clauses above against it, and a verdict on any [Detector], declared or not, rests on that exercise and not on the claim. A declaration tells the reader what the maintainer believes and what they know to be missing; diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 5b4d4aa..1f4348b 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -40,14 +40,15 @@ Wrong readings the text invites, which count against the document on one reader: conformance is judged on evidence and a declaration is optional, is explanation: an answer that gives it, or gives it instead of a plain yes, states element (2). - **2b.** Required: (1) the two things, both named: let the project switch the route off in its - configuration, or make the route detectable (by a rule the project can write, or by a - mechanical check the detector documents; naming either means of detection, or just - "detectable", states that alternative); an answer naming only one of the two things misses - this element; (2) yes, either one alone is enough. Wrong if it says the detector must remove or forbid the - route, or must require a reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD). Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each one it - MUST do at least one of two things, and either alone satisfies this clause: make the route - disableable by configuration; or make the route detectable, by a Rule the project can write in - the Detector itself or by a mechanical check the Detector documents". An answer need not say + configuration, or make the route detectable (by a bespoke rule the project writes for the + detector, or by a mechanical check the detector documents; naming either means of detection, + or just "detectable", states that alternative); an answer naming only one of the two things + misses this element; (2) yes, either one alone is enough. Wrong if it says the detector must + remove or forbid the route, or must require a reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD). Key quotation, + [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each one it MUST do at least one of two things, and either alone satisfies this + clause: make the route disableable by configuration; or make the route detectable, by a + bespoke Rule the project writes for the Detector under clause [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) or by a mechanical check the + Detector documents". An answer need not say that the detector is not required to remove or forbid the route; that follows from element (2) and the question does not ask it. - **2c.** Required: (1) with the rule, in the installed copy; (2) at a version tracked together From 4081aa45e771f14fb832e708ba612c88f5008d8e Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:16:08 +0100 Subject: [PATCH 071/105] Toolchain key and routed findings: open points from runs 61 and 62 The toolchain key counts a bullet on not describing partial conformance as conformance, and says a bullet calling section 7 recommended or a SHOULD asserts no obligation. acceptance/ROUTED.md gains clause 3.4's "tremendous" sentence from run 61 and notes that run on the two rows it raised again. Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 5 +++-- acceptance/toolchain/KEY.md | 6 ++++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 901caed..76acd8f 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -31,8 +31,9 @@ untouched text of the change those runs tested, and are listed here so that they | `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56 | `open` | | `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | | `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59, 61 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59, 61 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | run 61 | `open` | ## Closed diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index e0388d4..aadab7a 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -26,11 +26,13 @@ no bullet making an assertion listed below as wrong, is a correct restatement. defences on its own source ([8.1](../../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation), [8.2](../../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source)). - Make everything the method requires of a practitioner possible without the project building the mechanism first (section [1](../../TOOLING-SPEC.md#1-what-this-document-is-for), the governing principle). +- Not describe partial conformance as conformance (section [9](../../TOOLING-SPEC.md#9-conformance)). Wrong if it asserts: the toolchain must be a single product; a declaration is required to conform; the agent summary (section [7](../../TOOLING-SPEC.md#7-agent-context)) is a MUST. A bullet that calls agent support or the -self-audit a further condition "where it applies", or says a declaration is not a condition, -asserts no obligation and is neither correct nor wrong. So is a bullet that states how +self-audit a further condition "where it applies", says that section [7](../../TOOLING-SPEC.md#7-agent-context) is recommended or a +SHOULD, or says a declaration is not a condition, asserts no obligation and is neither correct +nor wrong. So is a bullet that states how conformance is measured rather than what is required: the toolchain judged as one assembled whole at the project level (sections [1](../../TOOLING-SPEC.md#1-what-this-document-is-for) and [9](../../TOOLING-SPEC.md#9-conformance)), the project-level verdict a single grade (section [9](../../TOOLING-SPEC.md#9-conformance)), or a shipped artefact graded separately ([9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately)). From c1c986e0804af06129ee4f4308168f71e227341a Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:17:27 +0100 Subject: [PATCH 072/105] Run 060: detector route rule cohort, failed on two findings Every keyed item within threshold, one reader leaving clause 7.1 ungraded. Two findings in text the change wrote or altered: clause 6.1's sentence on a documentation path in the Message, whose term two readers could not find defined, and clause 8.1's "anyone who exercises the clauses above". Nothing routed. Co-Authored-By: Claude Fable 5.1 --- .../runs/060-detector-route-rule-run-1.md | 138 ++++++++++++++++++ 1 file changed, 138 insertions(+) create mode 100644 acceptance/runs/060-detector-route-rule-run-1.md diff --git a/acceptance/runs/060-detector-route-rule-run-1.md b/acceptance/runs/060-detector-route-rule-run-1.md new file mode 100644 index 0000000..20317b4 --- /dev/null +++ b/acceptance/runs/060-detector-route-rule-run-1.md @@ -0,0 +1,138 @@ +# Run 60: DETECTOR-SPEC.md, detector route rule, first run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: not yet opened at the time of the run; it is to be opened from branch + `editor/detector-route-rule`. The readers read commit ed30e69. Change: clause 7.1's detectable + route is a bespoke rule the project writes for the detector under clause 4.1, with a paragraph + on what detectable means; clause 8.1's gap duty falls only on a detector that declares a + version; clause 5.3 says what the remainder is; clause 6.1 says it asks for the mechanism, a + missing page failing clause 6.3, and that a path in the message is not enough on its own. +- **Run within the pull request**: 1. +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `d3b016b18288e3589df60de97f746a31179924dbdd5faa339c055fbf95d89125`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at ed30e69, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | --- | --- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | miss | ok | ok | + +Question 1: correct bullets per reader 6, 6, 6, 6 and 6, one for each line of the key's list +(4.1; 4.2; 4.3; 5.1 to 5.4; 6.1 to 6.3; 7.1). Reader 1's third bullet adds 4.4 as a SHOULD and +its sixth adds what conformance means beside 7.1; both are, by the key, neither correct nor +wrong, and each bullet still names its obligation. Reader 2's fourth bullet says every finding +must reach the output, which the key says names 5.3 and is not wrong. No bullet asserts anything +the key marks wrong. + +Question 2: every reader states every required element of each item and quotes the document. +For 2a all five quote section 8's "A Detector Conforms if it satisfies every MUST in sections 4 +to 7" or clause 8.1's "A Detector that predates this document, or whose maintainer has never +read it, MAY be graded Conforming on evidence"; for 2b all five quote clause 7.1's "For each one +it MUST do at least one of two things, and either alone satisfies this clause", and name both +things and that either is enough; for 2c all five quote the headings or first sentences of 6.2 +and 6.3. Readers 1 and 2, under 2b, add that the detectable route is closed to a detector whose +rules never see comments, so that it needs the disable route or a documented mechanical check; +that is not among the key's wrong assertions for 2b, and the answers are correct by the rule. + +Question 3: readers 1 to 4 give every accepted grade; reader 5 gives every accepted grade on the +ten rows it answers. S8: all five say Lintomatic does not conform, naming 6.3. 6.4: all five say +it does not change the verdict, because it is a SHOULD. + +### Misses + +- **Reader 5, question 3, clause 7.1**: unanswered. Its table ends at 6.3 and no grade for 7.1 + appears anywhere on the sheet, so the item is a miss under "An unanswered question". There is + no reasoning, so nothing quotes the document; it is not a reading the key names (7.1 graded + No on 7.2's reason). The other four readers are correct on the item, so the threshold of all + but one is met. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..ed30e69 adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | ------------------- | +| 1 | Clause 6.1, "For a Bundled rule" paragraph: "A documentation path printed in the [Message] is therefore not enough on its own"; both readers say [Message] is not defined in the document they were given | 1, 5 | yes | +| 2 | Clause 8.1, second paragraph: "MAY be graded [Conforming] on evidence by anyone who exercises the clauses above against it" (5, whether the MAY is a permission or a possibility); "by anyone who exercises the clauses above against it" (3, whether "the clauses above" are sections 4 to 7 or earlier clauses of section 8) | 3, 5 | yes | +| 3 | Clause 6.1, "For a project's own Rule" paragraph: "Where the [Identifier] is itself a URL, as method clause 3.6 allows"; the reader could not check method clause 3.6 | 1 | no | +| 4 | Clause 6.1 heading: "The Detector MUST provide a mechanism that resolves a printed identifier to its documentation", set against passage 5 | 2 | no | +| 5 | Clause 6.1 table, project's own row: "Print the Identifier unaltered; the lookup is the Toolchain's to give", set against passage 4 | 2 | no | +| 6 | Clause 5.3, first paragraph: "a report that only adds detail to those same findings, such as a code snippet, need not be named in the output" | 2 | yes | +| 7 | Clause 8.1, first paragraph: "a [Detector] that declares a version and has learnt ... that it fails a MUST in sections 4 to 7 MUST record that gap alongside that version", set against section 8's definition and the optional declaration | 3 | yes | +| 8 | Clause 4.3, first paragraph: "in its default output and in every machine-readable format it offers"; "machine-readable" is not defined | 4 | no | + +## Findings and where they lie + +- **Finding A, passage 1, clause 6.1 "For a Bundled rule" paragraph** (readers 1 and 5): the + sentence both readers quote was written by the change, replacing "A [Message] that carries its + own documentation path resolves that [Message], not the [Identifier]." Text the change wrote. + Blocks. +- **Finding B, passage 2, clause 8.1 second paragraph** (readers 3 and 5): the sentence both + readers quote is not itself reworded, but the change rewrote the paragraph's first sentence + ("The declaration is optional, and so the duty above falls only on a [Detector] that has one; + ...") and added its second. The rule places a finding by paragraph, so the paragraph is the + change's. Blocks. + +Passages 3 to 8 are each named by one reader: recorded, not findings. Nothing is routed to +`acceptance/ROUTED.md` from this run. + +## Points the rules did not settle + +1. **Passage 1, a term defined in a document not supplied.** Readers 1 and 5 each quote the + sentence with its clause number, which `ACCEPTANCE.md` defines as naming a passage, and the + confusion each gives is that [Message] is defined in the method specification, which they + were not given. The rule that a remark about another document not being supplied is a note, + not a passage, covers a remark "not about a passage of the document"; these remarks are about + a quoted passage, so they were read as naming it. Had it been open, the open-point rule on + whether a reader named a passage counts the passage in any case. Finding A turns on this + point. Reader 1's passage 3 (method clause 3.6) is the same case, counted the same way, and + is named by one reader only. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit e6bdf40, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item but + 7.1, where four of five are correct (reader 5 left it unanswered). Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: two (findings A + and B). Not met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A and B alone; every keyed item passed its threshold. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From 4cbecad23a52dbdf54b48292f3c1c8d422f4e956 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:17:27 +0100 Subject: [PATCH 073/105] Detector: a Message row in section 2, and who may grade a detector After run 60: section 2's table gains a Message row, since two readers could not find the term defined; clause 8.1 says anyone may grade a detector by checking it against each clause of sections 4 to 7, in place of "anyone who exercises the clauses above against it". Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 7 ++++++- DETECTOR-SPEC.md | 7 ++++--- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bfeb0df..9d6fc71 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -82,7 +82,12 @@ pass it. mechanism and not for every page, a missing page failing clause [6.3](DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), and that a documentation path printed in the message is not enough on its own, after three readers of run 54 set that sentence against the mechanism. Prompted by runs 32 to 54 on the defined-terms - change, in which two or more readers quoted each passage. Accepted by run NNN. + change, in which two or more readers quoted each passage. + [Run 60](acceptance/runs/060-detector-route-rule-run-1.md) failed on two findings in the + revised text, every keyed item within threshold: two readers could not find Message defined, + so section [2](DETECTOR-SPEC.md#2-terminology)'s table gains a Message row; and two readers stopped at clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps)'s + "anyone who exercises the clauses above", which now says anyone may grade a detector by + checking it against each clause of sections [4](DETECTOR-SPEC.md#4-authoring-rules) to [7](DETECTOR-SPEC.md#7-suppression). Accepted by run NNN. ### 1.0.0, 2026-09-08 diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index 7583801..06a4238 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -39,6 +39,7 @@ definition; the ones this document leans on most are, in short: | -------------- | ------------------------------------------------------------------------------ | | [Detector] | A tool that reads code without executing it and reports what it finds | | [Rule] | One check a [Detector] evaluates | +| [Message] | The text a [Rule] prints when it fires | | [Identifier] | The stable name printed with a finding that leads to its documentation | | [Practitioner] | Whoever is doing the work, a person or an [Agent] | | [Toolchain] | Everything a project assembles to run its checks through, [Detectors] included | @@ -334,9 +335,9 @@ with a non-empty gap record is a statement of where the [Detector] stands and is The declaration is optional, and so the duty above falls only on a [Detector] that has one; a gap a [Practitioner] finds is recorded under the method's clause 3.2 in any case. It is how a maintainer claims [Conformance]; it is not a condition of it. A [Detector] that predates this document, or whose maintainer has never read -it, MAY be graded [Conforming] on evidence by anyone who exercises the clauses above against -it, and a verdict on any [Detector], declared or not, rests on that exercise and not on the -claim. A declaration tells the reader what the maintainer believes and what they know to be missing; +it, MAY be graded [Conforming] on evidence: anyone may grade it, by checking it against each +clause of sections 4 to 7, and a verdict on any [Detector], declared or not, rests on that +check and not on the claim. A declaration tells the reader what the maintainer believes and what they know to be missing; the reader still runs the [Harness]. The shape is illustrative rather than prescribed. In a Composer manifest: From 99074928ac750fd2098b309725424dd221403539 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:29:03 +0100 Subject: [PATCH 074/105] Routed findings from runs 63 and 64 acceptance/ROUTED.md gains the detector's clause 6.1 paragraph on a project's own rule, from run 63, and notes run 64 on the four method rows it raised again. Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 76acd8f..55462f4 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -15,25 +15,26 @@ untouched text of the change those runs tested, and are listed here so that they ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------- | ---------------------------------------- | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | run 54 | `editor/detector-route-rule` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59, 61 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59, 61 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | run 61 | `open` | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ---------------------------------------- | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | run 54 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | run 63 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59, 64 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59, 61, 64 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59, 61, 64 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | run 61 | `open` | ## Closed From 890d8e3d40e327caaf8776d890b97e4940bad536 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:29:20 +0100 Subject: [PATCH 075/105] Run 063: detector route rule cohort, failed on one finding Every keyed item five of five. One finding in text the change wrote: clause 5.3's sentence on a report that only adds detail. One finding in untouched text, clause 6.1's paragraph on a project's own rule, routed to acceptance/ROUTED.md. Co-Authored-By: Claude Fable 5.1 --- .../runs/063-detector-route-rule-run-2.md | 144 ++++++++++++++++++ 1 file changed, 144 insertions(+) create mode 100644 acceptance/runs/063-detector-route-rule-run-2.md diff --git a/acceptance/runs/063-detector-route-rule-run-2.md b/acceptance/runs/063-detector-route-rule-run-2.md new file mode 100644 index 0000000..c3fa574 --- /dev/null +++ b/acceptance/runs/063-detector-route-rule-run-2.md @@ -0,0 +1,144 @@ +# Run 63: DETECTOR-SPEC.md, detector route rule, second run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: not yet opened at the time of the run; it is to be opened from branch + `editor/detector-route-rule`. The readers read commit 4cbecad. Change: clause 7.1's detectable + route is a bespoke rule the project writes for the detector under clause 4.1, with a paragraph + on what detectable means; clause 8.1's gap duty falls only on a detector that declares a + version; clause 5.3 says what the remainder is; clause 6.1 says it asks for the mechanism, a + missing page failing clause 6.3, and that a path in the message is not enough on its own. + Revised after run 60 in section 2's table (a Message row) and clause 8.1's second paragraph. +- **Run within the pull request**: 2 (run 60 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `30b41f341b26e48b3c8bf1b1d55a25283ffd43c835328eaf5cd9ec021d265e09`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at 4cbecad, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Question 1: correct bullets per reader 6, 6, 6, 6 and 6, one for each line of the key's list +(4.1; 4.2; 4.3; 5.1 to 5.4; 6.1 to 6.3; 7.1). Reader 3's third bullet adds 4.4 as a SHOULD; +readers 2 and 3 add to their sixth bullet that a written reason is a SHOULD; reader 5's sixth +bullet adds the SHOULDs of 7.2, 4.4 and 6.4. Each restates a SHOULD as advice, which the key +says is neither correct nor wrong, and each bullet still names its obligation. No bullet asserts +anything the key marks wrong. + +Question 2: every reader states every required element of each item and quotes the document. +For 2a all five state that conformance is every MUST in sections 4 to 7 and that a detector +predating the document can conform, quoting section 8's definition or clause 8.1's "A Detector +that predates this document, or whose maintainer has never read it, MAY be graded Conforming on +evidence" (reader 1's quotation of section 8 drops "[Detector]", and quotes the document by the +nine-word run "if it satisfies every MUST in sections 4 to 7"; its 8.1 quotation is verbatim). +For 2b all five name both things and that either is enough, quoting clause 7.1's "For each one +it MUST do at least one of two things, and either alone satisfies this clause". Readers 2, 3 +and 5 add that the bespoke-rule way of detection is closed to a detector whose rules never see +comments; reader 2 goes on to say that for an ignore comment the configuration switch or a +documented mechanical check is what remains. Neither is among the key's wrong assertions for +2b (removing or forbidding the route, or requiring a reason), and the answers are correct by the +rule. For 2c all five state the installed copy, the version tracked together and no network +access, quoting the headings of 6.2 and 6.3. + +Question 3: all five give every accepted grade. Reader 2 grades 4.3 "Yes, with a caveat" about +machine-readable formats; the key accepts Yes with no condition on the reasoning and says a +caveat with the accepted grade is correct. S8: all five say Lintomatic does not conform, naming +6.3. 6.4: all five say it does not change the verdict, because it is a SHOULD. + +### Misses + +None. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..4cbecad adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------- | +| 1 | Clause 5.3, first paragraph: "a report that only adds detail to those same findings, such as a code snippet, need not be named in the output" (1, took two readings to see whether the exemption covers the HTML report; 2, the subject of "need not be named" took two readings to settle) | 1, 2 | yes | +| 2 | Clause 6.1, "For a project's own Rule" paragraph: "Where the [Identifier] is itself a URL, as method clause 3.6 allows, the [Detector] prints that URL and has done what this clause asks" (1 and 4, could not check method clause 3.6); "The lookup for such a [Rule] is an obligation on the project's assembled [Toolchain]" (5, set against passages 7 and 8) | 1, 4, 5 | no | +| 3 | Clause 4.3, first paragraph: "in its default output and in every machine-readable format it offers"; "machine-readable" not said, so whether the HTML report is covered | 2 | no | +| 4 | Clause 6.4, Why paragraph: "the [toolchain specification]'s self-audit makes this check a MUST for the [Toolchain]"; the reader was not given that document | 2 | yes | +| 5 | Clause 8.1, second paragraph: "A Detector that predates this document, or whose maintainer has never read it, MAY be graded Conforming on evidence"; whether the MAY is a permission to the grader or an eligibility of the detector, settled by the next sentence | 3 | yes | +| 6 | Clause 7.1, second paragraph: "That way is open only where the [Detector]'s [Rules] can see the route: a [Detector] whose [Rules] never see comments cannot make an ignore comment detectable this way"; what it means for a rule to "see" a comment | 4 | yes | +| 7 | Section 1, the governing principle paragraph: "a [Conforming] [Detector] MUST make that possible without the project building the mechanism first", set against passages 2 and 8 | 5 | no | +| 8 | Clause 6.1 table, project's own row: "the lookup is the [Toolchain]'s to give", set against passage 7 | 5 | no | + +Notes, not passages: reader 5 says the fixture's `explain` paragraph and its last paragraph +disagree about which rules print `No documentation declared`; that is about the fixture, not +the document. + +## Findings and where they lie + +- **Finding A, passage 1, clause 5.3 first paragraph** (readers 1 and 2): the sentence both + readers quote was added by the change, with "The remainder is the findings the summary leaves + out." Text the change wrote. Blocks. +- **Finding B, passage 2, clause 6.1 "For a project's own Rule" paragraph** (readers 1, 4 and + 5): the diff does not touch this paragraph, and no reader naming it sets it against a + paragraph the diff changed (reader 5 sets it against section 1's governing principle and the + 6.1 table's project row, both unchanged). Not the change's text. Routed to + `acceptance/ROUTED.md`; does not block. + +Passages 3 to 8 are each named by one reader: recorded, not findings. Passages 4, 5 and 6 lie +in changed paragraphs but are single. + +## Points the rules did not settle + +1. **Finding B, readers 1 and 4: a passage named for a reference to a document not supplied.** + Both quote the URL sentence with its clause number, which `ACCEPTANCE.md` defines as naming a + passage, and the confusion each gives is that method clause 3.6 is in a document they were + not given. The rule that a remark that another document was not supplied is a note covers a + remark "not about a passage of the document"; these remarks are attached to a quoted + passage. Marked as the open-point rule says: an open point on whether a reader named a + passage counts the passage. Without readers 1 and 4, reader 5 alone names the paragraph and + there is no finding B. Finding A does not turn on this point. Reader 2's passage 4 is the + same case, single either way. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 4081aa4, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: one (finding A). + Not met. +- Passages named by two or more readers elsewhere: one (finding B), routed to + `acceptance/ROUTED.md`. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on finding A alone; every keyed item passed its threshold. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From d157143b7f4e6e36d62370de418ff7b69be5dce1 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:29:20 +0100 Subject: [PATCH 076/105] Detector 5.3: a fuller report elsewhere is not a remainder After run 63: the sentence on a report that only adds detail is a sentence of its own, with its subject stated: a fuller report written elsewhere, such as a page with a code snippet for each finding, only adds detail to findings the output already carries, is not a remainder, and need not be named. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 7 ++++++- DETECTOR-SPEC.md | 5 +++-- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9d6fc71..532b83d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -87,7 +87,12 @@ pass it. revised text, every keyed item within threshold: two readers could not find Message defined, so section [2](DETECTOR-SPEC.md#2-terminology)'s table gains a Message row; and two readers stopped at clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps)'s "anyone who exercises the clauses above", which now says anyone may grade a detector by - checking it against each clause of sections [4](DETECTOR-SPEC.md#4-authoring-rules) to [7](DETECTOR-SPEC.md#7-suppression). Accepted by run NNN. + checking it against each clause of sections [4](DETECTOR-SPEC.md#4-authoring-rules) to [7](DETECTOR-SPEC.md#7-suppression). + [Run 63](acceptance/runs/063-detector-route-rule-run-2.md) failed on one finding in the revised + text, every keyed item five of five: two readers needed two readings of clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran)'s + sentence on a report that only adds detail, which is now a sentence of its own with its subject + stated; three readers named clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation)'s paragraph on a project's own rule, which this + change does not touch and `acceptance/ROUTED.md` now lists. Accepted by run NNN. ### 1.0.0, 2026-09-08 diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index 06a4238..42369c6 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -160,8 +160,9 @@ any. Where the [Detector] writes fuller detail elsewhere, the invoked command's own output MUST carry both a usable summary and the location of the remainder. The remainder is the findings the summary leaves out. Where the output prints every finding, each with where it is and its -[Identifier], there is no remainder, and a report that only adds detail to those same findings, -such as a code snippet, need not be named in the output. +[Identifier], there is no remainder. A fuller report written elsewhere, such as a page with a +code snippet for each finding, only adds detail to findings the output already carries; it is +not a remainder, and the output need not name it. **Why**: method specification clause 8.2. A report the [Practitioner] has to go and find is a report that arrives after the decision it was meant to inform. From e764773ee29f700b869aa959f2b567e14d2a6681 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:31:16 +0100 Subject: [PATCH 077/105] Semantic detectors: the change reduced to what it changes The three documents are taken from next and the change re-applied as the obligation it proposes and nothing else: the kinds of detector in section 1 and the detector gloss, clause 3.2's strongest-kind obligation with its two grounds, the recording of the grounds and the naming of the detector and kind on the remediation docs, the section 3 map's 3.2 row, section 2's and clause 3.2's "of a kind that can express the Rule", clause 3.6's deferral, section 4's sign-off proviso, section 7's conforming defence, Appendix A, and the detector's clause 8.2 with its section 9 sentence and glossed terms. The rewrites runs 19 to 30 had prompted in untouched text are dropped; the passages are in acceptance/ROUTED.md. Headers read 1.2.0-dev and 1.1.0-dev, unpublished, with package.json to match; the changelog carries one Unreleased line per document naming those runs. The method key gains the 2d item, the 3.2 row and its wrong readings in the current key format, and question 2d asks for what the key requires; the detector key gains clause 8.2 and the reading that the kind changes a verdict. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 175 +++++---------------------- DETECTOR-SPEC.md | 92 ++++++--------- SPEC.md | 208 ++++++++++++++------------------- acceptance/detector/KEY.md | 13 ++- acceptance/method/KEY.md | 56 ++++++--- acceptance/method/QUESTIONS.md | 5 +- package.json | 2 +- 7 files changed, 214 insertions(+), 337 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 99dba54..1ae692c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,91 +16,29 @@ version entry with no cohort line has not been accepted and is not published. kind of detector that can express it, semantic, else syntactic, else textual, judged by the configuration it runs in; enabling an existing rule or tightening its thresholds is not writing one, and giving an existing rule a pattern of the project's own, or changing the pattern of one - the project wrote, is. A weaker - kind is allowed only where the stronger cannot express the rule or the language has no - detector of that kind that accepts bespoke rules; a stronger configuration of a detector the - project runs counts, and one only a hosted service or licence tier provides does not. A - detector accepts bespoke rules when the project can add its own through a documented extension - point, and a rule its extension points cannot reach is one it cannot express. The - remediation docs of a rule the practitioner writes MUST name its detector and kind, and for - each step down from semantic the reason, with the stronger detectors considered, MUST be - recorded there and in the description of the commit that adds or moves the rule or first - records the reason. The practitioner judges it; a project MAY require the owner's sign-off by - recording that requirement where it records its calibrations, the sign-off then recorded with - the reason, which section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) and the section [3](SPEC.md#3-the-method) map note. Clause [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation)'s list of what the - remediation docs state defers to what clause [3.2](SPEC.md#32-build-the-net) adds. Section [1](SPEC.md#1-terminology) - defines textual, syntactic and semantic by whether a rule can ask the type of any expression - and what any name refers to across the whole program, naming mypy as semantic only with - `check_untyped_defs`, since by default it does not check the bodies of unannotated functions. - Section [2](SPEC.md#2-when-the-method-applies) and clause [3.2](SPEC.md#32-build-the-net) send a class that no extensible detector of a kind the language - has can express within clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s bounds to a bespoke detector or a toolchain gap, not to - "no pattern exists". Section - [7](SPEC.md#7-conformance)'s conforming defence carries the kind, and Appendix A carries the requirement. Prompted by - project rules hosted in ast-grep, where the language already had a semantic host, that decided - by names: they reported unrelated calls of the same name, and clause [3.2](SPEC.md#32-build-the-net) said any detector - qualified whatever its category. The method question set gains load-bearing statement 2d, which - `ACCEPTANCE.md` now lists, and `index.md`, `README.md`, `PRIMER.md` and two register pages - defer to the clause. After run 19, clarity edits: clause [3.2](SPEC.md#32-build-the-net) sets out the two grounds - for a weaker kind as a numbered pair, calls them grounds throughout, as Appendix A now does, and - says the extension-points rule in two sentences; section [2](SPEC.md#2-when-the-method-applies) says what independent means instead - of pointing at clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s vocabulary; clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class) says what the level of a class is; clause [3.3](SPEC.md#33-prove-the-net-by-making-the-rule-fire)'s - Part B says an instance left behind an inline ignore comment, an ignore entry or a baseline is - not a narrowing, since the code still carries the hazard, but a suppression for clauses [3.4](SPEC.md#34-sweep-the-codebase-then-fix-every-instance) and - [3.5](SPEC.md#35-enforce-permanently-and-block) and section [4](SPEC.md#4-authority-which-decisions-belong-to-whom); and clause [3.4](SPEC.md#34-sweep-the-codebase-then-fix-every-instance) says plainly that a baseline is rare. After run 21, clarity - edits: section [2](SPEC.md#2-when-the-method-applies) names the order of the steps, since three readers stopped at its MUST - sentence; clause [3.2](SPEC.md#32-build-the-net) states both grounds for a weaker kind in the sentence that allows one, since - no reader named them, and says the extension-points rule in short sentences; clause [3.3](SPEC.md#33-prove-the-net-by-making-the-rule-fire)'s Part - B says an inline ignore is not judged under it, and Part C says the red run may report a - fixture and the green run does not; section [4](SPEC.md#4-authority-which-decisions-belong-to-whom)'s lead-in names the owner, and its bullet on the - next wider rule, with the section [3](SPEC.md#3-the-method) map's row, says the owner decides unless the practitioner - has written and confirmed why the hazard cannot arise there. After run 23, clarity edits: clause - [3.2](SPEC.md#32-build-the-net)'s record sentence names the order of kinds, both grounds and the two places first, since - no reader stated all three; clause [3.2](SPEC.md#32-build-the-net) asks the second ground as two questions of each - stronger detector; clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s lower bound says an exact-value rule matches one thing and - states the test of the wider rule's rejection in short sentences, without calling it a - narrowing; section [1](SPEC.md#1-terminology)'s Narrowing and Suppression entries say an exclusion of code that carries - the hazard is a suppression, not a narrowing; and the Owner entry says the default owner is the - human who instructed the practitioner, or, where an agent gave the instruction, the human who - instructed that agent, and takes section [4](SPEC.md#4-authority-which-decisions-belong-to-whom)'s - decisions. Each after readers of run 23 missed or named the passage. After run 25, clarity - edits: clause [3.2](SPEC.md#32-build-the-net)'s record sentence states both grounds in its main clause, not in brackets, - and its paragraph on the second ground says in fewer sentences when the language has a - detector of the stronger kind and when it accepts bespoke rules; clause [3.5](SPEC.md#35-enforce-permanently-and-block) says what a - bundled defence is and that a project installing somebody else's toolchain counts, without the - undefined term consuming project; clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s paragraph on independent techniques is shorter - and says plainly that a run of the rule is never one; and clause [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation)'s versioning sentence - says a rule's documentation is versioned with the rule. Each after readers of run 25 missed or - named the passage. After run 27, clarity edits: clause [3.2](SPEC.md#32-build-the-net) states both grounds for a weaker - kind in the sentence straight after the strongest-kind sentence, and its record paragraph names - the two places, both required, before what is recorded and which commit carries it; section - [7](SPEC.md#7-conformance)'s conforming defence satisfies each clause in full, and its summary of clause [3.5](SPEC.md#35-enforce-permanently-and-block) includes - permanence and the recorded decision behind every suppression; section [2](SPEC.md#2-when-the-method-applies) and clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class) each say - what a technique is there, and clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s paragraph on independent techniques is three short - paragraphs, saying that a run of the rule belongs to the proof or the sweep. Each after readers - of run 27 missed or named the passage. After run 29, clarity edits: section [2](SPEC.md#2-when-the-method-applies)'s sentence that - no pattern exists names two independent ways of writing the rule that were tried, in place of - "techniques", and says a way of writing the rule is not a technique, which clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class) defines; - the MUST it carries is reworded, not changed. Clause [3.2](SPEC.md#32-build-the-net)'s paragraph on the two kinds of gap - gives each kind a sentence of its own and names, for a missing mechanism, the substitutes - clauses [3.3](SPEC.md#33-prove-the-net-by-making-the-rule-fire) and [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) already allow, each only under the condition its clause sets and only for - what it stands in for: the smoke check for the fixture-based test clause [3.3](SPEC.md#33-prove-the-net-by-making-the-rule-fire)'s test-first - paragraph asks for, so that Part A's red run and Part C's proof on fixture code are still owed, - and the anchor for a resolver; it says every other obligation of those clauses still applies, - so nothing is relaxed; and clause [3.5](SPEC.md#35-enforce-permanently-and-block)'s paragraph on a bundled - defence says in shorter sentences that, where a mechanism gap stops the toolchain running its - own entry point, the demonstration is made through another project's entry point and owed - there in full, and that clause [3.2](SPEC.md#32-build-the-net)'s substitutes give no way round it. Each after two readers of - run 29 named the passage. Clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class) says a technique is a way of searching for instances - wherever the specification uses the word, after one reader of run 29 named run 25's open - passage there, and section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) calls suppressions and baselines practices, not techniques, to - follow it. Section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) says an exception exists only where the owner, not "a human", has - discussed, agreed and documented it, as the owner's terminology entry and the section's own - list already say, after a reader of run 29 answered 2b with "a human" and not the owner. Clause [3.2](SPEC.md#32-build-the-net) then says that the - smoke check may be the run that makes Part A's red run, on the originating instance, and Part - C's proof, on fixture code, as clause [3.3](SPEC.md#33-prove-the-net-by-making-the-rule-fire) sets neither proof's means, and clause [3.5](SPEC.md#35-enforce-permanently-and-block)'s summary - of clause [3.2](SPEC.md#32-build-the-net) gives its substitutes under the conditions clause [3.2](SPEC.md#32-build-the-net) sets, after an - independent review found "still owed" could be read as leaving a rule with no pattern unprovable. - Accepted by run NNN. Minor version when released. + the project wrote, is. A weaker kind is allowed only on two grounds, that the stronger cannot + express the rule or that the language has no detector of that kind which accepts bespoke + rules; a stronger configuration of a detector the project runs counts, and one only a hosted + service or licence tier provides does not. The remediation docs of a rule the practitioner + writes MUST name its detector and kind, and where the kind is weaker than semantic the grounds + MUST be recorded there and in the description of the commit that adds or moves the rule or + first records them, naming the stronger detectors considered. The practitioner judges the + grounds; a project MAY require the owner's sign-off by recording that requirement where it + records its calibrations, which section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) and the section [3](SPEC.md#3-the-method) map note. Clause [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation)'s + list of what the remediation docs state defers to what clause [3.2](SPEC.md#32-build-the-net) adds. Section + [1](SPEC.md#1-terminology) defines textual, syntactic and semantic by whether a rule can ask the type of any + expression and what any name refers to across the whole program, naming mypy as semantic only + with `check_untyped_defs`. Section [2](SPEC.md#2-when-the-method-applies) and clause [3.2](SPEC.md#32-build-the-net) send a class that no extensible + detector of a kind the language has can express within clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s bounds to a bespoke + detector or a toolchain gap, not to "no pattern exists". Section [7](SPEC.md#7-conformance)'s conforming defence + carries the kind, and Appendix A carries the requirement. Prompted by project rules hosted in + ast-grep, where the language already had a semantic host, that decided by names: they reported + unrelated calls of the same name, and clause [3.2](SPEC.md#32-build-the-net) said any detector qualified whatever its + category. The method question set gains load-bearing statement 2d. Runs 19 to 30 were made on + an earlier text of this change under the earlier acceptance rules and failed on passages the + change had not touched, as `acceptance/CHANGES.md` records; the change was then reduced to + the text above, and the rewrites those runs prompted elsewhere in the document are listed in + `acceptance/ROUTED.md` for changes of their own. Accepted by run NNN. ### 1.1.0, 2026-10-02 @@ -164,65 +102,16 @@ pass it. - New obligation, clause [8.2](DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades): a verdict on a detector MUST name the kind, semantic, syntactic or textual, of each configuration it grades, as the detector has it without any hosted service - or licence tier. It is not a level of conformance and does not change the verdict; it - is stated because method clause [3.2](SPEC.md#32-build-the-net) now requires a rule to be written in the strongest kind of - detector that can express it, and a verdict of conforming alone presents a textual detector as - an equal host. Section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) still adds no obligation to the method a practitioner + or licence tier. It is not a level of conformance and does not change the verdict; it is + stated because method clause [3.2](SPEC.md#32-build-the-net) now requires a rule to be written in the strongest kind + of detector that can express it, and a verdict of conforming alone presents a textual detector + as an equal host. Section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) still adds no obligation to the method a practitioner follows and names this one as binding whoever gives a verdict on a detector, a practitioner - included when they give one, and section [2](DETECTOR-SPEC.md#2-terminology)'s gloss carries the three terms. After run 20, changed obligation, clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran): the - usable summary gives each finding in it with its location and identifier, and the location of - the rest is owed only where the summary leaves a finding out, so output that prints every - finding conforms though extra detail goes to a file it does not name; the old text owed that - location unconditionally, run 20's five readers graded the fixture's [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) Partial on it, and the - key's intended Yes already assumed the new reading. Clarity edits: clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) says it asks - for the mechanism and that missing documentation for a bundled rule fails clause [6.3](DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), as - clause [6.4](DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation)'s Why now says too, and says plainly why a path in the message is not enough; - clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) says the project writes the detecting rule in the detector, of the kind clause - [4.1](DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) requires, and that the detector need not remove or forbid the route; clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) says - what a gap is, that a detector declaring a version records it there, and that conformance is - judged on evidence with or without a declaration. After run 22, clarity edits: section [8](DETECTOR-SPEC.md#8-conformance) - says beside the definition of conformance that it is judged on evidence and a declaration is - optional, and clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) says plainly which declaration claims conformance; clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) opens by - saying the detector need not remove or forbid the route and says what detectable means in short - sentences; clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) says the summary need not list every finding; clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) says - plainly why a path in the message is not enough, and section [2](DETECTOR-SPEC.md#2-terminology)'s gloss defines the message. - Each after readers of run 22 missed or named the passage. After run 24, clarity edits: clause - [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)'s sentence that either thing alone satisfies the clause adds that neither requires the - detector to remove or forbid the route, and the clause is in shorter sentences; clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps)'s - sentence that a detector may be graded conforming on evidence says a declaration is optional, - and its paragraph is in short sentences; section [8](DETECTOR-SPEC.md#8-conformance)'s paragraph on the kind says one - detector can be graded in more than one configuration and that one needing a hosted service or - licence tier does not count. Each after readers of run 24 missed or named the passage. After - run 26, clarity edits: clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)'s obligation sentence itself says the detector need not - remove or forbid the route, and its definition of detectable is one sentence; section [8](DETECTOR-SPEC.md#8-conformance)'s - paragraph on the kind is shorter, says what the kind is, and says the duty is on whoever gives - the verdict, not on the detector. Each after readers of run 26 missed or named the passage. - After run 28, clarity edits: clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) says after the two things that neither requires removing - or forbidding the route, and says in a paragraph of its own what detectable means, that the - project writes the detecting rule, which works only where the detector lets it see every use, and what a - documented check is; clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) says that for a project's own rule printing the identifier - unaltered is everything it asks of the detector and the resolving is the toolchain's, without - needing the other specifications; section [8](DETECTOR-SPEC.md#8-conformance)'s paragraph on the kind says what a - configuration is and, in a paragraph of its own, that the duty is not one of the MUSTs - conformance is judged on and that a verdict leaving the kind out still stands on conformance. - Each after readers of run 28 missed or named the passage. After run 30, the MUST to name the - kind moves, unchanged, from the body of section [8](DETECTOR-SPEC.md#8-conformance), beside the conformance test, to a clause - [8.2](DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades) of its own, with a Why; the paragraph after it says, in new words, that it binds whoever - gives the verdict, anyone grading under clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) and a maintainer grading their own detector - included, and asks nothing of the detector itself, nor of a declaration under clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps), which - is the maintainer's claim and not a verdict; section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) names the clause and says it - binds a practitioner who gives a verdict. Clarity edits: clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) says, route by route, what - a project's own rule must be able to match for the route to be detectable that way: for an - ignore comment or a per-line directive each place it is written, for a generated baseline each - of its entries; clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) says a gap is a MUST the detector does not satisfy, that a - declaration is the maintainer's claim, and that a detector with no declaration claims nothing. - Each after two or more readers of run 30 named the passage. Clause [6.2](DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) gains a paragraph - saying that resolution reads the documentation clause [6.3](DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) requires to ship with the rule, at a - version tracked together with it, in the installed copy, that clause [6.2](DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) asks that the - lookup reach it without network access, and that a missing page fails clause [6.3](DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), not - clause [6.2](DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access), after two readers of run 30 answered - 2c from clause [6.2](DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) alone and left out the version. Accepted by run NNN. Minor version - when released. + included when they give one, and section [2](DETECTOR-SPEC.md#2-terminology)'s gloss carries the three terms and the + message. Runs 20, 22, 24, 26, 28 and 30 were made on an earlier text of this change under the + earlier acceptance rules and failed on passages the change had not touched, as + `acceptance/CHANGES.md` records; the rewrites those runs prompted elsewhere in the document + are listed in `acceptance/ROUTED.md` for changes of their own. Accepted by run NNN. ### 1.0.0, 2026-09-08 diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index 40d0a1a..b74dc5c 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -1,6 +1,6 @@ # Defence Before Fix: Detector Specification -**Version**: 1.0.0, published 2026-09-08 +**Version**: 1.1.0-dev, unpublished **Companion to**: [the method specification](SPEC.md), version 1.1.0, and [the toolchain specification](TOOLING-SPEC.md), version 0.2.0 **Author**: [Joseph Edmonds](https://ltscommerce.dev), [Edmonds Commerce](https://edmondscommerce.co.uk) **Coined**: 22 February 2026, in [the original article](https://ltscommerce.dev/articles/defence-before-fix-static-analysis) @@ -159,10 +159,7 @@ any. ### 5.3 The result MUST reach the practitioner in the output of the command they ran Where the [Detector] writes fuller detail elsewhere, the invoked command's own output MUST -still carry a usable summary, each finding in it with where it is and its [Identifier]. The -summary need not list every finding. If it leaves any finding out, the output MUST also say where -the rest can be read. Output that lists every finding meets this clause, even where extra detail, -such as a code snippet, is written only to a file the output does not name. +carry both a usable summary and the location of the remainder. **Why**: method specification clause 8.2. A report the [Practitioner] has to go and find is a report that arrives after the decision it was meant to inform. @@ -188,27 +185,23 @@ Two cases, by who wrote the [Rule]: | The project's own | The project | Print the [Identifier] unaltered; the lookup is the [Toolchain]'s to give | In both, the mechanism is keyed on **the [Identifier] exactly as printed**. A command, an index -file or a URL are all acceptable forms. This clause asks for the mechanism, not for every page. -Where the mechanism works but a [Bundled rule]'s [Identifier] has no documentation to resolve -to, that fails clause 6.3, not this clause; for a project's own [Rule], the missing -documentation is the project's to write. Each case in turn: +file or a URL are all acceptable forms. Each case in turn: **For a [Bundled rule]**, the [Detector] supplies the documentation and the lookup, and clauses 6.2 and 6.3 say where. The mechanism MUST resolve an [Identifier] presented alone, because the reader who needs it most has the [Identifier] from a log, a -ticket or a colleague and not the [Message]. Printing a documentation path in the [Message] is -not enough. It helps only a reader who has the [Message]; the [Identifier] alone still leads -nowhere. - -**For a project's own [Rule]**, the [Detector] cannot know the documentation, so this clause -asks of it only the half it can give: the [Identifier] printed unaltered, as clause 4.3 already -requires, with no transformation of it. For such a [Rule], that is everything this clause asks of -the [Detector]; the resolving itself is done by the project's [Toolchain], not by the [Detector]. -Where the project has chosen a URL as the [Identifier], the [Detector] prints that URL unaltered -like any other [Identifier]. What this clause forbids is a documentation path printed alongside a -shorter [Identifier] that cannot be looked up on its own. A [Detector] that offers the lookup as -well has gone further than this clause asks. The [toolchain specification](TOOLING-SPEC.md) states -the [Toolchain]'s side, in its clause 4.2; it is not needed to judge a [Detector] here. +ticket or a colleague and not the [Message]. A [Message] that carries its own +documentation path resolves that [Message], not the [Identifier]. + +**For a project's own [Rule]**, the [Detector] cannot know the documentation, so what +it owes is the half it can give: the [Identifier] printed unaltered under clause 4.3, and no +transformation of it. Where the [Identifier] is itself a URL, as method clause 3.6 allows, +the [Detector] prints that URL and has done what this clause asks. What this clause forbids is +a documentation path printed alongside a shorter [Identifier] that cannot be looked up on its +own. The lookup for such +a [Rule] is an obligation on the project's assembled [Toolchain], under clause 4.2 +of the [toolchain specification](TOOLING-SPEC.md), and a [Detector] that offers it as well has +gone further than this clause asks. **Why**: method specification clause 8.3 requires the [Identifier] to resolve without a human. An index keyed on anything else does not resolve it. This is the most commonly failed clause in this @@ -218,11 +211,6 @@ given. The lookup they can actually perform is the only one that counts. ### 6.2 Resolution of a bundled rule's identifier MUST work from the installed copy, without network access -Resolution reads the documentation that clause 6.3 requires to ship with the [Rule], at a version -tracked together with it, and the installed copy is where that documentation is. This clause asks -that the lookup reach what is there without network access. A page missing from that -documentation fails clause 6.3, not this clause. - **Why**: an [Agent] working offline, behind a proxy, or against a URL that has since moved needs the answer to be on disk. A dependency the project already installed is on disk by definition. A catalogue on the [Detector]'s website, however complete, is the right documentation in the wrong @@ -250,7 +238,7 @@ An automated check over every [Identifier] a [Bundled rule] can print, applying family list or pattern of clause 6.5 where one is used, that blocks the release when any of them lands on no page. -**Why**: clause 6.3 is the clause most easily believed to be satisfied whilst being broken, because +**Why**: clause 6.1 is the clause most easily believed to be satisfied whilst being broken, because the documentation is written by the same person who wrote the [Rule] and its absence is invisible from the inside. Where the [Detector] is shipped inside a [Toolchain], the [toolchain specification](TOOLING-SPEC.md)'s self-audit makes this check a MUST for the [Toolchain]; a @@ -280,21 +268,12 @@ nothing to run. ### 7.1 A detector MAY offer an inline suppression route, but MUST make it detectable or disableable -An inline ignore comment, a per-line directive and a generated [Baseline] are all inline -[Suppression] routes, and the [Detector] MAY ship them. For each route it ships, it need not -remove or forbid the route, but it MUST do at least one of two things: let the project switch the -route off in the [Detector]'s configuration, or make the route detectable. Either alone satisfies -this clause; neither requires the [Detector] to remove or forbid the route. A route that can be -neither switched off nor detected does not [Conform]. - -A route is detectable when the project can find every use of it, in one of two ways. The first is -a [Rule] the project writes for itself, under clause 4.1, and the [Detector] runs; the [Detector] -need not ship the [Rule] itself. That way works only where such a [Rule] can match every use of -the route: for an ignore comment or a per-line directive, each place it is written in the code; -for a generated [Baseline], each of its entries. A [Detector] whose [Rules] never see comments, for example, cannot make an -ignore comment detectable this way. The second is a mechanical check that the [Detector] itself documents, such as a -text search it documents for the route's spelling. Switching the route off or detecting it is what -lets a project that decides to forbid the route enforce that decision. +An inline ignore comment, a per-line directive and a generated [Baseline] are all such +routes. The [Detector] MAY ship them. For each one it MUST do at least one of two things, and either +alone satisfies this clause: make the route disableable by configuration; or make the route +detectable, by a [Rule] the project can write in the [Detector] itself or by a mechanical check +the [Detector] documents. Either lets a project which decides to forbid the route enforce that +decision. A route that can be neither switched off nor seen does not [Conform]. **Why**: the method specification's position is that [Suppression] is an [Owner] decision under its clause 3.4 and section 4, and an [Owner] cannot decide something they are @@ -317,8 +296,7 @@ minute at the moment they have the reason in mind, and it is the only thing that ## 8. Conformance -**A [Detector] [Conforms]** if it satisfies every MUST in sections 4 to 7. It is judged on the -evidence, and a declaration under clause 8.1 is optional. +**A [Detector] [Conforms]** if it satisfies every MUST in sections 4 to 7. Partial [Conformance] MUST NOT be described as [Conformance]. A [Detector] that satisfies most of this document is in a normal and respectable condition; it is not @@ -333,18 +311,18 @@ for a [Detector]. ### 8.1 A maintainer MAY declare the version of this document the detector conforms to, and the declaration records known gaps The declaration is machine-readable, in whatever form the [Detector]'s ecosystem uses to -record dependencies. The same declaration is where a known gap is recorded. A gap is a MUST in -sections 4 to 7 that the [Detector] does not satisfy. Once its maintainer knows of a gap, from the -[Detector]'s own checks or from a [Practitioner]'s report under the method's clause 3.2, a -[Detector] that declares a version MUST record the gap alongside that version, in the same file -as the declaration or in a file the declaration names. A declaration is the maintainer's claim. -One that records no gap claims [Conformance]; one that records a gap does not, and says where the -[Detector] stands. A [Detector] with no declaration claims nothing, either way. - -A [Detector] that predates this document, or whose maintainer has never read it, MAY be graded -[Conforming] on evidence: a declaration is optional. Anyone may grade a [Detector], by checking it -against the clauses above. The verdict rests on that check, not on what a declaration claims, -whether or not the [Detector] has one. A declaration tells the reader what the maintainer believes and what they know to be missing; +record dependencies. The same declaration is where a gap against this document is recorded once it is +known: a [Detector] that has learnt, from its own checks or from a [Practitioner]'s +report under the method's clause 3.2, that it fails a MUST in sections 4 to 7 MUST record that gap +alongside the version it declares, in the same file or one it names. A declaration with a non-empty +gap record is a statement of where the [Detector] stands and is not a claim of +[Conformance]. + +The declaration is optional. It is how a maintainer claims [Conformance]; it is not a +condition of it. A [Detector] that predates this document, or whose maintainer has never read +it, MAY be graded [Conforming] on evidence by anyone who exercises the clauses above against +it, and a verdict on any [Detector], declared or not, rests on that exercise and not on the +claim. A declaration tells the reader what the maintainer believes and what they know to be missing; the reader still runs the [Harness]. The shape is illustrative rather than prescribed. In a Composer manifest: diff --git a/SPEC.md b/SPEC.md index 0255dba..ebbc79e 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,6 +1,6 @@ # Defence Before Fix: Method Specification -**Version**: 1.1.0, published 2026-10-02 +**Version**: 1.2.0-dev, unpublished **Companion to**: [the detector specification](DETECTOR-SPEC.md), version 1.0.0, and [the toolchain specification](TOOLING-SPEC.md), version 0.2.0 **Author**: [Joseph Edmonds](https://ltscommerce.dev), [Edmonds Commerce](https://edmondscommerce.co.uk) **Coined**: 22 February 2026, in [the original article](https://ltscommerce.dev/articles/defence-before-fix-static-analysis) @@ -29,7 +29,7 @@ US spelling: **Defense Before Fix**. Abbreviated [DBF] throughout. ## Status of this document -This is version 1.1.0 of the specification. It is normative: section 3 defines the method, +This is version 1.2.0-dev of the specification. It is normative: section 3 defines the method, section 4 states who decides what, and section 7 defines what [Conformance] means and who may claim it. @@ -126,11 +126,11 @@ A report on code that does not carry the [Hazard]. #### Narrowing -Reducing a [Rule]'s scope so it stops reporting code that does not carry the [Hazard]. If the code left out does carry the [Hazard], the reduction is not a [Narrowing] but a [Suppression]. +Reducing a [Rule]'s scope so it stops reporting code that does not carry the [Hazard]. [Narrowing] that excludes code which does carry the [Hazard] is a [Suppression]. #### Suppression -Any means of preventing a [Rule] from reporting an [Instance] that carries the [Hazard]: an ignore comment, an ignore entry, a [Baseline], or a reduction of the [Rule]'s scope that excludes it. +Any means of preventing a [Rule] from reporting an [Instance] that carries the [Hazard]: an ignore comment, an ignore entry, a [Baseline], or a [Narrowing] that excludes it. #### Baseline @@ -162,7 +162,7 @@ A [Practitioner] that is an automated system. Everything said of a [Practitioner #### Owner -Whoever holds authority over a project's [Exceptions] and [Calibrations]. The [Owner] is always a human. The project decides who, and clause 8.7 requires the answer to be findable. Where the project has recorded nobody, the [Owner] is the human who instructed the [Practitioner]. If an [Agent] gave that instruction, the [Owner] is the first human up the chain of instructions that led to it. [Escalation] goes to the [Owner], who takes the decisions section 4 gives the [Owner]. One limit applies to that default. The default [Owner] is usually the person under the same deadline as the [Practitioner], so a decision they take under section 4 to keep an [Instance] unfixed, [Suppress], [Baseline] or remove a [Rule] MUST be recorded in the project record with its justification. That record MUST mark the decision as taken by a default [Owner] and state why nobody better placed was reachable, where the next reader can see who decided and why. Such a decision SHOULD be revisited by someone not under that deadline before the next release. A project that wants the separation section 4 describes names an [Owner]; the default exists so work is not blocked, not to certify that a second judgement was applied. +Whoever holds authority over a project's [Exceptions] and [Calibrations]. Always a human. The project decides who, and clause 8.7 requires the answer to be findable. Where the project has recorded nobody, the [Owner] is whoever instructed the [Practitioner], and [Escalation] goes there. One limit applies to that default. The default [Owner] is usually the person under the same deadline as the [Practitioner], so a decision they take under section 4 to keep an [Instance] unfixed, [Suppress], [Baseline] or remove a [Rule] MUST be recorded in the project record with its justification. That record MUST mark the decision as taken by a default [Owner] and state why nobody better placed was reachable, where the next reader can see who decided and why. Such a decision SHOULD be revisited by someone not under that deadline before the next release. A project that wants the separation section 4 describes names an [Owner]; the default exists so work is not blocked, not to certify that a second judgement was applied. #### Escalation @@ -182,9 +182,7 @@ Defence Before Fix applies to a [Defect] when that [Defect] can be attributed to idiom or configuration that a [Detector] can be made to recognise. When it can, building the [Defence] is not optional and MUST be attempted before the [Defect] is -fixed. The order is section 3's: attribute the [Defect] to a [Class], build the [Rule], prove -that it fires, run the [Sweep], then fix every [Instance]. This is the substance of the method -rather than a preliminary to it. +fixed. This is the substance of the method rather than a preliminary to it. When it cannot, the [Defect] is outside the method's scope and is fixed conventionally. Not every [Defect] is an [Instance] of a detectable pattern, and forcing a [Rule] where no pattern exists @@ -194,14 +192,12 @@ produces [Detectors] that fire on innocent code, which is worse than having no [ rather than deciding in advance whether it is expressible. Failing to write a [Rule] that satisfies clause 3.1's bounds is itself the evidence that the [Defect] is out of scope, and it is cheaper and more reliable than a judgement made before trying. The conclusion that no pattern exists MUST be -stated in one sentence, naming at least two independent ways of writing the [Rule] that were -tried, and recorded where the project's other decisions are enumerable under clause 8.7, not -only with the fix. Two ways of writing the [Rule] are independent when they would miss different -things. That is the test clause 3.1 applies to its search techniques, but a way of writing the -[Rule] is not a technique, which clause 3.1 defines as a way of searching for [Instances]. -That conclusion is the cheapest way out of the method and the only one that would otherwise -leave no trace; enumerating it is how an [Owner] sees an [Agent] routing around the method, and -naming the ways tried is what lets them judge whether the attempt was one. +stated in one sentence, naming at least two independent techniques tried at expressing the +[Rule], in the vocabulary clause 3.1 uses for a search, and recorded where the project's +other decisions are enumerable under clause 8.7, not only with the fix. It is the cheapest way +out of the method and the only one that would otherwise leave no trace; enumerating it is how an +[Owner] sees an [Agent] routing around the method, and naming the techniques is what lets +them judge whether the attempt was one. There are three ways out of the method and they are recorded differently. No pattern exists: the [Defect] is out of scope under this section, and only the sentence above is recorded. A pattern exists but the @@ -222,8 +218,8 @@ puts it in front of them. Reporting it in conversation does not satisfy this: th ends, and with it the only trace that the [Defect] was ever seen. **Why**: each of the three ways out costs the [Practitioner] something to take and leaves -something an [Owner] can read: a recorded sentence naming two ways of writing the [Rule] that -were tried, or a [Toolchain] gap. Deferral costs nothing, and without this record it would leave nothing, which makes it +something an [Owner] can read: a recorded sentence naming two techniques tried, or a [Toolchain] +gap. Deferral costs nothing, and without this record it would leave nothing, which makes it cheaper still than the sentence above and unseen from outside. A [Defect] mentioned once and never written down cannot be told apart from a [Defect] nobody found, so neither the [Class] nor the [Defence] it implies is ever reached. @@ -232,14 +228,14 @@ the [Defence] it implies is ever reached. Six clauses, in order. In brief, before the detail: -| Clause | The [Practitioner] MUST | The record shows | Goes to the [Owner] under section 4 | -| ------ | ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -| 3.1 | Name the [Class] the [Defect] belongs to, bounded both ways, after an independent search | The [Class], the [Hazard] sentence, the two search techniques, the next wider [Rule] not built, whether a [Runner] check pins the reported behaviour | Leaving the next wider [Rule] unbuilt, unless the [Hazard] cannot arise in the code it would add; leaving the [Rule] narrower than the search showed | -| 3.2 | Express the [Class] as a [Rule] in the strongest kind of [Detector] that can express it, never a test | The [Rule], its [Detector] and kind, why it is in a weaker kind than [Semantic] where it is, and any [Toolchain] gap that stopped a bespoke one | Whether a [Class] that could be defended is left undefended, and a weaker kind where the project requires the [Owner]'s sign-off | -| 3.3 | Make the [Rule] fire, on the [Instance] or a [Fixture], in a commit of its own | The red run, and the sentence behind every [Narrowing] | Any exclusion whose sentence cannot be written: that is [Suppression] | -| 3.4 | [Sweep] the whole codebase, record the count, then fix every [Instance] | The count, corroborated, and what was fixed by hand or by pattern | Any [Instance] left unfixed, and any [Baseline] | -| 3.5 | Make the [Rule] permanent and [Blocking] in the project's own checks | The green run through the project's entry point, and the recorded decision behind any [Suppression] | Removing or disabling the [Rule], and any [Suppression] added to it | -| 3.6 | Print a terse [Message] with a stable [Identifier] that resolves to documentation versioned with the [Rule] | The [Remediation docs] | Nothing | +| Clause | The [Practitioner] MUST | The record shows | Goes to the [Owner] under section 4 | +| ------ | ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| 3.1 | Name the [Class] the [Defect] belongs to, bounded both ways, after an independent search | The [Class], the [Hazard] sentence, the two search techniques, the next wider [Rule] not built, whether a [Runner] check pins the reported behaviour | Whether the next wider [Rule] is built, and whether the [Rule] stays narrower than the search showed | +| 3.2 | Express the [Class] as a [Rule] in the strongest kind of [Detector] that can express it, never a test | The [Rule], its [Detector] and kind, why it is in a weaker kind than [Semantic] where it is, and any [Toolchain] gap that stopped a bespoke one | Whether a [Class] that could be defended is left undefended, and a weaker kind where the project requires the [Owner]'s sign-off | +| 3.3 | Make the [Rule] fire, on the [Instance] or a [Fixture], in a commit of its own | The red run, and the sentence behind every [Narrowing] | Any exclusion whose sentence cannot be written: that is [Suppression] | +| 3.4 | [Sweep] the whole codebase, record the count, then fix every [Instance] | The count, corroborated, and what was fixed by hand or by pattern | Any [Instance] left unfixed, and any [Baseline] | +| 3.5 | Make the [Rule] permanent and [Blocking] in the project's own checks | The green run through the project's entry point, and the recorded decision behind any [Suppression] | Removing or disabling the [Rule], and any [Suppression] added to it | +| 3.6 | Print a terse [Message] with a stable [Identifier] that resolves to documentation versioned with the [Rule] | The [Remediation docs] | Nothing | The fourth column is what an [Agent] or other [Practitioner] MUST NOT decide alone, clause by clause; section 4 states the full list and governs where the two differ. @@ -319,23 +315,23 @@ Two bounds apply, and both are checkable during the work rather than matters of too narrowly and MUST be widened, unless the independent search below has found no other [Instance] and the [Practitioner] records that the [Class] is genuinely singular today. A [Rule] that matches exactly one thing is an [Instance] [Detector] wearing a [Rule]'s clothes. - A [Rule] drawn to the exact value or name found in the [Defect] also matches exactly one thing, - however it is written. Where it stays because the search found nothing wider, the record MUST also name the - next wider [Rule], the one with the [Instance]'s distinguishing detail dropped, and state why it - was rejected. The reason is one sentence: why the [Hazard] cannot arise in the code the wider - [Rule] would add. A search confirms it, as clause 3.3 requires for the sentence behind a - [Narrowing]. Where the [Practitioner] is not sure, the decision goes to the [Owner] under section 4. + A [Rule] drawn to the exact value or name found in the [Defect] is the same [Narrowing] however + it is spelt, and where it stays because the search found nothing wider, the record MUST also name the next + wider [Rule], the one with the [Instance]'s distinguishing detail dropped, and state why it was + rejected. That rejection is tested as a [Narrowing] is under clause 3.3: a sentence stating why the + [Hazard] cannot arise in the code the wider [Rule] would add, confirmed by search, with uncertainty + routed upwards under section 4. - **Upper bound.** If the [Rule] matches code that does not carry the [Hazard], the [Class] is drawn too broadly and MUST be narrowed. [False positives] destroy a [Defence]'s credibility faster than a missing [Rule] does. One report on code that does not carry the [Hazard] is enough; there is no tolerated rate. -**The bounds say how broad the [Class] is. Its level is a separate question: does the [Class] -name the failure the report describes, or the mechanism in the code that caused it?** -Where the [Defect] was reported as a behaviour, such as a check that reported clean having run -nothing, or a process that stopped before its work, a [Detector] usually cannot read that failure. -The [Class] it can read sits one level lower: the mechanism of this [Instance], not the failure -the report opens with. That [Class] is the right one to build, because no [Detector] reads "this check still produces +**Both bounds measure breadth within one level; the level itself is a separate question, answered +against the report.** +Where the [Defect] was reported as a behaviour, a check that reported clean having run nothing, a +process that stopped before its work, the [Class] a [Detector] can read usually sits one level +below that [Hazard]: the mechanism of this [Instance], not the failure the report opens with. +That [Class] is the right one to build, because no [Detector] reads "this check still produces its signal". It is not the whole answer. The record MUST state whether the reported behaviour is also pinned by a check a [Runner] executes, under section 6, either one added with this remediation or an existing one named, and where neither is offered, why not. A [Class] drawn at the mechanism with @@ -359,24 +355,22 @@ second technique that turns up new [Instances] means a third is owed; a second t new means the search is done. That is a test the [Practitioner] can apply and record without a threshold, and it is the standard clause 3.4 refers back to. -A technique, here and wherever this specification uses the word, is a way of searching the -codebase for [Instances]. Two techniques are -independent when they would miss different things. A text search for the token and a reading of -the code that consumes the value are independent, and asking someone who knows the system is a -third. Two text searches for two spellings of the same token are one technique, because they miss -the same things. For a [Class] defined by how code is spelt, though, searching for each other -common spelling of the idiom is a technique of its own, because there the spelling is the [Class] -itself. - -A technique counts only if it does not run the [Detector] under proof in any configuration and -would exist unchanged had the [Rule] never been written, because this search exists to check the -[Rule]. A run of the [Rule], however wide, is therefore never a technique. Such a run belongs to -the proof of clause 3.3 or the [Sweep] of clause 3.4, and a clean one shows nothing about whether -this search is saturated. - +Two techniques are independent when they would miss different things. A text search for the token +and a reading of the code paths that consume the value are independent; two text searches for two +spellings of the same token are one technique; asking someone who knows the system is a third. For a +[Class] defined by how code is spelt, writing the idiom the other ways it is commonly spelt and searching +the codebase for each of them is a technique, because a search for one spelling misses every other; +whether the [Rule] catches those spellings is the confirmation step, not the technique. The +[Rule] itself is never one of the techniques, because the search exists to check the [Rule]. +A clean run of the [Rule] is therefore not evidence of saturation either: "the [Rule] found nothing +the reading had missed" is one technique and a [Rule] run, not two techniques, and the stopping +criterion cannot be applied to it. The record MUST name both techniques, chosen before the [Rule] is run against the codebase, and -MUST state what each found that the other could not have checked. Where a later technique shows -that two earlier checks shared a blind spot, they were one technique, and a third is owed. +MUST state what each found that the other could not have checked. A technique counts only if it does +not invoke the [Detector] under proof in any configuration and would exist unchanged had the +[Rule] never been written; a [Sweep]-shaped run of the [Rule], however comprehensive it looks, is +never one of the two. Where a later technique shows that two earlier checks shared a blind spot, they +were one technique, and a third is owed. That turns the single-[Instance] case into something checkable rather than a matter of taste. If an independent search finds [Instances] the [Rule] missed, the [Class] was drawn too narrowly and the [Rule] @@ -472,24 +466,13 @@ A gap is recorded in the project record where the [Toolchain] defines one, as th then the only requirement is that somebody deciding what to invest in the [Toolchain] would find it. A gap recorded where nobody looks has been forgotten with extra steps. -There are two kinds of gap and they lead to different work. The first is the [Toolchain] gap -of the paragraphs above: the language has no extensible [Detector] of a kind that can express the -[Rule] within clause 3.1's bounds, and no bespoke one is practical, so the [Class] waits for one. -The second is a mechanism gap: the project's [Toolchain] lacks a mechanism the -[toolchain specification](TOOLING-SPEC.md) requires, such as a proving harness or an [Identifier] -resolver. A mechanism gap does not put the [Class] out of reach. The [Practitioner] still builds -the [Rule] and records the mechanism gap alongside it. Where the missing mechanism would have -served clause 3.3 or 3.6, the [Practitioner] uses a substitute only where that clause allows one, -and only for what it stands in for. Clause 3.3's acceptance or smoke check that demonstrates the -[Rule] firing is allowed only where -a [Detector] makes [Fixture]-based testing impractical, and it stands in only for the -[Fixture]-based test that clause's test-first paragraph asks for. Part A's red run, and Part C's -proof using [Fixture] code where the pattern is absent, are still owed, and the smoke check may -be the run that makes either, run on the code that proof names: the originating [Instance] for -Part A, [Fixture] code for Part C. Clause 3.6's [Identifier] -in a form that can be followed without a resolver, such as an anchor in a documentation file, -stands in only for the resolver. Every other obligation of those clauses still applies in full. -A mechanism gap that is also a breach of the [toolchain specification](TOOLING-SPEC.md)'s +There are two kinds of gap and they lead to different work. The language having no extensible +[Detector] of a kind that can express the [Rule] within clause 3.1's bounds, and no practical +bespoke one, is the gap above, and the [Class] waits for one. The project's [Toolchain] +lacking a mechanism the [toolchain specification](TOOLING-SPEC.md) requires, such as a proving harness or an +[Identifier] resolver, is a mechanism gap, and it does not put the [Class] out of reach: the +[Practitioner] builds the [Rule], uses the substitutes clauses 3.3 and 3.6 already allow, and records +the mechanism gap alongside it. A mechanism gap that is also a breach of the [toolchain specification](TOOLING-SPEC.md)'s own obligations, such as a [Toolchain] that cannot run its [Defences] on its own source, is recorded against the [Toolchain]'s claim under section 7 as well, because that is the fact its [Owner] needs. Where that record lives is the [Toolchain]'s own [Conformance] declaration, under the @@ -545,11 +528,6 @@ what must survive the merge is not. **Part B: [Narrowing].** -**An inline ignore comment, an ignore entry or a [Baseline] left over an [Instance] that carries the [Hazard] is not judged under this clause.** -Leaving an [Instance] that carries the [Hazard] in place behind one is not a [Narrowing], whatever -note accompanies it. It is a [Suppression] of the [Instance]. Clauses 3.4 and 3.5 and section 4 -govern it, and it does not fail this clause. - **The decision comes first, and it turns on one sentence.** Can the [Practitioner] write down why the [Hazard] cannot arise in the code being excluded? If they can, the exclusion is a [Narrowing], it is theirs to make, and they record that sentence. If they cannot, or are not sure, the exclusion @@ -638,10 +616,9 @@ Where a [Detector] makes [Fixture]-based testing impractical, an acceptance or s demonstrates the [Rule] firing is an acceptable substitute. [Fixtures] are part of the [Rule] and are versioned with it, under the co-location requirement in -clause 3.6. A [Fixture] is input to the [Rule]'s test, not part of the code the [Defence] enforces. -The red run of Part A may report it; that is the proof. The [Sweep] excludes it under clause 3.4, -and the green run under clause 3.5 does not report it as an [Instance]. So there is no circularity in a [Fixture] -that deliberately fails the [Rule] it proves. Writing the [Fixture] is the [Practitioner]'s own decision under section 4, not something +clause 3.6. A [Fixture] is input to the [Rule]'s test, not part of the code the [Defence] enforces: +the [Sweep] excludes it under clause 3.4, and the enforced run does not report it, so there is no +circularity in a [Fixture] that deliberately fails the [Rule] it proves. Writing the [Fixture] is the [Practitioner]'s own decision under section 4, not something to refer upwards. Where they sit within the repository is the project's business. **Absent a convention, put them @@ -715,8 +692,8 @@ been seen. "All examined" with nothing behind it is a claim, not a record. stops only when the next [Instance] needs a decision they lack the authority to make, and section 4 says which decisions those are; a hundred [Instances], or a thousand, is work rather than a decision. A [Baseline] of the existing [Instances], so that the [Rule] blocks only new ones, MAY -be adopted where fixing every [Instance] is genuinely unfeasible. That is rare: fixing every -[Instance] becomes unfeasible only when the change it needs is very large indeed. **No threshold is given here deliberately.** Any figure would be +be adopted where fixing every [Instance] is genuinely unfeasible, and the scale at which that +becomes true is tremendous. **No threshold is given here deliberately.** Any figure would be calibrated to the [Toolchain] of the moment rather than to anything durable, and the honest measure is the size of the change rather than the hours it would take, since the hours depend entirely on what is doing the work. @@ -769,17 +746,14 @@ that invocation covers, and see the [Rule] reported there. A [Toolchain] that wr [Detectors] in its own command is asking for that command to be used, and a [Practitioner] who bypasses it has proven the [Rule] and not the [Defence]. -A [Bundled defence] is a [Defence] that a shipped [Toolchain] enables by default in every -project that installs it from somebody else. Its enforcement can be demonstrated through the -entry point of any project the [Toolchain] is genuinely installed into, whether the [Toolchain]'s -own or somebody else's, since that is where the [Rule] will be enforced. A project created for the -purpose, with the [Toolchain] installed into it from the source under test, is such a project, so -a [Toolchain] with no consumer yet is not excused. Where a mechanism gap, as clause 3.2 defines -it, stops the [Toolchain] running its own entry point on its own source, the demonstration is made -through the entry point of another project the [Toolchain] is genuinely installed into, and is -owed there in full. Clause 3.2 lets the [Practitioner] use the substitutes it names for clauses -3.3 and 3.6, under the conditions it sets; it allows none for this clause. -The demonstration here is through an entry point or it is not made. +For a [Bundled defence], the project whose entry point demonstrates it is any project the +[Toolchain] is genuinely installed into, the [Consuming project] included, since that is +where the [Rule] will be enforced. A mechanism gap that stops the [Toolchain] running its own entry +point on its own source moves the demonstration to such a project; it does not weaken it. The +substitutes clause 3.2 allows, for proving under clause 3.3 and for resolution under clause 3.6, do +not extend to this clause: the demonstration here is through an entry point or it is not made. A +project created for the purpose, with the [Toolchain] installed into it from the source under +test, is such a project, so a [Toolchain] with no consumer yet is not excused. **Why**: the purpose is that the mistakes of the past become structurally impossible to repeat, and a [Warning] is not structure. A [Warning] is a suggestion, and suggestions decay under deadline @@ -790,8 +764,7 @@ pressure, which is the condition under which the original [Defect] was written. The failure [Message] MUST be terse, and it MUST carry a stable [Identifier] that resolves to [Remediation docs] shipped with the project. -That documentation MUST state three things, besides what clause 3.2 adds for a [Rule] the -[Practitioner] writes: what the [Rule] is about, why it exists, and how to +That documentation MUST state three things: what the [Rule] is about, why it exists, and how to fix a violation correctly using the project's preferred approach. A [Practitioner] who finds that an existing [Rule]'s printed [Identifier] does not resolve records @@ -808,9 +781,9 @@ something else. The documentation carries the reasoning and the remedy; it is re somebody who has decided to understand the [Rule], and it is maintained as the project's thinking develops. Anything that would grow over time belongs in the documentation. -**A [Rule]'s documentation MUST be versioned with the [Rule].** The two are one artefact: they -live in the same repository and are committed together, or in a library the project depends on at -a clearly tracked version. A [Rule] carries knowledge specific to the project it defends, and its +**A [Rule] and its documentation are one artefact and MUST be versioned as one.** They live in the +same repository and are committed together, or in a library the project depends on at a clearly +tracked version. A [Rule] carries knowledge specific to the project it defends, and its documentation is where most of that knowledge actually sits, so anything that lets the two drift apart destroys the value of both. The same applies to a [Rule]'s [Fixtures] under clause 3.3. @@ -854,19 +827,18 @@ they do not. 3.4). - The wording of the failure [Message] and the [Remediation docs] (clause 3.6). -**Decisions that MUST go to the [Owner]**, which the [Practitioner] never takes alone: +**Decisions that MUST go to whoever owns the codebase:** - Adopting a [Baseline], in whole or in part (clause 3.4). - Suppressing an [Instance], or removing or disabling an existing [Rule] (clauses 3.4 and 3.5). - Accepting a known [Instance] as unfixed for any reason. - Deciding that a [Class] will not be defended at all, where a [Rule] for it is achievable. -- Leaving unbuilt the next wider [Rule] that clause 3.1 requires the [Practitioner] to name, - unless they have written why the [Hazard] cannot arise in the code it would add and confirmed it - by search, as clause 3.1 requires. That no [Instance] of it exists today is not enough. It is a - reason not to widen unasked, not authority to leave the wider [Rule] out, and recording it as a - known gap does not make it the [Practitioner]'s decision. This bullet covers only that one wider - [Rule], already named on the record. It does not reopen a [Class] whose bounds a [Hazard] - sentence has settled. +- Leaving unbuilt the next wider [Rule] that clause 3.1 required the [Practitioner] to name, where + it was rejected because no [Instance] of it exists today rather than because the [Hazard] cannot + arise there. The absence of an [Instance] is the [Practitioner]'s reason not to widen unasked; it + is not authority to exclude the extension, and recording it as a known gap does not change whose + decision it is. This does not reopen a [Class] whose bounds a [Hazard] sentence has settled, and + it reaches no further than the one wider [Rule] already on the record. - Leaving a [Rule] narrower than an independent search under clause 3.1 showed it should be, where the reason is that the wider check is harder to build without [False positives] rather than that the [Hazard] is absent from what it would add. The [Instances] the search found are @@ -879,8 +851,8 @@ knowingly leave an [Instance] unfixed on its own authority, whatever the [Instan be. **The default position is that none of these are permitted at all.** [Suppressions], [Baselines] and -every other means of evading a [Defence] are not practices with a threshold governing their use; they are -[Exceptions] to the method, and the standing answer is no. An [Exception] exists only where the [Owner] has +every other means of evading a [Defence] are not techniques with a threshold governing their use; they are +[Exceptions] to the method, and the standing answer is no. An [Exception] exists only where a human has discussed it, agreed it and documented it for that project. Nothing an [Agent] concludes on its own creates one. @@ -977,13 +949,12 @@ evidence is gone and the opportunity closes with it. the verdict that they were rests on reproduction, as this section closes by requiring, not on the report. -**A [Defence] [Conforms]** if it satisfies each of clauses 3.1, 3.2, 3.3, 3.5 and 3.6 in full. -In brief: it is drawn to a [Class] within both bounds and not to the reported [Instance], it is -evaluated by reading code, in the kind of [Detector] clause 3.2 requires, it was proven to fire -against either the originating [Instance] or [Fixture] code, it is a permanent part of the -project's checks and fails rather than warns, every [Suppression] it carries is covered by a -recorded decision, and its [Message] is terse and resolves to [Remediation docs] versioned -alongside it. A [Defence] can [Conform] whilst the [Sweep] is incomplete; what that describes is a +**A [Defence] [Conforms]** if it satisfies clauses 3.1, 3.2, 3.3, 3.5 and 3.6: it is drawn to a +[Class] within both bounds and not to the reported [Instance], it is evaluated by reading code, in +the kind of [Detector] clause 3.2 requires, it was proven to fire against either the originating +[Instance] or [Fixture] code, it fails rather +than warns, and its [Message] is terse and resolves to [Remediation docs] versioned alongside +it. A [Defence] can [Conform] whilst the [Sweep] is incomplete; what that describes is a [Conforming] [Defence] over a codebase with known [Instances], which section 4 requires to be recorded, not a [Conforming] remediation. @@ -1121,7 +1092,7 @@ re-open it. ## 9. Citation -> Edmonds, Joseph. *Defence Before Fix*, version 1.1.0. First published 22 February 2026. +> Edmonds, Joseph. *Defence Before Fix*, version 1.2.0-dev. First published 22 February 2026. > ## Appendix A: Instructing an agent @@ -1220,6 +1191,7 @@ This appendix restates sections 2 to 4; where the two differ, the sections gover [conformance]: #conform [conforming]: #conform [conforms]: #conform +[consuming project]: TOOLING-SPEC.md#consuming-project [coverage]: #coverage [dbf]: #dbf [defect]: #defect diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 5b4d4aa..cf76401 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -21,9 +21,12 @@ no bullet making an assertion listed below as wrong, is a correct restatement. - Resolve a printed identifier to documentation; for bundled rules from the installed copy, offline, with the documentation shipped and versioned with the rule ([6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) to [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together)). - Make any inline suppression route detectable or disableable ([7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)). +- A verdict on the detector names its kind, semantic, syntactic or textual, in each + configuration it grades ([8.2](../../DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades)); the obligation is on whoever gives the verdict, as section + [9](../../DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) says, and a bullet that puts it on the detector, or on a verdict, names it. -Wrong if it asserts: the detector must enforce a project's suppression policy; the detector must -declare a version to conform; the detector must list a project's active defences or produce an +Wrong if it asserts: the detector must be semantic, or of any one kind, to conform; the detector +must enforce a project's suppression policy; the detector must declare a version to conform; the detector must list a project's active defences or produce an agent summary (those belong to the toolchain specification). A bullet that states what conformance means, or restates a SHOULD as advice, asserts no obligation and is neither correct nor wrong. @@ -77,7 +80,8 @@ Wrong readings the text invites, which count against the document on one reader: No grade above carries a condition on the reasoning: the accepted grade is correct whatever the reasoning, and a reader who adds a caveat about what the fixture does not say and still gives -the accepted grade is correct. +the accepted grade is correct. The fixture does not say what Lintomatic can see: a reader who +names its kind as unknown, or names none, is not marked on it. The two answers the question then asks for, each with the condition its reason must meet: @@ -92,6 +96,9 @@ Wrong readings the text invites, which count against the document on one reader: resolve, which has read the clause as admitting a proportion. - [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) graded No with reasoning that says the comment takes no reason and quotes [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) or [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason), which has read [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason)'s SHOULD into [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable). +- The section [8](../../DETECTOR-SPEC.md#8-conformance) verdict made to turn on Lintomatic's kind, with reasoning that a detector of some + kind cannot conform or that the kind is a level of conformance, which has read [8.2](../../DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades) as + changing the verdict: "the kind is not a level of Conformance and does not change the verdict". A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Yes because they missed the last paragraph of the fixture has misread the fixture, which is the reader's miss. diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 86cf8ff..fdb3d39 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -12,6 +12,10 @@ no bullet making an assertion listed below as wrong, is a correct restatement. - Attribute the defect to a class, bounded on both sides, with the hazard stated ([3.1](../../SPEC.md#31-attribute-the-defect-to-a-class)). - Express the class as a rule in a detector that reads code, never a test ([3.2](../../SPEC.md#32-build-the-net)). +- Write the rule in the strongest kind of detector that can express it, semantic, else + syntactic, else textual, name the detector and its kind on the rule's remediation docs, and + record the grounds where it is in a weaker kind ([3.2](../../SPEC.md#32-build-the-net)). A bullet that names the semantic + kind and does not mention the fallback names this obligation. - Prove the rule fires, against the instance or a fixture, in a commit of its own ([3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire)). - Sweep the whole codebase, record the count, then fix every instance ([3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance)). - Make the rule permanent and blocking in the project's checks ([3.5](../../SPEC.md#35-enforce-permanently-and-block)). @@ -25,7 +29,8 @@ no bullet making an assertion listed below as wrong, is a correct restatement. place (sections [2](../../SPEC.md#2-when-the-method-applies) and [6](../../SPEC.md#6-relationship-to-test-driven-development)). Wrong if it asserts: the practitioner fixes the reported instance first and defends afterwards; -a test may serve as the detector; an agent may baseline the remaining instances. A bullet that +a test may serve as the detector; an agent may baseline the remaining instances; only a semantic +detector may host a rule, or no weaker kind is ever allowed. A bullet that says the verdict on a remediation rests on reproduction (section [7](../../SPEC.md#7-conformance)) describes the reviewer, not the practitioner, and is neither correct nor wrong. @@ -59,26 +64,46 @@ Wrong readings the text invites, which count against the document on one reader: reproduction"; "A verdict on a remediation or a Defence MUST rest on reproduction, not on the report". "Reproduction" or "reproducing the runs" states element (2); the red and green runs need not be named. +- **2d.** Required: (1) the strongest kind of detector that can express the rule, in the order + semantic, else syntactic, else textual; (2) the two grounds for a weaker kind, both: the + stronger kind cannot express the rule, or the language has no detector of the stronger kind + that accepts bespoke rules; (3) the grounds, which of the two holds and why, recorded in both + places, on the rule's remediation docs and in the description of the commit that adds the rule, + moves it or first records the grounds. Wrong if it says any detector is as good as another, + that the semantic kind is only preferred and may be passed over for another reason, that a + weaker kind is never allowed, that the owner must always approve a weaker kind (the + practitioner judges it unless the project requires the owner's sign-off), or that the two + places are alternatives. Key quotations, [3.2](../../SPEC.md#32-build-the-net): "A Rule the Practitioner writes MUST be + written in the strongest kind of Detector that can express it: a Semantic one, else a + Syntactic one, else a Textual one."; "A weaker kind is allowed only where the stronger kind + cannot express the Rule, or where the language has no Detector of the stronger kind that + accepts bespoke Rules; those are the two grounds."; "Where that kind is weaker than Semantic, + the Practitioner MUST also record the grounds in two places, both of them: on the Rule's + Remediation docs, and in the description of a commit." Supporting: "On the Remediation docs of + a Rule they write, the Practitioner MUST name the Detector and its kind." Naming the stronger + detectors considered, and which commit it is, is explanation: an answer that gives it, or + leaves it out, is not marked on it. An answer that names the kinds in order without the words + "strongest" or "that can express it" states element (1). ## 3. Application: the ledger remediation -| Clause | Accepted grades | Condition on the reasoning | What the fixture supports | -| ----------------------------------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | None | Class, hazard and the next wider rule are named; Partial or No is the reading that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | -| [3.2](../../SPEC.md#32-build-the-net) | Yes | None | A rule in the project's static analyser, reading code, not a test | -| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes, Partial or No | For Partial or No: attributes the inline ignore to this clause, a fair reading of Part B | Commit B is red on the fixture and the originating instance and survives as its own commit | -| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Mentions the instance left unfixed, the inline ignore, or `Report::render()` | Five findings were recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority | -| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | Mentions the suppression, the inline ignore, or that no recorded decision covers it | The rule is permanent and blocking, but the suppression was not a recorded decision. [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | -| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | Mentions the wiki, or that the documentation is not versioned or co-located with the rule | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule | +| Clause | Accepted grades | Condition on the reasoning | What the fixture supports | +| ----------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | None | Class, hazard and the next wider rule are named; Partial or No is the reading that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | +| [3.2](../../SPEC.md#32-build-the-net) | Partial or No | Mentions that the detector, or its kind, is not named on the remediation docs, or that the wiki page names neither | A rule in the project's static analyser, reading code, not a test, but the wiki page of commit D names neither the detector nor its kind; [3.2](../../SPEC.md#32-build-the-net): "On the Remediation docs of a Rule they write, the Practitioner MUST name the Detector and its kind." The fixture does not say what the analyser can see, so no grounds for a weaker kind are shown to be owed | +| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes, Partial or No | For Partial or No: attributes the inline ignore to this clause, a fair reading of Part B | Commit B is red on the fixture and the originating instance and survives as its own commit | +| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Mentions the instance left unfixed, the inline ignore, or `Report::render()` | Five findings were recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority | +| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | Mentions the suppression, the inline ignore, or that no recorded decision covers it | The rule is permanent and blocking, but the suppression was not a recorded decision. [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | +| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | Mentions the wiki, or that the documentation is not versioned or co-located with the rule | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule | The three answers the question then asks for, each with the condition its reason must meet: - **The remediation under section [7](../../SPEC.md#7-conformance).** Accepted answer: does not conform. Condition: the reason - names [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), [3.5](../../SPEC.md#35-enforce-permanently-and-block) or [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the clauses not followed; other clauses named alongside are not + names [3.2](../../SPEC.md#32-build-the-net), [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), [3.5](../../SPEC.md#35-enforce-permanently-and-block) or [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the clauses not followed; other clauses named alongside are not marked. -- **The defence alone.** Accepted answer: does not conform. Condition: the reason names [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation), or - [3.5](../../SPEC.md#35-enforce-permanently-and-block)'s unrecorded suppression, which section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence also - requires to be absent; other clauses named alongside are not marked. +- **The defence alone.** Accepted answer: does not conform. Condition: the reason names [3.2](../../SPEC.md#32-build-the-net), + [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation), or [3.5](../../SPEC.md#35-enforce-permanently-and-block)'s unrecorded suppression, which section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence + also requires to be absent; other clauses named alongside are not marked. - **The decision section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) says was not the practitioner's.** Accepted answer: the inline ignore on `Report::render()`, which was the owner's decision. Condition: none; any other decision named alongside it is not marked. An answer naming only the decision to leave the wider rule unbuilt, @@ -92,6 +117,11 @@ Wrong readings the text invites, which count against the document on one reader: mention versioning or co-location. - The defence graded as conforming with reasoning that says only the sweep failed and that quotes section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence. +- [3.2](../../SPEC.md#32-build-the-net) graded Yes with reasoning that says the rule is in a detector that reads code, or not a + test, and does not mention the detector's kind or the naming of the detector on the + remediation docs, which has read [3.2](../../SPEC.md#32-build-the-net) without its naming sentence. A reader who grades [3.2](../../SPEC.md#32-build-the-net) + Partial or No only because no grounds for a weaker kind are recorded has the grade on a + ground the fixture does not show, and is correct on the grade. A reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the fixture, which is the reader's miss, whether or not the inline ignore was attributed to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) instead. diff --git a/acceptance/method/QUESTIONS.md b/acceptance/method/QUESTIONS.md index cd4383d..859f7be 100644 --- a/acceptance/method/QUESTIONS.md +++ b/acceptance/method/QUESTIONS.md @@ -19,8 +19,9 @@ the document you are relying on. authority? - 2c. What does it mean for a remediation to conform, and on what must a reviewer's verdict on it rest? -- 2d. Which kind of detector must a rule the practitioner writes be written in, when may a weaker - kind be used, and what must the practitioner record then? +- 2d. In which kind of detector must a rule the practitioner writes be written, and in what order + of preference; on what grounds may a weaker kind be used; and what must the practitioner + record then, and where? ## 3. Application diff --git a/package.json b/package.json index 740ca4d..62845c6 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@longtermsupport/defence-before-fix", - "version": "1.1.0", + "version": "1.2.0-dev", "description": "Defence Before Fix: the method and toolchain specifications. Documentation only.", "license": "CC-BY-4.0", "author": "Joseph Edmonds (https://ltscommerce.dev)", From 5852a8562abd0828b9519726ddcce3490d13b388 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:40:51 +0100 Subject: [PATCH 078/105] Routed findings: four detector rows closed by run 66, two toolchain rows from run 67 acceptance/ROUTED.md gains a Closed table, with the detector's clauses 7.1, 8.1, 5.3 and 6.1 rows closed by editor/detector-route-rule's run 66, notes run 66 on the clause 6.1 row on a project's own rule, and gains the toolchain's clause 6.2 heading and documented-check paragraph from run 67. Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 47 ++++++++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 21 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 55462f4..f995c21 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -15,27 +15,32 @@ untouched text of the change those runs tested, and are listed here so that they ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ---------------------------------------- | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | run 54 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | run 63 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56, 64 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59, 64 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59, 61, 64 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59, 61, 64 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | run 61 | `open` | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ---------------------------------------- | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | run 67 | `open` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 63, 66 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59, 64 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59, 61, 64 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59, 61, 64 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | run 61 | `open` | ## Closed -None yet. +A row moves here when a change that reworks its passage passes a cohort, naming that run, or +when the editor closes it with a reason. + +| Document | Clause | Passage | Raised by | Closed by | +| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ------------------------------------ | +| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | run 54 | `editor/detector-route-rule`, run 66 | From c78b76c413f7b3c6617621b00a1590f6c53a5389 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:40:57 +0100 Subject: [PATCH 079/105] Run 066: detector route rule cohort passed Every keyed item five of five and no finding in text the change wrote or altered. One finding in untouched text, clause 6.1's paragraph on a project's own rule, raised again and kept in acceptance/ROUTED.md. The changelog names runs 60, 63 and 66. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 5 +- .../runs/066-detector-route-rule-run-3.md | 145 ++++++++++++++++++ 2 files changed, 149 insertions(+), 1 deletion(-) create mode 100644 acceptance/runs/066-detector-route-rule-run-3.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 532b83d..9cc1a76 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -92,7 +92,10 @@ pass it. text, every keyed item five of five: two readers needed two readings of clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran)'s sentence on a report that only adds detail, which is now a sentence of its own with its subject stated; three readers named clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation)'s paragraph on a project's own rule, which this - change does not touch and `acceptance/ROUTED.md` now lists. Accepted by run NNN. + change does not touch and `acceptance/ROUTED.md` now lists. Accepted by + [run 66](acceptance/runs/066-detector-route-rule-run-3.md), every keyed item five of five; its + one finding is that same clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) paragraph, named again by three readers, which stays in + `acceptance/ROUTED.md` for a change of its own. ### 1.0.0, 2026-09-08 diff --git a/acceptance/runs/066-detector-route-rule-run-3.md b/acceptance/runs/066-detector-route-rule-run-3.md new file mode 100644 index 0000000..0d80045 --- /dev/null +++ b/acceptance/runs/066-detector-route-rule-run-3.md @@ -0,0 +1,145 @@ +# Run 66: DETECTOR-SPEC.md, detector route rule, third run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: not yet opened at the time of the run; it is to be opened from branch + `editor/detector-route-rule`. The readers read commit 44fdd9e. Change: clause 7.1's detectable + route is a bespoke rule the project writes for the detector under clause 4.1, with a paragraph + on what detectable means; clause 8.1's gap duty falls only on a detector that declares a + version; clause 5.3 says what the remainder is; clause 6.1 says it asks for the mechanism, a + missing page failing clause 6.3, and that a path in the message is not enough on its own; + section 2's table gains a Message row. Revised after run 63 in clause 5.3's sentence on a + fuller report elsewhere. +- **Run within the pull request**: 3 (runs 60 and 63 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `5f991fd47d3b8d092464ca41bfdfd9f5707647f46e3b274929a0c5a44f5a50c4`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at 44fdd9e, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Question 1: correct bullets per reader 6, 6, 6, 6 and 6, one for each line of the key's list +(4.1; 4.2; 4.3; 5.1 to 5.4; 6.1 to 6.3; 7.1). Readers 2 and 3 add to their sixth bullet that a +detector conforms when it meets every MUST in sections 4 to 7, which the key says states what +conformance means and is neither correct nor wrong; the bullet still names 7.1. Reader 4's +fourth bullet says every finding goes in the command's output, which the key says names 5.3 and +is not marked wrong for the plural. Reader 5's fifth bullet adds that a project rule's +identifier is printed unaltered, which is 4.3 restated inside the 6.1 to 6.3 bullet. No bullet +asserts anything the key marks wrong. + +Question 2: every reader states every required element of each item and quotes the document. +For 2a all five state that conformance is every MUST in sections 4 to 7 and that a detector +predating the document can conform, quoting section 8's definition or clause 8.1's "A Detector +that predates this document, or whose maintainer has never read it, MAY be graded Conforming on +evidence". For 2b all five name both things, disableable by configuration or detectable, and +that either is enough, quoting clause 7.1's "For each one it MUST do at least one of two +things, and either alone satisfies this clause". Readers 1, 3, 4 and 5 add that the bespoke-rule +way of detection is closed to a detector whose rules never see comments, which is the +document's own sentence and is not among the key's wrong assertions for 2b (removing or +forbidding the route, or requiring a reason). For 2c readers 1 to 4 state the installed copy, a +version tracked together and no network access, quoting the 6.2 and 6.3 headings; reader 5 +states all three in its own words (citing the headings without quotation marks) and quotes +clause 6.3's "Every [Identifier] a [Bundled rule] can print MUST resolve, under clause 6.1, to a +page in that shipped documentation", which quotes the document. + +Question 3: all five give every accepted grade. Readers 1 to 5 add caveats to 4.2 (the fixture +does not say `--only` prints finding lines) and 4.3 (no machine-readable format described); +reader 5 adds to 4.2 "If it does not, this is No", and reader 3 adds to its 6.3 No that the +grade would need revisiting if its reading of the fixture were wrong. Each gives the accepted +grade as its grade for the fixture as written, and the key says a reader who adds a caveat about +what the fixture does not say and still gives the accepted grade is correct. S8: all five say +Lintomatic does not conform, naming 6.3. 6.4: all five say it does not change the verdict, +because it is a SHOULD. + +### Misses + +None. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..44fdd9e adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------- | +| 1 | Clause 6.1, "For a project's own Rule" paragraph: "Where the [Identifier] is itself a URL, as method clause 3.6 allows, the [Detector] prints that URL and has done what this clause asks" and "The lookup for such a [Rule] is an obligation on the project's assembled [Toolchain], under clause 4.2 of the [toolchain specification]" (1, could not check the documents cited); "What this clause forbids is a documentation path printed alongside a shorter [Identifier] that cannot be looked up on its own" (3 and 4, what "shorter" is measured against) | 1, 3, 4 | no | +| 2 | Clause 4.3, first paragraph: "A prefix the [Detector] adds from the invoking directory or the [Rule]'s location is derived from the file path. A namespace the [Rule author] assigns once in configuration is not." (no reason given) | 2 | no | +| 3 | Clause 6.3, the "dangling one" paragraph: "The failure to guard against is not the absent document but the **dangling one**", set against the clause's first paragraph, "Every Identifier a Bundled rule can print MUST resolve, under clause 6.1, to a page in that shipped documentation"; whether the MUST covers absent documentation | 3 | no | +| 4 | Section 2, "These are additional."; what the terms are additional to | 3 | no | +| 5 | Clause 7.1, second paragraph: "That way is open only where the [Detector]'s [Rules] can see the route: a [Detector] whose [Rules] never see comments cannot make an ignore comment detectable this way"; what a bespoke rule can see | 4 | yes | +| 6 | Clause 7.2, first paragraph: "The [Detector] SHOULD NOT supply a default one, including each entry of a generated [Baseline]"; what "one" refers back to | 5 | no | +| 7 | Clause 8.1, second paragraph: "a gap a [Practitioner] finds is recorded under the method's clause 3.2 in any case", set against the same paragraph's "falls only on a [Detector] that has one"; where the record is kept with no declaration | 5 | yes | + +Notes, not passages: none. + +## Findings and where they lie + +- **Finding A, passage 1, clause 6.1 "For a project's own Rule" paragraph** (readers 1, 3 and + 4): the diff 9e914fe..44fdd9e changes the paragraph before it ("In both, ...") and the "For a + Bundled rule" paragraph, but not this one; no reader naming it sets it against a paragraph the + diff changed (reader 1 points at documents not supplied, readers 3 and 4 at the comparison in + "shorter"). Not the change's text. Routed to `acceptance/ROUTED.md`, where the paragraph's row + raised by run 63 is noted as raised again by run 66; does not block. + +Passages 2 to 7 are each named by one reader: recorded, not findings. Passages 5 and 7 lie in +changed paragraphs but are single. + +## Points the rules did not settle + +1. **Finding A, reader 1: a passage named for a reference to a document not supplied.** Reader 1 + quotes two sentences of the paragraph with the clause number, which `ACCEPTANCE.md` defines + as naming a passage, and its confusion is that method clause 3.6 and the toolchain + specification were not given. The rule that a remark that another document was not supplied + is a note covers a remark "not about a passage of the document"; this remark is attached to + quoted passages. Marked as the open-point rule says: an open point on whether a reader named + a passage counts the passage. The finding does not turn on this point, since readers 3 and 4 + name the same paragraph, and the finding is routed either way. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 9907492, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: none. Met. +- Passages named by two or more readers elsewhere: one (finding A), routed to + `acceptance/ROUTED.md`. + +## Findings applied + +None at the time of marking; finding A is for `acceptance/ROUTED.md`. + +## Verdict + +PASS. Every keyed item passed its threshold, no miss counts against the document, and the one +finding lies outside the text the change wrote or altered. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From f3ec69ebb2b2fc10fc592765ba20848acbdcb00a Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:47:43 +0100 Subject: [PATCH 080/105] Routed findings: rows from runs 19 to 30, two closed by defined-terms run 64 acceptance/ROUTED.md gains the passages runs 19 to 30 on the semantic-detectors change named in text that change no longer touches (the Narrowing entry, section 4's lead-in to the owner's decisions, clause 3.1's paragraph on independent techniques and the section 3 table's 3.1 row), notes those runs on the rows they raised again, and closes the method's two-bounds paragraph and bundled-defence paragraph by editor/defined-terms's run 64. Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index f995c21..c6c3d28 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -10,7 +10,7 @@ A finding is open until a change that reworks its passage passes a cohort, when that run and moves below the rule, or until the editor closes it with a reason recorded here. A finding raised again by a later run is noted on its line, not listed twice. The status column holds `open`, or the `editor/` branch or pull request number of the change that is -reworking the passage. The rows raised by runs 31 to 52 were raised under the earlier rules, on +reworking the passage. The rows raised by runs 19 to 52 were raised under the earlier rules, on untouched text of the change those runs tested, and are listed here so that they are not lost. ## Open @@ -25,22 +25,28 @@ untouched text of the change those runs tested, and are listed here so that they | `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | | `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | | `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | run 67 | `open` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 63, 66 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56, 64 | `open` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | | `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59, 64 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59, 61, 64 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59, 61, 64 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | run 61 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | +| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | ## Closed A row moves here when a change that reworks its passage passes a cohort, naming that run, or when the editor closes it with a reason. -| Document | Clause | Passage | Raised by | Closed by | -| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ------------------------------------ | -| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule`, run 66 | -| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule`, run 66 | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule`, run 66 | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | run 54 | `editor/detector-route-rule`, run 66 | +| Document | Clause | Passage | Raised by | Closed by | +| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------------------------------------ | +| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 20, 22, 24, 30, 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 20, 30, 39, 42 | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | runs 20, 54 | `editor/detector-route-rule`, run 66 | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on the two bounds | "Both bounds measure breadth within one level; the level itself is a separate question, answered against the report." | runs 19, 56 | `editor/defined-terms`, run 64 | +| `SPEC.md` | [3.5](../SPEC.md#35-enforce-permanently-and-block), the paragraph on a bundled defence | "For a Bundled defence, the project whose entry point demonstrates it is any project the Toolchain is genuinely installed into", and its substitutes sentence | runs 25, 29, 53, 59 | `editor/defined-terms`, run 64 | From befe52c1eb074a5878dd148ad9e3db3369700f2a Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:48:25 +0100 Subject: [PATCH 081/105] Semantic detectors: review findings applied Clause 3.6 defers to what clause 3.2 adds; clause 3.2 states the first ground as "cannot express the Rule within clause 3.1's bounds" everywhere, drops its repeated "can reach" sentence, says which commit carries the grounds for a rule already in place, and makes the sign-off's recording a MUST on the practitioner; section 7's conforming defence names the recording clause 3.2 requires; the three kinds are defined by what a Rule can ask; Appendix A's "outside one" is spelt out; detector clause 8.2 rests its hosted-service exclusion on method clause 3.2 rather than clause 5.4. The method key's 2d quotations follow, the 3.2 row carries no condition, question 2d asks in what order the kinds are tried, and the detector key marks a bullet putting clause 8.2 on the detector as wrong. The changelog names the runs by document. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 15 +++++----- DETECTOR-SPEC.md | 3 +- SPEC.md | 41 +++++++++++++++------------- acceptance/detector/KEY.md | 6 ++-- acceptance/method/KEY.md | 37 +++++++++++++------------ acceptance/method/QUESTIONS.md | 2 +- defence-before-fix-project-prompt.md | 4 +-- 7 files changed, 58 insertions(+), 50 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1ae692c..483c8ea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,11 +34,12 @@ version entry with no cohort line has not been accepted and is not published. carries the kind, and Appendix A carries the requirement. Prompted by project rules hosted in ast-grep, where the language already had a semantic host, that decided by names: they reported unrelated calls of the same name, and clause [3.2](SPEC.md#32-build-the-net) said any detector qualified whatever its - category. The method question set gains load-bearing statement 2d. Runs 19 to 30 were made on - an earlier text of this change under the earlier acceptance rules and failed on passages the - change had not touched, as `acceptance/CHANGES.md` records; the change was then reduced to - the text above, and the rewrites those runs prompted elsewhere in the document are listed in - `acceptance/ROUTED.md` for changes of their own. Accepted by run NNN. + category. The method question set gains load-bearing statement 2d. Runs 19, 21, 23, 25, 27 + and 29 were made on an earlier text of this change under the earlier acceptance rules: runs + 19 and 29 failed on statement 2d, and every run failed on passages the change had not + touched, as `acceptance/CHANGES.md` records; the change was then reduced to the text above, + and the passages those runs named in untouched text are listed in `acceptance/ROUTED.md` for + changes of their own. Accepted by run NNN. ### 1.1.0, 2026-10-02 @@ -110,8 +111,8 @@ pass it. included when they give one, and section [2](DETECTOR-SPEC.md#2-terminology)'s gloss carries the three terms and the message. Runs 20, 22, 24, 26, 28 and 30 were made on an earlier text of this change under the earlier acceptance rules and failed on passages the change had not touched, as - `acceptance/CHANGES.md` records; the rewrites those runs prompted elsewhere in the document - are listed in `acceptance/ROUTED.md` for changes of their own. Accepted by run NNN. + `acceptance/CHANGES.md` records; the passages those runs named in untouched text are listed + in `acceptance/ROUTED.md` for changes of their own. Accepted by run NNN. ### 1.0.0, 2026-09-08 diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index b74dc5c..40f0161 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -357,7 +357,8 @@ configuration the verdict grades. A configuration here is one way of setting the run, and one [Detector] can have several: ESLint, for example, is [Syntactic] run without type information and [Semantic] run with it. The kind is what a [Rule] can ask in that configuration, as section 2 glosses it. The kind named is the one the [Detector] has without a hosted service or -a licence tier, since under clause 5.4 a finding reported only there does not [Conform]. +a licence tier, the same configurations the method specification's clause 3.2 counts when it +asks for the strongest kind. This clause binds whoever gives the verdict, anyone grading a [Detector] under clause 8.1 included, and asks nothing of the [Detector] itself; a maintainer who gives a verdict on their diff --git a/SPEC.md b/SPEC.md index ebbc79e..1795fbd 100644 --- a/SPEC.md +++ b/SPEC.md @@ -78,11 +78,11 @@ A tool that **executes** code and reports what happened: a test [Runner], a comp #### Textual -Of a [Detector]: seeing the code only as text, as grep does. It sees how the code is spelt and nothing of its structure, so it cannot tell a call from a comment that names it. +Of a [Detector]: one in which a [Rule] can ask only how the code is spelt, as grep does. It sees the text and nothing of its structure, so it cannot tell a call from a comment that names it. #### Syntactic -Of a [Detector]: seeing the parse tree of the code, and at most some of the types of values, as ast-grep does. It sees the structure, but cannot always tell apart two calls of the same name on values of different types. +Of a [Detector]: one in which a [Rule] can ask about the parse tree of the code, and at most some of the types of values, as ast-grep does. It sees the structure, but cannot always tell apart two calls of the same name on values of different types. #### Semantic @@ -412,10 +412,10 @@ are examples rather than a permitted list, and a [Detector] is not disqualified **Qualifying is not the same as being the right host.** A [Rule] the [Practitioner] writes MUST be written in the strongest kind of [Detector] that can express it: a [Semantic] one, else a [Syntactic] one, else a [Textual] one. A weaker kind is allowed only where the stronger kind -cannot express the [Rule], or where the language has no [Detector] of the stronger kind that -accepts bespoke [Rules]; those are the two grounds. For example, the stronger kind cannot express -a [Rule] that reads a file the [Semantic] [Detector] does not parse. The kind is that of the -configuration the [Rule] runs in. Enabling an existing [Rule], or tightening its thresholds, is +cannot express the [Rule] within clause 3.1's bounds, or where the language has no [Detector] +of the stronger kind that accepts bespoke [Rules]; those are the two grounds. For example, the +stronger kind cannot express a [Rule] that reads a file the [Semantic] [Detector] does not +parse. The kind is that of the configuration the [Rule] runs in. Enabling an existing [Rule], or tightening its thresholds, is not writing one, and the off-the-shelf paragraph above covers it; giving an existing [Rule] a pattern of the project's own, or changing the pattern of a [Rule] the project wrote, is writing one. @@ -425,20 +425,20 @@ already runs one or the language's own [Detector] ecosystem offers one. A strong of a [Detector] the project runs counts; a [Detector] or configuration that only a hosted service or a licence tier provides does not. A [Detector] accepts bespoke [Rules] when the project can add its own [Rule] to it through a documented extension point, such as a plugin interface; it need not meet the rest -of the [detector specification](DETECTOR-SPEC.md). If no [Detector] of the stronger kind can reach -the construct the [Rule] is about, that is the first ground. +of the [detector specification](DETECTOR-SPEC.md). On the [Remediation docs] of a [Rule] they write, the [Practitioner] MUST name the [Detector] and its kind. Where that kind is weaker than [Semantic], the [Practitioner] MUST also record the grounds in two places, both of them: on the [Rule]'s [Remediation docs], and in the description -of a commit. That commit is the one that adds the [Rule], moves it to another [Detector], or first -records the grounds. What is recorded, in both places, is which of the two grounds holds at each +of a commit. That commit is the one that adds the [Rule] or moves it to another [Detector]; for +a [Rule] already in place when its grounds are first written down, it is the commit that writes +them. What is recorded, in both places, is which of the two grounds holds at each step down the order [Semantic], [Syntactic], [Textual], and why, naming the [Detectors] of the stronger kind that were considered. The [Practitioner] judges whether the grounds hold, as -section 4 gives them the choice of [Detector]; a -project MAY require its [Owner]'s sign-off on a weaker kind as well, by recording that -requirement where it records its [Calibrations]; the sign-off is then recorded with the grounds, -and where the project has recorded no such requirement none is needed. +section 4 gives them the choice of [Detector]. A project MAY require its [Owner]'s sign-off on +a weaker kind as well, by recording that requirement where it records its [Calibrations]; where +it has, the [Practitioner] MUST record the sign-off with the grounds, and where the project has +recorded no such requirement none is needed. **A worked example.** Take a [Rule] against calling a method named `context` on one type. A [Detector] that cannot see types reports every call named `context`, whatever type it is called @@ -764,7 +764,8 @@ pressure, which is the condition under which the original [Defect] was written. The failure [Message] MUST be terse, and it MUST carry a stable [Identifier] that resolves to [Remediation docs] shipped with the project. -That documentation MUST state three things: what the [Rule] is about, why it exists, and how to +That documentation MUST state three things, besides what clause 3.2 adds for a [Rule] the +[Practitioner] writes: what the [Rule] is about, why it exists, and how to fix a violation correctly using the project's preferred approach. A [Practitioner] who finds that an existing [Rule]'s printed [Identifier] does not resolve records @@ -949,9 +950,11 @@ evidence is gone and the opportunity closes with it. the verdict that they were rests on reproduction, as this section closes by requiring, not on the report. -**A [Defence] [Conforms]** if it satisfies clauses 3.1, 3.2, 3.3, 3.5 and 3.6: it is drawn to a +**A [Defence] [Conforms]** if it satisfies each of clauses 3.1, 3.2, 3.3, 3.5 and 3.6 in full. +In brief: it is drawn to a [Class] within both bounds and not to the reported [Instance], it is evaluated by reading code, in -the kind of [Detector] clause 3.2 requires, it was proven to fire against either the originating +the kind of [Detector] clause 3.2 requires and with the [Detector], its kind and any grounds +recorded as that clause requires, it was proven to fire against either the originating [Instance] or [Fixture] code, it fails rather than warns, and its [Message] is terse and resolves to [Remediation docs] versioned alongside it. A [Defence] can [Conform] whilst the [Sweep] is incomplete; what that describes is a @@ -1111,8 +1114,8 @@ This appendix restates sections 2 to 4; where the two differ, the sections gover > what any name refers to, across the whole program; failing that, one that sees syntax; and only > then one that sees text. Use a weaker one only where the stronger one cannot express the [Rule], > or the language has none that takes custom [Rules]: none the project runs, in any configuration -> it can use without a hosted service or a licence tier, and none its ecosystem offers outside -> one. Name the [Detector] and its kind in the [Rule]'s documentation. Where you use a weaker one, +> it can use without a hosted service or a licence tier, and none its ecosystem offers without +> one either. Name the [Detector] and its kind in the [Rule]'s documentation. Where you use a weaker one, > write down which of those two grounds holds, and why, naming the stronger [Detectors] you > considered, there and in the description of the commit that adds the [Rule], moves it to another > [Detector], or first records the grounds, and where the project requires its [Owner]'s sign-off diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index cf76401..446a17e 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -23,10 +23,12 @@ no bullet making an assertion listed below as wrong, is a correct restatement. - Make any inline suppression route detectable or disableable ([7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)). - A verdict on the detector names its kind, semantic, syntactic or textual, in each configuration it grades ([8.2](../../DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades)); the obligation is on whoever gives the verdict, as section - [9](../../DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) says, and a bullet that puts it on the detector, or on a verdict, names it. + [9](../../DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) says, and a bullet that says a verdict, or a grading, names the kind names it. Wrong if it asserts: the detector must be semantic, or of any one kind, to conform; the detector -must enforce a project's suppression policy; the detector must declare a version to conform; the detector must list a project's active defences or produce an +must itself state or declare its kind, which clause [8.2](../../DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades) asks of the verdict and not of the +detector; the detector must enforce a project's suppression policy; the detector must declare a +version to conform; the detector must list a project's active defences or produce an agent summary (those belong to the toolchain specification). A bullet that states what conformance means, or restates a SHOULD as advice, asserts no obligation and is neither correct nor wrong. diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index fdb3d39..7149a2d 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -14,8 +14,10 @@ no bullet making an assertion listed below as wrong, is a correct restatement. - Express the class as a rule in a detector that reads code, never a test ([3.2](../../SPEC.md#32-build-the-net)). - Write the rule in the strongest kind of detector that can express it, semantic, else syntactic, else textual, name the detector and its kind on the rule's remediation docs, and - record the grounds where it is in a weaker kind ([3.2](../../SPEC.md#32-build-the-net)). A bullet that names the semantic - kind and does not mention the fallback names this obligation. + record the grounds where it is in a weaker kind ([3.2](../../SPEC.md#32-build-the-net)). This line and the one above are + separate obligations of the same clause, and a bullet naming each counts once for each. A + bullet that says a rule goes in a semantic detector, without saying that a weaker kind is + never allowed, names this obligation. - Prove the rule fires, against the instance or a fixture, in a commit of its own ([3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire)). - Sweep the whole codebase, record the count, then fix every instance ([3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance)). - Make the rule permanent and blocking in the project's checks ([3.5](../../SPEC.md#35-enforce-permanently-and-block)). @@ -68,16 +70,15 @@ Wrong readings the text invites, which count against the document on one reader: semantic, else syntactic, else textual; (2) the two grounds for a weaker kind, both: the stronger kind cannot express the rule, or the language has no detector of the stronger kind that accepts bespoke rules; (3) the grounds, which of the two holds and why, recorded in both - places, on the rule's remediation docs and in the description of the commit that adds the rule, - moves it or first records the grounds. Wrong if it says any detector is as good as another, - that the semantic kind is only preferred and may be passed over for another reason, that a - weaker kind is never allowed, that the owner must always approve a weaker kind (the - practitioner judges it unless the project requires the owner's sign-off), or that the two - places are alternatives. Key quotations, [3.2](../../SPEC.md#32-build-the-net): "A Rule the Practitioner writes MUST be + places, on the rule's remediation docs and in the description of a commit. Wrong if it says + any detector is as good as another, that a weaker kind may be used for a reason other than + the two grounds, that a weaker kind is never allowed, that the owner must always approve a + weaker kind (the practitioner judges it unless the project requires the owner's sign-off), or + that the two places are alternatives. Key quotations, [3.2](../../SPEC.md#32-build-the-net): "A Rule the Practitioner writes MUST be written in the strongest kind of Detector that can express it: a Semantic one, else a Syntactic one, else a Textual one."; "A weaker kind is allowed only where the stronger kind - cannot express the Rule, or where the language has no Detector of the stronger kind that - accepts bespoke Rules; those are the two grounds."; "Where that kind is weaker than Semantic, + cannot express the Rule within clause [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class)'s bounds, or where the language has no Detector of + the stronger kind that accepts bespoke Rules; those are the two grounds."; "Where that kind is weaker than Semantic, the Practitioner MUST also record the grounds in two places, both of them: on the Rule's Remediation docs, and in the description of a commit." Supporting: "On the Remediation docs of a Rule they write, the Practitioner MUST name the Detector and its kind." Naming the stronger @@ -87,14 +88,14 @@ Wrong readings the text invites, which count against the document on one reader: ## 3. Application: the ledger remediation -| Clause | Accepted grades | Condition on the reasoning | What the fixture supports | -| ----------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | None | Class, hazard and the next wider rule are named; Partial or No is the reading that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | -| [3.2](../../SPEC.md#32-build-the-net) | Partial or No | Mentions that the detector, or its kind, is not named on the remediation docs, or that the wiki page names neither | A rule in the project's static analyser, reading code, not a test, but the wiki page of commit D names neither the detector nor its kind; [3.2](../../SPEC.md#32-build-the-net): "On the Remediation docs of a Rule they write, the Practitioner MUST name the Detector and its kind." The fixture does not say what the analyser can see, so no grounds for a weaker kind are shown to be owed | -| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes, Partial or No | For Partial or No: attributes the inline ignore to this clause, a fair reading of Part B | Commit B is red on the fixture and the originating instance and survives as its own commit | -| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Mentions the instance left unfixed, the inline ignore, or `Report::render()` | Five findings were recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority | -| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | Mentions the suppression, the inline ignore, or that no recorded decision covers it | The rule is permanent and blocking, but the suppression was not a recorded decision. [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | -| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | Mentions the wiki, or that the documentation is not versioned or co-located with the rule | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule | +| Clause | Accepted grades | Condition on the reasoning | What the fixture supports | +| ----------------------------------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | None | Class, hazard and the next wider rule are named; Partial or No is the reading that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | +| [3.2](../../SPEC.md#32-build-the-net) | Partial or No | None | A rule in the project's static analyser, reading code, not a test, but the wiki page of commit D names neither the detector nor its kind; [3.2](../../SPEC.md#32-build-the-net): "On the Remediation docs of a Rule they write, the Practitioner MUST name the Detector and its kind." The fixture does not say what the analyser can see, so no grounds for a weaker kind are shown to be owed | +| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes, Partial or No | For Partial or No: attributes the inline ignore to this clause, a fair reading of Part B | Commit B is red on the fixture and the originating instance and survives as its own commit | +| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Mentions the instance left unfixed, the inline ignore, or `Report::render()` | Five findings were recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority | +| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | Mentions the suppression, the inline ignore, or that no recorded decision covers it | The rule is permanent and blocking, but the suppression was not a recorded decision. [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | +| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | Mentions the wiki, or that the documentation is not versioned or co-located with the rule | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule | The three answers the question then asks for, each with the condition its reason must meet: diff --git a/acceptance/method/QUESTIONS.md b/acceptance/method/QUESTIONS.md index 859f7be..ba0efaa 100644 --- a/acceptance/method/QUESTIONS.md +++ b/acceptance/method/QUESTIONS.md @@ -20,7 +20,7 @@ the document you are relying on. - 2c. What does it mean for a remediation to conform, and on what must a reviewer's verdict on it rest? - 2d. In which kind of detector must a rule the practitioner writes be written, and in what order - of preference; on what grounds may a weaker kind be used; and what must the practitioner + are the kinds tried; on what grounds may a weaker kind be used; and what must the practitioner record then, and where? ## 3. Application diff --git a/defence-before-fix-project-prompt.md b/defence-before-fix-project-prompt.md index 84b5a88..f34cbe1 100644 --- a/defence-before-fix-project-prompt.md +++ b/defence-before-fix-project-prompt.md @@ -17,8 +17,8 @@ Put it in a Semantic Detector, one in which a Rule can ask the type of any expre what any name refers to, across the whole program; failing that, one that sees syntax; and only then one that sees text. Use a weaker one only where the stronger one cannot express the Rule, or the language has none that takes custom Rules: none the project runs, in any configuration -it can use without a hosted service or a licence tier, and none its ecosystem offers outside -one. Name the Detector and its kind in the Rule's documentation. Where you use a weaker one, +it can use without a hosted service or a licence tier, and none its ecosystem offers without +one either. Name the Detector and its kind in the Rule's documentation. Where you use a weaker one, write down which of those two grounds holds, and why, naming the stronger Detectors you considered, there and in the description of the commit that adds the Rule, moves it to another Detector, or first records the grounds, and where the project requires its Owner's sign-off From ae5f8dab06df43dd63f0a8b34d1fcf17940974db Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:58:29 +0100 Subject: [PATCH 082/105] Runs 060, 063 and 066: pull request 23 and the posted answer sheets Each record names pull request 23 and links the three comments carrying its evidence. Co-Authored-By: Claude Fable 5.1 --- acceptance/runs/060-detector-route-rule-run-1.md | 7 +++++-- acceptance/runs/063-detector-route-rule-run-2.md | 7 +++++-- acceptance/runs/066-detector-route-rule-run-3.md | 7 +++++-- 3 files changed, 15 insertions(+), 6 deletions(-) diff --git a/acceptance/runs/060-detector-route-rule-run-1.md b/acceptance/runs/060-detector-route-rule-run-1.md index 20317b4..1b11d48 100644 --- a/acceptance/runs/060-detector-route-rule-run-1.md +++ b/acceptance/runs/060-detector-route-rule-run-1.md @@ -1,7 +1,7 @@ # Run 60: DETECTOR-SPEC.md, detector route rule, first run - **Document**: DETECTOR-SPEC.md -- **Pull request**: not yet opened at the time of the run; it is to be opened from branch +- **Pull request**: [23](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23), opened after the run from branch `editor/detector-route-rule`. The readers read commit ed30e69. Change: clause 7.1's detectable route is a bespoke rule the project writes for the detector under clause 4.1, with a paragraph on what detectable means; clause 8.1's gap duty falls only on a detector that declares a @@ -135,4 +135,7 @@ FAIL, on findings A and B alone; every keyed item passed its threshold. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request in three comments: +[the command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124578731), +[the answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124580496) and +[the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124582261). diff --git a/acceptance/runs/063-detector-route-rule-run-2.md b/acceptance/runs/063-detector-route-rule-run-2.md index c3fa574..61604ee 100644 --- a/acceptance/runs/063-detector-route-rule-run-2.md +++ b/acceptance/runs/063-detector-route-rule-run-2.md @@ -1,7 +1,7 @@ # Run 63: DETECTOR-SPEC.md, detector route rule, second run - **Document**: DETECTOR-SPEC.md -- **Pull request**: not yet opened at the time of the run; it is to be opened from branch +- **Pull request**: [23](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23), opened after the run from branch `editor/detector-route-rule`. The readers read commit 4cbecad. Change: clause 7.1's detectable route is a bespoke rule the project writes for the detector under clause 4.1, with a paragraph on what detectable means; clause 8.1's gap duty falls only on a detector that declares a @@ -141,4 +141,7 @@ FAIL, on finding A alone; every keyed item passed its threshold. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request in three comments: +[the command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124582736), +[the answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124583990) and +[the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124585481). diff --git a/acceptance/runs/066-detector-route-rule-run-3.md b/acceptance/runs/066-detector-route-rule-run-3.md index 0d80045..72cc618 100644 --- a/acceptance/runs/066-detector-route-rule-run-3.md +++ b/acceptance/runs/066-detector-route-rule-run-3.md @@ -1,7 +1,7 @@ # Run 66: DETECTOR-SPEC.md, detector route rule, third run - **Document**: DETECTOR-SPEC.md -- **Pull request**: not yet opened at the time of the run; it is to be opened from branch +- **Pull request**: [23](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23), opened after the run from branch `editor/detector-route-rule`. The readers read commit 44fdd9e. Change: clause 7.1's detectable route is a bespoke rule the project writes for the detector under clause 4.1, with a paragraph on what detectable means; clause 8.1's gap duty falls only on a detector that declares a @@ -142,4 +142,7 @@ finding lies outside the text the change wrote or altered. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request in three comments: +[the command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124587014), +[the answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124587505) and +[the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124587875). From 06d26fc9d8cd574e95f8310bd3ce226b6e3a4f3b Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:01:16 +0100 Subject: [PATCH 083/105] Runs 070 and 071: semantic detectors, method passed, detector failed Run 70 on SPEC.md passed: every keyed item within threshold, no finding in text the change wrote or altered, and six findings in untouched text for acceptance/ROUTED.md. Run 71 on DETECTOR-SPEC.md failed: clause 5.3 missed by five readers and 6.1 by four, on text this branch has not yet taken from editor/detector-route-rule, and clause 8.2's reference to the method's clause 3.2 confused two readers; clauses 7.1 and 8.1 routed, both reworked on that branch. Co-Authored-By: Claude Fable 5.1 --- .../runs/070-semantic-detectors-run-13.md | 234 ++++++++++++++++++ .../runs/071-semantic-detectors-run-14.md | 170 +++++++++++++ 2 files changed, 404 insertions(+) create mode 100644 acceptance/runs/070-semantic-detectors-run-13.md create mode 100644 acceptance/runs/071-semantic-detectors-run-14.md diff --git a/acceptance/runs/070-semantic-detectors-run-13.md b/acceptance/runs/070-semantic-detectors-run-13.md new file mode 100644 index 0000000..b9febf3 --- /dev/null +++ b/acceptance/runs/070-semantic-detectors-run-13.md @@ -0,0 +1,234 @@ +# Run 70: SPEC.md, semantic detectors, thirteenth run + +- **Document**: SPEC.md + +- **Pull request**: 14, open on branch `editor/semantic-detectors`. The readers were given commit + befe52c. The change, read against its merge base with `next`, 9e914fe, adds to clause 3.2 the + obligation that a [Rule] the [Practitioner] writes is written in the strongest kind of + [Detector] that can express it, with the two grounds for a weaker kind, the naming of the + [Detector] and its kind on the [Remediation docs] and the recording of the grounds on the + [Remediation docs] and in a commit description, the owner's sign-off where the project requires + it, and a worked example; adds the Textual, Syntactic and Semantic terminology entries and a + sentence to the Detector entry; rewrites the section 3 map's 3.2 row; adds "of a kind that can + express the [Rule] within clause 3.1's bounds" to section 2's three-ways-out paragraph and to + clause 3.2's bespoke-detector and two-kinds-of-gap paragraphs; adds clause 3.6's deferral to + what clause 3.2 adds; adds the sign-off proviso to section 4's first list and dividing-line + paragraph; rewrites section 7's conforming-[Defence] sentence; extends Appendix A's "Write a + custom [Rule]" paragraph; adds three link definitions; and sets the version to 1.2.0-dev. The + question set gains item 2d for the new obligation; the key gains 2d and the 3.2 lines that go + with it. + +- **Cohort**: five readers, none reused from any earlier run; model `claude-haiku-5-5`, the + cohort's model named for the run before its readers started. Every reader reported + `claude-haiku-5-5`. + +- **Isolation**, as the editor supplied it and attests: each reader a separate `claude -p` + process, started through the `ccy-claude` wrapper (which only reattaches the session's + credential), in a fresh empty directory made by `mktemp -d` outside any repository, with the + packet on standard input; no tools, no settings, no project context, no saved session, and no + managed settings on the machine. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt's sha256 is `659a8d793ec9f5e94261703c695c94c17f3045a72f6988899c5734ac7ce59f9c`, + checked by the marker against the file; the editor states it is byte for byte + `acceptance/method/QUESTIONS.md`, `SPEC.md` and `acceptance/fixtures/remediation-ledger.md` at + befe52c, in that order, each after one `===== FILE: =====` line, one blank line between + files. + +- **Readers set aside**: none. Every answer text is non-empty, every reader's JSON has `is_error` + false and subtype `success`, and every reader reported the cohort's model. + +- **Run within the pull request**: 13. Runs 19 to 30 came before, on both documents, under the + earlier acceptance rules and on an earlier, larger text of the change; runs 19, 21, 23, 25, 27 + and 29 read this document. + +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + editor's marking instruction and its agent definition, having seen neither the drafting nor + the review. It marked against `ACCEPTANCE.md` and `acceptance/method/KEY.md` at befe52c, by the + Marking section and the key alone. No judgement was taken from the records of runs 19 to 30; + run 64 was read for the shape of a record only. + +## Marks + +| Reader | Q1 | 2a | 2b | 2c | 2d | 3.1 | 3.2 | 3.3 | 3.4 | 3.5 | 3.6 | Remediation | Defence | Section 4 | +| ------ | ------- | --- | --- | --- | --- | ------- | ------- | ------- | --- | --- | --- | ----------- | ------- | --------- | +| 1 | correct | ok | ok | ok | ok | Partial | Partial | Yes | No | No | No | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | No | Partial | Yes | No | No | No | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | No | Partial | Yes | No | No | No | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | Partial | Partial | Partial | No | No | No | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | Partial | Partial | miss | No | No | No | ok | ok | ok | + +One miss in the run: reader 5 on 3.3, graded Partial without the key's condition. Every other +cell gives the reader's grade, accepted, or "ok" for a correct answer. No answer matches a wrong +reading the key names. + +### Question 1 + +Each reader has at least four bullets naming obligations the key lists and none the key marks +wrong. + +- Reader 1: 3.1 (two bullets, counted once), 3.2's detector and its strongest kind with 3.3, + 3.4 with section 4, 3.5 with 3.6, the original defect fixed with a reproduction test and the + deferred defect recorded. +- Reader 2: 3.1 (two bullets, counted once), 3.2's strongest kind, 3.3, 3.4 with section 4, 3.5 + with 3.6. Its sixth bullet's "A verdict on the result rests on reproduction" describes the + reviewer, which the key says is neither correct nor wrong. +- Reader 3: 3.1 (two bullets, counted once), 3.2's strongest kind and never a test, 3.3, 3.4 with + section 4, 3.5 with 3.6 and the reproduction test. +- Reader 4: 3.1 (two bullets, counted once), 3.2's strongest kind, 3.3, 3.4 with section 4, 3.5 + with 3.6 and the reproduction test. +- Reader 5: 3.1 (three bullets, counted once), 3.2's strongest kind with 3.6's identifier, 3.3, + 3.4 with section 4, 3.5 with 3.6. + +No reader has the fix first, a test as the detector, an agent baselining, or a weaker kind never +allowed; readers 1, 2, 3 and 5 give the order semantic, syntactic, textual, which the key counts +as naming the strongest-kind line. Readers 3, 4 and 5 also assert, inside a bullet that names +clause 3.3, that an exclusion needs a written sentence or goes to the owner; see the points below. + +### Question 2 + +- **2a**, five of five. Each states the four steps, attribution, the rule, its proof and the + sweep, in that order with the fixing after the sweep, and each quotes clause 3.4's "The + [Practitioner] MUST run the [Rule] across the entire codebase and record the total [Instance] + count before fixing anything" (reader 4 from "record the total"). Reader 2 adds that the + document "does not place those steps in one consistent sequence", citing Appendix A's fix of + the original defect last; that is not an assertion the key marks wrong, and its own sequence + has the fix after the sweep. +- **2b**, five of five. Each names the owner, a human, and says an agent may not; each quotes + section 4's "An [Agent] MUST NOT [Baseline], suppress or knowingly leave an [Instance] unfixed + on its own authority, whatever the [Instance] count turns out to be." +- **2c**, five of five. Each states that all six clauses were followed and that the verdict rests + on reproduction. Readers 1, 2, 3 and 5 quote section 7's conforming-remediation sentence; + reader 4 states it in its own words and quotes section 7's "A verdict on a remediation or a + [Defence] MUST rest on reproduction, not on the report". +- **2d**, five of five. Each states the order semantic, else syntactic, else textual, quoting + clause 3.2's "A [Rule] the [Practitioner] writes MUST be written in the strongest kind of + [Detector] that can express it". Each quotes the two-grounds sentence, which states both + grounds; reader 3 also gives them in its own words. Each states that the grounds are + recorded in both places, the [Remediation docs] and a commit description: readers 1 and 3 in + their own words, readers 2 and 4 by "two places" with the quotation "on the [Rule]'s + [Remediation docs], and in the description of a commit", reader 5 by that quotation. No reader + says the places are alternatives, that a weaker kind is never allowed, or that the owner must + always approve; reader 1 says sign-off "applies only where the project requires it". + +### Question 3 + +- **3.1**: three Partial (readers 1, 4, 5), two No (readers 2, 3); the key accepts any grade. +- **3.2**: five Partial, each on the detector or its kind not named. No reader graded Yes. +- **3.3**: three Yes (readers 1, 2, 3), accepted without condition. Reader 4 Partial, attributing + the inline ignore on `Report::render` to Part B: correct. **Reader 5 Partial: a miss.** Its + reasoning is that the enforced run reports the [Fixture], which "3.3 says it must not", and + that the [Fixture]'s status after commit C is not explained; it does not attribute the inline + ignore to this clause, the key's condition for Partial or No. The reasoning paraphrases clause + 3.3 Part C ("the enforced run does not report it") and gives no quotation of the document. The + key names no wrong reading for 3.3, so this is a single miss, tolerated by the threshold. +- **3.4**: five No, each naming `Report::render()` left under the inline ignore. +- **3.5**: five No, each on the inline ignore no recorded decision covers. +- **3.6**: five No, each naming the wiki and the documentation not versioned with the rule. +- **Remediation**: five say it does not conform, each naming 3.4, 3.5 and 3.6 among the clauses + not followed. +- **Defence**: five say it does not conform, each naming 3.5's uncovered suppression and 3.6. +- **Section 4**: five name the inline ignore on `Report::render()`. All five also name, as a + second decision, leaving the wider rule unbuilt; not marked. + +## Passages named under question 4 + +| Passage | Clause | Readers | Text | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------ | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| "Attempt rather than pre-judge": "in the vocabulary clause 3.1 uses for a search" (reader 1); "naming at least two independent techniques tried at expressing the [Rule], in the vocabulary clause 3.1 uses for a search" (readers 2, 4) | 2 | three: 1, 2, 4 | Untouched. Readers 2 and 4 set it against clause 3.1's search techniques, untouched | +| The two-bounds paragraph: "Both bounds measure breadth within one level; the level itself is a separate question, answered against the report." (readers 1, 5) | 3.1 | two: 1, 5 | Untouched. Reader 5 sets it against the lower and upper bounds, untouched | +| The lower-bound list item: "That rejection is tested as a [Narrowing] is under clause 3.3" (reader 1); "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt." (reader 3) | 3.1 | two: 1, 3 | Untouched. Reader 1 sets it against clause 3.3's test, reader 3 against the Narrowing entry; both untouched | +| "A remediation MAY be delivered across several changes": "provided the [Rule] does not become [Blocking] until every [Instance] is fixed" (reader 2); "does not become [Blocking] until every [Instance] is fixed" (reader 4) | 4 | two: 2, 4 | Untouched. Reader 2 sets it against "What to do whilst waiting", reader 4 against Appendix A's "Make the [Rule] a permanent part" paragraph; both untouched | +| Appendix A's closing paragraph: "Only then fix the original [Defect]" (reader 2, against clause 3.4 and section 2); "Only then fix the original Defect." (reader 3, against clause 3.3's "the fix is committed after it"); "Only then fix the original [Defect] in the normal way, with a test that reproduces it." (reader 4) | App A | three: 2, 3, 4 | Untouched; the change altered only Appendix A's "Write a custom [Rule]" paragraph, which no reader names. The paragraphs set against it are untouched | +| Section 4's list item on leaving the next wider [Rule] unbuilt: "The absence of an Instance is the Practitioner's reason not to widen unasked." (reader 3); described as "§4 reserves that decision to the owner", set against clause 3.1's worked example (reader 2) | 4 | two: 3, and 2 (open point) | Untouched | +| "What to do whilst waiting": "the [Practitioner] … leaves the [Rule] unmerged rather than merged in a weakened form" and "the [Rule] MAY merge at full width with that case recorded as a known [Instance] awaiting the [Owner]" | 4 | one: 2 | Untouched | +| Clause 3.1's worked example: "was not built because most of the codebase's catches are legitimate and the [Owner] has not decided" | 3.1 | one: 2 | Untouched | +| Section 2's "When it can" paragraph: "before the [Defect] is fixed", and clause 3.4's opening sentence: "fix every [Instance]", both set against Appendix A's closing paragraph | 2, 3.4 | one: 2 | Untouched | +| Clause 3.2's recording paragraph: "which of the two grounds holds at each step down the order [Semantic], [Syntactic], [Textual], and why" | 3.2 | one: 2 | Written by the change | +| Clause 3.3 Part A, "The proof MUST survive as a commit of its own": "the fix is committed after it" | 3.3 | one: 3 | Untouched | +| Clause 3.4, "Fix them all.": "the scale at which that becomes true is tremendous." | 3.4 | one: 3 | Untouched | +| Appendix A's "Make the [Rule] a permanent part of the project's quality checks, failing rather than [Warning]." paragraph | App A | one: 4 | Untouched | +| Section 4's lead-in "Decisions that MUST go to whoever owns the codebase:", set against the Owner entry, "Whoever holds authority over a project's [Exceptions] and [Calibrations]" | 4, 1 | one: 5 | Untouched, and the Owner entry untouched | +| Clause 3.3 Part B, "A [Narrowing] reduces the [Practitioner]'s own work": "a [Narrowing] that excludes more code than the [Rule] still covers MUST be reported to the [Owner] as if it were a [Suppression]." | 3.3 | one: 5 | Untouched | + +Notes, not passages: none. Suggestions volunteered: none. + +## Findings and where they lie + +1. **Section 2, "Attempt rather than pre-judge"**, readers 1, 2 and 4. Untouched; the change's + words in section 2 are in the next paragraph, "There are three ways out", which no reader + names. Readers 2 and 4 set it against clause 3.1's search techniques, untouched. Routed to + `acceptance/ROUTED.md` (its existing row, raised again); does not block. +2. **Clause 3.1, the two-bounds paragraph**, readers 1 and 5. Untouched by this change, and set + against the bounds list items, untouched. Its row is closed by `editor/defined-terms` run 64, + whose rewrite is not in this branch; routed to `acceptance/ROUTED.md`, noted on that row as + raised again on the text before that rewrite; does not block. +3. **Clause 3.1, the lower-bound list item**, readers 1 and 3. Untouched; set against clause + 3.3's test and the Narrowing entry, both untouched. Routed to `acceptance/ROUTED.md` (its + existing row, raised again); does not block. +4. **Section 4, "A remediation MAY be delivered across several changes"**, readers 2 and 4. + Untouched; set against "What to do whilst waiting" and Appendix A's "Make the [Rule] a + permanent part" paragraph, both untouched. Routed to `acceptance/ROUTED.md`, on the existing + section 4 row that sets "MAY merge at full width" against this sentence; does not block. +5. **Appendix A, "Only then fix the original [Defect] in the normal way"**, readers 2, 3 and 4. + Untouched: each paragraph of Appendix A's block quote is a paragraph of its own, and the change + altered only "Write a custom [Rule]". Set against section 2's "When it can", clause 3.4's + opening sentence, clause 3.3 Part A's "The proof MUST survive" and section 4's several-changes + paragraph, all untouched. Routed to `acceptance/ROUTED.md` as a new row; does not block. +6. **Section 4, the list item on leaving the next wider [Rule] unbuilt**, reader 3, and reader 2 + by the open-point rule. Untouched; reader 2 sets it against clause 3.1's worked example, + untouched. Routed to `acceptance/ROUTED.md` (its existing row, raised again); does not block. + +No finding lies in text the change wrote or altered. Of the passages from one reader only, one +lies there: clause 3.2's recording paragraph, "at each step down the order" (reader 2). No reader +names the terminology entries for Textual, Syntactic or Semantic, the section 3 map's 3.2 row, +clause 3.2's other new paragraphs, clause 3.6's deferral, section 4's proviso or section 7's +conforming-[Defence] sentence. + +## Points the rules did not settle + +- **Whether reader 2 names section 4's next-wider list item.** Reader 2 quotes clause 3.1's worked + example and sets it against "§4 reserves that decision to the owner", a clause number with a + description that identifies the list item on leaving the next wider [Rule] unbuilt. Marked by + the open-point rule as naming the passage, which makes finding 6. It lies in untouched text + either way, so the verdict is the same. +- **Question 1 bullets on [Narrowing].** Readers 3, 4 and 5 each assert, in a bullet that also + names clause 3.3's proof, that an exclusion or narrowing needs a written sentence, or goes to + the owner (clause 3.3 Part B). That obligation is in neither of the key's lists. Not counted + either way; each bullet counts for its clause 3.3 content, and each reader has four or more + correct bullets without it. An open point for the key. +- **Considered, and settled by the rules.** The marker's agent definition says a miss whose + reasoning quotes the document is a finding whatever the count; `ACCEPTANCE.md` has no such + rule, and the marker marked by `ACCEPTANCE.md`. The one miss, reader 5 on 3.3, quotes nothing, + so the point does not arise in this run. + +## Process change + +Made under the process change of pull request 22 before it merged, at commit f3ec69e, named +before the readers started. f3ec69e is on `editor/acceptance-coherent`, one commit after +5852a85, which this branch merged; it changes only `acceptance/ROUTED.md`. `ACCEPTANCE.md`, +`acceptance/README.md`, `acceptance/runs/README.md` and `CONTRIBUTING.md` are identical at +f3ec69e and befe52c. + +## Findings applied + +None at the time of marking; what to change is the editor's decision. + +## Against the criteria + +- Load-bearing statements, every reader correct on every item: met; 2a, 2b, 2c and 2d five of + five. +- Restatement and application, all but one reader correct on each keyed item: met; 3.3 four of + five, every other item five of five. +- A miss the key says counts against the document: none. +- A passage found confusing by two or more readers, in text the change wrote or altered: none. +- A passage found confusing by two or more readers, elsewhere: six, section 2's "Attempt rather + than pre-judge", clause 3.1's two-bounds paragraph and lower-bound list item, section 4's + several-changes paragraph and next-wider list item, and Appendix A's closing paragraph, routed + to `acceptance/ROUTED.md`. +- From one reader only: nine passages, recorded. + +**Verdict: PASS.** + +## Answer sheets + +Answer sheets, prompt, command and marker instruction: LINK TO PULL REQUEST COMMENT diff --git a/acceptance/runs/071-semantic-detectors-run-14.md b/acceptance/runs/071-semantic-detectors-run-14.md new file mode 100644 index 0000000..de82038 --- /dev/null +++ b/acceptance/runs/071-semantic-detectors-run-14.md @@ -0,0 +1,170 @@ +# Run 71: DETECTOR-SPEC.md, semantic detectors, fourteenth run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: 14, open on branch `editor/semantic-detectors`. The readers read commit + befe52c. Change: a new clause 8.2, that a verdict on a detector names its kind, semantic, + syntactic or textual, in each configuration it grades; section 9's sentence on that + obligation; section 2's table rows for the three kinds and for Message. +- **Run within the pull request**: 14 (runs 19 to 30 and run 70 came before; runs 20, 22, 24, + 26, 28 and 30 read this document on an earlier, larger text of the change under the earlier + acceptance rules). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `d45ab8c1ff1dc2ecabd1bad36eca32c3b0858327c5079cd4b6aea2abcb4e5cc3`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at befe52c, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model; each sheet is that output's `result`. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | --- | ---- | --- | --- | --- | --- | --- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | miss | ok | miss | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | miss | ok | miss | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | miss | ok | miss | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | miss | ok | miss | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | miss | ok | ok | ok | ok | ok | ok | ok | + +Question 1: every reader's first six bullets name, one each, the key's lines for 4.1; 4.2; 4.3; +5.1 to 5.4; 6.1 to 6.3; and 7.1. Reader 2's sixth bullet adds 7.2 as a SHOULD and what +conformance means, and reader 4's sixth adds what conformance means; the key says such a +statement is neither correct nor wrong, and both bullets still name 7.1. Reader 3's seventh +bullet, on conformance, is not marked, since only the first six are. No reader names the 8.2 +line, which the count does not need. No bullet asserts anything the key marks wrong. + +Question 2: every reader states every required element of each item and quotes the document. +2a: all five state that conformance is every MUST in sections 4 to 7 and that a detector +predating the document can conform, quoting clause 8.1's "A Detector that predates this +document, or whose maintainer has never read it, MAY be graded Conforming on evidence" +(readers 2, 3 and 5 also quote section 8's definition). 2b: all five name both things, +disableable by configuration or detectable, and that either alone is enough, quoting clause +7.1's "For each one it MUST do at least one of two things, and either alone satisfies this +clause". 2c: all five state that the documentation ships with the rule or lives in the installed +copy, at a version tracked together, and resolves without network access, quoting the 6.2 and +6.3 headings; readers 2, 3 and 5 add that a website does not satisfy it, which is not among the +key's wrong assertions. + +Question 3: every reader gives the accepted grade on 4.1, 4.2, 4.3, 5.1, 5.2, 5.4, 6.2, 6.3 and +7.1, several with caveats about what the fixture does not say (4.2's output, 4.3's +machine-readable formats), which the key accepts. S8: all five say Lintomatic does not conform, +naming 6.3; all five note that the verdict cannot name Lintomatic's kind because the fixture +does not say it, and that this does not alter the conformance finding, which the key does not +mark. 6.4: all five say it does not change the verdict, because it is a SHOULD. + +### Misses + +The key's wrong readings for question 3 are 6.3 graded Partial on a proportion, 7.1 graded No +on the missing reason, and the section 8 verdict made to turn on the kind. No miss matches one. +None of the reasoning below quotes the document as `ACCEPTANCE.md` defines quoting (a run of six +or more consecutive words); each paraphrases it or quotes the fixture. + +- **5.3, readers 1 to 5, Partial (key: Yes).** Every reader reasons that the one-line output is a + usable summary but that the fixture never says the command's output gives the location of + `.lintomatic/report.html`, which they read clause 5.3's second sentence ("both a usable summary + and the location of the remainder") as requiring. Reader 1 adds, under question 4, that what + makes a summary "usable" decided its grade. Five of five: the item fails its threshold. +- **6.1, readers 1 to 4, Partial (key: Yes).** Each reasons that `explain` resolves an identifier + presented alone, but that the two bundled rules printing `No documentation declared` do not + resolve to documentation, so the mechanism fails for them; reader 4 says it counts the same + defect under 6.1 and 6.3. Reader 5 gives Yes. Four of five: the item fails its threshold. + +Both items fail as keyed items, which `ACCEPTANCE.md` says are fixed in the text that carries +the obligation wherever it lies; clauses 5.3 and 6.1 are not text the diff 9e914fe..befe52c +changes. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..befe52c adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------- | +| 1 | Clause 8.1, first paragraph: "a [Detector] that has learnt, from its own checks or from a [Practitioner]'s report under the method's clause 3.2, that it fails a MUST in sections 4 to 7 MUST record that gap alongside the version it declares" (readers 4 and 5; reader 1 a shorter quotation of the same sentence); whether the duty binds a detector that makes no declaration, set against passage 2 by readers 1 and 4, and against "the version it declares" within the sentence by reader 5 | 1, 4, 5 | no | +| 2 | Clause 8.1, second paragraph: "The declaration is optional" (readers 1 and 4) and "it is not a condition of it" (reader 4), set against passage 1 | 1, 4 | no | +| 3 | Clause 7.1, first paragraph: "or make the route detectable, by a [Rule] the project can write in the [Detector] itself" (readers 2 and 3, reader 3 with the rest of the sentence); whether that rule is clause 4.1's bespoke rule, added to configuration, or a change to the detector's own codebase | 2, 3 | no | +| 4 | Clause 8.2, first paragraph: "the same configurations the method specification's clause 3.2 counts when it asks for the strongest kind" (readers 2 and 3); which configurations are meant, the method specification not having been supplied. Reader 1 names "the method specification's clause 3.2" with clause 8.2 among terms it read as ordinary English for want of `SPEC.md` (point 2 below) | 1, 2, 3 | yes | +| 5 | Clause 5.3, first paragraph: "the invoked command's own output MUST carry both a usable summary and the location of the remainder"; what makes a summary usable | 1 | no | +| 6 | Section 2, introductory paragraph: "Terms from the method specification carry over unchanged", with [Class] and [Owner] pointing to `SPEC.md`, which reader 1 was not given (point 2 below) | 1 | no | + +Notes, not passages: none beyond what point 2 below counts as passages. + +## Findings and where they lie + +- **Finding A, passage 1, clause 8.1 first paragraph** (readers 1, 4 and 5): not changed by the + diff, and the paragraph the readers set it against (passage 2) is not changed either. Routed + to `acceptance/ROUTED.md`; does not block. +- **Finding B, passage 2, clause 8.1 second paragraph** (readers 1 and 4): not changed by the + diff, and set only against passage 1. Routed to `acceptance/ROUTED.md`; does not block. +- **Finding C, passage 3, clause 7.1 first paragraph** (readers 2 and 3): not changed by the diff, + and set against no changed paragraph. Routed to `acceptance/ROUTED.md`; does not block. +- **Finding D, passage 4, clause 8.2 first paragraph** (readers 2 and 3, and reader 1 under point + 2): the diff adds the whole of clause 8.2. Text the change wrote; blocks. + +`acceptance/ROUTED.md` at befe52c lists, under Closed, rows on the passages of findings A and B +(clause 8.1, "MUST record that gap alongside the version it declares" against "The declaration +is optional.") and of finding C (clause 7.1, "by a Rule the project can write in the Detector +itself"), closed by run 66 on `editor/detector-route-rule`, whose rework of those clauses is not +in the text these readers read. Where the routed findings are entered is the editor's decision. + +## Points the rules did not settle + +1. **Passage 4, readers 2 and 3: a passage or a note.** Each quotes a sentence of clause 8.2 with + the clause number and says it could not tell which configurations the sentence means, + because the method specification was not supplied. `ACCEPTANCE.md` makes a remark that + another document was not supplied a note only where it is "not about a passage of the + document"; these remarks are about what a quoted sentence means. Marked as passages, which + is also what the open-point rule says: an open point on whether a reader named a passage + counts the passage. Finding D turns on this point. +2. **Reader 1's third item: a passage or a note.** Its subject is that `SPEC.md` was not given, + which reads as a note, but it quotes section 2's "Terms from the method specification carry + over unchanged" (a run of eight words) and names "the method specification's clause 3.2" + with clause 8.2, whose first occurrence in 8.2 is the first paragraph. Counted, by the + open-point rule, as naming section 2's introductory paragraph (passage 6) and clause 8.2's + first paragraph (passage 4). Finding D does not turn on this point, since readers 2 and 3 + name the same paragraph. +3. **Readers 1 and 4 naming two paragraphs in one item.** Each quotes sentences from both the + first and the second paragraph of clause 8.1 and sets them against each other. Counted as + naming both paragraphs, by the open-point rule. Finding A stands on readers 4 and 5 without + reader 1; finding B depends on this point, and is routed either way. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit f3ec69e, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item except + 5.3 (none of five) and 6.1 (one of five). Not met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: one (finding D, + clause 8.2 first paragraph). Not met. +- Passages named by two or more readers elsewhere: three (findings A, B and C), for + `acceptance/ROUTED.md`. + +## Findings applied + +None at the time of marking. + +## Verdict + +FAIL. Question 3's items 5.3 and 6.1 fail the all-but-one threshold, and finding D lies in +clause 8.2's first paragraph, which the change wrote. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From 8defdae4aeb9e7feb7c0531ab2d5559d3bb1e195 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:01:19 +0100 Subject: [PATCH 084/105] Detector 5.3 and 8.1: the location owed only where the summary leaves a finding out After run 69, a defined-terms cohort on this document with this change merged in: clause 5.3's obligation owes the location of the remainder only where the summary leaves a finding out, so it no longer pulls against "need not name it"; clause 8.1 says a practitioner's record of a gap lives in their own project's record, and that a detector without a declaration can still be found to conform. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 14 ++++++++++---- DETECTOR-SPEC.md | 15 ++++++++------- 2 files changed, 18 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9cc1a76..11f482c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -92,10 +92,16 @@ pass it. text, every keyed item five of five: two readers needed two readings of clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran)'s sentence on a report that only adds detail, which is now a sentence of its own with its subject stated; three readers named clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation)'s paragraph on a project's own rule, which this - change does not touch and `acceptance/ROUTED.md` now lists. Accepted by - [run 66](acceptance/runs/066-detector-route-rule-run-3.md), every keyed item five of five; its - one finding is that same clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) paragraph, named again by three readers, which stays in - `acceptance/ROUTED.md` for a change of its own. + change does not touch and `acceptance/ROUTED.md` now lists. + [Run 66](acceptance/runs/066-detector-route-rule-run-3.md) passed, every keyed item five of + five, its one finding being that same clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) paragraph, named again by three readers, + which stays in `acceptance/ROUTED.md` for a change of its own. Run 69, a cohort of the + defined-terms change on this document with this change merged in, then had two readers set + clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran)'s "MUST carry both a usable summary and the location of the remainder" against its + "need not name it", and two readers stop at clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps)'s gap sentence and its MAY: clause + [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) now owes the location only where the summary leaves a finding out, and clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) says + where a practitioner's record of a gap lives and that a detector without a declaration can + still be found to conform. Accepted by run NNN. ### 1.0.0, 2026-09-08 diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index 42369c6..b4e8160 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -158,8 +158,8 @@ any. ### 5.3 The result MUST reach the practitioner in the output of the command they ran Where the [Detector] writes fuller detail elsewhere, the invoked command's own output MUST -carry both a usable summary and the location of the remainder. The remainder is the findings -the summary leaves out. Where the output prints every finding, each with where it is and its +carry a usable summary and, where the summary leaves any finding out, the location of the +remainder. The remainder is the findings the summary leaves out. Where the output prints every finding, each with where it is and its [Identifier], there is no remainder. A fuller report written elsewhere, such as a page with a code snippet for each finding, only adds detail to findings the output already carries; it is not a remainder, and the output need not name it. @@ -334,11 +334,12 @@ with a non-empty gap record is a statement of where the [Detector] stands and is [Conformance]. The declaration is optional, and so the duty above falls only on a [Detector] that has one; a -gap a [Practitioner] finds is recorded under the method's clause 3.2 in any case. It is how a -maintainer claims [Conformance]; it is not a condition of it. A [Detector] that predates this document, or whose maintainer has never read -it, MAY be graded [Conforming] on evidence: anyone may grade it, by checking it against each -clause of sections 4 to 7, and a verdict on any [Detector], declared or not, rests on that -check and not on the claim. A declaration tells the reader what the maintainer believes and what they know to be missing; +gap a [Practitioner] finds is recorded in their own project's record under the method's clause +3.2 in any case. The declaration is how a maintainer claims [Conformance]; it is not a condition +of it, and a [Detector] without one can still be found to [Conform]. A [Detector] that predates +this document, or whose maintainer has never read it, MAY be graded [Conforming] on evidence: +anyone may grade it, by checking it against each clause of sections 4 to 7, and a verdict on +any [Detector], declared or not, rests on that check and not on the claim. A declaration tells the reader what the maintainer believes and what they know to be missing; the reader still runs the [Harness]. The shape is illustrative rather than prescribed. In a Composer manifest: From bc6ae14586c1a392c1808989114ebc71a46b9d27 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:01:43 +0100 Subject: [PATCH 085/105] Method key: clause 3.3's narrowing sentence; routed findings from runs 68 to 71 The method key lists the sentence behind a narrowing, and the owner's decision where it cannot be written, among the obligations, after three readers of run 70 restated it. acceptance/ROUTED.md gains Appendix A's closing paragraph from run 70 and the detector's clause 5.3 from run 69, and notes runs 68, 70 and 71 on the rows they raised again. Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 60 +++++++++++++++++++++------------------- acceptance/method/KEY.md | 2 ++ 2 files changed, 33 insertions(+), 29 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index c6c3d28..eced9c7 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -15,38 +15,40 @@ untouched text of the change those runs tested, and are listed here so that they ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ---------------------------------------- | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | run 67 | `open` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | -| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | ---------------------------------------- | +| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | run 69 | `editor/detector-route-rule` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68 | `open` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | +| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | ## Closed A row moves here when a change that reworks its passage passes a cohort, naming that run, or when the editor closes it with a reason. -| Document | Clause | Passage | Raised by | Closed by | -| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------------------------------------ | -| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 20, 22, 24, 30, 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule`, run 66 | -| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 20, 30, 39, 42 | `editor/detector-route-rule`, run 66 | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule`, run 66 | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | runs 20, 54 | `editor/detector-route-rule`, run 66 | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on the two bounds | "Both bounds measure breadth within one level; the level itself is a separate question, answered against the report." | runs 19, 56 | `editor/defined-terms`, run 64 | -| `SPEC.md` | [3.5](../SPEC.md#35-enforce-permanently-and-block), the paragraph on a bundled defence | "For a Bundled defence, the project whose entry point demonstrates it is any project the Toolchain is genuinely installed into", and its substitutes sentence | runs 25, 29, 53, 59 | `editor/defined-terms`, run 64 | +| Document | Clause | Passage | Raised by | Closed by | +| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------ | +| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 20, 22, 24, 30, 36, 42, 45, 48, 51, 54, and 71 on a branch without the rework | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 20, 30, 39, 42, and 71 on a branch without the rework | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | runs 20, 54 | `editor/detector-route-rule`, run 66 | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on the two bounds | "Both bounds measure breadth within one level; the level itself is a separate question, answered against the report." | runs 19, 56, and 70 on a branch without the rework | `editor/defined-terms`, run 64 | +| `SPEC.md` | [3.5](../SPEC.md#35-enforce-permanently-and-block), the paragraph on a bundled defence | "For a Bundled defence, the project whose entry point demonstrates it is any project the Toolchain is genuinely installed into", and its substitutes sentence | runs 25, 29, 53, 59 | `editor/defined-terms`, run 64 | diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 86cf8ff..e2ffd60 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -23,6 +23,8 @@ no bullet making an assertion listed below as wrong, is a correct restatement. practitioner's own authority (section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom)). - Fix the original defect conventionally, with a test that reproduces it, once the defence is in place (sections [2](../../SPEC.md#2-when-the-method-applies) and [6](../../SPEC.md#6-relationship-to-test-driven-development)). +- Write down, for any code the rule is narrowed to exclude, why the hazard cannot arise there, + and take the exclusion to the owner where that sentence cannot be written ([3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire), Part B). Wrong if it asserts: the practitioner fixes the reported instance first and defends afterwards; a test may serve as the detector; an agent may baseline the remaining instances. A bullet that From aa21deedeaa0c0c2b8f7db88b0ce4fbc85434ea7 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:02:33 +0100 Subject: [PATCH 086/105] Detector 8.2: the configuration whose kind is named, in the clause's own words After run 71: the sentence naming which configuration's kind a verdict states no longer points at the method's clause 3.2, which two readers given this document alone could not follow; it says the configuration that needs no hosted service and no licence tier. Co-Authored-By: Claude Fable 5.1 --- DETECTOR-SPEC.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index 40f0161..ddd9a8b 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -356,9 +356,8 @@ A verdict on a [Detector] MUST name its kind, [Semantic], [Syntactic] or [Textua configuration the verdict grades. A configuration here is one way of setting the [Detector] up to run, and one [Detector] can have several: ESLint, for example, is [Syntactic] run without type information and [Semantic] run with it. The kind is what a [Rule] can ask in that configuration, -as section 2 glosses it. The kind named is the one the [Detector] has without a hosted service or -a licence tier, the same configurations the method specification's clause 3.2 counts when it -asks for the strongest kind. +as section 2 glosses it. The kind named is the one the [Detector] has in a configuration that +needs no hosted service and no licence tier. This clause binds whoever gives the verdict, anyone grading a [Detector] under clause 8.1 included, and asks nothing of the [Detector] itself; a maintainer who gives a verdict on their From dbdf841674660faea021eb5cb534c3763fb886dd Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:11:45 +0100 Subject: [PATCH 087/105] Detector 5.3 and 7.1: a usable summary defined, detectability in one paragraph of two ways --- DETECTOR-SPEC.md | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index b4e8160..1de2e6e 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -159,10 +159,10 @@ any. Where the [Detector] writes fuller detail elsewhere, the invoked command's own output MUST carry a usable summary and, where the summary leaves any finding out, the location of the -remainder. The remainder is the findings the summary leaves out. Where the output prints every finding, each with where it is and its -[Identifier], there is no remainder. A fuller report written elsewhere, such as a page with a -code snippet for each finding, only adds detail to findings the output already carries; it is -not a remainder, and the output need not name it. +remainder. A summary is usable when each finding in it carries where it is and its +[Identifier]. The remainder is the findings the summary leaves out: where the output prints +every finding there is none, and a fuller report written elsewhere, such as a page with a code +snippet for each finding, is not one and need not be named. **Why**: method specification clause 8.2. A report the [Practitioner] has to go and find is a report that arrives after the decision it was meant to inform. @@ -281,14 +281,12 @@ detectable, by a bespoke [Rule] the project writes for the [Detector] under clau mechanical check the [Detector] documents. Either lets a project which decides to forbid the route enforce that decision. A route that can be neither switched off nor seen does not [Conform]. -A route is detectable when the project can find every use of it. The first way is a bespoke -[Rule]: the project registers a [Rule] of its own, as clause 4.1 provides for, that matches the -route wherever it is used, and the [Detector] runs it like any other bespoke [Rule]; the -[Detector] ships no such [Rule] itself, and need not. That way is open only where the -[Detector]'s [Rules] can see the route: a [Detector] whose [Rules] never see comments cannot make -an ignore comment detectable this way. The second way is a mechanical check the [Detector] -documents, such as a text search for the route's spelling that the [Detector]'s documentation -gives. +A route is detectable when the project can find every use of it, in either of two ways: by a +bespoke [Rule] the project registers under clause 4.1 that matches the route wherever it is +used, which the [Detector] runs like any other bespoke [Rule] and need not ship itself; or by a +mechanical check the [Detector] documents, such as a text search for the route's spelling. The +first way needs the [Detector]'s [Rules] to be able to match the route's text, so a [Detector] +whose [Rules] never see comments offers only the second for an ignore comment. **Why**: the method specification's position is that [Suppression] is an [Owner] decision under its clause 3.4 and section 4, and an [Owner] cannot decide something they are From 13b497ba3ddc0eba771e3c516bc89e4ffe14cb2d Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:18:48 +0100 Subject: [PATCH 088/105] Toolchain key: the 4.3 row says which case the fixture presents; ROUTED 5.3 row adds run 75 Run 72's open point: three readers graded clause 4.3 on the condition that the ignore comment is Lintomatic's only suppression route, and the key did not say which case the fixture presents. The 4.3 row now says so, and such a grade is the grade for this case. Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 2 +- acceptance/toolchain/KEY.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index eced9c7..68a0be9 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -18,7 +18,7 @@ untouched text of the change those runs tested, and are listed here so that they | Document | Clause | Passage | Raised by | Status | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | ---------------------------------------- | | `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | run 69 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | | `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | | `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | | `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index aadab7a..e098c90 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -63,7 +63,7 @@ Wrong readings the text invites, which count against the document on one reader: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | None | Lintomatic fails detector [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) on two undocumented bundled rules and nothing in the wrapping closes that gap; a reader who notes that the detector specification was not supplied and grades from this document's summary of it is correct | | [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | None. A No given on the condition that the two bundled rules are blocking is a No: their firing stops `zig-qa run`, so they are | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)` | -| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | None | Ignore comments disabled project-wide; exceptions only through the record the toolchain reads | +| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | None | The fixture names one suppression route of Lintomatic's, the inline ignore comment, disabled project-wide, with exceptions only through the record the toolchain reads; a grade given on the condition that the comment is Lintomatic's only route is the grade for this case | | [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | None | `zig-qa run` is local, prints Lintomatic's output unchanged, one line per finding | | [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | None | Lintomatic, then the tests, stopping on a Lintomatic failure | | [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes, Partial or No | None | `zig-qa list` prints identifier, summary and documentation path without running the rules; Partial or No is the reading that counts the two `(none)` routes, since [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) says the listing MUST provide the route and [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) point 2 says a gap may be named again by a later clause | From f32e54224a351627a5412cfb2cfee9f9942d7aea Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:19:41 +0100 Subject: [PATCH 089/105] Run 073: detector route rule cohort, failed on clauses 5.3 and 7.1 Every keyed item at threshold. Two findings in text the change wrote: clause 5.3's "usable summary" and whether a report written elsewhere is the remainder, and clause 7.1's second paragraph on when a route is detectable. Nothing routed. Co-Authored-By: Claude Fable 5.1 --- .../runs/073-detector-route-rule-run-4.md | 151 ++++++++++++++++++ 1 file changed, 151 insertions(+) create mode 100644 acceptance/runs/073-detector-route-rule-run-4.md diff --git a/acceptance/runs/073-detector-route-rule-run-4.md b/acceptance/runs/073-detector-route-rule-run-4.md new file mode 100644 index 0000000..2f070f2 --- /dev/null +++ b/acceptance/runs/073-detector-route-rule-run-4.md @@ -0,0 +1,151 @@ +# Run 73: DETECTOR-SPEC.md, detector route rule, fourth run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [23](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23), from branch + `editor/detector-route-rule`. The readers read commit d67174c. Change: clause 7.1's detectable + route is a bespoke rule the project writes for the detector under clause 4.1, with a paragraph + on what detectable means; clause 8.1's gap duty falls only on a detector that declares a + version; clause 5.3 says what the remainder is; clause 6.1 says it asks for the mechanism, a + missing page failing clause 6.3, and that a path in the message is not enough on its own; + section 2's table gains a Message row. Revised after run 69 (a cohort of another change with + this one merged in) in clause 5.3's obligation sentence, the location of the remainder owed + only where the summary leaves a finding out, and in clause 8.1's second paragraph, where a + practitioner's record of a gap lives and that a detector without a declaration can still be + found to conform; so this is a fresh run. +- **Run within the pull request**: 4 (runs 60, 63 and 66 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `0df0b01c02c5d2fd83b37d218f88785980384c3a0ecadce92abf7088211c9249`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at d67174c, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Question 1: correct bullets per reader 6, 6, 6, 6 and 6. Readers 1, 2, 3 and 5 give one bullet +per line of the key's list (4.1; 4.2; 4.3; 5.1 to 5.4; 6.1 to 6.3; 7.1). Reader 4 splits 5.1 to +5.3 from 5.4, which are different clauses of one line and each count, and joins 6.1 to 6.3 with +7.1 in its sixth bullet. Reader 2's fourth bullet says every finding must reach the command's +output, which the key says names 5.3 and is not marked wrong for the plural; its fifth bullet +says every identifier the detector can print must have a page, which names the 6.1 to 6.3 line +and is not among the key's wrong assertions. Reader 5 adds, after its six bullets, a line on +what conformance means; it is not among the first six bullets and states no obligation. No +bullet asserts anything the key marks wrong. + +Question 2: every reader states every required element of each item and quotes the document. +For 2a all five state that conformance is every MUST in sections 4 to 7 and that a detector +predating the document can conform; readers 1 to 4 quote section 8's definition, and all five +quote clause 8.1's "A Detector that predates this document, or whose maintainer has never read +it, MAY be graded Conforming on evidence" (reader 5 also its "a Detector without one can still +be found to Conform"). For 2b all five name both things, disableable by configuration or +detectable, and that either alone is enough, quoting clause 7.1's "For each one it MUST do at +least one of two things, and either alone satisfies this clause". Readers 2, 3 and 5 add that +the bespoke-rule way is closed to a detector whose rules never see comments, the document's own +sentence, which is not among the key's wrong assertions for 2b. For 2c all five state that the +documentation ships with the rule (in the installed copy), at a version tracked together, and +is reachable without network access, quoting the 6.2 and 6.3 headings; reader 4 adds clause +6.2's sentence that a website catalogue is the wrong place, which agrees with the key. + +Question 3: readers 1, 2, 4 and 5 give every accepted grade, each with caveats on 4.2 (the +fixture does not say `--only` prints finding lines) and 4.3 (no machine-readable format +described); the key says a caveat with the accepted grade is correct. S8: all five say +Lintomatic does not conform, naming 6.3. 6.4: all five say it does not change the verdict, +because it is a SHOULD. + +### Misses + +- **Reader 3, clause 4.2: Partial** (accepted: Yes). Reasoning: `--only` runs one rule and exits + 1 when it fires, but the clause requires the harness to report "what it printed" and the + fixture never says `--only` prints the finding line; "it becomes Yes if `--only` output + includes the finding". The reasoning quotes three words of the document, below the six that + `ACCEPTANCE.md` defines as quoting it, so it does not quote the document. The key names no + wrong reading for 4.2. A single miss, the reader's, tolerated by the threshold. Reader 3's + question 4 note puts the close call on the fixture, not the document. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..d67174c adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | ------------------- | +| 1 | Clause 8.1, first paragraph: "The same declaration is where a gap against this document is recorded once it is known: a [Detector] that declares a version and has learnt… that it fails a MUST in sections 4 to 7 MUST record that gap…"; whether a MUST in section 8 counts toward conformance | 1 | yes | +| 2 | Clause 5.3, first paragraph: "the invoked command's own output MUST carry a usable summary" (2, what makes a summary usable); "The remainder is the findings the summary leaves out." (5, whether a fuller HTML report is a remainder, settled only by the sentences after) | 2, 5 | yes | +| 3 | Clause 7.1, second paragraph: "a [Detector] whose [Rules] never see comments cannot make an ignore comment detectable this way" (2, whether a property of the detector's API or of its rules); "That way is open only where the [Detector]'s [Rules] can see the route: …" (4, whether the route was open to Lintomatic) | 2, 4 | yes | +| 4 | Section 1, "The governing principle" paragraph: "a [Conforming] [Detector] MUST make that possible without the project building the mechanism first"; an unnumbered MUST outside sections 4 to 7, and whether it bears on conformance | 4 | no | +| 5 | Clause 4.2, first paragraph: "a [Detector] failing 4.1 is graded here on its [Bundled rules]", set beside the sentence before it that the clause applies to every rule | 5 | no | + +Notes, not passages: reader 3 wrote "none", adding that its 4.2 close call came from the fixture +not stating what `--only` prints, not from the document. + +## Findings and where they lie + +- **Finding A, passage 2, clause 5.3 first paragraph** (readers 2 and 5): the diff 9e914fe..d67174c + rewords this paragraph's obligation sentence and adds its three following sentences, so it is + text the change wrote or altered. Reader 2's words ("usable summary") are in the part of the + sentence the change kept, but the rule is by paragraph. Blocks. +- **Finding B, passage 3, clause 7.1 second paragraph** (readers 2 and 4): the diff adds this + paragraph. Text the change wrote. Blocks. + +Passages 1, 4 and 5 are each named by one reader: recorded, not findings. Passage 1 lies in a +changed paragraph but is single. Nothing is routed to `acceptance/ROUTED.md` from this run. + +## Points the rules did not settle + +1. **Reader 3's 4.2 grade carries a condition for another grade.** `ACCEPTANCE.md` reads a grade + given on a stated condition as the grade for the case the fixture presents where the key says + which case that is, and otherwise as unanswered; the key says no row carries a condition. + Reader 3 gives Partial as its grade and states the condition only for Yes. Marked as Partial, + the grade given; read either way (Partial, or unanswered) the item is a miss, so the mark does + not turn on the point. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit bc6ae14, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item but + 4.2, which is four of five. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: two (findings A and + B). Not met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking; findings A and B are on the change's own text, to be fixed there +before a fresh cohort. + +## Verdict + +FAIL. Every keyed item passed its threshold and no miss counts against the document, but +findings A (clause 5.3, first paragraph) and B (clause 7.1, second paragraph) lie in text the +change wrote or altered, and each blocks. + +## Answer sheets + +Posted on the pull request: LINK PLACEHOLDER. From cca75a72e4d7f3887e02cd0b5d8d4c9e12933b91 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:19:41 +0100 Subject: [PATCH 090/105] Run 077: detector route rule cohort, failed on clause 5.3's remainder sentence Every keyed item at threshold, one miss on clause 4.2 tolerated. One finding in text the change wrote: clause 5.3's remainder sentence, four readers unable to tell what "is not one" refers to. Nothing routed. Co-Authored-By: Claude Fable 5.1 --- .../runs/077-detector-route-rule-run-5.md | 154 ++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 acceptance/runs/077-detector-route-rule-run-5.md diff --git a/acceptance/runs/077-detector-route-rule-run-5.md b/acceptance/runs/077-detector-route-rule-run-5.md new file mode 100644 index 0000000..7d1852f --- /dev/null +++ b/acceptance/runs/077-detector-route-rule-run-5.md @@ -0,0 +1,154 @@ +# Run 77: DETECTOR-SPEC.md, detector route rule, fifth run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [23](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23), from branch + `editor/detector-route-rule`. The readers read commit dbdf841. Change: clause 7.1's detectable + route is a bespoke rule the project writes for the detector under clause 4.1, with a paragraph + on what detectable means; clause 8.1's gap duty falls only on a detector that declares a + version; clause 5.3 says what the remainder is; clause 6.1 says it asks for the mechanism, a + missing page failing clause 6.3, and that a path in the message is not enough on its own; + section 2's table gains a Message row. Revised after run 73 in the two paragraphs its findings + named: clause 5.3's paragraph now defines a usable summary (each finding carries where it is + and its Identifier) and the remainder (the findings the summary leaves out; none where the + output prints every finding; a fuller report written elsewhere is not one and need not be + named), and clause 7.1's second paragraph now states the two ways a route is detectable in one + paragraph (a bespoke Rule registered under clause 4.1, or a mechanical check the Detector + documents); so this is a fresh run. +- **Run within the pull request**: 5 (runs 60, 63, 66 and 73 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `2a96d5b0c6339f5402c9883018d35c0353c42dfad581c94d9b80a8b428882bc9`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at dbdf841, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Question 1: correct bullets among the first six, per reader, 6, 6, 6, 5 and 6. Readers 1 and 5 +give one bullet per line of the key's list. Reader 2 joins 4.1 and 4.2 in its first bullet and +splits 5.1 to 5.3 from 5.4. Reader 3 gives seven bullets; only the first six are marked (4.1, +4.2, 4.3, 5.1 to 5.3, 5.4, 6.1 to 6.3), and its seventh, on 7.1, is not marked. Reader 4 gives +five bullets, each naming a line of the key's list. Readers 1, 2 and 4 say the command's output +carries each finding with its location and identifier, which names 5.3 and is not marked wrong +for the plural. No bullet asserts anything the key marks wrong. + +Question 2: every reader states every required element of each item and quotes the document. +For 2a all five state that conformance is every MUST in sections 4 to 7 and that a detector +predating the document can conform, quoting section 8's definition and clause 8.1's "A Detector +that predates this document, or whose maintainer has never read it, MAY be graded Conforming on +evidence"; readers 1 and 4 also quote "a Detector without one can still be found to Conform". +For 2b all five name both things, disableable by configuration or detectable, and that either +alone is enough, quoting clause 7.1's "For each one it MUST do at least one of two things, and +either alone satisfies this clause" (reader 2 with an ellipsis; its longest verbatim run is well +over six words). Reader 4 adds that a detector whose rules never see comments has only the +documented mechanical check, the document's own sentence, which is not among the key's wrong +assertions for 2b. For 2c all five state that the documentation ships with the rule in the +installed copy, at a version tracked together, and is reachable without network access, +quoting the 6.2 and 6.3 headings; readers 3, 4 and 5 add clause 6.3's "Every Identifier a +Bundled rule can print MUST resolve", which agrees with the key. + +Question 3: readers 1, 2, 3 and 5 give every accepted grade, several with caveats on 4.2 (the +fixture does not say `--only` prints finding lines) and 4.3 (no machine-readable format +described); the key says a caveat with the accepted grade is correct. Readers 1, 3 and 5 state +that they read the two undocumented rules as bundled; reader 3 adds what would follow if they +were project rules, but gives No as its grade without condition. All five grade 5.3 Yes, every +finding being printed so no remainder needs naming, and 6.1 Yes. S8: all five say Lintomatic +does not conform, naming 6.3. 6.4: all five say it does not change the verdict, because it is a +SHOULD. + +### Misses + +- **Reader 4, clause 4.2: Partial** (accepted: Yes). Reasoning: `--only` runs a single rule and + exits 1 when it fires, which covers "whether the Rule fired", but the fixture never says a + single-rule run prints its findings, so "what it printed" is not established. The reasoning + quotes four and three words of clause 4.2, below the six that `ACCEPTANCE.md` defines as + quoting the document, so it does not quote the document; it puts the gap on the fixture's + silence. The key names no wrong reading for 4.2. A single miss, the reader's, tolerated by the + threshold. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..dbdf841 adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ------------------- | +| 1 | Clause 5.3, first paragraph: "The remainder is the findings the summary leaves out: where the output prints every finding there is none, and a fuller report written elsewhere, such as a page with a code snippet for each finding, is not one and need not be named."; what "is not one" refers to (the remainder or the summary), and the double negative | 2, 3, 4, 5 | yes | +| 2 | Clause 7.1, second paragraph: "The first way needs the [Detector]'s [Rules] to be able to match the route's text, so a [Detector] whose [Rules] never see comments offers only the second for an ignore comment."; how the project's bespoke rule relates to the detector's matching, given "need not ship itself" | 1 | yes | +| 3 | Clause 6.4, Why paragraph: "Where the [Detector] is shipped inside a [Toolchain], the [toolchain specification]'s self-audit makes this check a MUST for the [Toolchain]; a [Detector] released on its own is asked for it as a SHOULD."; whether a detector inside a toolchain keeps the SHOULD | 1 | yes | +| 4 | Clause 6.1, "For a project's own [Rule]" paragraph: "The lookup for such a [Rule] is an obligation on the project's assembled [Toolchain], under clause 4.2 of the [toolchain specification]"; the referenced clause could not be checked | 3 | no | +| 5 | Clause 7.1, Why paragraph: "the method specification's position is that [Suppression] is an [Owner] decision under its clause 3.4 and section 4"; it depends on definitions in SPEC.md | 3 | no | + +Notes, not passages: reader 1's third item is about the fixture's `explain` paragraph and its +last paragraph (whether the two rules printing `No documentation declared` are bundled), not a +passage of the document. Reader 2 notes that defined terms link to SPEC.md, which was not +supplied: a remark about another document. + +## Findings and where they lie + +- **Finding A, passage 1, clause 5.3 first paragraph** (readers 2, 3, 4 and 5): the diff + 9e914fe..dbdf841 rewords this paragraph's obligation sentence and adds the sentences after it, + including the one all four readers quote. Text the change wrote. Blocks. + +Passages 2 to 5 are each named by one reader: recorded, not findings. Passages 2 and 3 lie in +changed paragraphs but are single. Nothing is routed to `acceptance/ROUTED.md` from this run. + +## Points the rules did not settle + +1. **Reader 3's passages 4 and 5.** Each quotes six or more words of the document with its + clause number, which `ACCEPTANCE.md` counts as naming a passage, but the confusion stated is + that SPEC.md or TOOLING-SPEC.md was not supplied, which it records as a note and not a + passage. Under the open-point rule, an open point on whether a reader named a passage counts + the passage, so both are listed as passages. Each is named by reader 3 alone, so the verdict + does not turn on the point. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit bc6ae14, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item but + 4.2, which is four of five. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: one (finding A). + Not met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking; finding A is on the change's own text, to be fixed there before a +fresh cohort. + +## Verdict + +FAIL. Every keyed item passed its threshold and no miss counts against the document, but +finding A (clause 5.3, first paragraph, the sentence defining the remainder, named by four of +five readers) lies in text the change wrote, and blocks. + +## Answer sheets + +Posted on the pull request: LINK PLACEHOLDER. From c5f17b5e6cfca24aa0dfe12f45fafee38a6e4987 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:19:41 +0100 Subject: [PATCH 091/105] Detector 5.3: the remainder in three plain sentences After run 77: the remainder is the findings the summary leaves out; where the output prints every finding there is no remainder; a fuller report written elsewhere is not a remainder and the output need not name it. The changelog names runs 73 and 77. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 8 +++++++- DETECTOR-SPEC.md | 6 +++--- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 11f482c..e24f7ea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -101,7 +101,13 @@ pass it. "need not name it", and two readers stop at clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps)'s gap sentence and its MAY: clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) now owes the location only where the summary leaves a finding out, and clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) says where a practitioner's record of a gap lives and that a detector without a declaration can - still be found to conform. Accepted by run NNN. + still be found to conform. [Run 73](acceptance/runs/073-detector-route-rule-run-4.md) failed + on clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran)'s "usable summary" and on clause [7.1](DETECTOR-SPEC.md#71-every-route-that-silences-a-finding-must-be-detectable)'s second paragraph; clause + [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) now says what a usable summary is and what the remainder is, and clause [7.1](DETECTOR-SPEC.md#71-every-route-that-silences-a-finding-must-be-detectable) states + the two ways a route is detectable in one paragraph. [Run + 77](acceptance/runs/077-detector-route-rule-run-5.md) failed on the remainder sentence alone, + four readers unable to tell what "is not one" referred to; it is now three plain sentences. + Accepted by run NNN. ### 1.0.0, 2026-09-08 diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index 1de2e6e..d9823de 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -160,9 +160,9 @@ any. Where the [Detector] writes fuller detail elsewhere, the invoked command's own output MUST carry a usable summary and, where the summary leaves any finding out, the location of the remainder. A summary is usable when each finding in it carries where it is and its -[Identifier]. The remainder is the findings the summary leaves out: where the output prints -every finding there is none, and a fuller report written elsewhere, such as a page with a code -snippet for each finding, is not one and need not be named. +[Identifier]. The remainder is the findings the summary leaves out. Where the output prints +every finding, there is no remainder. A fuller report written elsewhere, such as a page with a +code snippet for each finding, is not a remainder, and the output need not name it. **Why**: method specification clause 8.2. A report the [Practitioner] has to go and find is a report that arrives after the decision it was meant to inform. From 74fffacfa8ecffa25550ec9e3f795ad47be2a4a5 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:19:54 +0100 Subject: [PATCH 092/105] Run 074: semantic detectors, detector cohort failed on clause 8.2 Every keyed item five of five. One finding in text the change wrote: clause 8.2's sentence on the configuration whose kind is named, which three readers could not place. Nothing routed. Co-Authored-By: Claude Fable 5.1 --- .../runs/074-semantic-detectors-run-15.md | 145 ++++++++++++++++++ 1 file changed, 145 insertions(+) create mode 100644 acceptance/runs/074-semantic-detectors-run-15.md diff --git a/acceptance/runs/074-semantic-detectors-run-15.md b/acceptance/runs/074-semantic-detectors-run-15.md new file mode 100644 index 0000000..61f794a --- /dev/null +++ b/acceptance/runs/074-semantic-detectors-run-15.md @@ -0,0 +1,145 @@ +# Run 74: DETECTOR-SPEC.md, semantic detectors, fifteenth run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: 14, open on branch `editor/semantic-detectors`. The readers read commit + f7e55e1. Change: a new clause 8.2, that a verdict on a detector names its kind, semantic, + syntactic or textual, in each configuration it grades; section 9's sentence on that + obligation; section 2's table row for the three kinds. Since run 71 the branch has merged + `editor/detector-route-rule` (pull request 23), whose rework of clauses 5.3, 6.1, 7.1 and 8.1 + and whose Message row in section 2 are that change's text, not this one's; clause 8.2's + sentence on which configuration's kind is named was reworded after run 71. +- **Run within the pull request**: 15 (runs 19 to 30, 70 and 71 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `003d030ade886b2008da3abc90e75d8c62be8f02f79c3b4aec05818379bcc550`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at f7e55e1, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model; each sheet is that output's `result`. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Question 1: readers 1, 3 and 4 name, one bullet each, the key's lines for 4.1; 4.2; 4.3; 5.1 to +5.4; 6.1 to 6.3; and 7.1. Reader 2 names the same six, its sixth bullet adding what conformance +means, which the key says is neither correct nor wrong; its fourth says every finding must +appear in the command's output, which the key says is not marked wrong for the plural. Reader 5 +names 4.1 with 4.2, 4.3, 5.1 to 5.4, 6.1 to 6.3 and 7.1, and its sixth bullet says any verdict +on a detector must name its kind, which names the 8.2 line and does not put the duty on the +detector. No bullet asserts anything the key marks wrong. + +Question 2: every reader states every required element of each item and quotes the document. +2a: all five state that conformance is every MUST in sections 4 to 7 and that a detector +predating the document can conform, quoting section 8's definition and clause 8.1's "A Detector +that predates this document, or whose maintainer has never read it, MAY be graded Conforming on +evidence". 2b: all five name both things, disableable by configuration or detectable, and that +either alone is enough, quoting clause 7.1's "For each one it MUST do at least one of two +things, and either alone satisfies this clause" (reader 2 with an ellipsis inside the sentence, +reader 3 ending in an ellipsis; each run before the ellipsis is well over six words). 2c: all +five state that the documentation ships with the rule in the installed copy, at a version +tracked together, and resolves without network access, quoting the 6.2 and 6.3 headings (reader +4 also 6.3's first sentence); readers 3 and 5 add that a website does not satisfy it, which is +not among the key's wrong assertions. + +Question 3: every reader gives the accepted grade on all eleven clauses, several with caveats on +what the fixture does not say (what `--only` prints under 4.2, machine-readable formats under +4.3), which the key accepts since no row carries a condition. S8: all five say Lintomatic does +not conform, naming 6.3 or the two undocumented bundled rules; all five say the verdict cannot +name Lintomatic's kind because the fixture does not say it, and that this leaves the +conformance finding standing, which the key does not mark. 6.4: all five say it does not change +the verdict, because it is a SHOULD. + +### Misses + +None. No answer matches a wrong reading the key names for question 1 or question 3. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "this change's text" means a paragraph the diff +d67174c..f7e55e1 adds, rewords, moves, deletes a sentence from, or changes a word, term or link +in. The diff 9e914fe..f7e55e1 shows this change and the merged detector change together. + +| # | Passage (clause, words or description) | Readers | This change's text | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------ | +| 1 | Clause 8.2, first paragraph: "The kind named is the one the [Detector] has in a configuration that needs no hosted service and no licence tier." Reader 2: whether a verdict grading a hosted configuration names the local kind or the hosted one; reader 3: whether it means the strongest kind available locally or the kind of whichever local configuration is named; reader 4 gives no reason | 2, 3, 4 | yes | +| 2 | Clause 6.1, paragraph "For a project's own Rule": "Where the [Identifier] is itself a URL, as method clause 3.6 allows, the [Detector] prints that URL and has done what this clause asks."; the method specification was not supplied (point 1 below) | 1 | no | +| 3 | Clause 8.1, first paragraph: "the method's clause 3.2", given with clause 8.1 as another reference to an unsupplied document (point 1 below) | 1 | no | +| 4 | Clause 6.3, first paragraph: "One [Identifier] that does not resolve fails this clause, however many others do. A page per [Identifier] satisfies this; a page per family of [Identifiers] MAY be used instead, under clause 6.5."; whether a prose-only family page meets 6.3's MUST | 5 | no | +| 5 | Clause 6.5, first paragraph: "Such a page SHOULD let the check clause 6.4 asks for confirm that each member resolves to it."; set against passage 4 | 5 | no | +| 6 | Clause 6.5, table row "Says in prose that everything under `LM-01` is documented here": "No: a person can infer it, a check cannot"; set against passage 4 | 5 | no | +| 7 | Clause 6.1, paragraph "In both, the mechanism is keyed on the Identifier exactly as printed", described, not quoted: "6.1 requires a lookup keyed on the identifier exactly as printed"; set against passage 4 (point 2 below) | 5 | no | + +Notes, not passages: reader 1's general remark that it took references to the method and +toolchain specifications on trust, not having been given them. + +## Findings and where they lie + +- **Finding A, passage 1, clause 8.2 first paragraph** (readers 2, 3 and 4): the diff + d67174c..f7e55e1 adds the whole of clause 8.2, and the sentence the readers quote was reworded + after run 71. Text the change wrote; blocks. + +No other passage is named by two or more readers, so nothing is routed to +`acceptance/ROUTED.md` from this run. Passages 2 to 7 are each one reader's and are recorded +only. + +## Points the rules did not settle + +1. **Reader 1's question 4: passages or a note.** Its subject is that the method specification + was not given, which reads as a note, but it quotes a sentence of clause 6.1 (passage 2) and + names "the method's clause 3.2" with clause 8.1, a four-word quotation that occurs in both + paragraphs of 8.1. Counted, by the open-point rule (an open point on whether a reader named a + passage counts the passage), as naming clause 6.1's paragraph and clause 8.1's first + paragraph, where the words first occur; the general remark is recorded as a note. No finding + turns on this point: no other reader names either paragraph. +2. **Reader 5's description of clause 6.1.** It gives a clause number and a description, not a + quotation, that identifies the sentence "In both, the mechanism is keyed on the Identifier + exactly as printed"; counted as a passage by the open-point rule. It is not the paragraph + reader 1 names in clause 6.1, so no finding turns on it. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit bc6ae14, named before the readers started. `ACCEPTANCE.md`, +`acceptance/README.md` and `acceptance/runs/README.md` are identical at bc6ae14 and f7e55e1. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. Met. +- Passages named by two or more readers in text the change wrote or altered: one (finding A, + clause 8.2 first paragraph). Not met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking. + +## Verdict + +FAIL. Every keyed item passes, but finding A, named by readers 2, 3 and 4, lies in the first +paragraph of clause 8.2, which the change wrote: the sentence "The kind named is the one the +Detector has in a configuration that needs no hosted service and no licence tier." + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From 9841bc907745bdbb1677c10a809968ffb5a1861c Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:19:54 +0100 Subject: [PATCH 093/105] Detector 8.2: the hosted-service sentence dropped After run 74: clause 8.2 no longer says which configuration's kind is named beyond "each configuration the verdict grades"; the sentence on a hosted service or a licence tier, which three readers could not place, is gone, and the changelog line follows. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 12 +++++++++--- DETECTOR-SPEC.md | 3 +-- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f7fe9c7..af92d51 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -102,8 +102,7 @@ pass it. ### Unreleased - New obligation, clause [8.2](DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades): a verdict on a detector MUST name the kind, semantic, syntactic - or textual, of each configuration it grades, as the detector has it without any hosted service - or licence tier. It is not a level of conformance and does not change the verdict; it is + or textual, of each configuration it grades. It is not a level of conformance and does not change the verdict; it is stated because method clause [3.2](SPEC.md#32-build-the-net) now requires a rule to be written in the strongest kind of detector that can express it, and a verdict of conforming alone presents a textual detector as an equal host. Section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) still adds no obligation to the method a practitioner @@ -112,7 +111,14 @@ pass it. message. Runs 20, 22, 24, 26, 28 and 30 were made on an earlier text of this change under the earlier acceptance rules and failed on passages the change had not touched, as `acceptance/CHANGES.md` records; the passages those runs named in untouched text are listed - in `acceptance/ROUTED.md` for changes of their own. Accepted by run NNN. + in `acceptance/ROUTED.md` for changes of their own. [Run + 71](acceptance/runs/071-semantic-detectors-run-14.md), under the rules of pull request 22, + failed on this clause's first paragraph, which two readers could not place, and on misses in + clauses [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) and [6.1](DETECTOR-SPEC.md#61-every-finding-must-carry-the-route-to-its-remediation-docs) in text the change had not touched; the paragraph then gained a sentence + naming the configuration whose kind is named, and [run + 74](acceptance/runs/074-semantic-detectors-run-15.md) failed on that sentence, three readers + unable to tell which configuration it meant. The sentence is withdrawn: the verdict names the + kind of each configuration it grades. Accepted by run NNN. - Clarity, no obligation changed: four passages that cold readers of an unrelated change read as the document pulling against itself or could not follow, each listed in `acceptance/ROUTED.md`. Clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)'s detectable route is now a bespoke rule the project writes for the detector under diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index a8591ec..8e2412a 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -374,8 +374,7 @@ A verdict on a [Detector] MUST name its kind, [Semantic], [Syntactic] or [Textua configuration the verdict grades. A configuration here is one way of setting the [Detector] up to run, and one [Detector] can have several: ESLint, for example, is [Syntactic] run without type information and [Semantic] run with it. The kind is what a [Rule] can ask in that configuration, -as section 2 glosses it. The kind named is the one the [Detector] has in a configuration that -needs no hosted service and no licence tier. +as section 2 glosses it. This clause binds whoever gives the verdict, anyone grading a [Detector] under clause 8.1 included, and asks nothing of the [Detector] itself; a maintainer who gives a verdict on their From ac149963ff6916d0c6bbb34d6a5678400216082f Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:20:49 +0100 Subject: [PATCH 094/105] Changelog: clause links point at the headings as they are Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e24f7ea..20c5805 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -102,8 +102,8 @@ pass it. [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) now owes the location only where the summary leaves a finding out, and clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) says where a practitioner's record of a gap lives and that a detector without a declaration can still be found to conform. [Run 73](acceptance/runs/073-detector-route-rule-run-4.md) failed - on clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran)'s "usable summary" and on clause [7.1](DETECTOR-SPEC.md#71-every-route-that-silences-a-finding-must-be-detectable)'s second paragraph; clause - [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) now says what a usable summary is and what the remainder is, and clause [7.1](DETECTOR-SPEC.md#71-every-route-that-silences-a-finding-must-be-detectable) states + on clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran)'s "usable summary" and on clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)'s second paragraph; clause + [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) now says what a usable summary is and what the remainder is, and clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) states the two ways a route is detectable in one paragraph. [Run 77](acceptance/runs/077-detector-route-rule-run-5.md) failed on the remainder sentence alone, four readers unable to tell what "is not one" referred to; it is now three plain sentences. From 23f5bf030f54cf32f4036674626aa8cbddc14098 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:20:54 +0100 Subject: [PATCH 095/105] Changelog: clause links point at the headings as they are Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index af92d51..8f156d7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -114,7 +114,7 @@ pass it. in `acceptance/ROUTED.md` for changes of their own. [Run 71](acceptance/runs/071-semantic-detectors-run-14.md), under the rules of pull request 22, failed on this clause's first paragraph, which two readers could not place, and on misses in - clauses [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) and [6.1](DETECTOR-SPEC.md#61-every-finding-must-carry-the-route-to-its-remediation-docs) in text the change had not touched; the paragraph then gained a sentence + clauses [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) and [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) in text the change had not touched; the paragraph then gained a sentence naming the configuration whose kind is named, and [run 74](acceptance/runs/074-semantic-detectors-run-15.md) failed on that sentence, three readers unable to tell which configuration it meant. The sentence is withdrawn: the verdict names the From 08b9e5658bbfa022a59cede83305d00513c7d401 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:33:37 +0100 Subject: [PATCH 096/105] Run 078: detector route rule cohort passed Every keyed item at threshold, one miss on clause 4.2 tolerated. No finding in text the change wrote; clause 4.2's sentence on a detector failing 4.1, named by two readers, routed to acceptance/ROUTED.md. The changelog names the run. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 2 +- .../runs/078-detector-route-rule-run-6.md | 177 ++++++++++++++++++ 2 files changed, 178 insertions(+), 1 deletion(-) create mode 100644 acceptance/runs/078-detector-route-rule-run-6.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 20c5805..0325039 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -107,7 +107,7 @@ pass it. the two ways a route is detectable in one paragraph. [Run 77](acceptance/runs/077-detector-route-rule-run-5.md) failed on the remainder sentence alone, four readers unable to tell what "is not one" referred to; it is now three plain sentences. - Accepted by run NNN. + Accepted by [run 78](acceptance/runs/078-detector-route-rule-run-6.md). ### 1.0.0, 2026-09-08 diff --git a/acceptance/runs/078-detector-route-rule-run-6.md b/acceptance/runs/078-detector-route-rule-run-6.md new file mode 100644 index 0000000..a2ebfda --- /dev/null +++ b/acceptance/runs/078-detector-route-rule-run-6.md @@ -0,0 +1,177 @@ +# Run 78: DETECTOR-SPEC.md, detector route rule, sixth run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [23](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23), from branch + `editor/detector-route-rule`. The readers read commit ac14996. Change: clause 7.1's detectable + route is a bespoke rule the project writes for the detector under clause 4.1, with a paragraph + on what detectable means; clause 8.1's gap duty falls only on a detector that declares a + version; clause 5.3 says what the remainder is; clause 6.1 says it asks for the mechanism, a + missing page failing clause 6.3, and that a path in the message is not enough on its own; + section 2's table gains a Message row. Revised after run 77 in the one paragraph its finding + named: clause 5.3's remainder sentence is now three plain sentences (the remainder is the + findings the summary leaves out; where the output prints every finding there is no remainder; + a fuller report written elsewhere is not a remainder and the output need not name it); so this + is a fresh run. +- **Run within the pull request**: 6 (runs 60, 63, 66, 73 and 77 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `9d52a2f0c2c15cb1f4d1b7d628fc403560534fa35cdde941e26efe717846f35c`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at ac14996, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 1 | correct | ok | ok | ok | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Question 1: every reader has six correct bullets among the first six, one per line of the key's +list (4.1, 4.2, 4.3, 5.1 to 5.4, 6.1 to 6.3, 7.1). Reader 1 gives seven bullets; its seventh, on +section 8, is not marked. Readers 3 and 5 also restate SHOULDs (4.4, 6.4, 7.2) as SHOULDs, and +reader 4's sixth bullet adds what conformance means; the key says neither is correct nor wrong. +Reader 4 says the output carries a pointer to any findings the summary leaves out, which names +5.3. No bullet asserts anything the key marks wrong. + +Question 2: every reader states every required element of each item and quotes the document. +For 2a all five state that conformance is every MUST in sections 4 to 7 and that a detector +predating the document can conform, quoting clause 8.1's "A Detector that predates this +document, or whose maintainer has never read it, MAY be graded Conforming on evidence"; readers +1, 3 and 4 also quote section 8's definition, reader 2 its "satisfies every MUST in sections 4 +to 7". For 2b all five name both things, disableable by configuration or detectable, and that +either alone is enough, quoting clause 7.1's "For each one it MUST do at least one of two +things, and either alone satisfies this clause"; readers 2, 4 and 5 add that 7.2's reason is a +SHOULD, and reader 3 that a detector whose rules never see comments has only the mechanical +check, the document's own sentence; neither is among the key's wrong assertions. For 2c all five +state that the documentation ships with the rule (reader 2 "inside the installed rule package"), +at a version tracked together or the same version, and reachable without network access, +quoting clause 6.2's heading and clause 6.3's heading or its "Every Identifier a Bundled rule can +print MUST resolve" sentence. None names a website as acceptable; reader 4 quotes 6.2's Why +saying it is not. + +Question 3: readers 2 to 5 give every accepted grade; readers 2, 3 and 4 add caveats on 4.2 (the +fixture does not say `--only` prints finding lines) and 4.3 (no machine-readable format +described), and the key says a caveat with the accepted grade is correct. Reader 3 states that +it read the two undocumented rules as bundled and gives No for 6.3 without condition. All five +grade 5.3 Yes because every finding is printed so there is no remainder, and 6.1 Yes; readers 2, +3 and 4 say the two missing pages fail 6.3, not 6.1. No reader grades 6.3 Partial or 7.1 No. S8: +all five say Lintomatic does not conform, naming 6.3 (reader 1 adds its own 4.2 Partial as a +second reason). 6.4: all five say it does not change the verdict, because it is a SHOULD. + +### Misses + +- **Reader 1, clause 4.2: Partial** (accepted: Yes). Reasoning: `--only` exits 1 when the rule + fires, so whether it fired is reported, but the fixture never says `--only` prints what the + rule printed, which 4.2 requires. The quotation in the reasoning is from the fixture ("exits 1 + when the rule fires, 0 when it does not"); clause 4.2 is paraphrased, not quoted, so the + reasoning does not quote the document. It puts the gap on the fixture's silence. The key names + no wrong reading for 4.2. A single miss, the reader's, tolerated by the threshold. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..ac14996 adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. +That diff changes section 2's Message row, clause 5.3's first paragraph, clause 6.1's "In both" +and "For a Bundled rule" paragraphs, clause 6.4's Why paragraph, clause 7.1's first and second +paragraphs, and clause 8.1's first and second paragraphs. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------- | +| 1 | Clause 4.2, first paragraph: "a [Detector] failing 4.1 is graded here on its [Bundled rules]."; reader 2: the sentence's purpose unclear until read as applying 4.2 to bundled rules alone | 1, 2 | no | +| 2 | Clause 6.1, the table row for a project's own Rule: "Print the [Identifier] unaltered; the lookup is the [Toolchain]'s to give", set against the clause heading's "MUST provide a mechanism that resolves a printed identifier to its documentation" | 1 | no | +| 3 | Clause 6.1, the heading, which reader 1 sets against passage 2; counted as named by reader 1 under the open-point rule | 1 | no | +| 4 | Clause 8.1, the JSON example: `"method": "1.0.1"`, against the header's method specification version 1.1.0; what the field means | 2 | no | +| 5 | Clause 8.1, second paragraph: "A [Detector] that predates this document, or whose maintainer has never read it, MAY be graded [Conforming] on evidence: anyone may grade it, ... rests on that check and not on the claim."; how every detector is graded, under a heading about the optional declaration | 3 | yes | +| 6 | Clause 4.3, first paragraph: "The [Detector] MUST print it, unaltered, alongside every finding the [Rule] reports, in its default output and in every machine-readable format it offers."; whether the HTML report is a machine-readable format, set against 5.3's "fuller report" | 5 | no | +| 7 | Clause 5.3, first paragraph, which reader 5 cites ("5.3 calls the HTML a 'fuller report'") in passage 6; counted as named by reader 5 under the open-point rule | 5 | yes | + +Notes, not passages: reader 2's items on clause 6.1 ("Where the [Identifier] is itself a URL, as +method clause 3.6 allows": the method specification was not provided, so the cross-reference +could not be checked) and on clause 7.1's Why paragraph ([Owner], [Class] and [Hazard] are +defined only in the method specification, which was not given); reader 4's single item, the same +6.1 sentence and the references to method clauses 3.3 (4.2's Why) and 3.2 (8.1), with *Class* +and *Owner*, all because the method specification was not provided. Each is a remark that +another document was not supplied, which `ACCEPTANCE.md` records as a note and not a passage; +see the first point below. + +## Findings and where they lie + +- **Finding A, passage 1, clause 4.2 first paragraph** (readers 1 and 2): the diff + 9e914fe..ac14996 does not touch clause 4.2, and neither reader sets the paragraph against a + paragraph the change wrote or altered. A finding against the document, not the change: to be + recorded in `acceptance/ROUTED.md` as a new open row and routed to a change of its own. Does + not block. + +Passages 2 to 7 are each named by one reader: recorded, not findings. Passages 5 and 7 lie in +changed paragraphs but are single. + +## Points the rules did not settle + +1. **Remarks that the method specification was not supplied, made by quoting a sentence.** + Readers 2 and 4 each quote six or more words of clause 6.1's "For a project's own Rule" + paragraph, with the clause number, and say only that the method specification was not + provided, so the cross-reference could not be checked. `ACCEPTANCE.md` both says a quotation + with a clause number names a passage and says a remark "such as that another document was not + supplied" is a note and not a passage. Run 77 recorded this as an open point and counted the + passages; `ACCEPTANCE.md` and the key have not changed on it since, so it was not settled + before this run as "Changing the test" asks. This marker holds that the rule settles it: the + stated confusion in each item is exactly the rule's own example of a note, and the quotation + only locates the reference whose target was not supplied; so these are notes, and no open + point arises. The verdict turns on this. Were it held open and the passages counted, readers 2 + and 4 would form a finding on clause 6.1's "For a project's own Rule" paragraph, unchanged + and routed (it is already open in `acceptance/ROUTED.md`, runs 28, 63, 66); and reader 4's + "8.1's 'the method's clause 3.2'", whose words occur in both of clause 8.1's paragraphs, would + leave open whether reader 4 and reader 3 (passage 5) named the same changed paragraph, which + the open-point rule counts, so that finding would block. The editor settles the point in + `ACCEPTANCE.md` before any further run. +2. **Reader 1's passage on clause 6.1** sets the table row against the heading; whether it names + one passage or both is not settled, so both are counted (passages 2 and 3). Reader 5's + citation of clause 5.3 is counted the same way (passage 7). No other reader names any of + these paragraphs, so the verdict does not turn on either. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 13b497b, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item but + 4.2, which is four of five. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: none, on the + marking of point 1. Met. +- Passages named by two or more readers elsewhere: one (finding A, clause 4.2), routed to + `acceptance/ROUTED.md`; does not block. + +## Findings applied + +None at the time of marking; finding A is to be added to `acceptance/ROUTED.md` by the editor. + +## Verdict + +PASS, on the marking of point 1. Every keyed item meets its threshold, no miss counts against +the document, and the one finding, clause 4.2's first paragraph, lies outside the text the change +wrote and is routed. + +## Answer sheets + +Posted on the pull request: LINK PLACEHOLDER. From ed889dd23e96c62455872da9af348fb826142a23 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:33:41 +0100 Subject: [PATCH 097/105] Run 079: semantic detectors, detector cohort passed Every keyed item five of five; no passage named by more than one reader. The changelog names runs 70 and 79 as the accepting runs. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 5 +- .../runs/079-semantic-detectors-run-16.md | 150 ++++++++++++++++++ 2 files changed, 153 insertions(+), 2 deletions(-) create mode 100644 acceptance/runs/079-semantic-detectors-run-16.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 8f156d7..137d503 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -39,7 +39,7 @@ version entry with no cohort line has not been accepted and is not published. 19 and 29 failed on statement 2d, and every run failed on passages the change had not touched, as `acceptance/CHANGES.md` records; the change was then reduced to the text above, and the passages those runs named in untouched text are listed in `acceptance/ROUTED.md` for - changes of their own. Accepted by run NNN. + changes of their own. Accepted by [run 70](acceptance/runs/070-semantic-detectors-run-13.md). ### 1.1.0, 2026-10-02 @@ -118,7 +118,8 @@ pass it. naming the configuration whose kind is named, and [run 74](acceptance/runs/074-semantic-detectors-run-15.md) failed on that sentence, three readers unable to tell which configuration it meant. The sentence is withdrawn: the verdict names the - kind of each configuration it grades. Accepted by run NNN. + kind of each configuration it grades. Accepted by [run + 79](acceptance/runs/079-semantic-detectors-run-16.md). - Clarity, no obligation changed: four passages that cold readers of an unrelated change read as the document pulling against itself or could not follow, each listed in `acceptance/ROUTED.md`. Clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)'s detectable route is now a bespoke rule the project writes for the detector under diff --git a/acceptance/runs/079-semantic-detectors-run-16.md b/acceptance/runs/079-semantic-detectors-run-16.md new file mode 100644 index 0000000..fa5feb0 --- /dev/null +++ b/acceptance/runs/079-semantic-detectors-run-16.md @@ -0,0 +1,150 @@ +# Run 79: DETECTOR-SPEC.md, semantic detectors, sixteenth run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: 14, open on branch `editor/semantic-detectors`. The readers read commit + 23f5bf0. Change: a new clause 8.2, that a verdict on a detector names its kind, semantic, + syntactic or textual, in each configuration it grades; section 9's sentence on that + obligation; section 2's table row for the three kinds. The branch carries + `editor/detector-route-rule` (pull request 23) at its commit d67174c, whose rework of clauses + 5.3, 6.1, 7.1 and 8.1 and whose Message row in section 2 are that change's text, not this + one's. Since run 74, clause 8.2's sentence on which configuration's kind is named (one needing + no hosted service and no licence tier) has been removed; the clause's first paragraph now ends + at "as section 2 glosses it". +- **Run within the pull request**: 16 (runs 19 to 30, 70, 71 and 74 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `7b94f116b3f875170cdfd9fd61f845b3cf7f233723bbd749dd38e93adb6a1fd9`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at 23f5bf0, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model; each sheet is that output's `result`. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Question 1: readers 1, 2 and 4 name, one bullet each, the key's lines for 4.1; 4.2; 4.3; 5.1 to +5.4; 6.1 to 6.3; and 7.1. Reader 3 names the same six, its sixth bullet adding what conformance +means, which the key says is neither correct nor wrong. Reader 5's first bullet states what +conformance means (neither correct nor wrong); its others name 4.1, 4.2, 4.3 (as "every rule has +a stable identifier", which the key says names 4.3), 5.1 to 5.4, and, in its sixth bullet, both +6.1 to 6.3 and 7.1 (point 2 below). No reader names the 8.2 line, and no bullet asserts anything +the key marks wrong: none says the detector must be of one kind or must state its own kind. + +Question 2: every reader states every required element of each item and quotes the document. +2a: all five state that conformance is every MUST in sections 4 to 7 and that a detector +predating the document can conform; readers 1, 2, 3 and 5 quote section 8's definition, and all +five quote clause 8.1's "A Detector that predates this document, or whose maintainer has never +read it, MAY be graded Conforming on evidence". 2b: all five name both things, disableable by +configuration or detectable, and say either alone is enough, quoting clause 7.1's "For each one +it MUST do at least one of two things, and either alone satisfies this clause" (readers 1, 3 and +4 with an ellipsis or ending in one; the run before it is well over six words). Readers 2 and 5 +add that a written reason is a SHOULD under 7.2, which is not a wrong assertion. 2c: all five +state that the documentation ships with the rule in the installed copy, at a version tracked +together, and resolves without network access, quoting the 6.2 and 6.3 headings (reader 2 also +6.3's first sentence, reader 3 also 6.5's sentence on the installed file); reader 1 adds that a +website does not satisfy it, which is not among the key's wrong assertions. + +Question 3: every reader gives the accepted grade on all eleven clauses, several with caveats on +what the fixture does not say (what `--only` prints under 4.2, machine-readable formats under +4.3), which the key accepts since no row carries a condition. Reader 2's 4.2 is Yes with the +remark that it would be Partial if `--only` printed nothing; the grade given is Yes, and the key +says a caveat on what the fixture does not say leaves the accepted grade correct. S8: all five +say Lintomatic does not conform, naming 6.3 or the two undocumented bundled rules; all five say +the verdict cannot name Lintomatic's kind because the fixture does not say it, and that the +conformance finding stands, which the key does not mark. None makes the verdict turn on the +kind. 6.4: all five say it does not change the verdict, because it is a SHOULD. + +### Misses + +None. No answer matches a wrong reading the key names for question 1 or question 3. + +## Question 4: passages named + +Reader 1 wrote "none". Paragraphs as `ACCEPTANCE.md` defines them; "this change's text" means a +paragraph the diff d67174c..23f5bf0 adds, rewords, moves, deletes a sentence from, or changes a +word, term or link in. The diff 9e914fe..23f5bf0 shows this change and the merged detector +change together. + +| # | Passage (clause, words or description) | Readers | This change's text | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------ | +| 1 | Clause 6.4, **Why** paragraph: "Where the [Detector] is shipped inside a [Toolchain], the [toolchain specification]'s self-audit makes this check a MUST for the [Toolchain]; a [Detector] released on its own is asked for it as a SHOULD because ..."; the "because" reads as a reason for a MUST as well as for a SHOULD | 2 | no | +| 2 | Clause 7.1, **Why** paragraph: "is an [Owner] decision under its clause 3.4 and section 4"; Owner is defined in the unsupplied method specification (point 1 below) | 3 | no | +| 3 | Clause 4.1, **Why** paragraph: "the [Classes] that matter most to a project"; Class is defined in the unsupplied method specification (point 1 below) | 3 | no | +| 4 | Clause 7.1, second paragraph ("A route is detectable when ..."): "That way is open only where the [Detector]'s [Rules] can see the route: a [Detector] whose [Rules] never see comments cannot make an ignore comment detectable this way.", read with "the [Detector] ships no such [Rule] itself, and need not"; how the detector's MUST is discharged when the project does the work | 4 | no | +| 5 | Clause 4.3, first paragraph: "A prefix the [Detector] adds from the invoking directory or the [Rule]'s location is derived from the file path. A namespace the [Rule author] assigns once in configuration is not."; whether a detector-added prefix not derived from location is allowed | 4 | no | +| 6 | Clause 6.1, heading: "The [Detector] MUST provide a mechanism that resolves a printed identifier to its documentation"; whether the MUST applies to project rules at all | 5 | no | +| 7 | Clause 6.1, paragraph "For a project's own Rule": "what it owes is the half it can give: the [Identifier] printed unaltered under clause 4.3"; set against passage 6 | 5 | no | + +Notes, not passages: reader 2's remark that the fixture does not say whether Lintomatic sits +inside a toolchain, so it read it as standalone. + +Passages 2 and 4 are both in clause 7.1 but in different paragraphs (the **Why** paragraph and +the second paragraph), so they are not the same passage; passages 6 and 7 are both reader 5's. + +## Findings and where they lie + +None. No passage is named by two or more readers, so nothing blocks and nothing is routed to +`acceptance/ROUTED.md` from this run. Passages 1 to 7 are each one reader's and are recorded +only; none lies in text this change wrote or altered (clauses 6.1 and 7.1 are the detector +change's text at d67174c, and clauses 4.1, 4.3 and 6.4 are unchanged by d67174c..23f5bf0). +No reader named clause 8.2, section 9 or section 2's kinds row. + +## Points the rules did not settle + +1. **Reader 3's question 4: passages or notes.** Each item's subject is a term defined only in + the method specification, which was not supplied, which reads as a note; but each quotes six + or more consecutive words of the document with a clause number. Counted, by the open-point + rule (an open point on whether a reader named a passage counts the passage), as naming + clause 7.1's **Why** paragraph and clause 4.1's **Why** paragraph. No finding turns on this + point: no other reader names either paragraph. +2. **Reader 5's sixth question 1 bullet names two lines of the key's list** (6.1 to 6.3, and + 7.1). The rules say how bullets naming one line count, not how one bullet naming two lines + counts. Marked for the reader, by the open-point rule; no mark turns on it, since reader 5's + bullets name at least four lines of the key without it (4.1, 4.2, 4.3, 5.1 to 5.4). + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 13b497b, named before the readers started. `ACCEPTANCE.md`, +`acceptance/README.md`, `acceptance/runs/README.md` and `CONTRIBUTING.md` are identical at +13b497b and 23f5bf0. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. Met. +- Passages named by two or more readers in text the change wrote or altered: none. Met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking. + +## Verdict + +PASS. Every reader is correct on every keyed item, no miss counts against the document, and no +passage is named by two or more readers. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From 389ef3f967fe910150b14f77623c1150423210a5 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:35:36 +0100 Subject: [PATCH 098/105] Acceptance: a missing-document remark is a note however it quotes; a parted paragraph is the change's text Settles the open points of runs 76, 77, 78 and 80. ACCEPTANCE.md says a remark that another document was not supplied, or that a term is defined only there, is a note however much of the document it quotes to locate the reference; and that a paragraph the pull request parts from what it refers to, by inserting a paragraph between them or moving either, is the change's text. The Lintomatic fixture says `check --only` prints its findings as `check` does, and the detector key's 4.2 row says so and which case a conditional grade is for. ROUTED gains the toolchain 6.1 Project record sentence (run 76) and the detector 4.2 sentence on a detector failing 4.1 (run 78, taken by editor/defined-terms); the 4.3 row links the section, whose heading the toolchain change rewrites. Co-Authored-By: Claude Fable 5.1 --- ACCEPTANCE.md | 10 ++++--- acceptance/ROUTED.md | 48 ++++++++++++++++--------------- acceptance/detector/KEY.md | 2 +- acceptance/fixtures/lintomatic.md | 3 +- 4 files changed, 34 insertions(+), 29 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index d1bd6af..77eac8f 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -134,15 +134,17 @@ rule. gives a clause number with a description or a shorter quotation that identifies a sentence or paragraph. A quotation whose words occur in more than one paragraph, given with no clause number, names the paragraph where they first occur. A remark that is not about a passage of - the document, such as that another document was not supplied, is recorded as a note and is - not a passage. Two readers name the same passage when the words or sentences they name lie in + the document, such as that another document was not supplied or that a term is defined only + in another document, is recorded as a note and is not a passage, however much of the document + it quotes to locate the reference. Two readers name the same passage when the words or sentences they name lie in the same paragraph; a heading, a list item, a table row and a terminology entry are each a paragraph of their own. A passage named by two or more readers of the run is a finding. - **Where a finding lies.** The record says, for each finding, whether it lies in text the change wrote or altered: a paragraph in which the pull request, read against its merge base with the branch it targets, adds, rewords, moves or deletes a sentence, or changes a word, a term or a - link; or a paragraph that any reader naming the finding sets against such a paragraph. A - finding there blocks the change. A finding anywhere else is a finding against the document, not + link; a paragraph the pull request parts from what it refers to, by inserting a paragraph + between them or moving either; or a paragraph that any reader naming the finding sets against + such a paragraph. A finding there blocks the change. A finding anywhere else is a finding against the document, not against the change: it is recorded in [acceptance/ROUTED.md](acceptance/ROUTED.md), routed to a change of its own, and does not block. The change is judged on what it changed; the document is proved by the keyed questions, which every run answers over the whole of it. diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 68a0be9..f322fc5 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -15,29 +15,31 @@ untouched text of the change those runs tested, and are listed here so that they ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | ---------------------------------------- | -| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68 | `open` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | -| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | -------------------------------------------------- | +| `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind | run 76 | `open` | +| `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | +| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68 | `open` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | +| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | ## Closed diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 5b4d4aa..dd559a8 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -64,7 +64,7 @@ Wrong readings the text invites, which count against the document on one reader: | Clause | Accepted grade | What the fixture supports (no row carries a condition on the reasoning) | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules via the `Rule` interface, first class in every report | -| [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `check --only ` runs one rule, bundled or project, with a distinguishing exit code | +| [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `check --only ` runs one rule, bundled or project, prints its findings as `check` does, and exits with a distinguishing code; the fixture says all three, so a grade given on the condition that `--only` prints as `check` does is the grade for this case | | [4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `LM-NNNN` fixed at first release; project names taken as given, not rewritten; printed on every finding line | | [5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | Local, no account, no server, no network | | [5.2](../../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Yes | `check ` accepts a single file | diff --git a/acceptance/fixtures/lintomatic.md b/acceptance/fixtures/lintomatic.md index f168d56..9d334fc 100644 --- a/acceptance/fixtures/lintomatic.md +++ b/acceptance/fixtures/lintomatic.md @@ -21,7 +21,8 @@ src/db.zig:88:3 proj.no-raw-sql Query built by string concatenation Fuller detail, including a snippet, is written to `.lintomatic/report.html`. The one-line form is printed whether or not the HTML report is enabled. -`lintomatic check --only ` runs a single rule. It accepts any rule name, +`lintomatic check --only ` runs a single rule and prints its findings as +`check` does. It accepts any rule name, bundled or project, and exits 1 when the rule fires, 0 when it does not, and 2 when the rule name is unknown. From 87e62de5f2154c7d39aa93e15d25f549740689af Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:36:18 +0100 Subject: [PATCH 099/105] Run records: links to the answer sheets posted on the pull request Co-Authored-By: Claude Fable 5.1 --- acceptance/runs/070-semantic-detectors-run-13.md | 2 +- acceptance/runs/071-semantic-detectors-run-14.md | 2 +- acceptance/runs/074-semantic-detectors-run-15.md | 2 +- acceptance/runs/079-semantic-detectors-run-16.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/acceptance/runs/070-semantic-detectors-run-13.md b/acceptance/runs/070-semantic-detectors-run-13.md index b9febf3..154a021 100644 --- a/acceptance/runs/070-semantic-detectors-run-13.md +++ b/acceptance/runs/070-semantic-detectors-run-13.md @@ -231,4 +231,4 @@ None at the time of marking; what to change is the editor's decision. ## Answer sheets -Answer sheets, prompt, command and marker instruction: LINK TO PULL REQUEST COMMENT +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125136762); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125136402); the prompt, [part 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125137201) and [part 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125137748). diff --git a/acceptance/runs/071-semantic-detectors-run-14.md b/acceptance/runs/071-semantic-detectors-run-14.md index de82038..e7d4322 100644 --- a/acceptance/runs/071-semantic-detectors-run-14.md +++ b/acceptance/runs/071-semantic-detectors-run-14.md @@ -167,4 +167,4 @@ clause 8.2's first paragraph, which the change wrote. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125138630); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125138207); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125139111). diff --git a/acceptance/runs/074-semantic-detectors-run-15.md b/acceptance/runs/074-semantic-detectors-run-15.md index 61f794a..adcd679 100644 --- a/acceptance/runs/074-semantic-detectors-run-15.md +++ b/acceptance/runs/074-semantic-detectors-run-15.md @@ -142,4 +142,4 @@ Detector has in a configuration that needs no hosted service and no licence tier ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125140101); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125139595); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125140560). diff --git a/acceptance/runs/079-semantic-detectors-run-16.md b/acceptance/runs/079-semantic-detectors-run-16.md index fa5feb0..f17d355 100644 --- a/acceptance/runs/079-semantic-detectors-run-16.md +++ b/acceptance/runs/079-semantic-detectors-run-16.md @@ -147,4 +147,4 @@ passage is named by two or more readers. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125344415); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125343899); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125344785). From 0da928d9830d12101122821650daacd4e050a6b7 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:43:19 +0100 Subject: [PATCH 100/105] ROUTED: run 81 names the toolchain 6.1 Project record sentence again; 6.2 and 6.3 speak only of exceptions Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 51 ++++++++++++++++++++++---------------------- 1 file changed, 26 insertions(+), 25 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index f322fc5..fe7d05c 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -15,31 +15,32 @@ untouched text of the change those runs tested, and are listed here so that they ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | -------------------------------------------------- | -| `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind | run 76 | `open` | -| `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | -| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68 | `open` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | -| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | -------------------------------------------------- | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) and [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Both clauses speak only of exceptions, where section 2's Project record entry and clause 6.1 cover calibrations and conventions too | run 81 | `open` | +| `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind; named again by four readers of run 81 | run 76 | `open` | +| `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | +| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68 | `open` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | +| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | ## Closed From ae6faf2b34d4f8e21d146fd62cef9445102d00fc Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:48:34 +0100 Subject: [PATCH 101/105] ROUTED: run 82 names the toolchain 6.2 last paragraph again Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index fe7d05c..c16596a 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -29,7 +29,7 @@ untouched text of the change those runs tested, and are listed here so that they | `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | | `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | | `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68, 82 | `open` | | `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | | `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | | `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | From a095fa7ad823b41b0abe3f932d8484dee9a3fd10 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:53:10 +0100 Subject: [PATCH 102/105] ROUTED: toolchain 6.2's first paragraph against clause 6.4's defaults, from run 83 Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 1 + 1 file changed, 1 insertion(+) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index c16596a..5ae6e16 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -17,6 +17,7 @@ untouched text of the change those runs tested, and are listed here so that they | Document | Clause | Passage | Raised by | Status | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | -------------------------------------------------- | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), the first paragraph | "It MUST NOT supply a default, and MUST reject an Exception that omits one" against clause [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)'s defaults | run 83 | `open` | | `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) and [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Both clauses speak only of exceptions, where section 2's Project record entry and clause 6.1 cover calibrations and conventions too | run 81 | `open` | | `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind; named again by four readers of run 81 | run 76 | `open` | | `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | From 852dcd80f61a418609cd36375ca7631d3648a92e Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:58:23 +0100 Subject: [PATCH 103/105] ROUTED: run 84 names the toolchain Project record entry against clauses 6.1, 6.2 and 6.4 Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 54 ++++++++++++++++++++++---------------------- 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 5ae6e16..1423a48 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -15,33 +15,33 @@ untouched text of the change those runs tested, and are listed here so that they ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | -------------------------------------------------- | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), the first paragraph | "It MUST NOT supply a default, and MUST reject an Exception that omits one" against clause [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)'s defaults | run 83 | `open` | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) and [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Both clauses speak only of exceptions, where section 2's Project record entry and clause 6.1 cover calibrations and conventions too | run 81 | `open` | -| `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind; named again by four readers of run 81 | run 76 | `open` | -| `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | -| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68, 82 | `open` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | -| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------- | -------------------------------------------------- | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), the first paragraph | "It MUST NOT supply a default, and MUST reject an Exception that omits one" against clause [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)'s defaults | run 83 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) and [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Both clauses speak only of exceptions, where section 2's Project record entry and clause 6.1 cover calibrations and conventions too | run 81 | `open` | +| `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind; named again by four readers of run 81, and by two readers of run 84 as the Project record entry against clauses 6.1, 6.2 and 6.4 | run 76 | `open` | +| `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | +| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68, 82 | `open` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | +| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | ## Closed From 568e323782847e3f03baeded0303a0979d7d41da Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 15:02:37 +0100 Subject: [PATCH 104/105] ROUTED: run 85 names the toolchain 6.2 first paragraph against clause 6.4 again Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 1423a48..fe7182a 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -17,7 +17,7 @@ untouched text of the change those runs tested, and are listed here so that they | Document | Clause | Passage | Raised by | Status | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------- | -------------------------------------------------- | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), the first paragraph | "It MUST NOT supply a default, and MUST reject an Exception that omits one" against clause [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)'s defaults | run 83 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), the first paragraph | "It MUST NOT supply a default, and MUST reject an Exception that omits one" against clause [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)'s defaults; named again by two readers of run 85 | run 83 | `open` | | `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) and [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Both clauses speak only of exceptions, where section 2's Project record entry and clause 6.1 cover calibrations and conventions too | run 81 | `open` | | `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind; named again by four readers of run 81, and by two readers of run 84 as the Project record entry against clauses 6.1, 6.2 and 6.4 | run 76 | `open` | | `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | From 40f7895867706c8bfdc7f5cb27a9ff1507ba2a8c Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 15:08:03 +0100 Subject: [PATCH 105/105] ROUTED: run 86's findings in untouched toolchain text, clause 6.4 against 6.2 new Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 55 ++++++++++++++++++++++---------------------- 1 file changed, 28 insertions(+), 27 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index fe7182a..e53b671 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -15,33 +15,34 @@ untouched text of the change those runs tested, and are listed here so that they ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------- | -------------------------------------------------- | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), the first paragraph | "It MUST NOT supply a default, and MUST reject an Exception that omits one" against clause [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)'s defaults; named again by two readers of run 85 | run 83 | `open` | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) and [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Both clauses speak only of exceptions, where section 2's Project record entry and clause 6.1 cover calibrations and conventions too | run 81 | `open` | -| `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind; named again by four readers of run 81, and by two readers of run 84 as the Project record entry against clauses 6.1, 6.2 and 6.4 | run 76 | `open` | -| `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | -| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68, 82 | `open` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | -| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | -------------------------------------------------- | +| `TOOLING-SPEC.md` | [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project), the heading and first paragraph | "SHOULD state its own defaults for anything the method leaves to the project" against clause [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one)'s "MUST NOT supply a default" for a justification | run 86 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), the first paragraph | "It MUST NOT supply a default, and MUST reject an Exception that omits one" against clause [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)'s defaults; named again by two readers of run 85 and two of run 86 | run 83 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) and [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Both clauses speak only of exceptions, where section 2's Project record entry and clause 6.1 cover calibrations and conventions too | run 81 | `open` | +| `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind; named again by four readers of run 81, and by two readers of run 84 as the Project record entry against clauses 6.1, 6.2 and 6.4; and by two readers of run 86, the entry and clause 6.1's first paragraph together | run 76 | `open` | +| `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | +| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68, 82 | `open` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | +| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | ## Closed