From 4383fb8a45812f01f141dc695445ce74f8bead6d Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 11:34:10 +0100 Subject: [PATCH 01/52] Acceptance process: the test proves the clarity of the change, with keys that ask what the question asks Keys list the required elements per part of each question, questions ask for exactly those, a confusion finding is settled run by run and blocks only in text the change wrote or altered (others go to acceptance/ROUTED.md), single misses count against the document only where the key names the reading, and open marking points are settled in the rules before the next run rather than invented per run. Co-Authored-By: Claude Fable 5.1 --- ACCEPTANCE.md | 252 ++++++++++++++++++++++-------- CONTRIBUTING.md | 32 ++-- acceptance/CHANGES.md | 36 +++++ acceptance/README.md | 51 ++++-- acceptance/ROUTED.md | 32 ++++ acceptance/detector/KEY.md | 129 +++++++++------ acceptance/detector/QUESTIONS.md | 9 +- acceptance/method/KEY.md | 107 +++++++------ acceptance/method/QUESTIONS.md | 3 +- acceptance/runs/README.md | 25 +-- acceptance/toolchain/KEY.md | 116 ++++++++------ acceptance/toolchain/QUESTIONS.md | 3 +- 12 files changed, 546 insertions(+), 249 deletions(-) create mode 100644 acceptance/CHANGES.md create mode 100644 acceptance/ROUTED.md diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 31c8df0..84ab39b 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -6,49 +6,76 @@ document that only a strong reader can follow is not a specification, and the re matter most, coding agents meeting the method in a failing pipeline, are often the weakest. The test is run with a low-strength model so that clarity is measured at the floor, not the ceiling. +The test proves one thing: that a capable reader, reading the document cold and once, takes from +it what the document exists to say, and is not confused by the text the change put there. The +keyed questions prove the document; the confusion question proves the change. Everything below +serves that, and a rule that stops it proving that is a defect in this file, fixed as +[Changes to this process](#changes-to-this-process) describes. + ## The cohort -- **At least five independent readers**, each a fresh instance of a low-strength model (the - smallest current model in its family, for example Haiku), with no shared context and no - access to the change history, the critique findings or each other. -- Each reader receives **only the changed document or documents** and the fixed question set - below. Nothing else is read first. +- **An odd number of independent readers, at least five**, each a fresh instance of a + low-strength model (the smallest current model in its family, for example Haiku), with no + shared context and no access to the change history, the critique findings or each other. + +- **Each reader is a separate process with no tools and no project context, only its packet**: + started from an empty directory outside any repository, with no user, project or local settings + files, no `CLAUDE.md`, skills, plugins, hooks or MCP servers, and no saved session, so that no + project instruction reaches it. The model tool's own fixed system prompt is not project + context. For example, with the packet on standard input: + + ``` + claude -p --safe-mode --model haiku --tools "" --setting-sources "" --strict-mcp-config --no-session-persistence --output-format json < packet.md + ``` + + A reader is set aside, and its answers do not count, when its answer text is empty, when its + process reports an error, or when the model version it reports is not the cohort's model. A run + with a reader set aside is void: it takes a run number and has a record, and a fresh cohort + runs. No reader is set aside for any other cause. + +- Each reader receives **only its packet**: the document's question set, the changed document + and the fixture the questions name, each in full in the reader's prompt, the question set + first. Each file is preceded by exactly one line, `===== FILE: =====`, where `` is + the name the question set gives the file (so `fixtures/remediation-ledger.md`, and + `QUESTIONS.md` for the question set itself). The question set carries the instruction to the + reader, and nothing else is added. Every reader in a run receives the same packet. + - Readers are run **after** every stronger review has been applied. They are the last gate, not the first draft. ## The question set -Written before the run, with an **answer key** for every keyed question drawn verbatim from the -document, so that marking is mechanical. Each reader answers all of them. +Written before the run, with an **answer key** for every keyed question, so that marking is +mechanical. The key states, for each keyed item, exactly what an answer must contain and what +makes it wrong; the question asks for exactly that and nothing the key does not expect. A key +asks for nothing the question does not. Each reader answers all of them. 1. **Restatement.** In their own words, what the document asks of its subject, in at most six bullets. Keyed against the list of MUST clauses: a bullet is correct when it names an obligation the document actually states. -2. **Load-bearing statements.** Three questions, fixed per document, each asking for one - sentence and a quotation: +2. **Load-bearing statements.** Three or more questions, fixed per document, each asking for a + short answer and the sentence or sentences of the document it rests on, quoted. Each has one + or more parts, and the key lists the element an answer must state for each part: - `SPEC.md`: what a practitioner does before fixing a defect, in order; who may baseline or - suppress; what "conforms" means for a remediation. + suppress, and whether an agent may; what "conforms" means for a remediation and what a + verdict rests on. - `DETECTOR-SPEC.md`: what "conforms" means and whether a tool that predates the document can - conform; what a detector must do about an inline suppression route; where a bundled rule's - documentation must live. - - `TOOLING-SPEC.md`: where conformance is measured; how a grade is assigned when parts come - from different sources; how many grades a project that ships tooling has. + conform; what a detector must do about an inline suppression route and whether one of those + things is enough; where a bundled rule's documentation must live, at what version, and + whether it must be reachable offline. + - `TOOLING-SPEC.md`: where conformance is measured and what the toolchain is; how a grade is + assigned when parts come from different sources; how many grades a project that ships + tooling has and whether one implies the other. 3. **Application.** A fixture case supplied with the questions, graded clause by clause with - Yes, Partial or No and one line of reasoning. Keyed against the intended verdicts; a divergence - counts against the document only when the reader's reasoning shows the clause was read - differently, not when the fixture was misread. + Yes, Partial or No and one line of reasoning. Keyed against the intended verdicts, with the + readings the key names as ones the text invites counting against the document. 4. **Confusion.** Any passage the reader found confusing on its one reading, quoted with its clause number, and nothing else: not what would make it clearer, not a proposed change. -The cohort settles one thing: is the document clear, coherent and comprehensive. A reader getting -everything right after one reading proves it; a reader getting confused fails it. The document -is designed by its editor to resolve the comprehension failures the cohort shows; the cohort is -the acceptance layer, not the designer. - ## What the cohort is not asked The cohort may point at what it found confusing. It is never asked to rate the document, to say @@ -58,61 +85,162 @@ not there to design the text, and a document shaped by what its weakest reader s a document designed by that reader. The signals this test takes from the cohort are a keyed answer missed, a wrong verdict reasoned from the document, and a passage pointed at; what to do about any of them is the editor's decision. An earlier version of this test asked the reader -what would make a clause clearer and for a clarity rating; both were removed on the author's +what would make a clause clearer and for a clarity rating; both were removed on the editor's instruction for that reason, and neither is reinstated. +## Marking + +The editor marks each answer sheet against the document's key, or delegates the marking to a +separate marker that has seen neither the drafting nor the review. A delegated marker receives +the document, its question set, fixture and key at the commit the readers read, this file, the +answer sheets, and the facts the record needs that the sheets cannot show. Its marks are final +unless the editor shows that a rule here was misapplied, and a re-mark says which rule and why. + +Every mark is given by a rule in this file or in the key, as follows. Where neither settles a +mark, the marker gives the reader the benefit of the doubt, records the point in the run record, +and the editor settles it by a change to this file or to the key before the next run. A run +record states no judgement that is not written here or in the key; a judgement an earlier run +recorded is not a rule. + +- **Keyed items.** Question 1 as a whole is one keyed item, and so is each item of question 2 + and each answer of question 3 for which the key gives an intended answer. +- **Reading an answer.** An answer is read for its sense, not its wording: a paraphrase, a + synonym, or a defined term used in its defined sense states what it means. A reader quotes the + document when it gives consecutive words of the document, in the document's order and of any + length, in quotation marks or a block quote, verbatim apart from letter case, quotation-mark or + apostrophe style, and markdown link or emphasis markup. +- **Question 1** is correct when at least four bullets each name an obligation the key lists and + no bullet asserts one the key marks wrong. +- **Question 2.** An answer is the reader's sentences and its quotation read together. It is + correct when it states every element the key lists as required for that item, in the + document's words or the reader's own, asserts nothing the key marks wrong, and quotes the + document. An element stated only in the quotation is stated. Anything in the key beyond the + required elements and the wrong assertions is explanation for the marker, and an answer is + never marked down for leaving it out. +- **Question 3.** Each item is correct when its grade is one the key accepts and its reasoning + mentions what the key says it must mention for that grade. A grade the key accepts without a + condition is correct whatever the reasoning. +- **A miss that counts against the document.** The key names, for question 1 and for each item + of question 3, the wrong readings the text invites. A miss of that kind blocks whatever the + count, since one reader in five taking that reading means one agent in five acting on it. Every + other single miss under questions 1 and 3 is the reader's and is tolerated by the threshold + below; a second reader making the same miss is a failed threshold, not a judgement. +- **An unanswered question.** Each keyed item left unanswered is a miss. Question 4 left + unanswered counts as no passage named. +- **Question 4.** A reader names a passage when it quotes words of the document or gives a clause + number with a description that identifies a sentence or paragraph. A remark that is not about + a passage of the document, such as that another document was not supplied, is recorded as a + note and is not a passage. Two readers name the same passage when the words or sentences they + name lie in the same paragraph; a heading, a list item, a table row and a terminology entry are + each a paragraph of their own. A passage named by two or more readers of the run is a finding. +- **Where a finding lies.** The record says, for each finding, whether it lies in text the change + wrote or altered: a sentence or paragraph the pull request adds, rewords or moves, read against + the branch it targets. A finding there blocks the change. A finding anywhere else is a finding + against the document, not against the change: it is recorded in + [acceptance/ROUTED.md](acceptance/ROUTED.md), routed to a change of its own, and does not + block. The change is judged on what it changed; the document is proved by the keyed questions, + which every run answers over the whole of it. + ## Pass criteria -The change passes when all of the following hold on a single run: +The change passes when all of the following hold on a single run. The thresholds apply to the +readers whose answers count; with five readers, "every reader" is five of five and "all but one" +is four of five. -| Criterion | Threshold | -| ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | -| Load-bearing statements (question 2) | **5 of 5** readers correct on every one | -| Restatement and application (questions 1 and 3) | **At least 4 of 5** readers correct on each keyed item | -| A wrong answer traceable to the text | **None.** A miss whose reasoning quotes the document is a finding, whatever the count | -| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it | -| Anything from one reader only | Recorded; applied at the editor's discretion; does not block | +| Criterion | Threshold | +| ------------------------------------------------------------------------------------- | -------------------------------------------------------------- | +| Load-bearing statements (question 2) | **Every** reader correct on every item | +| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | +| A miss the key says counts against the document | **None**, whatever the count | +| A passage found confusing by two or more readers, in text the change wrote or altered | **None.** It is a finding and blocks until the text is changed | +| A passage found confusing by two or more readers, elsewhere | Recorded in `acceptance/ROUTED.md` and routed; does not block | +| Anything from one reader only | Recorded; applied at the editor's discretion; does not block | The asymmetry is deliberate. A load-bearing statement is the sentence the whole document exists to make, and one reader in five getting it wrong means one agent in five will act on the wrong reading; unanimity is the only acceptable result there. Everywhere else a single divergent reader -is more likely to be the reader than the text, so four of five is tolerated, but a miss that the -reader can justify from the document is always the document's fault. - -## The bar only moves up - -No criterion in this document is ever relaxed to let a change through. If the cohort is confused, -the test is working and the text is wrong; the fix is to the text, never to the threshold, the -question, the fixture or the key. A threshold may be raised, a keyed question may be added, and a -key may be corrected where it misquoted the document; none of those makes a failing run pass. The -one time a threshold was relaxed, after three runs, the relaxation was reversed on the author's -instruction and is recorded here so that it is not tried again. Removing the clarity rating and -the request for suggestions was not a relaxation: it changed what the cohort is asked, not how -strictly an answer is marked. +is more likely to be the reader than the text, so one such reader is tolerated, unless the key +says the text invites that reading. A finding is settled run by run: a passage two readers named +in one run has no standing in the next, which is judged on its own readers, and a passage the +fresh cohort names by fewer than two is clear. + +## Changing the test + +Nothing in this test is changed to let a run through. A threshold, a question, a key or a +fixture is changed only before a run's readers start, never changes a mark already given, and +is recorded with its reason: a threshold, a question or the rules here by a change to this +process, a key by step 4 of `CONTRIBUTING.md` when the text it quotes is reworded or when a +load-bearing obligation is added, and each in [acceptance/CHANGES.md](acceptance/CHANGES.md). A +question is reworded only so that it asks for what its key requires, and a key's required +elements are never more than its question asks for. A run made after such a change is a fresh +run under the new rules, not a re-mark of an old one. The one time a threshold was relaxed to +let a change through, after three runs, the relaxation was reversed on the editor's instruction +and is recorded here so that it is not tried again. Removing the clarity rating and the request +for suggestions was not that: it changed what the cohort is asked, not how strictly an answer is +marked. ## After a failure -Fix the text, never the questions, and run a fresh cohort. Readers are not reused across runs. -The number of runs it took is recorded; a document that needed three runs is telling its editor -something. +Fix the text the findings name and run a fresh cohort; readers are never reused. A keyed miss is +fixed in the text that carries the obligation, wherever the change left it. A finding on text the +change wrote or altered is fixed there. A finding elsewhere goes to `acceptance/ROUTED.md` and +is not fixed in this change. The number of runs it took is recorded; a document that needed three +runs is telling its editor something. ## When it runs -Once per pull request, on the final text, after the editor's rewrite and any key update. It is -the last gate before the merge to `next`, so that every change on `next` is one the cohort has -accepted, and the release branch needs no cohort of its own unless it changes specification text. -A wording change after the run means a fresh run; readers are never reused. +On the final text of each pull request, after the editor's rewrite and any key update, one cohort +per changed document. It is the last gate before the merge to `next`, so that every change on +`next` is one the cohort has accepted, and the release branch needs no cohort of its own unless it +changes specification text. A fresh cohort runs after every change to the pull request's own +text that follows a run, and after every void run; a text that failed is never run again +unchanged. A version number or date that `PUBLISHING.md` or step 5 of `CONTRIBUTING.md` has the +editor set is not such a change, and neither is a merge of `next` into the branch that brings in +text another cohort accepted and alters no sentence this change wrote. ## The record -Each run has a file under `acceptance/runs/`, named `NNN-.md` with `NNN` the next number in -the one sequence shared by the three documents, stating: the document, the pull request and the -commit read, the cohort size and model, the run number in that pull request, the marks per -reader per question, the findings applied, and whether it passed. The changelog line for the -change names the file. CI refuses a specification change whose pull request adds no run file; -it checks for at least one, and a pull request that changes two documents adds a run for each. - -Each published version's changelog entry carries the lines of the changes it contains, so a -version's cohort record is the set of runs those lines name. A version with no such line has not -been accepted and MUST NOT be described as published. The question sets and answer keys live -under `acceptance/` in this repository so that a run can be repeated and compared. +Each run has a file under `acceptance/runs/`, named `NNN--run-.md` with `NNN` the next +number in the one sequence shared by the three documents and `` the run number within the pull +request, stating: the document; the pull request and the commit read; the cohort size and model, +with the model version each reader reported; how the readers were isolated and the command they +were started with; any reader set aside and why; who marked and, for a delegated marker, how it +was started; the marks per reader per keyed item, with each miss and the reader's reasoning in +brief; every passage named under question 4, with the words quoted or the description given, its +clause and the readers naming it; each finding and whether it lies in text the change wrote or +altered; any point the rules did not settle; whether the run was made under a process change +before it merged and, if so, at which commit; the findings applied afterwards; the verdict; and +a link to the answer sheets. + +The answer sheets are not kept in this repository. Every reader's whole output as its process +returned it, the exact prompt every reader received, the command the readers were started with, +and the instruction a delegated marker was given are posted in one or more comments on the pull +request, linked from the record and never edited, hidden or deleted after posting. + +The changelog line for the change names the record. CI refuses a specification change whose pull +request adds no run file; it checks for at least one, and a pull request that changes two +documents adds a run for each. Each published version's changelog entry carries the lines of the +changes it contains, so a version's cohort record is the set of runs those lines name. A version +with no such line has not been accepted and MUST NOT be described as published. The question sets +and answer keys live under `acceptance/` in this repository so that a run can be repeated and +compared. + +`acceptance/ROUTED.md` lists every finding routed from a run, open until a change that fixes +its passage passes a cohort, or until the editor closes it with a recorded reason. The editor +reads that list before cutting a release. + +## Changes to this process + +The process files are this file, `acceptance/README.md`, `acceptance/runs/README.md`, +`acceptance/ROUTED.md`, `CONTRIBUTING.md` and the checks that enforce them: `tools/changes.py`, +the key-quotation checks in `spec-qa.py`, the CI workflow and the pull request template. A change +to any of them, or to a question set, is a change to this process. They are not specifications, +and a change to them is not tested by a cohort: the cohort measures whether a low-strength reader +can follow a specification, and the process is followed by the editor. Such a change names the +runs or the case that showed the need for it, and goes through steps 1, 3 and 6 of +`CONTRIBUTING.md`, with no scope, key or cohort step; its review reads every process file it +changes. It applies to every run whose readers start after it merges; a run made before then may +be made under it, following all of it at a commit named in the record before the readers start. +It is recorded by a line added at the top of [acceptance/CHANGES.md](acceptance/CHANGES.md), +naming its pull request, not in the changelog, which is for the three specifications. A change to +a key or a fixture follows `acceptance/README.md`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fa1743e..013bb9d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,7 +29,9 @@ review. It is a patch version when released. SHOULD or MAY in a numbered section or appendix. It needs everything a clarity edit needs, plus the five tests in SCOPE.md answered in the pull request, and it is at least a minor version when released. `tools/changes.py` tells the two apart mechanically and CI runs it on every pull -request. +request. A change to the process files ACCEPTANCE.md names, this file among them, is neither: +ACCEPTANCE.md says how it is made, and it is recorded in `acceptance/CHANGES.md`, not in +CHANGELOG.md. ## What a pull request contains @@ -55,20 +57,26 @@ survives, the change is yours. the editor sets the pre-release header. 2. **Scope.** For an obligation, the editor confirms each SCOPE.md test. A failed test closes the pull request with a row in `DECLINED.md`. -3. **Review and rewrite.** An independent review of the diff against the document, then the - editor's rewrite, pushed to the pull request branch where the contributor allows it or to an - editor's branch that supersedes it. +3. **Review and rewrite.** An independent review of the diff against the whole document, by a + reviewer who did not draft the change, then the editor's rewrite, pushed to the pull request + branch where the contributor allows it or to an editor's branch that supersedes it. The pull + request names the reviewer and links or contains the review. 4. **Keys.** Where the change touches a sentence an acceptance key quotes, or adds a load-bearing - obligation, the editor updates the key before the cohort runs, never after. -5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text. Its record is - committed under `acceptance/runs/`, the `Unreleased` changelog line names it, and CI refuses - a specification change that lacks one. For an obligation, the editor sets the document's - header to `-dev, unpublished` in the same commit if it still names a published - version. A wording change after the run means a fresh run. + obligation, the editor updates the key before the cohort runs, never after. The key's + required elements stay what its question asks for; `acceptance/README.md` says what a key + update may and may not do. +5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text, one cohort per + changed document. Its record is committed under `acceptance/runs/`, the `Unreleased` + changelog line names it, and CI refuses a specification change that lacks one. For an + obligation, the editor sets the document's header to `-dev, unpublished` in the + same commit if it still names a published version. A change to the pull request's own text + after the run means a fresh run, as ACCEPTANCE.md says under "When it runs". A finding the + run routes to `acceptance/ROUTED.md` is not fixed in this pull request. 6. **Merge** into `next` by merge commit, and the branch is deleted. -Steps 2 to 5 are done by a person with a Claude Code session, not by CI, and they happen in -batches. A quiet pull request is not a declined one; a declined one says so. +Steps 2 to 5 are the editor's, not CI's, and they happen in batches. The editor may have an +agent draft, review or mark, as ACCEPTANCE.md allows for marking; every decision in these steps +stays the editor's. A quiet pull request is not a declined one; a declined one says so. ## Versions and releases diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md new file mode 100644 index 0000000..a076ff1 --- /dev/null +++ b/acceptance/CHANGES.md @@ -0,0 +1,36 @@ +# Changes to the acceptance process + +Changes to the process files that [ACCEPTANCE.md](../ACCEPTANCE.md) names, recorded here as it +describes: a line is added at the top for each change, naming its pull request. The process is +not a specification and is not versioned, and a change to it is not tested by a cohort. Changes +to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). + +- Pull request NNN: the test is made to prove what it exists to prove, the clarity of the + document to a cold reader, after thirty-four runs on two changes (19 to 30, 31 to 33 and 35 + to 52) in which keyed answers reached five of five and the runs still failed. Four defects in + the process were found. Keys required elements the question never asked for (detector 2b's + "need not remove or forbid", detector 2c's "at a version tracked together", and on pull + request 14 method 2d's order of the kinds), so readers who answered the question as put were + marked wrong: now each key lists, per item, the elements an answer must state, one for + each part of the question, and the questions are reworded to ask for exactly those; anything + else in the key only catches a wrong assertion. A passage two readers found confusing stayed + open whilst any single reader of a later cohort named any part of its paragraph, so cleared + passages kept coming back: now a finding is settled run by run, raised by two readers of one + run and clear when the next names it by fewer than two. A finding landed on text the change + never touched, so a change's cohort acted as a review of the whole document and could not + converge: now a finding in text the change wrote or altered blocks the change, and a finding + elsewhere is routed to `acceptance/ROUTED.md` and to a change of its own, whilst the keyed + questions still prove the whole document on every run. Markers invented judgements the rules + left open and later runs cited them as precedent: now an open point is marked for the reader + and settled in this file or the key before the next run, and a run record states no judgement + the rules do not. The rule that any miss whose reasoning quoted the document was a finding + whatever the count is replaced by the key naming the wrong readings the text invites, which + count against the document on one reader, every other single miss being tolerated by the + all-but-one threshold. Question 2's instruction asks for the sentence or sentences relied on, + since three keys rest on two. The cohort, the packet, set-aside readers, the posted sheets, + the fuller record and the route for a process change are kept from the earlier proposal in + pull request 13, which this change supersedes. Relative to the rules on `next`, the + traceable-miss row is narrower and the confusion row is applied to the change rather than + the document; both are recorded here as the reason the earlier rules could not converge, and + neither changes a mark already given. Prompted by runs 19 to 30 on pull request 14 and runs + 31 to 52 on the defined-terms change. diff --git a/acceptance/README.md b/acceptance/README.md index 4cfb6ec..49a097b 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -2,8 +2,10 @@ The question sets and answer keys that [ACCEPTANCE.md](../ACCEPTANCE.md) requires. One directory per specification document, each holding the fixed questions a cold reader answers and the key an -editor marks against. The fixture cases the application question uses live in `fixtures/`, and -the record of every run lives in `runs/`. +editor marks against. The fixture cases the application question uses live in `fixtures/`, the +record of every run lives in `runs/`, findings routed out of a run to a change of their own are +listed in [ROUTED.md](ROUTED.md), and changes to the acceptance process itself are recorded +in [CHANGES.md](CHANGES.md). | Document | Questions | Key | | ------------------ | ------------------------------------------------ | ------------------------------------ | @@ -14,24 +16,47 @@ the record of every run lives in `runs/`. ## How a run works 1. The editor identifies which documents the change touched. Only those are tested. -2. For each, five or more fresh low-strength readers each receive the document's current text, - its `QUESTIONS.md`, and the fixture the questions name. Nothing else: no key, no changelog, no - critique findings, no other reader's answers. -3. The editor marks each answer sheet against `KEY.md` and applies the pass criteria in - `ACCEPTANCE.md`. -4. The result is recorded as a file under [runs/](runs/), with cohort size, model, run number, - marks and the findings applied, and the change's `Unreleased` changelog line names that file. - A run that fails is followed by a change to the text, never to the questions, and a fresh - cohort. +2. For each, an odd number of fresh low-strength readers, at least five, each isolated as + `ACCEPTANCE.md` describes, receive the document's current text, its `QUESTIONS.md`, and the + fixture the questions name. Nothing else: no key, no changelog, no critique findings, no other + reader's answers. +3. The editor, or a separate marker the editor delegates to, marks each answer sheet against + `KEY.md` under the marking rules in `ACCEPTANCE.md` and applies its pass criteria. +4. The result is recorded as a file under [runs/](runs/), stating what `ACCEPTANCE.md` lists, + with the answer sheets posted on the pull request and linked from it, and the change's + `Unreleased` changelog line names that file. A run that fails is followed by a change to the + text the findings name and a fresh cohort; a finding on text the change did not write or + alter is added to [ROUTED.md](ROUTED.md) instead. + +## How a key is written + +A key says, for each keyed item, exactly what is marked and nothing a marker has to guess: + +- **Question 1**: the obligations a bullet may name, and the assertions that make a bullet wrong. +- **Question 2**: for each item, the elements an answer is required to state, numbered, one for + each part of the question; the assertions that make an answer wrong; and the sentence or + sentences of the document the answer rests on, quoted verbatim. The required elements are + what the question asks for, no more: an element the question does not ask for is not + required, however load-bearing, until the question is reworded to ask for it. +- **Question 3**: for each item, the grade or grades accepted, any condition on the reasoning, + and the wrong readings the text invites, which count against the document when one reader + takes them. + +Everything else in a key is explanation for the marker and never marks an answer down. ## Keeping the keys honest A key quotes the document. When a clause is reworded, the key line that quotes it is updated in the same commit, and the change is one the acceptance run itself then tests. A key that no longer -matches the document is a defect in this directory, not a reason to mark a reader wrong. +matches the document is a defect in this directory, not a reason to mark a reader wrong. A key +update under step 4 of `CONTRIBUTING.md` changes a quotation to follow the text, or adds a keyed +item for a new load-bearing obligation; it does not change what an existing item requires. A +change to what an item requires, or to a question, is a change to the process and is recorded in +[CHANGES.md](CHANGES.md) with its reason, before the readers of the next run start. Questions are fixed so that runs can be compared across versions. Adding a keyed question is -allowed when a new clause is load-bearing; removing or softening one to make a run pass is not. +allowed when a new clause is load-bearing; removing or softening one to make a run pass is not; +rewording one is allowed so that it asks for what its key requires, and for no other reason. The reader may quote a passage it found confusing; it is never asked to rate the document, say what would make a passage clearer, or propose a change: the cohort confirms comprehension and does not design the text. diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md new file mode 100644 index 0000000..bb82718 --- /dev/null +++ b/acceptance/ROUTED.md @@ -0,0 +1,32 @@ +# Passages routed out of a cohort + +A finding is a passage two or more readers of one cohort named as confusing, under question 4, +that lies outside the text the change being tested wrote or altered. `ACCEPTANCE.md` routes such +a finding here rather than blocking the change whose cohort raised it, so that each change is +judged on what it changed and the document is still made clearer, one change at a time. The +editor reads this list before cutting a release. + +A finding is open until a change that reworks its passage passes a cohort, when the line names +that run and moves below the rule, or until the editor closes it with a reason recorded here. A +finding raised again by a later run is noted on its line, not listed twice. + +## Open + +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | -------------------------------------- | +| `TOOLING-SPEC.md` | 9 and 9.1 | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | 2, 4.2, 8.1 | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | 9.2 | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | 4.3 | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | run 33 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | 4.1 | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | 5.3 | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | open | +| `DETECTOR-SPEC.md` | 7.1 | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | 8.1 | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | 5.3 | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | +| `SPEC.md` | 1 (Owner) | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50 | open | +| `SPEC.md` | 4 | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 (one reader, 29) | open; a method question for the editor | +| `SPEC.md` | 2 and 3.1 | "technique" used for a way of writing the Rule in section 2 and for a way of searching in clause 3.1 | runs 29, 38 | pull request 14 rewords both | + +## Closed + +None yet. diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 132e6cb..b0a3f97 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -1,8 +1,9 @@ # Answer key for DETECTOR-SPEC.md -Marked against the document at the version named in the changelog entry that records the run. -Quotations below are from the document; when a clause is reworded, update the quotation here in -the same commit. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, +update the quotation here in the same commit. `ACCEPTANCE.md` says how each item is marked; +`acceptance/README.md` says what each part of this key is for. ## 1. Restatement @@ -11,67 +12,93 @@ bullet asserting an obligation the document does not state, is a correct restate - Support bespoke rules written by the project ([4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it)). - Provide a harness that runs a single rule against supplied code ([4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code)). -- Allow a stable identifier per rule and print it, unaltered, with every finding ([4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding)). +- Allow a stable identifier per rule and print it, unaltered, with every finding ([4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding)); a + bullet that says each rule carries a stable identifier names this obligation. - Be invocable locally by the practitioner, over a subset down to one file, with the result in - the command's own output and never only through a hosted service or CI-only mode ([5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally)). + the command's own output and never only through a hosted service or CI-only mode ([5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally)); + a bullet that says every finding, or each finding, must appear in that output names [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran), + whose summary may leave findings out, and is not marked wrong for the plural. - Resolve a printed identifier to documentation; for bundled rules from the installed copy, offline, with the documentation shipped and versioned with the rule ([6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) to [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together)). - Make any inline suppression route detectable or disableable ([7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)). Wrong if it asserts: the detector must enforce a project's suppression policy; the detector must declare a version to conform; the detector must list a project's active defences or produce an -agent summary (those belong to the toolchain specification). +agent summary (those belong to the toolchain specification). A bullet that states what +conformance means, or restates a SHOULD as advice, asserts no obligation and is neither correct +nor wrong. + +Wrong readings the text invites, which count against the document on one reader: none named. ## 2. Load-bearing statements -- **2a.** Conformance is satisfying every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression), judged on evidence; a detector - that predates the document can conform, and the declaration is optional. Key quotation, section - 8: "A Detector Conforms if it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression)." Supporting, [8.1](../../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps): "A - maintainer MAY declare the version of this document the detector conforms to". An answer that - says a declaration is required, or that partial satisfaction is conformance, is wrong. -- **2b.** It must make the route either disableable by configuration or detectable by a rule the - project can write or a mechanical check it documents; it is not required to remove or forbid - it. Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each one it MUST do at least one of two things, and either alone - satisfies this clause: make the route disableable by configuration; or make the route - detectable, by a Rule the project can write in the Detector itself or by a mechanical check the - Detector documents". An answer that says the detector must forbid the route, or must require a - reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD), is wrong. -- **2c.** With the rule, in the installed copy, at a version tracked together, and yes, offline. - Key quotations, [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access): "Resolution of a bundled rule's identifier MUST work from the installed - copy, without network access"; [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together): "A bundled rule's documentation MUST ship with the rule, at - a version tracked together". An answer naming the detector's website as an acceptable location - is wrong. +- **2a.** Required: (1) a detector conforms when it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression); + (2) yes, a detector that predates the document can conform, because conformance is judged on + the evidence and a declaration is not a condition of it. Wrong if it says a declaration is + required, or that partial satisfaction is conformance. Key quotation, section [8](../../DETECTOR-SPEC.md#8-conformance): "A Detector + Conforms if it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression)." Supporting, [8.1](../../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps): "A maintainer + MAY declare the version of this document the detector conforms to". "Judged on evidence", "the + declaration is optional" or "graded on evidence by anyone" each state the reason in element + (2); one of them is enough. +- **2b.** Required: (1) it must let the project switch the route off in its configuration, or + make the route detectable (by a rule the project can write, or by a mechanical check the + detector documents; naming either means of detection, or just "detectable", states this); + (2) yes, either one alone is enough. Wrong if it says the detector must remove or forbid the + route, or must require a reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD). Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each one it + MUST do at least one of two things, and either alone satisfies this clause: make the route + disableable by configuration; or make the route detectable, by a Rule the project can write in + the Detector itself or by a mechanical check the Detector documents". An answer need not say + that the detector is not required to remove or forbid the route; that follows from element (2) + and the question does not ask it. +- **2c.** Required: (1) with the rule, in the installed copy; (2) at a version tracked together + with the rule; (3) yes, it must be reachable without network access. Wrong if it names the + detector's website, or any location reached over the network, as acceptable. Key quotations, + [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access): "Resolution of a bundled rule's identifier MUST work from the installed copy, without + network access"; [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together): "A bundled rule's documentation MUST ship with the rule, at a version + tracked together". "Shipped with the rule", "in the installed package" or "on disk with the + detector" each state element (1); "versioned with the rule" or "at the same version" states + element (2). ## 3. Application: Lintomatic 4.2 -| Clause | Intended verdict | Reasoning the fixture supports | -| ------ | ---------------- | -------------------------------------------------------------------------------------------------------------------- | -| [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules via the `Rule` interface, first class in every report | -| [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `check --only ` runs one rule, bundled or project, with a distinguishing exit code | -| [4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `LM-NNNN` fixed at first release; project names taken as given, not rewritten; printed on every finding line | -| [5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | Local, no account, no server, no network | -| [5.2](../../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Yes | `check ` accepts a single file | -| [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | One line per finding on standard output whether or not the HTML report is enabled | -| [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | The hosted dashboard shows nothing the command line does not print | -| [6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Yes | `explain ` resolves a printed identifier; project rules resolve to their declared `doc` string | -| [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Yes | `explain` reads the installed package; there is no website | -| [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | No | Two of forty bundled rules print `No documentation declared`; every identifier a bundled rule can print MUST resolve | -| [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | The ignore comment is disableable with `ignore_comments = false` | - -Section [8](../../DETECTOR-SPEC.md#8-conformance) verdict: **does not conform**, because [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) fails and partial conformance MUST NOT be -described as conformance. A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Partial on the grounds that thirty-eight resolve -has read the clause differently and that counts against the document; a reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Yes -because they missed the last paragraph of the fixture has misread the fixture, not the document. - -Clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation): **does not change the verdict**. It is a SHOULD, and Lintomatic's failure to check its -own identifiers in CI is a missed SHOULD, not a further MUST failure; the MUST that fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). - -A reader who marks [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) No because the comment takes no reason has confused [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) with [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason), which is -SHOULD; that counts against the document if the reasoning quotes [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) or [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason). +| Clause | Accepted grade | Condition on the reasoning, and what the fixture supports | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules via the `Rule` interface, first class in every report | +| [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `check --only ` runs one rule, bundled or project, with a distinguishing exit code | +| [4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `LM-NNNN` fixed at first release; project names taken as given, not rewritten; printed on every finding line | +| [5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | Local, no account, no server, no network | +| [5.2](../../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Yes | `check ` accepts a single file | +| [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | One line per finding on standard output whether or not the HTML report is enabled; a reader who notes that the HTML report need not be named for that reason is right | +| [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | The hosted dashboard shows nothing the command line does not print | +| [6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Yes | `explain ` resolves a printed identifier; project rules resolve to their declared `doc` string; the two undocumented bundled rules are [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together)'s failure | +| [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Yes | `explain` reads the installed package; there is no website | +| [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | No | Two of forty bundled rules print `No documentation declared`; every identifier a bundled rule can print MUST resolve | +| [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | The ignore comment is disableable with `ignore_comments = false` | + +No grade above carries a condition on the reasoning: the accepted grade is correct whatever the +reasoning, and a reader who adds a caveat about what the fixture does not say and still gives +the accepted grade is correct. + +Section [8](../../DETECTOR-SPEC.md#8-conformance) verdict: **does not conform**; a reader is correct who says so and names [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), the +two undocumented bundled rules, or partial conformance not counting, as the reason. + +Clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation): **does not change the verdict**; a reader is correct who says so and gives as the +reason that it is a SHOULD, or that the MUST which fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). + +Wrong readings the text invites, which count against the document on one reader: + +- [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) graded Partial on the grounds that thirty-eight of forty resolve, which has read the + clause as admitting a proportion. +- [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) graded No because the comment takes no reason, where the reasoning quotes [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) or [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason), + which has read [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason)'s SHOULD into [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable). + +A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Yes because they missed the last paragraph of the fixture has misread the +fixture, which is the reader's miss. ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage named, as `ACCEPTANCE.md` defines naming and the same +passage. A passage named by two or more readers of the run is a finding: it blocks the change +where it lies in text the change wrote or altered, and is routed to `acceptance/ROUTED.md` +otherwise. What to change is the editor's decision; the reader is not asked and any suggestion +it volunteers is discarded unread. diff --git a/acceptance/detector/QUESTIONS.md b/acceptance/detector/QUESTIONS.md index 58d06d1..2666b2f 100644 --- a/acceptance/detector/QUESTIONS.md +++ b/acceptance/detector/QUESTIONS.md @@ -10,14 +10,15 @@ In your own words and in at most six bullets, what does this document require of ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. +Answer each part of each question in a sentence or two, then quote the sentence or sentences of +the document you are relying on. - 2a. What does it mean for a detector to conform, and can a detector that was written before this document existed, and has never heard of it, conform? - 2b. A detector ships an inline ignore comment. What must the detector do about that route for - it to conform? -- 2c. Where must the documentation for a bundled rule's identifier live, and must it be - reachable without network access? + it to conform, and is doing one of those things enough? +- 2c. Where must the documentation for a bundled rule's identifier live, at what version, and + must it be reachable without network access? ## 3. Application diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 3ca1a80..182d6b2 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -1,8 +1,9 @@ # Answer key for SPEC.md -Marked against the document at the version named in the changelog entry that records the run. -Quotations below are from the document; when a clause is reworded, update the quotation here in -the same commit. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, +update the quotation here in the same commit. `ACCEPTANCE.md` says how each item is marked; +`acceptance/README.md` says what each part of this key is for. ## 1. Restatement @@ -18,62 +19,80 @@ bullet asserting an obligation the document does not state, is a correct restate with the rule ([3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation)). - Record a defect whose fix is deferred where the project's other decisions are enumerable, rather than only mentioning it (section [2](../../SPEC.md#2-when-the-method-applies)). +- Take a baseline, a suppression or an instance left unfixed to the owner, never on the + practitioner's own authority (section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom)). Wrong if it asserts: the practitioner fixes the reported instance first and defends afterwards; a test may serve as the detector; an agent may baseline the remaining instances. +Wrong readings the text invites, which count against the document on one reader: none named. + ## 2. Load-bearing statements -- **2a.** Attribute the defect to a class, express it as a detector rule, prove the rule fires, - then sweep and fix every instance, in that order. Key quotations, section [3](../../SPEC.md#3-the-method): "Six clauses, in +- **2a.** Required: (1) the four steps before the fix: attribute the defect to a class, express + the class as a rule in a detector, prove the rule fires, and sweep the codebase; (2) in that + order, with the fix after the sweep. Wrong if it has the fix before the rule, or the sweep + before the proof, or a test as the detector. Key quotations, section [3](../../SPEC.md#3-the-method): "Six clauses, in order."; [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance): "The Practitioner MUST run the Rule across the entire codebase and record the total Instance count before fixing anything, and MUST then fix every Instance found rather - than only the one". An answer that has the fix before the rule, or the sweep before the proof, - is wrong. -- **2b.** The owner of the codebase, always a human; an agent may not. Key quotation, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): - "An Agent MUST NOT Baseline, suppress or knowingly leave an Instance unfixed on its own - authority, whatever the Instance count turns out to be." An answer that lets the practitioner - decide because the count was small, or because the project recorded no rule, is wrong (the - default owner paragraph in the terminology entry for Owner still requires a recorded decision). -- **2c.** All six clauses of section [3](../../SPEC.md#3-the-method) were followed for that defect, and the verdict rests on the - reviewer reproducing the red and green runs, not on the report. Key quotations, section [7](../../SPEC.md#7-conformance): "A - remediation Conforms if all six clauses of section [3](../../SPEC.md#3-the-method) were followed for that Defect, and the - verdict that they were rests on reproduction"; "A verdict on a remediation or a Defence MUST - rest on reproduction, not on the report". An answer that accepts the pull request description - as the verdict is wrong, and so is one that states the six clauses and says nothing of - reproduction. + than only the one". An answer that names the steps by their clause numbers in order states + both elements. +- **2b.** Required: (1) the owner of the codebase, the human who holds that authority; (2) no, + an agent may not do so on its own authority. Wrong if it lets the practitioner or an agent + decide because the count was small, because the project recorded no owner, or at all. Key + quotation, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): "An Agent MUST NOT Baseline, suppress or knowingly leave an Instance + unfixed on its own authority, whatever the Instance count turns out to be." "The owner" names + the defined term and states that the decision is a human's; an answer need not add "a human" + to be correct. The default owner paragraph in the terminology entry for Owner still requires a + recorded decision, so an answer naming whoever instructed the practitioner, for a project that + has recorded nobody, names the owner and is correct. +- **2c.** Required: (1) all six clauses of section [3](../../SPEC.md#3-the-method) were followed for that defect; (2) the verdict + rests on reproduction, the reviewer rerunning the defence, not on the report. Wrong if it + accepts the pull request description or the practitioner's report as the verdict, or says + that most clauses suffice. Key quotations, section [7](../../SPEC.md#7-conformance): "A remediation Conforms if all six clauses + of section [3](../../SPEC.md#3-the-method) were followed for that Defect, and the verdict that they were rests on + reproduction"; "A verdict on a remediation or a Defence MUST rest on reproduction, not on the + report". "Reproduction" or "reproducing the runs" states element (2); the red and green runs + need not be named. ## 3. Application: the ledger remediation -| Clause | Intended verdict | Reasoning the fixture supports | -| ----------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | Class, hazard and the next wider rule are named; Partial or No is accepted where the reader notes that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | -| [3.2](../../SPEC.md#32-build-the-net) | Yes | A rule in the project's static analyser, reading code, not a test | -| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | Commit B is red on the fixture and the originating instance and survives as its own commit | -| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Five findings recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority; Partial accepted where the reasoning names that suppression | -| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | The rule is permanent and blocking, but the suppression was not a recorded decision; [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | -| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule; Partial accepted where the reasoning quotes the versioning sentence | +| Clause | Accepted grades | Condition on the reasoning, and what the fixture supports | +| ----------------------------------------------------------------------------------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | No condition. Class, hazard and the next wider rule are named; Partial or No is the reading that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | +| [3.2](../../SPEC.md#32-build-the-net) | Yes | A rule in the project's static analyser, reading code, not a test | +| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | Commit B is red on the fixture and the originating instance and survives as its own commit | +| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | The reasoning names the instance left unfixed behind an inline ignore on the agent's own authority; five findings were recorded | +| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | The reasoning names the suppression no recorded decision covers; the rule is otherwise permanent and blocking. [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | +| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | The reasoning names the documentation on a wiki, not versioned with the rule; the message is terse and carries an identifier | -Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**, because [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) and [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) were not followed. The -**defence does not conform** either, because [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) fails; a reader who says the defence conforms -because the sweep is what failed has missed [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) and that counts against the document only if the -reasoning quotes the section [7](../../SPEC.md#7-conformance) definition of a conforming defence. +Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**; a reader is correct who says so and names +[3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) or [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the clauses not followed. The **defence does not conform** either; a reader is +correct who says so and names [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the reasons, or [3.5](../../SPEC.md#35-enforce-permanently-and-block)'s unrecorded suppression, which +section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence also requires to be absent. Section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): the **inline ignore on `Report::render()`** was the owner's decision, not the -practitioner's. An answer naming instead the decision not to build the wider rule is half right: -that is also listed in section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) as the owner's to take, and an answer naming both is correct. -An answer naming neither is wrong. +practitioner's. A reader who names it is correct, with or without the decision to leave the +wider rule unbuilt, which section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) also gives to the owner. A reader who names only the wider +rule, or neither, is wrong. + +Wrong readings the text invites, which count against the document on one reader: + +- [3.5](../../SPEC.md#35-enforce-permanently-and-block) graded Yes because the rule fails the build, which has not applied the clause's opening + sentence on unrecorded suppression. +- [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) graded Yes because the URL resolves, which has read the versioning sentence of [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) + differently. +- The defence graded as conforming because the sweep is what failed, where the reasoning quotes + section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence. -A reader who grades [3.5](../../SPEC.md#35-enforce-permanently-and-block) Yes because the rule fails the build has not applied the clause's opening -sentence on unrecorded suppression, and that counts against the document. A reader who grades -[3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) Yes because the URL resolves has read the versioning sentence of [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) -differently, which counts against the document; a reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the -fixture. A reader who attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing -passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as covering it, which is a fair reading and is not marked wrong. +A reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the fixture, which is the reader's miss. A reader who +attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as +covering it, which is a fair reading and is not marked wrong. ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage named, as `ACCEPTANCE.md` defines naming and the same +passage. A passage named by two or more readers of the run is a finding: it blocks the change +where it lies in text the change wrote or altered, and is routed to `acceptance/ROUTED.md` +otherwise. What to change is the editor's decision; the reader is not asked and any suggestion +it volunteers is discarded unread. diff --git a/acceptance/method/QUESTIONS.md b/acceptance/method/QUESTIONS.md index 072a3c0..34fa82d 100644 --- a/acceptance/method/QUESTIONS.md +++ b/acceptance/method/QUESTIONS.md @@ -11,7 +11,8 @@ do when they are handed a defect? ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. +Answer each part of each question in a sentence or two, then quote the sentence or sentences of +the document you are relying on. - 2a. What does a practitioner do before fixing the defect they were handed, and in what order? - 2b. Who may adopt a baseline or suppress an instance, and may an agent do so on its own diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index 4994f74..3f7ad64 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -1,15 +1,22 @@ # Acceptance run records -One file per cohort run, `NNN-.md`, numbered in the one sequence the three documents -share. Runs 1 to 15 predate this directory and are summarised in the changelog entries for -1.0.0 and 1.0.1 of the method specification, 1.0.0 of the detector specification and 0.2.0 of -the toolchain specification; numbering here continues from 16. +One file per cohort run, `NNN--run-.md`, numbered in the one sequence the three +documents share, with `` the run number within its pull request. Runs 1 to 15 predate this +directory and are summarised in the changelog entries for 1.0.0 and 1.0.1 of the method +specification, 1.0.0 of the detector specification and 0.2.0 of the toolchain specification; +numbering here continues from 16. -A record states, in this order: the document; the pull request and the commit the readers were -given; the cohort size and model; the run number within that pull request; the marks per reader -per question against the key; the passages quoted as confusing and by how many readers; the -findings applied to the text as a result; and the verdict against the criteria in -`ACCEPTANCE.md`. Answer sheets are not pasted in full; the marks are the record. +A record states what `ACCEPTANCE.md` lists under "The record", in this order: the document; the +pull request and the commit the readers were given; the cohort size and model, with the model +version each reader reported; how the readers were isolated and the command they were started +with; any reader set aside and why; who marked and, for a delegated marker, how it was started; +the marks per reader per keyed item, with each miss and the reader's reasoning in brief; every +passage named under question 4, with the words quoted or the description given, its clause and +the readers naming it; each finding and whether it lies in text the change wrote or altered; any +point the rules did not settle; whether the run was made under a process change before it merged +and, if so, at which commit; the findings applied afterwards; the verdict; and a link to the +answer sheets posted on the pull request. Earlier records show the style only; where one differs +from this order, this file wins. Answer sheets are not kept here; the marks are the record. The `Unreleased` changelog line for the change names the record, and `tools/changes.py` refuses a specification change whose pull request adds no file here. diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 5a2602b..c11ae32 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -1,8 +1,9 @@ # Answer key for TOOLING-SPEC.md -Marked against the document at the version named in the changelog entry that records the run. -Quotations below are from the document; when a clause is reworded, update the quotation here in -the same commit. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, +update the quotation here in the same commit. `ACCEPTANCE.md` says how each item is marked; +`acceptance/README.md` says what each part of this key is for. ## 1. Restatement @@ -25,61 +26,72 @@ bullet asserting an obligation the document does not state, is a correct restate defences on its own source ([8.1](../../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation), [8.2](../../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source)). Wrong if it asserts: the toolchain must be a single product; a declaration is required to -conform; the agent summary (section [7](../../TOOLING-SPEC.md#7-agent-context)) is a MUST. +conform; the agent summary (section [7](../../TOOLING-SPEC.md#7-agent-context)) is a MUST. A bullet that calls agent support or the +self-audit a further condition "where it applies", or says a declaration is not a condition, +asserts no obligation and is neither correct nor wrong. + +Wrong readings the text invites, which count against the document on one reader: none named. ## 2. Load-bearing statements -- **2a.** At the project level; the toolchain is whatever the project has assembled from - third-party, first-party and project-level parts. Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "A project's - Toolchain Conforms if every MUST in sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record) holds across the assembled parts, wherever - each part came from". An answer that says conformance is graded per tool, or that only a - packaged product can conform, is wrong. -- **2b.** One grade; which part satisfied each clause is evidence, not a grade of its own. Key - quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "The project-level verdict is a single grade; which part of the Toolchain - satisfied each clause is evidence for that grade, not a second grade." -- **2c.** Two, graded and declared separately, and neither implies the other. Key quotation, - [9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately): "The two verdicts MUST be graded and declared separately, and neither implies the other." - An answer giving one grade, or saying the artefact grade follows from the project grade, is - wrong. +- **2a.** Required: (1) at the project level; (2) the toolchain is whatever the project has + assembled to run its checks through, from third-party, first-party and project-level parts. + Wrong if it says conformance is graded per tool, or that only a packaged product can conform. + Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "A project's Toolchain Conforms if every MUST in sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record) + holds across the assembled parts, wherever each part came from". An answer that says everything the + project runs its checks through, or the assembled parts from wherever they came, or that lists + the three kinds of part, states element (2). +- **2b.** Required: (1) one grade; (2) which part satisfied each clause is evidence for that + grade, not a grade of its own. Wrong if it gives a grade per part. Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): + "The project-level verdict is a single grade; which part of the Toolchain satisfied each + clause is evidence for that grade, not a second grade." +- **2c.** Required: (1) two, graded and declared separately; (2) neither implies the other. + Wrong if it gives one grade, or says the artefact grade follows from the project grade or the + reverse. Key quotation, [9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately): "The two verdicts MUST be graded and declared separately, and + neither implies the other." ## 3. Application: the Zigpipe project -| Clause | Intended verdict | Reasoning the fixture supports | -| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | Lintomatic fails detector [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) on two undocumented bundled rules and nothing in the wrapping closes that gap | -| [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)` | -| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | Ignore comments disabled project-wide; exceptions only through the record the toolchain reads | -| [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | `zig-qa run` is local, prints Lintomatic's output unchanged, one line per finding | -| [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | Lintomatic, then the tests, stopping on a Lintomatic failure | -| [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes, Partial or No | `zig-qa list` prints identifier, summary and documentation path without running the rules; Partial or No is accepted where the reasoning counts the two `(none)` routes, since 5.1 says the listing MUST provide the route and [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) point 2 says a gap may be named again by a later clause | -| [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | Derived from `lintomatic.toml`, not hand-maintained | -| [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | Yes | The `proj.*` rules appear in the same listing | -| [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes | `quality-exceptions.toml` is a defined path that `zig-qa run` loads | -| [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Partial or No | A justification is required and an empty one rejected, but nothing rejects a justification that could be pasted onto any entry | -| [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | `zig-qa list --exceptions` uses the same format and operation as the rule listing | - -Either Partial or No is accepted for [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one); the reasoning must mention the generic-justification -requirement. A Yes on [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) with reasoning that quotes only the first sentence of the clause is a -finding against the document. - -Section [9](../../TOOLING-SPEC.md#9-conformance) verdict: **does not conform**, because [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) and [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) fail; partial conformance MUST NOT be -described as conformance. **Not with agent support**, because section [7](../../TOOLING-SPEC.md#7-agent-context) is not met (no summary). -**Section [8](../../TOOLING-SPEC.md#8-self-audit) does not apply**, because Zigpipe ships nothing to other projects; a reader who applies -[8.1](../../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) to Zigpipe on the basis that Zigpipe's tooling includes a shipped package (zig-qa) has read -"where the Toolchain is one the project ships" (section [9](../../TOOLING-SPEC.md#9-conformance)) as reaching the consuming project, and that counts against the document. The -correct reading is that section [8](../../TOOLING-SPEC.md#8-self-audit) bears on zig-qa's own artefact grade, not on Zigpipe. - -Declaration: **does not affect the verdict**. Key quotation, [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance): "The declaration is optional." -A reader who says Zigpipe cannot be graded without one has read [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) wrongly. - -A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes on the grounds that Lintomatic "mostly" conforms has read either [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) -or the detector document's section [8](../../DETECTOR-SPEC.md#8-conformance) differently; it counts against this document only if the -reasoning quotes [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification). A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes because they missed the two undocumented rules -has misread the fixture. +| Clause | Accepted grades | Condition on the reasoning, and what the fixture supports | +| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | Lintomatic fails detector [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) on two undocumented bundled rules and nothing in the wrapping closes that gap; a reader who notes that the detector specification was not supplied and grades from this document's summary of it is correct | +| [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)`. The two are blocking, since their firing stops `zig-qa run`; a reader who grades No on that condition is correct | +| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | Ignore comments disabled project-wide; exceptions only through the record the toolchain reads | +| [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | `zig-qa run` is local, prints Lintomatic's output unchanged, one line per finding | +| [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | Lintomatic, then the tests, stopping on a Lintomatic failure | +| [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes, Partial or No | No condition. `zig-qa list` prints identifier, summary and documentation path without running the rules; Partial or No is the reading that counts the two `(none)` routes, since 5.1 says the listing MUST provide the route and [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) point 2 says a gap may be named again by a later clause | +| [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | Derived from `lintomatic.toml`, not hand-maintained | +| [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | Yes | The `proj.*` rules appear in the same listing | +| [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes or Partial | `quality-exceptions.toml` is a defined path that `zig-qa run` loads. Partial is accepted only where the reasoning says the fixture names a place for exceptions and none for calibrations or conventions, which the project record entry also covers | +| [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Partial or No | The reasoning mentions that nothing rejects a justification that could be pasted onto any entry; a justification is required and an empty one rejected | +| [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | `zig-qa list --exceptions` uses the same format and operation as the rule listing | + +Section [9](../../TOOLING-SPEC.md#9-conformance) verdict: **does not conform**; a reader is correct who says so and names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) or +[4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) among the failing clauses. **Not with agent support**; a reader is correct who says so because +section [7](../../TOOLING-SPEC.md#7-agent-context) is not met, there being no summary. **Section [8](../../TOOLING-SPEC.md#8-self-audit) does not apply**; a reader is correct +who says so because Zigpipe ships nothing to other projects, or because section [8](../../TOOLING-SPEC.md#8-self-audit) bears on +zig-qa's own artefact grade and not on Zigpipe. **The absence of a declaration does not affect +the verdict**; a reader is correct who says so. Key quotation, [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance): "The declaration is +optional." + +Wrong readings the text invites, which count against the document on one reader: + +- [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) graded Yes with reasoning that quotes only the first sentence of the clause, which has + not read the sentence on a generic justification. +- [8.1](../../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) applied to Zigpipe on the basis that its tooling includes a shipped package, which has + read "where the Toolchain is one the project ships" (section [9](../../TOOLING-SPEC.md#9-conformance)) as reaching the consuming + project. +- [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) graded Yes on the grounds that Lintomatic "mostly" conforms, where the reasoning quotes + [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), which has read it as admitting a proportion. +- Zigpipe held ungradeable for want of a declaration, which has read [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) as a condition. + +A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes because they missed the two undocumented rules has misread the +fixture, which is the reader's miss. ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage named, as `ACCEPTANCE.md` defines naming and the same +passage. A passage named by two or more readers of the run is a finding: it blocks the change +where it lies in text the change wrote or altered, and is routed to `acceptance/ROUTED.md` +otherwise. What to change is the editor's decision; the reader is not asked and any suggestion +it volunteers is discarded unread. diff --git a/acceptance/toolchain/QUESTIONS.md b/acceptance/toolchain/QUESTIONS.md index a4da266..35a2b91 100644 --- a/acceptance/toolchain/QUESTIONS.md +++ b/acceptance/toolchain/QUESTIONS.md @@ -11,7 +11,8 @@ toolchain? ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. +Answer each part of each question in a sentence or two, then quote the sentence or sentences of +the document you are relying on. - 2a. At what level is toolchain conformance measured, and what is the toolchain? - 2b. A project's tooling is assembled from a third-party detector, a first-party package and From 36d87d18fe3b2aa586472a45b5888812a6831aae Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 11:43:12 +0100 Subject: [PATCH 02/52] Acceptance process: review 1 findings applied One route for each kind of change to the test (quotation and new item in the specification pull request, everything else a process change); ROUTED.md and runs/ are records, not process files; question 1 defined once; open points marked in a stated direction; quoting needs a six-word run; a finding lies in the change's text when any sentence of its paragraph changed; wrong readings stated as grade plus reasoning; keys ask only what their questions ask (detector 2a, toolchain 2c); the question 3 tables separate the condition from the explanation. Co-Authored-By: Claude Fable 5.1 --- ACCEPTANCE.md | 123 ++++++++++++++++++------------ acceptance/CHANGES.md | 4 +- acceptance/README.md | 17 +++-- acceptance/ROUTED.md | 7 +- acceptance/detector/KEY.md | 17 ++--- acceptance/method/KEY.md | 37 ++++----- acceptance/toolchain/KEY.md | 49 ++++++------ acceptance/toolchain/QUESTIONS.md | 2 +- 8 files changed, 144 insertions(+), 112 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 84ab39b..c5bd8cd 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -51,8 +51,8 @@ makes it wrong; the question asks for exactly that and nothing the key does not asks for nothing the question does not. Each reader answers all of them. 1. **Restatement.** In their own words, what the document asks of its subject, in at most six - bullets. Keyed against the list of MUST clauses: a bullet is correct when it names an - obligation the document actually states. + bullets. Keyed against the key's list of the document's obligations: a bullet is correct when + it names one of them. 2. **Load-bearing statements.** Three or more questions, fixed per document, each asking for a short answer and the sentence or sentences of the document it rests on, quoted. Each has one @@ -97,34 +97,41 @@ answer sheets, and the facts the record needs that the sheets cannot show. Its m unless the editor shows that a rule here was misapplied, and a re-mark says which rule and why. Every mark is given by a rule in this file or in the key, as follows. Where neither settles a -mark, the marker gives the reader the benefit of the doubt, records the point in the run record, -and the editor settles it by a change to this file or to the key before the next run. A run -record states no judgement that is not written here or in the key; a judgement an earlier run -recorded is not a rule. +mark, the marker records the point in the run record and marks it as the open-point rule below +says, and the editor settles it in this file or in the key, as [Changing the test](#changing-the-test) +allows, before the next run. A run record states no judgement that is not written here or in +the key; a judgement an earlier run recorded is not a rule. - **Keyed items.** Question 1 as a whole is one keyed item, and so is each item of question 2 - and each answer of question 3 for which the key gives an intended answer. + and each answer of question 3 for which the key gives an accepted grade or answer. - **Reading an answer.** An answer is read for its sense, not its wording: a paraphrase, a synonym, or a defined term used in its defined sense states what it means. A reader quotes the - document when it gives consecutive words of the document, in the document's order and of any - length, in quotation marks or a block quote, verbatim apart from letter case, quotation-mark or - apostrophe style, and markdown link or emphasis markup. + document when it gives, in quotation marks or a block quote, a run of six or more consecutive + words of the document in the document's order, verbatim apart from letter case, quotation-mark + or apostrophe style, and markdown link or emphasis markup; a quotation that drops or changes a + word quotes the document by its longest such run. - **Question 1** is correct when at least four bullets each name an obligation the key lists and - no bullet asserts one the key marks wrong. + no bullet asserts one the key marks wrong. Several bullets that name different clauses of one + line of the key's list each count. A bullet asserting an obligation that is in neither list is + recorded as an open point for the key and is not counted either way. - **Question 2.** An answer is the reader's sentences and its quotation read together. It is correct when it states every element the key lists as required for that item, in the document's words or the reader's own, asserts nothing the key marks wrong, and quotes the document. An element stated only in the quotation is stated. Anything in the key beyond the required elements and the wrong assertions is explanation for the marker, and an answer is never marked down for leaving it out. -- **Question 3.** Each item is correct when its grade is one the key accepts and its reasoning - mentions what the key says it must mention for that grade. A grade the key accepts without a - condition is correct whatever the reasoning. +- **Question 3.** Each item is correct when its grade is one the key accepts and, where the key + sets a condition on the reasoning for that grade, the reasoning meets it. A grade the key + accepts with no condition is correct whatever the reasoning. A grade given on a stated + condition is read as the grade for the case the fixture presents, where the key says which + case that is; otherwise the item is unanswered. - **A miss that counts against the document.** The key names, for question 1 and for each item - of question 3, the wrong readings the text invites. A miss of that kind blocks whatever the - count, since one reader in five taking that reading means one agent in five acting on it. Every + of question 3, the wrong readings the text invites, each as a grade or answer together with + what the reasoning states or leaves out. A miss that matches one blocks whatever the count, + since one reader in five taking that reading means one agent in five acting on it, and it + blocks wherever the text lies, because the keyed questions are what prove the document. Every other single miss under questions 1 and 3 is the reader's and is tolerated by the threshold - below; a second reader making the same miss is a failed threshold, not a judgement. + below; a second miss on the same item fails the threshold, and no judgement is needed. - **An unanswered question.** Each keyed item left unanswered is a miss. Question 4 left unanswered counts as no passage named. - **Question 4.** A reader names a passage when it quotes words of the document or gives a clause @@ -134,12 +141,15 @@ recorded is not a rule. name lie in the same paragraph; a heading, a list item, a table row and a terminology entry are each a paragraph of their own. A passage named by two or more readers of the run is a finding. - **Where a finding lies.** The record says, for each finding, whether it lies in text the change - wrote or altered: a sentence or paragraph the pull request adds, rewords or moves, read against - the branch it targets. A finding there blocks the change. A finding anywhere else is a finding - against the document, not against the change: it is recorded in - [acceptance/ROUTED.md](acceptance/ROUTED.md), routed to a change of its own, and does not - block. The change is judged on what it changed; the document is proved by the keyed questions, - which every run answers over the whole of it. + wrote or altered: a paragraph in which the pull request, read against the branch it targets, + adds, rewords, moves or deletes a sentence, or changes a word, a term or a link. A finding + there blocks the change. A finding anywhere else is a finding against the document, not + against the change: it is recorded in [acceptance/ROUTED.md](acceptance/ROUTED.md), routed to a + change of its own, and does not block. The change is judged on what it changed; the document + is proved by the keyed questions, which every run answers over the whole of it. +- **Open points.** An open point in marking an answer is marked for the reader. An open point on + whether a reader named a passage, or whether two readers named the same one, counts the + passage. An open point on where a finding lies treats it as the change's own text. ## Pass criteria @@ -166,26 +176,37 @@ fresh cohort names by fewer than two is clear. ## Changing the test -Nothing in this test is changed to let a run through. A threshold, a question, a key or a -fixture is changed only before a run's readers start, never changes a mark already given, and -is recorded with its reason: a threshold, a question or the rules here by a change to this -process, a key by step 4 of `CONTRIBUTING.md` when the text it quotes is reworded or when a -load-bearing obligation is added, and each in [acceptance/CHANGES.md](acceptance/CHANGES.md). A -question is reworded only so that it asks for what its key requires, and a key's required +Nothing in this test is changed to let a particular run through. A threshold, a question, a +key, a fixture or a rule here is changed only before a run's readers start, never changes a mark +already given, and is changed in one of three ways, each recorded with its reason: + +- A key's quotation follows the text it quotes, and a key gains an item for a new load-bearing + obligation, with the question that asks for it, under step 4 of `CONTRIBUTING.md`, in the + specification pull request that makes the change; `acceptance/README.md` says what that may + and may not do. +- A threshold, a fixture, a rule here, what an existing key item requires, or the wording of a + question changes by a change to this process, with its own pull request. +- An open point a run recorded is settled by either route, whichever it needs. + +A question is reworded only so that it asks for what its key requires, and a key's required elements are never more than its question asks for. A run made after such a change is a fresh -run under the new rules, not a re-mark of an old one. The one time a threshold was relaxed to -let a change through, after three runs, the relaxation was reversed on the editor's instruction -and is recorded here so that it is not tried again. Removing the clarity rating and the request -for suggestions was not that: it changed what the cohort is asked, not how strictly an answer is -marked. +run under the new rules, not a re-mark of an old one. The test for a change to a rule is what +the rule measured: a rule that measured something other than whether a cold reader could follow +the text is a defect in this file and is corrected, and the correction says what the rule +measured instead; a rule that measured that, and found the text wanting, is not touched, and the +text is fixed. The one time a threshold was relaxed to let a change through, after three runs, +the relaxation was reversed on the editor's instruction and is recorded here so that it is not +tried again. Removing the clarity rating and the request for suggestions was not that: it +changed what the cohort is asked, not how strictly an answer is marked. ## After a failure -Fix the text the findings name and run a fresh cohort; readers are never reused. A keyed miss is -fixed in the text that carries the obligation, wherever the change left it. A finding on text the -change wrote or altered is fixed there. A finding elsewhere goes to `acceptance/ROUTED.md` and -is not fixed in this change. The number of runs it took is recorded; a document that needed three -runs is telling its editor something. +Fix the text the run's misses and findings name and run a fresh cohort; readers are never reused. +A keyed miss is fixed in the text that carries the obligation, wherever it lies, since the keyed +questions prove the document and a document one reader in five misreads is not merged to, whoever +wrote the sentence. A finding on text the change wrote or altered is fixed there. A finding +elsewhere goes to `acceptance/ROUTED.md` and is not fixed in this change. The number of runs it +took is recorded; a document that needed three runs is telling its editor something. ## When it runs @@ -217,8 +238,8 @@ returned it, the exact prompt every reader received, the command the readers wer and the instruction a delegated marker was given are posted in one or more comments on the pull request, linked from the record and never edited, hidden or deleted after posting. -The changelog line for the change names the record. CI refuses a specification change whose pull -request adds no run file; it checks for at least one, and a pull request that changes two +The changelog line for the change names every run made on it, the passing run last. CI refuses a +specification change whose pull request adds no run file; it checks for at least one, and a pull request that changes two documents adds a run for each. Each published version's changelog entry carries the lines of the changes it contains, so a version's cohort record is the set of runs those lines name. A version with no such line has not been accepted and MUST NOT be described as published. The question sets @@ -232,15 +253,17 @@ reads that list before cutting a release. ## Changes to this process The process files are this file, `acceptance/README.md`, `acceptance/runs/README.md`, -`acceptance/ROUTED.md`, `CONTRIBUTING.md` and the checks that enforce them: `tools/changes.py`, -the key-quotation checks in `spec-qa.py`, the CI workflow and the pull request template. A change -to any of them, or to a question set, is a change to this process. They are not specifications, -and a change to them is not tested by a cohort: the cohort measures whether a low-strength reader -can follow a specification, and the process is followed by the editor. Such a change names the -runs or the case that showed the need for it, and goes through steps 1, 3 and 6 of -`CONTRIBUTING.md`, with no scope, key or cohort step; its review reads every process file it +`CONTRIBUTING.md` and the checks that enforce them: `tools/changes.py`, the key-quotation checks +in `spec-qa.py`, the CI workflow and the pull request template. A change to any of them is a +change to this process, and so is a change to a threshold, a fixture, the wording of a question +or what a key item requires, as [Changing the test](#changing-the-test) says. They are not +specifications, and a change to them is not tested by a cohort: the cohort measures whether a +low-strength reader can follow a specification, and the process is followed by the editor. Such +a change names the runs or the case that showed the need for it, and goes through steps 1, 3 and +6 of `CONTRIBUTING.md`, with no scope, key or cohort step; its review reads every process file it changes. It applies to every run whose readers start after it merges; a run made before then may be made under it, following all of it at a commit named in the record before the readers start. It is recorded by a line added at the top of [acceptance/CHANGES.md](acceptance/CHANGES.md), -naming its pull request, not in the changelog, which is for the three specifications. A change to -a key or a fixture follows `acceptance/README.md`. +naming its pull request, not in the changelog, which is for the three specifications. +`acceptance/ROUTED.md` and `acceptance/runs/` are records, not process files: a specification +pull request writes to them. diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index a076ff1..cc01848 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -6,7 +6,7 @@ not a specification and is not versioned, and a change to it is not tested by a to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). - Pull request NNN: the test is made to prove what it exists to prove, the clarity of the - document to a cold reader, after thirty-four runs on two changes (19 to 30, 31 to 33 and 35 + document to a cold reader, after thirty-three runs on two changes (19 to 30, 31 to 33 and 35 to 52) in which keyed answers reached five of five and the runs still failed. Four defects in the process were found. Keys required elements the question never asked for (detector 2b's "need not remove or forbid", detector 2c's "at a version tracked together", and on pull @@ -33,4 +33,4 @@ to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). traceable-miss row is narrower and the confusion row is applied to the change rather than the document; both are recorded here as the reason the earlier rules could not converge, and neither changes a mark already given. Prompted by runs 19 to 30 on pull request 14 and runs - 31 to 52 on the defined-terms change. + 31 to 33 and 35 to 52 on the defined-terms change. diff --git a/acceptance/README.md b/acceptance/README.md index 49a097b..dcf39c0 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -48,11 +48,18 @@ Everything else in a key is explanation for the marker and never marks an answer A key quotes the document. When a clause is reworded, the key line that quotes it is updated in the same commit, and the change is one the acceptance run itself then tests. A key that no longer -matches the document is a defect in this directory, not a reason to mark a reader wrong. A key -update under step 4 of `CONTRIBUTING.md` changes a quotation to follow the text, or adds a keyed -item for a new load-bearing obligation; it does not change what an existing item requires. A -change to what an item requires, or to a question, is a change to the process and is recorded in -[CHANGES.md](CHANGES.md) with its reason, before the readers of the next run start. +matches the document is a defect in this directory, not a reason to mark a reader wrong. + +A key, a question set or a fixture changes in one of two ways, as `ACCEPTANCE.md` says under +"Changing the test", and always before the readers of the next run start: + +- **In a specification pull request, under step 4 of `CONTRIBUTING.md`**: a quotation follows + the text it quotes, or a keyed item is added for a new load-bearing obligation together with + the question that asks for it. The pull request's own changelog line records it. Such an + update does not change what an existing item requires, nor an existing question's wording. +- **In a process change, with its own pull request**: what an existing item requires, the + wording of a question, a fixture, or the rules of marking. It is recorded in + [CHANGES.md](CHANGES.md) with its reason. Questions are fixed so that runs can be compared across versions. Adding a keyed question is allowed when a new clause is load-bearing; removing or softening one to make a run pass is not; diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index bb82718..eb35815 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -8,7 +8,9 @@ editor reads this list before cutting a release. A finding is open until a change that reworks its passage passes a cohort, when the line names that run and moves below the rule, or until the editor closes it with a reason recorded here. A -finding raised again by a later run is noted on its line, not listed twice. +finding raised again by a later run is noted on its line, not listed twice. The rows below were +raised by runs 31 to 52 under the earlier rules, on untouched text of the change those runs +tested, and are listed here so that they are not lost. ## Open @@ -24,8 +26,7 @@ finding raised again by a later run is noted on its line, not listed twice. | `DETECTOR-SPEC.md` | 8.1 | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | | `DETECTOR-SPEC.md` | 5.3 | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | | `SPEC.md` | 1 (Owner) | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50 | open | -| `SPEC.md` | 4 | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 (one reader, 29) | open; a method question for the editor | -| `SPEC.md` | 2 and 3.1 | "technique" used for a way of writing the Rule in section 2 and for a way of searching in clause 3.1 | runs 29, 38 | pull request 14 rewords both | +| `SPEC.md` | 4 | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | open; a method question for the editor | ## Closed diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index b0a3f97..2f6fd96 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -33,13 +33,12 @@ Wrong readings the text invites, which count against the document on one reader: ## 2. Load-bearing statements - **2a.** Required: (1) a detector conforms when it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression); - (2) yes, a detector that predates the document can conform, because conformance is judged on - the evidence and a declaration is not a condition of it. Wrong if it says a declaration is + (2) yes, a detector that predates the document can conform. Wrong if it says a declaration is required, or that partial satisfaction is conformance. Key quotation, section [8](../../DETECTOR-SPEC.md#8-conformance): "A Detector Conforms if it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression)." Supporting, [8.1](../../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps): "A maintainer - MAY declare the version of this document the detector conforms to". "Judged on evidence", "the - declaration is optional" or "graded on evidence by anyone" each state the reason in element - (2); one of them is enough. + MAY declare the version of this document the detector conforms to". The reason, that + conformance is judged on evidence and a declaration is optional, is explanation: an answer + that gives it, or gives it instead of a plain yes, states element (2). - **2b.** Required: (1) it must let the project switch the route off in its configuration, or make the route detectable (by a rule the project can write, or by a mechanical check the detector documents; naming either means of detection, or just "detectable", states this); @@ -61,7 +60,7 @@ Wrong readings the text invites, which count against the document on one reader: ## 3. Application: Lintomatic 4.2 -| Clause | Accepted grade | Condition on the reasoning, and what the fixture supports | +| Clause | Accepted grade | What the fixture supports (no row carries a condition on the reasoning) | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules via the `Rule` interface, first class in every report | | [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `check --only ` runs one rule, bundled or project, with a distinguishing exit code | @@ -87,9 +86,9 @@ reason that it is a SHOULD, or that the MUST which fails is [6.3](../../DETECTOR Wrong readings the text invites, which count against the document on one reader: -- [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) graded Partial on the grounds that thirty-eight of forty resolve, which has read the - clause as admitting a proportion. -- [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) graded No because the comment takes no reason, where the reasoning quotes [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) or [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason), +- [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) graded Partial with reasoning that says most, or thirty-eight of forty, bundled rules + resolve, which has read the clause as admitting a proportion. +- [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) graded No with reasoning that says the comment takes no reason and quotes [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) or [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason), which has read [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason)'s SHOULD into [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable). A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Yes because they missed the last paragraph of the fixture has misread the diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 182d6b2..36bfafd 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -57,33 +57,34 @@ Wrong readings the text invites, which count against the document on one reader: ## 3. Application: the ledger remediation -| Clause | Accepted grades | Condition on the reasoning, and what the fixture supports | -| ----------------------------------------------------------------------------------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | No condition. Class, hazard and the next wider rule are named; Partial or No is the reading that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | -| [3.2](../../SPEC.md#32-build-the-net) | Yes | A rule in the project's static analyser, reading code, not a test | -| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | Commit B is red on the fixture and the originating instance and survives as its own commit | -| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | The reasoning names the instance left unfixed behind an inline ignore on the agent's own authority; five findings were recorded | -| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | The reasoning names the suppression no recorded decision covers; the rule is otherwise permanent and blocking. [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | -| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | The reasoning names the documentation on a wiki, not versioned with the rule; the message is terse and carries an identifier | +| Clause | Accepted grades | Condition on the reasoning | What the fixture supports | +| ----------------------------------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | None | Class, hazard and the next wider rule are named; Partial or No is the reading that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | +| [3.2](../../SPEC.md#32-build-the-net) | Yes | None | A rule in the project's static analyser, reading code, not a test | +| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | None | Commit B is red on the fixture and the originating instance and survives as its own commit | +| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Mentions the instance left unfixed, the inline ignore, or `Report::render()` | Five findings were recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority | +| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | Mentions the suppression, the inline ignore, or that no recorded decision covers it | The rule is permanent and blocking, but the suppression was not a recorded decision. [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | +| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | Mentions the wiki, or that the documentation is not versioned or co-located with the rule | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule | Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**; a reader is correct who says so and names [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) or [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the clauses not followed. The **defence does not conform** either; a reader is correct who says so and names [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the reasons, or [3.5](../../SPEC.md#35-enforce-permanently-and-block)'s unrecorded suppression, which -section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence also requires to be absent. +section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence also requires to be absent. Other clauses named +alongside are not marked. Section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): the **inline ignore on `Report::render()`** was the owner's decision, not the -practitioner's. A reader who names it is correct, with or without the decision to leave the -wider rule unbuilt, which section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) also gives to the owner. A reader who names only the wider -rule, or neither, is wrong. +practitioner's. A reader who names it is correct; any other decision named alongside it is not +marked. A reader who names only the decision to leave the wider rule unbuilt, or neither, is +wrong. Wrong readings the text invites, which count against the document on one reader: -- [3.5](../../SPEC.md#35-enforce-permanently-and-block) graded Yes because the rule fails the build, which has not applied the clause's opening - sentence on unrecorded suppression. -- [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) graded Yes because the URL resolves, which has read the versioning sentence of [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) - differently. -- The defence graded as conforming because the sweep is what failed, where the reasoning quotes - section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence. +- [3.5](../../SPEC.md#35-enforce-permanently-and-block) graded Yes with reasoning that says the rule blocks, or is permanent, and does not mention + the suppression. +- [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) graded Yes with reasoning that says the identifier resolves, or the URL works, and does not + mention versioning or co-location. +- The defence graded as conforming with reasoning that says only the sweep failed and that + quotes section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence. A reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the fixture, which is the reader's miss. A reader who attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index c11ae32..af28752 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -45,26 +45,26 @@ Wrong readings the text invites, which count against the document on one reader: grade, not a grade of its own. Wrong if it gives a grade per part. Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "The project-level verdict is a single grade; which part of the Toolchain satisfied each clause is evidence for that grade, not a second grade." -- **2c.** Required: (1) two, graded and declared separately; (2) neither implies the other. - Wrong if it gives one grade, or says the artefact grade follows from the project grade or the - reverse. Key quotation, [9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately): "The two verdicts MUST be graded and declared separately, and +- **2c.** Required: (1) two; (2) neither implies the other. Wrong if it gives one grade, or says + the artefact grade follows from the project grade or the reverse. That the two are graded and + declared separately is explanation; an answer that says it has stated element (1). Key quotation, [9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately): "The two verdicts MUST be graded and declared separately, and neither implies the other." ## 3. Application: the Zigpipe project -| Clause | Accepted grades | Condition on the reasoning, and what the fixture supports | -| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | Lintomatic fails detector [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) on two undocumented bundled rules and nothing in the wrapping closes that gap; a reader who notes that the detector specification was not supplied and grades from this document's summary of it is correct | -| [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)`. The two are blocking, since their firing stops `zig-qa run`; a reader who grades No on that condition is correct | -| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | Ignore comments disabled project-wide; exceptions only through the record the toolchain reads | -| [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | `zig-qa run` is local, prints Lintomatic's output unchanged, one line per finding | -| [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | Lintomatic, then the tests, stopping on a Lintomatic failure | -| [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes, Partial or No | No condition. `zig-qa list` prints identifier, summary and documentation path without running the rules; Partial or No is the reading that counts the two `(none)` routes, since 5.1 says the listing MUST provide the route and [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) point 2 says a gap may be named again by a later clause | -| [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | Derived from `lintomatic.toml`, not hand-maintained | -| [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | Yes | The `proj.*` rules appear in the same listing | -| [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes or Partial | `quality-exceptions.toml` is a defined path that `zig-qa run` loads. Partial is accepted only where the reasoning says the fixture names a place for exceptions and none for calibrations or conventions, which the project record entry also covers | -| [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Partial or No | The reasoning mentions that nothing rejects a justification that could be pasted onto any entry; a justification is required and an empty one rejected | -| [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | `zig-qa list --exceptions` uses the same format and operation as the rule listing | +| Clause | Accepted grades | Condition on the reasoning | What the fixture supports | +| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | None | Lintomatic fails detector [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) on two undocumented bundled rules and nothing in the wrapping closes that gap; a reader who notes that the detector specification was not supplied and grades from this document's summary of it is correct | +| [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | None. A No given on the condition that the two bundled rules are blocking is a No: their firing stops `zig-qa run`, so they are | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)` | +| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | None | Ignore comments disabled project-wide; exceptions only through the record the toolchain reads | +| [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | None | `zig-qa run` is local, prints Lintomatic's output unchanged, one line per finding | +| [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | None | Lintomatic, then the tests, stopping on a Lintomatic failure | +| [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes, Partial or No | None | `zig-qa list` prints identifier, summary and documentation path without running the rules; Partial or No is the reading that counts the two `(none)` routes, since [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) says the listing MUST provide the route and [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) point 2 says a gap may be named again by a later clause | +| [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | None | Derived from `lintomatic.toml`, not hand-maintained | +| [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | Yes | None | The `proj.*` rules appear in the same listing | +| [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes or Partial | For Partial: mentions that the fixture names a place for exceptions and none for calibrations or conventions | `quality-exceptions.toml` is a defined path that `zig-qa run` loads; the project record entry also covers calibrations and conventions | +| [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Partial or No | Mentions that nothing rejects a generic justification, one that could be pasted onto any entry | A justification is required and an empty one rejected | +| [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | None | `zig-qa list --exceptions` uses the same format and operation as the rule listing | Section [9](../../TOOLING-SPEC.md#9-conformance) verdict: **does not conform**; a reader is correct who says so and names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) or [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) among the failing clauses. **Not with agent support**; a reader is correct who says so because @@ -76,14 +76,15 @@ optional." Wrong readings the text invites, which count against the document on one reader: -- [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) graded Yes with reasoning that quotes only the first sentence of the clause, which has - not read the sentence on a generic justification. -- [8.1](../../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) applied to Zigpipe on the basis that its tooling includes a shipped package, which has - read "where the Toolchain is one the project ships" (section [9](../../TOOLING-SPEC.md#9-conformance)) as reaching the consuming - project. -- [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) graded Yes on the grounds that Lintomatic "mostly" conforms, where the reasoning quotes - [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), which has read it as admitting a proportion. -- Zigpipe held ungradeable for want of a declaration, which has read [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) as a condition. +- [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) graded Yes with reasoning that says a justification is required or an empty one rejected + and does not mention a generic justification, which has not read the sentence on one. +- Section [8](../../TOOLING-SPEC.md#8-self-audit) said to apply to Zigpipe with reasoning that says its tooling includes a shipped + package, which has read "where the Toolchain is one the project ships" (section [9](../../TOOLING-SPEC.md#9-conformance)) as + reaching the consuming project. +- [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) graded Yes with reasoning that says Lintomatic mostly or largely conforms and quotes [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), + which has read it as admitting a proportion. +- The declaration said to affect the verdict, with reasoning that Zigpipe cannot be graded without + one, which has read [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) as a condition. A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes because they missed the two undocumented rules has misread the fixture, which is the reader's miss. diff --git a/acceptance/toolchain/QUESTIONS.md b/acceptance/toolchain/QUESTIONS.md index 35a2b91..86de411 100644 --- a/acceptance/toolchain/QUESTIONS.md +++ b/acceptance/toolchain/QUESTIONS.md @@ -26,7 +26,7 @@ the document you are relying on. Grade the Zigpipe project's toolchain, as described in the fixture, against every MUST clause in sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record): [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record), [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure), [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration), [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones), [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences). For each give Yes, Partial or No and one line of reasoning. Then state whether the toolchain conforms under section -9, whether it conforms with agent support, and whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies to it. Finally, state +[9](../../TOOLING-SPEC.md#9-conformance), whether it conforms with agent support, and whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies to it. Finally, state whether the absence of a declaration affects the verdict. ## 4. Confusion From 8a63448820785166faeef81263fc1143b3028710 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 11:52:20 +0100 Subject: [PATCH 03/52] Acceptance process: review 2 findings applied Question 1 defined once (first six bullets, same clause counts once, a bullet on neither list not counted); the verdict items of question 3 ask for their reason and the keys state each as an accepted answer with a condition; a finding lies in the change's text also where the readers set it against that text, read against the merge base; the cohort's model is the version the editor names; a keyed miss whose fix would change an obligation is routed and waited for; the question set is the one list of load-bearing questions; ROUTED.md defines its status column and links its clauses. Co-Authored-By: Claude Fable 5.1 --- ACCEPTANCE.md | 102 +++++++++++++++--------------- CONTRIBUTING.md | 6 +- acceptance/CHANGES.md | 11 ++-- acceptance/README.md | 9 +-- acceptance/ROUTED.md | 33 +++++----- acceptance/detector/KEY.md | 11 ++-- acceptance/detector/QUESTIONS.md | 2 +- acceptance/method/KEY.md | 30 ++++----- acceptance/method/QUESTIONS.md | 4 +- acceptance/toolchain/KEY.md | 22 ++++--- acceptance/toolchain/QUESTIONS.md | 4 +- 11 files changed, 123 insertions(+), 111 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index c5bd8cd..39110a7 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -29,7 +29,8 @@ serves that, and a rule that stops it proving that is a defect in this file, fix ``` A reader is set aside, and its answers do not count, when its answer text is empty, when its - process reports an error, or when the model version it reports is not the cohort's model. A run + process reports an error, or when the model version it reports is not the cohort's model, the + one version the editor names for the run before its readers start. A run with a reader set aside is void: it takes a run number and has a record, and a fresh cohort runs. No reader is set aside for any other cause. @@ -54,24 +55,17 @@ asks for nothing the question does not. Each reader answers all of them. bullets. Keyed against the key's list of the document's obligations: a bullet is correct when it names one of them. -2. **Load-bearing statements.** Three or more questions, fixed per document, each asking for a - short answer and the sentence or sentences of the document it rests on, quoted. Each has one - or more parts, and the key lists the element an answer must state for each part: - - - `SPEC.md`: what a practitioner does before fixing a defect, in order; who may baseline or - suppress, and whether an agent may; what "conforms" means for a remediation and what a - verdict rests on. - - `DETECTOR-SPEC.md`: what "conforms" means and whether a tool that predates the document can - conform; what a detector must do about an inline suppression route and whether one of those - things is enough; where a bundled rule's documentation must live, at what version, and - whether it must be reachable offline. - - `TOOLING-SPEC.md`: where conformance is measured and what the toolchain is; how a grade is - assigned when parts come from different sources; how many grades a project that ships - tooling has and whether one implies the other. +2. **Load-bearing statements.** Three or more questions, fixed per document and listed in its + question set under `acceptance/`, each asking for a short answer and the sentence or + sentences of the document it rests on, quoted. Each has one or more parts, and the key lists + the element an answer must state for each part. An obligation is load-bearing, and gets a + question, when a reader who misreads it would act against the method; the editor decides, and + the pull request that adds the question says why. 3. **Application.** A fixture case supplied with the questions, graded clause by clause with - Yes, Partial or No and one line of reasoning. Keyed against the intended verdicts, with the - readings the key names as ones the text invites counting against the document. + Yes, Partial or No and one line of reasoning, then the verdicts the question set asks for, + each with its reason. Keyed against the accepted grades and answers, with the readings the + key names as ones the text invites counting against the document. 4. **Confusion.** Any passage the reader found confusing on its one reading, quoted with its clause number, and nothing else: not what would make it clearer, not a proposed change. @@ -98,9 +92,10 @@ unless the editor shows that a rule here was misapplied, and a re-mark says whic Every mark is given by a rule in this file or in the key, as follows. Where neither settles a mark, the marker records the point in the run record and marks it as the open-point rule below -says, and the editor settles it in this file or in the key, as [Changing the test](#changing-the-test) -allows, before the next run. A run record states no judgement that is not written here or in -the key; a judgement an earlier run recorded is not a rule. +says, and the editor settles it in this file or in the key, as +[Changing the test](#changing-the-test) allows, before the next run. A run record states no +judgement that is not written here or in the key; a judgement an earlier run recorded is not a +rule. - **Keyed items.** Question 1 as a whole is one keyed item, and so is each item of question 2 and each answer of question 3 for which the key gives an accepted grade or answer. @@ -111,20 +106,21 @@ the key; a judgement an earlier run recorded is not a rule. or apostrophe style, and markdown link or emphasis markup; a quotation that drops or changes a word quotes the document by its longest such run. - **Question 1** is correct when at least four bullets each name an obligation the key lists and - no bullet asserts one the key marks wrong. Several bullets that name different clauses of one - line of the key's list each count. A bullet asserting an obligation that is in neither list is - recorded as an open point for the key and is not counted either way. + no bullet asserts one the key marks wrong. Only the first six bullets are marked. Several + bullets that name different clauses of one line of the key's list each count; two that name + the same clause count once. A bullet asserting an obligation that is in neither list is not + counted either way, and is recorded as an open point for the key. - **Question 2.** An answer is the reader's sentences and its quotation read together. It is correct when it states every element the key lists as required for that item, in the document's words or the reader's own, asserts nothing the key marks wrong, and quotes the document. An element stated only in the quotation is stated. Anything in the key beyond the required elements and the wrong assertions is explanation for the marker, and an answer is never marked down for leaving it out. -- **Question 3.** Each item is correct when its grade is one the key accepts and, where the key - sets a condition on the reasoning for that grade, the reasoning meets it. A grade the key - accepts with no condition is correct whatever the reasoning. A grade given on a stated - condition is read as the grade for the case the fixture presents, where the key says which - case that is; otherwise the item is unanswered. +- **Question 3.** Each item is correct when its grade or answer is one the key accepts and, + where the key sets a condition on the reasoning for it, the reasoning meets it. A grade or + answer the key accepts with no condition is correct whatever the reasoning. A grade given on a + stated condition is read as the grade for the case the fixture presents, where the key says + which case that is; otherwise the item is unanswered. - **A miss that counts against the document.** The key names, for question 1 and for each item of question 3, the wrong readings the text invites, each as a grade or answer together with what the reasoning states or leaves out. A miss that matches one blocks whatever the count, @@ -134,22 +130,26 @@ the key; a judgement an earlier run recorded is not a rule. below; a second miss on the same item fails the threshold, and no judgement is needed. - **An unanswered question.** Each keyed item left unanswered is a miss. Question 4 left unanswered counts as no passage named. -- **Question 4.** A reader names a passage when it quotes words of the document or gives a clause - number with a description that identifies a sentence or paragraph. A remark that is not about - a passage of the document, such as that another document was not supplied, is recorded as a - note and is not a passage. Two readers name the same passage when the words or sentences they - name lie in the same paragraph; a heading, a list item, a table row and a terminology entry are - each a paragraph of their own. A passage named by two or more readers of the run is a finding. +- **Question 4.** A reader names a passage when it quotes the document, as defined above, or + gives a clause number with a description or a shorter quotation that identifies a sentence or + paragraph. A quotation whose words occur in more than one paragraph, given with no clause + number, names the paragraph where they first occur. A remark that is not about a passage of + the document, such as that another document was not supplied, is recorded as a note and is + not a passage. Two readers name the same passage when the words or sentences they name lie in + the same paragraph; a heading, a list item, a table row and a terminology entry are each a + paragraph of their own. A passage named by two or more readers of the run is a finding. - **Where a finding lies.** The record says, for each finding, whether it lies in text the change - wrote or altered: a paragraph in which the pull request, read against the branch it targets, - adds, rewords, moves or deletes a sentence, or changes a word, a term or a link. A finding + wrote or altered: a paragraph in which the pull request, read against its merge base with the + branch it targets, adds, rewords, moves or deletes a sentence, or changes a word, a term or a + link; or a paragraph the readers naming the finding set against such a paragraph. A finding there blocks the change. A finding anywhere else is a finding against the document, not against the change: it is recorded in [acceptance/ROUTED.md](acceptance/ROUTED.md), routed to a change of its own, and does not block. The change is judged on what it changed; the document is proved by the keyed questions, which every run answers over the whole of it. -- **Open points.** An open point in marking an answer is marked for the reader. An open point on - whether a reader named a passage, or whether two readers named the same one, counts the - passage. An open point on where a finding lies treats it as the change's own text. +- **Open points.** An open point in marking an answer is marked for the reader, except that a + question 1 bullet on neither list is not counted, as above. An open point on whether a reader + named a passage, or whether two readers named the same one, counts the passage. An open point + on where a finding lies treats it as the change's own text. ## Pass criteria @@ -164,7 +164,7 @@ is four of five. | A miss the key says counts against the document | **None**, whatever the count | | A passage found confusing by two or more readers, in text the change wrote or altered | **None.** It is a finding and blocks until the text is changed | | A passage found confusing by two or more readers, elsewhere | Recorded in `acceptance/ROUTED.md` and routed; does not block | -| Anything from one reader only | Recorded; applied at the editor's discretion; does not block | +| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | The asymmetry is deliberate. A load-bearing statement is the sentence the whole document exists to make, and one reader in five getting it wrong means one agent in five will act on the wrong @@ -202,11 +202,12 @@ changed what the cohort is asked, not how strictly an answer is marked. ## After a failure Fix the text the run's misses and findings name and run a fresh cohort; readers are never reused. -A keyed miss is fixed in the text that carries the obligation, wherever it lies, since the keyed -questions prove the document and a document one reader in five misreads is not merged to, whoever -wrote the sentence. A finding on text the change wrote or altered is fixed there. A finding -elsewhere goes to `acceptance/ROUTED.md` and is not fixed in this change. The number of runs it -took is recorded; a document that needed three runs is telling its editor something. +A failed keyed item is fixed in the text that carries the obligation, wherever it lies, since the +keyed questions prove the document as a whole, whoever wrote the sentence; where that fix would +change an obligation, it is routed to a change of its own, and this change waits for it. A +finding on text the change wrote or altered is fixed there. A finding elsewhere goes to +`acceptance/ROUTED.md` and is not fixed in this change. The number of runs it took is recorded; +a document that needed three runs is telling its editor something. ## When it runs @@ -239,11 +240,12 @@ and the instruction a delegated marker was given are posted in one or more comme request, linked from the record and never edited, hidden or deleted after posting. The changelog line for the change names every run made on it, the passing run last. CI refuses a -specification change whose pull request adds no run file; it checks for at least one, and a pull request that changes two -documents adds a run for each. Each published version's changelog entry carries the lines of the -changes it contains, so a version's cohort record is the set of runs those lines name. A version -with no such line has not been accepted and MUST NOT be described as published. The question sets -and answer keys live under `acceptance/` in this repository so that a run can be repeated and +specification change whose pull request adds no run file; it checks for at least one, and a pull +request that changes two documents adds a run for each. Each published version's changelog entry +carries the lines of the changes it contains, so a version's cohort record is the set of runs +those lines name. A version with no such line has not been accepted and MUST NOT be described as +published. The question sets and answer keys live under `acceptance/` in this repository so that +a run can be repeated and compared. `acceptance/ROUTED.md` lists every finding routed from a run, open until a change that fixes diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 013bb9d..733e05b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,9 +29,9 @@ review. It is a patch version when released. SHOULD or MAY in a numbered section or appendix. It needs everything a clarity edit needs, plus the five tests in SCOPE.md answered in the pull request, and it is at least a minor version when released. `tools/changes.py` tells the two apart mechanically and CI runs it on every pull -request. A change to the process files ACCEPTANCE.md names, this file among them, is neither: -ACCEPTANCE.md says how it is made, and it is recorded in `acceptance/CHANGES.md`, not in -CHANGELOG.md. +request. A change to the acceptance process, as ACCEPTANCE.md defines one under "Changes to +this process", is neither: ACCEPTANCE.md says how it is made, and it is recorded in +`acceptance/CHANGES.md`, not in CHANGELOG.md. ## What a pull request contains diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index cc01848..29a20db 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -1,13 +1,14 @@ # Changes to the acceptance process -Changes to the process files that [ACCEPTANCE.md](../ACCEPTANCE.md) names, recorded here as it -describes: a line is added at the top for each change, naming its pull request. The process is -not a specification and is not versioned, and a change to it is not tested by a cohort. Changes -to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). +Changes to the acceptance process, as [ACCEPTANCE.md](../ACCEPTANCE.md) defines one under +"Changes to this process", recorded here as it describes: a line is added at the top for each +change, naming its pull request. The process is not a specification and is not versioned, and a +change to it is not tested by a cohort. Changes to the three specifications are recorded in +[CHANGELOG.md](../CHANGELOG.md). - Pull request NNN: the test is made to prove what it exists to prove, the clarity of the document to a cold reader, after thirty-three runs on two changes (19 to 30, 31 to 33 and 35 - to 52) in which keyed answers reached five of five and the runs still failed. Four defects in + to 52) in which keyed answers reached five of five and the runs still failed. Five defects in the process were found. Keys required elements the question never asked for (detector 2b's "need not remove or forbid", detector 2c's "at a version tracked together", and on pull request 14 method 2d's order of the kinds), so readers who answered the question as put were diff --git a/acceptance/README.md b/acceptance/README.md index dcf39c0..aaeba1f 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -25,8 +25,8 @@ in [CHANGES.md](CHANGES.md). 4. The result is recorded as a file under [runs/](runs/), stating what `ACCEPTANCE.md` lists, with the answer sheets posted on the pull request and linked from it, and the change's `Unreleased` changelog line names that file. A run that fails is followed by a change to the - text the findings name and a fresh cohort; a finding on text the change did not write or - alter is added to [ROUTED.md](ROUTED.md) instead. + text its misses and findings name and a fresh cohort; a finding on text the change did not + write or alter is added to [ROUTED.md](ROUTED.md) instead. ## How a key is written @@ -50,8 +50,9 @@ A key quotes the document. When a clause is reworded, the key line that quotes i the same commit, and the change is one the acceptance run itself then tests. A key that no longer matches the document is a defect in this directory, not a reason to mark a reader wrong. -A key, a question set or a fixture changes in one of two ways, as `ACCEPTANCE.md` says under -"Changing the test", and always before the readers of the next run start: +A key, a question set or a fixture changes by one of two routes, as `ACCEPTANCE.md` says under +"Changing the test", always before the readers of the next run start, and an open point a run +recorded is settled by whichever route it needs: - **In a specification pull request, under step 4 of `CONTRIBUTING.md`**: a quotation follows the text it quotes, or a keyed item is added for a new load-bearing obligation together with diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index eb35815..60fc616 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -8,25 +8,26 @@ editor reads this list before cutting a release. A finding is open until a change that reworks its passage passes a cohort, when the line names that run and moves below the rule, or until the editor closes it with a reason recorded here. A -finding raised again by a later run is noted on its line, not listed twice. The rows below were -raised by runs 31 to 52 under the earlier rules, on untouched text of the change those runs -tested, and are listed here so that they are not lost. +finding raised again by a later run is noted on its line, not listed twice. The status column +holds `open`, or the `editor/` branch or pull request number of the change that is +reworking the passage. The rows below were raised by runs 31 to 52 under the earlier rules, on +untouched text of the change those runs tested, and are listed here so that they are not lost. ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | -------------------------------------- | -| `TOOLING-SPEC.md` | 9 and 9.1 | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | 2, 4.2, 8.1 | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | 9.2 | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | 4.3 | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | run 33 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | 4.1 | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | 5.3 | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | open | -| `DETECTOR-SPEC.md` | 7.1 | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | 8.1 | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | 5.3 | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | -| `SPEC.md` | 1 (Owner) | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50 | open | -| `SPEC.md` | 4 | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | open; a method question for the editor | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | ---------------------------------------- | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | run 33 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | ## Closed diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 2f6fd96..aaddb69 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -8,7 +8,7 @@ update the quotation here in the same commit. `ACCEPTANCE.md` says how each item ## 1. Restatement A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting an obligation the document does not state, is a correct restatement. +bullet asserting one the list of wrong assertions below marks wrong, is a correct restatement. - Support bespoke rules written by the project ([4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it)). - Provide a harness that runs a single rule against supplied code ([4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code)). @@ -78,11 +78,12 @@ No grade above carries a condition on the reasoning: the accepted grade is corre reasoning, and a reader who adds a caveat about what the fixture does not say and still gives the accepted grade is correct. -Section [8](../../DETECTOR-SPEC.md#8-conformance) verdict: **does not conform**; a reader is correct who says so and names [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), the -two undocumented bundled rules, or partial conformance not counting, as the reason. +The two answers the question then asks for, each with the condition its reason must meet: -Clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation): **does not change the verdict**; a reader is correct who says so and gives as the -reason that it is a SHOULD, or that the MUST which fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). +- **Conformance under section [8](../../DETECTOR-SPEC.md#8-conformance).** Accepted answer: does not conform. Condition: the reason + names [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), the two undocumented bundled rules, or partial conformance not counting. +- **Whether clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation) changes the verdict.** Accepted answer: it does not. Condition: the reason + says that it is a SHOULD, or that the MUST which fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). Wrong readings the text invites, which count against the document on one reader: diff --git a/acceptance/detector/QUESTIONS.md b/acceptance/detector/QUESTIONS.md index 2666b2f..5c288f8 100644 --- a/acceptance/detector/QUESTIONS.md +++ b/acceptance/detector/QUESTIONS.md @@ -25,7 +25,7 @@ the document you are relying on. Grade Lintomatic 4.2, as described in the fixture, against every MUST clause in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression): [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it), [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), [4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding), [5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure), [5.2](../../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file), [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran), [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally), [6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation), [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access), [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable). For each give Yes, Partial or No and one line of reasoning. Then state whether Lintomatic conforms under section [8](../../DETECTOR-SPEC.md#8-conformance), and why. Finally, -state whether clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation) changes the verdict. +state whether clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation) changes the verdict, and why. ## 4. Confusion diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 36bfafd..b756f47 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -8,7 +8,7 @@ update the quotation here in the same commit. `ACCEPTANCE.md` says how each item ## 1. Restatement A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting an obligation the document does not state, is a correct restatement. +bullet asserting one the list of wrong assertions below marks wrong, is a correct restatement. - Attribute the defect to a class, bounded on both sides, with the hazard stated ([3.1](../../SPEC.md#31-attribute-the-defect-to-a-class)). - Express the class as a rule in a detector that reads code, never a test ([3.2](../../SPEC.md#32-build-the-net)). @@ -43,7 +43,7 @@ Wrong readings the text invites, which count against the document on one reader: quotation, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): "An Agent MUST NOT Baseline, suppress or knowingly leave an Instance unfixed on its own authority, whatever the Instance count turns out to be." "The owner" names the defined term and states that the decision is a human's; an answer need not add "a human" - to be correct. The default owner paragraph in the terminology entry for Owner still requires a + to be correct. The default-owner sentences of the terminology entry for Owner still require a recorded decision, so an answer naming whoever instructed the practitioner, for a project that has recorded nobody, names the owner and is correct. - **2c.** Required: (1) all six clauses of section [3](../../SPEC.md#3-the-method) were followed for that defect; (2) the verdict @@ -61,21 +61,22 @@ Wrong readings the text invites, which count against the document on one reader: | ----------------------------------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | None | Class, hazard and the next wider rule are named; Partial or No is the reading that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | | [3.2](../../SPEC.md#32-build-the-net) | Yes | None | A rule in the project's static analyser, reading code, not a test | -| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | None | Commit B is red on the fixture and the originating instance and survives as its own commit | +| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes, Partial or No | For Partial or No: attributes the inline ignore to this clause, a fair reading of Part B | Commit B is red on the fixture and the originating instance and survives as its own commit | | [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Mentions the instance left unfixed, the inline ignore, or `Report::render()` | Five findings were recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority | | [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | Mentions the suppression, the inline ignore, or that no recorded decision covers it | The rule is permanent and blocking, but the suppression was not a recorded decision. [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | | [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | Mentions the wiki, or that the documentation is not versioned or co-located with the rule | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule | -Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**; a reader is correct who says so and names -[3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) or [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the clauses not followed. The **defence does not conform** either; a reader is -correct who says so and names [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the reasons, or [3.5](../../SPEC.md#35-enforce-permanently-and-block)'s unrecorded suppression, which -section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence also requires to be absent. Other clauses named -alongside are not marked. +The three answers the question then asks for, each with the condition its reason must meet: -Section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): the **inline ignore on `Report::render()`** was the owner's decision, not the -practitioner's. A reader who names it is correct; any other decision named alongside it is not -marked. A reader who names only the decision to leave the wider rule unbuilt, or neither, is -wrong. +- **The remediation under section [7](../../SPEC.md#7-conformance).** Accepted answer: does not conform. Condition: the reason + names [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) or [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the clauses not followed; other clauses named alongside are not marked. +- **The defence alone.** Accepted answer: does not conform. Condition: the reason names [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation), or + [3.5](../../SPEC.md#35-enforce-permanently-and-block)'s unrecorded suppression, which section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence also + requires to be absent; other clauses named alongside are not marked. +- **The decision section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) says was not the practitioner's.** Accepted answer: the inline ignore on + `Report::render()`, which was the owner's decision. Condition: none; any other decision named + alongside it is not marked. An answer naming only the decision to leave the wider rule unbuilt, + or naming none, is wrong. Wrong readings the text invites, which count against the document on one reader: @@ -86,9 +87,8 @@ Wrong readings the text invites, which count against the document on one reader: - The defence graded as conforming with reasoning that says only the sweep failed and that quotes section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence. -A reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the fixture, which is the reader's miss. A reader who -attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as -covering it, which is a fair reading and is not marked wrong. +A reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the fixture, which is the reader's miss, whether or not +the inline ignore was attributed to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) instead. ## 4. Confusion diff --git a/acceptance/method/QUESTIONS.md b/acceptance/method/QUESTIONS.md index 34fa82d..9a2f393 100644 --- a/acceptance/method/QUESTIONS.md +++ b/acceptance/method/QUESTIONS.md @@ -24,8 +24,8 @@ the document you are relying on. Grade the ledger remediation, as described in the fixture, against each of the six clauses of section [3](../../SPEC.md#3-the-method): [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class), [3.2](../../SPEC.md#32-build-the-net), [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire), [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), [3.5](../../SPEC.md#35-enforce-permanently-and-block), [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation). For each give Yes, Partial or No and one line of -reasoning. Then state whether the remediation conforms under section [7](../../SPEC.md#7-conformance), whether the defence -alone conforms, and which decision in the fixture, if any, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) says was not the +reasoning. Then state whether the remediation conforms under section [7](../../SPEC.md#7-conformance) and why, whether the defence +alone conforms and why, and which decision in the fixture, if any, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) says was not the practitioner's to take. ## 4. Confusion diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index af28752..7353852 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -8,7 +8,7 @@ update the quotation here in the same commit. `ACCEPTANCE.md` says how each item ## 1. Restatement A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting an obligation the document does not state, is a correct restatement. +bullet asserting one the list of wrong assertions below marks wrong, is a correct restatement. - Route defences only through detectors that conform to the detector specification, wrapping permitted ([4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification)). @@ -66,13 +66,19 @@ Wrong readings the text invites, which count against the document on one reader: | [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Partial or No | Mentions that nothing rejects a generic justification, one that could be pasted onto any entry | A justification is required and an empty one rejected | | [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | None | `zig-qa list --exceptions` uses the same format and operation as the rule listing | -Section [9](../../TOOLING-SPEC.md#9-conformance) verdict: **does not conform**; a reader is correct who says so and names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) or -[4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) among the failing clauses. **Not with agent support**; a reader is correct who says so because -section [7](../../TOOLING-SPEC.md#7-agent-context) is not met, there being no summary. **Section [8](../../TOOLING-SPEC.md#8-self-audit) does not apply**; a reader is correct -who says so because Zigpipe ships nothing to other projects, or because section [8](../../TOOLING-SPEC.md#8-self-audit) bears on -zig-qa's own artefact grade and not on Zigpipe. **The absence of a declaration does not affect -the verdict**; a reader is correct who says so. Key quotation, [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance): "The declaration is -optional." +The four answers the question then asks for, each with the condition its reason must meet: + +- **Conformance under section [9](../../TOOLING-SPEC.md#9-conformance).** Accepted answer: does not conform. Condition: the reason + names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) or [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) among the failing clauses; other clauses named alongside are not marked. +- **Conformance with agent support.** Accepted answer: no. Condition: the reason says that + section [7](../../TOOLING-SPEC.md#7-agent-context) is not met, or that there is no summary. +- **Whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies.** Accepted answer: it does not. Condition: the reason says that + Zigpipe ships nothing to other projects, or that section [8](../../TOOLING-SPEC.md#8-self-audit) bears on zig-qa's own artefact + grade and not on Zigpipe. +- **Whether the absence of a declaration affects the verdict.** Accepted answer: it does not. + Condition: the reason says that the declaration is optional, or not a condition of + conformance, or that a verdict rests on the evidence. Key quotation, [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance): "The declaration is + optional." Wrong readings the text invites, which count against the document on one reader: diff --git a/acceptance/toolchain/QUESTIONS.md b/acceptance/toolchain/QUESTIONS.md index 86de411..d4e5cd7 100644 --- a/acceptance/toolchain/QUESTIONS.md +++ b/acceptance/toolchain/QUESTIONS.md @@ -26,8 +26,8 @@ the document you are relying on. Grade the Zigpipe project's toolchain, as described in the fixture, against every MUST clause in sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record): [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record), [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure), [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration), [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones), [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences). For each give Yes, Partial or No and one line of reasoning. Then state whether the toolchain conforms under section -[9](../../TOOLING-SPEC.md#9-conformance), whether it conforms with agent support, and whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies to it. Finally, state -whether the absence of a declaration affects the verdict. +[9](../../TOOLING-SPEC.md#9-conformance) and why, whether it conforms with agent support and why, and whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies to it +and why. Finally, state whether the absence of a declaration affects the verdict, and why. ## 4. Confusion From 4cc568884c13ed12c7b31a56f41459e26519de30 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 11:58:31 +0100 Subject: [PATCH 04/52] Acceptance process: review 3 findings applied Two routes for changing the test, with a changed or removed obligation's key item following the text in the specification pull request; an open point holds up only its own pull request; a fresh run follows a change to the document, not to the record; detector 2b names both things; the section 7 and section 9 reasons accept every failing clause; the restatement rule reads plainly; ROUTED.md defines a finding as ACCEPTANCE.md does. Co-Authored-By: Claude Fable 5.1 --- ACCEPTANCE.md | 30 ++++++++++++++++-------------- CONTRIBUTING.md | 4 ++-- acceptance/README.md | 20 +++++++++++++------- acceptance/ROUTED.md | 6 +++--- acceptance/detector/KEY.md | 13 +++++++------ acceptance/method/KEY.md | 7 ++++--- acceptance/toolchain/KEY.md | 10 ++++++---- 7 files changed, 51 insertions(+), 39 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 39110a7..d1bd6af 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -141,8 +141,8 @@ rule. - **Where a finding lies.** The record says, for each finding, whether it lies in text the change wrote or altered: a paragraph in which the pull request, read against its merge base with the branch it targets, adds, rewords, moves or deletes a sentence, or changes a word, a term or a - link; or a paragraph the readers naming the finding set against such a paragraph. A finding - there blocks the change. A finding anywhere else is a finding against the document, not + link; or a paragraph that any reader naming the finding sets against such a paragraph. A + finding there blocks the change. A finding anywhere else is a finding against the document, not against the change: it is recorded in [acceptance/ROUTED.md](acceptance/ROUTED.md), routed to a change of its own, and does not block. The change is judged on what it changed; the document is proved by the keyed questions, which every run answers over the whole of it. @@ -178,15 +178,18 @@ fresh cohort names by fewer than two is clear. Nothing in this test is changed to let a particular run through. A threshold, a question, a key, a fixture or a rule here is changed only before a run's readers start, never changes a mark -already given, and is changed in one of three ways, each recorded with its reason: +already given, and is changed by one of two routes, each recorded with its reason: -- A key's quotation follows the text it quotes, and a key gains an item for a new load-bearing - obligation, with the question that asks for it, under step 4 of `CONTRIBUTING.md`, in the - specification pull request that makes the change; `acceptance/README.md` says what that may - and may not do. -- A threshold, a fixture, a rule here, what an existing key item requires, or the wording of a - question changes by a change to this process, with its own pull request. -- An open point a run recorded is settled by either route, whichever it needs. +- In the specification pull request that changes the text, under step 4 of `CONTRIBUTING.md`: + a key's quotation follows the text it quotes; a key gains an item, with the question that asks + for it, for a new load-bearing obligation; and where the change alters or removes the + obligation an existing item tests, the item and its question follow the text, since that is + the text changing and not the test. `acceptance/README.md` says what that may and may not do. +- By a change to this process, with its own pull request: a threshold, a fixture, a rule here, + what an existing key item requires of an unchanged obligation, or the wording of a question. + +An open point a run recorded is settled by whichever route it needs, before the next run of the +pull request whose run recorded it; a run of another pull request is not held up by it. A question is reworded only so that it asks for what its key requires, and a key's required elements are never more than its question asks for. A run made after such a change is a fresh @@ -214,9 +217,9 @@ a document that needed three runs is telling its editor something. On the final text of each pull request, after the editor's rewrite and any key update, one cohort per changed document. It is the last gate before the merge to `next`, so that every change on `next` is one the cohort has accepted, and the release branch needs no cohort of its own unless it -changes specification text. A fresh cohort runs after every change to the pull request's own +changes specification text. A fresh cohort runs after every change to the changed document's text that follows a run, and after every void run; a text that failed is never run again -unchanged. A version number or date that `PUBLISHING.md` or step 5 of `CONTRIBUTING.md` has the +unchanged, and a commit that adds the run record or the changelog line changes no document. A version number or date that `PUBLISHING.md` or step 5 of `CONTRIBUTING.md` has the editor set is not such a change, and neither is a merge of `next` into the branch that brings in text another cohort accepted and alters no sentence this change wrote. @@ -245,8 +248,7 @@ request that changes two documents adds a run for each. Each published version's carries the lines of the changes it contains, so a version's cohort record is the set of runs those lines name. A version with no such line has not been accepted and MUST NOT be described as published. The question sets and answer keys live under `acceptance/` in this repository so that -a run can be repeated and -compared. +a run can be repeated and compared. `acceptance/ROUTED.md` lists every finding routed from a run, open until a change that fixes its passage passes a cohort, or until the editor closes it with a recorded reason. The editor diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 733e05b..ea5a525 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -69,8 +69,8 @@ survives, the change is yours. changed document. Its record is committed under `acceptance/runs/`, the `Unreleased` changelog line names it, and CI refuses a specification change that lacks one. For an obligation, the editor sets the document's header to `-dev, unpublished` in the - same commit if it still names a published version. A change to the pull request's own text - after the run means a fresh run, as ACCEPTANCE.md says under "When it runs". A finding the + same commit if it still names a published version. A change to the document's text after the + run means a fresh run, as ACCEPTANCE.md says under "When it runs". A finding the run routes to `acceptance/ROUTED.md` is not fixed in this pull request. 6. **Merge** into `next` by merge commit, and the branch is deleted. diff --git a/acceptance/README.md b/acceptance/README.md index aaeba1f..ec5c3b2 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -32,12 +32,15 @@ in [CHANGES.md](CHANGES.md). A key says, for each keyed item, exactly what is marked and nothing a marker has to guess: -- **Question 1**: the obligations a bullet may name, and the assertions that make a bullet wrong. +- **Question 1**: the obligations a bullet may name, the assertions that make a bullet wrong, + and the wrong readings the text invites, if any. - **Question 2**: for each item, the elements an answer is required to state, numbered, one for each part of the question; the assertions that make an answer wrong; and the sentence or - sentences of the document the answer rests on, quoted verbatim. The required elements are - what the question asks for, no more: an element the question does not ask for is not - required, however load-bearing, until the question is reworded to ask for it. + sentences of the document the answer rests on, quoted verbatim, which show the marker where + the elements come from and which `spec-qa.py` checks against the document; a reader's + quotation need not be that one. The required elements are what the question asks for, no + more: an element the question does not ask for is not required, however load-bearing, until + the question is reworded to ask for it. - **Question 3**: for each item, the grade or grades accepted, any condition on the reasoning, and the wrong readings the text invites, which count against the document when one reader takes them. @@ -55,9 +58,12 @@ A key, a question set or a fixture changes by one of two routes, as `ACCEPTANCE. recorded is settled by whichever route it needs: - **In a specification pull request, under step 4 of `CONTRIBUTING.md`**: a quotation follows - the text it quotes, or a keyed item is added for a new load-bearing obligation together with - the question that asks for it. The pull request's own changelog line records it. Such an - update does not change what an existing item requires, nor an existing question's wording. + the text it quotes; a keyed item is added for a new load-bearing obligation together with the + question that asks for it; or, where the change alters or removes the obligation an existing + item tests, that item and its question follow the text, which is the text changing and not + the test. The pull request's own changelog line records it. Such an update does not change + what an item requires of an obligation the change leaves as it was, nor the wording of a + question whose obligation is unchanged. - **In a process change, with its own pull request**: what an existing item requires, the wording of a question, a fixture, or the rules of marking. It is recorded in [CHANGES.md](CHANGES.md) with its reason. diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 60fc616..24beef9 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -1,8 +1,8 @@ # Passages routed out of a cohort -A finding is a passage two or more readers of one cohort named as confusing, under question 4, -that lies outside the text the change being tested wrote or altered. `ACCEPTANCE.md` routes such -a finding here rather than blocking the change whose cohort raised it, so that each change is +A finding is a passage two or more readers of one cohort named as confusing under question 4. +`ACCEPTANCE.md` routes here a finding that lies outside the text the change being tested wrote +or altered, rather than blocking the change whose cohort raised it, so that each change is judged on what it changed and the document is still made clearer, one change at a time. The editor reads this list before cutting a release. diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index aaddb69..5b4d4aa 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -7,8 +7,8 @@ update the quotation here in the same commit. `ACCEPTANCE.md` says how each item ## 1. Restatement -A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting one the list of wrong assertions below marks wrong, is a correct restatement. +A bullet is correct when it names one of these obligations. Four or more correct bullets, with +no bullet making an assertion listed below as wrong, is a correct restatement. - Support bespoke rules written by the project ([4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it)). - Provide a harness that runs a single rule against supplied code ([4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code)). @@ -39,10 +39,11 @@ Wrong readings the text invites, which count against the document on one reader: MAY declare the version of this document the detector conforms to". The reason, that conformance is judged on evidence and a declaration is optional, is explanation: an answer that gives it, or gives it instead of a plain yes, states element (2). -- **2b.** Required: (1) it must let the project switch the route off in its configuration, or - make the route detectable (by a rule the project can write, or by a mechanical check the - detector documents; naming either means of detection, or just "detectable", states this); - (2) yes, either one alone is enough. Wrong if it says the detector must remove or forbid the +- **2b.** Required: (1) the two things, both named: let the project switch the route off in its + configuration, or make the route detectable (by a rule the project can write, or by a + mechanical check the detector documents; naming either means of detection, or just + "detectable", states that alternative); an answer naming only one of the two things misses + this element; (2) yes, either one alone is enough. Wrong if it says the detector must remove or forbid the route, or must require a reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD). Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each one it MUST do at least one of two things, and either alone satisfies this clause: make the route disableable by configuration; or make the route detectable, by a Rule the project can write in diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index b756f47..a990c7b 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -7,8 +7,8 @@ update the quotation here in the same commit. `ACCEPTANCE.md` says how each item ## 1. Restatement -A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting one the list of wrong assertions below marks wrong, is a correct restatement. +A bullet is correct when it names one of these obligations. Four or more correct bullets, with +no bullet making an assertion listed below as wrong, is a correct restatement. - Attribute the defect to a class, bounded on both sides, with the hazard stated ([3.1](../../SPEC.md#31-attribute-the-defect-to-a-class)). - Express the class as a rule in a detector that reads code, never a test ([3.2](../../SPEC.md#32-build-the-net)). @@ -69,7 +69,8 @@ Wrong readings the text invites, which count against the document on one reader: The three answers the question then asks for, each with the condition its reason must meet: - **The remediation under section [7](../../SPEC.md#7-conformance).** Accepted answer: does not conform. Condition: the reason - names [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) or [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the clauses not followed; other clauses named alongside are not marked. + names [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), [3.5](../../SPEC.md#35-enforce-permanently-and-block) or [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the clauses not followed; other clauses named alongside are not + marked. - **The defence alone.** Accepted answer: does not conform. Condition: the reason names [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation), or [3.5](../../SPEC.md#35-enforce-permanently-and-block)'s unrecorded suppression, which section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence also requires to be absent; other clauses named alongside are not marked. diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 7353852..2d2bea3 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -7,8 +7,8 @@ update the quotation here in the same commit. `ACCEPTANCE.md` says how each item ## 1. Restatement -A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting one the list of wrong assertions below marks wrong, is a correct restatement. +A bullet is correct when it names one of these obligations. Four or more correct bullets, with +no bullet making an assertion listed below as wrong, is a correct restatement. - Route defences only through detectors that conform to the detector specification, wrapping permitted ([4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification)). @@ -69,9 +69,11 @@ Wrong readings the text invites, which count against the document on one reader: The four answers the question then asks for, each with the condition its reason must meet: - **Conformance under section [9](../../TOOLING-SPEC.md#9-conformance).** Accepted answer: does not conform. Condition: the reason - names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) or [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) among the failing clauses; other clauses named alongside are not marked. + names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) or [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) among the failing clauses; other clauses named alongside are not + marked. - **Conformance with agent support.** Accepted answer: no. Condition: the reason says that - section [7](../../TOOLING-SPEC.md#7-agent-context) is not met, or that there is no summary. + section [7](../../TOOLING-SPEC.md#7-agent-context) is not met, or that there is no summary, or that the toolchain does not conform + under section [9](../../TOOLING-SPEC.md#9-conformance) at all. - **Whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies.** Accepted answer: it does not. Condition: the reason says that Zigpipe ships nothing to other projects, or that section [8](../../TOOLING-SPEC.md#8-self-audit) bears on zig-qa's own artefact grade and not on Zigpipe. From c31d49a0ac8e66931784ac09ef342d78347b0957 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 12:00:29 +0100 Subject: [PATCH 05/52] Acceptance process: CHANGES.md names pull request 22 Co-Authored-By: Claude Fable 5.1 --- acceptance/CHANGES.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md index 29a20db..daa3d44 100644 --- a/acceptance/CHANGES.md +++ b/acceptance/CHANGES.md @@ -6,7 +6,8 @@ change, naming its pull request. The process is not a specification and is not v change to it is not tested by a cohort. Changes to the three specifications are recorded in [CHANGELOG.md](../CHANGELOG.md). -- Pull request NNN: the test is made to prove what it exists to prove, the clarity of the +- [Pull request #22](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/22): + the test is made to prove what it exists to prove, the clarity of the document to a cold reader, after thirty-three runs on two changes (19 to 30, 31 to 33 and 35 to 52) in which keyed answers reached five of five and the runs still failed. Five defects in the process were found. Keys required elements the question never asked for (detector 2b's From 4d6ebbdf3ef78c90d2b33c994947b13b01747f89 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 12:25:41 +0100 Subject: [PATCH 06/52] Toolchain 2, 4.1, 4.2, 4.3, 8.1, 9 and 9.2: four passages that pulled against themselves Section 9 no longer counts section 8 in a shipping project's project grade, as clause 9.1 already said; clauses 4.2 and 8.1 say a blocking rule whose documentation is missing is graded as a defence with a gap; clause 9.2's gap duty falls only on one that declares, with the method's clause 3.2 record applying either way; clause 4.3's heading names both ways its body allows; clause 4.1 says every MUST of the detector specification applies and its point 2 says the wrapped pair is what is judged. Links to the renamed 4.3 heading follow it. Raised by runs 33 to 52 and 55 on the defined-terms change, as acceptance/ROUTED.md lists. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 21 ++++++++- TOOLING-SPEC.md | 49 +++++++++++++-------- acceptance/toolchain/KEY.md | 4 +- acceptance/toolchain/QUESTIONS.md | 2 +- tools/php-qa-ci.md | 70 +++++++++++++++--------------- tools/ts-qa-ci.md | 72 +++++++++++++++---------------- 6 files changed, 124 insertions(+), 94 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f5f8122..ab6229f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -95,6 +95,25 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be ## Toolchain specification (TOOLING-SPEC.md) +### Unreleased + +- Clarity, no obligation changed: four passages that cold readers of an unrelated change read as + the document pulling against itself, each listed in `acceptance/ROUTED.md`. Section [9](TOOLING-SPEC.md#9-conformance)'s + conformance sentence now says that section [8](TOOLING-SPEC.md#8-self-audit) bears on a shipping project's artefact grade + under clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately), not on its project grade, which is what clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) already said and the + answer key assumed. Clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) says that one identifier that does not resolve fails it and + that a blocking rule whose documentation is missing is graded as a defence with a gap, and + clause [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) says the same of a bundled defence, after readers took section [2](TOOLING-SPEC.md#2-terminology)'s Defence row to + put such a rule outside both clauses. Clause [9.2](TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance)'s duty to record a known gap in the + declaration falls only on a project or artefact that declares a version, and says so beside + "The declaration is optional", with the method's clause [3.2](SPEC.md#32-build-the-net) record applying either way. + Clause [4.3](TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)'s heading names both ways its body allows, disabling the route or a defence of + the toolchain's own. Clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) says every MUST of the detector specification's sections [4](DETECTOR-SPEC.md#4-authoring-rules) to [7](DETECTOR-SPEC.md#7-suppression) + applies and the six it lists are a summary, and its point 2 says the wrapped pair is what the + clause judges whilst the detector's own grade is a separate verdict. Prompted by runs 33 to 52 + and 55 on the defined-terms change, in which two or more readers quoted each passage. Accepted + by run NNN. + ### 0.2.0, 2026-09-08 Reissued the same day as 0.1.0 with sections [4](TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](TOOLING-SPEC.md#6-the-project-record) moved to the detector specification and @@ -121,7 +140,7 @@ again by a later clause; three readers of run 8 graded it so from the text. Find across the runs: an eleven-term gloss and a reading note in sections [1](TOOLING-SPEC.md#1-what-this-document-is-for) and [2](TOOLING-SPEC.md#2-terminology); long sentences split; [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) opens by listing the detector specification's MUSTs and its point 2 leads with a three-case table of wrapped pairs, then the one question behind it; [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) says it -widens the detector document's reach to the project's own rules; [4.3](TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) loses its double negative; +widens the detector document's reach to the project's own rules; [4.3](TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) loses its double negative; [5.2](TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) gives the derived-listing test; [6.2](TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one)'s heading carries the generic-justification rejection; [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) is consistent with [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) on third-party catalogues. Runs 1 to 7 also asked readers what would make a clause clearer and for a clarity rating; both questions were withdrawn before run 8, as diff --git a/TOOLING-SPEC.md b/TOOLING-SPEC.md index 209495a..1c97502 100644 --- a/TOOLING-SPEC.md +++ b/TOOLING-SPEC.md @@ -108,13 +108,15 @@ any row and the failure mode is visible too: either half alone leaves the [Pract ### 4.1 Every detector the toolchain routes a defence through MUST conform to the [detector specification](DETECTOR-SPEC.md) A [Toolchain] MUST NOT route a [Defence] through a [Detector] that does not -[Conform] to [the detector specification](DETECTOR-SPEC.md). That document's MUSTs, in brief, -are six. The [Detector] hosts bespoke [Rules]. It runs one [Rule] on its own against supplied +[Conform] to [the detector specification](DETECTOR-SPEC.md). That document's MUSTs are those of +its sections 4 to 7, and every one of them applies; the six summarised here are the ones most +often failed. The [Detector] hosts bespoke [Rules]. It runs one [Rule] on its own against supplied code. It prints a stable [Identifier] with every finding. It runs locally, down to one file, with the result in the command's own output and nothing reportable only through a hosted service. It resolves every [Identifier] a [Bundled rule] can print to documentation shipped with it, offline. It makes any inline [Suppression] route detectable or disableable. A [Detector] that fails any -one of those, after wrapping, fails this clause. Three points govern how that is met. +one of that document's MUSTs, after wrapping, fails this clause. Three points govern how that is +met. 1. **Wrapping is permitted.** The [Toolchain] MAY supply a mechanism the [Detector] lacks by wrapping it, with a [Harness] script or a @@ -134,8 +136,9 @@ one of those, after wrapping, fails this clause. Three points govern how that is meet every MUST of sections 4 to 7 of the [detector specification](DETECTOR-SPEC.md), exercised as its clause 8.1 describes? If yes, this clause holds. If no, it fails, however much else the [Detector] does well, and whether or not a clause below names the same gap - again. Wrapping can add a mechanism; it cannot excuse a gap. The [Detector]'s own verdict - under that document is unchanged by the wrapping. Satisfying most of that document is not + again. Wrapping can add a mechanism; it cannot excuse a gap. What this clause judges is the + wrapped pair; the [Detector]'s own grade under that document, as an artefact on its own, is a + separate verdict that the wrapping does not change. Satisfying most of that document is not [Conformance] to it, any more than satisfying most of this one is. 3. **What cannot be wrapped.** A [Detector] that cannot host a bespoke [Rule] at all @@ -156,7 +159,10 @@ clause 3.2. A gap recorded is not a gap closed. This clause widens the [detector specification](DETECTOR-SPEC.md)'s reach: that document resolves [Bundled rules] only, and this one covers every [Identifier] the project can see, the project's own [Rules] included. The [Toolchain] MUST resolve every [Identifier] its [Defences] -print to [Remediation docs] on disk. The mechanism is keyed on the +print to [Remediation docs] on disk. One [Identifier] that does not resolve fails this clause, +however many others do. A [Blocking] [Rule] whose documentation is missing is the case this +clause exists to catch: it is graded here as a [Defence] with a gap, not left out as no +[Defence] at all. The mechanism is keyed on the [Identifier] exactly as printed, and it covers the project's own [Rules] and every [Bundled defence]. Those [Remediation docs] MUST ship with the project or with the [Toolchain], at a version tracked together. The @@ -179,7 +185,7 @@ reader, which for an [Agent] means mechanically and on disk, and the project's o has a [Message] that names a pattern without leading anywhere, which method clause 3.6 says does not [Conform]. -### 4.3 The toolchain MUST forbid, through a defence of its own, every suppression route that bypasses the project record +### 4.3 The toolchain MUST forbid every suppression route that bypasses the project record, by disabling it or by a defence of its own A [Conforming] [Toolchain] MUST disable every [Suppression] route that bypasses the [Project record], or MUST run a [Blocking] @@ -362,7 +368,9 @@ mistake rather than after it, which is why they are worth stating even as SHOULD The [Toolchain] MUST run an automated check that blocks its own release. The check covers every [Identifier] printed by a [Rule] the [Toolchain] authors or bundles -as its own [Defence], whatever kind of [Detector] carries it. Where a documentation +as its own [Defence], whatever kind of [Detector] carries it; a [Bundled defence] whose +documentation is missing is still a [Bundled defence] for this clause, and that gap is what the +check exists to find. Where a documentation page covers a family of [Identifiers] by a pattern, as clause 6.5 of the [detector specification](DETECTOR-SPEC.md) allows, this audit MUST apply the pattern to every [Identifier] printed and confirm it lands on that page. A member added to the @@ -400,8 +408,9 @@ own [Defences] already run on its own source, because that is the only source th ## 9. Conformance **A project's [Toolchain] [Conforms]** if every MUST in sections 4 to 6 holds across the -assembled parts, wherever each part came from, and, where the [Toolchain] is one the project -ships, every MUST in section 8 as well. The project-level verdict is a single grade; which part of the +assembled parts, wherever each part came from; where the [Toolchain] is one the project +ships, every MUST in section 8 bears on its artefact grade under clause 9.1, not on this one. The +project-level verdict is a single grade; which part of the [Toolchain] satisfied each clause is evidence for that grade, not a second grade. **A [Toolchain] [Conforms] with [Agent] support** if it additionally satisfies section 7. @@ -433,15 +442,17 @@ record dependencies, the version of the method specification it follows and the document or the [detector specification](DETECTOR-SPEC.md) it [Conforms] to. A project that ships an artefact carries both levels of clause 9.1 in that declaration, each named separately. -The same declaration is where a gap is recorded once it is known. A project or artefact that has -learnt that it fails a MUST of the document it declares against MUST record that gap alongside the -version, in the same file or one it names. It may learn that from its own self-audit under section 8 -or from a [Practitioner]'s report under the method's clause 3.2. A declaration with a non-empty gap record is -a statement of where the project stands and is not a claim of [Conformance]. A mechanism gap -is by its nature one the [Toolchain] could not detect for itself, so the record is the only -place its [Owner] and its consumers can learn of it. - -The declaration is optional. A [Toolchain] assembled before this document existed, or by a +The same declaration is where a gap is recorded once it is known. A project or artefact that +declares a version and has learnt that it fails a MUST of the document it declares against MUST +record that gap alongside that version, in the same file or one it names. It may learn that from +its own self-audit under section 8 or from a [Practitioner]'s report under the method's clause +3.2. A declaration with a non-empty gap record is a statement of where the project stands and is +not a claim of [Conformance]. A mechanism gap is by its nature one the [Toolchain] could not +detect for itself, so the record is the only place its [Owner] and its consumers can learn of it. + +The declaration is optional, and so the duty above falls only on a project or artefact that has +one; a gap a [Practitioner] finds is recorded under the method's clause 3.2 in any case, whether +or not the project declares. A [Toolchain] assembled before this document existed, or by a project that has never read it, MAY be graded [Conforming] on evidence by anyone who exercises the clauses above against it. A verdict on any [Toolchain], declared or not, rests on that exercise and not on the claim. diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 2d2bea3..43053c9 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -15,7 +15,7 @@ no bullet making an assertion listed below as wrong, is a correct restatement. - Resolve every identifier a routed defence can print, including the project's own, offline, to documentation stating the correct construction ([4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access)). - Forbid, by disabling or by a blocking defence, every suppression route that bypasses the - project record ([4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record)). + project record ([4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)). - The toolchain's own entry point meets the detector reporting clauses and prints identifiers unaltered ([4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes)), and runs detectors before runners, stopping on a detector failure ([4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure)). - List the active defences, derived from configuration, including the project's own ([5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) to @@ -56,7 +56,7 @@ Wrong readings the text invites, which count against the document on one reader: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | None | Lintomatic fails detector [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) on two undocumented bundled rules and nothing in the wrapping closes that gap; a reader who notes that the detector specification was not supplied and grades from this document's summary of it is correct | | [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | None. A No given on the condition that the two bundled rules are blocking is a No: their firing stops `zig-qa run`, so they are | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)` | -| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | None | Ignore comments disabled project-wide; exceptions only through the record the toolchain reads | +| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) | Yes | None | Ignore comments disabled project-wide; exceptions only through the record the toolchain reads | | [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | None | `zig-qa run` is local, prints Lintomatic's output unchanged, one line per finding | | [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | None | Lintomatic, then the tests, stopping on a Lintomatic failure | | [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes, Partial or No | None | `zig-qa list` prints identifier, summary and documentation path without running the rules; Partial or No is the reading that counts the two `(none)` routes, since [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) says the listing MUST provide the route and [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) point 2 says a gap may be named again by a later clause | diff --git a/acceptance/toolchain/QUESTIONS.md b/acceptance/toolchain/QUESTIONS.md index d4e5cd7..3b58dba 100644 --- a/acceptance/toolchain/QUESTIONS.md +++ b/acceptance/toolchain/QUESTIONS.md @@ -24,7 +24,7 @@ the document you are relying on. ## 3. Application Grade the Zigpipe project's toolchain, as described in the fixture, against every MUST clause in -sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record): [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record), [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure), [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration), [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones), [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences). For each give Yes, +sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record): [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own), [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure), [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration), [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones), [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences). For each give Yes, Partial or No and one line of reasoning. Then state whether the toolchain conforms under section [9](../../TOOLING-SPEC.md#9-conformance) and why, whether it conforms with agent support and why, and whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies to it and why. Finally, state whether the absence of a declaration affects the verdict, and why. diff --git a/tools/php-qa-ci.md b/tools/php-qa-ci.md index d28c8b3..1337e56 100644 --- a/tools/php-qa-ci.md +++ b/tools/php-qa-ci.md @@ -22,7 +22,7 @@ Every mechanism below was checked by running it in a checkout: `bin/qa`, `bin/qa For its own PHPStan bundle, every detector clause holds. A project's rules go under `rules:` in its `qaConfig/phpstan.neon` (detector [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it)). `bin/phpstan-rule phpqaci.nullCoalescingFalse src/ZzProbe.php` on a probe containing `?? false` printed `FIRED (1)` with the location and exited 1, and the same command for `phpqaci.nestedTernary` printed `did not fire` (detector [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code)). PHPStan prints the `phpqaci.*` identifier under every finding, and `RequireRuleIdentifierConstantRule`, on by default, rejects a magic-string identifier (detector [4.3](../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding), detector [4.4](../DETECTOR-SPEC.md#44-the-detector-should-enforce-43-with-a-rule-of-its-own)). `bin/qa -t phpstan -p ` runs one lane over one file locally with the result in the terminal, and the CI script runs the same `bin/qa` (detector [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to detector [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally)). `bin/rule-doc phpqaci.nestedTernary` resolves the identifier as printed to its rule, bundle, source and shipped page, offline, and the page carries a `The correct construction` section (detector [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) to detector [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) for the bundle). `RuleDocumentationTest` fails the build when a declared identifier resolves to no page, or to a page whose construction section is missing or only restates the summary (detector [6.4](../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation)). Inline `@phpstan-ignore` in every form is a finding of `ForbidInlinePhpstanIgnoreRule` (detector [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)), and behind it the default tier turns on PHPStan's `reportIgnoresWithoutComments`: a probe ignore with no reason printed both `phpqaci.inlinePhpstanIgnore` and `ignore.noComment (non-ignorable)` (detector [7.2](../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason)). -At toolchain level, every lane prints a stable identifier when it fails: a run of `bin/qa -t branchNamePolicy` on a `plan/*` branch ended with `🪪 phpqaci.branchNamePolicy (vendor/bin/rule-doc phpqaci.branchNamePolicy)`, and `EveryLaneNamesItselfTest` fails the build on a lane without one. `bin/rule-doc phpqaci.packageType` resolves a lane identifier to its page as it does a rule's, and `bin/rules .` prints no `no documentation page` row: all 69 indexed identifiers (`bin/rule-doc --list`) resolve to a page ([4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access)). `bin/rules .` lists every rule in the resolved neon chain, including the rules `phpstan/extension-installer` delivers, with the package each came from, and every lane with its identifier and documentation route; phase runners alone carry none ([5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) to [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones)). The record is the `ignoreErrors` block PHPStan itself loads ([6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself)); the `pij` lane reads it through every `includes:` file, and a probe baseline included from `qaConfig/phpstan.neon` with an uncommented entry failed it, naming the included file and line ([6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record)). A `phparkitect-baseline.json` is never read: on a probe project with one violation, PHPArkitect with a generated baseline printed `No violations detected`, and with `--skip-baseline`, which the lane always passes, reported the violation; the lane names the file when one is present ([4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record)). The same listing enumerates the record beside the defences, together with the deferred-defect and no-pattern record `qaConfig/defect-record.neon` (`defectRecord` in `--json`), which the same lane fails on when an entry does not read in full ([6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences)). `docs/defence-before-fix-defaults.md` states the toolchain's default for the owner, the project record, the sweep scope, fixtures and the method's three calibrations, linked from the agent block ([6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)). The entry point runs coding standards, linting and static analysis before tests and stops on a failure ([4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure)). `bin/rules --write-agent-summary` writes one line per active defence, identifier and page, into a marked region of the project's `CLAUDE.md`, refreshed on every install and update ([7.1](../TOOLING-SPEC.md#71-the-toolchain-should-generate-a-summary-of-the-active-defences-suitable-for-an-agents-context), [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically)). The release guard covers rules and lanes and requires a page stating a correct construction ([8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation)), and the self-check runs both bundled rule sets and the PHPArkitect tier on the package's own source ([8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source)). The manifest declares method 1.1.0 and toolchain 0.2.0 at both levels, with the gaps below recorded against their clauses, and `DefenceBeforeFixDeclarationTest` holds each recorded gap to a decision and a probe that fails when the gap closes ([9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance)). +At toolchain level, every lane prints a stable identifier when it fails: a run of `bin/qa -t branchNamePolicy` on a `plan/*` branch ended with `🪪 phpqaci.branchNamePolicy (vendor/bin/rule-doc phpqaci.branchNamePolicy)`, and `EveryLaneNamesItselfTest` fails the build on a lane without one. `bin/rule-doc phpqaci.packageType` resolves a lane identifier to its page as it does a rule's, and `bin/rules .` prints no `no documentation page` row: all 69 indexed identifiers (`bin/rule-doc --list`) resolve to a page ([4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access)). `bin/rules .` lists every rule in the resolved neon chain, including the rules `phpstan/extension-installer` delivers, with the package each came from, and every lane with its identifier and documentation route; phase runners alone carry none ([5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) to [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones)). The record is the `ignoreErrors` block PHPStan itself loads ([6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself)); the `pij` lane reads it through every `includes:` file, and a probe baseline included from `qaConfig/phpstan.neon` with an uncommented entry failed it, naming the included file and line ([6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)). A `phparkitect-baseline.json` is never read: on a probe project with one violation, PHPArkitect with a generated baseline printed `No violations detected`, and with `--skip-baseline`, which the lane always passes, reported the violation; the lane names the file when one is present ([4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)). The same listing enumerates the record beside the defences, together with the deferred-defect and no-pattern record `qaConfig/defect-record.neon` (`defectRecord` in `--json`), which the same lane fails on when an entry does not read in full ([6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences)). `docs/defence-before-fix-defaults.md` states the toolchain's default for the owner, the project record, the sweep scope, fixtures and the method's three calibrations, linked from the agent block ([6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)). The entry point runs coding standards, linting and static analysis before tests and stops on a failure ([4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure)). `bin/rules --write-agent-summary` writes one line per active defence, identifier and page, into a marked region of the project's `CLAUDE.md`, refreshed on every install and update ([7.1](../TOOLING-SPEC.md#71-the-toolchain-should-generate-a-summary-of-the-active-defences-suitable-for-an-agents-context), [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically)). The release guard covers rules and lanes and requires a page stating a correct construction ([8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation)), and the self-check runs both bundled rule sets and the PHPArkitect tier on the package's own source ([8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source)). The manifest declares method 1.1.0 and toolchain 0.2.0 at both levels, with the gaps below recorded against their clauses, and `DefenceBeforeFixDeclarationTest` holds each recorded gap to a decision and a probe that fails when the gap closes ([9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance)). ## How it is not conformant @@ -47,40 +47,40 @@ php-qa-ci is a toolchain rather than a tool, so the question for each lane it ru Detector rows grade PHPStan as wrapped, with PHPArkitect as wrapped noted where it differs. -| Document | Clause | Result | Evidence | -| --------- | ------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------- | -| Detector | [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules under `rules:` in `qaConfig/phpstan.neon`; PHPArkitect project rules in `phparkitect.php` | -| Detector | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `bin/phpstan-rule ` fired red on a probe and stayed silent on another rule; PHPArkitect as wrapped has none | -| Detector | [4.3](../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `🪪 phpqaci.nullCoalescingFalse` printed under the finding, identifier in JSON mode too; PHPArkitect prints prose only | -| Detector | [4.4](../DETECTOR-SPEC.md#44-the-detector-should-enforce-43-with-a-rule-of-its-own) | Yes | `RequireRuleIdentifierConstantRule` in `rules-default.neon` | -| Detector | [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | Every command ran in the checkout with no service | -| Detector | [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Yes | `bin/qa -t phpstan -p src/ZzProbe.php` scanned that path alone; PHPArkitect's lane is non-path-supporting | -| Detector | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | Findings in the terminal, archived log named after them | -| Detector | [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | `ci.bash` runs `bin/qa` | -| Detector | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Yes | `bin/rule-doc phpqaci.nestedTernary` resolved as printed to its page; an unknown identifier exits 1 | -| Detector | [6.2](../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Partial | Bundle and lanes resolve offline, 69 of 69; `bin/rule-doc method.notFound` names phpstan.org, PHPStan's catalogue is online only | -| Detector | [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | Partial | Bundle and lane pages ship with the package; PHPStan's own rules ship none; PHPArkitect's tier rules have no page | -| Detector | [6.4](../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation) | Yes | `RuleDocumentationTest` fails the build on an identifier with no page or a page stating no correct construction | -| Detector | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | Inline ignores detectable by a bundled rule; PHPArkitect's baseline never read (`--skip-baseline` always passed) | -| Detector | [7.2](../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) | Yes | `reportIgnoresWithoutComments` on in the default tier; a probe printed `ignore.noComment (non-ignorable)` | -| Toolchain | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | PHPArkitect's tier rules carry no identifier, single-file run or resolver; PHPStan's native catalogue is online only; both declared | -| Toolchain | [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | Yes | `bin/rules .`: no `no documentation page` row; `bin/rule-doc phpqaci.packageType` resolves to its page | -| Toolchain | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | Inline ignore forbidden; `phparkitect-baseline.json` never read; an included PHPStan baseline fails the justification lane | -| Toolchain | [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | `bin/qa -t -p ` for every path-supporting lane | -| Toolchain | [4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | `bin/qa` runs coding standards, linting, static analysis, then tests, failing fast or reporting every failure before the success banner | -| Toolchain | [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes | Rules listed with identifier, summary, route and source package; lanes with identifier and page | -| Toolchain | [5.2](../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | Derived from the resolved neon chain and the lane registry | -| Toolchain | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | Yes | Contributor-only rules in the self-listing | -| Toolchain | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes | `ignoreErrors` in `qaConfig/phpstan.neon`, loaded by PHPStan | -| Toolchain | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Yes | `bin/qa -t pij` reads the whole include chain; an uncommented entry in an included file failed it | -| Toolchain | [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | `bin/rules .` lists the record, an included baseline's entries among it, and the deferred-defect record `qaConfig/defect-record.neon`, in text and JSON | -| Toolchain | [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project) | Yes | `docs/defence-before-fix-defaults.md` states the owner, record, sweep, fixture and calibration defaults | -| Toolchain | [7.1](../TOOLING-SPEC.md#71-the-toolchain-should-generate-a-summary-of-the-active-defences-suitable-for-an-agents-context) | Yes | `bin/rules --write-agent-summary` writes one line per active defence into the `CLAUDE.md` region | -| Toolchain | [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically) | Yes | Block written and refreshed on install and update | -| Toolchain | [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | Yes | `RuleDocumentationTest` covers rules and lanes and requires a page stating a correct construction | -| Toolchain | [8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source) | Yes | `SelfCheckRunsBundledRulesTest`; own `qaConfig/phpstan.neon` includes both bundles; `bin/qa -t arch` runs the tier on `src/` | -| Toolchain | [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Yes | Artefact and project graded separately above | -| Toolchain | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | Yes | `composer.json` `extra.defence-before-fix`: method 1.1.0, toolchain 0.2.0, two gaps; `project` with the same keys and one gap | +| Document | Clause | Result | Evidence | +| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Detector | [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules under `rules:` in `qaConfig/phpstan.neon`; PHPArkitect project rules in `phparkitect.php` | +| Detector | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `bin/phpstan-rule ` fired red on a probe and stayed silent on another rule; PHPArkitect as wrapped has none | +| Detector | [4.3](../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `🪪 phpqaci.nullCoalescingFalse` printed under the finding, identifier in JSON mode too; PHPArkitect prints prose only | +| Detector | [4.4](../DETECTOR-SPEC.md#44-the-detector-should-enforce-43-with-a-rule-of-its-own) | Yes | `RequireRuleIdentifierConstantRule` in `rules-default.neon` | +| Detector | [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | Every command ran in the checkout with no service | +| Detector | [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Yes | `bin/qa -t phpstan -p src/ZzProbe.php` scanned that path alone; PHPArkitect's lane is non-path-supporting | +| Detector | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | Findings in the terminal, archived log named after them | +| Detector | [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | `ci.bash` runs `bin/qa` | +| Detector | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Yes | `bin/rule-doc phpqaci.nestedTernary` resolved as printed to its page; an unknown identifier exits 1 | +| Detector | [6.2](../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Partial | Bundle and lanes resolve offline, 69 of 69; `bin/rule-doc method.notFound` names phpstan.org, PHPStan's catalogue is online only | +| Detector | [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | Partial | Bundle and lane pages ship with the package; PHPStan's own rules ship none; PHPArkitect's tier rules have no page | +| Detector | [6.4](../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation) | Yes | `RuleDocumentationTest` fails the build on an identifier with no page or a page stating no correct construction | +| Detector | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | Inline ignores detectable by a bundled rule; PHPArkitect's baseline never read (`--skip-baseline` always passed) | +| Detector | [7.2](../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) | Yes | `reportIgnoresWithoutComments` on in the default tier; a probe printed `ignore.noComment (non-ignorable)` | +| Toolchain | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | PHPArkitect's tier rules carry no identifier, single-file run or resolver; PHPStan's native catalogue is online only; both declared | +| Toolchain | [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | Yes | `bin/rules .`: no `no documentation page` row; `bin/rule-doc phpqaci.packageType` resolves to its page | +| Toolchain | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) | Yes | Inline ignore forbidden; `phparkitect-baseline.json` never read; an included PHPStan baseline fails the justification lane | +| Toolchain | [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | `bin/qa -t -p ` for every path-supporting lane | +| Toolchain | [4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | `bin/qa` runs coding standards, linting, static analysis, then tests, failing fast or reporting every failure before the success banner | +| Toolchain | [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes | Rules listed with identifier, summary, route and source package; lanes with identifier and page | +| Toolchain | [5.2](../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | Derived from the resolved neon chain and the lane registry | +| Toolchain | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | Yes | Contributor-only rules in the self-listing | +| Toolchain | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes | `ignoreErrors` in `qaConfig/phpstan.neon`, loaded by PHPStan | +| Toolchain | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Yes | `bin/qa -t pij` reads the whole include chain; an uncommented entry in an included file failed it | +| Toolchain | [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | `bin/rules .` lists the record, an included baseline's entries among it, and the deferred-defect record `qaConfig/defect-record.neon`, in text and JSON | +| Toolchain | [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project) | Yes | `docs/defence-before-fix-defaults.md` states the owner, record, sweep, fixture and calibration defaults | +| Toolchain | [7.1](../TOOLING-SPEC.md#71-the-toolchain-should-generate-a-summary-of-the-active-defences-suitable-for-an-agents-context) | Yes | `bin/rules --write-agent-summary` writes one line per active defence into the `CLAUDE.md` region | +| Toolchain | [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically) | Yes | Block written and refreshed on install and update | +| Toolchain | [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | Yes | `RuleDocumentationTest` covers rules and lanes and requires a page stating a correct construction | +| Toolchain | [8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source) | Yes | `SelfCheckRunsBundledRulesTest`; own `qaConfig/phpstan.neon` includes both bundles; `bin/qa -t arch` runs the tier on `src/` | +| Toolchain | [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Yes | Artefact and project graded separately above | +| Toolchain | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | Yes | `composer.json` `extra.defence-before-fix`: method 1.1.0, toolchain 0.2.0, two gaps; `project` with the same keys and one gap | ## Notes for a practitioner diff --git a/tools/ts-qa-ci.md b/tools/ts-qa-ci.md index aed07db..844ddbf 100644 --- a/tools/ts-qa-ci.md +++ b/tools/ts-qa-ci.md @@ -39,7 +39,7 @@ The three conformance mechanisms cover the ESLint lanes in full and nothing else Bespoke rules under detector clause [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) enter through `tsQaConfig/eslint.config.js`, which is merged into the resolved flat config, so any ESLint plugin object the project writes is a first-class defence. Clause [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) is met by `ts-qa rule `, which runs the ordinary lint lane over one path and reports whether that one rule fired, exiting 1 with locations when it did; a fixture carrying an `eslint-disable` comment made `ts-qa/no-eslint-disable` fire and a clean rule stay silent on the same file. Identifiers are `ts-qa/`, chosen once in the plugin map and printed by ESLint with every finding, which is clause [4.3](../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding). Section [5](../DETECTOR-SPEC.md#5-reporting) is met by the entry point, its `-p ` subset flag and the `--llm` verdict table, which names the failing tool and the path to the full result. Clauses [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) to [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) are met for every bundled plugin rule by `ts-qa rule-doc `, which resolves the identifier exactly as printed to its section of the shipped `docs/cdd-rules.md` and prints it, offline, and a test in the suite walks every plugin rule to prove the page is there. That closes for the toolchain what ESLint alone fails, since ESLint's own rule pages are online only. -On the toolchain document, the listing under clauses [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) and [5.2](../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) is `ts-qa rules`, which walks the resolved configuration and prints each active defence's identifier, severity, a terse description and the on-disk route, then the project record beneath it in the same listing; the repository's own rule against its own source appears alongside the bundled ones. The project record under [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) is `tsQaConfig/tier-a-exemptions.json`, loaded by the toolchain itself; an entry without a justification throws, and so do "legacy" and "needed for now", each rejection naming the entry and asking for the hazard and the scope, which is [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) ([configuration](https://github.com/LongTermSupport/ts-qa-ci/blob/main/docs/configuration.md)). Inline suppression is forbidden under [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) by `ts-qa/no-eslint-disable`, which fires on every `eslint-disable`, `oxlint-disable` and `@ts-` directive form and points the reader at the record. Clause [4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) holds: phases 0 to 3 are detectors, phase 4 is the test runner, and a run that fails at phase 1 runs nothing after it. Clause [8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source) is the continuous integration workflow, which runs the toolchain on the package itself and fails on anything it cannot fix. The declaration under [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) is the `defenceBeforeFix` key in `package.json`, carrying both levels of clause [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) with their gaps recorded, so neither level claims conformance. +On the toolchain document, the listing under clauses [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) and [5.2](../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) is `ts-qa rules`, which walks the resolved configuration and prints each active defence's identifier, severity, a terse description and the on-disk route, then the project record beneath it in the same listing; the repository's own rule against its own source appears alongside the bundled ones. The project record under [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) is `tsQaConfig/tier-a-exemptions.json`, loaded by the toolchain itself; an entry without a justification throws, and so do "legacy" and "needed for now", each rejection naming the entry and asking for the hazard and the scope, which is [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) ([configuration](https://github.com/LongTermSupport/ts-qa-ci/blob/main/docs/configuration.md)). Inline suppression is forbidden under [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) by `ts-qa/no-eslint-disable`, which fires on every `eslint-disable`, `oxlint-disable` and `@ts-` directive form and points the reader at the record. Clause [4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) holds: phases 0 to 3 are detectors, phase 4 is the test runner, and a run that fails at phase 1 runs nothing after it. Clause [8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source) is the continuous integration workflow, which runs the toolchain on the package itself and fails on anything it cannot fix. The declaration under [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) is the `defenceBeforeFix` key in `package.json`, carrying both levels of clause [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) with their gaps recorded, so neither level claims conformance. ## How it is not conformant @@ -47,7 +47,7 @@ Three causes, each verified through the entry point. **The dependency-cruiser lane is outside the three commands.** The shipped `no-circular` defence prints its identifier with a two-file cycle and fails the run, but `ts-qa rule-doc no-circular` exits 2 as unknown, `ts-qa rules` does not list it, the lane ignores `-p`, and the release self-audit walks plugin rules only. Through the wrapping, dependency-cruiser fails detector clauses [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) and [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation), which fails toolchain [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), and the toolchain fails [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) and [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) on the same fact. `ts-qa rule no-circular src` compounds it by answering "did not fire" with exit 0 for an identifier the harness cannot see. -**ESLint's bulk suppressions file is an open route.** A file containing the literal `PLACEHOLDER` fails the lint lane. With an `eslint-suppressions.json` in the project root naming that file and `ts-qa/no-placeholder`, the same run passes with every phase clean and the project record unchanged. Nothing disables the file or fails on its presence, so toolchain [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) fails, though every inline form is banned. +**ESLint's bulk suppressions file is an open route.** A file containing the literal `PLACEHOLDER` fails the lint lane. With an `eslint-suppressions.json` in the project root naming that file and `ts-qa/no-placeholder`, the same run passes with every phase clean and the project record unchanged. Nothing disables the file or fails on its presence, so toolchain [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) fails, though every inline form is banned. **Resolution of a core rule stops at a URL.** The always-on as/enum ban is routed through ESLint's `no-restricted-syntax`, and `ts-qa rule-doc no-restricted-syntax` resolves to a one-line description and eslint.org, not to the toolchain's own shipped page for the ban. Detector [6.2](../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) and [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) fail for that identifier and toolchain [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) with them. @@ -55,40 +55,40 @@ The derived listing also exposed a defect: `ts-qa/no-naive-datetime-template` is ## Clause by clause -| Document | Clause | Result | Evidence | -| --------- | ------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | -| Detector | [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | `tsQaConfig/eslint.config.js` merged into the resolved config; a project dependency-cruiser config replaces the shipped one | -| Detector | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Partial | `ts-qa rule ts-qa/no-eslint-disable src/probe.ts` exited 1 with the location and a silent rule exited 0; no route for a dependency-cruiser rule | -| Detector | [4.3](../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `ts-qa/` chosen in the plugin map and printed unaltered; `no-circular` printed with its cycle | -| Detector | [4.4](../DETECTOR-SPEC.md#44-the-detector-should-enforce-43-with-a-rule-of-its-own) | No | No rule over the rules | -| Detector | [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | `npx ts-qa`, no infrastructure | -| Detector | [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Partial | `-p ` and the single-path `rule` command for the ESLint lane; the dependency-cruiser lane ignores `-p` | -| Detector | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | `--llm` verdict table names the failing tool and the path to the full result | -| Detector | [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | CI runs the same command | -| Detector | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Partial | `rule-doc ts-qa/no-eslint-disable` printed the shipped section; `rule-doc no-circular` exited 2 as unknown | -| Detector | [6.2](../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Partial | `docs/` ships and the printed route is a local path for plugin rules; a core rule resolves to eslint.org only | -| Detector | [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | Partial | Every plugin rule has a section in `docs/cdd-rules.md`, pinned by a test; no page for `no-circular`, none shipped for core rules | -| Detector | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | Directives detected by `ts-qa/no-eslint-disable`; the known-violations baseline needs a flag the toolchain never passes | -| Detector | [7.2](../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) | No | A directive's reason is optional and a baseline entry carries none | -| Toolchain | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | dependency-cruiser plus the wrapping fails detector [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) and [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) and a defence is routed through it | -| Toolchain | [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | `no-circular` unresolvable; `no-restricted-syntax` resolves to an upstream URL, not the shipped as/enum ban page | -| Toolchain | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | No | A `PLACEHOLDER` finding failed the lint lane; with `eslint-suppressions.json` naming it the run passed, all phases clean | -| Toolchain | [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | No | Entry point meets detector [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) for the ESLint lane; no subset for the dependency-cruiser lane | -| Toolchain | [4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | Detectors in phases 0 to 3, tests in 4; a failure at phase 1 ran nothing after it | -| Toolchain | [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | No | `ts-qa rules` printed eighteen defences with identifier, description and route; `no-circular` absent | -| Toolchain | [5.2](../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | The listing walks the resolved ESLint configuration, and showed a declared Tier A rule that the configuration never enables | -| Toolchain | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | No | The repository's own plugin rule is listed; its dependency-cruiser defence is not | -| Toolchain | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes | `tsQaConfig/tier-a-exemptions.json`, loaded by the toolchain on every run | -| Toolchain | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Yes | Missing, "legacy" and "needed for now" justifications each rejected with the entry named | -| Toolchain | [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | The project record is printed by `ts-qa rules` beneath the defences | -| Toolchain | [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project) | Yes | The always-on tier and its reasoning are documented in `docs/cdd-rules.md` and `docs/coding-standards.md` | -| Toolchain | [7.1](../TOOLING-SPEC.md#71-the-toolchain-should-generate-a-summary-of-the-active-defences-suitable-for-an-agents-context) | Partial | One line per defence with identifier and route, phrased as a description | -| Toolchain | [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically) | Partial | `deploy-skills` delivers static skills, not the generated summary | -| Toolchain | [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | No | The self-audit test walks plugin rules only; `no-circular` is outside it | -| Toolchain | [8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source) | No | CI runs the toolchain on itself, but `ts-qa/no-naive-datetime-template` ships and is active nowhere | -| Toolchain | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | Yes | `package.json` `defenceBeforeFix`: method 1.0.0, toolchain 0.2.0 at both levels, with the gaps above in `knownGaps` | -| Method 7 | [3.3](../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Partial | Twenty-five bundled rules have a fixture test; `no-duplicate-section-ids` and `no-placeholder` have none | -| Method 7 | record | Yes | Two justified, scoped exemptions, printed on every run and by `rules` | +| Document | Clause | Result | Evidence | +| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Detector | [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | `tsQaConfig/eslint.config.js` merged into the resolved config; a project dependency-cruiser config replaces the shipped one | +| Detector | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Partial | `ts-qa rule ts-qa/no-eslint-disable src/probe.ts` exited 1 with the location and a silent rule exited 0; no route for a dependency-cruiser rule | +| Detector | [4.3](../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `ts-qa/` chosen in the plugin map and printed unaltered; `no-circular` printed with its cycle | +| Detector | [4.4](../DETECTOR-SPEC.md#44-the-detector-should-enforce-43-with-a-rule-of-its-own) | No | No rule over the rules | +| Detector | [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | `npx ts-qa`, no infrastructure | +| Detector | [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Partial | `-p ` and the single-path `rule` command for the ESLint lane; the dependency-cruiser lane ignores `-p` | +| Detector | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | `--llm` verdict table names the failing tool and the path to the full result | +| Detector | [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | CI runs the same command | +| Detector | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Partial | `rule-doc ts-qa/no-eslint-disable` printed the shipped section; `rule-doc no-circular` exited 2 as unknown | +| Detector | [6.2](../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Partial | `docs/` ships and the printed route is a local path for plugin rules; a core rule resolves to eslint.org only | +| Detector | [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | Partial | Every plugin rule has a section in `docs/cdd-rules.md`, pinned by a test; no page for `no-circular`, none shipped for core rules | +| Detector | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | Directives detected by `ts-qa/no-eslint-disable`; the known-violations baseline needs a flag the toolchain never passes | +| Detector | [7.2](../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) | No | A directive's reason is optional and a baseline entry carries none | +| Toolchain | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | dependency-cruiser plus the wrapping fails detector [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) and [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) and a defence is routed through it | +| Toolchain | [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | `no-circular` unresolvable; `no-restricted-syntax` resolves to an upstream URL, not the shipped as/enum ban page | +| Toolchain | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) | No | A `PLACEHOLDER` finding failed the lint lane; with `eslint-suppressions.json` naming it the run passed, all phases clean | +| Toolchain | [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | No | Entry point meets detector [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) for the ESLint lane; no subset for the dependency-cruiser lane | +| Toolchain | [4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | Detectors in phases 0 to 3, tests in 4; a failure at phase 1 ran nothing after it | +| Toolchain | [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | No | `ts-qa rules` printed eighteen defences with identifier, description and route; `no-circular` absent | +| Toolchain | [5.2](../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | The listing walks the resolved ESLint configuration, and showed a declared Tier A rule that the configuration never enables | +| Toolchain | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | No | The repository's own plugin rule is listed; its dependency-cruiser defence is not | +| Toolchain | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes | `tsQaConfig/tier-a-exemptions.json`, loaded by the toolchain on every run | +| Toolchain | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Yes | Missing, "legacy" and "needed for now" justifications each rejected with the entry named | +| Toolchain | [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | The project record is printed by `ts-qa rules` beneath the defences | +| Toolchain | [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project) | Yes | The always-on tier and its reasoning are documented in `docs/cdd-rules.md` and `docs/coding-standards.md` | +| Toolchain | [7.1](../TOOLING-SPEC.md#71-the-toolchain-should-generate-a-summary-of-the-active-defences-suitable-for-an-agents-context) | Partial | One line per defence with identifier and route, phrased as a description | +| Toolchain | [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically) | Partial | `deploy-skills` delivers static skills, not the generated summary | +| Toolchain | [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | No | The self-audit test walks plugin rules only; `no-circular` is outside it | +| Toolchain | [8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source) | No | CI runs the toolchain on itself, but `ts-qa/no-naive-datetime-template` ships and is active nowhere | +| Toolchain | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | Yes | `package.json` `defenceBeforeFix`: method 1.0.0, toolchain 0.2.0 at both levels, with the gaps above in `knownGaps` | +| Method 7 | [3.3](../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Partial | Twenty-five bundled rules have a fixture test; `no-duplicate-section-ids` and `no-placeholder` have none | +| Method 7 | record | Yes | Two justified, scoped exemptions, printed on every run and by `rules` | ## Notes for a practitioner From 0652ee89c59bcbe5d5d6178c7ff36cf154c95da4 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 12:38:57 +0100 Subject: [PATCH 07/52] Run 057: toolchain-contradictions cohort, failed Every load-bearing statement five of five; clause 4.3 three of five, two readers grading Partial because the fixture names no route other than the ignore comment; five findings in text the change wrote, chiefly clause 4.2's sentence calling an undocumented blocking rule "a Defence with a gap", which every reader set against the section 2 Defence row; one finding routed, section 9's agent-support sentence against section 7's SHOULDs. Co-Authored-By: Claude Fable 5.1 --- .../057-toolchain-contradictions-run-1.md | 148 ++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100644 acceptance/runs/057-toolchain-contradictions-run-1.md diff --git a/acceptance/runs/057-toolchain-contradictions-run-1.md b/acceptance/runs/057-toolchain-contradictions-run-1.md new file mode 100644 index 0000000..b222a30 --- /dev/null +++ b/acceptance/runs/057-toolchain-contradictions-run-1.md @@ -0,0 +1,148 @@ +# Run 57: TOOLING-SPEC.md, toolchain contradictions, first run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: not yet opened at the time of the run; it is to be opened from branch + `editor/toolchain-contradictions`. The readers read commit 4d6ebbd. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. +- **Run within the pull request**: 1 +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `00f3043ad8fef7967b0915398bcfaca27d646f8112b0b6df4f9d7d75065eda60`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at 4d6ebbd, each after one + `===== FILE: =====` line; the marker rebuilt it from that commit and the hash matched. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. +- **Process change**: the run was made under the acceptance process change of pull request 22 + before it merged, following all of it at commit c31d49a, named before the readers started. +- **Verdict**: FAIL + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | ------------ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | miss (Part.) | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | miss (Part.) | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Partial (1, 2, 5), No (3, 4); +6.1 Yes (1, 4, 5), Partial (2, 3), both Partials mentioning that no place is given for +calibrations or conventions; 6.2 No (all five), each mentioning that nothing rejects a generic +justification. + +Question 1: correct bullets per reader 4, 5, 5, 5 and 5 (reader 1: 4.1, 4.2, 4.3 with 4.4 and +4.5, sections 5 and 6). No bullet asserts anything the key marks wrong. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote the section 1 sentence "It is whatever the project assembles to run its +checks through", which states element (2). + +### Misses + +- **Reader 1, 4.3, Partial.** The inline `// lintomatic-ignore` route is disabled, but "the + fixture is silent on any other route, such as a baseline, so 'every route' is not shown". The + reasoning does not quote the document (its only quotation is two words). +- **Reader 2, 4.3, Partial.** The inline route is disabled and exceptions go to the record, but + the fixture does not say whether Lintomatic has a baseline or another route, "and the clause + covers every route. I cannot confirm the list is complete." The reasoning does not quote the + document. + +The key accepts Yes only for 4.3 and names no wrong reading for it, so neither miss counts +against the document on its own; two misses on one item fail the threshold for questions 1 and +3\. Readers 3, 4 and 5 also noted the fixture's silence on baselines and graded Yes on what is +written. No miss matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..4d6ebbd adds, +rewords or deletes a sentence in the paragraph. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------- | ------------------- | +| 1 | Section 2 terminology table, [Defence] row: "A [Rule] with its documentation, in force and [Blocking]", set against clause 4.2's first paragraph | 1, 2, 3, 5 | no | +| 2 | Clause 4.2, first paragraph: "A [Blocking] [Rule] whose documentation is missing ... graded here as a [Defence] with a gap" (1, 2, 3, 5); "covers every [Identifier] the project can see" (4) | 1, 2, 3, 4, 5 | yes | +| 3 | Clause 4.2, second paragraph: "A third-party [Detector]'s native catalogue ... is resolved under that document's clause 6 and is not re-shipped here", set against the first paragraph | 3, 4 | no | +| 4 | Section 1, "How to read this document": "It is self-contained for grading a [Toolchain]" (2, set against clause 4.1's "every one of"); "The table in section 2 glosses every term it borrows" (3) | 2, 3 | no | +| 5 | Clause 9.1, first paragraph: "as it stands when installed into a [Consuming project] with nothing else built around it" (1); "is graded against this document and section 7 of the method specification" (5) | 1, 5 | no | +| 6 | Section 9, second paragraph: "A [Toolchain] [Conforms] with [Agent] support if it additionally satisfies section 7", section 7 being all SHOULDs | 3, 5 | no | +| 7 | Clause 4.1, first paragraph: requires "every one of" the detector specification's MUSTs in its sections 4 to 7 | 2 | yes | +| 8 | Section 9, first paragraph: "assembled parts, wherever each part came from" | 1 | yes | +| 9 | Clause 4.1, point 1: "is then what the [Practitioner] uses and what is judged" | 1 | no | +| 10 | Section 2 terminology, [Bundled defence] entry: "with its documentation, [Blocking], and routed through the [Toolchain]'s own entry point" | 4 | no | +| 11 | Clause 8.1, first paragraph: "a [Bundled defence] whose documentation is missing is still a [Bundled defence] for this clause" | 4 | yes | +| 12 | Section 3 table, [Identifier] resolution row, routing [Bundled rules] to the detector specification | 3 | no | +| 13 | Clause 4.3, first paragraph: "MUST disable every [Suppression] route ... or MUST run a [Blocking] [Defence] that fails on its use" | 3 | no | +| 14 | Clause 4.3, second paragraph: "This clause is where the project's decision is made and enforced, and the decision is no." | 3 | no | + +## Findings and where they lie + +- **Finding A, passage 2, clause 4.2 first paragraph** (five readers): text the change wrote. + Blocks. +- **Finding B, passage 1, section 2 [Defence] row** (four readers): every reader naming it sets + it against clause 4.2's first paragraph, which the change altered. Blocks. +- **Finding C, passage 3, clause 4.2 second paragraph** (readers 3 and 4): both set it against + clause 4.2's first paragraph, which the change altered. Blocks. +- **Finding D, passage 4, section 1 "How to read this document"** (readers 2 and 3): reader 2 + sets it against clause 4.1's first paragraph, which the change altered. Blocks. +- **Finding E, passage 5, clause 9.1 first paragraph** (readers 1 and 5): reader 1 sets it + against section 9's first paragraph, which the change altered. Blocks. +- **Finding F, passage 6, section 9 second paragraph** (readers 3 and 5): neither reader sets it + against a paragraph the change altered; both set it against section 7, which is unchanged. + Routed to `acceptance/ROUTED.md`; does not block. + +Passages 7 to 14 are each named by one reader: recorded, not findings. + +## Points the rules did not settle + +1. **Reader 5, clause 9.1.** The reader quotes "is graded against this document and section 7 of + the method specification" and says it could not resolve the reference because it had only + this document. That is close to the example of a remark that is not about a passage (another + document not supplied), but it quotes the document and names its clause. Marked by the + open-point rule: an open point on whether a reader named a passage counts the passage. It + therefore counts, and with reader 1 makes finding E. +2. **Question 1 bullets that state how conformance is measured rather than what the toolchain + must do**: reader 1's first and sixth ("judged on the whole toolchain"; "one project-level + grade"), reader 2's first, reader 4's first sentence ("a single conformance grade"), reader + 5's first and reader 3's sixth ("Self-audit (8) and the two-grade split (9.1) apply only to a + toolchain that a project ships"). They are on neither of the key's lists. Marked by the rule + for a bullet on neither list: not counted either way. Every reader has four or more correct + bullets without them, so no mark turns on it. For the key to settle. +3. **Section 7 described as "all SHOULDs"** (readers 3 and 5) is a description of a whole + section, which identifies no sentence or paragraph, so under the definition of naming it is + not a passage of its own; it is the reason both readers give for naming section 9's second + paragraph (passage 6). Had it counted, it would be a passage the change did not touch and + would be routed, not block. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item except + 4.3, which is three of five. Not met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: five (findings A + to E). Not met. +- Passages named by two or more readers elsewhere: one (finding F), routed to + `acceptance/ROUTED.md`. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From 90a99361618281c8e7f75dbc270deec9c95b4d55 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 12:38:57 +0100 Subject: [PATCH 08/52] Toolchain 1, 4.1, 4.2, 4.3, 8.1, 9 and 9.1: the passages run 57 failed on Clause 4.2 now says Defence there reaches every blocking rule the toolchain runs, whether or not its documentation exists, in place of "a Defence with a gap"; its native-catalogue sentence says an identifier the detector resolves meets the clause and one nothing resolves fails it; clause 8.1's new sentence reads the same way; section 1 says the document defers to the detector specification only in clause 4.1, and clause 4.1 says a verdict from this document alone rests on the six it lists; clause 4.3 says the routes are the ones each detector offers; clause 9.1 says the artefact grade counts what the shipped toolchain supplies and nothing the consuming project adds; section 9's agent-support sentence says section 7 recommends rather than requires. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 11 +++++++++-- TOOLING-SPEC.md | 38 ++++++++++++++++++++++++-------------- 2 files changed, 33 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ab6229f..ad5f0e6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -111,8 +111,15 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be the toolchain's own. Clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) says every MUST of the detector specification's sections [4](DETECTOR-SPEC.md#4-authoring-rules) to [7](DETECTOR-SPEC.md#7-suppression) applies and the six it lists are a summary, and its point 2 says the wrapped pair is what the clause judges whilst the detector's own grade is a separate verdict. Prompted by runs 33 to 52 - and 55 on the defined-terms change, in which two or more readers quoted each passage. Accepted - by run NNN. + and 55 on the defined-terms change, in which two or more readers quoted each passage. + [Run 57](acceptance/runs/057-toolchain-contradictions-run-1.md) failed on the first wording: every reader set clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access)'s + "a defence with a gap" against the Defence row, so the sentence now says Defence there + reaches every blocking rule whether or not its documentation exists; clause [4.3](TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) says the + routes are the ones each detector offers, after two readers graded it Partial for routes the + fixture does not describe; section [1](TOOLING-SPEC.md#1-what-this-document-is-for) says the document defers to the detector specification + only in clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification); clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) says what the artefact grade counts; and section [9](TOOLING-SPEC.md#9-conformance)'s agent-support + sentence says section [7](TOOLING-SPEC.md#7-agent-context) recommends, after two readers set "additionally satisfies" against + that section's SHOULDs. Accepted by run NNN. ### 0.2.0, 2026-09-08 diff --git a/TOOLING-SPEC.md b/TOOLING-SPEC.md index 1c97502..5ad5f51 100644 --- a/TOOLING-SPEC.md +++ b/TOOLING-SPEC.md @@ -39,7 +39,9 @@ building the mechanism first. A [Toolchain] that leaves the project to construct moved the obligation rather than met it. Where the project builds the means itself, those scripts are part of its [Toolchain] and are judged as such. -**How to read this document.** It is self-contained for grading a [Toolchain]. The +**How to read this document.** It is self-contained for grading a [Toolchain], apart from clause +4.1, which holds each [Detector] to the [detector specification](DETECTOR-SPEC.md) and summarises +that document's MUSTs. The table in section 2 glosses every term it borrows. Each clause states its obligation in its heading and first sentence, and the **Why** paragraph beneath is reasoning, not a further requirement. A link is there for depth and is not a prerequisite for the sentence that carries it. @@ -110,7 +112,8 @@ any row and the failure mode is visible too: either half alone leaves the [Pract A [Toolchain] MUST NOT route a [Defence] through a [Detector] that does not [Conform] to [the detector specification](DETECTOR-SPEC.md). That document's MUSTs are those of its sections 4 to 7, and every one of them applies; the six summarised here are the ones most -often failed. The [Detector] hosts bespoke [Rules]. It runs one [Rule] on its own against supplied +often failed, and a verdict on this clause taken from this document alone rests on them. The +[Detector] hosts bespoke [Rules]. It runs one [Rule] on its own against supplied code. It prints a stable [Identifier] with every finding. It runs locally, down to one file, with the result in the command's own output and nothing reportable only through a hosted service. It resolves every [Identifier] a [Bundled rule] can print to documentation shipped with it, offline. @@ -160,9 +163,10 @@ This clause widens the [detector specification](DETECTOR-SPEC.md)'s reach: that [Bundled rules] only, and this one covers every [Identifier] the project can see, the project's own [Rules] included. The [Toolchain] MUST resolve every [Identifier] its [Defences] print to [Remediation docs] on disk. One [Identifier] that does not resolve fails this clause, -however many others do. A [Blocking] [Rule] whose documentation is missing is the case this -clause exists to catch: it is graded here as a [Defence] with a gap, not left out as no -[Defence] at all. The mechanism is keyed on the +however many others do. Here [Defence] reaches every [Blocking] [Rule] the [Toolchain] runs, +bundled or the project's own, whether or not its documentation exists: a [Rule] whose +documentation is missing is the failure this clause names, not a [Rule] outside it. The mechanism +is keyed on the [Identifier] exactly as printed, and it covers the project's own [Rules] and every [Bundled defence]. Those [Remediation docs] MUST ship with the project or with the [Toolchain], at a version tracked together. The @@ -175,8 +179,9 @@ only the prohibition, specifically enough to act on, as method clause 8.4 requir This clause closes what the [detector specification](DETECTOR-SPEC.md) leaves open. That document holds a [Detector] to on-disk resolution of its own [Bundled rules] and no further, because it cannot know a project's documentation. A third-party [Detector]'s -native catalogue, which the [Toolchain] orchestrates without claiming as its own, is -resolved under that document's clause 6 and is not re-shipped here. +native catalogue, which the [Toolchain] orchestrates without claiming as its own, is not +re-shipped here: an [Identifier] the [Detector] resolves under that document's clause 6 meets +this clause, and one that nothing resolves fails it. **Why**: method specification clauses 3.6 and 8.3. The [Identifier] has to resolve for the reader, which for an [Agent] means mechanically and on disk, and the project's own @@ -193,7 +198,10 @@ route that bypasses the [Project record], or MUST run a [Blocking] silenced through it without an entry appearing in the [Project record]: an inline [Suppression] comment, a per-line ignore and a silently generated [Baseline] all do. The obligation covers every such route in every [Detector] the [Toolchain] routes a -[Defence] through. Irreducible cases MUST be directed to the [Project record], +[Defence] through. The routes are the ones each [Detector] offers: where a [Detector]'s only +route is an ignore comment, this clause is met for that [Detector] once the comment is disabled +or a [Blocking] [Defence] fails on its use, and nothing is owed for a route it does not have. +Irreducible cases MUST be directed to the [Project record], where clause 6.2 requires a justification. The [detector specification](DETECTOR-SPEC.md) permits a [Detector] to ship such a route provided @@ -368,9 +376,9 @@ mistake rather than after it, which is why they are worth stating even as SHOULD The [Toolchain] MUST run an automated check that blocks its own release. The check covers every [Identifier] printed by a [Rule] the [Toolchain] authors or bundles -as its own [Defence], whatever kind of [Detector] carries it; a [Bundled defence] whose -documentation is missing is still a [Bundled defence] for this clause, and that gap is what the -check exists to find. Where a documentation +as its own [Defence], whatever kind of [Detector] carries it, and it reaches every such [Rule] +whether or not its documentation exists: a missing page is the failure the check is there to +find. Where a documentation page covers a family of [Identifiers] by a pattern, as clause 6.5 of the [detector specification](DETECTOR-SPEC.md) allows, this audit MUST apply the pattern to every [Identifier] printed and confirm it lands on that page. A member added to the @@ -413,7 +421,8 @@ ships, every MUST in section 8 bears on its artefact grade under clause 9.1, not project-level verdict is a single grade; which part of the [Toolchain] satisfied each clause is evidence for that grade, not a second grade. -**A [Toolchain] [Conforms] with [Agent] support** if it additionally satisfies section 7. +**A [Toolchain] [Conforms] with [Agent] support** if, in addition, it does each thing section 7 +recommends. Partial [Conformance] MUST NOT be described as [Conformance]. A [Toolchain] that satisfies most of this document is in a normal and respectable condition; it is not [Conforming], and describing it as such @@ -424,8 +433,9 @@ removes the only value the word has. As a project, it follows the method with its own assembled [Toolchain], like any other project, and is graded against this document and section 7 of the method specification on that basis. As an artefact, what it ships is graded for its consumers: a [Detector] against -[the detector specification](DETECTOR-SPEC.md), a [Toolchain] against this document as it stands -when installed into a [Consuming project] with nothing else built around it. The two +[the detector specification](DETECTOR-SPEC.md), a [Toolchain] against this document, on what the +shipped [Toolchain] itself supplies once installed into a [Consuming project], counting nothing +the [Consuming project] adds around it. The two verdicts MUST be graded and declared separately, and neither implies the other. Section 8 bears on the artefact grade only; clause 5.3 is its project-level counterpart. From e6bdf402e051155cf4b7752f62ad4dd8a5ef41a1 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 12:57:49 +0100 Subject: [PATCH 09/52] Keys and routed findings: open points from runs 53 to 59 The method key lists fixing the original defect with a reproducing test among the obligations, says a bullet on the verdict resting on reproduction describes the reviewer, and says that an answer to 2a naming no step does not state the steps. The toolchain key lists the governing principle, says a bullet on how conformance is measured is neither correct nor wrong, and accepts a section 9 reason that points to the grades given. acceptance/ROUTED.md gains the findings runs 53 to 59 routed, and the runs that raised existing rows again. Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 33 +++++++++++++++++++-------------- acceptance/method/KEY.md | 9 +++++++-- acceptance/toolchain/KEY.md | 12 +++++++++--- 3 files changed, 35 insertions(+), 19 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 24beef9..901caed 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -10,24 +10,29 @@ A finding is open until a change that reworks its passage passes a cohort, when that run and moves below the rule, or until the editor closes it with a reason recorded here. A finding raised again by a later run is noted on its line, not listed twice. The status column holds `open`, or the `editor/` branch or pull request number of the change that is -reworking the passage. The rows below were raised by runs 31 to 52 under the earlier rules, on +reworking the passage. The rows raised by runs 31 to 52 were raised under the earlier rules, on untouched text of the change those runs tested, and are listed here so that they are not lost. ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | ---------------------------------------- | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | run 33 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------- | ---------------------------------------- | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | run 54 | `editor/detector-route-rule` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59 | `open` | ## Closed diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index a990c7b..86cf8ff 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -21,9 +21,13 @@ no bullet making an assertion listed below as wrong, is a correct restatement. than only mentioning it (section [2](../../SPEC.md#2-when-the-method-applies)). - Take a baseline, a suppression or an instance left unfixed to the owner, never on the practitioner's own authority (section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom)). +- Fix the original defect conventionally, with a test that reproduces it, once the defence is in + place (sections [2](../../SPEC.md#2-when-the-method-applies) and [6](../../SPEC.md#6-relationship-to-test-driven-development)). Wrong if it asserts: the practitioner fixes the reported instance first and defends afterwards; -a test may serve as the detector; an agent may baseline the remaining instances. +a test may serve as the detector; an agent may baseline the remaining instances. A bullet that +says the verdict on a remediation rests on reproduction (section [7](../../SPEC.md#7-conformance)) describes the reviewer, +not the practitioner, and is neither correct nor wrong. Wrong readings the text invites, which count against the document on one reader: none named. @@ -36,7 +40,8 @@ Wrong readings the text invites, which count against the document on one reader: order."; [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance): "The Practitioner MUST run the Rule across the entire codebase and record the total Instance count before fixing anything, and MUST then fix every Instance found rather than only the one". An answer that names the steps by their clause numbers in order states - both elements. + both elements; one that says only that section [3](../../SPEC.md#3-the-method)'s six clauses are followed in order, + naming no step, does not state element (1). - **2b.** Required: (1) the owner of the codebase, the human who holds that authority; (2) no, an agent may not do so on its own authority. Wrong if it lets the practitioner or an agent decide because the count was small, because the project recorded no owner, or at all. Key diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 2d2bea3..e0388d4 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -24,11 +24,16 @@ no bullet making an assertion listed below as wrong, is a correct restatement. same way as the defences ([6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) to [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences)). - A shipped toolchain also fails its release on undocumented bundled defences and runs its own defences on its own source ([8.1](../../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation), [8.2](../../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source)). +- Make everything the method requires of a practitioner possible without the project building + the mechanism first (section [1](../../TOOLING-SPEC.md#1-what-this-document-is-for), the governing principle). Wrong if it asserts: the toolchain must be a single product; a declaration is required to conform; the agent summary (section [7](../../TOOLING-SPEC.md#7-agent-context)) is a MUST. A bullet that calls agent support or the self-audit a further condition "where it applies", or says a declaration is not a condition, -asserts no obligation and is neither correct nor wrong. +asserts no obligation and is neither correct nor wrong. So is a bullet that states how +conformance is measured rather than what is required: the toolchain judged as one assembled +whole at the project level (sections [1](../../TOOLING-SPEC.md#1-what-this-document-is-for) and [9](../../TOOLING-SPEC.md#9-conformance)), the project-level verdict a single grade +(section [9](../../TOOLING-SPEC.md#9-conformance)), or a shipped artefact graded separately ([9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately)). Wrong readings the text invites, which count against the document on one reader: none named. @@ -69,8 +74,9 @@ Wrong readings the text invites, which count against the document on one reader: The four answers the question then asks for, each with the condition its reason must meet: - **Conformance under section [9](../../TOOLING-SPEC.md#9-conformance).** Accepted answer: does not conform. Condition: the reason - names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) or [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) among the failing clauses; other clauses named alongside are not - marked. + names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) or [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) among the failing clauses, or points to the grades the reader gave above, as "the + clauses graded No", where those grades have one of the three at No or Partial; other clauses + named alongside are not marked. - **Conformance with agent support.** Accepted answer: no. Condition: the reason says that section [7](../../TOOLING-SPEC.md#7-agent-context) is not met, or that there is no summary, or that the toolchain does not conform under section [9](../../TOOLING-SPEC.md#9-conformance) at all. From 302e496ba4b0ded8ad6363b49008b79a13dc06ff Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:03:11 +0100 Subject: [PATCH 10/52] Run 058: toolchain contradictions cohort, failed on four findings Every keyed item five of five. Four findings in text the change wrote or altered: clause 4.2's "whether or not its documentation exists" against the Defence row (four readers, two passages), clause 4.1's summary sentences against its Why, and section 9's first paragraph against clause 9.1. Nothing routed. Co-Authored-By: Claude Fable 5.1 --- .../058-toolchain-contradictions-run-2.md | 150 ++++++++++++++++++ 1 file changed, 150 insertions(+) create mode 100644 acceptance/runs/058-toolchain-contradictions-run-2.md diff --git a/acceptance/runs/058-toolchain-contradictions-run-2.md b/acceptance/runs/058-toolchain-contradictions-run-2.md new file mode 100644 index 0000000..51bf6e2 --- /dev/null +++ b/acceptance/runs/058-toolchain-contradictions-run-2.md @@ -0,0 +1,150 @@ +# Run 58: TOOLING-SPEC.md, toolchain contradictions, second run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: not yet opened at the time of the run; it is to be opened from branch + `editor/toolchain-contradictions`. The readers read commit 90a9936. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 57 in section 1 and clauses 4.1, 4.2, 4.3, 8.1, 9 and 9.1. +- **Run within the pull request**: 2 (run 57 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `43a220ebce6d3db005704ef829656671ecded332c3f3daf275b73db8abf0b825`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at 90a9936, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Partial (1, 2, 4, 5), No (3); +6.1 Yes (all five); 6.2 No (all five), each mentioning that nothing rejects a generic +justification. Every reader graded 4.3 Yes; readers 1 to 5 each noted that the fixture names no +route beyond the ignore comment. + +Question 1: correct bullets per reader 5, 5, 5, 5 and 5 (reader 1: 4.1; 4.2; 4.3; 5.1 to 5.3 +with 4.5; 6.1 to 6.3). No bullet asserts anything the key marks wrong. Bullets on neither list +are listed under the points the rules did not settle. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through", which states element (2); for 2b all five quote section 9's "The project-level +verdict is a single grade; ..."; for 2c all five quote clause 9.1's "The two verdicts MUST be +graded and declared separately, and neither implies the other." + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says there is no agent summary (all +five); S8 says Zigpipe ships no tooling (all five; reader 5 answers "No, to Zigpipe's project +grade" and places section 8 on zig-qa's artefact grade); Decl says the declaration is optional +(all five). + +### Misses + +None. No answer matches a wrong reading the key names: reader 5's 4.1 reasoning says Lintomatic +"meets most of the detector specification" but grades No. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..90a9936 adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------------------- | +| 1 | Clause 4.2, first paragraph: "Here [Defence] reaches every [Blocking] [Rule] the [Toolchain] runs, bundled or the project's own, whether or not its documentation exists" | 1, 3, 4, 5 | yes | +| 2 | Section 2 terminology table, [Defence] row: "A [Rule] with its documentation, in force and [Blocking]", set against clause 4.2's first paragraph | 1, 3, 4, 5 | no | +| 3 | Clause 4.1, first paragraph: "the six summarised here are the ones most often failed" (3, set against the Why paragraph's "six clauses of the method"); "That document's MUSTs are those of its sections 4 to 7, and every one of them applies; ... rests on them" (4) | 3, 4 | yes | +| 4 | Section 9, first paragraph: "The project-level verdict is a single grade" (1, set against the clause 9.1 heading); "every MUST in section 8 bears on its artefact grade under clause 9.1, not on this one", what "this one" refers to (2) | 1, 2 | yes | +| 5 | Clause 5.1, first paragraph: listing "every [Defence] active", set against clause 4.2 and the [Defence] row | 1 | no | +| 6 | Clause 9.1 heading: "has two levels of conformance", set against section 9's first paragraph | 1 | no | +| 7 | Clause 4.1, point 2: "whether or not a clause below names the same gap again" | 1 | yes | +| 8 | Clause 8.2, last paragraph: "has no release and no [Consuming project], so this section does not bear on its [Conformance]" | 2 | no | +| 9 | Clause 4.1, Why paragraph: "the six clauses of the method are carried out in a [Detector]" | 3 | no | +| 10 | Clause 6.2, last paragraph: "The [Toolchain] MUST reject a justification that could be pasted onto any [Exception] unchanged, by a check it documents", against the requirement to name hazard and scope | 3 | no | +| 11 | Section 2 terminology, [Bundled defence] entry: "one with its documentation" | 4 | no | +| 12 | Clause 8.1 heading: "if a bundled defence lacks resolvable documentation" | 4 | no | +| 13 | Clause 5.3, second paragraph: "its entry MAY be sourced from wherever it is enabled", what "its entry" refers to | 4 | no | +| 14 | Section 9, second paragraph: "does each thing section 7 recommends", which document's section 7 | 5 | yes | +| 15 | Clause 9.1, first paragraph: "section 7 of the method specification", set against section 9's second paragraph | 5 | yes | +| 16 | Clause 6.4: a SHOULD, and whether an unmet 6.4 affects any grade | 5 | no | + +## Findings and where they lie + +- **Finding A, passage 1, clause 4.2 first paragraph** (readers 1, 3, 4 and 5): the sentence + named was added by the change. Text the change wrote. Blocks. +- **Finding B, passage 2, section 2 [Defence] row** (readers 1, 3, 4 and 5): the row is + unchanged, but every reader naming it sets it against clause 4.2's first paragraph, which the + change altered. Blocks. +- **Finding C, passage 3, clause 4.1 first paragraph** (readers 3 and 4): the change rewrote the + sentences both readers quote. Text the change wrote. Blocks. +- **Finding D, passage 4, section 9 first paragraph** (readers 1 and 2): the change reworded the + paragraph's first sentence, which reader 2 quotes; reader 1 quotes its unchanged second + sentence, which lies in the same paragraph. Text the change altered. Blocks. + +Passages 5 to 16 are each named by one reader: recorded, not findings. Nothing is routed to +`acceptance/ROUTED.md` from this run. + +## Points the rules did not settle + +1. **Question 1 bullets on neither of the key's lists**: reader 1's first and reader 3's first + (the toolchain is judged as one assembled whole), reader 4's first ("must provide every + mechanism the method needs", the governing principle of section 1), reader 2's sixth and + reader 3's sixth (a shipped artefact graded separately, clause 9.1). Marked by the rule for a + bullet on neither list: not counted either way. Every reader has five correct bullets + without them, so no mark turns on it. For the key to settle. +2. **Reader 1's passage 5** gives clause 5.1 with the three-word quotation "every [Defence] + active"; **reader 5's passage 16** gives clause 6.4 with a description ("It is a SHOULD") + that identifies its heading rather than a quoted sentence. Whether each names a passage was + marked by the open-point rule: counted. Each is named by one reader only, so neither makes a + finding either way. +3. **Reader 1's passage 4** quotes section 9's second sentence, which the change did not reword, + in a paragraph whose first sentence it did. The rule places a finding by paragraph, so the + paragraph is the change's; had it been open, the open-point rule would have treated it as the + change's own text in any case. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit c31d49a, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: four (findings A to + D). Not met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A to D alone; every keyed item passed. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From 352cfcc366e4755481606beb59d275145296bc52 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:03:11 +0100 Subject: [PATCH 11/52] Toolchain: a blocking rule without documentation is not yet a defence After run 58: clauses 4.2 and 8.1 no longer say Defence reaches a rule whether or not its documentation exists; each says that a blocking rule whose documentation is missing is not yet a defence and is the failure the clause names. Clause 4.1 says its six sentences are a summary and not the test, and its Why names the method's clauses 3.1 to 3.6; its point 2 names clause 4.2 as a later clause that may name the same gap. Section 9 says every project has one project grade and that section 8 bears on a shipping project's separate artefact grade; clause 9.1 names the two levels; the agent-support sentence says section 7 of this document. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 10 +++++++- TOOLING-SPEC.md | 47 +++++++++++++++++++++---------------- acceptance/toolchain/KEY.md | 4 ++-- 3 files changed, 38 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ad5f0e6..c619684 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -119,7 +119,15 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be fixture does not describe; section [1](TOOLING-SPEC.md#1-what-this-document-is-for) says the document defers to the detector specification only in clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification); clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) says what the artefact grade counts; and section [9](TOOLING-SPEC.md#9-conformance)'s agent-support sentence says section [7](TOOLING-SPEC.md#7-agent-context) recommends, after two readers set "additionally satisfies" against - that section's SHOULDs. Accepted by run NNN. + that section's SHOULDs. [Run 58](acceptance/runs/058-toolchain-contradictions-run-2.md) failed + on four findings in the revised text, every keyed item five of five: four readers set clause + [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access)'s "whether or not its documentation exists" against the Defence row again, so clauses + [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) and [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) now say that a blocking rule whose documentation is missing is not yet a + defence and is the failure each names; clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) says its six sentences are a summary and + not the test, and its Why names the method's clauses [3.1](SPEC.md#31-attribute-the-defect-to-a-class) to [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) rather than "six clauses"; + section [9](TOOLING-SPEC.md#9-conformance) says every project has one project grade and that section [8](TOOLING-SPEC.md#8-self-audit) bears on a shipping + project's separate artefact grade, clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) names the two levels, and the agent-support + sentence says which document's section [7](TOOLING-SPEC.md#7-agent-context) it means. Accepted by run NNN. ### 0.2.0, 2026-09-08 diff --git a/TOOLING-SPEC.md b/TOOLING-SPEC.md index 5ad5f51..fff1fd8 100644 --- a/TOOLING-SPEC.md +++ b/TOOLING-SPEC.md @@ -111,15 +111,16 @@ any row and the failure mode is visible too: either half alone leaves the [Pract A [Toolchain] MUST NOT route a [Defence] through a [Detector] that does not [Conform] to [the detector specification](DETECTOR-SPEC.md). That document's MUSTs are those of -its sections 4 to 7, and every one of them applies; the six summarised here are the ones most -often failed, and a verdict on this clause taken from this document alone rests on them. The +its sections 4 to 7, and this clause holds the [Detector] to every one of them. The sentences +that follow summarise the ones most often failed, so that a reader with this document alone can +see what the clause mostly turns on; they are a summary, not the test. The [Detector] hosts bespoke [Rules]. It runs one [Rule] on its own against supplied code. It prints a stable [Identifier] with every finding. It runs locally, down to one file, with the result in the command's own output and nothing reportable only through a hosted service. It resolves every [Identifier] a [Bundled rule] can print to documentation shipped with it, offline. It makes any inline [Suppression] route detectable or disableable. A [Detector] that fails any -one of that document's MUSTs, after wrapping, fails this clause. Three points govern how that is -met. +one of that document's MUSTs, summarised here or not, fails this clause after wrapping. Three +points govern how that is met. 1. **Wrapping is permitted.** The [Toolchain] MAY supply a mechanism the [Detector] lacks by wrapping it, with a [Harness] script or a @@ -138,8 +139,9 @@ met. Behind the table is one question. With the wrapping in place, does the [Detector] meet every MUST of sections 4 to 7 of the [detector specification](DETECTOR-SPEC.md), exercised as its clause 8.1 describes? If yes, this clause holds. If no, it fails, however - much else the [Detector] does well, and whether or not a clause below names the same gap - again. Wrapping can add a mechanism; it cannot excuse a gap. What this clause judges is the + much else the [Detector] does well, and whether or not a later clause of this document, such + as clause 4.2 on resolution, names the same gap again. Wrapping can add a mechanism; it + cannot excuse a gap. What this clause judges is the wrapped pair; the [Detector]'s own grade under that document, as an artefact on its own, is a separate verdict that the wrapping does not change. Satisfying most of that document is not [Conformance] to it, any more than satisfying most of this one is. @@ -149,7 +151,7 @@ met. It MAY still run as one of the [Toolchain]'s checks, as a formatter or a [Runner] does. -**Why**: the six clauses of the method are carried out in a [Detector], and everything this +**Why**: the method's clauses 3.1 to 3.6 are carried out in a [Detector], and everything this document adds presumes those clauses can be followed there. A [Toolchain] that lists, records and resolves beautifully around a [Detector] in which no [Rule] can be written or proven has governed nothing. The wrapping is permitted because a [Practitioner] cannot tell where a @@ -163,9 +165,10 @@ This clause widens the [detector specification](DETECTOR-SPEC.md)'s reach: that [Bundled rules] only, and this one covers every [Identifier] the project can see, the project's own [Rules] included. The [Toolchain] MUST resolve every [Identifier] its [Defences] print to [Remediation docs] on disk. One [Identifier] that does not resolve fails this clause, -however many others do. Here [Defence] reaches every [Blocking] [Rule] the [Toolchain] runs, -bundled or the project's own, whether or not its documentation exists: a [Rule] whose -documentation is missing is the failure this clause names, not a [Rule] outside it. The mechanism +however many others do. A [Defence] is a [Rule] with its documentation, as section 2 says, so a +[Blocking] [Rule] the [Toolchain] runs whose documentation is missing is not yet a [Defence]: +that [Rule] is the failure this clause names, not a [Rule] outside its reach, and the clause +reaches every [Blocking] [Rule] the [Toolchain] runs, bundled or the project's own. The mechanism is keyed on the [Identifier] exactly as printed, and it covers the project's own [Rules] and every [Bundled defence]. Those [Remediation docs] MUST ship @@ -376,9 +379,9 @@ mistake rather than after it, which is why they are worth stating even as SHOULD The [Toolchain] MUST run an automated check that blocks its own release. The check covers every [Identifier] printed by a [Rule] the [Toolchain] authors or bundles -as its own [Defence], whatever kind of [Detector] carries it, and it reaches every such [Rule] -whether or not its documentation exists: a missing page is the failure the check is there to -find. Where a documentation +as its own, whatever kind of [Detector] carries it. A [Bundled defence] is a [Rule] with its +documentation, so a [Rule] shipped that way whose page is missing is not yet one; it is what +this check exists to find, and the check reaches every such [Rule]. Where a documentation page covers a family of [Identifiers] by a pattern, as clause 6.5 of the [detector specification](DETECTOR-SPEC.md) allows, this audit MUST apply the pattern to every [Identifier] printed and confirm it lands on that page. A member added to the @@ -416,13 +419,15 @@ own [Defences] already run on its own source, because that is the only source th ## 9. Conformance **A project's [Toolchain] [Conforms]** if every MUST in sections 4 to 6 holds across the -assembled parts, wherever each part came from; where the [Toolchain] is one the project -ships, every MUST in section 8 bears on its artefact grade under clause 9.1, not on this one. The -project-level verdict is a single grade; which part of the -[Toolchain] satisfied each clause is evidence for that grade, not a second grade. +assembled parts, wherever each part came from; where the project also ships a [Detector] or a +[Toolchain], every MUST in section 8 bears on the artefact grade of clause 9.1 and not on this +project grade. The project-level verdict is a single grade; which part of the +[Toolchain] satisfied each clause is evidence for that grade, not a second grade. The artefact +a project ships is graded on its own, and that grade is the second level clause 9.1 names; a +project that ships nothing has only the one. **A [Toolchain] [Conforms] with [Agent] support** if, in addition, it does each thing section 7 -recommends. +of this document recommends. Partial [Conformance] MUST NOT be described as [Conformance]. A [Toolchain] that satisfies most of this document is in a normal and respectable condition; it is not [Conforming], and describing it as such @@ -436,8 +441,10 @@ basis. As an artefact, what it ships is graded for its consumers: a [Detector] a [the detector specification](DETECTOR-SPEC.md), a [Toolchain] against this document, on what the shipped [Toolchain] itself supplies once installed into a [Consuming project], counting nothing the [Consuming project] adds around it. The two -verdicts MUST be graded and declared separately, and neither implies the other. Section 8 bears on -the artefact grade only; clause 5.3 is its project-level counterpart. +verdicts MUST be graded and declared separately, and neither implies the other. The two levels +are the project grade of section 9, which every project has, and the artefact grade, which only +a project that ships has. Section 8 bears on the artefact grade only; clause 5.3 is its +project-level counterpart. **Why**: the two questions have different readers. A contributor to the artefact wants to know whether the project practises what it ships; a [Consuming project] wants to know what it diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index f3fba7b..29e1833 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -93,8 +93,8 @@ Wrong readings the text invites, which count against the document on one reader: - [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) graded Yes with reasoning that says a justification is required or an empty one rejected and does not mention a generic justification, which has not read the sentence on one. - Section [8](../../TOOLING-SPEC.md#8-self-audit) said to apply to Zigpipe with reasoning that says its tooling includes a shipped - package, which has read "where the Toolchain is one the project ships" (section [9](../../TOOLING-SPEC.md#9-conformance)) as - reaching the consuming project. + package, which has read "where the project also ships a Detector or a Toolchain" (section [9](../../TOOLING-SPEC.md#9-conformance)) + as reaching the consuming project. - [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) graded Yes with reasoning that says Lintomatic mostly or largely conforms and quotes [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), which has read it as admitting a proportion. - The declaration said to affect the verdict, with reasoning that Zigpipe cannot be graded without From 4081aa45e771f14fb832e708ba612c88f5008d8e Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:16:08 +0100 Subject: [PATCH 12/52] Toolchain key and routed findings: open points from runs 61 and 62 The toolchain key counts a bullet on not describing partial conformance as conformance, and says a bullet calling section 7 recommended or a SHOULD asserts no obligation. acceptance/ROUTED.md gains clause 3.4's "tremendous" sentence from run 61 and notes that run on the two rows it raised again. Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 5 +++-- acceptance/toolchain/KEY.md | 6 ++++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 901caed..76acd8f 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -31,8 +31,9 @@ untouched text of the change those runs tested, and are listed here so that they | `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56 | `open` | | `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | | `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59, 61 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59, 61 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | run 61 | `open` | ## Closed diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index e0388d4..aadab7a 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -26,11 +26,13 @@ no bullet making an assertion listed below as wrong, is a correct restatement. defences on its own source ([8.1](../../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation), [8.2](../../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source)). - Make everything the method requires of a practitioner possible without the project building the mechanism first (section [1](../../TOOLING-SPEC.md#1-what-this-document-is-for), the governing principle). +- Not describe partial conformance as conformance (section [9](../../TOOLING-SPEC.md#9-conformance)). Wrong if it asserts: the toolchain must be a single product; a declaration is required to conform; the agent summary (section [7](../../TOOLING-SPEC.md#7-agent-context)) is a MUST. A bullet that calls agent support or the -self-audit a further condition "where it applies", or says a declaration is not a condition, -asserts no obligation and is neither correct nor wrong. So is a bullet that states how +self-audit a further condition "where it applies", says that section [7](../../TOOLING-SPEC.md#7-agent-context) is recommended or a +SHOULD, or says a declaration is not a condition, asserts no obligation and is neither correct +nor wrong. So is a bullet that states how conformance is measured rather than what is required: the toolchain judged as one assembled whole at the project level (sections [1](../../TOOLING-SPEC.md#1-what-this-document-is-for) and [9](../../TOOLING-SPEC.md#9-conformance)), the project-level verdict a single grade (section [9](../../TOOLING-SPEC.md#9-conformance)), or a shipped artefact graded separately ([9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately)). From 37191a4d371308ab5b704c45213741c9cf22275e Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:17:33 +0100 Subject: [PATCH 13/52] Run 062: toolchain contradictions cohort, failed on clause 4.2 again Every keyed item five of five. Two findings in text the change wrote or altered: clause 4.2's "not yet a Defence" sentence (four readers), and the Defence row set against it. Nothing routed. Co-Authored-By: Claude Fable 5.1 --- .../062-toolchain-contradictions-run-3.md | 149 ++++++++++++++++++ 1 file changed, 149 insertions(+) create mode 100644 acceptance/runs/062-toolchain-contradictions-run-3.md diff --git a/acceptance/runs/062-toolchain-contradictions-run-3.md b/acceptance/runs/062-toolchain-contradictions-run-3.md new file mode 100644 index 0000000..8564b18 --- /dev/null +++ b/acceptance/runs/062-toolchain-contradictions-run-3.md @@ -0,0 +1,149 @@ +# Run 62: TOOLING-SPEC.md, toolchain contradictions, third run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: not yet opened at the time of the run; it is to be opened from branch + `editor/toolchain-contradictions`. The readers read commit 352cfcc. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 58 in clauses 4.1, 4.2, 8.1, 9 and 9.1. +- **Run within the pull request**: 3 (runs 57 and 58 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `7ceebd159dd90a77d1844c6f31d01d93fce2e3e6e38bf33e68f53044056c74aa`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at 352cfcc, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Partial (1, 4), No (2), Yes +(3, 5); 6.1 Yes (1, 4, 5), Partial (2, 3), each Partial mentioning that no location is given for +calibrations or conventions; 6.2 No (all five), each mentioning that nothing rejects a generic +justification. Every reader graded 4.3 Yes, noting that the fixture names no route beyond the +ignore comment. Reader 2 states that it assumed the two undocumented rules are Blocking; the +key reads a 4.2 No on that condition as a No, and the reader's other grades do not turn on it. + +Question 1: correct bullets per reader 6, 5, 4, 5 and 4 (reader 1: section 1's governing +principle; 4.1; 4.2; 4.3; 4.4 with 4.5; 5.1 to 5.2 with 6.2 and 6.3. Reader 3: 4.1; 4.2; 4.3 with +4.5; 5.1, 5.2, 6.2 and 6.3. Reader 5: 4.1; 4.2; 4.3 with 4.5; 5.1 to 5.3 with 6.1 and 6.2). No +bullet asserts anything the key marks wrong. The first bullet of readers 2 to 5 (the toolchain +judged as one assembled whole) states how conformance is measured, which the key counts neither +way. Bullets on neither list are listed under the points the rules did not settle. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through", which states element (2); for 2b all five quote section 9's "The project-level +verdict is a single grade; ..."; for 2c all five quote clause 9.1's "The two verdicts MUST be +graded and declared separately, and neither implies the other." Reader 1 states element (2) of +2c only in that quotation, which the Marking section counts as stated. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says section 7 is not met or there is no +agent summary (all five); S8 says Zigpipe ships no tooling (all five; readers 2, 3, 4 and 5 also +place section 8 on zig-qa's own artefact grade, reader 5 answering "No, not to Zigpipe's project +grade"); Decl says the declaration is optional, or that the verdict rests on the exercise (all +five). + +### Misses + +None. No answer matches a wrong reading the key names: reader 3's 4.1 reasoning says Lintomatic +"meets most of the detector specification" but grades No. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..352cfcc adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------------------- | +| 1 | Clause 4.2, first paragraph: "a [Blocking] [Rule] the [Toolchain] runs whose documentation is missing is not yet a \[Defence\]: that [Rule] is the failure this clause names, not a [Rule] outside its reach" (2, 3, 4, 5; 5 adds "and the clause reaches every [Blocking] [Rule] the [Toolchain] runs"); whether the clause's scope is set by "its [Defences]" or by every Blocking rule | 2, 3, 4, 5 | yes | +| 2 | Section 2 terminology table, [Defence] row: "section 2 defines a defence as a rule with its documentation" (3), "section 2 defines a Defence as having documentation" (5), each set against clause 4.2's first paragraph | 3, 5 | no | +| 3 | Clause 4.3, first paragraph: "Irreducible cases MUST be directed to the [Project record], where clause 6.2 requires a justification", "irreducible" not defined | 1 | yes | +| 4 | Section 2 terminology, [Project record] entry: "[Calibrations], [Exceptions] and conventions", set against clause 6.1 | 2 | no | +| 5 | Clause 6.1, first paragraph: "The [Toolchain] MUST define where the [Project record] lives and MUST load it", whether one location for all three kinds or only for exceptions | 2 | no | +| 6 | Clause 5.1, first paragraph: "every [Defence]", set against clause 4.2's first paragraph | 3 | no | +| 7 | Clause 6.2 heading: "reject a generic one" | 4 | no | +| 8 | Clause 6.2, last paragraph: "name the [Hazard] being accepted and the scope", the documented check of phrases, "cannot verify truth"; whether the check is the phrase list, a check that hazard and scope are named, or both | 4 | no | +| 9 | Clause 4.4, first paragraph: "so that clause 4.3 of the [detector specification] holds through the wrapping", set against section 1's "How to read" paragraph | 5 | no | +| 10 | Section 1, "How to read this document" paragraph: "self-contained for grading a [Toolchain], apart from clause 4.1", set against clause 4.4 | 5 | yes | + +Notes, not passages: reader 1 (question 3 preamble) and reader 4 (4.1 reasoning) remark that +the detector specification was not supplied; neither is under question 4. + +## Findings and where they lie + +- **Finding A, passage 1, clause 4.2 first paragraph** (readers 2, 3, 4 and 5): the sentences + named were added by the change. Text the change wrote. Blocks. +- **Finding B, passage 2, section 2 [Defence] row** (readers 3 and 5): the row is unchanged, but + both readers naming it set it against clause 4.2's first paragraph, which the change altered. + Blocks. + +Passages 3 to 10 are each named by one reader: recorded, not findings. Nothing is routed to +`acceptance/ROUTED.md` from this run. + +## Points the rules did not settle + +1. **Question 1 bullets on neither of the key's lists**: reader 3's sixth ("Agent-context + summary and automatic delivery are recommended (SHOULD)", with "a shipped artefact has a + separate grade with a self-audit") and reader 5's sixth ("partial conformance must not be + called conformance", with the single grade and the separate artefact grade). The SHOULD half + of reader 3's bullet and the section 9 partial-conformance MUST in reader 5's are on neither + list; the rest of each states how conformance is measured, which the key counts neither way. + Marked by the rule for a bullet on neither list: not counted either way. Readers 3 and 5 each + have four correct bullets without them, so no mark turns on it. For the key to settle. +2. **Passage 2**: readers 3 and 5 identify the [Defence] row by a section number and a + description, inside their remark on clause 4.2, not as a passage named on its own. Whether + that names the row was marked by the open-point rule: counted. Finding B rests on it; finding + A, and so the verdict, does not. +3. **Reader 4's 6.2 remark** names the heading and the last paragraph, each a paragraph of its + own, and **reader 2's 6.1 remark** names clause 6.1 and the [Project record] entry; each is + recorded as two passages. No other reader names any of them, so no finding turns on it. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit e6bdf40, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: two (findings A + and B). Not met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A and B alone; every keyed item passed. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From dfd4b4d8d45d3a52aea083a02ddfc19f2242efe7 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:17:33 +0100 Subject: [PATCH 14/52] Toolchain 4.2: a defence or a blocking rule it routes After run 62: clause 4.2's heading and obligation sentence name both what the clause reaches, a defence and any other blocking rule the toolchain routes, and the body says the blocking rules are named because a defence is a rule with its documentation, so one without is the failure the clause exists to catch. The heading's anchor follows in the keys, the question set, the changelog, the routed list and the tools pages. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 19 ++++++++++------ TOOLING-SPEC.md | 12 +++++----- acceptance/ROUTED.md | 38 +++++++++++++++---------------- acceptance/toolchain/KEY.md | 6 ++--- acceptance/toolchain/QUESTIONS.md | 2 +- tools/php-qa-ci.md | 4 ++-- tools/ts-qa-ci.md | 6 ++--- 7 files changed, 46 insertions(+), 41 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c619684..e2ea1f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -101,7 +101,7 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be the document pulling against itself, each listed in `acceptance/ROUTED.md`. Section [9](TOOLING-SPEC.md#9-conformance)'s conformance sentence now says that section [8](TOOLING-SPEC.md#8-self-audit) bears on a shipping project's artefact grade under clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately), not on its project grade, which is what clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) already said and the - answer key assumed. Clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) says that one identifier that does not resolve fails it and + answer key assumed. Clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) says that one identifier that does not resolve fails it and that a blocking rule whose documentation is missing is graded as a defence with a gap, and clause [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) says the same of a bundled defence, after readers took section [2](TOOLING-SPEC.md#2-terminology)'s Defence row to put such a rule outside both clauses. Clause [9.2](TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance)'s duty to record a known gap in the @@ -112,7 +112,7 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be applies and the six it lists are a summary, and its point 2 says the wrapped pair is what the clause judges whilst the detector's own grade is a separate verdict. Prompted by runs 33 to 52 and 55 on the defined-terms change, in which two or more readers quoted each passage. - [Run 57](acceptance/runs/057-toolchain-contradictions-run-1.md) failed on the first wording: every reader set clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access)'s + [Run 57](acceptance/runs/057-toolchain-contradictions-run-1.md) failed on the first wording: every reader set clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)'s "a defence with a gap" against the Defence row, so the sentence now says Defence there reaches every blocking rule whether or not its documentation exists; clause [4.3](TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) says the routes are the ones each detector offers, after two readers graded it Partial for routes the @@ -121,13 +121,18 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be sentence says section [7](TOOLING-SPEC.md#7-agent-context) recommends, after two readers set "additionally satisfies" against that section's SHOULDs. [Run 58](acceptance/runs/058-toolchain-contradictions-run-2.md) failed on four findings in the revised text, every keyed item five of five: four readers set clause - [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access)'s "whether or not its documentation exists" against the Defence row again, so clauses - [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) and [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) now say that a blocking rule whose documentation is missing is not yet a + [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)'s "whether or not its documentation exists" against the Defence row again, so clauses + [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) and [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) now say that a blocking rule whose documentation is missing is not yet a defence and is the failure each names; clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) says its six sentences are a summary and not the test, and its Why names the method's clauses [3.1](SPEC.md#31-attribute-the-defect-to-a-class) to [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) rather than "six clauses"; section [9](TOOLING-SPEC.md#9-conformance) says every project has one project grade and that section [8](TOOLING-SPEC.md#8-self-audit) bears on a shipping project's separate artefact grade, clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) names the two levels, and the agent-support - sentence says which document's section [7](TOOLING-SPEC.md#7-agent-context) it means. Accepted by run NNN. + sentence says which document's section [7](TOOLING-SPEC.md#7-agent-context) it means. + [Run 62](acceptance/runs/062-toolchain-contradictions-run-3.md) failed on the same clause + [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) sentence, every keyed item five of five: four readers could not settle whether the + clause covers its defences or every blocking rule, so its heading and its obligation sentence + now name both, a defence or a blocking rule it routes, and the body says why the blocking + rules are named. Accepted by run NNN. ### 0.2.0, 2026-09-08 @@ -154,10 +159,10 @@ since the clause says the listing MUST provide the route and [4.1](TOOLING-SPEC. again by a later clause; three readers of run 8 graded it so from the text. Findings applied across the runs: an eleven-term gloss and a reading note in sections [1](TOOLING-SPEC.md#1-what-this-document-is-for) and [2](TOOLING-SPEC.md#2-terminology); long sentences split; [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) opens by listing the detector specification's MUSTs and its point 2 -leads with a three-case table of wrapped pairs, then the one question behind it; [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) says it +leads with a three-case table of wrapped pairs, then the one question behind it; [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) says it widens the detector document's reach to the project's own rules; [4.3](TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) loses its double negative; [5.2](TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) gives the derived-listing test; [6.2](TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one)'s heading carries the generic-justification rejection; -[8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) is consistent with [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) on third-party catalogues. Runs 1 to 7 also asked readers what would +[8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) is consistent with [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) on third-party catalogues. Runs 1 to 7 also asked readers what would make a clause clearer and for a clarity rating; both questions were withdrawn before run 8, as `ACCEPTANCE.md` records. diff --git a/TOOLING-SPEC.md b/TOOLING-SPEC.md index fff1fd8..458367e 100644 --- a/TOOLING-SPEC.md +++ b/TOOLING-SPEC.md @@ -159,16 +159,16 @@ mechanism lives, and the method does not care. It is bounded because a mechanism does not offer and the [Toolchain] does not add is a gap the project writes down under method clause 3.2. A gap recorded is not a gap closed. -### 4.2 The toolchain MUST resolve every identifier a defence it routes can print, from the installed copy, without network access +### 4.2 The toolchain MUST resolve every identifier a defence or a blocking rule it routes can print, from the installed copy, without network access This clause widens the [detector specification](DETECTOR-SPEC.md)'s reach: that document resolves [Bundled rules] only, and this one covers every [Identifier] the project can see, the project's own [Rules] included. The [Toolchain] MUST resolve every [Identifier] its [Defences] -print to [Remediation docs] on disk. One [Identifier] that does not resolve fails this clause, -however many others do. A [Defence] is a [Rule] with its documentation, as section 2 says, so a -[Blocking] [Rule] the [Toolchain] runs whose documentation is missing is not yet a [Defence]: -that [Rule] is the failure this clause names, not a [Rule] outside its reach, and the clause -reaches every [Blocking] [Rule] the [Toolchain] runs, bundled or the project's own. The mechanism +print, and every [Identifier] its other [Blocking] [Rules] print, to [Remediation docs] on disk. +One [Identifier] that does not resolve fails this clause, however many others do. The other +[Blocking] [Rules] are named because a [Defence] is a [Rule] with its documentation, as section +2 says: a [Blocking] [Rule] whose documentation is missing is not yet a [Defence], and it is +exactly the failure this clause exists to catch. The mechanism is keyed on the [Identifier] exactly as printed, and it covers the project's own [Rules] and every [Bundled defence]. Those [Remediation docs] MUST ship diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 76acd8f..65acc8a 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -15,25 +15,25 @@ untouched text of the change those runs tested, and are listed here so that they ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------- | ---------------------------------------- | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | run 54 | `editor/detector-route-rule` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59, 61 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59, 61 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | run 61 | `open` | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------- | ---------------------------------------- | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | run 54 | `editor/detector-route-rule` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59, 61 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59, 61 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | run 61 | `open` | ## Closed diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index b78577b..9fe09d6 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -13,7 +13,7 @@ no bullet making an assertion listed below as wrong, is a correct restatement. - Route defences only through detectors that conform to the detector specification, wrapping permitted ([4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification)). - Resolve every identifier a routed defence can print, including the project's own, offline, - to documentation stating the correct construction ([4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access)). + to documentation stating the correct construction ([4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)). - Forbid, by disabling or by a blocking defence, every suppression route that bypasses the project record ([4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)). - The toolchain's own entry point meets the detector reporting clauses and prints identifiers @@ -62,7 +62,7 @@ Wrong readings the text invites, which count against the document on one reader: | Clause | Accepted grades | Condition on the reasoning | What the fixture supports | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | None | Lintomatic fails detector [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) on two undocumented bundled rules and nothing in the wrapping closes that gap; a reader who notes that the detector specification was not supplied and grades from this document's summary of it is correct | -| [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | None. A No given on the condition that the two bundled rules are blocking is a No: their firing stops `zig-qa run`, so they are | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)` | +| [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) | No | None. A No given on the condition that the two bundled rules are blocking is a No: their firing stops `zig-qa run`, so they are | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)` | | [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) | Yes | None | Ignore comments disabled project-wide; exceptions only through the record the toolchain reads | | [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | None | `zig-qa run` is local, prints Lintomatic's output unchanged, one line per finding | | [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | None | Lintomatic, then the tests, stopping on a Lintomatic failure | @@ -76,7 +76,7 @@ Wrong readings the text invites, which count against the document on one reader: The four answers the question then asks for, each with the condition its reason must meet: - **Conformance under section [9](../../TOOLING-SPEC.md#9-conformance).** Accepted answer: does not conform. Condition: the reason - names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) or [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) among the failing clauses, or points to the grades the reader gave above, as "the + names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) or [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) among the failing clauses, or points to the grades the reader gave above, as "the clauses graded No", where those grades have one of the three at No or Partial; other clauses named alongside are not marked. - **Conformance with agent support.** Accepted answer: no. Condition: the reason says that diff --git a/acceptance/toolchain/QUESTIONS.md b/acceptance/toolchain/QUESTIONS.md index 3b58dba..21a8d5e 100644 --- a/acceptance/toolchain/QUESTIONS.md +++ b/acceptance/toolchain/QUESTIONS.md @@ -24,7 +24,7 @@ the document you are relying on. ## 3. Application Grade the Zigpipe project's toolchain, as described in the fixture, against every MUST clause in -sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record): [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own), [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure), [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration), [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones), [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences). For each give Yes, +sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record): [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access), [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own), [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure), [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration), [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones), [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences). For each give Yes, Partial or No and one line of reasoning. Then state whether the toolchain conforms under section [9](../../TOOLING-SPEC.md#9-conformance) and why, whether it conforms with agent support and why, and whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies to it and why. Finally, state whether the absence of a declaration affects the verdict, and why. diff --git a/tools/php-qa-ci.md b/tools/php-qa-ci.md index 1337e56..feb7567 100644 --- a/tools/php-qa-ci.md +++ b/tools/php-qa-ci.md @@ -22,7 +22,7 @@ Every mechanism below was checked by running it in a checkout: `bin/qa`, `bin/qa For its own PHPStan bundle, every detector clause holds. A project's rules go under `rules:` in its `qaConfig/phpstan.neon` (detector [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it)). `bin/phpstan-rule phpqaci.nullCoalescingFalse src/ZzProbe.php` on a probe containing `?? false` printed `FIRED (1)` with the location and exited 1, and the same command for `phpqaci.nestedTernary` printed `did not fire` (detector [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code)). PHPStan prints the `phpqaci.*` identifier under every finding, and `RequireRuleIdentifierConstantRule`, on by default, rejects a magic-string identifier (detector [4.3](../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding), detector [4.4](../DETECTOR-SPEC.md#44-the-detector-should-enforce-43-with-a-rule-of-its-own)). `bin/qa -t phpstan -p ` runs one lane over one file locally with the result in the terminal, and the CI script runs the same `bin/qa` (detector [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to detector [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally)). `bin/rule-doc phpqaci.nestedTernary` resolves the identifier as printed to its rule, bundle, source and shipped page, offline, and the page carries a `The correct construction` section (detector [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) to detector [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) for the bundle). `RuleDocumentationTest` fails the build when a declared identifier resolves to no page, or to a page whose construction section is missing or only restates the summary (detector [6.4](../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation)). Inline `@phpstan-ignore` in every form is a finding of `ForbidInlinePhpstanIgnoreRule` (detector [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)), and behind it the default tier turns on PHPStan's `reportIgnoresWithoutComments`: a probe ignore with no reason printed both `phpqaci.inlinePhpstanIgnore` and `ignore.noComment (non-ignorable)` (detector [7.2](../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason)). -At toolchain level, every lane prints a stable identifier when it fails: a run of `bin/qa -t branchNamePolicy` on a `plan/*` branch ended with `🪪 phpqaci.branchNamePolicy (vendor/bin/rule-doc phpqaci.branchNamePolicy)`, and `EveryLaneNamesItselfTest` fails the build on a lane without one. `bin/rule-doc phpqaci.packageType` resolves a lane identifier to its page as it does a rule's, and `bin/rules .` prints no `no documentation page` row: all 69 indexed identifiers (`bin/rule-doc --list`) resolve to a page ([4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access)). `bin/rules .` lists every rule in the resolved neon chain, including the rules `phpstan/extension-installer` delivers, with the package each came from, and every lane with its identifier and documentation route; phase runners alone carry none ([5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) to [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones)). The record is the `ignoreErrors` block PHPStan itself loads ([6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself)); the `pij` lane reads it through every `includes:` file, and a probe baseline included from `qaConfig/phpstan.neon` with an uncommented entry failed it, naming the included file and line ([6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)). A `phparkitect-baseline.json` is never read: on a probe project with one violation, PHPArkitect with a generated baseline printed `No violations detected`, and with `--skip-baseline`, which the lane always passes, reported the violation; the lane names the file when one is present ([4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)). The same listing enumerates the record beside the defences, together with the deferred-defect and no-pattern record `qaConfig/defect-record.neon` (`defectRecord` in `--json`), which the same lane fails on when an entry does not read in full ([6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences)). `docs/defence-before-fix-defaults.md` states the toolchain's default for the owner, the project record, the sweep scope, fixtures and the method's three calibrations, linked from the agent block ([6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)). The entry point runs coding standards, linting and static analysis before tests and stops on a failure ([4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure)). `bin/rules --write-agent-summary` writes one line per active defence, identifier and page, into a marked region of the project's `CLAUDE.md`, refreshed on every install and update ([7.1](../TOOLING-SPEC.md#71-the-toolchain-should-generate-a-summary-of-the-active-defences-suitable-for-an-agents-context), [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically)). The release guard covers rules and lanes and requires a page stating a correct construction ([8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation)), and the self-check runs both bundled rule sets and the PHPArkitect tier on the package's own source ([8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source)). The manifest declares method 1.1.0 and toolchain 0.2.0 at both levels, with the gaps below recorded against their clauses, and `DefenceBeforeFixDeclarationTest` holds each recorded gap to a decision and a probe that fails when the gap closes ([9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance)). +At toolchain level, every lane prints a stable identifier when it fails: a run of `bin/qa -t branchNamePolicy` on a `plan/*` branch ended with `🪪 phpqaci.branchNamePolicy (vendor/bin/rule-doc phpqaci.branchNamePolicy)`, and `EveryLaneNamesItselfTest` fails the build on a lane without one. `bin/rule-doc phpqaci.packageType` resolves a lane identifier to its page as it does a rule's, and `bin/rules .` prints no `no documentation page` row: all 69 indexed identifiers (`bin/rule-doc --list`) resolve to a page ([4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)). `bin/rules .` lists every rule in the resolved neon chain, including the rules `phpstan/extension-installer` delivers, with the package each came from, and every lane with its identifier and documentation route; phase runners alone carry none ([5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) to [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones)). The record is the `ignoreErrors` block PHPStan itself loads ([6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself)); the `pij` lane reads it through every `includes:` file, and a probe baseline included from `qaConfig/phpstan.neon` with an uncommented entry failed it, naming the included file and line ([6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)). A `phparkitect-baseline.json` is never read: on a probe project with one violation, PHPArkitect with a generated baseline printed `No violations detected`, and with `--skip-baseline`, which the lane always passes, reported the violation; the lane names the file when one is present ([4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)). The same listing enumerates the record beside the defences, together with the deferred-defect and no-pattern record `qaConfig/defect-record.neon` (`defectRecord` in `--json`), which the same lane fails on when an entry does not read in full ([6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences)). `docs/defence-before-fix-defaults.md` states the toolchain's default for the owner, the project record, the sweep scope, fixtures and the method's three calibrations, linked from the agent block ([6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)). The entry point runs coding standards, linting and static analysis before tests and stops on a failure ([4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure)). `bin/rules --write-agent-summary` writes one line per active defence, identifier and page, into a marked region of the project's `CLAUDE.md`, refreshed on every install and update ([7.1](../TOOLING-SPEC.md#71-the-toolchain-should-generate-a-summary-of-the-active-defences-suitable-for-an-agents-context), [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically)). The release guard covers rules and lanes and requires a page stating a correct construction ([8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation)), and the self-check runs both bundled rule sets and the PHPArkitect tier on the package's own source ([8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source)). The manifest declares method 1.1.0 and toolchain 0.2.0 at both levels, with the gaps below recorded against their clauses, and `DefenceBeforeFixDeclarationTest` holds each recorded gap to a decision and a probe that fails when the gap closes ([9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance)). ## How it is not conformant @@ -64,7 +64,7 @@ Detector rows grade PHPStan as wrapped, with PHPArkitect as wrapped noted where | Detector | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | Inline ignores detectable by a bundled rule; PHPArkitect's baseline never read (`--skip-baseline` always passed) | | Detector | [7.2](../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) | Yes | `reportIgnoresWithoutComments` on in the default tier; a probe printed `ignore.noComment (non-ignorable)` | | Toolchain | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | PHPArkitect's tier rules carry no identifier, single-file run or resolver; PHPStan's native catalogue is online only; both declared | -| Toolchain | [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | Yes | `bin/rules .`: no `no documentation page` row; `bin/rule-doc phpqaci.packageType` resolves to its page | +| Toolchain | [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) | Yes | `bin/rules .`: no `no documentation page` row; `bin/rule-doc phpqaci.packageType` resolves to its page | | Toolchain | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) | Yes | Inline ignore forbidden; `phparkitect-baseline.json` never read; an included PHPStan baseline fails the justification lane | | Toolchain | [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | `bin/qa -t -p ` for every path-supporting lane | | Toolchain | [4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | `bin/qa` runs coding standards, linting, static analysis, then tests, failing fast or reporting every failure before the success banner | diff --git a/tools/ts-qa-ci.md b/tools/ts-qa-ci.md index 844ddbf..3df23c5 100644 --- a/tools/ts-qa-ci.md +++ b/tools/ts-qa-ci.md @@ -45,11 +45,11 @@ On the toolchain document, the listing under clauses [5.1](../TOOLING-SPEC.md#51 Three causes, each verified through the entry point. -**The dependency-cruiser lane is outside the three commands.** The shipped `no-circular` defence prints its identifier with a two-file cycle and fails the run, but `ts-qa rule-doc no-circular` exits 2 as unknown, `ts-qa rules` does not list it, the lane ignores `-p`, and the release self-audit walks plugin rules only. Through the wrapping, dependency-cruiser fails detector clauses [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) and [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation), which fails toolchain [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), and the toolchain fails [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) and [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) on the same fact. `ts-qa rule no-circular src` compounds it by answering "did not fire" with exit 0 for an identifier the harness cannot see. +**The dependency-cruiser lane is outside the three commands.** The shipped `no-circular` defence prints its identifier with a two-file cycle and fails the run, but `ts-qa rule-doc no-circular` exits 2 as unknown, `ts-qa rules` does not list it, the lane ignores `-p`, and the release self-audit walks plugin rules only. Through the wrapping, dependency-cruiser fails detector clauses [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) and [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation), which fails toolchain [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), and the toolchain fails [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access), [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) and [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) on the same fact. `ts-qa rule no-circular src` compounds it by answering "did not fire" with exit 0 for an identifier the harness cannot see. **ESLint's bulk suppressions file is an open route.** A file containing the literal `PLACEHOLDER` fails the lint lane. With an `eslint-suppressions.json` in the project root naming that file and `ts-qa/no-placeholder`, the same run passes with every phase clean and the project record unchanged. Nothing disables the file or fails on its presence, so toolchain [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) fails, though every inline form is banned. -**Resolution of a core rule stops at a URL.** The always-on as/enum ban is routed through ESLint's `no-restricted-syntax`, and `ts-qa rule-doc no-restricted-syntax` resolves to a one-line description and eslint.org, not to the toolchain's own shipped page for the ban. Detector [6.2](../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) and [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) fail for that identifier and toolchain [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) with them. +**Resolution of a core rule stops at a URL.** The always-on as/enum ban is routed through ESLint's `no-restricted-syntax`, and `ts-qa rule-doc no-restricted-syntax` resolves to a one-line description and eslint.org, not to the toolchain's own shipped page for the ban. Detector [6.2](../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) and [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) fail for that identifier and toolchain [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) with them. The derived listing also exposed a defect: `ts-qa/no-naive-datetime-template` is declared Tier A in the source and the documentation and is absent from the shipped generic configuration, so it is neither listed nor enforced anywhere, which fails [8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source) as a shipped rule inactive on the package itself. At project level, two of the bundled rules, `no-duplicate-section-ids` and `no-placeholder`, have no fixture test, so the red proof method clause [3.3](../SPEC.md#33-prove-the-net-by-making-the-rule-fire) requires is not in the repository for them. Two SHOULDs remain open from the earlier audit: detector [4.4](../DETECTOR-SPEC.md#44-the-detector-should-enforce-43-with-a-rule-of-its-own), a rule over the rules, and toolchain [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically), since `deploy-skills` delivers static files rather than the generated listing. @@ -71,7 +71,7 @@ The derived listing also exposed a defect: `ts-qa/no-naive-datetime-template` is | Detector | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | Directives detected by `ts-qa/no-eslint-disable`; the known-violations baseline needs a flag the toolchain never passes | | Detector | [7.2](../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) | No | A directive's reason is optional and a baseline entry carries none | | Toolchain | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | dependency-cruiser plus the wrapping fails detector [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) and [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) and a defence is routed through it | -| Toolchain | [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | `no-circular` unresolvable; `no-restricted-syntax` resolves to an upstream URL, not the shipped as/enum ban page | +| Toolchain | [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) | No | `no-circular` unresolvable; `no-restricted-syntax` resolves to an upstream URL, not the shipped as/enum ban page | | Toolchain | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) | No | A `PLACEHOLDER` finding failed the lint lane; with `eslint-suppressions.json` naming it the run passed, all phases clean | | Toolchain | [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | No | Entry point meets detector [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) for the ESLint lane; no subset for the dependency-cruiser lane | | Toolchain | [4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | Detectors in phases 0 to 3, tests in 4; a failure at phase 1 ran nothing after it | From 99074928ac750fd2098b309725424dd221403539 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:29:03 +0100 Subject: [PATCH 15/52] Routed findings from runs 63 and 64 acceptance/ROUTED.md gains the detector's clause 6.1 paragraph on a project's own rule, from run 63, and notes run 64 on the four method rows it raised again. Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 76acd8f..55462f4 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -15,25 +15,26 @@ untouched text of the change those runs tested, and are listed here so that they ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------- | ---------------------------------------- | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | run 54 | `editor/detector-route-rule` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59, 61 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59, 61 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | run 61 | `open` | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ---------------------------------------- | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | run 54 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | run 63 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59, 64 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59, 61, 64 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59, 61, 64 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | run 61 | `open` | ## Closed From 7dea25afd6a883b32d4cf9598d71cb2d2595aea9 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:29:24 +0100 Subject: [PATCH 16/52] Run 065: toolchain contradictions cohort, failed on clause 4.2 a third time Every keyed item five of five. Four findings in text the change wrote or altered: clause 4.2's "not yet a Defence" sentence, the Defence row and clause 5.1 set against it, and clause 9.1's "section 7 of the method specification". Nothing routed. Co-Authored-By: Claude Fable 5.1 --- .../065-toolchain-contradictions-run-4.md | 163 ++++++++++++++++++ 1 file changed, 163 insertions(+) create mode 100644 acceptance/runs/065-toolchain-contradictions-run-4.md diff --git a/acceptance/runs/065-toolchain-contradictions-run-4.md b/acceptance/runs/065-toolchain-contradictions-run-4.md new file mode 100644 index 0000000..a959ed2 --- /dev/null +++ b/acceptance/runs/065-toolchain-contradictions-run-4.md @@ -0,0 +1,163 @@ +# Run 65: TOOLING-SPEC.md, toolchain contradictions, fourth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: not yet opened at the time of the run; it is to be opened from branch + `editor/toolchain-contradictions`. The readers read commit dfd4b4d. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 62 in clause 4.2's heading and first paragraph. +- **Run within the pull request**: 4 (runs 57, 58 and 62 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `1186cb6938da03e55dfbec783676bac2414b667e1005d7cf831a67424e0ed90e`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at dfd4b4d, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (all five); 6.1 Yes (1, 4, +5), Partial (2, 3), each Partial mentioning that no location is given for calibrations or +conventions; 6.2 No (all five), each mentioning that nothing rejects generic text or a generic +justification. Reader 1 grades 6.1 Yes and adds that it would be Partial if calibrations and +conventions have no location; the grade in its table is Yes, and Yes and Partial are both +accepted. Readers 2 and 3 note that the fixture does not say the two undocumented rules are +Blocking; both grade 4.2 No, which the key accepts with no condition. + +Question 1: correct bullets per reader 6, 5, 4, 5 and 6 (reader 1: section 1's governing +principle; 4.1; 4.2; 4.3; 4.5 with 4.4; 5.1 to 5.3 with 6.1 to 6.3. Reader 2: 4.1; 4.2; 4.3; +4.4 and 4.5 with 5.1 to 5.3; 6.1 to 6.3. Reader 3: 4.1; 4.2; 4.3 with 4.5; 5.1 to 5.3 with 6.1 +to 6.3. Reader 4: 4.1; 4.2; 4.3 with 6.2; 4.5; 5.1, 5.2, 6.1 and 6.3. Reader 5: 4.1; 4.2; 4.3; +4.4; 4.5; sections 5 and 6). Reader 5's fourth and fifth bullets name different clauses of one +line of the key's list and each count. No bullet asserts anything the key marks wrong. The +first bullet of readers 2, 3 and 4 (the toolchain judged as one assembled whole) and reader 3's +sixth (one project grade, a separate artefact grade, the agent summary a SHOULD) state how +conformance is measured or call section 7 a SHOULD, which the key counts neither way; so does +reader 4's sixth, in calling the agent summary and the self-audit "SHOULD or conditional". A +bullet on neither list is listed under the points the rules did not settle. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through", which states element (2); for 2b all five quote section 9's "The project-level +verdict is a single grade; ..."; for 2c all five quote clause 9.1's "The two verdicts MUST be +graded and declared separately, and neither implies the other." No answer gives a grade per +part or a single grade for 2c. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says section 7 is not met or there is no +agent summary, and that the toolchain does not conform (all five); S8 says Zigpipe ships no +tooling (all five, each also placing zig-qa's grade on its own artefact); Decl says the +declaration is optional and the verdict rests on the exercise (all five). + +### Misses + +None. No answer matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..dfd4b4d adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------- | +| 1 | Clause 4.2, first paragraph: "a [Blocking] [Rule] whose documentation is missing is not yet a [Defence]" (2, 4, 5); the clause covers Blocking rules that by the definition are not Defences (2); whether clause 5.1 then covers an undocumented blocking rule (4, 5) | 2, 4, 5 | yes | +| 2 | Section 2 terminology table, [Defence] row: "A [Rule] with its documentation, in force and [Blocking]", each set against clause 4.2's first paragraph | 2, 4, 5 | no | +| 3 | Clause 5.1, first paragraph: "every [Defence] active in the project" (5), "clause 5.1's listing obligation" (4), each set against clause 4.2's first paragraph | 4, 5 | no | +| 4 | Clause 9.1, first paragraph: "graded against this document and section 7 of the method specification" (1, 3); which section 7 governs the project grade, set against section 1's "How to read" paragraph and section 9 (1) or section 9's first paragraph (3) | 1, 3 | yes | +| 5 | Clause 4.2 heading: "[Blocking] [Rules] it routes" (2), given with the clause number inside its remark on the first paragraph | 2 | yes | +| 6 | Section 1, "How to read this document" paragraph: self-contained "apart from 4.1", set against clause 9.1 | 1 | yes | +| 7 | Section 9, agent-support paragraph: "section 7 of this document", set against clause 9.1 | 1 | yes | +| 8 | Section 9, first paragraph: "defines the project grade by sections 4 to 6 of this document", set against clause 9.1 | 3 | yes | +| 9 | Clause 4.3, first paragraph: "A route bypasses the [Project record] when a finding can be silenced through it without an entry appearing in the [Project record]"; whether disabling an active rule in configuration is a route | 1 | yes | +| 10 | Section 2 terminology, [Project record] entry: "[Calibrations], [Exceptions] and conventions", set against clause 6.1 | 3 | no | +| 11 | Clause 6.1, first paragraph: "The [Toolchain] MUST define where the [Project record] lives and MUST load it"; whether the location covers all three kinds of judgement or only exceptions | 3 | no | +| 12 | Clause 4.1, point 2, paragraph after the table: "exercised as its clause 8.1 describes"; the reader could not follow the reference to the detector specification | 4 | yes | +| 13 | Clause 4.5, first paragraph: "MUST stop the levels below it from being treated as meaningful"; the direction of "below" | 4 | no | + +Notes, not passages: every reader remarks under question 3 that the detector specification, the +method specification or both were not supplied; none of those remarks is under question 4. + +## Findings and where they lie + +- **Finding A, passage 1, clause 4.2 first paragraph** (readers 2, 4 and 5): the sentence named + was added by the change. Text the change wrote. Blocks. +- **Finding B, passage 2, section 2 [Defence] row** (readers 2, 4 and 5): the row is unchanged, + but every reader naming it sets it against clause 4.2's first paragraph, which the change + altered. Blocks. +- **Finding C, passage 3, clause 5.1 first paragraph** (readers 4 and 5): the paragraph is + unchanged, but both readers set it against clause 4.2's first paragraph. Blocks. +- **Finding D, passage 4, clause 9.1 first paragraph** (readers 1 and 3): the sentence quoted is + unchanged, but the change rewords and adds sentences in the same paragraph, and both readers + set it against section 9 or section 1's "How to read" paragraph, which the change also altered. + Text the change altered. Blocks. + +Passages 5 to 13 are each named by one reader: recorded, not findings. Nothing is routed to +`acceptance/ROUTED.md` from this run. + +## Points the rules did not settle + +1. **A question 1 bullet on neither of the key's lists**: reader 3's sixth bullet says "default + values are SHOULDs", which is clause 6.4 and on neither list. Marked by the rule for a bullet + on neither list: not counted either way. Reader 3 has four correct bullets without it, so no + mark turns on it. For the key to settle. +2. **Passages set against another and named by description**: reader 4 identifies clause 5.1 as + "clause 5.1's listing obligation", and reader 3 identifies section 9's first paragraph by a + description, each inside a remark on another passage. Whether that names the paragraph was + marked by the open-point rule: counted. Finding C rests on reader 4's being counted; findings + A, B and D, and so the verdict, do not. +3. **Reader 2's short quotation "[Blocking] [Rules] it routes"** matches no run of the body and + is nearest the clause 4.2 heading. Whether it names the heading was marked by the open-point + rule: counted, as passage 5. No other reader names the heading, so no finding turns on it. +4. **Reader 4's remark on "exercised as its clause 8.1 describes"** quotes a passage with its + clause number, but its substance is that the detector specification was not supplied, which + the Marking section calls a note. Marked by the open-point rule: counted as a passage. Named + by reader 4 alone, so no finding turns on it. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 4081aa4, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: four (findings A, + B, C and D). Not met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A, B, C and D; every keyed item passed. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From 54e4780a38449ba9abc7338a9fc2c748c1de353e Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:29:24 +0100 Subject: [PATCH 17/52] Toolchain 4.2: blocking rules, not defences After run 65: clause 4.2's heading and obligation name blocking rules alone, with one sentence on why the clause names them rather than defences; clause 8.1 drops its sentence on the same contrast; clause 9.1 says the toolchain is graded against this document under section 9 and the project's remediations against the method's section 7. The heading's anchor follows in the keys, the question set, the changelog, the routed list and the tools pages. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 21 ++++++++++------- TOOLING-SPEC.md | 22 ++++++++---------- acceptance/ROUTED.md | 38 +++++++++++++++---------------- acceptance/toolchain/KEY.md | 6 ++--- acceptance/toolchain/QUESTIONS.md | 2 +- tools/php-qa-ci.md | 4 ++-- tools/ts-qa-ci.md | 6 ++--- 7 files changed, 51 insertions(+), 48 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e2ea1f0..4c3afb1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -101,7 +101,7 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be the document pulling against itself, each listed in `acceptance/ROUTED.md`. Section [9](TOOLING-SPEC.md#9-conformance)'s conformance sentence now says that section [8](TOOLING-SPEC.md#8-self-audit) bears on a shipping project's artefact grade under clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately), not on its project grade, which is what clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) already said and the - answer key assumed. Clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) says that one identifier that does not resolve fails it and + answer key assumed. Clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) says that one identifier that does not resolve fails it and that a blocking rule whose documentation is missing is graded as a defence with a gap, and clause [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) says the same of a bundled defence, after readers took section [2](TOOLING-SPEC.md#2-terminology)'s Defence row to put such a rule outside both clauses. Clause [9.2](TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance)'s duty to record a known gap in the @@ -112,7 +112,7 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be applies and the six it lists are a summary, and its point 2 says the wrapped pair is what the clause judges whilst the detector's own grade is a separate verdict. Prompted by runs 33 to 52 and 55 on the defined-terms change, in which two or more readers quoted each passage. - [Run 57](acceptance/runs/057-toolchain-contradictions-run-1.md) failed on the first wording: every reader set clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)'s + [Run 57](acceptance/runs/057-toolchain-contradictions-run-1.md) failed on the first wording: every reader set clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)'s "a defence with a gap" against the Defence row, so the sentence now says Defence there reaches every blocking rule whether or not its documentation exists; clause [4.3](TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) says the routes are the ones each detector offers, after two readers graded it Partial for routes the @@ -121,18 +121,23 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be sentence says section [7](TOOLING-SPEC.md#7-agent-context) recommends, after two readers set "additionally satisfies" against that section's SHOULDs. [Run 58](acceptance/runs/058-toolchain-contradictions-run-2.md) failed on four findings in the revised text, every keyed item five of five: four readers set clause - [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)'s "whether or not its documentation exists" against the Defence row again, so clauses - [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) and [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) now say that a blocking rule whose documentation is missing is not yet a + [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)'s "whether or not its documentation exists" against the Defence row again, so clauses + [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) and [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) now say that a blocking rule whose documentation is missing is not yet a defence and is the failure each names; clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) says its six sentences are a summary and not the test, and its Why names the method's clauses [3.1](SPEC.md#31-attribute-the-defect-to-a-class) to [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) rather than "six clauses"; section [9](TOOLING-SPEC.md#9-conformance) says every project has one project grade and that section [8](TOOLING-SPEC.md#8-self-audit) bears on a shipping project's separate artefact grade, clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) names the two levels, and the agent-support sentence says which document's section [7](TOOLING-SPEC.md#7-agent-context) it means. [Run 62](acceptance/runs/062-toolchain-contradictions-run-3.md) failed on the same clause - [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) sentence, every keyed item five of five: four readers could not settle whether the + [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) sentence, every keyed item five of five: four readers could not settle whether the clause covers its defences or every blocking rule, so its heading and its obligation sentence now name both, a defence or a blocking rule it routes, and the body says why the blocking - rules are named. Accepted by run NNN. + rules are named. [Run 65](acceptance/runs/065-toolchain-contradictions-run-4.md) failed on it + a third time, three readers unable to square a clause on defences with rules that are not + yet one, so the heading and the obligation now name blocking rules alone, with one sentence + on why, and clause [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) drops its sentence on the same contrast; clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) says which + document grades what, after two readers asked which section [7](TOOLING-SPEC.md#7-agent-context) the project grade rests on. + Accepted by run NNN. ### 0.2.0, 2026-09-08 @@ -159,10 +164,10 @@ since the clause says the listing MUST provide the route and [4.1](TOOLING-SPEC. again by a later clause; three readers of run 8 graded it so from the text. Findings applied across the runs: an eleven-term gloss and a reading note in sections [1](TOOLING-SPEC.md#1-what-this-document-is-for) and [2](TOOLING-SPEC.md#2-terminology); long sentences split; [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) opens by listing the detector specification's MUSTs and its point 2 -leads with a three-case table of wrapped pairs, then the one question behind it; [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) says it +leads with a three-case table of wrapped pairs, then the one question behind it; [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) says it widens the detector document's reach to the project's own rules; [4.3](TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) loses its double negative; [5.2](TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) gives the derived-listing test; [6.2](TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one)'s heading carries the generic-justification rejection; -[8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) is consistent with [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) on third-party catalogues. Runs 1 to 7 also asked readers what would +[8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) is consistent with [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) on third-party catalogues. Runs 1 to 7 also asked readers what would make a clause clearer and for a clarity rating; both questions were withdrawn before run 8, as `ACCEPTANCE.md` records. diff --git a/TOOLING-SPEC.md b/TOOLING-SPEC.md index 458367e..0fd4436 100644 --- a/TOOLING-SPEC.md +++ b/TOOLING-SPEC.md @@ -159,16 +159,15 @@ mechanism lives, and the method does not care. It is bounded because a mechanism does not offer and the [Toolchain] does not add is a gap the project writes down under method clause 3.2. A gap recorded is not a gap closed. -### 4.2 The toolchain MUST resolve every identifier a defence or a blocking rule it routes can print, from the installed copy, without network access +### 4.2 The toolchain MUST resolve every identifier a blocking rule it routes can print, from the installed copy, without network access This clause widens the [detector specification](DETECTOR-SPEC.md)'s reach: that document resolves [Bundled rules] only, and this one covers every [Identifier] the project can see, the project's -own [Rules] included. The [Toolchain] MUST resolve every [Identifier] its [Defences] -print, and every [Identifier] its other [Blocking] [Rules] print, to [Remediation docs] on disk. -One [Identifier] that does not resolve fails this clause, however many others do. The other -[Blocking] [Rules] are named because a [Defence] is a [Rule] with its documentation, as section -2 says: a [Blocking] [Rule] whose documentation is missing is not yet a [Defence], and it is -exactly the failure this clause exists to catch. The mechanism +own [Rules] included. The [Toolchain] MUST resolve every [Identifier] its [Blocking] [Rules] +print to [Remediation docs] on disk. One [Identifier] that does not resolve fails this clause, +however many others do. The clause names [Blocking] [Rules] rather than [Defences] because the +failure it catches is a [Rule] that blocks without the documentation that would make it a +[Defence]. The mechanism is keyed on the [Identifier] exactly as printed, and it covers the project's own [Rules] and every [Bundled defence]. Those [Remediation docs] MUST ship @@ -379,9 +378,8 @@ mistake rather than after it, which is why they are worth stating even as SHOULD The [Toolchain] MUST run an automated check that blocks its own release. The check covers every [Identifier] printed by a [Rule] the [Toolchain] authors or bundles -as its own, whatever kind of [Detector] carries it. A [Bundled defence] is a [Rule] with its -documentation, so a [Rule] shipped that way whose page is missing is not yet one; it is what -this check exists to find, and the check reaches every such [Rule]. Where a documentation +as its own, whatever kind of [Detector] carries it, and a missing page is what the check exists +to find. Where a documentation page covers a family of [Identifiers] by a pattern, as clause 6.5 of the [detector specification](DETECTOR-SPEC.md) allows, this audit MUST apply the pattern to every [Identifier] printed and confirm it lands on that page. A member added to the @@ -436,8 +434,8 @@ removes the only value the word has. ### 9.1 A project that ships a detector or a toolchain has two levels of conformance, graded separately As a project, it follows the method with its own assembled [Toolchain], like any other -project, and is graded against this document and section 7 of the method specification on that -basis. As an artefact, what it ships is graded for its consumers: a [Detector] against +project: that [Toolchain] is graded against this document, under section 9, and the project's +remediations against section 7 of the method specification, as any project's are. As an artefact, what it ships is graded for its consumers: a [Detector] against [the detector specification](DETECTOR-SPEC.md), a [Toolchain] against this document, on what the shipped [Toolchain] itself supplies once installed into a [Consuming project], counting nothing the [Consuming project] adds around it. The two diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 65acc8a..5a8de0b 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -15,25 +15,25 @@ untouched text of the change those runs tested, and are listed here so that they ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------- | ---------------------------------------- | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | run 54 | `editor/detector-route-rule` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59, 61 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59, 61 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | run 61 | `open` | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------- | ---------------------------------------- | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | run 54 | `editor/detector-route-rule` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59, 61 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59, 61 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | run 61 | `open` | ## Closed diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 9fe09d6..6967ab4 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -13,7 +13,7 @@ no bullet making an assertion listed below as wrong, is a correct restatement. - Route defences only through detectors that conform to the detector specification, wrapping permitted ([4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification)). - Resolve every identifier a routed defence can print, including the project's own, offline, - to documentation stating the correct construction ([4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)). + to documentation stating the correct construction ([4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)). - Forbid, by disabling or by a blocking defence, every suppression route that bypasses the project record ([4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)). - The toolchain's own entry point meets the detector reporting clauses and prints identifiers @@ -62,7 +62,7 @@ Wrong readings the text invites, which count against the document on one reader: | Clause | Accepted grades | Condition on the reasoning | What the fixture supports | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | None | Lintomatic fails detector [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) on two undocumented bundled rules and nothing in the wrapping closes that gap; a reader who notes that the detector specification was not supplied and grades from this document's summary of it is correct | -| [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) | No | None. A No given on the condition that the two bundled rules are blocking is a No: their firing stops `zig-qa run`, so they are | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)` | +| [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) | No | None. A No given on the condition that the two bundled rules are blocking is a No: their firing stops `zig-qa run`, so they are | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)` | | [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) | Yes | None | Ignore comments disabled project-wide; exceptions only through the record the toolchain reads | | [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | None | `zig-qa run` is local, prints Lintomatic's output unchanged, one line per finding | | [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | None | Lintomatic, then the tests, stopping on a Lintomatic failure | @@ -76,7 +76,7 @@ Wrong readings the text invites, which count against the document on one reader: The four answers the question then asks for, each with the condition its reason must meet: - **Conformance under section [9](../../TOOLING-SPEC.md#9-conformance).** Accepted answer: does not conform. Condition: the reason - names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) or [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) among the failing clauses, or points to the grades the reader gave above, as "the + names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) or [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) among the failing clauses, or points to the grades the reader gave above, as "the clauses graded No", where those grades have one of the three at No or Partial; other clauses named alongside are not marked. - **Conformance with agent support.** Accepted answer: no. Condition: the reason says that diff --git a/acceptance/toolchain/QUESTIONS.md b/acceptance/toolchain/QUESTIONS.md index 21a8d5e..fa45dd3 100644 --- a/acceptance/toolchain/QUESTIONS.md +++ b/acceptance/toolchain/QUESTIONS.md @@ -24,7 +24,7 @@ the document you are relying on. ## 3. Application Grade the Zigpipe project's toolchain, as described in the fixture, against every MUST clause in -sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record): [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access), [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own), [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure), [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration), [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones), [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences). For each give Yes, +sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record): [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access), [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own), [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure), [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration), [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones), [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences). For each give Yes, Partial or No and one line of reasoning. Then state whether the toolchain conforms under section [9](../../TOOLING-SPEC.md#9-conformance) and why, whether it conforms with agent support and why, and whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies to it and why. Finally, state whether the absence of a declaration affects the verdict, and why. diff --git a/tools/php-qa-ci.md b/tools/php-qa-ci.md index feb7567..f241cb4 100644 --- a/tools/php-qa-ci.md +++ b/tools/php-qa-ci.md @@ -22,7 +22,7 @@ Every mechanism below was checked by running it in a checkout: `bin/qa`, `bin/qa For its own PHPStan bundle, every detector clause holds. A project's rules go under `rules:` in its `qaConfig/phpstan.neon` (detector [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it)). `bin/phpstan-rule phpqaci.nullCoalescingFalse src/ZzProbe.php` on a probe containing `?? false` printed `FIRED (1)` with the location and exited 1, and the same command for `phpqaci.nestedTernary` printed `did not fire` (detector [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code)). PHPStan prints the `phpqaci.*` identifier under every finding, and `RequireRuleIdentifierConstantRule`, on by default, rejects a magic-string identifier (detector [4.3](../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding), detector [4.4](../DETECTOR-SPEC.md#44-the-detector-should-enforce-43-with-a-rule-of-its-own)). `bin/qa -t phpstan -p ` runs one lane over one file locally with the result in the terminal, and the CI script runs the same `bin/qa` (detector [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to detector [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally)). `bin/rule-doc phpqaci.nestedTernary` resolves the identifier as printed to its rule, bundle, source and shipped page, offline, and the page carries a `The correct construction` section (detector [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) to detector [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) for the bundle). `RuleDocumentationTest` fails the build when a declared identifier resolves to no page, or to a page whose construction section is missing or only restates the summary (detector [6.4](../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation)). Inline `@phpstan-ignore` in every form is a finding of `ForbidInlinePhpstanIgnoreRule` (detector [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)), and behind it the default tier turns on PHPStan's `reportIgnoresWithoutComments`: a probe ignore with no reason printed both `phpqaci.inlinePhpstanIgnore` and `ignore.noComment (non-ignorable)` (detector [7.2](../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason)). -At toolchain level, every lane prints a stable identifier when it fails: a run of `bin/qa -t branchNamePolicy` on a `plan/*` branch ended with `🪪 phpqaci.branchNamePolicy (vendor/bin/rule-doc phpqaci.branchNamePolicy)`, and `EveryLaneNamesItselfTest` fails the build on a lane without one. `bin/rule-doc phpqaci.packageType` resolves a lane identifier to its page as it does a rule's, and `bin/rules .` prints no `no documentation page` row: all 69 indexed identifiers (`bin/rule-doc --list`) resolve to a page ([4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)). `bin/rules .` lists every rule in the resolved neon chain, including the rules `phpstan/extension-installer` delivers, with the package each came from, and every lane with its identifier and documentation route; phase runners alone carry none ([5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) to [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones)). The record is the `ignoreErrors` block PHPStan itself loads ([6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself)); the `pij` lane reads it through every `includes:` file, and a probe baseline included from `qaConfig/phpstan.neon` with an uncommented entry failed it, naming the included file and line ([6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)). A `phparkitect-baseline.json` is never read: on a probe project with one violation, PHPArkitect with a generated baseline printed `No violations detected`, and with `--skip-baseline`, which the lane always passes, reported the violation; the lane names the file when one is present ([4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)). The same listing enumerates the record beside the defences, together with the deferred-defect and no-pattern record `qaConfig/defect-record.neon` (`defectRecord` in `--json`), which the same lane fails on when an entry does not read in full ([6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences)). `docs/defence-before-fix-defaults.md` states the toolchain's default for the owner, the project record, the sweep scope, fixtures and the method's three calibrations, linked from the agent block ([6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)). The entry point runs coding standards, linting and static analysis before tests and stops on a failure ([4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure)). `bin/rules --write-agent-summary` writes one line per active defence, identifier and page, into a marked region of the project's `CLAUDE.md`, refreshed on every install and update ([7.1](../TOOLING-SPEC.md#71-the-toolchain-should-generate-a-summary-of-the-active-defences-suitable-for-an-agents-context), [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically)). The release guard covers rules and lanes and requires a page stating a correct construction ([8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation)), and the self-check runs both bundled rule sets and the PHPArkitect tier on the package's own source ([8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source)). The manifest declares method 1.1.0 and toolchain 0.2.0 at both levels, with the gaps below recorded against their clauses, and `DefenceBeforeFixDeclarationTest` holds each recorded gap to a decision and a probe that fails when the gap closes ([9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance)). +At toolchain level, every lane prints a stable identifier when it fails: a run of `bin/qa -t branchNamePolicy` on a `plan/*` branch ended with `🪪 phpqaci.branchNamePolicy (vendor/bin/rule-doc phpqaci.branchNamePolicy)`, and `EveryLaneNamesItselfTest` fails the build on a lane without one. `bin/rule-doc phpqaci.packageType` resolves a lane identifier to its page as it does a rule's, and `bin/rules .` prints no `no documentation page` row: all 69 indexed identifiers (`bin/rule-doc --list`) resolve to a page ([4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)). `bin/rules .` lists every rule in the resolved neon chain, including the rules `phpstan/extension-installer` delivers, with the package each came from, and every lane with its identifier and documentation route; phase runners alone carry none ([5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) to [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones)). The record is the `ignoreErrors` block PHPStan itself loads ([6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself)); the `pij` lane reads it through every `includes:` file, and a probe baseline included from `qaConfig/phpstan.neon` with an uncommented entry failed it, naming the included file and line ([6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)). A `phparkitect-baseline.json` is never read: on a probe project with one violation, PHPArkitect with a generated baseline printed `No violations detected`, and with `--skip-baseline`, which the lane always passes, reported the violation; the lane names the file when one is present ([4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)). The same listing enumerates the record beside the defences, together with the deferred-defect and no-pattern record `qaConfig/defect-record.neon` (`defectRecord` in `--json`), which the same lane fails on when an entry does not read in full ([6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences)). `docs/defence-before-fix-defaults.md` states the toolchain's default for the owner, the project record, the sweep scope, fixtures and the method's three calibrations, linked from the agent block ([6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)). The entry point runs coding standards, linting and static analysis before tests and stops on a failure ([4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure)). `bin/rules --write-agent-summary` writes one line per active defence, identifier and page, into a marked region of the project's `CLAUDE.md`, refreshed on every install and update ([7.1](../TOOLING-SPEC.md#71-the-toolchain-should-generate-a-summary-of-the-active-defences-suitable-for-an-agents-context), [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically)). The release guard covers rules and lanes and requires a page stating a correct construction ([8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation)), and the self-check runs both bundled rule sets and the PHPArkitect tier on the package's own source ([8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source)). The manifest declares method 1.1.0 and toolchain 0.2.0 at both levels, with the gaps below recorded against their clauses, and `DefenceBeforeFixDeclarationTest` holds each recorded gap to a decision and a probe that fails when the gap closes ([9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance)). ## How it is not conformant @@ -64,7 +64,7 @@ Detector rows grade PHPStan as wrapped, with PHPArkitect as wrapped noted where | Detector | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | Inline ignores detectable by a bundled rule; PHPArkitect's baseline never read (`--skip-baseline` always passed) | | Detector | [7.2](../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) | Yes | `reportIgnoresWithoutComments` on in the default tier; a probe printed `ignore.noComment (non-ignorable)` | | Toolchain | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | PHPArkitect's tier rules carry no identifier, single-file run or resolver; PHPStan's native catalogue is online only; both declared | -| Toolchain | [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) | Yes | `bin/rules .`: no `no documentation page` row; `bin/rule-doc phpqaci.packageType` resolves to its page | +| Toolchain | [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) | Yes | `bin/rules .`: no `no documentation page` row; `bin/rule-doc phpqaci.packageType` resolves to its page | | Toolchain | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) | Yes | Inline ignore forbidden; `phparkitect-baseline.json` never read; an included PHPStan baseline fails the justification lane | | Toolchain | [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | `bin/qa -t -p ` for every path-supporting lane | | Toolchain | [4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | `bin/qa` runs coding standards, linting, static analysis, then tests, failing fast or reporting every failure before the success banner | diff --git a/tools/ts-qa-ci.md b/tools/ts-qa-ci.md index 3df23c5..aeeb1cd 100644 --- a/tools/ts-qa-ci.md +++ b/tools/ts-qa-ci.md @@ -45,11 +45,11 @@ On the toolchain document, the listing under clauses [5.1](../TOOLING-SPEC.md#51 Three causes, each verified through the entry point. -**The dependency-cruiser lane is outside the three commands.** The shipped `no-circular` defence prints its identifier with a two-file cycle and fails the run, but `ts-qa rule-doc no-circular` exits 2 as unknown, `ts-qa rules` does not list it, the lane ignores `-p`, and the release self-audit walks plugin rules only. Through the wrapping, dependency-cruiser fails detector clauses [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) and [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation), which fails toolchain [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), and the toolchain fails [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access), [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) and [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) on the same fact. `ts-qa rule no-circular src` compounds it by answering "did not fire" with exit 0 for an identifier the harness cannot see. +**The dependency-cruiser lane is outside the three commands.** The shipped `no-circular` defence prints its identifier with a two-file cycle and fails the run, but `ts-qa rule-doc no-circular` exits 2 as unknown, `ts-qa rules` does not list it, the lane ignores `-p`, and the release self-audit walks plugin rules only. Through the wrapping, dependency-cruiser fails detector clauses [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) and [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation), which fails toolchain [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), and the toolchain fails [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access), [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) and [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) on the same fact. `ts-qa rule no-circular src` compounds it by answering "did not fire" with exit 0 for an identifier the harness cannot see. **ESLint's bulk suppressions file is an open route.** A file containing the literal `PLACEHOLDER` fails the lint lane. With an `eslint-suppressions.json` in the project root naming that file and `ts-qa/no-placeholder`, the same run passes with every phase clean and the project record unchanged. Nothing disables the file or fails on its presence, so toolchain [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) fails, though every inline form is banned. -**Resolution of a core rule stops at a URL.** The always-on as/enum ban is routed through ESLint's `no-restricted-syntax`, and `ts-qa rule-doc no-restricted-syntax` resolves to a one-line description and eslint.org, not to the toolchain's own shipped page for the ban. Detector [6.2](../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) and [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) fail for that identifier and toolchain [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) with them. +**Resolution of a core rule stops at a URL.** The always-on as/enum ban is routed through ESLint's `no-restricted-syntax`, and `ts-qa rule-doc no-restricted-syntax` resolves to a one-line description and eslint.org, not to the toolchain's own shipped page for the ban. Detector [6.2](../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) and [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) fail for that identifier and toolchain [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) with them. The derived listing also exposed a defect: `ts-qa/no-naive-datetime-template` is declared Tier A in the source and the documentation and is absent from the shipped generic configuration, so it is neither listed nor enforced anywhere, which fails [8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source) as a shipped rule inactive on the package itself. At project level, two of the bundled rules, `no-duplicate-section-ids` and `no-placeholder`, have no fixture test, so the red proof method clause [3.3](../SPEC.md#33-prove-the-net-by-making-the-rule-fire) requires is not in the repository for them. Two SHOULDs remain open from the earlier audit: detector [4.4](../DETECTOR-SPEC.md#44-the-detector-should-enforce-43-with-a-rule-of-its-own), a rule over the rules, and toolchain [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically), since `deploy-skills` delivers static files rather than the generated listing. @@ -71,7 +71,7 @@ The derived listing also exposed a defect: `ts-qa/no-naive-datetime-template` is | Detector | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | Directives detected by `ts-qa/no-eslint-disable`; the known-violations baseline needs a flag the toolchain never passes | | Detector | [7.2](../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) | No | A directive's reason is optional and a baseline entry carries none | | Toolchain | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | dependency-cruiser plus the wrapping fails detector [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) and [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) and a defence is routed through it | -| Toolchain | [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-or-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) | No | `no-circular` unresolvable; `no-restricted-syntax` resolves to an upstream URL, not the shipped as/enum ban page | +| Toolchain | [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) | No | `no-circular` unresolvable; `no-restricted-syntax` resolves to an upstream URL, not the shipped as/enum ban page | | Toolchain | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) | No | A `PLACEHOLDER` finding failed the lint lane; with `eslint-suppressions.json` naming it the run passed, all phases clean | | Toolchain | [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | No | Entry point meets detector [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) for the ESLint lane; no subset for the dependency-cruiser lane | | Toolchain | [4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | Detectors in phases 0 to 3, tests in 4; a failure at phase 1 ran nothing after it | From 5852a8562abd0828b9519726ddcce3490d13b388 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:40:51 +0100 Subject: [PATCH 18/52] Routed findings: four detector rows closed by run 66, two toolchain rows from run 67 acceptance/ROUTED.md gains a Closed table, with the detector's clauses 7.1, 8.1, 5.3 and 6.1 rows closed by editor/detector-route-rule's run 66, notes run 66 on the clause 6.1 row on a project's own rule, and gains the toolchain's clause 6.2 heading and documented-check paragraph from run 67. Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 47 ++++++++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 21 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 55462f4..f995c21 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -15,27 +15,32 @@ untouched text of the change those runs tested, and are listed here so that they ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ---------------------------------------- | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | run 54 | `editor/detector-route-rule` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | run 63 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56, 64 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59, 64 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59, 61, 64 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59, 61, 64 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | run 61 | `open` | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ---------------------------------------- | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | run 67 | `open` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 63, 66 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59, 64 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59, 61, 64 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59, 61, 64 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | run 61 | `open` | ## Closed -None yet. +A row moves here when a change that reworks its passage passes a cohort, naming that run, or +when the editor closes it with a reason. + +| Document | Clause | Passage | Raised by | Closed by | +| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ------------------------------------ | +| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | run 54 | `editor/detector-route-rule`, run 66 | From 22c3a9a8672c6f0b75e684142bd327d96505c1a1 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:45:36 +0100 Subject: [PATCH 19/52] Run 067: toolchain contradictions cohort, failed on clause 4.2 and section 1 Every keyed item five of five. Two findings in text the change wrote or altered: clause 4.2's first paragraph, three readers each stopping at a different sentence of it, and section 1's "How to read this document" against clause 4.4. Two findings in untouched text, clause 6.2's heading and its paragraph on the documented check, routed to acceptance/ROUTED.md. Co-Authored-By: Claude Fable 5.1 --- .../067-toolchain-contradictions-run-5.md | 171 ++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 acceptance/runs/067-toolchain-contradictions-run-5.md diff --git a/acceptance/runs/067-toolchain-contradictions-run-5.md b/acceptance/runs/067-toolchain-contradictions-run-5.md new file mode 100644 index 0000000..f5ce03d --- /dev/null +++ b/acceptance/runs/067-toolchain-contradictions-run-5.md @@ -0,0 +1,171 @@ +# Run 67: TOOLING-SPEC.md, toolchain contradictions, fifth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: not yet opened at the time of the run; it is to be opened from branch + `editor/toolchain-contradictions`. The readers read commit 645d811. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 65 in clause 4.2's heading and first paragraph, clause 8.1's first paragraph + and clause 9.1's first paragraph. +- **Run within the pull request**: 5 (runs 57, 58, 62 and 65 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `efd52c615ca1e070ef6fc016a6d6eaeaac069cb9baf184ba36805631a0fabcf3`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at 645d811, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (1), Partial (2, 4, 5), No +(3); 6.1 Yes (1, 4, 5), Partial (2, 3), each Partial mentioning that no location is given for +calibrations or conventions; 6.2 No (all five), each mentioning that nothing rejects generic +text or a generic justification. Readers 1, 3, 4 and 5 note that the fixture does not say the +two undocumented rules are Blocking and grade 4.2 No on that assumption or inference, which the +key accepts with no condition. + +Question 1: correct bullets per reader 5, 5, 4, 5 and 5 (reader 1: 4.1; 4.2; 4.3 with 4.5; 5.1 +to 5.3; 6.1 to 6.3. Reader 2: 4.1; 4.2; 4.3; 4.4 with 4.5; 5.1 to 5.3 with 6.1 and 6.2. Reader +3: 4.1 with 4.2; 4.3 with 4.5; 5.1 to 5.3; 6.1 to 6.3. Reader 4: 4.1; 4.2; 4.3 with 4.5; 5.1 to +5.3; 6.1 to 6.3. Reader 5: 4.1; 4.2; 4.3 with 6.2; 4.4 with 4.5; 5.1 to 5.3 with 6.3 and 8.2). +Reader 5's fourth and sixth bullets name different clauses of one line of the key's list and +each count. No bullet asserts anything the key marks wrong. The first bullet of every reader +(the toolchain judged as one assembled whole, or one grade) states how conformance is measured, +and reader 3's sixth (a shipped artefact graded separately, section 7 SHOULDs, the declaration +optional) is of the kinds the key counts neither way. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through", which states element (2); for 2b all five quote section 9's "The project-level +verdict is a single grade; ..."; for 2c all five quote clause 9.1's "The two verdicts MUST be +graded and declared separately, and neither implies the other." No answer gives a grade per +part or a single grade for 2c. Reader 1's 2c says "two grades if it ships an artefact"; see the +points the rules did not settle. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says section 7 is not met or there is +no agent summary, and that the toolchain does not conform (all five); S8 says Zigpipe ships no +tooling (all five, readers 1, 2, 3 and 5 also placing zig-qa's grade on its own artefact); Decl +says the declaration is optional, not a condition, or that the verdict rests on the exercise +(all five). + +### Misses + +None. No answer matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..645d811 adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------- | +| 1 | Clause 4.2, first paragraph: "this one covers every [Identifier] the project can see, the project's own [Rules] included" and "covers the project's own [Rules] and every [Bundled defence]", set against the heading (1); "The mechanism is keyed on the [Identifier] exactly as printed", the mechanism not named first (3); "The [Toolchain] MUST resolve every [Identifier] its [Blocking] [Rules] print to [Remediation docs] on disk", the term not glossed in section 2 (5) | 1, 3, 5 | yes | +| 2 | Section 1, "How to read this document" paragraph: "self-contained for grading a [Toolchain], apart from clause 4.1", set against clause 4.4's citation of detector clauses 5.1 to 5.4 (4); described as where "the document says" the section 2 table "is how terms are defined for grading", set against clause 4.2 (5) | 4, 5 | yes | +| 3 | Clause 6.2 heading: "MUST carry a written justification that names the hazard and the scope" (3); "Clause 6.2's heading", "name the hazard and the scope" (4); each set against clause 6.2's fourth paragraph | 3, 4 | no | +| 4 | Clause 6.2, fourth paragraph: "by a check it documents" (3); "by a check it documents: 'needed for now', 'legacy', 'TODO' and their like" and "cannot verify truth" (4); whether the check must test that hazard and scope are named | 3, 4 | no | +| 5 | Clause 4.2 heading: "every [Identifier] a [Blocking] [Rule] it routes can print"; whether non-blocking rules are in scope | 1 | yes | +| 6 | Clause 6.2, first paragraph: "It MUST NOT supply a default", set against clause 6.4 | 1 | no | +| 7 | Clause 6.4 heading: "The [Toolchain] SHOULD state its own defaults for anything the method leaves to the project" | 1 | no | +| 8 | Clause 5.1, first paragraph: "MUST list every Defence active in the project" and "MUST provide the route to each one's full documentation"; whether an undocumented rule is listed | 2 | no | +| 9 | Section 2 terminology table, [Defence] row: "A Rule with its documentation, in force and Blocking", set against clause 5.1 | 2 | no | +| 10 | Clause 4.3, second paragraph: "This clause is where the project's decision is made and enforced, and the decision is no" | 2 | no | +| 11 | Clause 9.1, first paragraph: "MUST be graded and declared separately", set against clause 9.2 | 3 | yes | +| 12 | Clause 9.2, third paragraph: "The declaration is optional." | 3 | yes | +| 13 | Clause 6.1, first paragraph: "define where the [Project record] lives"; whether one path holds all three kinds of judgement | 3 | no | +| 14 | Section 2 terminology, [Project record] entry: the record covering calibrations, exceptions and conventions, set against clause 6.1 | 3 | no | +| 15 | Clause 4.4, first paragraph: "clauses 5.1 to 5.4 of the [detector specification]", set against section 1's "How to read" paragraph | 4 | no | +| 16 | Clause 4.5, first paragraph: "in the order method section 5 states"; the method specification not supplied | 4 | no | +| 17 | Clause 5.3, second paragraph: "Where such a [Defence] is not expressible in the [Toolchain]'s own [Detectors], its entry MAY be sourced from wherever it is enabled"; what "enabled" refers to | 5 | no | + +Notes, not passages: readers 2, 3 and 4 remark under question 3 that the detector specification +was not supplied; none of those remarks is under question 4. Reader 5's reference to "the +section 2 table" as a whole names no row of it. + +## Findings and where they lie + +- **Finding A, passage 1, clause 4.2 first paragraph** (readers 1, 3 and 5): the diff rewrites + this paragraph, adding the sentences on [Blocking] [Rules] and rewrapping the sentence reader 3 + quotes. Text the change wrote or altered. Blocks. +- **Finding B, passage 2, section 1 "How to read this document" paragraph** (readers 4 and 5, + reader 5 counted by the open-point rule, point 1 below): the diff adds "apart from clause 4.1, + which holds each [Detector] to the [detector specification](DETECTOR-SPEC.md) and summarises + that document's MUSTs" to this paragraph. Text the change wrote or altered. Blocks. +- **Finding C, passage 3, clause 6.2 heading** (readers 3 and 4): unchanged by the diff, and each + reader sets it only against clause 6.2's fourth paragraph, also unchanged. Elsewhere: to be + recorded in `acceptance/ROUTED.md` and routed; does not block. +- **Finding D, passage 4, clause 6.2 fourth paragraph** (readers 3 and 4): unchanged by the diff, + and set only against clause 6.2's heading. Elsewhere: to be recorded in `acceptance/ROUTED.md` + and routed; does not block. + +Passages 5 to 17 are each named by one reader: recorded, not findings. + +## Points the rules did not settle + +1. **Reader 5's description of section 1's "How to read" paragraph.** Reader 5 gives no clause + number for it, describing it as where "the document says" the section 2 table "is how terms + are defined for grading"; the nearest text is "The table in section 2 glosses every term it + borrows" in that paragraph, with section 2's opening paragraph the other candidate. Whether + that names the paragraph was marked by the open-point rule: counted. Finding B rests on it; + finding A, and so the verdict, does not. +2. **Remarks on another document not supplied, given as passages.** Reader 4's remarks on clauses + 4.4 and 4.5 and reader 5's on clause 4.2 each quote a passage with its clause number, but turn + in part on the detector or method specification not being supplied, which the Marking section + calls a note. Marked by the open-point rule: counted as passages. Finding A stands on readers 1 + and 3 without reader 5; finding B rests on reader 4's clause 4.4 remark being counted. +3. **Reader 1's 2c answer, "two grades if it ships an artefact".** The Marking section gives a + rule for a grade on a stated condition only under question 3. The question's company ships + the package, so read for its sense the answer states two for the case asked about, and its + quotation of clause 9.1 states element (2). Marked for the reader under the open-point rule: + correct. No other mark turns on it. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 9907492, named before the readers started. The process files at +645d811 are identical to those at 9907492. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: two (findings A + and B). Not met. +- Passages named by two or more readers elsewhere: two (findings C and D), for + `acceptance/ROUTED.md`; they do not block. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A and B; every keyed item passed. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From 8fe50565c2cb19740b6a789324ea339807f799a8 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:46:21 +0100 Subject: [PATCH 20/52] Toolchain 4.2 rewritten in three paragraphs; section 1 on grading from this document alone After run 67: clause 4.2 states the obligation first, then what is in scope and why it names blocking rules, then the mechanism and the documentation's home, with Remediation docs glossed in section 2's table; section 1's "How to read this document" says a toolchain can be graded from this document alone because each clause that rests on the detector specification says in its own words what that document requires. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 8 ++++++- TOOLING-SPEC.md | 55 +++++++++++++++++++++++++------------------------ 2 files changed, 35 insertions(+), 28 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c3afb1..bb27cd2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -137,7 +137,13 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be yet one, so the heading and the obligation now name blocking rules alone, with one sentence on why, and clause [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) drops its sentence on the same contrast; clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) says which document grades what, after two readers asked which section [7](TOOLING-SPEC.md#7-agent-context) the project grade rests on. - Accepted by run NNN. + [Run 67](acceptance/runs/067-toolchain-contradictions-run-5.md) failed on clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)'s first + paragraph, three readers each stopping at a different sentence of it, and on section [1](TOOLING-SPEC.md#1-what-this-document-is-for)'s + "How to read this document" set against clause [4.4](TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes): the paragraph is rewritten in three, the + obligation first, then what is in scope, then the mechanism, with Remediation docs glossed in + section [2](TOOLING-SPEC.md#2-terminology)'s table; and section [1](TOOLING-SPEC.md#1-what-this-document-is-for) says that a toolchain can be graded from this document + alone because each clause that rests on the detector specification says in its own words what + that document requires. Accepted by run NNN. ### 0.2.0, 2026-09-08 diff --git a/TOOLING-SPEC.md b/TOOLING-SPEC.md index 0fd4436..b9a5350 100644 --- a/TOOLING-SPEC.md +++ b/TOOLING-SPEC.md @@ -39,9 +39,9 @@ building the mechanism first. A [Toolchain] that leaves the project to construct moved the obligation rather than met it. Where the project builds the means itself, those scripts are part of its [Toolchain] and are judged as such. -**How to read this document.** It is self-contained for grading a [Toolchain], apart from clause -4.1, which holds each [Detector] to the [detector specification](DETECTOR-SPEC.md) and summarises -that document's MUSTs. The +**How to read this document.** A [Toolchain] can be graded from this document alone. Where a +clause rests on the [detector specification](DETECTOR-SPEC.md), as clauses 4.1, 4.2 and 4.4 do, +the clause says in its own words what that document requires of the [Detector]. The table in section 2 glosses every term it borrows. Each clause states its obligation in its heading and first sentence, and the **Why** paragraph beneath is reasoning, not a further requirement. A link is there for depth and is not a prerequisite for the sentence that carries it. @@ -51,19 +51,20 @@ link is there for depth and is not a prerequisite for the sentence that carries Terms from the method specification and the [detector specification](DETECTOR-SPEC.md) carry over unchanged. Every capitalised term links to its definition; the ones this document leans on most are, in short: -| Term | In one line | -| -------------- | ----------------------------------------------------------------------------------- | -| [Detector] | A tool that reads code without executing it and reports what it finds | -| [Runner] | A tool that executes code, a test suite or a compilation, and reports what happened | -| [Rule] | One check a [Detector] evaluates | -| [Defence] | A [Rule] with its documentation, in force and [Blocking] | -| [Identifier] | The stable name printed with a finding that leads to its documentation | -| [Blocking] | Fails the run rather than issuing a [Warning] | -| [Exception] | A recorded, justified decision to leave an [Instance] unfixed or a [Rule] unapplied | -| [Practitioner] | Whoever is doing the work, a person or an [Agent] | -| [Owner] | The human who decides what the codebase may keep | -| [Bundled rule] | A [Rule] a [Detector] ships rather than one the project wrote | -| [Harness] | The route by which one [Rule] is run against supplied code on its own | +| Term | In one line | +| ------------------ | --------------------------------------------------------------------------------------------------- | +| [Detector] | A tool that reads code without executing it and reports what it finds | +| [Runner] | A tool that executes code, a test suite or a compilation, and reports what happened | +| [Rule] | One check a [Detector] evaluates | +| [Defence] | A [Rule] with its documentation, in force and [Blocking] | +| [Identifier] | The stable name printed with a finding that leads to its documentation | +| [Remediation docs] | The documentation an [Identifier] leads to: what the [Rule] is about, why, and how to fix a finding | +| [Blocking] | Fails the run rather than issuing a [Warning] | +| [Exception] | A recorded, justified decision to leave an [Instance] unfixed or a [Rule] unapplied | +| [Practitioner] | Whoever is doing the work, a person or an [Agent] | +| [Owner] | The human who decides what the codebase may keep | +| [Bundled rule] | A [Rule] a [Detector] ships rather than one the project wrote | +| [Harness] | The route by which one [Rule] is run against supplied code on its own | These are additional. @@ -161,16 +162,15 @@ clause 3.2. A gap recorded is not a gap closed. ### 4.2 The toolchain MUST resolve every identifier a blocking rule it routes can print, from the installed copy, without network access -This clause widens the [detector specification](DETECTOR-SPEC.md)'s reach: that document resolves -[Bundled rules] only, and this one covers every [Identifier] the project can see, the project's -own [Rules] included. The [Toolchain] MUST resolve every [Identifier] its [Blocking] [Rules] -print to [Remediation docs] on disk. One [Identifier] that does not resolve fails this clause, -however many others do. The clause names [Blocking] [Rules] rather than [Defences] because the -failure it catches is a [Rule] that blocks without the documentation that would make it a -[Defence]. The mechanism -is keyed on the -[Identifier] exactly as printed, and it covers the project's own [Rules] and -every [Bundled defence]. Those [Remediation docs] MUST ship +The [Toolchain] MUST resolve every [Identifier] its [Blocking] [Rules] print to +[Remediation docs] on disk. The [Rules] in scope are every [Blocking] [Rule] the [Toolchain] +runs, bundled or the project's own; a [Rule] that only warns is outside this clause. The clause +names [Blocking] [Rules] rather than [Defences] because the failure it catches is a [Rule] that +blocks without the documentation that would make it a [Defence]. One [Identifier] that does not +resolve fails this clause, however many others do. + +The [Toolchain] resolves an [Identifier] by a mechanism of its own, a command, an index file or +a URL scheme, keyed on the [Identifier] exactly as printed. Those [Remediation docs] MUST ship with the project or with the [Toolchain], at a version tracked together. The [Toolchain] MUST also give that documentation a place to live, so that a [Rule author] adding a [Rule] knows where its documentation goes. @@ -180,7 +180,8 @@ only the prohibition, specifically enough to act on, as method clause 8.4 requir This clause closes what the [detector specification](DETECTOR-SPEC.md) leaves open. That document holds a [Detector] to on-disk resolution of its own [Bundled rules] and -no further, because it cannot know a project's documentation. A third-party [Detector]'s +no further, because it cannot know a project's documentation; this clause reaches the project's +own [Rules] as well. A third-party [Detector]'s native catalogue, which the [Toolchain] orchestrates without claiming as its own, is not re-shipped here: an [Identifier] the [Detector] resolves under that document's clause 6 meets this clause, and one that nothing resolves fails it. From f3ec69ebb2b2fc10fc592765ba20848acbdcb00a Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 13:47:43 +0100 Subject: [PATCH 21/52] Routed findings: rows from runs 19 to 30, two closed by defined-terms run 64 acceptance/ROUTED.md gains the passages runs 19 to 30 on the semantic-detectors change named in text that change no longer touches (the Narrowing entry, section 4's lead-in to the owner's decisions, clause 3.1's paragraph on independent techniques and the section 3 table's 3.1 row), notes those runs on the rows they raised again, and closes the method's two-bounds paragraph and bundled-defence paragraph by editor/defined-terms's run 64. Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index f995c21..c6c3d28 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -10,7 +10,7 @@ A finding is open until a change that reworks its passage passes a cohort, when that run and moves below the rule, or until the editor closes it with a reason recorded here. A finding raised again by a later run is noted on its line, not listed twice. The status column holds `open`, or the `editor/` branch or pull request number of the change that is -reworking the passage. The rows raised by runs 31 to 52 were raised under the earlier rules, on +reworking the passage. The rows raised by runs 19 to 52 were raised under the earlier rules, on untouched text of the change those runs tested, and are listed here so that they are not lost. ## Open @@ -25,22 +25,28 @@ untouched text of the change those runs tested, and are listed here so that they | `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | | `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | | `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | run 67 | `open` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 63, 66 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, and the bracketed verb "Suppress" among nouns | runs 41, 47, 50, 56, 64 | `open` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | | `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 56, 59, 64 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 53, 59, 61, 64 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 53, 59, 61, 64 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | run 61 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | +| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | ## Closed A row moves here when a change that reworks its passage passes a cohort, naming that run, or when the editor closes it with a reason. -| Document | Clause | Passage | Raised by | Closed by | -| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ------------------------------------ | -| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule`, run 66 | -| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 39, 42 | `editor/detector-route-rule`, run 66 | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule`, run 66 | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | run 54 | `editor/detector-route-rule`, run 66 | +| Document | Clause | Passage | Raised by | Closed by | +| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------------------------------------ | +| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 20, 22, 24, 30, 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 20, 30, 39, 42 | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | runs 20, 54 | `editor/detector-route-rule`, run 66 | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on the two bounds | "Both bounds measure breadth within one level; the level itself is a separate question, answered against the report." | runs 19, 56 | `editor/defined-terms`, run 64 | +| `SPEC.md` | [3.5](../SPEC.md#35-enforce-permanently-and-block), the paragraph on a bundled defence | "For a Bundled defence, the project whose entry point demonstrates it is any project the Toolchain is genuinely installed into", and its substitutes sentence | runs 25, 29, 53, 59 | `editor/defined-terms`, run 64 | From bc6ae14586c1a392c1808989114ebc71a46b9d27 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:01:43 +0100 Subject: [PATCH 22/52] Method key: clause 3.3's narrowing sentence; routed findings from runs 68 to 71 The method key lists the sentence behind a narrowing, and the owner's decision where it cannot be written, among the obligations, after three readers of run 70 restated it. acceptance/ROUTED.md gains Appendix A's closing paragraph from run 70 and the detector's clause 5.3 from run 69, and notes runs 68, 70 and 71 on the rows they raised again. Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 60 +++++++++++++++++++++------------------- acceptance/method/KEY.md | 2 ++ 2 files changed, 33 insertions(+), 29 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index c6c3d28..eced9c7 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -15,38 +15,40 @@ untouched text of the change those runs tested, and are listed here so that they ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ---------------------------------------- | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | run 67 | `open` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | -| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | ---------------------------------------- | +| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | run 69 | `editor/detector-route-rule` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68 | `open` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | +| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | ## Closed A row moves here when a change that reworks its passage passes a cohort, naming that run, or when the editor closes it with a reason. -| Document | Clause | Passage | Raised by | Closed by | -| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------------------------------------ | -| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 20, 22, 24, 30, 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule`, run 66 | -| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 20, 30, 39, 42 | `editor/detector-route-rule`, run 66 | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule`, run 66 | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | runs 20, 54 | `editor/detector-route-rule`, run 66 | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on the two bounds | "Both bounds measure breadth within one level; the level itself is a separate question, answered against the report." | runs 19, 56 | `editor/defined-terms`, run 64 | -| `SPEC.md` | [3.5](../SPEC.md#35-enforce-permanently-and-block), the paragraph on a bundled defence | "For a Bundled defence, the project whose entry point demonstrates it is any project the Toolchain is genuinely installed into", and its substitutes sentence | runs 25, 29, 53, 59 | `editor/defined-terms`, run 64 | +| Document | Clause | Passage | Raised by | Closed by | +| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------ | +| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 20, 22, 24, 30, 36, 42, 45, 48, 51, 54, and 71 on a branch without the rework | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 20, 30, 39, 42, and 71 on a branch without the rework | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | runs 20, 54 | `editor/detector-route-rule`, run 66 | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on the two bounds | "Both bounds measure breadth within one level; the level itself is a separate question, answered against the report." | runs 19, 56, and 70 on a branch without the rework | `editor/defined-terms`, run 64 | +| `SPEC.md` | [3.5](../SPEC.md#35-enforce-permanently-and-block), the paragraph on a bundled defence | "For a Bundled defence, the project whose entry point demonstrates it is any project the Toolchain is genuinely installed into", and its substitutes sentence | runs 25, 29, 53, 59 | `editor/defined-terms`, run 64 | diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 86cf8ff..e2ffd60 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -23,6 +23,8 @@ no bullet making an assertion listed below as wrong, is a correct restatement. practitioner's own authority (section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom)). - Fix the original defect conventionally, with a test that reproduces it, once the defence is in place (sections [2](../../SPEC.md#2-when-the-method-applies) and [6](../../SPEC.md#6-relationship-to-test-driven-development)). +- Write down, for any code the rule is narrowed to exclude, why the hazard cannot arise there, + and take the exclusion to the owner where that sentence cannot be written ([3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire), Part B). Wrong if it asserts: the practitioner fixes the reported instance first and defends afterwards; a test may serve as the detector; an agent may baseline the remaining instances. A bullet that From c2d722bcd737f72b858b0db690328ae6bc7e6dd0 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:01:46 +0100 Subject: [PATCH 23/52] Run 068: toolchain contradictions cohort, failed on six findings Every keyed item five of five. Six findings in text the change wrote or altered, five in or set against clause 4.2 and one on clause 4.1 against section 1. One finding in untouched text, clause 6.2's paragraph on generic phrases, noted on its row in acceptance/ROUTED.md. Co-Authored-By: Claude Fable 5.1 --- .../068-toolchain-contradictions-run-6.md | 174 ++++++++++++++++++ 1 file changed, 174 insertions(+) create mode 100644 acceptance/runs/068-toolchain-contradictions-run-6.md diff --git a/acceptance/runs/068-toolchain-contradictions-run-6.md b/acceptance/runs/068-toolchain-contradictions-run-6.md new file mode 100644 index 0000000..4091b2e --- /dev/null +++ b/acceptance/runs/068-toolchain-contradictions-run-6.md @@ -0,0 +1,174 @@ +# Run 68: TOOLING-SPEC.md, toolchain contradictions, sixth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: not yet opened at the time of the run; it is to be opened from branch + `editor/toolchain-contradictions`. The readers read commit 0080b8e. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 67 in clause 4.2 (rewritten in three paragraphs), section 2's table (a + Remediation docs row) and section 1's "How to read this document". +- **Run within the pull request**: 6 (runs 57, 58, 62, 65 and 67 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `dbce8540d4259779cdced1129c5fa71306bbe31597700618177c4a5f48662919`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at 0080b8e, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Partial (1), Yes (2, 3, 4, 5); +6.1 Yes (1, 2, 4, 5), Partial (3), the Partial mentioning that no location is given for +calibrations or conventions; 6.2 No (all five), each mentioning that nothing rejects generic +text or a generic justification. Readers 1, 2, 4 and 5 note that the fixture does not say the +two undocumented rules are Blocking and grade 4.2 No on that assumption or inference, which the +key accepts with no condition. + +Question 1: correct bullets per reader 6, 5, 6, 5 and 4 (reader 1: the governing principle; +4.1; 4.2; 4.3; 4.4 with 4.5; 5.1 to 5.3 with 6.1 to 6.3. Reader 2: 4.1; 4.2; 4.3; 4.4 with 4.5; +5.1 to 5.3 with 6.1 to 6.3. Reader 3: 4.1; 4.2; 4.3; 4.4 with 4.5; 5.1 to 5.3; 6.1 to 6.3. +Reader 4: 4.1 with 4.3; 4.2; 4.5; 5.1 to 5.3 with 6.3; 6.1 and 6.2. Reader 5: 4.1 with 4.2; 4.3; +4.4 with 4.5; 5.1 to 5.3 with 6.1 to 6.3). Reader 4's fourth and fifth bullets name different +clauses of one line of the key's list and each count. No bullet asserts anything the key marks +wrong. The first bullets of readers 2 and 5 (the toolchain judged as one assembled whole) state +how conformance is measured; reader 2's sixth calls the agent summary recommended; reader 4's +sixth (a declaration permitted, a project grade and a separate artefact grade) and reader 5's +sixth (one project grade, a shipped artefact graded separately, section 7 and the declaration +SHOULD or MAY, not conditions) are of the kinds the key counts neither way. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through", which states element (2); for 2b all five quote section 9's "The project-level +verdict is a single grade; ..."; for 2c all five quote clause 9.1's "The two verdicts MUST be +graded and declared separately, and neither implies the other." No answer gives a grade per +part or a single grade for 2c. Reader 1's 2c says "Two grades, if the company's project actually +ships the package"; see the points the rules did not settle. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says the toolchain does not conform +and that there is no agent summary (all five); S8 says Zigpipe ships no tooling (all five, +readers 1, 2 and 3 also placing zig-qa's grade on its own artefact); Decl says the declaration +is optional and the verdict rests on the exercise (all five). + +### Misses + +None. No answer matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..0080b8e adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | ------------------- | +| 1 | Clause 4.2, second paragraph: "The [Toolchain] resolves an [Identifier] by a mechanism of its own, a command, an index file or a URL scheme, keyed on the [Identifier] exactly as printed" and "MUST ship with the project or with the [Toolchain]", a URL scheme set against the heading (1, 5); "MUST ship with the project or with the [Toolchain]" set against the third paragraph (2) | 1, 2, 5 | yes | +| 2 | Clause 4.2 heading: "from the installed copy, without network access", set against the second paragraph's URL scheme | 1, 5 | yes | +| 3 | Clause 4.2, first paragraph: "names [Blocking] [Rules] rather than [Defences] because the failure it catches is a [Rule] that blocks without the documentation that would make it a [Defence]", set against the section 2 Defence row and clause 5.1 | 3, 5 | yes | +| 4 | Section 2 terminology table, [Defence] row: "A [Rule] with its documentation, in force and [Blocking]", set against clause 4.2's first paragraph (3, 5) and clause 5.1 (3, 4, 5) | 3, 4, 5 | no | +| 5 | Clause 5.1, first paragraph: "every [Defence] active in the project"; whether the two undocumented rules, which `zig-qa list` shows, are within it; set against the Defence row (3, 4, 5) and clause 4.2's first paragraph (3, 5) | 3, 4, 5 | no | +| 6 | Clause 4.1, first paragraph: "That document's MUSTs are those of its sections 4 to 7, and this clause holds the [Detector] to every one of them. ... they are a summary, not the test" (3); described as the clause taking its MUSTs from sections 4 to 7 of a document not supplied, graded from the summary (2) | 2, 3 | yes | +| 7 | Clause 6.2, fourth paragraph: "'needed for now', 'legacy', 'TODO' and their like", where the open phrase stops (2); "could be pasted onto any Exception unchanged, by a check it documents", set against the heading (5) | 2, 5 | no | +| 8 | Clause 4.2, third paragraph: "is not re-shipped here", set against the second paragraph | 2 | yes | +| 9 | Clause 6.1, first paragraph: "The [Toolchain] MUST define where the [Project record] lives and MUST load it", set against the Project record entry | 3 | no | +| 10 | Section 2 terminology, [Project record] entry: "[Calibrations], [Exceptions] and conventions", set against clause 6.1 | 3 | no | +| 11 | Clause 6.2 heading: "names the hazard and the scope", set against the fourth paragraph | 5 | no | +| 12 | Clause 4.3, second paragraph: "This clause is where the project's decision is made and enforced, and the decision is no", set against the Why paragraph | 5 | no | +| 13 | Clause 4.3, Why paragraph: suppression "an Owner decision", set against the second paragraph | 5 | no | +| 14 | Clause 4.1, point 2, "Behind the table" paragraph: its reference to the detector specification's "clause 8.1", that document not supplied | 5 | yes | + +Notes, not passages: readers 2, 3, 4 and 5 remark under question 3 that the detector +specification was not supplied; none of those remarks is under question 4. The section 2 table +was realigned by the diff for the new Remediation docs row; the Defence row's words, terms and +links are unchanged, so passage 4 is marked unchanged. + +## Findings and where they lie + +- **Finding A, passage 1, clause 4.2 second paragraph** (readers 1, 2 and 5): the diff writes + this paragraph's first sentence and rewraps the "MUST ship" sentence into it. Text the change + wrote or altered. Blocks. +- **Finding B, passage 2, clause 4.2 heading** (readers 1 and 5): the diff changes "a defence" + to "a blocking rule" in it. Text the change wrote or altered. Blocks. +- **Finding C, passage 3, clause 4.2 first paragraph** (readers 3 and 5): written by the diff. + Text the change wrote or altered. Blocks. +- **Finding D, passage 4, section 2 Defence row** (readers 3, 4 and 5): unchanged, but readers + 3 and 5 set it against clause 4.2's first paragraph, which the diff wrote. Text the change + wrote or altered, under the Marking section's rule on a paragraph a reader sets against one. + Blocks. +- **Finding E, passage 5, clause 5.1 first paragraph** (readers 3, 4 and 5): unchanged, but + readers 3 and 5 set it against clause 4.2's first paragraph. Text the change wrote or altered, + as for finding D. Blocks. +- **Finding F, passage 6, clause 4.1 first paragraph** (readers 2 and 3, counted by the + open-point rule, point 1 below): the diff rewrites the paragraph, adding "That document's + MUSTs are those of its sections 4 to 7" and "they are a summary, not the test". Text the + change wrote or altered. Blocks. +- **Finding G, passage 7, clause 6.2 fourth paragraph** (readers 2 and 5): unchanged by the + diff, and set only against clause 6.2's heading, also unchanged. Elsewhere: to be recorded in + `acceptance/ROUTED.md`, on the open clause 6.2 line run 67 raised, and routed; does not block. + +Passages 8 to 14 are each named by one reader: recorded, not findings. + +## Points the rules did not settle + +1. **Remarks on another document not supplied, given as passages.** Readers 2 and 3 name clause + 4.1's first paragraph under question 4, reader 3 by quotation and reader 2 by clause number + and a description identifying its "sections 4 to 7" sentence, but each turns in part on the + detector specification not being supplied, which the Marking section calls a note. Reader 5's + item on clause 4.1 point 2 is of the same kind. Marked by the open-point rule: counted as + passages. Finding F rests on readers 2 and 3 both being counted; findings A to E, and so the + verdict, do not. +2. **Reader 1's 2c answer, "Two grades, if the company's project actually ships the package".** + The Marking section gives a rule for a grade on a stated condition only under question 3. The + question's company publishes the package, so read for its sense the answer states two for the + case asked about, and its quotation of clause 9.1 states element (2). Marked for the reader + under the open-point rule: correct. No other mark turns on it. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit f3ec69e, named before the readers started. The process files at +0080b8e are identical to those at f3ec69e. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: six (findings A + to F). Not met. +- Passages named by two or more readers elsewhere: one (finding G), for `acceptance/ROUTED.md`; + it does not block. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A to F; every keyed item passed. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From d272e23614378c3f93b2d6259fdbadd3514e06a5 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:01:46 +0100 Subject: [PATCH 24/52] Toolchain 4.2 says only what it requires; 4.1 and section 1 agree on restating After run 68: clause 4.2 states its obligation on the project's own rules and bundled rules alike, with no sentence on defences, no list of mechanisms and no third-party catalogue "not re-shipped", and its remediation docs may ship with the detector whose rule they document; clause 4.1 and section 1 both say a clause that rests on the detector specification restates it far enough to grade what a toolchain's evidence shows. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 11 ++++++++++- TOOLING-SPEC.md | 37 +++++++++++++++++-------------------- 2 files changed, 27 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bb27cd2..14ba863 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -143,7 +143,16 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be obligation first, then what is in scope, then the mechanism, with Remediation docs glossed in section [2](TOOLING-SPEC.md#2-terminology)'s table; and section [1](TOOLING-SPEC.md#1-what-this-document-is-for) says that a toolchain can be graded from this document alone because each clause that rests on the detector specification says in its own words what - that document requires. Accepted by run NNN. + that document requires. [Run 68](acceptance/runs/068-toolchain-contradictions-run-6.md) failed + on six findings in that text: readers set a URL scheme against "without network access", "MUST + ship with the project or with the Toolchain" against a third-party catalogue "not re-shipped + here", the sentence on why the clause names blocking rules against the Defence row and clause + [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) yet again, and clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification)'s "a summary, not the test" against section [1](TOOLING-SPEC.md#1-what-this-document-is-for). Clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) now + says only what it requires, of the project's own rules and bundled rules alike, with no + sentence on defences and no list of mechanisms, and its remediation docs may ship with the + detector whose rule they document; clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) and section [1](TOOLING-SPEC.md#1-what-this-document-is-for) both say the clause restates + the detector specification far enough to grade what a toolchain's evidence shows. Accepted by + run NNN. ### 0.2.0, 2026-09-08 diff --git a/TOOLING-SPEC.md b/TOOLING-SPEC.md index b9a5350..714d0da 100644 --- a/TOOLING-SPEC.md +++ b/TOOLING-SPEC.md @@ -41,7 +41,8 @@ part of its [Toolchain] and are judged as such. **How to read this document.** A [Toolchain] can be graded from this document alone. Where a clause rests on the [detector specification](DETECTOR-SPEC.md), as clauses 4.1, 4.2 and 4.4 do, -the clause says in its own words what that document requires of the [Detector]. The +it restates what that document requires of the [Detector], far enough to grade what a +[Toolchain]'s evidence shows. The table in section 2 glosses every term it borrows. Each clause states its obligation in its heading and first sentence, and the **Why** paragraph beneath is reasoning, not a further requirement. A link is there for depth and is not a prerequisite for the sentence that carries it. @@ -113,14 +114,14 @@ any row and the failure mode is visible too: either half alone leaves the [Pract A [Toolchain] MUST NOT route a [Defence] through a [Detector] that does not [Conform] to [the detector specification](DETECTOR-SPEC.md). That document's MUSTs are those of its sections 4 to 7, and this clause holds the [Detector] to every one of them. The sentences -that follow summarise the ones most often failed, so that a reader with this document alone can -see what the clause mostly turns on; they are a summary, not the test. The +that follow restate the ones most often failed, far enough for a reader with this document +alone to grade what a [Toolchain]'s evidence shows. The [Detector] hosts bespoke [Rules]. It runs one [Rule] on its own against supplied code. It prints a stable [Identifier] with every finding. It runs locally, down to one file, with the result in the command's own output and nothing reportable only through a hosted service. It resolves every [Identifier] a [Bundled rule] can print to documentation shipped with it, offline. It makes any inline [Suppression] route detectable or disableable. A [Detector] that fails any -one of that document's MUSTs, summarised here or not, fails this clause after wrapping. Three +one of that document's MUSTs, restated here or not, fails this clause after wrapping. Three points govern how that is met. 1. **Wrapping is permitted.** The [Toolchain] MAY supply a mechanism the @@ -163,28 +164,24 @@ clause 3.2. A gap recorded is not a gap closed. ### 4.2 The toolchain MUST resolve every identifier a blocking rule it routes can print, from the installed copy, without network access The [Toolchain] MUST resolve every [Identifier] its [Blocking] [Rules] print to -[Remediation docs] on disk. The [Rules] in scope are every [Blocking] [Rule] the [Toolchain] -runs, bundled or the project's own; a [Rule] that only warns is outside this clause. The clause -names [Blocking] [Rules] rather than [Defences] because the failure it catches is a [Rule] that -blocks without the documentation that would make it a [Defence]. One [Identifier] that does not -resolve fails this clause, however many others do. - -The [Toolchain] resolves an [Identifier] by a mechanism of its own, a command, an index file or -a URL scheme, keyed on the [Identifier] exactly as printed. Those [Remediation docs] MUST ship -with the project or with the [Toolchain], at a version tracked together. The +[Remediation docs] on disk. That is every [Blocking] [Rule] the [Toolchain] runs, the project's +own and the bundled alike. One [Identifier] that does not resolve fails this clause, however +many others do. + +The resolving mechanism is keyed on the [Identifier] exactly as printed. Those +[Remediation docs] MUST ship with the project, with the [Toolchain] or with the [Detector] +whose [Rule] they document, at a version tracked together. The [Toolchain] MUST also give that documentation a place to live, so that a [Rule author] adding a [Rule] knows where its documentation goes. The documentation an [Identifier] resolves to MUST state the correct construction, not only the prohibition, specifically enough to act on, as method clause 8.4 requires. The [Toolchain] supplies the place and the [Rule author] the text. -This clause closes what the [detector specification](DETECTOR-SPEC.md) leaves open. That document holds a -[Detector] to on-disk resolution of its own [Bundled rules] and -no further, because it cannot know a project's documentation; this clause reaches the project's -own [Rules] as well. A third-party [Detector]'s -native catalogue, which the [Toolchain] orchestrates without claiming as its own, is not -re-shipped here: an [Identifier] the [Detector] resolves under that document's clause 6 meets -this clause, and one that nothing resolves fails it. +This clause closes what the [detector specification](DETECTOR-SPEC.md) leaves open: that +document holds a [Detector] to on-disk resolution of its own [Bundled rules] and no further, +because it cannot know a project's documentation, and this clause reaches the project's own +[Rules] as well. An [Identifier] a third-party [Detector] resolves under that document's clause +6 meets this clause, and one that nothing resolves fails it. **Why**: method specification clauses 3.6 and 8.3. The [Identifier] has to resolve for the reader, which for an [Agent] means mechanically and on disk, and the project's own From 2024b8a4c4ead2b0ff0bfcf6c7852541967952a5 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:02:05 +0100 Subject: [PATCH 25/52] Routed list taken from the process branch, clause 4.2 linked by its section --- acceptance/ROUTED.md | 60 +++++++++++++++++++++++--------------------- 1 file changed, 31 insertions(+), 29 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 3226aae..eced9c7 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -15,38 +15,40 @@ untouched text of the change those runs tested, and are listed here so that they ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ---------------------------------------- | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | run 67 | `open` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | run 50 | `open`; a method question for the editor | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | -| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | ---------------------------------------- | +| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | run 69 | `editor/detector-route-rule` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68 | `open` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | +| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | ## Closed A row moves here when a change that reworks its passage passes a cohort, naming that run, or when the editor closes it with a reason. -| Document | Clause | Passage | Raised by | Closed by | -| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------------------------------------ | -| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 20, 22, 24, 30, 36, 42, 45, 48, 51, 54 | `editor/detector-route-rule`, run 66 | -| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 20, 30, 39, 42 | `editor/detector-route-rule`, run 66 | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule`, run 66 | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | runs 20, 54 | `editor/detector-route-rule`, run 66 | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on the two bounds | "Both bounds measure breadth within one level; the level itself is a separate question, answered against the report." | runs 19, 56 | `editor/defined-terms`, run 64 | -| `SPEC.md` | [3.5](../SPEC.md#35-enforce-permanently-and-block), the paragraph on a bundled defence | "For a Bundled defence, the project whose entry point demonstrates it is any project the Toolchain is genuinely installed into", and its substitutes sentence | runs 25, 29, 53, 59 | `editor/defined-terms`, run 64 | +| Document | Clause | Passage | Raised by | Closed by | +| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------ | +| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 20, 22, 24, 30, 36, 42, 45, 48, 51, 54, and 71 on a branch without the rework | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 20, 30, 39, 42, and 71 on a branch without the rework | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | runs 20, 54 | `editor/detector-route-rule`, run 66 | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on the two bounds | "Both bounds measure breadth within one level; the level itself is a separate question, answered against the report." | runs 19, 56, and 70 on a branch without the rework | `editor/defined-terms`, run 64 | +| `SPEC.md` | [3.5](../SPEC.md#35-enforce-permanently-and-block), the paragraph on a bundled defence | "For a Bundled defence, the project whose entry point demonstrates it is any project the Toolchain is genuinely installed into", and its substitutes sentence | runs 25, 29, 53, 59 | `editor/defined-terms`, run 64 | From 13b497ba3ddc0eba771e3c516bc89e4ffe14cb2d Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:18:48 +0100 Subject: [PATCH 26/52] Toolchain key: the 4.3 row says which case the fixture presents; ROUTED 5.3 row adds run 75 Run 72's open point: three readers graded clause 4.3 on the condition that the ignore comment is Lintomatic's only suppression route, and the key did not say which case the fixture presents. The 4.3 row now says so, and such a grade is the grade for this case. Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 2 +- acceptance/toolchain/KEY.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index eced9c7..68a0be9 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -18,7 +18,7 @@ untouched text of the change those runs tested, and are listed here so that they | Document | Clause | Passage | Raised by | Status | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | ---------------------------------------- | | `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | run 69 | `editor/detector-route-rule` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | | `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | | `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | | `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index aadab7a..e098c90 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -63,7 +63,7 @@ Wrong readings the text invites, which count against the document on one reader: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | None | Lintomatic fails detector [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) on two undocumented bundled rules and nothing in the wrapping closes that gap; a reader who notes that the detector specification was not supplied and grades from this document's summary of it is correct | | [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | None. A No given on the condition that the two bundled rules are blocking is a No: their firing stops `zig-qa run`, so they are | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)` | -| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | None | Ignore comments disabled project-wide; exceptions only through the record the toolchain reads | +| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | None | The fixture names one suppression route of Lintomatic's, the inline ignore comment, disabled project-wide, with exceptions only through the record the toolchain reads; a grade given on the condition that the comment is Lintomatic's only route is the grade for this case | | [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | None | `zig-qa run` is local, prints Lintomatic's output unchanged, one line per finding | | [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | None | Lintomatic, then the tests, stopping on a Lintomatic failure | | [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes, Partial or No | None | `zig-qa list` prints identifier, summary and documentation path without running the rules; Partial or No is the reading that counts the two `(none)` routes, since [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) says the listing MUST provide the route and [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) point 2 says a gap may be named again by a later clause | From 7d7b18dba4530c1e6a70b600f93af0568f80e60d Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:19:47 +0100 Subject: [PATCH 27/52] Run 072: toolchain contradictions cohort, failed on four findings Every keyed item five of five. Four findings in text the change wrote or altered or set against it: the Defence row and clause 5.1 against clause 4.2, by four readers each, and section 1's "graded from this document alone" against clause 4.1, by two. Nothing routed. Co-Authored-By: Claude Fable 5.1 --- .../072-toolchain-contradictions-run-7.md | 167 ++++++++++++++++++ 1 file changed, 167 insertions(+) create mode 100644 acceptance/runs/072-toolchain-contradictions-run-7.md diff --git a/acceptance/runs/072-toolchain-contradictions-run-7.md b/acceptance/runs/072-toolchain-contradictions-run-7.md new file mode 100644 index 0000000..8df3d61 --- /dev/null +++ b/acceptance/runs/072-toolchain-contradictions-run-7.md @@ -0,0 +1,167 @@ +# Run 72: TOOLING-SPEC.md, toolchain contradictions, seventh run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: not yet opened at the time of the run; it is to be opened from branch + `editor/toolchain-contradictions`. The readers read commit cb9b4e5. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 68 in clause 4.2 (which now states only what it requires, with no sentence on + defences and no list of mechanisms), clause 4.1's first paragraph and section 1's "How to read + this document". +- **Run within the pull request**: 7 (runs 57, 58, 62, 65, 67 and 68 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `6c9683df16a39ddd12169c438fe1f9c8a32050bd5346a81de4edb1c0d6990472`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at cb9b4e5, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`, which + the marker confirmed byte for byte. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Partial (1, 2), Yes (3, 4, 5); +6.1 Partial (1, 5), Yes (2, 3, 4), each Partial mentioning that the fixture gives no location for +calibrations or conventions; 6.2 No (all five), each mentioning that nothing rejects generic +text or a generic justification. Readers 1, 2, 3 and 4 note that the fixture does not say the +two undocumented rules are Blocking and grade 4.2 No on that assumption or inference, which the +key reads as a No. Readers 1, 4 and 5 grade 4.3 Yes on what the fixture writes, reader 1 by +assuming Lintomatic has no other route; see the points the rules did not settle. + +Question 1: correct bullets per reader 6, 5, 5, 6 and 5 (reader 1: 4.1; 4.2; 4.3; 4.5 with 4.4; +5.1 to 5.3 with 6.1 to 6.3; 8.1 and 8.2. Readers 2, 3 and 5: 4.1; 4.2; 4.3 with 4.5; 5.1 to 5.3; +6.1 to 6.3. Reader 4: the governing principle; 4.1; 4.2; 4.3 with 4.5 and 4.4; 5.1 to 5.3; 6.1 +to 6.3). No bullet asserts anything the key marks wrong. The first bullets of readers 2, 3 and 5 +(the toolchain judged as one assembled whole) and the first sentence of reader 1's sixth (the +project level and a shipped artefact graded separately) state how conformance is measured, which +the key counts neither way. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through", which states element (2); for 2b all five quote section 9's "The project-level +verdict is a single grade; ..."; for 2c all five quote clause 9.1's "The two verdicts MUST be +graded and declared separately, and neither implies the other." No answer gives a grade per +part or a single grade for 2c. Readers 2 and 4 add to 2b that Zigpipe, shipping nothing, has +only the one grade, which the key does not mark wrong. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says the toolchain does not conform +and that there is no agent summary (all five); S8 says Zigpipe ships no tooling (all five, +readers 1, 2, 4 and 5 also placing zig-qa's grade on its own artefact); Decl says the +declaration is optional and the verdict rests on the exercise (all five). + +### Misses + +None. No answer matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..cb9b4e5 adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------------------- | +| 1 | Section 2 terminology table, [Defence] row: "A [Rule] with its documentation, in force and [Blocking]", whether the two undocumented rules are Defences; set against clause 4.2's first paragraph (1) and clause 5.1 (1, 2, 3, 4) | 1, 2, 3, 4 | no | +| 2 | Clause 5.1, first paragraph: "every [Defence] active in the project" (1, 3, 4), "MUST provide the route to each one's full documentation" (2); set against the Defence row (1, 2, 3, 4) and, by reader 1, against clause 4.2's first paragraph, asking which of 4.2 and 5.1 governs the undocumented rules | 1, 2, 3, 4 | no | +| 3 | Section 1, "How to read this document": "A [Toolchain] can be graded from this document alone", set against clause 4.1's first paragraph | 2, 3 | yes | +| 4 | Clause 4.1, first paragraph: "that fails any one of that document's MUSTs, restated here or not, fails this clause" (2, 3) and "MUSTs are those of its sections 4 to 7" (2), set against section 1's "graded from this document alone" | 2, 3 | yes | +| 5 | Clause 4.2, first paragraph: "every [Identifier] its [Blocking] [Rules] print", set against the Defence row and clause 5.1 | 1 | yes | +| 6 | Clause 9.2, first paragraph: "A project or a shipped artefact MAY declare" against "A project that ships an artefact carries both levels of clause 9.1 in that declaration", read as unconditional | 1 | no | +| 7 | Clause 6.2, first paragraph: "MUST NOT supply a default", set against the clause 6.4 heading | 3 | no | +| 8 | Clause 6.4 heading: "SHOULD state its own defaults for anything the method leaves to the project", set against clause 6.2's first paragraph | 3 | no | +| 9 | Clause 6.2, fourth paragraph: "The justification MUST name the [Hazard] being accepted and the scope of the [Exception]. The [Toolchain] MUST reject a justification that could be pasted onto any [Exception] unchanged, by a check it documents", whether the check must test hazard and scope or only generic text | 4 | no | +| 10 | Clause 5.3, second paragraph: "Where such a [Defence] is not expressible in the [Toolchain]'s own [Detectors], its entry MAY be sourced from wherever it is enabled, provided the listing stays derived under clause 5.2 rather than hand-maintained", what "wherever it is enabled" covers | 5 | no | + +Notes, not passages: readers 2, 3 and 5 remark under question 3 that the detector specification +was not supplied; none of those remarks is under question 4. The section 2 table was realigned +by the diff for the Remediation docs row; the Defence row's words, terms and links are +unchanged, so passage 1 is marked unchanged. + +## Findings and where they lie + +- **Finding A, passage 1, section 2 Defence row** (readers 1, 2, 3 and 4): unchanged, but + reader 1 sets it against clause 4.2's first paragraph, which the diff rewrote. Text the change + wrote or altered, under the Marking section's rule on a paragraph a reader sets against one. + Blocks. +- **Finding B, passage 2, clause 5.1 first paragraph** (readers 1, 2, 3 and 4): unchanged, but + reader 1 sets it against clause 4.2's first paragraph, asking which of the two governs the + undocumented rules. Text the change wrote or altered, as for finding A. Blocks. +- **Finding C, passage 3, section 1 "How to read this document"** (readers 2 and 3): the diff + rewrites the paragraph's opening, adding "A [Toolchain] can be graded from this document + alone". Text the change wrote or altered. Blocks. +- **Finding D, passage 4, clause 4.1 first paragraph** (readers 2 and 3): the diff rewrites the + paragraph, adding "That document's MUSTs are those of its sections 4 to 7" and "restated here + or not". Text the change wrote or altered. Blocks. + +Passages 5 to 10 are each named by one reader: recorded, not findings. Passages 7 and 9 both lie +in clause 6.2 but in different paragraphs, so they are not the same passage. No finding lies +elsewhere, so nothing from this run goes to `acceptance/ROUTED.md`. + +## Points the rules did not settle + +1. **Clause 4.3 graded Yes on a stated assumption.** Reader 1 grades 4.3 Yes and says the + fixture does not say whether Lintomatic offers any other suppression route, "so this verdict + assumes it does not"; readers 4 and 5 say their Yes holds "only for what is written". The + Marking section reads a grade given on a stated condition as the grade for the case the + fixture presents only where the key says which case that is, and the key's 4.3 row does not + say whether Lintomatic's only route is the ignore comment. Marked for the reader under the + open-point rule: correct. Had the three been marked unanswered, 4.3 would have failed the + all-but-one threshold; the verdict below does not turn on it, since the run fails on the + findings. For the editor to settle in the key before the next run of this pull request. +2. **Readers 2 and 3 on section 1 against clause 4.1.** Each turns on grading 4.1 without the + detector specification, which the Marking section calls a note when the remark is not about a + passage. Both quote section 1's and clause 4.1's sentences and ask how they fit together, + which is a remark about those passages, so the marker counts them as passages; were that in + doubt, the open-point rule counts the passage. Findings C and D rest on it; findings A and B, + and so the verdict, do not. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit bc6ae14, named before the readers started. The process files at +cb9b4e5 (`ACCEPTANCE.md`, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md`, `tools/changes.py`, `spec-qa.py` and the CI workflow) are identical to those +at bc6ae14. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: four (findings A + to D). Not met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A to D; every keyed item passed. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From 43d720f586dba5c47b0b0bc25aca1afc1e62ae91 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:19:47 +0100 Subject: [PATCH 28/52] Toolchain: an undocumented blocking rule named under section 2's table; 5.1 lists it After run 72: section 2 says, under its table, that a blocking rule whose documentation is missing is not yet a defence and is what clauses 4.2, 5.1 and 8.1 exist to catch; clause 5.1 says the listing covers every blocking rule, documented or not, with a missing route showing; section 1 and clause 4.1 both say a verdict from this document alone rests on what is restated and a grader with the detector specification holds the detector to all of it. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 12 ++++++++++-- TOOLING-SPEC.md | 22 ++++++++++++++-------- 2 files changed, 24 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 14ba863..c66c189 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -151,8 +151,16 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be says only what it requires, of the project's own rules and bundled rules alike, with no sentence on defences and no list of mechanisms, and its remediation docs may ship with the detector whose rule they document; clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) and section [1](TOOLING-SPEC.md#1-what-this-document-is-for) both say the clause restates - the detector specification far enough to grade what a toolchain's evidence shows. Accepted by - run NNN. + the detector specification far enough to grade what a toolchain's evidence shows. + [Run 72](acceptance/runs/072-toolchain-contradictions-run-7.md) failed on four findings: four + readers set the Defence row and clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) against clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) once more, asking whether an + undocumented blocking rule is a defence and is listed, and two set section [1](TOOLING-SPEC.md#1-what-this-document-is-for)'s "graded + from this document alone" against clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification)'s "restated here or not". Section [2](TOOLING-SPEC.md#2-terminology) now says, + under its table, that a blocking rule whose documentation is missing is not yet a defence and + is what clauses [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access), [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) and [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) exist to catch; clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) says the listing covers every + blocking rule, documented or not, with a missing route showing; and section [1](TOOLING-SPEC.md#1-what-this-document-is-for) and clause + [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) both say a verdict from this document alone rests on what is restated and a grader with + the detector specification holds the detector to all of it. Accepted by run NNN. ### 0.2.0, 2026-09-08 diff --git a/TOOLING-SPEC.md b/TOOLING-SPEC.md index 714d0da..1b638d7 100644 --- a/TOOLING-SPEC.md +++ b/TOOLING-SPEC.md @@ -39,10 +39,10 @@ building the mechanism first. A [Toolchain] that leaves the project to construct moved the obligation rather than met it. Where the project builds the means itself, those scripts are part of its [Toolchain] and are judged as such. -**How to read this document.** A [Toolchain] can be graded from this document alone. Where a -clause rests on the [detector specification](DETECTOR-SPEC.md), as clauses 4.1, 4.2 and 4.4 do, -it restates what that document requires of the [Detector], far enough to grade what a -[Toolchain]'s evidence shows. The +**How to read this document.** A [Toolchain] can be graded from this document alone, on the +evidence a project shows. Where a clause rests on the [detector specification](DETECTOR-SPEC.md), +as clauses 4.1, 4.2 and 4.4 do, it restates what that document requires of the [Detector], and a +grader with that document to hand holds the [Detector] to all of it. The table in section 2 glosses every term it borrows. Each clause states its obligation in its heading and first sentence, and the **Why** paragraph beneath is reasoning, not a further requirement. A link is there for depth and is not a prerequisite for the sentence that carries it. @@ -67,6 +67,10 @@ capitalised term links to its definition; the ones this document leans on most a | [Bundled rule] | A [Rule] a [Detector] ships rather than one the project wrote | | [Harness] | The route by which one [Rule] is run against supplied code on its own | +A [Blocking] [Rule] whose documentation is missing is not yet a [Defence]. It is what clauses +4.2, 5.1 and 8.1 exist to catch, so where those clauses reach a [Toolchain]'s [Rules] they reach +every [Blocking] [Rule] it runs, documented or not. + These are additional. #### Consuming project @@ -114,15 +118,15 @@ any row and the failure mode is visible too: either half alone leaves the [Pract A [Toolchain] MUST NOT route a [Defence] through a [Detector] that does not [Conform] to [the detector specification](DETECTOR-SPEC.md). That document's MUSTs are those of its sections 4 to 7, and this clause holds the [Detector] to every one of them. The sentences -that follow restate the ones most often failed, far enough for a reader with this document -alone to grade what a [Toolchain]'s evidence shows. The +that follow restate the ones most often failed, and a verdict from this document alone rests on +them; a grader with that document to hand holds the [Detector] to every one of its MUSTs, +restated here or not. The [Detector] hosts bespoke [Rules]. It runs one [Rule] on its own against supplied code. It prints a stable [Identifier] with every finding. It runs locally, down to one file, with the result in the command's own output and nothing reportable only through a hosted service. It resolves every [Identifier] a [Bundled rule] can print to documentation shipped with it, offline. It makes any inline [Suppression] route detectable or disableable. A [Detector] that fails any -one of that document's MUSTs, restated here or not, fails this clause after wrapping. Three -points govern how that is met. +one of them, after wrapping, fails this clause. Three points govern how that is met. 1. **Wrapping is permitted.** The [Toolchain] MAY supply a mechanism the [Detector] lacks by wrapping it, with a [Harness] script or a @@ -258,6 +262,8 @@ kind of property that lives in the [Toolchain] and nowhere else. The [Toolchain] MUST list every [Defence] active in the project without running it. The listing MUST include each [Defence]'s [Identifier] and a terse statement of what it forbids or requires. It MUST provide the route to each one's full documentation. +The listing covers every [Blocking] [Rule] the [Toolchain] runs, documented or not, and a +[Rule] whose route to documentation is missing is listed with that gap showing. **Why**: method specification clause 8.5. An [Agent] arriving at a codebase has no colleague to ask. Without a listing, a project's standards can only be learned by violating them one at a time. From 1a757d9105f5a0c62d0770463d1b7a37a0ea9414 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:33:47 +0100 Subject: [PATCH 29/52] Run 076: toolchain contradictions cohort, failed on clause 4.1 and section 2 Every keyed item five of five. Two findings in text the change wrote or altered: clause 4.1's first paragraph, all five readers unable to tell which test governs without the detector specification, and section 2's "These are additional.", parted from its table by the inserted paragraph. Clause 6.1's Project record sentence routed to acceptance/ROUTED.md. Co-Authored-By: Claude Fable 5.1 --- .../076-toolchain-contradictions-run-8.md | 167 ++++++++++++++++++ 1 file changed, 167 insertions(+) create mode 100644 acceptance/runs/076-toolchain-contradictions-run-8.md diff --git a/acceptance/runs/076-toolchain-contradictions-run-8.md b/acceptance/runs/076-toolchain-contradictions-run-8.md new file mode 100644 index 0000000..4b32b4c --- /dev/null +++ b/acceptance/runs/076-toolchain-contradictions-run-8.md @@ -0,0 +1,167 @@ +# Run 76: TOOLING-SPEC.md, toolchain contradictions, eighth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: not yet opened at the time of the run; it is to be opened from branch + `editor/toolchain-contradictions`. The readers read commit fa79f8a. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 72: section 2 gains a paragraph under its table saying a blocking rule whose + documentation is missing is not yet a defence and is what clauses 4.2, 5.1 and 8.1 exist to + catch; clause 5.1 says the listing covers every blocking rule, documented or not, with a + missing route showing; section 1's "How to read this document" and clause 4.1's first + paragraph both say a verdict from this document alone rests on what is restated and a grader + with the detector specification holds the detector to all of it. The key's 4.3 row now says + which case the fixture presents. +- **Run within the pull request**: 8 (runs 57, 58, 62, 65, 67, 68 and 72 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `18f9a24ca270450e83434b252e766ca54bcc6fa744f855bcc76bc0c59613fe3a`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at fa79f8a, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`, which + the marker confirmed byte for byte. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (all five); 6.1 Yes (all +five); 6.2 No (all five), each mentioning that nothing rejects generic text. Readers 1, 3, 4 and +5 note that the fixture does not say the two undocumented rules are Blocking and grade 4.2 No on +that assumption, which the key's 4.2 row reads as a No. Readers 1, 2, 3, 4 and 5 grade 4.3 Yes +on the suppression route the fixture describes, several saying the fixture is silent on other +routes; the key's 4.3 row says the fixture presents one route, the ignore comment, so under the +Marking section a grade on that condition is the grade for this case. + +Question 1: correct clauses named per reader, distinct within the key's list: reader 1, the +governing principle; 4.1; 4.2; 4.3, 4.4 and 4.5; 5.1 to 5.3 with 6.1 to 6.3; section 8 as +shipped toolchains' self-audit duties. Readers 2 and 3: 4.1; 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 +with 6.1 to 6.3. Reader 4: 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3; 6.1 to 6.3. Reader 5: 4.1 and +4.2; 4.3 with 6.2; 4.4 and 4.5; 5.1 to 5.3 with 6.1 and 6.3. No bullet asserts anything the key +marks wrong. Every reader's first bullet (the toolchain judged as one assembled whole), reader +1's two levels of grading, reader 5's single project-level grade, and the remarks that section 7 +is recommended or a SHOULD (readers 1, 2, 3, 5) and that section 8 applies only to shipped +artefacts (reader 5) are counted neither way, as the key says. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through"; for 2b all five quote section 9's "The project-level verdict is a single grade; ..."; +for 2c all five quote clause 9.1's "The two verdicts MUST be graded and declared separately, and +neither implies the other." No answer gives a grade per part or a single grade for 2c. Reader 2 +adds to 2b that a second grade exists only where the project ships something, and reader 5 that +a failure in any part a clause touches fails that clause for the whole; the key marks neither +wrong. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says there is no agent summary (all +five), readers 3, 4 and 5 adding that the toolchain does not conform at all; S8 says Zigpipe +ships no tooling (all five), readers 2, 3, 4 and 5 placing section 8 on zig-qa's own artefact +grade; Decl says the declaration is optional and the verdict rests on the exercise (all five). + +### Misses + +None. No answer matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..fa79f8a (9e914fe +being the merge base with `next`) adds, rewords, moves or deletes a sentence in the paragraph, +or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------- | ------------------- | +| 1 | Clause 4.1, first paragraph: "The sentences that follow restate the ones most often failed, and a verdict from this document alone rests on them; a grader with that document to hand holds the [Detector] to every one of its MUSTs, restated here or not" (1, 2, 4, 5), and "rests on them" beside "holds the [Detector] to every one of them" (3); which governs without the detector specification | 1, 2, 3, 4, 5 | yes | +| 2 | Section 2, the paragraph "These are additional.", after the terminology table and before the added term entries: what it is additional to | 2, 5 | open; see below | +| 3 | Clause 6.1, first paragraph: "The [Toolchain] MUST define where the [Project record] lives and MUST load it", set against section 2's Project record entry ("[Calibrations], [Exceptions] and conventions"): one location for the whole record or one per kind | 3, 4 | no | +| 4 | Section 1, "How to read this document": "The table in section 2 glosses every term it borrows", where Suppression, Baseline, Agent, Conform, Warning and Fixture are not in the table | 1 | yes | +| 5 | Clause 4.3, first paragraph: "Irreducible cases MUST be directed to the [Project record], where clause 6.2 requires a justification": what "directed" requires of a toolchain | 3 | yes | + +Notes, not passages: readers 1, 3 and 4 remark under question 3 that the detector specification +was not supplied. Readers 1 to 5 tie the same remark to passage 1 under question 4, where it is +part of naming that passage. + +## Findings and where they lie + +- **Finding A, passage 1, clause 4.1 first paragraph** (all five readers): the diff rewrites the + paragraph, adding "That document's MUSTs are those of its sections 4 to 7", "a verdict from + this document alone rests on them" and "restated here or not". Text the change wrote or + altered. Blocks. +- **Finding B, passage 2, section 2 "These are additional."** (readers 2 and 5): the paragraph's + own words are unchanged, but the diff inserts the new paragraph "A [Blocking] [Rule] whose + documentation is missing is not yet a [Defence]. ..." between the terminology table and it, so + that it no longer follows the table it refers to. Whether that is text the change wrote or + altered is an open point (below); marked as the change's own text. Blocks. +- **Finding C, passage 3, clause 6.1 first paragraph** (readers 3 and 4): unchanged, and set + against section 2's Project record entry, which is also unchanged. Elsewhere: recorded in + `acceptance/ROUTED.md` as a finding against the document, routed to a change of its own, and + does not block. The marker writes only this record; the ROUTED row is for the editor to add. + +Passages 4 and 5 are each named by one reader, in paragraphs the diff changed: recorded, not +findings, applied at the editor's discretion. + +## Points the rules did not settle + +1. **Where finding B lies.** The Marking section counts a paragraph as the change's when the + diff adds, rewords, moves or deletes a sentence in it, or changes a word, term or link, or + when a reader sets it against such a paragraph. The diff does none of those to "These are + additional." itself; it inserts a paragraph immediately before it, between it and the table + whose terms it says the following entries are additional to. Reader 2 places it by "It + follows the terminology table", reader 5 by "after the terminology table and before the new + term definitions"; neither names the inserted paragraph. Whether a paragraph displaced from + its referent by an insertion has been moved, or is set against the inserted paragraph by a + reader who is confused by what it refers to, the rules do not say. The open-point rule treats + a finding whose location is open as the change's own text, so it is marked as blocking. + Run 72's ten passages did not include it, which is consistent with the insertion being its + cause. For the editor to settle in `ACCEPTANCE.md`, if it needs settling, before the next run + of this pull request. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 13b497b, named before the readers started. The process files at +fa79f8a (`ACCEPTANCE.md`, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md`, `tools/changes.py`, `spec-qa.py` and the CI workflows) are identical to those +at 13b497b. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: two (findings A + and B). Not met. +- Passages named by two or more readers elsewhere: one (finding C), routed to + `acceptance/ROUTED.md`; does not block. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A and B; every keyed item passed. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From e3a75d043facdf8faebb5d10b881e5c91fb3ff82 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:34:29 +0100 Subject: [PATCH 30/52] Toolchain: clause 4.1 and section 1 give one test; the undocumented-rule paragraph ends section 2 After run 76: a grader of clause 4.1 checks the restated sentences and takes a verdict already given on the detector under the detector specification as settled, and section 1 says the same; the paragraph on a blocking rule whose documentation is missing moves from under section 2's table, where it parted "These are additional." from the table, to the end of the section after the Project record entry. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 9 ++++++++- TOOLING-SPEC.md | 21 +++++++++++---------- 2 files changed, 19 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c66c189..121c3b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -160,7 +160,14 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be is what clauses [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access), [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) and [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) exist to catch; clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) says the listing covers every blocking rule, documented or not, with a missing route showing; and section [1](TOOLING-SPEC.md#1-what-this-document-is-for) and clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) both say a verdict from this document alone rests on what is restated and a grader with - the detector specification holds the detector to all of it. Accepted by run NNN. + the detector specification holds the detector to all of it. [Run + 76](acceptance/runs/076-toolchain-contradictions-run-8.md) failed on clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification)'s first + paragraph, all five readers unable to tell which of its two tests governs a grader without the + detector specification, and on section [2](TOOLING-SPEC.md#2-terminology)'s "These are additional.", which the new + paragraph had parted from its table. Clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) and section [1](TOOLING-SPEC.md#1-what-this-document-is-for) now give one test: a + grader checks the restated sentences and takes a verdict already given on the detector under + the detector specification as settled; the new paragraph sits at the end of section [2](TOOLING-SPEC.md#2-terminology), after + the Project record entry. Accepted by run NNN. ### 0.2.0, 2026-09-08 diff --git a/TOOLING-SPEC.md b/TOOLING-SPEC.md index 1b638d7..6972f05 100644 --- a/TOOLING-SPEC.md +++ b/TOOLING-SPEC.md @@ -41,8 +41,8 @@ part of its [Toolchain] and are judged as such. **How to read this document.** A [Toolchain] can be graded from this document alone, on the evidence a project shows. Where a clause rests on the [detector specification](DETECTOR-SPEC.md), -as clauses 4.1, 4.2 and 4.4 do, it restates what that document requires of the [Detector], and a -grader with that document to hand holds the [Detector] to all of it. The +as clauses 4.1, 4.2 and 4.4 do, it restates what the grader checks, and a verdict already given +on the [Detector] under that document is taken as settled. The table in section 2 glosses every term it borrows. Each clause states its obligation in its heading and first sentence, and the **Why** paragraph beneath is reasoning, not a further requirement. A link is there for depth and is not a prerequisite for the sentence that carries it. @@ -67,10 +67,6 @@ capitalised term links to its definition; the ones this document leans on most a | [Bundled rule] | A [Rule] a [Detector] ships rather than one the project wrote | | [Harness] | The route by which one [Rule] is run against supplied code on its own | -A [Blocking] [Rule] whose documentation is missing is not yet a [Defence]. It is what clauses -4.2, 5.1 and 8.1 exist to catch, so where those clauses reach a [Toolchain]'s [Rules] they reach -every [Blocking] [Rule] it runs, documented or not. - These are additional. #### Consuming project @@ -85,6 +81,10 @@ A [Defence] a shipped [Toolchain] carries and enables by default in every [Consu The place a project writes down the judgements the method specification delegates to it: [Calibrations], [Exceptions] and conventions. +A [Blocking] [Rule] whose documentation is missing is not yet a [Defence]. It is what clauses +4.2, 5.1 and 8.1 exist to catch, so where those clauses reach a [Toolchain]'s [Rules] they reach +every [Blocking] [Rule] it runs, documented or not. + The distinction between a [Bundled defence] and a project's own is significant throughout. A bundled [Defence] is authored once and runs in codebases its author will never see, so everything a [Practitioner] needs in order to act on it has to travel with it. @@ -117,10 +117,11 @@ any row and the failure mode is visible too: either half alone leaves the [Pract A [Toolchain] MUST NOT route a [Defence] through a [Detector] that does not [Conform] to [the detector specification](DETECTOR-SPEC.md). That document's MUSTs are those of -its sections 4 to 7, and this clause holds the [Detector] to every one of them. The sentences -that follow restate the ones most often failed, and a verdict from this document alone rests on -them; a grader with that document to hand holds the [Detector] to every one of its MUSTs, -restated here or not. The +its sections 4 to 7, and this clause holds the [Detector] to every one of them. A grader of this +clause checks the sentences that follow, which restate the ones most often failed, and takes a +verdict already given on the [Detector] under that document as settled: a [Detector] that +document finds not to [Conform] fails this clause, whether or not the sentences that follow +show why. The [Detector] hosts bespoke [Rules]. It runs one [Rule] on its own against supplied code. It prints a stable [Identifier] with every finding. It runs locally, down to one file, with the result in the command's own output and nothing reportable only through a hosted service. It From 389ef3f967fe910150b14f77623c1150423210a5 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:35:36 +0100 Subject: [PATCH 31/52] Acceptance: a missing-document remark is a note however it quotes; a parted paragraph is the change's text Settles the open points of runs 76, 77, 78 and 80. ACCEPTANCE.md says a remark that another document was not supplied, or that a term is defined only there, is a note however much of the document it quotes to locate the reference; and that a paragraph the pull request parts from what it refers to, by inserting a paragraph between them or moving either, is the change's text. The Lintomatic fixture says `check --only` prints its findings as `check` does, and the detector key's 4.2 row says so and which case a conditional grade is for. ROUTED gains the toolchain 6.1 Project record sentence (run 76) and the detector 4.2 sentence on a detector failing 4.1 (run 78, taken by editor/defined-terms); the 4.3 row links the section, whose heading the toolchain change rewrites. Co-Authored-By: Claude Fable 5.1 --- ACCEPTANCE.md | 10 ++++--- acceptance/ROUTED.md | 48 ++++++++++++++++--------------- acceptance/detector/KEY.md | 2 +- acceptance/fixtures/lintomatic.md | 3 +- 4 files changed, 34 insertions(+), 29 deletions(-) diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index d1bd6af..77eac8f 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -134,15 +134,17 @@ rule. gives a clause number with a description or a shorter quotation that identifies a sentence or paragraph. A quotation whose words occur in more than one paragraph, given with no clause number, names the paragraph where they first occur. A remark that is not about a passage of - the document, such as that another document was not supplied, is recorded as a note and is - not a passage. Two readers name the same passage when the words or sentences they name lie in + the document, such as that another document was not supplied or that a term is defined only + in another document, is recorded as a note and is not a passage, however much of the document + it quotes to locate the reference. Two readers name the same passage when the words or sentences they name lie in the same paragraph; a heading, a list item, a table row and a terminology entry are each a paragraph of their own. A passage named by two or more readers of the run is a finding. - **Where a finding lies.** The record says, for each finding, whether it lies in text the change wrote or altered: a paragraph in which the pull request, read against its merge base with the branch it targets, adds, rewords, moves or deletes a sentence, or changes a word, a term or a - link; or a paragraph that any reader naming the finding sets against such a paragraph. A - finding there blocks the change. A finding anywhere else is a finding against the document, not + link; a paragraph the pull request parts from what it refers to, by inserting a paragraph + between them or moving either; or a paragraph that any reader naming the finding sets against + such a paragraph. A finding there blocks the change. A finding anywhere else is a finding against the document, not against the change: it is recorded in [acceptance/ROUTED.md](acceptance/ROUTED.md), routed to a change of its own, and does not block. The change is judged on what it changed; the document is proved by the keyed questions, which every run answers over the whole of it. diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 68a0be9..f322fc5 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -15,29 +15,31 @@ untouched text of the change those runs tested, and are listed here so that they ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | ---------------------------------------- | -| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68 | `open` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | -| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | -------------------------------------------------- | +| `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind | run 76 | `open` | +| `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | +| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68 | `open` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | +| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | ## Closed diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 5b4d4aa..dd559a8 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -64,7 +64,7 @@ Wrong readings the text invites, which count against the document on one reader: | Clause | Accepted grade | What the fixture supports (no row carries a condition on the reasoning) | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules via the `Rule` interface, first class in every report | -| [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `check --only ` runs one rule, bundled or project, with a distinguishing exit code | +| [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `check --only ` runs one rule, bundled or project, prints its findings as `check` does, and exits with a distinguishing code; the fixture says all three, so a grade given on the condition that `--only` prints as `check` does is the grade for this case | | [4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `LM-NNNN` fixed at first release; project names taken as given, not rewritten; printed on every finding line | | [5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | Local, no account, no server, no network | | [5.2](../../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Yes | `check ` accepts a single file | diff --git a/acceptance/fixtures/lintomatic.md b/acceptance/fixtures/lintomatic.md index f168d56..9d334fc 100644 --- a/acceptance/fixtures/lintomatic.md +++ b/acceptance/fixtures/lintomatic.md @@ -21,7 +21,8 @@ src/db.zig:88:3 proj.no-raw-sql Query built by string concatenation Fuller detail, including a snippet, is written to `.lintomatic/report.html`. The one-line form is printed whether or not the HTML report is enabled. -`lintomatic check --only ` runs a single rule. It accepts any rule name, +`lintomatic check --only ` runs a single rule and prints its findings as +`check` does. It accepts any rule name, bundled or project, and exits 1 when the rule fires, 0 when it does not, and 2 when the rule name is unknown. From 660787d042450e7b8bea9e66915a98a2b2977ac9 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:42:40 +0100 Subject: [PATCH 32/52] Run 081: toolchain contradictions cohort, failed on clause 5.1 Every keyed item at threshold, one miss on clause 6.3 tolerated. One finding in text the change wrote: clause 5.1's new sentence on the listing covering every blocking rule, set by two readers against "every Defence active". Clause 6.1's Project record sentence (four readers) and clauses 6.2 and 6.3 speaking only of exceptions (two) routed to acceptance/ROUTED.md. Co-Authored-By: Claude Fable 5.1 --- .../081-toolchain-contradictions-run-9.md | 183 ++++++++++++++++++ 1 file changed, 183 insertions(+) create mode 100644 acceptance/runs/081-toolchain-contradictions-run-9.md diff --git a/acceptance/runs/081-toolchain-contradictions-run-9.md b/acceptance/runs/081-toolchain-contradictions-run-9.md new file mode 100644 index 0000000..17b2c25 --- /dev/null +++ b/acceptance/runs/081-toolchain-contradictions-run-9.md @@ -0,0 +1,183 @@ +# Run 81: TOOLING-SPEC.md, toolchain contradictions, ninth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: not yet opened at the time of the run; it is to be opened from branch + `editor/toolchain-contradictions`. The readers read commit 0373a27. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 76: clause 4.1's first paragraph and section 1's "How to read this document" + now give one test, a grader checks the restated sentences and takes a verdict already given on + the detector under the detector specification as settled; and the paragraph on a blocking rule + whose documentation is missing, which sat under section 2's table and parted "These are + additional." from it, now ends section 2 after the Project record entry. +- **Run within the pull request**: 9 (runs 57, 58, 62, 65, 67, 68, 72 and 76 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `09c3a3ba5bcd441d513797fcd973e2d0873473ca867aa774ff94643902147e6d`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at 0373a27, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`, which + the marker confirmed byte for byte. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | miss | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (all five); 6.1 Partial +(readers 1, 2 and 5, each mentioning that the fixture names a place for exceptions and none for +calibrations or conventions) and Yes (readers 3 and 4); 6.2 No (all five), each mentioning that +nothing rejects generic text. Readers 1, 2, 3, 4 and 5 grade 4.2 No reading the two undocumented +rules as Blocking, which the key's 4.2 row reads as a No. All five grade 4.3 Yes on the one route +the fixture describes, which the key's 4.3 row says is the case the fixture presents. Readers 3, +4 and 5 give their 6.1 grade with a remark on what it would be in another case; see the points +below. + +Question 1: correct clauses named per reader, distinct within the key's list: reader 1, 4.1; +4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 with 6.1 to 6.3. Reader 2: 4.1; 4.2; 4.3; 5.1 to 5.3 with +6.1 and 6.2; 4.5. Reader 3: 4.1 with 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3; 6.1 to 6.3. Reader 4: +4.1; 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 with 6.1 to 6.3. Reader 5: 4.1; 4.2 with 4.3; 4.4 and +4.5; 5.1 to 5.3 with 6.1 and 6.3; 6.2 with 8.1 and 8.2. No bullet asserts anything the key marks +wrong. Every reader's first bullet (the toolchain judged as one assembled whole) and reader 2's +remark that section 7 is SHOULDs and the declaration optional are counted neither way, as the +key says. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through"; for 2b all five quote section 9's "The project-level verdict is a single grade; ..."; +for 2c all five quote clause 9.1's "The two verdicts MUST be graded and declared separately, and +neither implies the other." No answer gives a grade per part or a single grade for 2c. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says there is no agent summary (all +five), readers 3, 4 and 5 adding that the toolchain does not conform at all; S8 says Zigpipe +ships no tooling (all five), reader 1 adding that zig-qa's own artefact grade is not described +and giving no verdict on it; Decl says the declaration is optional and the verdict rests on the +exercise (all five). + +### Misses + +- **Reader 2, 6.3, Partial.** The key accepts Yes only. Reasoning: `zig-qa list --exceptions` + lists exceptions in the rule listing's format, but calibrations and conventions are not + described as enumerable, "so the same gap as 6.1 applies". The reasoning quotes nothing of the + document; it carries over the reader's 6.1 reading of section 2's Project record entry. The key + names no wrong reading for 6.3, so this is the reader's single miss on the item, tolerated by + the threshold. + +No answer matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..0373a27 (9e914fe +being the merge base with `next`) adds, rewords, moves or deletes a sentence in the paragraph, +changes a word, term or link in it, or parts it from what it refers to. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------ | +| 1 | Clause 5.1, first paragraph: "every [Defence] active in the project" with "The listing covers every [Blocking] [Rule] the [Toolchain] runs, documented or not"; a listing of Defences that covers what is not a Defence | 1, 5 | yes: the second sentence quoted is added by the diff | +| 2 | Clause 6.1, first paragraph: "The [Toolchain] MUST define where the [Project record] lives and MUST load it"; one location for the whole record or one per kind | 1, 2, 3, 4 | no | +| 3 | Section 2, the Project record entry: "[Calibrations], [Exceptions] and conventions" (1, 3), the whole entry quoted (2, 4), set against passage 2 | 1, 2, 3, 4 | no: its words are unchanged; the diff inserts a paragraph after it, but it refers to nothing that follows it | +| 4 | Clauses 6.2 and 6.3, described as speaking only of exceptions, set against passage 3 | 1, 2 | no; whether this names a passage is open, see below | +| 5 | Section 2, the Defence row: "A [Rule] with its documentation, in force and [Blocking]", set against passage 1 | 1 | no: the diff re-pads the table for an added row; the row's words are unchanged | +| 6 | Section 2, the paragraph "A [Blocking] [Rule] whose documentation is missing is not yet a [Defence]", set against passage 1 | 5 | yes: added by the diff | +| 7 | Clause 5.3, first paragraph: "Every [Defence] the project has written itself MUST appear in the listing clause 5.1 requires, meeting its content requirements in full"; what the content requirements are | 2 | no | +| 8 | Clause 4.1, first paragraph: "takes a verdict already given on the [Detector] under that document as settled"; whether to grade the detector itself or only the restated sentences, with no verdict given | 3 | yes: rewritten by the diff | +| 9 | Clause 9.2, first paragraph: "A project or a shipped artefact MAY declare" followed by "A project that ships an artefact carries both levels of clause 9.1 in that declaration"; an obligation inside an optional provision | 3 | no | + +Passage 8 is a passage and not a note: reader 3 says it was given no verdict on Lintomatic, but +its confusion is what the sentence asks a grader to do, not that another document was missing. +Reader 3 cites the next paragraph of 9.2 as confirming the declaration optional, which does not +set passage 9 against it. + +Notes, not passages: under question 3, readers 1 and 5 remark that the detector specification +was not supplied or not seen in full, and readers 2 and 4 that the fixture gives no verdict on +Lintomatic under it. + +## Findings and where they lie + +- **Finding A, passage 1, clause 5.1 first paragraph** (readers 1 and 5): the diff adds the + sentence "The listing covers every [Blocking] [Rule] the [Toolchain] runs, documented or not, + and a [Rule] whose route to documentation is missing is listed with that gap showing." Reader 5 + also sets the paragraph against section 2's added paragraph (passage 6). Text the change wrote + or altered. Blocks. +- **Finding B, passages 2 and 3, clause 6.1 first paragraph against section 2's Project record + entry** (readers 1, 2, 3 and 4): both paragraphs unchanged, and no reader naming them sets + either against a changed paragraph. Elsewhere: a finding against the document, recorded in + `acceptance/ROUTED.md` (its 6.1 row, raised by run 76, gains run 81), routed to a change of its + own, and does not block. The marker writes only this record; the ROUTED entry is for the + editor. +- **Finding C, passage 4, clauses 6.2 and 6.3** (readers 1 and 2), counted as named under the + open-point rule (below): unchanged, and set against the unchanged Project record entry. + Elsewhere: routed to `acceptance/ROUTED.md` with finding B, whose passage it belongs to; does + not block. + +Passages 5 to 9 are each named by one reader: recorded, not findings, applied at the editor's +discretion. + +## Points the rules did not settle + +1. **6.1 grades given with a remark on another case.** Reader 3 gives "Yes, with a caveat", that + it would be Partial if calibrations or conventions are recorded elsewhere and never read; + reader 4 gives Yes and says it would be Partial if those belong to the project record; reader + 5 gives Partial and says it becomes Yes if the project has neither. The Marking section reads + a grade given on a stated condition as the grade for the case the fixture presents only where + the key says which case that is, and the key's 6.1 row does not say it in those terms. + Judgement: each reader states one grade outright and the remark names what it would be in a + different case, so the stated grade is the grade; each is one the key accepts, and readers 1, + 2 and 5's Partial meets the key's condition. Had the remark made the grade conditional, the + open-point rule marks it for the reader. Marked for the readers. +2. **Whether passage 4 is named.** Readers 1 and 2 say clauses 6.1 to 6.3, or 6.2 and 6.3, + "mention only exceptions" or "speak only of exceptions", which describes the clauses and + identifies no one sentence or paragraph. The open-point rule counts the passage; it lies in + unchanged text and does not affect the verdict. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 389ef3f, named before the readers started. The process files at +0373a27 (`ACCEPTANCE.md`, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md`, `tools/changes.py`, `spec-qa.py` and the CI workflows) are identical to those +at 389ef3f. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item but 6.3, + where four of five. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: one (finding A). + Not met. +- Passages named by two or more readers elsewhere: findings B and C, routed to + `acceptance/ROUTED.md`; do not block. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on finding A; every keyed item passed. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From dbcab02d290409461ce947887f40dcab58144c44 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:42:58 +0100 Subject: [PATCH 33/52] Toolchain 5.1: an undocumented blocking rule is not yet a defence, and the listing can show it After run 81: clause 5.1 no longer says the listing covers every blocking rule, documented or not, which contradicted "every Defence active"; it says such a rule is not yet a defence, that clause 4.2 fails it, and that the listing can show it with the gap showing. Section 2's paragraph names clauses 4.2 and 8.1 as the clauses that catch it. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 7 ++++++- TOOLING-SPEC.md | 10 +++++----- 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 121c3b4..fc8defd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -167,7 +167,12 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be paragraph had parted from its table. Clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) and section [1](TOOLING-SPEC.md#1-what-this-document-is-for) now give one test: a grader checks the restated sentences and takes a verdict already given on the detector under the detector specification as settled; the new paragraph sits at the end of section [2](TOOLING-SPEC.md#2-terminology), after - the Project record entry. Accepted by run NNN. + the Project record entry. [Run 81](acceptance/runs/081-toolchain-contradictions-run-9.md) + failed on clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), two readers setting "every Defence active" against the new sentence that the + listing covers every blocking rule, documented or not: a listing of defences said to cover + what is not one. Clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) now says such a rule is not yet a defence, that clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) fails it, + and that the listing can show it with the gap showing; section [2](TOOLING-SPEC.md#2-terminology)'s paragraph names + clauses [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) and [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) as the clauses that catch it. Accepted by run NNN. ### 0.2.0, 2026-09-08 diff --git a/TOOLING-SPEC.md b/TOOLING-SPEC.md index 6972f05..64f837b 100644 --- a/TOOLING-SPEC.md +++ b/TOOLING-SPEC.md @@ -81,9 +81,9 @@ A [Defence] a shipped [Toolchain] carries and enables by default in every [Consu The place a project writes down the judgements the method specification delegates to it: [Calibrations], [Exceptions] and conventions. -A [Blocking] [Rule] whose documentation is missing is not yet a [Defence]. It is what clauses -4.2, 5.1 and 8.1 exist to catch, so where those clauses reach a [Toolchain]'s [Rules] they reach -every [Blocking] [Rule] it runs, documented or not. +A [Blocking] [Rule] whose documentation is missing is not yet a [Defence]. Clauses 4.2 and 8.1 +exist to catch it, so where they reach a [Toolchain]'s [Rules] they reach every [Blocking] +[Rule] it runs, documented or not. The distinction between a [Bundled defence] and a project's own is significant throughout. A bundled [Defence] is authored once and runs in codebases its author will never see, so everything a @@ -263,8 +263,8 @@ kind of property that lives in the [Toolchain] and nowhere else. The [Toolchain] MUST list every [Defence] active in the project without running it. The listing MUST include each [Defence]'s [Identifier] and a terse statement of what it forbids or requires. It MUST provide the route to each one's full documentation. -The listing covers every [Blocking] [Rule] the [Toolchain] runs, documented or not, and a -[Rule] whose route to documentation is missing is listed with that gap showing. +A [Blocking] [Rule] whose route to documentation is missing is not yet a [Defence]: clause 4.2 +fails it, and the listing can show it with that gap showing. **Why**: method specification clause 8.5. An [Agent] arriving at a codebase has no colleague to ask. Without a listing, a project's standards can only be learned by violating them one at a time. From 0da928d9830d12101122821650daacd4e050a6b7 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:43:19 +0100 Subject: [PATCH 34/52] ROUTED: run 81 names the toolchain 6.1 Project record sentence again; 6.2 and 6.3 speak only of exceptions Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 51 ++++++++++++++++++++++---------------------- 1 file changed, 26 insertions(+), 25 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index f322fc5..fe7d05c 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -15,31 +15,32 @@ untouched text of the change those runs tested, and are listed here so that they ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | -------------------------------------------------- | -| `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind | run 76 | `open` | -| `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | -| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68 | `open` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | -| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | -------------------------------------------------- | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) and [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Both clauses speak only of exceptions, where section 2's Project record entry and clause 6.1 cover calibrations and conventions too | run 81 | `open` | +| `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind; named again by four readers of run 81 | run 76 | `open` | +| `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | +| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68 | `open` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | +| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | ## Closed From 1207b6a4c436f57a3d883b11ed7666d291c6885d Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:48:09 +0100 Subject: [PATCH 35/52] Run 082: toolchain contradictions cohort, failed on clauses 5.1 and 4.1 Every keyed item five of five. Two findings in text the change wrote or altered: clause 5.1's sentence on an undocumented blocking rule, and clause 4.1's "a verdict already given". Clause 6.2's last paragraph (two readers) routed to acceptance/ROUTED.md, on its open row. Co-Authored-By: Claude Fable 5.1 --- .../082-toolchain-contradictions-run-10.md | 178 ++++++++++++++++++ 1 file changed, 178 insertions(+) create mode 100644 acceptance/runs/082-toolchain-contradictions-run-10.md diff --git a/acceptance/runs/082-toolchain-contradictions-run-10.md b/acceptance/runs/082-toolchain-contradictions-run-10.md new file mode 100644 index 0000000..1b99335 --- /dev/null +++ b/acceptance/runs/082-toolchain-contradictions-run-10.md @@ -0,0 +1,178 @@ +# Run 82: TOOLING-SPEC.md, toolchain contradictions, tenth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: not yet opened at the time of the run; it is to be opened from branch + `editor/toolchain-contradictions`. The readers read commit de34774. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 81: clause 5.1 no longer says the listing covers every blocking rule, + documented or not; it says a blocking rule whose route to documentation is missing is not yet + a defence, that clause 4.2 fails it, and that the listing can show it with the gap showing; and + section 2's closing paragraph names clauses 4.2 and 8.1 as the clauses that catch such a rule. +- **Run within the pull request**: 10 (runs 57, 58, 62, 65, 67, 68, 72, 76 and 81 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `1fa098d5a7901d8211c09387ca7bffd0bcabcd05caba5565a489557ae5761028`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at de34774, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`, which + the marker confirmed byte for byte. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (all five); 6.1 Partial +(readers 2, 3 and 5, each mentioning that the fixture names a place for exceptions and none for +calibrations or conventions) and Yes (readers 1 and 4); 6.2 No (all five), each mentioning that +nothing rejects a generic justification or generic text. All five grade 4.2 No reading the two +undocumented rules as Blocking because `zig-qa run` stops when Lintomatic fails, which the key's +4.2 row reads as a No. All five grade 4.3 Yes on the one route the fixture describes, which the +key's 4.3 row says is the case the fixture presents. Readers 1, 2 and 3 cite the 5.1 sentence +that the listing "can show it with that gap showing" for their 5.1 Yes; reader 5 says the two +`(none)` entries are the gap clause 4.2 reports. + +Question 1: correct clauses named per reader within the first six bullets. Reader 1: 4.1 with +4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 with 6.3; 6.1 and 6.2. Reader 2: 4.1; 4.2; 4.3 with 4.5; +5.1 to 5.3 with 6.2 and 6.3. Reader 3: 4.1 with 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3; 6.1 to 6.3. +Reader 4 (seven bullets, the first six marked): 4.1; 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3. Reader +5: 4.1; 4.2; 4.3; 5.1 to 5.3 with 6.2 and 6.3; 4.5. No bullet asserts anything the key marks +wrong. Every reader's first bullet (the toolchain judged as one assembled whole), the remarks of +readers 1, 2, 3 and 5 that a shipped artefact has a second grade, and the remarks of readers 2 +and 3 that section 7 is SHOULDs, are counted neither way, as the key says. Reader 2's sixth +bullet also calls clause 6.4's defaults recommended; a bullet on a SHOULD asserts no obligation +and is not counted. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through"; for 2b all five quote section 9's "The project-level verdict is a single grade; ..."; +for 2c all five quote clause 9.1's "The two verdicts MUST be graded and declared separately, and +neither implies the other." No answer gives a grade per part or a single grade for 2c. Reader 1's +2b adds that Zigpipe ships nothing and so has only the one grade, quoting section 9; it asserts +nothing the key marks wrong. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says there is no agent summary (all +five), each adding that the toolchain does not conform at all; S8 says Zigpipe ships no tooling +(all five), readers 1, 2, 4 and 5 adding that zig-qa would carry its own artefact grade, which +none grades; Decl says the declaration is optional and the verdict rests on the exercise (all +five). + +### Misses + +None. No answer matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..de34774 (9e914fe +being the merge base with `next`) adds, rewords, moves or deletes a sentence in the paragraph, +changes a word, term or link in it, or parts it from what it refers to. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | ------------------------------------------------------------------------------------------------ | +| 1 | Clause 5.1, first paragraph: "It MUST provide the route to each one's full documentation" with "the listing can show it with that gap showing", whether an empty route meets the MUST (1); "MUST list every [Defence] active in the project" with "A [Blocking] [Rule] whose route to documentation is missing is not yet a [Defence] ... can show it", whether the undocumented rules must appear (3) | 1, 3 | yes: the second sentence each quotes is added by the diff | +| 2 | Clause 4.1, first paragraph: "takes a verdict already given on the [Detector] under that document as settled" (1, 2), with "a [Detector] that document finds not to [Conform] fails this clause, whether or not the sentences that follow show why" (2); which verdict is meant, and what a grader does when none has been given | 1, 2 | yes: rewritten by the diff | +| 3 | Clause 6.2, last paragraph: "by a check it documents: 'needed for now', 'legacy', 'TODO' and their like" against "could be pasted onto any [Exception] unchanged", which test governs (1); "name the [Hazard] ... and the scope" and "cannot verify truth", with the heading's "MUST reject a generic one", whether the check must test that hazard and scope are named (4) | 1, 4 | no: neither the paragraph nor the 6.2 heading is touched by the diff | +| 4 | Clause 4.3, second paragraph: "This clause is where the project's decision is made and enforced, and the decision is no"; what the "no" answers | 2 | no: the diff adds sentences to the first paragraph only and inserts no paragraph between them | +| 5 | Clause 4.4, first paragraph: "The [Toolchain]'s own entry point MUST meet clauses 5.1 to 5.4 of the [detector specification] for every [Defence] it routes"; the restatement covers three properties, so whether 5.4 is graded | 2 | no | +| 6 | Clause 6.1, first paragraph: "The [Toolchain] MUST define where the [Project record] lives and MUST load it", set against section 2's Project record entry; one location for all three kinds or not | 2 | no | +| 7 | Section 2: "These are additional."; what the terms are additional to | 2 | no: the diff re-pads the table above it and adds a row; the sentence and its place are unchanged | +| 8 | Section 2, the paragraph "A [Blocking] [Rule] whose documentation is missing is not yet a [Defence]"; it sits under the Project record entry, so which term it governs | 4 | yes: added by the diff | +| 9 | Clause 5.3, second paragraph: "Where such a [Defence] is not expressible in the [Toolchain]'s own [Detectors], its entry MAY be sourced from wherever it is enabled ..."; whether that reaches configuration outside the toolchain's own files | 5 | no | + +Notes, not passages: reader 4's remark on clause 9.1, "the project's remediations against +section 7 of the method specification", says it could not work out what that section requires +because the method specification was not in its materials; that is a remark that another document +was not supplied, and the Marking section records it as a note. Under question 3, readers 1, 2 +and 3 remark that no verdict on Lintomatic under the detector specification was given, and +reader 2 that it did not have the detector specification, when grading 4.1 and 4.4. + +## Findings and where they lie + +- **Finding A, passage 1, clause 5.1 first paragraph** (readers 1 and 3): the diff adds the + sentence "A [Blocking] [Rule] whose route to documentation is missing is not yet a \[Defence\]: + clause 4.2 fails it, and the listing can show it with that gap showing." Reader 1 sets it + against the paragraph's MUST on the route to documentation, reader 3 against its MUST to list + every Defence. Text the change wrote or altered. Blocks. +- **Finding B, passage 2, clause 4.1 first paragraph** (readers 1 and 2): the diff rewrites the + paragraph, adding the sentence each quotes, "A grader of this clause checks the sentences that + follow ... and takes a verdict already given on the [Detector] under that document as settled: + ...". Text the change wrote or altered. Blocks. Whether it is a passage rather than a note is + recorded below. +- **Finding C, passage 3, clause 6.2 last paragraph** (readers 1 and 4): unchanged, and neither + reader sets it against a changed paragraph; reader 4 sets it against the 6.2 heading, also + unchanged. Elsewhere: a finding against the document, recorded in `acceptance/ROUTED.md`, where + the open 6.2 row raised by runs 67 and 68 (whether the check must test that the hazard and the + scope are named) covers it and gains run 82; routed to a change of its own, and does not block. + The marker writes only this record; the ROUTED entry is for the editor. + +Passages 4 to 9 are each named by one reader: recorded, not findings, applied at the editor's +discretion. + +## Points the rules did not settle + +1. **Whether passage 2 is a passage or a note.** Readers 1 and 2 tie their confusion to the + fixture giving no verdict on Lintomatic under the detector specification, and the Marking + section makes a remark that another document was not supplied a note. Judgement: each reader's + confusion is what the sentence asks a grader to do, which verdict it means (1) and what to do + when none has been given (2), and that is about the passage, not about a missing document; so + it is a passage. Had it stayed open, the open-point rule counts the passage. Marked as named, + which makes finding B. +2. **6.1 grades given with a remark on another case.** Readers 2, 3 and 5 grade Partial and say + it would be Yes if the exceptions file is the whole project record (3 implicitly, 2 and 5 in + terms). The Marking section reads a grade given on a stated condition as the grade for the + case the fixture presents only where the key says which case that is, and the key's 6.1 row + does not. Judgement: each reader states Partial outright and the remark names a different + case, so Partial is the grade; it meets the key's condition. Had the remark made the grade + conditional, the open-point rule marks it for the reader. Marked for the readers. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 0da928d, named before the readers started. The process files at +de34774 (`ACCEPTANCE.md`, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md`, `tools/changes.py`, `spec-qa.py` and the CI workflows) are identical to those +at 0da928d. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: two (findings A + and B). Not met. +- Passages named by two or more readers elsewhere: finding C, routed to `acceptance/ROUTED.md`; + does not block. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A and B; every keyed item passed. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From ae6faf2b34d4f8e21d146fd62cef9445102d00fc Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:48:34 +0100 Subject: [PATCH 36/52] ROUTED: run 82 names the toolchain 6.2 last paragraph again Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index fe7d05c..c16596a 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -29,7 +29,7 @@ untouched text of the change those runs tested, and are listed here so that they | `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | | `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | | `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68, 82 | `open` | | `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | | `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | | `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | From 0b2078066350856c0daddd934772bf5a7e5f9b1b Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:48:36 +0100 Subject: [PATCH 37/52] Toolchain: 5.1 says nothing of undocumented rules; 4.1 names the verdict it takes and the case without one After run 82: clause 5.1 drops the sentence on a blocking rule whose documentation is missing, section 2's closing paragraph being where that is said, and that paragraph ends the section; clause 4.1 and section 1 name the verdict as one the detector already holds under the detector specification's section 8, and say that without one the restated sentences are the test. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 9 ++++++++- TOOLING-SPEC.md | 22 ++++++++++------------ 2 files changed, 18 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fc8defd..c314107 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -172,7 +172,14 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be listing covers every blocking rule, documented or not: a listing of defences said to cover what is not one. Clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) now says such a rule is not yet a defence, that clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) fails it, and that the listing can show it with the gap showing; section [2](TOOLING-SPEC.md#2-terminology)'s paragraph names - clauses [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) and [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) as the clauses that catch it. Accepted by run NNN. + clauses [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) and [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) as the clauses that catch it. [Run 82](acceptance/runs/082-toolchain-contradictions-run-10.md) + failed on that clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) sentence again, two readers asking whether an empty route meets the MUST + and whether "can show" means such a rule need not appear, and on clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification)'s "a verdict + already given", two readers unable to tell which verdict or what a grader does without one. + Clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) says nothing of undocumented rules now, section [2](TOOLING-SPEC.md#2-terminology)'s closing paragraph being where + that is said; clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) and section [1](TOOLING-SPEC.md#1-what-this-document-is-for) name the verdict as one given under the detector + specification's section 8 and say that without one the restated sentences are the test. + Accepted by run NNN. ### 0.2.0, 2026-09-08 diff --git a/TOOLING-SPEC.md b/TOOLING-SPEC.md index 64f837b..b7dbce4 100644 --- a/TOOLING-SPEC.md +++ b/TOOLING-SPEC.md @@ -41,8 +41,8 @@ part of its [Toolchain] and are judged as such. **How to read this document.** A [Toolchain] can be graded from this document alone, on the evidence a project shows. Where a clause rests on the [detector specification](DETECTOR-SPEC.md), -as clauses 4.1, 4.2 and 4.4 do, it restates what the grader checks, and a verdict already given -on the [Detector] under that document is taken as settled. The +as clauses 4.1, 4.2 and 4.4 do, it restates what the grader checks, and a verdict the [Detector] +already holds under that document's section 8 counts too. The table in section 2 glosses every term it borrows. Each clause states its obligation in its heading and first sentence, and the **Why** paragraph beneath is reasoning, not a further requirement. A link is there for depth and is not a prerequisite for the sentence that carries it. @@ -81,14 +81,14 @@ A [Defence] a shipped [Toolchain] carries and enables by default in every [Consu The place a project writes down the judgements the method specification delegates to it: [Calibrations], [Exceptions] and conventions. -A [Blocking] [Rule] whose documentation is missing is not yet a [Defence]. Clauses 4.2 and 8.1 -exist to catch it, so where they reach a [Toolchain]'s [Rules] they reach every [Blocking] -[Rule] it runs, documented or not. - The distinction between a [Bundled defence] and a project's own is significant throughout. A bundled [Defence] is authored once and runs in codebases its author will never see, so everything a [Practitioner] needs in order to act on it has to travel with it. +Throughout this document, a [Blocking] [Rule] whose documentation is missing is not yet a +[Defence]. Clauses 4.2 and 8.1 exist to catch it, so where they reach a [Toolchain]'s [Rules] +they reach every [Blocking] [Rule] it runs, documented or not. + ## 3. The division of responsibility Almost every requirement in section 8 of the method specification has two halves. Stating only one @@ -118,10 +118,10 @@ any row and the failure mode is visible too: either half alone leaves the [Pract A [Toolchain] MUST NOT route a [Defence] through a [Detector] that does not [Conform] to [the detector specification](DETECTOR-SPEC.md). That document's MUSTs are those of its sections 4 to 7, and this clause holds the [Detector] to every one of them. A grader of this -clause checks the sentences that follow, which restate the ones most often failed, and takes a -verdict already given on the [Detector] under that document as settled: a [Detector] that -document finds not to [Conform] fails this clause, whether or not the sentences that follow -show why. The +clause checks the sentences that follow, which restate the ones most often failed. Where the +[Detector] has already been graded against that document, under its section 8, that verdict +counts too: a [Detector] found not to [Conform] fails this clause whether or not the sentences +that follow show why. Where it has not, the sentences that follow are the test. The [Detector] hosts bespoke [Rules]. It runs one [Rule] on its own against supplied code. It prints a stable [Identifier] with every finding. It runs locally, down to one file, with the result in the command's own output and nothing reportable only through a hosted service. It @@ -263,8 +263,6 @@ kind of property that lives in the [Toolchain] and nowhere else. The [Toolchain] MUST list every [Defence] active in the project without running it. The listing MUST include each [Defence]'s [Identifier] and a terse statement of what it forbids or requires. It MUST provide the route to each one's full documentation. -A [Blocking] [Rule] whose route to documentation is missing is not yet a [Defence]: clause 4.2 -fails it, and the listing can show it with that gap showing. **Why**: method specification clause 8.5. An [Agent] arriving at a codebase has no colleague to ask. Without a listing, a project's standards can only be learned by violating them one at a time. From c770722857c27f5773998f57e1026e5f2fd103b1 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:48:49 +0100 Subject: [PATCH 38/52] Changelog: clause links point at the headings as they are Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c314107..6875767 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -178,7 +178,7 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be already given", two readers unable to tell which verdict or what a grader does without one. Clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) says nothing of undocumented rules now, section [2](TOOLING-SPEC.md#2-terminology)'s closing paragraph being where that is said; clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) and section [1](TOOLING-SPEC.md#1-what-this-document-is-for) name the verdict as one given under the detector - specification's section 8 and say that without one the restated sentences are the test. + specification's conformance section and say that without one the restated sentences are the test. Accepted by run NNN. ### 0.2.0, 2026-09-08 From 6dd7928581cce8962844b6a7f2746983a238b488 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:52:53 +0100 Subject: [PATCH 39/52] Run 083: toolchain contradictions cohort, failed on section 2's closing paragraph and clause 5.1 Every keyed item at threshold, one miss on clause 4.3 tolerated. Two findings in text the change wrote or set against it: section 2's closing paragraph on an undocumented blocking rule, and clause 5.1's first paragraph set against it, three readers asking whether such a rule has to appear in the listing. Clause 6.2's first paragraph against clause 6.4's defaults (two readers) routed to acceptance/ROUTED.md. Co-Authored-By: Claude Fable 5.1 --- .../083-toolchain-contradictions-run-11.md | 175 ++++++++++++++++++ 1 file changed, 175 insertions(+) create mode 100644 acceptance/runs/083-toolchain-contradictions-run-11.md diff --git a/acceptance/runs/083-toolchain-contradictions-run-11.md b/acceptance/runs/083-toolchain-contradictions-run-11.md new file mode 100644 index 0000000..1a10aa2 --- /dev/null +++ b/acceptance/runs/083-toolchain-contradictions-run-11.md @@ -0,0 +1,175 @@ +# Run 83: TOOLING-SPEC.md, toolchain contradictions, eleventh run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: not yet opened at the time of the run; it is to be opened from branch + `editor/toolchain-contradictions`. The readers read commit c770722. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 82: clause 5.1 says nothing of undocumented rules any more; the paragraph on + a blocking rule whose documentation is missing now ends section 2, after the paragraph on the + distinction between a bundled defence and a project's own, and opens "Throughout this + document"; clause 4.1 and section 1 name the verdict a grader takes as one the detector already + holds under the detector specification's section 8, and clause 4.1 says that without one the + restated sentences are the test. +- **Run within the pull request**: 11 (runs 57, 58, 62, 65, 67, 68, 72, 76, 81 and 82 came + before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `1f981586432ce938dba63935e5be59cd7e1b36b1569fedc019b576877b34c06d`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at c770722, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`, which + the marker confirmed byte for byte. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (readers 1, 3 and 4) and +Partial (readers 2 and 5); 6.1 Partial (readers 1, 3, 4 and 5, each mentioning that the fixture +names a place for exceptions and none for calibrations or conventions) and Yes (reader 2); 6.2 +No (all five), each mentioning that nothing rejects a generic justification or generic text. +All five grade 4.1 No on the table's "Nothing for those two" row, and 4.2 No reading the two +undocumented rules as Blocking because `zig-qa run` stops when Lintomatic fails, which the key's +4.2 row reads as a No. Readers 3 and 4 give 5.1 Yes because section 2 says the two undocumented +rules are not yet Defences; readers 2 and 5 give Partial on the `(none)` routes. The key accepts +all three grades with no condition. + +Question 1: correct clauses named per reader within the first six bullets. Reader 1: 4.1 with +4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 with 6.3; 6.1 and 6.2. Reader 2: 4.1; 4.2 with 4.3; 4.4 and +4.5; 5.1 to 5.3 with 6.3; 6.1 and 6.2. Reader 3: 4.1 with 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 +with 6.1 to 6.3; 8.1 and 8.2. Reader 4: 4.1; 4.2; 4.3; 4.5 with 5.1 to 5.3; 6.1 and 6.2. +Reader 5: 4.1 with 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 with 6.3; 6.1 and 6.2. No bullet asserts +anything the key marks wrong. Every reader's first bullet (the toolchain judged as one assembled +whole), reader 3's remark that agent-facing summaries are SHOULDs, and reader 5's remark that a +shipped artefact gets a separate grade with section 8 applying only to it are counted neither +way, as the key says. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through"; for 2b all five quote section 9's "The project-level verdict is a single grade; ..."; +for 2c all five quote clause 9.1's "The two verdicts MUST be graded and declared separately, and +neither implies the other." No answer gives a grade per part or a single grade for 2c. Reader 2's +2b adds that a second grade arises only for a shipped artefact and Zigpipe ships none; it asserts +nothing the key marks wrong. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says the toolchain does not conform at +all and there is no agent summary (all five); S8 says Zigpipe ships no tooling to other projects +(all five); Decl says the declaration is optional and the verdict rests on the exercise or the +evidence (all five). + +### Misses + +- **Reader 5, clause 4.3: Partial** (key accepts Yes). Reasoning: the inline route is closed by + `ignore_comments = false` and exceptions go to `quality-exceptions.toml`, but the fixture does + not say whether Lintomatic has any other route, such as a silent baseline, so the reader cannot + confirm the clause covers every route. The grade is stated outright, not on a condition. The + reasoning quotes neither the document nor any sentence of clause 4.3; it is a reading of the + fixture's silence. The key names no wrong reading for 4.3, so this is a single miss on the item, + tolerated by the threshold (four of five correct). + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..c770722 (9e914fe +being the merge base with `next`) adds, rewords, moves or deletes a sentence in the paragraph, +changes a word, term or link in it, or parts it from what it refers to. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------- | +| 1 | Section 2, closing paragraph: "Throughout this document, a [Blocking] [Rule] whose documentation is missing is not yet a [Defence]. ..." (5 quotes it whole; 1 and 4 quote "a [Blocking] [Rule] whose documentation is missing is not yet a [Defence]"); each sets it against clause 5.1, whether undocumented blocking rules are listed | 1, 4, 5 | yes: added by the diff | +| 2 | Clause 5.1, first paragraph: "The [Toolchain] MUST list every [Defence] active in the project" (4, 5), and "every [Defence]" with clause number 5.1 and its listing requirement (1); set against passage 1 | 1, 4, 5 | no in itself; each reader sets it against passage 1, which the diff added | +| 3 | Clause 6.2, first paragraph: "It MUST NOT supply a default, and MUST reject an [Exception] that omits one"; set against clause 6.4's defaults, whether a justification is a delegated judgement a default may fill | 2, 3 | no; nor is either 6.4 paragraph the readers set it against | +| 4 | Clause 4.1, first paragraph: "Where it has not, the sentences that follow are the test" against "this clause holds the [Detector] to every one of them"; whether the restated sentences are the full test or a sample | 1 | yes: rewritten by the diff | +| 5 | Clause 6.2, last paragraph: "could be pasted onto any [Exception] unchanged, by a check it documents: 'needed for now', 'legacy', 'TODO' and their like"; which test the check applies | 1 | no | +| 6 | Clause 6.4, first paragraph: "Where the method specification delegates a judgement and the project has recorded nothing, a documented [Toolchain] default is what the [Practitioner] falls back to" | 2 | no | +| 7 | Clause 6.4 heading: "The [Toolchain] SHOULD state its own defaults for anything the method leaves to the project" | 3 | no | +| 8 | Clause 9.1, first paragraph: "The two verdicts MUST be graded and declared separately", set against 9.2's optional declaration | 3 | yes: the diff rewords sentences in the paragraph | +| 9 | Clause 9.2, third paragraph: "The declaration is optional", set against passage 8 | 3 | yes: the diff extends the sentence quoted | +| 10 | Section 2, the Consuming project entry: "A codebase that installs a [Detector] or a [Toolchain] shipped by somebody else", with the Bundled defence entry named by its term; whether the `zq.*` rules are bundled defences | 4 | no | +| 11 | Clause 6.1 heading: "MUST define a location for the project record", set against section 2's Project record entry; one location for all three kinds or not | 4 | no | + +Passages 6 and 7 are different paragraphs: the Marking section makes a heading a paragraph of its +own, so readers 2 and 3 do not name the same 6.4 passage. Passages 3 and 5 are different +paragraphs of 6.2. + +Notes, not passages: under question 3, readers 1, 3 and 5 remark that the detector specification +was not supplied and that they graded 4.1 (and, for reader 5, 4.4) on the restated sentences. + +## Findings and where they lie + +- **Finding A, passage 1, section 2 closing paragraph** (readers 1, 4 and 5): the diff adds the + paragraph. Text the change wrote or altered. Blocks. +- **Finding B, passage 2, clause 5.1 first paragraph** (readers 1, 4 and 5): the paragraph is + unchanged by the diff, but every reader naming it sets it against passage 1, which the diff + added: whether a Blocking Rule that section 2 says is "not yet a [Defence]" falls within the + duty to list "every [Defence] active in the project". A paragraph a reader naming the finding + sets against a changed paragraph is the change's own text under the Marking section. Blocks. +- **Finding C, passage 3, clause 6.2 first paragraph** (readers 2 and 3): unchanged, and both + readers set it against clause 6.4 (reader 2 its first paragraph, reader 3 its heading), also + unchanged. Elsewhere: a finding against the document, to be recorded in `acceptance/ROUTED.md`, + where no open row covers the default of 6.2 against the defaults of 6.4; routed to a change of + its own, and does not block. The marker writes only this record; the ROUTED entry is for the + editor. + +Passages 4 to 11 are each named by one reader: recorded, not findings, applied at the editor's +discretion. + +## Points the rules did not settle + +1. **Whether reader 4 names the Bundled defence entry.** Reader 4 quotes the Consuming project + entry and names "Bundled defence" only by its term in its item heading. Judgement: the term is + a description that identifies the entry, so it is named; had it stayed open, the open-point + rule counts the passage. No other reader names that entry, so it makes no finding either way. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit ae6faf2, named before the readers started. The process files at +c770722 (`ACCEPTANCE.md`, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md`, `tools/changes.py`, `spec-qa.py` and the CI workflows) are identical to those +at ae6faf2. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item but 4.3, + where four of five are correct. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: two (findings A + and B). Not met. +- Passages named by two or more readers elsewhere: finding C, routed to `acceptance/ROUTED.md`; + does not block. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A and B; every keyed item passed its threshold. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From a095fa7ad823b41b0abe3f932d8484dee9a3fd10 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:53:10 +0100 Subject: [PATCH 40/52] ROUTED: toolchain 6.2's first paragraph against clause 6.4's defaults, from run 83 Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 1 + 1 file changed, 1 insertion(+) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index c16596a..5ae6e16 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -17,6 +17,7 @@ untouched text of the change those runs tested, and are listed here so that they | Document | Clause | Passage | Raised by | Status | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | -------------------------------------------------- | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), the first paragraph | "It MUST NOT supply a default, and MUST reject an Exception that omits one" against clause [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)'s defaults | run 83 | `open` | | `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) and [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Both clauses speak only of exceptions, where section 2's Project record entry and clause 6.1 cover calibrations and conventions too | run 81 | `open` | | `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind; named again by four readers of run 81 | run 76 | `open` | | `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | From cdadc77e7c444a23def07231e4a5eb93bb544686 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:53:12 +0100 Subject: [PATCH 41/52] Toolchain section 2: an undocumented blocking rule is asked nothing the document asks of defences After run 83: the closing paragraph says that nothing this document asks of defences, clause 5.1's listing included, is asked of a blocking rule whose documentation is missing, and that a toolchain running one has failed clause 4.2 whether or not its listing shows it. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 6 +++++- TOOLING-SPEC.md | 6 ++++-- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6875767..ef8dbe0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -179,7 +179,11 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be Clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) says nothing of undocumented rules now, section [2](TOOLING-SPEC.md#2-terminology)'s closing paragraph being where that is said; clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) and section [1](TOOLING-SPEC.md#1-what-this-document-is-for) name the verdict as one given under the detector specification's conformance section and say that without one the restated sentences are the test. - Accepted by run NNN. + [Run 83](acceptance/runs/083-toolchain-contradictions-run-11.md) failed on section [2](TOOLING-SPEC.md#2-terminology)'s + closing paragraph and on clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) set against it, three readers asking whether an + undocumented blocking rule has to appear in the listing. The paragraph now answers: nothing + the document asks of defences, the listing included, is asked of such a rule, and a toolchain + that runs one has failed clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) whether or not its listing shows it. Accepted by run NNN. ### 0.2.0, 2026-09-08 diff --git a/TOOLING-SPEC.md b/TOOLING-SPEC.md index b7dbce4..27f6ede 100644 --- a/TOOLING-SPEC.md +++ b/TOOLING-SPEC.md @@ -86,8 +86,10 @@ The distinction between a [Bundled defence] and a project's own is significant t [Practitioner] needs in order to act on it has to travel with it. Throughout this document, a [Blocking] [Rule] whose documentation is missing is not yet a -[Defence]. Clauses 4.2 and 8.1 exist to catch it, so where they reach a [Toolchain]'s [Rules] -they reach every [Blocking] [Rule] it runs, documented or not. +[Defence], and nothing this document asks of [Defences], clause 5.1's listing included, is asked +of it. It is what clauses 4.2 and 8.1 exist to catch: where they reach a [Toolchain]'s [Rules] +they reach every [Blocking] [Rule] it runs, documented or not, and a [Toolchain] that runs such +a [Rule] has failed clause 4.2 whether or not its listing shows it. ## 3. The division of responsibility From c17a1790319dcdd11e694842f9981fd5043d62b2 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:58:13 +0100 Subject: [PATCH 42/52] Run 084: toolchain contradictions cohort, failed on clause 4.1 and section 2's closing paragraph Every keyed item five of five. Three findings in text the change wrote or altered: clause 4.1's first paragraph and its point 2, readers unable to tell which of two tests governs, and section 2's closing paragraph. Section 2's Project record entry against clauses 6.1, 6.2 and 6.4 (two readers) routed to acceptance/ROUTED.md. Co-Authored-By: Claude Fable 5.1 --- .../084-toolchain-contradictions-run-12.md | 170 ++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100644 acceptance/runs/084-toolchain-contradictions-run-12.md diff --git a/acceptance/runs/084-toolchain-contradictions-run-12.md b/acceptance/runs/084-toolchain-contradictions-run-12.md new file mode 100644 index 0000000..9aabaaa --- /dev/null +++ b/acceptance/runs/084-toolchain-contradictions-run-12.md @@ -0,0 +1,170 @@ +# Run 84: TOOLING-SPEC.md, toolchain contradictions, twelfth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: not yet opened at the time of the run; it is to be opened from branch + `editor/toolchain-contradictions`. The readers read commit c2e3249. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 83 in section 2's closing paragraph alone: it now says that nothing this + document asks of defences, clause 5.1's listing included, is asked of a blocking rule whose + documentation is missing, and that a toolchain running one has failed clause 4.2 whether or + not its listing shows it. +- **Run within the pull request**: 12 (runs 57, 58, 62, 65, 67, 68, 72, 76, 81, 82 and 83 came + before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `eb3d2cc18c08b54960dd96080fcac5066e04ef556aaa983194c13e79bef930dc`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at c2e3249, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`, which + the marker confirmed byte for byte. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (all five), each saying the +two `(none)` rules are not Defences under section 2 and so outside 5.1 (readers 3, 4 and 5 add +that clause 4.2 catches them); 6.1 Yes (readers 1, 2, 3 and 5) and Partial (reader 4, mentioning +that section 2's Project record also holds calibrations and conventions and the fixture does not +say where those are, which meets the key's condition); 6.2 No (all five), each mentioning that +nothing rejects a generic justification or generic text. All five grade 4.1 No on the two +undocumented bundled rules with nothing supplied for them, and 4.2 No; readers 2, 3, 4 and 5 +read the two rules as Blocking because `zig-qa run` stops when Lintomatic fails, which the key's +4.2 row reads as a No, and reader 1 grades No outright. + +Question 1: correct clauses named per reader within the first six bullets. Reader 1: 4.1 with +4.2; 4.3 with 4.5; 4.4; 5.1 to 5.3; 6.1 to 6.3. Reader 2: 4.1; 4.2; 4.3 with 6.2; 4.4 and 4.5; +5.1 to 5.3 with 6.3. Reader 3: 4.1 with 4.2; 4.3; 4.5; 5.1 to 5.3; 6.1 to 6.3. Reader 4: 4.1 +with 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 with 6.3; 6.2. Reader 5: 4.1 with 4.2; 4.3 with 4.5; 4.4; +5.1 to 5.3 with 6.1 and 6.2; 6.3. Five correct bullets each. No bullet asserts anything the key +marks wrong. Every reader's first bullet (the toolchain judged as one assembled whole), reader +2's and reader 4's remarks that the agent summary is a SHOULD, reader 4's that a declaration is +optional, and readers 4's and 5's that a shipped artefact gets its own grade are counted neither +way, as the key says. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through"; for 2b all five quote section 9's "The project-level verdict is a single grade; ..."; +for 2c all five quote clause 9.1's "The two verdicts MUST be graded and declared separately, and +neither implies the other." No answer gives a grade per part or a single grade for 2c. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says the toolchain does not conform at +all, or that there is no summary, or both (all five); S8 says Zigpipe ships no tooling to other +projects (all five); Decl says the declaration is optional, or that the verdict rests on the +exercise of the clauses (all five). + +### Misses + +None. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..c2e3249 (9e914fe +being the merge base with `next`) adds, rewords, moves or deletes a sentence in the paragraph, +changes a word, term or link in it, or parts it from what it refers to. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------- | +| 1 | Clause 4.1, first paragraph: "Where it has not, the sentences that follow are the test" (2, 3), "A grader of this clause checks the sentences that follow" (3, 5), "this clause holds the [Detector] to every one of them" (3; 5 quotes "every one of them"); whether the restated sentences or every MUST of the detector specification is the test | 2, 3, 5 | yes: rewritten by the diff | +| 2 | Clause 4.1, point 2: "Does the [Detector] meet every MUST of sections 4 to 7 of the [detector specification](DETECTOR-SPEC.md), exercised as its clause 8.1 describes?" (2), "every MUST of sections 4 to 7 … exercised as its clause 8.1 describes" (5); set against passage 1 | 2, 5 | yes: the diff rewords sentences in the item | +| 3 | Section 2, closing paragraph: "not yet a [Defence]" and "clause 5.1's listing included", with the paragraph identified by its opening words, set against clause 5.1 (3); "where they reach a [Toolchain]'s [Rules] they reach every [Blocking] [Rule] it runs, documented or not", which clauses "they" refers to (5) | 3, 5 | yes: added by the diff, and the revision after run 83 | +| 4 | Section 2, the Project record entry: described as listing the project-record judgements, exceptions among them, set against clauses 6.2 and 6.4 (1); "[Calibrations], [Exceptions] and conventions", set against clause 6.1 (4) | 1, 4 | no | +| 5 | Clause 6.2, first paragraph: "MUST NOT supply a default", set against clause 6.4 | 1 | no | +| 6 | Clause 6.4 heading: "SHOULD state its own defaults for anything the method leaves to the project" | 1 | no | +| 7 | Clause 5.1, first paragraph: "MUST list every [Defence] active", set against passage 3 | 3 | no in itself; set against passage 3, which the diff added | +| 8 | Clause 4.3, first paragraph: "A route bypasses the [Project record] when a finding can be silenced through it without an entry appearing in the [Project record]"; whether disabling a whole rule in configuration is a suppression route | 4 | yes: the diff adds a sentence to the paragraph | +| 9 | Section 2, the Exception row of the table: "a recorded, justified decision to leave an [Instance] unfixed or a [Rule] unapplied", set against passage 8 | 4 | no | +| 10 | Clause 6.1, first paragraph: "The [Toolchain] MUST define where the [Project record] lives and MUST load it"; one location for all three kinds or not | 4 | no | +| 11 | Clause 9.1, first paragraph: "MUST be graded and declared separately", set against passage 12 | 5 | yes: the diff rewords sentences in the paragraph | +| 12 | Clause 9.2, third paragraph: "The declaration is optional", set against passage 11 | 5 | yes: the diff extends the sentence quoted | + +Notes, not passages: reader 4's third item, "exercised as its clause 8.1 describes" with clause +4.1, is a remark that the detector specification was not supplied, and so a note however much +of the document it quotes; under question 3, readers 1, 2, 3 and 4 remark the same of 4.1 or +4.4. Passages 5 and 6 are different paragraphs (a heading is a paragraph of its own). + +## Findings and where they lie + +- **Finding A, passage 1, clause 4.1 first paragraph** (readers 2, 3 and 5): the diff rewrites + the paragraph. Text the change wrote or altered. Blocks. +- **Finding B, passage 2, clause 4.1 point 2** (readers 2 and 5): the diff rewords the item's + sentences, and both readers set it against passage 1. Text the change wrote or altered. Blocks. +- **Finding C, passage 3, section 2 closing paragraph** (readers 3 and 5): the diff adds the + paragraph, and its revision after run 83 is the only change since that run. Text the change + wrote or altered. Blocks. +- **Finding D, passage 4, section 2's Project record entry** (readers 1 and 4): unchanged by the + diff, not parted from anything it refers to, and each reader sets it only against unchanged + paragraphs (reader 1 against 6.2 and 6.4, reader 4 against 6.1). Elsewhere: a finding against + the document, to be recorded in `acceptance/ROUTED.md`, where the open rows from runs 76 and 81 + on clause 6.1 and on 6.2 and 6.3 against this entry are the nearest; routed, and does not + block. The marker writes only this record; the ROUTED entry is for the editor. + +Passages 5 to 12 are each named by one reader: recorded, not findings, applied at the editor's +discretion. + +## Points the rules did not settle + +1. **Whether a paragraph cited under question 4 as context is named.** Readers 1 and 4 name the + Project record entry as what they set another clause against, not as itself confusing. + Judgement: the Marking section defines naming by quotation or by a clause number with an + identifying description, not by what the reader calls confusing, so it is named; had it + stayed open, the open-point rule counts the passage. Either way finding D lies elsewhere and + does not block. +2. **Whether reader 4 names clause 6.2 or 6.3** by "Clauses 6.2 and 6.3 speak only of + exceptions". Judgement: not named, because the description identifies a clause and no + sentence or paragraph within it, which the Marking section requires. Had it been counted, it + would at most join passage 5 in a finding on unchanged text set against unchanged text, which + is routed and does not block; the verdict is the same either way. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit a095fa7, named before the readers started. The process files at +c2e3249 (`ACCEPTANCE.md`, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md`, `tools/changes.py`, `spec-qa.py` and the CI workflows) are identical to those +at a095fa7. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: three (findings A, + B and C). Not met. +- Passages named by two or more readers elsewhere: finding D, routed to `acceptance/ROUTED.md`; + does not block. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A, B and C; every keyed item passed its threshold. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From 852dcd80f61a418609cd36375ca7631d3648a92e Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:58:23 +0100 Subject: [PATCH 43/52] ROUTED: run 84 names the toolchain Project record entry against clauses 6.1, 6.2 and 6.4 Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 54 ++++++++++++++++++++++---------------------- 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 5ae6e16..1423a48 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -15,33 +15,33 @@ untouched text of the change those runs tested, and are listed here so that they ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | -------------------------------------------------- | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), the first paragraph | "It MUST NOT supply a default, and MUST reject an Exception that omits one" against clause [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)'s defaults | run 83 | `open` | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) and [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Both clauses speak only of exceptions, where section 2's Project record entry and clause 6.1 cover calibrations and conventions too | run 81 | `open` | -| `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind; named again by four readers of run 81 | run 76 | `open` | -| `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | -| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68, 82 | `open` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | -| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------- | -------------------------------------------------- | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), the first paragraph | "It MUST NOT supply a default, and MUST reject an Exception that omits one" against clause [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)'s defaults | run 83 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) and [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Both clauses speak only of exceptions, where section 2's Project record entry and clause 6.1 cover calibrations and conventions too | run 81 | `open` | +| `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind; named again by four readers of run 81, and by two readers of run 84 as the Project record entry against clauses 6.1, 6.2 and 6.4 | run 76 | `open` | +| `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | +| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68, 82 | `open` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | +| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | ## Closed From 9a06b9f1f449d0171a47d08b75829a506779a4fb Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:58:25 +0100 Subject: [PATCH 44/52] Toolchain 4.1: one test, every MUST of sections 4 to 7; section 2's paragraph in plain sentences After run 84: clause 4.1 holds the detector to every MUST of the detector specification's sections 4 to 7 and says the restated sentences are where to look first and not the whole test, a grader without that document checking only them and saying so; section 1 says the same. Section 2's closing paragraph says clause 5.1 does not require an undocumented blocking rule to be listed, and that clauses 4.2 and 8.1 reach it. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 10 +++++++++- TOOLING-SPEC.md | 26 +++++++++++++------------- 2 files changed, 22 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ef8dbe0..d5457c8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -183,7 +183,15 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be closing paragraph and on clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) set against it, three readers asking whether an undocumented blocking rule has to appear in the listing. The paragraph now answers: nothing the document asks of defences, the listing included, is asked of such a rule, and a toolchain - that runs one has failed clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) whether or not its listing shows it. Accepted by run NNN. + that runs one has failed clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) whether or not its listing shows it. [Run + 84](acceptance/runs/084-toolchain-contradictions-run-12.md) failed on clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), three + readers unable to tell whether "every one of them" or the restated sentences set the test, and + on the paragraph again, one reader still asking whether such a rule is listed and one unable + to place "they". Clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) now has one test, every MUST of sections 4 to 7, with the + restated sentences as where to look first and a grader without the detector specification + checking only them and saying so; section [1](TOOLING-SPEC.md#1-what-this-document-is-for) says the same; the paragraph says in plain + sentences that clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) does not require such a rule to be listed and that clauses [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) and + [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) reach it. Accepted by run NNN. ### 0.2.0, 2026-09-08 diff --git a/TOOLING-SPEC.md b/TOOLING-SPEC.md index 27f6ede..cbe13fd 100644 --- a/TOOLING-SPEC.md +++ b/TOOLING-SPEC.md @@ -41,8 +41,9 @@ part of its [Toolchain] and are judged as such. **How to read this document.** A [Toolchain] can be graded from this document alone, on the evidence a project shows. Where a clause rests on the [detector specification](DETECTOR-SPEC.md), -as clauses 4.1, 4.2 and 4.4 do, it restates what the grader checks, and a verdict the [Detector] -already holds under that document's section 8 counts too. The +as clauses 4.1, 4.2 and 4.4 do, it says what it holds the [Detector] to and restates the parts +most often failed; a grader without that document to hand checks what is restated and says so. +The table in section 2 glosses every term it borrows. Each clause states its obligation in its heading and first sentence, and the **Why** paragraph beneath is reasoning, not a further requirement. A link is there for depth and is not a prerequisite for the sentence that carries it. @@ -86,10 +87,9 @@ The distinction between a [Bundled defence] and a project's own is significant t [Practitioner] needs in order to act on it has to travel with it. Throughout this document, a [Blocking] [Rule] whose documentation is missing is not yet a -[Defence], and nothing this document asks of [Defences], clause 5.1's listing included, is asked -of it. It is what clauses 4.2 and 8.1 exist to catch: where they reach a [Toolchain]'s [Rules] -they reach every [Blocking] [Rule] it runs, documented or not, and a [Toolchain] that runs such -a [Rule] has failed clause 4.2 whether or not its listing shows it. +[Defence]. Clause 5.1 does not require it to be listed, because that clause lists [Defences]. +Clauses 4.2 and 8.1 reach every [Blocking] [Rule] a [Toolchain] runs, documented or not, and a +[Toolchain] that runs such a [Rule] fails clause 4.2. ## 3. The division of responsibility @@ -119,17 +119,17 @@ any row and the failure mode is visible too: either half alone leaves the [Pract A [Toolchain] MUST NOT route a [Defence] through a [Detector] that does not [Conform] to [the detector specification](DETECTOR-SPEC.md). That document's MUSTs are those of -its sections 4 to 7, and this clause holds the [Detector] to every one of them. A grader of this -clause checks the sentences that follow, which restate the ones most often failed. Where the -[Detector] has already been graded against that document, under its section 8, that verdict -counts too: a [Detector] found not to [Conform] fails this clause whether or not the sentences -that follow show why. Where it has not, the sentences that follow are the test. The +its sections 4 to 7, and this clause holds the [Detector] to every one of them: a [Detector] +that fails any of them, after wrapping, fails this clause. The sentences that follow restate +the ones most often failed, so that a grader knows where to look first; they are not the whole +test, and a grader without that document to hand can check only them, and says so in the +verdict. The [Detector] hosts bespoke [Rules]. It runs one [Rule] on its own against supplied code. It prints a stable [Identifier] with every finding. It runs locally, down to one file, with the result in the command's own output and nothing reportable only through a hosted service. It resolves every [Identifier] a [Bundled rule] can print to documentation shipped with it, offline. -It makes any inline [Suppression] route detectable or disableable. A [Detector] that fails any -one of them, after wrapping, fails this clause. Three points govern how that is met. +It makes any inline [Suppression] route detectable or disableable. Three points govern how that +is met. 1. **Wrapping is permitted.** The [Toolchain] MAY supply a mechanism the [Detector] lacks by wrapping it, with a [Harness] script or a From 58924a8762ed0f939b38ec7b6fac884af6d70245 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 14:58:37 +0100 Subject: [PATCH 45/52] Changelog: clause links point at the headings as they are Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d5457c8..21fe83b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -187,7 +187,7 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be 84](acceptance/runs/084-toolchain-contradictions-run-12.md) failed on clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), three readers unable to tell whether "every one of them" or the restated sentences set the test, and on the paragraph again, one reader still asking whether such a rule is listed and one unable - to place "they". Clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) now has one test, every MUST of sections 4 to 7, with the + to place "they". Clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) now has one test, every MUST of the detector specification's authoring, reporting, resolving and suppression sections, with the restated sentences as where to look first and a grader without the detector specification checking only them and saying so; section [1](TOOLING-SPEC.md#1-what-this-document-is-for) says the same; the paragraph says in plain sentences that clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) does not require such a rule to be listed and that clauses [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) and From 6e5697901bfb0f4f530a6a56ef0a41ad353d4988 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 15:02:27 +0100 Subject: [PATCH 46/52] Run 085: toolchain contradictions cohort, failed on section 2's closing paragraph against clause 8.1 Every keyed item five of five. Two findings in text the change wrote or altered: section 2's closing paragraph, "Clauses 4.2 and 8.1 reach every Blocking Rule", and clause 8.1's first paragraph set against it, whose exclusion of a third-party catalogue it contradicted. Clause 6.2's first paragraph against clause 6.4 (two readers) routed to acceptance/ROUTED.md. Co-Authored-By: Claude Fable 5.1 --- .../085-toolchain-contradictions-run-13.md | 168 ++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 acceptance/runs/085-toolchain-contradictions-run-13.md diff --git a/acceptance/runs/085-toolchain-contradictions-run-13.md b/acceptance/runs/085-toolchain-contradictions-run-13.md new file mode 100644 index 0000000..705a277 --- /dev/null +++ b/acceptance/runs/085-toolchain-contradictions-run-13.md @@ -0,0 +1,168 @@ +# Run 85: TOOLING-SPEC.md, toolchain contradictions, thirteenth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: not yet opened at the time of the run; it is to be opened from branch + `editor/toolchain-contradictions`. The readers read commit 58924a8. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 84: clause 4.1's first paragraph now has one test, every MUST of the detector + specification's sections 4 to 7, with the restated sentences as where a grader looks first and + not the whole test, a grader without that document checking only them and saying so; section + 1's "How to read this document" says the same; section 2's closing paragraph says in plain + sentences that clause 5.1 does not require an undocumented blocking rule to be listed and that + clauses 4.2 and 8.1 reach it. +- **Run within the pull request**: 13 (runs 57, 58, 62, 65, 67, 68, 72, 76, 81, 82, 83 and 84 + came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `6d37cc625287ae0258a845657c48a96b7a53c698d998d6cde43c5c9e064a53ea`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at 58924a8, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`, which + the marker confirmed byte for byte. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (all five; readers 4 and 5 +add that the two `(none)` rules are not Defences under section 2 and so outside 5.1); 6.1 +Partial (readers 1 and 5, each mentioning that the Project record also holds calibrations and +conventions and the fixture gives no location for those, which meets the key's condition) and +Yes (readers 2, 3 and 4); 6.2 No (all five), each mentioning that nothing rejects a generic +justification or generic phrases. All five grade 4.1 No on the two undocumented bundled rules +with nothing supplied for them, each saying the detector specification was not supplied and +only the restated points were checked, which the key's 4.1 row accepts. All five grade 4.2 No; +readers 1, 2, 3 and 5 state that they read the two rules as Blocking because `zig-qa run` stops +when Lintomatic fails, which the key's 4.2 row reads as a No, and reader 4 states they are +Blocking for the same reason. + +Question 1: correct clauses named per reader within the first six bullets. Reader 1: 4.1 with +4.2; 4.3 with 4.4 and 4.5; 5.1 to 5.3; 6.1 to 6.3. Reader 2: 4.1; 4.2; 4.3 with 6.1 and 6.2; +4.4 and 4.5; 5.1, 5.2 and 6.3. Reader 3: 4.1 with 4.2; 4.3 with 4.4 and 4.5; 5.1 to 5.3; 6.1 to +6.3. Reader 4: the governing principle with 4.1 and 4.2; 4.3 with 4.5; 5.1 to 5.3 with 6.3; 6.1 +and 6.2. Reader 5: 4.1; 4.2; 4.3 with 6.2; 5.1, 5.2, 6.1 and 6.3; 4.4 and 4.5. Four or more +correct bullets each. No bullet asserts anything the key marks wrong. Every reader's first +bullet (the toolchain judged as one assembled whole), the remarks that section 7 or clause 6.4 +is a SHOULD or recommended (readers 1, 2, 3 and 5), that a declaration is optional or not a +condition (readers 1, 3 and 4), and that a shipped artefact gets its own grade (readers 1, 3, 4 +and 5) are counted neither way, as the key says. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through"; for 2b all five quote section 9's "The project-level verdict is a single grade; ..."; +for 2c all five quote clause 9.1's "The two verdicts MUST be graded and declared separately, and +neither implies the other." No answer gives a grade per part, or a single grade for 2c. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says the toolchain does not conform at +all, or that there is no summary, or both (all five); S8 says Zigpipe ships no tooling to other +projects (all five); Decl says the declaration is optional, or that the verdict rests on the +exercise of the clauses (all five). + +### Misses + +None. No answer matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..58924a8 (9e914fe +being the merge base with `next`) adds, rewords, moves or deletes a sentence in the paragraph, +changes a word, term or link in it, or parts it from what it refers to. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------- | +| 1 | Section 2, closing paragraph: "Clauses 4.2 and 8.1 reach every [Blocking] [Rule] a [Toolchain] runs, documented or not" (3, 5, each set against passage 2); "a [Blocking] [Rule] whose documentation is missing is not yet a [Defence]. Clause 5.1 does not require it to be listed, because that clause lists [Defences]" (4, set against passage 3) | 3, 4, 5 | yes: added by the diff, and reworded in the revision after run 84 | +| 2 | Clause 8.1, first paragraph: "A third-party [Detector]'s native catalogue, which the [Toolchain] orchestrates without claiming as its own, is outside this audit", set against passage 1 | 3, 5 | yes: the diff rewords a sentence in the paragraph; set against passage 1 | +| 3 | Section 2, the Defence row of the table: "A [Rule] with its documentation, in force and [Blocking]", set against passage 1 | 4 | the row's words unchanged (the table is realigned); set against passage 1 | +| 4 | Clause 6.2, first paragraph: "It MUST NOT supply a default" (1, set against passage 5; 2, quoting the whole sentence, set against passage 6) | 1, 2 | no | +| 5 | Clause 6.4 heading: "The toolchain SHOULD state its own defaults for anything the method leaves to the project" | 1 | no | +| 6 | Clause 6.4, first paragraph: "a documented [Toolchain] default is what the [Practitioner] falls back to" | 2 | no | +| 7 | Section 3, the Identifier resolution row: "clause 4.2 here for the rest", set against passage 8 | 2 | no in itself; set against passage 8, which the diff wrote | +| 8 | Clause 4.2, first paragraph: "every [Blocking] [Rule] the [Toolchain] runs, the project's own and the bundled alike", set against passage 7 | 2 | yes: added by the diff | +| 9 | Clause 6.2, last paragraph: "That check cannot verify truth, and a [Toolchain]'s [Conformance] MUST NOT be read as having verified it" (what "it" refers to) and "MUST reject a justification that could be pasted onto any [Exception] unchanged, by a check it documents" (whether the phrases are examples or the full set) | 4 | no | + +Notes, not passages: none under question 4. Under question 3, every reader remarks that the +detector specification was not supplied; those remarks are not question 4 answers and are not +marked as passages. Passages 4, 5 and 6 are three paragraphs (a heading is a paragraph of its +own), so readers 1 and 2 name the same passage only in passage 4. + +## Findings and where they lie + +- **Finding A, passage 1, section 2 closing paragraph** (readers 3, 4 and 5): the diff adds the + paragraph and the revision after run 84 rewrote it. Text the change wrote or altered. Blocks. + Readers 3 and 5 read its claim that clause 8.1 reaches every Blocking Rule a Toolchain runs as + contradicting clause 8.1's exclusion of a third-party Detector's native catalogue. +- **Finding B, passage 2, clause 8.1 first paragraph** (readers 3 and 5): the diff rewords the + sentence on what the check covers ("as its own, whatever kind of [Detector] carries it, and a + missing page is what the check exists to find"), and both readers set the paragraph against + passage 1, which the diff wrote. Text the change wrote or altered. Blocks. +- **Finding C, passage 4, clause 6.2 first paragraph** (readers 1 and 2): unchanged by the diff, + not parted from anything it refers to, and each reader sets it only against unchanged text + (reader 1 against the clause 6.4 heading, reader 2 against clause 6.4's first paragraph). + Elsewhere: a finding against the document, to be recorded in `acceptance/ROUTED.md`, where the + open row from run 83 on this sentence against clause 6.4's defaults is the same passage; + routed, and does not block. The marker writes only this record; the ROUTED entry is for the + editor. + +Passages 3 and 5 to 9 are each named by one reader: recorded, not findings, applied at the +editor's discretion. + +## Points the rules did not settle + +1. **Whether reader 4's third item names passage 1 as confusing**, since the reader says it read + the paragraph and the Defence row as consistent after checking both. Judgement: the Marking + section defines naming by quotation or by a clause number with an identifying description, + not by what the reader calls confusing, so it is named. The verdict does not turn on it: + readers 3 and 5 alone make passage 1 a finding. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 852dcd8, named before the readers started. The process files at +58924a8 (`ACCEPTANCE.md`, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md`, `tools/changes.py`, `spec-qa.py` and the CI workflows) are identical to those +at 852dcd8. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: two (findings A + and B). Not met. +- Passages named by two or more readers elsewhere: finding C, routed to `acceptance/ROUTED.md`; + does not block. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A and B; every keyed item passed its threshold. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From 568e323782847e3f03baeded0303a0979d7d41da Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 15:02:37 +0100 Subject: [PATCH 47/52] ROUTED: run 85 names the toolchain 6.2 first paragraph against clause 6.4 again Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index 1423a48..fe7182a 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -17,7 +17,7 @@ untouched text of the change those runs tested, and are listed here so that they | Document | Clause | Passage | Raised by | Status | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------- | -------------------------------------------------- | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), the first paragraph | "It MUST NOT supply a default, and MUST reject an Exception that omits one" against clause [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)'s defaults | run 83 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), the first paragraph | "It MUST NOT supply a default, and MUST reject an Exception that omits one" against clause [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)'s defaults; named again by two readers of run 85 | run 83 | `open` | | `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) and [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Both clauses speak only of exceptions, where section 2's Project record entry and clause 6.1 cover calibrations and conventions too | run 81 | `open` | | `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind; named again by four readers of run 81, and by two readers of run 84 as the Project record entry against clauses 6.1, 6.2 and 6.4 | run 76 | `open` | | `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | From 3d3b2d465b596882991a8dd56a8f4598853631c0 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 15:02:39 +0100 Subject: [PATCH 48/52] Toolchain section 2: the closing paragraph names clause 4.2 alone After run 85: the paragraph on an undocumented blocking rule no longer says clause 8.1 reaches every blocking rule a toolchain runs, which contradicted that clause's exclusion of a third-party detector's native catalogue; clause 4.2 is the clause that reaches them all. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 5 ++++- TOOLING-SPEC.md | 2 +- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 21fe83b..1378604 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -191,7 +191,10 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be restated sentences as where to look first and a grader without the detector specification checking only them and saying so; section [1](TOOLING-SPEC.md#1-what-this-document-is-for) says the same; the paragraph says in plain sentences that clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) does not require such a rule to be listed and that clauses [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) and - [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) reach it. Accepted by run NNN. + [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) reach it. [Run 85](acceptance/runs/085-toolchain-contradictions-run-13.md) failed on that + paragraph's "Clauses 4.2 and 8.1 reach every Blocking Rule", two readers setting it against + clause [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation)'s exclusion of a third-party detector's native catalogue; the paragraph names + clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) alone. Accepted by run NNN. ### 0.2.0, 2026-09-08 diff --git a/TOOLING-SPEC.md b/TOOLING-SPEC.md index cbe13fd..8ed7aa6 100644 --- a/TOOLING-SPEC.md +++ b/TOOLING-SPEC.md @@ -88,7 +88,7 @@ The distinction between a [Bundled defence] and a project's own is significant t Throughout this document, a [Blocking] [Rule] whose documentation is missing is not yet a [Defence]. Clause 5.1 does not require it to be listed, because that clause lists [Defences]. -Clauses 4.2 and 8.1 reach every [Blocking] [Rule] a [Toolchain] runs, documented or not, and a +Clause 4.2 reaches every [Blocking] [Rule] a [Toolchain] runs, documented or not, and a [Toolchain] that runs such a [Rule] fails clause 4.2. ## 3. The division of responsibility From 6da74098250fefbe4092d5149caa9eb990644ff0 Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 15:02:49 +0100 Subject: [PATCH 49/52] Changelog: clause links point at the headings as they are Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1378604..813492f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -192,7 +192,7 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be checking only them and saying so; section [1](TOOLING-SPEC.md#1-what-this-document-is-for) says the same; the paragraph says in plain sentences that clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) does not require such a rule to be listed and that clauses [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) and [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) reach it. [Run 85](acceptance/runs/085-toolchain-contradictions-run-13.md) failed on that - paragraph's "Clauses 4.2 and 8.1 reach every Blocking Rule", two readers setting it against + paragraph's sentence naming both clauses as reaching every blocking rule, two readers setting it against clause [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation)'s exclusion of a third-party detector's native catalogue; the paragraph names clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) alone. Accepted by run NNN. From c1ad65b48b750030252a0d18371fa78c2c32ae6c Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 15:07:51 +0100 Subject: [PATCH 50/52] Run 086: toolchain contradictions cohort passed Every keyed item at threshold, one miss on clause 6.3 tolerated. No finding in text the change wrote or altered. Four findings in untouched text routed to acceptance/ROUTED.md: clause 6.1's first paragraph and section 2's Project record entry (two readers each), clause 6.2's first paragraph against clause 6.4, and clause 6.4 itself. The changelog names the run. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 2 +- .../086-toolchain-contradictions-run-14.md | 187 ++++++++++++++++++ 2 files changed, 188 insertions(+), 1 deletion(-) create mode 100644 acceptance/runs/086-toolchain-contradictions-run-14.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 813492f..34686d0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -194,7 +194,7 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) reach it. [Run 85](acceptance/runs/085-toolchain-contradictions-run-13.md) failed on that paragraph's sentence naming both clauses as reaching every blocking rule, two readers setting it against clause [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation)'s exclusion of a third-party detector's native catalogue; the paragraph names - clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) alone. Accepted by run NNN. + clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) alone. Accepted by [run 86](acceptance/runs/086-toolchain-contradictions-run-14.md). ### 0.2.0, 2026-09-08 diff --git a/acceptance/runs/086-toolchain-contradictions-run-14.md b/acceptance/runs/086-toolchain-contradictions-run-14.md new file mode 100644 index 0000000..2ed5ec9 --- /dev/null +++ b/acceptance/runs/086-toolchain-contradictions-run-14.md @@ -0,0 +1,187 @@ +# Run 86: TOOLING-SPEC.md, toolchain contradictions, fourteenth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: not yet opened at the time of the run; it is to be opened from branch + `editor/toolchain-contradictions`. The readers read commit 6da7409. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 85 in section 2's closing paragraph alone: it now names clause 4.2 alone as + the clause that reaches every blocking rule a toolchain runs, documented or not; clause 8.1 is + no longer named there. +- **Run within the pull request**: 14 (runs 57, 58, 62, 65, 67, 68, 72, 76, 81, 82, 83, 84 and + 85 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `60c864ced29d5011fb73ca0f35139b8bfcc146303dd001952fb43a99bcaa910e`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at 6da7409, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`, which + the marker confirmed byte for byte. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | miss | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (all five; readers 2 and 3 +add that the two `(none)` rules are not Defences under section 2 and so outside 5.1); 6.1 +Partial (readers 1, 4 and 5, each mentioning that the Project record also holds calibrations and +conventions and the fixture gives no location for those, which meets the key's condition; +reader 5 adds that it would be Yes if the record is read as exceptions only, and the Partial is +the grade it gives for the case) and Yes (readers 2 and 3); 6.2 No (all five), each mentioning +that nothing rejects a generic justification or generic phrasing. All five grade 4.1 No on the +two undocumented bundled rules with nothing supplied for them; readers 2, 3, 4 and 5 say the +detector specification was not supplied and only the restated points were checked, which the +key's 4.1 row accepts. All five grade 4.2 No; readers 1, 4 and 5 state that they read the two +rules as Blocking because they are enabled or because `zig-qa run` stops when Lintomatic fails, +which the key's 4.2 row reads as a No. + +Question 1: correct clauses named per reader within the first six bullets. Reader 1: 4.1 with +4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3; 6.1 to 6.3 (five bullets; its first bullet is not needed for +the count and is not decided). Reader 2: 4.1; 4.2; 4.3 with 6.1 and 6.2; 5.1, 5.2, 6.3 and 4.5. +Reader 3: 4.4 and 4.5; 4.1; 4.2; 4.3 with 6.1; 6.2; 5.1 to 5.3 with 6.3. Reader 4: 4.1; 4.2; 4.3 +with 4.5; 5.1 to 5.3 with 6.3; 6.1 and 6.2. Reader 5: 4.1; 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 +with 6.3; 6.1 and 6.2. Four or more correct bullets each. No bullet asserts anything the key +marks wrong. The first bullets of readers 2 and 4 (the toolchain judged as one assembled +whole), the remarks that section 7 or the defaults are SHOULDs (readers 2, 4 and 5), that a +declaration is optional (reader 2) and that a shipping project has two levels (reader 2) are +counted neither way, as the key says. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through"; for 2b all five quote section 9's "The project-level verdict is a single grade; ..."; +for 2c all five quote clause 9.1's "The two verdicts MUST be graded and declared separately, and +neither implies the other." No answer gives a grade per part, or a single grade for 2c. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says the toolchain does not conform at +all, or that there is no summary, or both (all five); S8 says Zigpipe ships no tooling to other +projects, and readers 4 and 5 add that section 8 bears on zig-qa's artefact grade (all five); +Decl says the declaration is optional, or that the verdict rests on the exercise of the clauses +(all five). + +### Misses + +- **Reader 4, clause 6.3: Partial.** The key accepts Yes only, with no condition. The reader + grants that `zig-qa list --exceptions` lists exceptions in the same format as the rule listing, + and grades Partial because calibrations and conventions are not described and so cannot be + shown to be enumerable. The reasoning does not quote the document in that row; it rests on + section 2's Project record entry, which the reader cites by section in its 6.1 row. It matches + no wrong reading the key names. A single miss on the item, the reader's, tolerated by the + threshold: four of five correct. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..6da7409 (9e914fe +being the merge base with `next`) adds, rewords, moves or deletes a sentence in the paragraph, +changes a word, term or link in it, or parts it from what it refers to. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------- | +| 1 | Clause 6.1, first paragraph: "where the Project record lives" (1); "The [Toolchain] MUST define where the [Project record] lives and MUST load it. Not a documentation convention. A path the [Toolchain] loads." (5); each set against passage 2 | 1, 5 | no | +| 2 | Section 2, the Project record terminology entry: "Calibrations, Exceptions and conventions", given with "the definition in section 2" (1) and "Section 2 defines the project record as" (5); each set against passage 1 | 1, 5 | no; the paragraph the diff inserts in section 2 follows it and parts it from nothing it refers to | +| 3 | Clause 5.3, second paragraph: "its entry MAY be sourced from wherever it is enabled" (what "wherever" covers) | 1 | no | +| 4 | Clause 4.5, first paragraph: "A [Blocking] failure at the [Detector] level MUST stop the levels below it from being treated as meaningful, in the order method section 5 states." (what the levels below a detector are) | 2 | no | +| 5 | Clause 6.2, first paragraph: "It MUST NOT supply a default." (2, set against passage 6); "MUST NOT supply a default" (4, set against passage 6) | 2, 4 | no | +| 6 | Clause 6.4: "6.4, which SHOULDs a documented toolchain default" (2, a description); "SHOULD state its own defaults for anything the method leaves to the project" (4, the heading's words) | 2, 4 | no | +| 7 | Clause 4.1, first paragraph: "The sentences that follow restate the ones most often failed, so that a grader knows where to look first; they are not the whole test, and a grader without that document to hand can check only them, and says so in the verdict." (set against passage 8) | 3 | yes: the diff wrote the sentence | +| 8 | Clause 4.1, point 2: "does the [Detector] meet every MUST of sections 4 to 7 of the [detector specification], exercised as its clause 8.1 describes?" (set against passage 7) | 3 | yes: the diff rewords sentences in point 2 | + +Notes, not passages: reader 2's first item quotes clause 4.1 point 2, "with the wrapping in +place, does the [Detector] meet every MUST of sections 4 to 7 ... exercised as its clause 8.1 +describes?", and says only that it could not tell what the exercise is "because the detector +specification was not in what I was given". That is a remark that another document was not +supplied, which the Marking section's rule on question 4 records as a note and not a passage, +"however much of the document it quotes to locate the reference"; it is not counted with +reader 3's passage 8. Reader 2's second item also says method section 5 was not given, but +its remark that the clause does not define the levels below a detector is about the passage, so +passage 4 is counted. Under question 3, readers 2, 3, 4 and 5 remark that the detector +specification was not supplied; those are not question 4 answers. + +## Findings and where they lie + +- **Finding A, passage 1, clause 6.1 first paragraph** (readers 1 and 5): unchanged by the diff, + not parted from anything it refers to, and each reader sets it only against passage 2, which + is unchanged. Elsewhere: a finding against the document, recorded in `acceptance/ROUTED.md`, + where the open row on clause 6.1's first paragraph against section 2's Project record entry is + the same passage; routed, and does not block. +- **Finding B, passage 2, section 2's Project record entry** (readers 1 and 5): unchanged by the + diff; the paragraph the diff inserts at the end of section 2 comes after the entry and the + paragraph that follows it, and parts the entry from nothing it refers to; each reader sets it + only against passage 1. Elsewhere: routed to `acceptance/ROUTED.md` with finding A, whose open + row names both; does not block. +- **Finding C, passage 5, clause 6.2 first paragraph** (readers 2 and 4): unchanged by the diff, + not parted from anything it refers to, each reader setting it only against clause 6.4, which + is unchanged. Elsewhere: routed to `acceptance/ROUTED.md`, where the open row on this sentence + against clause 6.4's defaults is the same passage; does not block. +- **Finding D, passage 6, clause 6.4** (readers 2 and 4, counted as the same passage under the + open-point rule; see below): unchanged by the diff. Elsewhere: routed to + `acceptance/ROUTED.md` with finding C; does not block. + +The marker writes only this record; the ROUTED entries are for the editor. Passages 3, 4, 7 and +8 are each named by one reader: recorded, not findings, applied at the editor's discretion. +Passages 7 and 8 lie in text the change wrote, and are named by reader 3 alone. + +## Points the rules did not settle + +1. **Whether readers 2 and 4 name the same paragraph of clause 6.4.** Reader 4 quotes the + heading's words; reader 2 describes 6.4 as one "which SHOULDs a documented toolchain + default", which could identify the heading (its SHOULD) or the first paragraph ("a documented + [Toolchain] default"), and a heading is a paragraph of its own. Marked in the direction the + open-point rule gives: counted as the same passage, so finding D. The verdict does not turn on + it, since the passage lies outside the change either way. + +Considered and settled by a rule, so not open: reader 2's clause 4.1 item is a note under the +Marking section's question 4 rule, as above; and readers 1 and 5 each name both passage 1 and +passage 2, since each gives a clause or section number with words that identify each paragraph. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 568e323, named before the readers started. The process files at +6da7409 (`ACCEPTANCE.md`, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md`, `tools/changes.py`, `spec-qa.py` and the CI workflows and pull request +template under `.github/`) are identical to those at 568e323. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item but + 6.3, which is four of five. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: none. Met. +- Passages named by two or more readers elsewhere: findings A to D, routed to + `acceptance/ROUTED.md`; do not block. + +## Findings applied + +None at the time of marking. + +## Verdict + +PASS. Every keyed item met its threshold, no miss matches a wrong reading the key names, and +no finding lies in text the change wrote or altered. + +## Answer sheets + +Posted on the pull request: LINK TO BE ADDED. From 40f7895867706c8bfdc7f5cb27a9ff1507ba2a8c Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 15:08:03 +0100 Subject: [PATCH 51/52] ROUTED: run 86's findings in untouched toolchain text, clause 6.4 against 6.2 new Co-Authored-By: Claude Fable 5.1 --- acceptance/ROUTED.md | 55 ++++++++++++++++++++++---------------------- 1 file changed, 28 insertions(+), 27 deletions(-) diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md index fe7182a..e53b671 100644 --- a/acceptance/ROUTED.md +++ b/acceptance/ROUTED.md @@ -15,33 +15,34 @@ untouched text of the change those runs tested, and are listed here so that they ## Open -| Document | Clause | Passage | Raised by | Status | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------- | -------------------------------------------------- | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), the first paragraph | "It MUST NOT supply a default, and MUST reject an Exception that omits one" against clause [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)'s defaults; named again by two readers of run 85 | run 83 | `open` | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) and [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Both clauses speak only of exceptions, where section 2's Project record entry and clause 6.1 cover calibrations and conventions too | run 81 | `open` | -| `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind; named again by four readers of run 81, and by two readers of run 84 as the Project record entry against clauses 6.1, 6.2 and 6.4 | run 76 | `open` | -| `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | -| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | -| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | -| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | -| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68, 82 | `open` | -| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | -| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | -| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | -| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | -| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | -| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | -| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | -------------------------------------------------- | +| `TOOLING-SPEC.md` | [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project), the heading and first paragraph | "SHOULD state its own defaults for anything the method leaves to the project" against clause [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one)'s "MUST NOT supply a default" for a justification | run 86 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), the first paragraph | "It MUST NOT supply a default, and MUST reject an Exception that omits one" against clause [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)'s defaults; named again by two readers of run 85 and two of run 86 | run 83 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) and [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Both clauses speak only of exceptions, where section 2's Project record entry and clause 6.1 cover calibrations and conventions too | run 81 | `open` | +| `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind; named again by four readers of run 81, and by two readers of run 84 as the Project record entry against clauses 6.1, 6.2 and 6.4; and by two readers of run 86, the entry and clause 6.1's first paragraph together | run 76 | `open` | +| `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | +| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68, 82 | `open` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | +| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | ## Closed From 241ca4bd52a9fb20b1da938d1055065f3b0f4c1e Mon Sep 17 00:00:00 2001 From: joseph Date: Mon, 12 Oct 2026 15:11:31 +0100 Subject: [PATCH 52/52] Run records: links to the answer sheets posted on the pull request Co-Authored-By: Claude Fable 5.1 --- acceptance/runs/057-toolchain-contradictions-run-1.md | 4 ++-- acceptance/runs/058-toolchain-contradictions-run-2.md | 4 ++-- acceptance/runs/062-toolchain-contradictions-run-3.md | 4 ++-- acceptance/runs/065-toolchain-contradictions-run-4.md | 4 ++-- acceptance/runs/067-toolchain-contradictions-run-5.md | 4 ++-- acceptance/runs/068-toolchain-contradictions-run-6.md | 4 ++-- acceptance/runs/072-toolchain-contradictions-run-7.md | 4 ++-- acceptance/runs/076-toolchain-contradictions-run-8.md | 4 ++-- acceptance/runs/081-toolchain-contradictions-run-9.md | 4 ++-- acceptance/runs/082-toolchain-contradictions-run-10.md | 4 ++-- acceptance/runs/083-toolchain-contradictions-run-11.md | 4 ++-- acceptance/runs/084-toolchain-contradictions-run-12.md | 4 ++-- acceptance/runs/085-toolchain-contradictions-run-13.md | 4 ++-- acceptance/runs/086-toolchain-contradictions-run-14.md | 4 ++-- 14 files changed, 28 insertions(+), 28 deletions(-) diff --git a/acceptance/runs/057-toolchain-contradictions-run-1.md b/acceptance/runs/057-toolchain-contradictions-run-1.md index b222a30..fdd3244 100644 --- a/acceptance/runs/057-toolchain-contradictions-run-1.md +++ b/acceptance/runs/057-toolchain-contradictions-run-1.md @@ -1,7 +1,7 @@ # Run 57: TOOLING-SPEC.md, toolchain contradictions, first run - **Document**: TOOLING-SPEC.md -- **Pull request**: not yet opened at the time of the run; it is to be opened from branch +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch `editor/toolchain-contradictions`. The readers read commit 4d6ebbd. Change: section 9 against clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; @@ -145,4 +145,4 @@ None at the time of marking; for the editor, under "After a failure" in `ACCEPTA ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125956319); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125955854); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125956709). diff --git a/acceptance/runs/058-toolchain-contradictions-run-2.md b/acceptance/runs/058-toolchain-contradictions-run-2.md index 51bf6e2..7adaa89 100644 --- a/acceptance/runs/058-toolchain-contradictions-run-2.md +++ b/acceptance/runs/058-toolchain-contradictions-run-2.md @@ -1,7 +1,7 @@ # Run 58: TOOLING-SPEC.md, toolchain contradictions, second run - **Document**: TOOLING-SPEC.md -- **Pull request**: not yet opened at the time of the run; it is to be opened from branch +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch `editor/toolchain-contradictions`. The readers read commit 90a9936. Change: section 9 against clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; @@ -147,4 +147,4 @@ FAIL, on findings A to D alone; every keyed item passed. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125958931); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125957078); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125959274). diff --git a/acceptance/runs/062-toolchain-contradictions-run-3.md b/acceptance/runs/062-toolchain-contradictions-run-3.md index 8564b18..4d083f1 100644 --- a/acceptance/runs/062-toolchain-contradictions-run-3.md +++ b/acceptance/runs/062-toolchain-contradictions-run-3.md @@ -1,7 +1,7 @@ # Run 62: TOOLING-SPEC.md, toolchain contradictions, third run - **Document**: TOOLING-SPEC.md -- **Pull request**: not yet opened at the time of the run; it is to be opened from branch +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch `editor/toolchain-contradictions`. The readers read commit 352cfcc. Change: section 9 against clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; @@ -146,4 +146,4 @@ FAIL, on findings A and B alone; every keyed item passed. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125960137); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125959666); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125960585). diff --git a/acceptance/runs/065-toolchain-contradictions-run-4.md b/acceptance/runs/065-toolchain-contradictions-run-4.md index a959ed2..00d637f 100644 --- a/acceptance/runs/065-toolchain-contradictions-run-4.md +++ b/acceptance/runs/065-toolchain-contradictions-run-4.md @@ -1,7 +1,7 @@ # Run 65: TOOLING-SPEC.md, toolchain contradictions, fourth run - **Document**: TOOLING-SPEC.md -- **Pull request**: not yet opened at the time of the run; it is to be opened from branch +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch `editor/toolchain-contradictions`. The readers read commit dfd4b4d. Change: section 9 against clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; @@ -160,4 +160,4 @@ FAIL, on findings A, B, C and D; every keyed item passed. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125961653); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125961175); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125962019). diff --git a/acceptance/runs/067-toolchain-contradictions-run-5.md b/acceptance/runs/067-toolchain-contradictions-run-5.md index f5ce03d..1cc3e13 100644 --- a/acceptance/runs/067-toolchain-contradictions-run-5.md +++ b/acceptance/runs/067-toolchain-contradictions-run-5.md @@ -1,7 +1,7 @@ # Run 67: TOOLING-SPEC.md, toolchain contradictions, fifth run - **Document**: TOOLING-SPEC.md -- **Pull request**: not yet opened at the time of the run; it is to be opened from branch +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch `editor/toolchain-contradictions`. The readers read commit 645d811. Change: section 9 against clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; @@ -168,4 +168,4 @@ FAIL, on findings A and B; every keyed item passed. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125962818); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125962391); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125963215). diff --git a/acceptance/runs/068-toolchain-contradictions-run-6.md b/acceptance/runs/068-toolchain-contradictions-run-6.md index 4091b2e..cbbbcf9 100644 --- a/acceptance/runs/068-toolchain-contradictions-run-6.md +++ b/acceptance/runs/068-toolchain-contradictions-run-6.md @@ -1,7 +1,7 @@ # Run 68: TOOLING-SPEC.md, toolchain contradictions, sixth run - **Document**: TOOLING-SPEC.md -- **Pull request**: not yet opened at the time of the run; it is to be opened from branch +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch `editor/toolchain-contradictions`. The readers read commit 0080b8e. Change: section 9 against clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; @@ -171,4 +171,4 @@ FAIL, on findings A to F; every keyed item passed. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125964042); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125963595); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125964458). diff --git a/acceptance/runs/072-toolchain-contradictions-run-7.md b/acceptance/runs/072-toolchain-contradictions-run-7.md index 8df3d61..e508d14 100644 --- a/acceptance/runs/072-toolchain-contradictions-run-7.md +++ b/acceptance/runs/072-toolchain-contradictions-run-7.md @@ -1,7 +1,7 @@ # Run 72: TOOLING-SPEC.md, toolchain contradictions, seventh run - **Document**: TOOLING-SPEC.md -- **Pull request**: not yet opened at the time of the run; it is to be opened from branch +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch `editor/toolchain-contradictions`. The readers read commit cb9b4e5. Change: section 9 against clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; @@ -164,4 +164,4 @@ FAIL, on findings A to D; every keyed item passed. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125965287); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125964789); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125965683). diff --git a/acceptance/runs/076-toolchain-contradictions-run-8.md b/acceptance/runs/076-toolchain-contradictions-run-8.md index 4b32b4c..2dc28eb 100644 --- a/acceptance/runs/076-toolchain-contradictions-run-8.md +++ b/acceptance/runs/076-toolchain-contradictions-run-8.md @@ -1,7 +1,7 @@ # Run 76: TOOLING-SPEC.md, toolchain contradictions, eighth run - **Document**: TOOLING-SPEC.md -- **Pull request**: not yet opened at the time of the run; it is to be opened from branch +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch `editor/toolchain-contradictions`. The readers read commit fa79f8a. Change: section 9 against clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; @@ -164,4 +164,4 @@ FAIL, on findings A and B; every keyed item passed. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125966495); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125966048); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125966891). diff --git a/acceptance/runs/081-toolchain-contradictions-run-9.md b/acceptance/runs/081-toolchain-contradictions-run-9.md index 17b2c25..e7533de 100644 --- a/acceptance/runs/081-toolchain-contradictions-run-9.md +++ b/acceptance/runs/081-toolchain-contradictions-run-9.md @@ -1,7 +1,7 @@ # Run 81: TOOLING-SPEC.md, toolchain contradictions, ninth run - **Document**: TOOLING-SPEC.md -- **Pull request**: not yet opened at the time of the run; it is to be opened from branch +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch `editor/toolchain-contradictions`. The readers read commit 0373a27. Change: section 9 against clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; @@ -180,4 +180,4 @@ FAIL, on finding A; every keyed item passed. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125967702); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125967247); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125968173). diff --git a/acceptance/runs/082-toolchain-contradictions-run-10.md b/acceptance/runs/082-toolchain-contradictions-run-10.md index 1b99335..690c7a3 100644 --- a/acceptance/runs/082-toolchain-contradictions-run-10.md +++ b/acceptance/runs/082-toolchain-contradictions-run-10.md @@ -1,7 +1,7 @@ # Run 82: TOOLING-SPEC.md, toolchain contradictions, tenth run - **Document**: TOOLING-SPEC.md -- **Pull request**: not yet opened at the time of the run; it is to be opened from branch +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch `editor/toolchain-contradictions`. The readers read commit de34774. Change: section 9 against clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; @@ -175,4 +175,4 @@ FAIL, on findings A and B; every keyed item passed. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125969059); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125968623); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125969436). diff --git a/acceptance/runs/083-toolchain-contradictions-run-11.md b/acceptance/runs/083-toolchain-contradictions-run-11.md index 1a10aa2..6b21bac 100644 --- a/acceptance/runs/083-toolchain-contradictions-run-11.md +++ b/acceptance/runs/083-toolchain-contradictions-run-11.md @@ -1,7 +1,7 @@ # Run 83: TOOLING-SPEC.md, toolchain contradictions, eleventh run - **Document**: TOOLING-SPEC.md -- **Pull request**: not yet opened at the time of the run; it is to be opened from branch +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch `editor/toolchain-contradictions`. The readers read commit c770722. Change: section 9 against clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; @@ -172,4 +172,4 @@ FAIL, on findings A and B; every keyed item passed its threshold. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125970205); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125969785); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125970613). diff --git a/acceptance/runs/084-toolchain-contradictions-run-12.md b/acceptance/runs/084-toolchain-contradictions-run-12.md index 9aabaaa..1342b61 100644 --- a/acceptance/runs/084-toolchain-contradictions-run-12.md +++ b/acceptance/runs/084-toolchain-contradictions-run-12.md @@ -1,7 +1,7 @@ # Run 84: TOOLING-SPEC.md, toolchain contradictions, twelfth run - **Document**: TOOLING-SPEC.md -- **Pull request**: not yet opened at the time of the run; it is to be opened from branch +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch `editor/toolchain-contradictions`. The readers read commit c2e3249. Change: section 9 against clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; @@ -167,4 +167,4 @@ FAIL, on findings A, B and C; every keyed item passed its threshold. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125971464); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125971023); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125972035). diff --git a/acceptance/runs/085-toolchain-contradictions-run-13.md b/acceptance/runs/085-toolchain-contradictions-run-13.md index 705a277..0ab5d1f 100644 --- a/acceptance/runs/085-toolchain-contradictions-run-13.md +++ b/acceptance/runs/085-toolchain-contradictions-run-13.md @@ -1,7 +1,7 @@ # Run 85: TOOLING-SPEC.md, toolchain contradictions, thirteenth run - **Document**: TOOLING-SPEC.md -- **Pull request**: not yet opened at the time of the run; it is to be opened from branch +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch `editor/toolchain-contradictions`. The readers read commit 58924a8. Change: section 9 against clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; @@ -165,4 +165,4 @@ FAIL, on findings A and B; every keyed item passed its threshold. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125972905); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125972491); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125973283). diff --git a/acceptance/runs/086-toolchain-contradictions-run-14.md b/acceptance/runs/086-toolchain-contradictions-run-14.md index 2ed5ec9..84667ae 100644 --- a/acceptance/runs/086-toolchain-contradictions-run-14.md +++ b/acceptance/runs/086-toolchain-contradictions-run-14.md @@ -1,7 +1,7 @@ # Run 86: TOOLING-SPEC.md, toolchain contradictions, fourteenth run - **Document**: TOOLING-SPEC.md -- **Pull request**: not yet opened at the time of the run; it is to be opened from branch +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch `editor/toolchain-contradictions`. The readers read commit 6da7409. Change: section 9 against clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; @@ -184,4 +184,4 @@ no finding lies in text the change wrote or altered. ## Answer sheets -Posted on the pull request: LINK TO BE ADDED. +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125974067); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125973613); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125974453).