diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 31c8df0..77eac8f 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -6,49 +6,70 @@ document that only a strong reader can follow is not a specification, and the re matter most, coding agents meeting the method in a failing pipeline, are often the weakest. The test is run with a low-strength model so that clarity is measured at the floor, not the ceiling. +The test proves one thing: that a capable reader, reading the document cold and once, takes from +it what the document exists to say, and is not confused by the text the change put there. The +keyed questions prove the document; the confusion question proves the change. Everything below +serves that, and a rule that stops it proving that is a defect in this file, fixed as +[Changes to this process](#changes-to-this-process) describes. + ## The cohort -- **At least five independent readers**, each a fresh instance of a low-strength model (the - smallest current model in its family, for example Haiku), with no shared context and no - access to the change history, the critique findings or each other. -- Each reader receives **only the changed document or documents** and the fixed question set - below. Nothing else is read first. +- **An odd number of independent readers, at least five**, each a fresh instance of a + low-strength model (the smallest current model in its family, for example Haiku), with no + shared context and no access to the change history, the critique findings or each other. + +- **Each reader is a separate process with no tools and no project context, only its packet**: + started from an empty directory outside any repository, with no user, project or local settings + files, no `CLAUDE.md`, skills, plugins, hooks or MCP servers, and no saved session, so that no + project instruction reaches it. The model tool's own fixed system prompt is not project + context. For example, with the packet on standard input: + + ``` + claude -p --safe-mode --model haiku --tools "" --setting-sources "" --strict-mcp-config --no-session-persistence --output-format json < packet.md + ``` + + A reader is set aside, and its answers do not count, when its answer text is empty, when its + process reports an error, or when the model version it reports is not the cohort's model, the + one version the editor names for the run before its readers start. A run + with a reader set aside is void: it takes a run number and has a record, and a fresh cohort + runs. No reader is set aside for any other cause. + +- Each reader receives **only its packet**: the document's question set, the changed document + and the fixture the questions name, each in full in the reader's prompt, the question set + first. Each file is preceded by exactly one line, `===== FILE: =====`, where `` is + the name the question set gives the file (so `fixtures/remediation-ledger.md`, and + `QUESTIONS.md` for the question set itself). The question set carries the instruction to the + reader, and nothing else is added. Every reader in a run receives the same packet. + - Readers are run **after** every stronger review has been applied. They are the last gate, not the first draft. ## The question set -Written before the run, with an **answer key** for every keyed question drawn verbatim from the -document, so that marking is mechanical. Each reader answers all of them. +Written before the run, with an **answer key** for every keyed question, so that marking is +mechanical. The key states, for each keyed item, exactly what an answer must contain and what +makes it wrong; the question asks for exactly that and nothing the key does not expect. A key +asks for nothing the question does not. Each reader answers all of them. 1. **Restatement.** In their own words, what the document asks of its subject, in at most six - bullets. Keyed against the list of MUST clauses: a bullet is correct when it names an - obligation the document actually states. - -2. **Load-bearing statements.** Three questions, fixed per document, each asking for one - sentence and a quotation: + bullets. Keyed against the key's list of the document's obligations: a bullet is correct when + it names one of them. - - `SPEC.md`: what a practitioner does before fixing a defect, in order; who may baseline or - suppress; what "conforms" means for a remediation. - - `DETECTOR-SPEC.md`: what "conforms" means and whether a tool that predates the document can - conform; what a detector must do about an inline suppression route; where a bundled rule's - documentation must live. - - `TOOLING-SPEC.md`: where conformance is measured; how a grade is assigned when parts come - from different sources; how many grades a project that ships tooling has. +2. **Load-bearing statements.** Three or more questions, fixed per document and listed in its + question set under `acceptance/`, each asking for a short answer and the sentence or + sentences of the document it rests on, quoted. Each has one or more parts, and the key lists + the element an answer must state for each part. An obligation is load-bearing, and gets a + question, when a reader who misreads it would act against the method; the editor decides, and + the pull request that adds the question says why. 3. **Application.** A fixture case supplied with the questions, graded clause by clause with - Yes, Partial or No and one line of reasoning. Keyed against the intended verdicts; a divergence - counts against the document only when the reader's reasoning shows the clause was read - differently, not when the fixture was misread. + Yes, Partial or No and one line of reasoning, then the verdicts the question set asks for, + each with its reason. Keyed against the accepted grades and answers, with the readings the + key names as ones the text invites counting against the document. 4. **Confusion.** Any passage the reader found confusing on its one reading, quoted with its clause number, and nothing else: not what would make it clearer, not a proposed change. -The cohort settles one thing: is the document clear, coherent and comprehensive. A reader getting -everything right after one reading proves it; a reader getting confused fails it. The document -is designed by its editor to resolve the comprehension failures the cohort shows; the cohort is -the acceptance layer, not the designer. - ## What the cohort is not asked The cohort may point at what it found confusing. It is never asked to rate the document, to say @@ -58,61 +79,197 @@ not there to design the text, and a document shaped by what its weakest reader s a document designed by that reader. The signals this test takes from the cohort are a keyed answer missed, a wrong verdict reasoned from the document, and a passage pointed at; what to do about any of them is the editor's decision. An earlier version of this test asked the reader -what would make a clause clearer and for a clarity rating; both were removed on the author's +what would make a clause clearer and for a clarity rating; both were removed on the editor's instruction for that reason, and neither is reinstated. +## Marking + +The editor marks each answer sheet against the document's key, or delegates the marking to a +separate marker that has seen neither the drafting nor the review. A delegated marker receives +the document, its question set, fixture and key at the commit the readers read, this file, the +answer sheets, and the facts the record needs that the sheets cannot show. Its marks are final +unless the editor shows that a rule here was misapplied, and a re-mark says which rule and why. + +Every mark is given by a rule in this file or in the key, as follows. Where neither settles a +mark, the marker records the point in the run record and marks it as the open-point rule below +says, and the editor settles it in this file or in the key, as +[Changing the test](#changing-the-test) allows, before the next run. A run record states no +judgement that is not written here or in the key; a judgement an earlier run recorded is not a +rule. + +- **Keyed items.** Question 1 as a whole is one keyed item, and so is each item of question 2 + and each answer of question 3 for which the key gives an accepted grade or answer. +- **Reading an answer.** An answer is read for its sense, not its wording: a paraphrase, a + synonym, or a defined term used in its defined sense states what it means. A reader quotes the + document when it gives, in quotation marks or a block quote, a run of six or more consecutive + words of the document in the document's order, verbatim apart from letter case, quotation-mark + or apostrophe style, and markdown link or emphasis markup; a quotation that drops or changes a + word quotes the document by its longest such run. +- **Question 1** is correct when at least four bullets each name an obligation the key lists and + no bullet asserts one the key marks wrong. Only the first six bullets are marked. Several + bullets that name different clauses of one line of the key's list each count; two that name + the same clause count once. A bullet asserting an obligation that is in neither list is not + counted either way, and is recorded as an open point for the key. +- **Question 2.** An answer is the reader's sentences and its quotation read together. It is + correct when it states every element the key lists as required for that item, in the + document's words or the reader's own, asserts nothing the key marks wrong, and quotes the + document. An element stated only in the quotation is stated. Anything in the key beyond the + required elements and the wrong assertions is explanation for the marker, and an answer is + never marked down for leaving it out. +- **Question 3.** Each item is correct when its grade or answer is one the key accepts and, + where the key sets a condition on the reasoning for it, the reasoning meets it. A grade or + answer the key accepts with no condition is correct whatever the reasoning. A grade given on a + stated condition is read as the grade for the case the fixture presents, where the key says + which case that is; otherwise the item is unanswered. +- **A miss that counts against the document.** The key names, for question 1 and for each item + of question 3, the wrong readings the text invites, each as a grade or answer together with + what the reasoning states or leaves out. A miss that matches one blocks whatever the count, + since one reader in five taking that reading means one agent in five acting on it, and it + blocks wherever the text lies, because the keyed questions are what prove the document. Every + other single miss under questions 1 and 3 is the reader's and is tolerated by the threshold + below; a second miss on the same item fails the threshold, and no judgement is needed. +- **An unanswered question.** Each keyed item left unanswered is a miss. Question 4 left + unanswered counts as no passage named. +- **Question 4.** A reader names a passage when it quotes the document, as defined above, or + gives a clause number with a description or a shorter quotation that identifies a sentence or + paragraph. A quotation whose words occur in more than one paragraph, given with no clause + number, names the paragraph where they first occur. A remark that is not about a passage of + the document, such as that another document was not supplied or that a term is defined only + in another document, is recorded as a note and is not a passage, however much of the document + it quotes to locate the reference. Two readers name the same passage when the words or sentences they name lie in + the same paragraph; a heading, a list item, a table row and a terminology entry are each a + paragraph of their own. A passage named by two or more readers of the run is a finding. +- **Where a finding lies.** The record says, for each finding, whether it lies in text the change + wrote or altered: a paragraph in which the pull request, read against its merge base with the + branch it targets, adds, rewords, moves or deletes a sentence, or changes a word, a term or a + link; a paragraph the pull request parts from what it refers to, by inserting a paragraph + between them or moving either; or a paragraph that any reader naming the finding sets against + such a paragraph. A finding there blocks the change. A finding anywhere else is a finding against the document, not + against the change: it is recorded in [acceptance/ROUTED.md](acceptance/ROUTED.md), routed to a + change of its own, and does not block. The change is judged on what it changed; the document + is proved by the keyed questions, which every run answers over the whole of it. +- **Open points.** An open point in marking an answer is marked for the reader, except that a + question 1 bullet on neither list is not counted, as above. An open point on whether a reader + named a passage, or whether two readers named the same one, counts the passage. An open point + on where a finding lies treats it as the change's own text. + ## Pass criteria -The change passes when all of the following hold on a single run: +The change passes when all of the following hold on a single run. The thresholds apply to the +readers whose answers count; with five readers, "every reader" is five of five and "all but one" +is four of five. -| Criterion | Threshold | -| ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | -| Load-bearing statements (question 2) | **5 of 5** readers correct on every one | -| Restatement and application (questions 1 and 3) | **At least 4 of 5** readers correct on each keyed item | -| A wrong answer traceable to the text | **None.** A miss whose reasoning quotes the document is a finding, whatever the count | -| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it | -| Anything from one reader only | Recorded; applied at the editor's discretion; does not block | +| Criterion | Threshold | +| ------------------------------------------------------------------------------------- | -------------------------------------------------------------- | +| Load-bearing statements (question 2) | **Every** reader correct on every item | +| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | +| A miss the key says counts against the document | **None**, whatever the count | +| A passage found confusing by two or more readers, in text the change wrote or altered | **None.** It is a finding and blocks until the text is changed | +| A passage found confusing by two or more readers, elsewhere | Recorded in `acceptance/ROUTED.md` and routed; does not block | +| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | The asymmetry is deliberate. A load-bearing statement is the sentence the whole document exists to make, and one reader in five getting it wrong means one agent in five will act on the wrong reading; unanimity is the only acceptable result there. Everywhere else a single divergent reader -is more likely to be the reader than the text, so four of five is tolerated, but a miss that the -reader can justify from the document is always the document's fault. +is more likely to be the reader than the text, so one such reader is tolerated, unless the key +says the text invites that reading. A finding is settled run by run: a passage two readers named +in one run has no standing in the next, which is judged on its own readers, and a passage the +fresh cohort names by fewer than two is clear. + +## Changing the test + +Nothing in this test is changed to let a particular run through. A threshold, a question, a +key, a fixture or a rule here is changed only before a run's readers start, never changes a mark +already given, and is changed by one of two routes, each recorded with its reason: + +- In the specification pull request that changes the text, under step 4 of `CONTRIBUTING.md`: + a key's quotation follows the text it quotes; a key gains an item, with the question that asks + for it, for a new load-bearing obligation; and where the change alters or removes the + obligation an existing item tests, the item and its question follow the text, since that is + the text changing and not the test. `acceptance/README.md` says what that may and may not do. +- By a change to this process, with its own pull request: a threshold, a fixture, a rule here, + what an existing key item requires of an unchanged obligation, or the wording of a question. -## The bar only moves up +An open point a run recorded is settled by whichever route it needs, before the next run of the +pull request whose run recorded it; a run of another pull request is not held up by it. -No criterion in this document is ever relaxed to let a change through. If the cohort is confused, -the test is working and the text is wrong; the fix is to the text, never to the threshold, the -question, the fixture or the key. A threshold may be raised, a keyed question may be added, and a -key may be corrected where it misquoted the document; none of those makes a failing run pass. The -one time a threshold was relaxed, after three runs, the relaxation was reversed on the author's -instruction and is recorded here so that it is not tried again. Removing the clarity rating and -the request for suggestions was not a relaxation: it changed what the cohort is asked, not how -strictly an answer is marked. +A question is reworded only so that it asks for what its key requires, and a key's required +elements are never more than its question asks for. A run made after such a change is a fresh +run under the new rules, not a re-mark of an old one. The test for a change to a rule is what +the rule measured: a rule that measured something other than whether a cold reader could follow +the text is a defect in this file and is corrected, and the correction says what the rule +measured instead; a rule that measured that, and found the text wanting, is not touched, and the +text is fixed. The one time a threshold was relaxed to let a change through, after three runs, +the relaxation was reversed on the editor's instruction and is recorded here so that it is not +tried again. Removing the clarity rating and the request for suggestions was not that: it +changed what the cohort is asked, not how strictly an answer is marked. ## After a failure -Fix the text, never the questions, and run a fresh cohort. Readers are not reused across runs. -The number of runs it took is recorded; a document that needed three runs is telling its editor -something. +Fix the text the run's misses and findings name and run a fresh cohort; readers are never reused. +A failed keyed item is fixed in the text that carries the obligation, wherever it lies, since the +keyed questions prove the document as a whole, whoever wrote the sentence; where that fix would +change an obligation, it is routed to a change of its own, and this change waits for it. A +finding on text the change wrote or altered is fixed there. A finding elsewhere goes to +`acceptance/ROUTED.md` and is not fixed in this change. The number of runs it took is recorded; +a document that needed three runs is telling its editor something. ## When it runs -Once per pull request, on the final text, after the editor's rewrite and any key update. It is -the last gate before the merge to `next`, so that every change on `next` is one the cohort has -accepted, and the release branch needs no cohort of its own unless it changes specification text. -A wording change after the run means a fresh run; readers are never reused. +On the final text of each pull request, after the editor's rewrite and any key update, one cohort +per changed document. It is the last gate before the merge to `next`, so that every change on +`next` is one the cohort has accepted, and the release branch needs no cohort of its own unless it +changes specification text. A fresh cohort runs after every change to the changed document's +text that follows a run, and after every void run; a text that failed is never run again +unchanged, and a commit that adds the run record or the changelog line changes no document. A version number or date that `PUBLISHING.md` or step 5 of `CONTRIBUTING.md` has the +editor set is not such a change, and neither is a merge of `next` into the branch that brings in +text another cohort accepted and alters no sentence this change wrote. ## The record -Each run has a file under `acceptance/runs/`, named `NNN-.md` with `NNN` the next number in -the one sequence shared by the three documents, stating: the document, the pull request and the -commit read, the cohort size and model, the run number in that pull request, the marks per -reader per question, the findings applied, and whether it passed. The changelog line for the -change names the file. CI refuses a specification change whose pull request adds no run file; -it checks for at least one, and a pull request that changes two documents adds a run for each. - -Each published version's changelog entry carries the lines of the changes it contains, so a -version's cohort record is the set of runs those lines name. A version with no such line has not -been accepted and MUST NOT be described as published. The question sets and answer keys live -under `acceptance/` in this repository so that a run can be repeated and compared. +Each run has a file under `acceptance/runs/`, named `NNN--run-.md` with `NNN` the next +number in the one sequence shared by the three documents and `` the run number within the pull +request, stating: the document; the pull request and the commit read; the cohort size and model, +with the model version each reader reported; how the readers were isolated and the command they +were started with; any reader set aside and why; who marked and, for a delegated marker, how it +was started; the marks per reader per keyed item, with each miss and the reader's reasoning in +brief; every passage named under question 4, with the words quoted or the description given, its +clause and the readers naming it; each finding and whether it lies in text the change wrote or +altered; any point the rules did not settle; whether the run was made under a process change +before it merged and, if so, at which commit; the findings applied afterwards; the verdict; and +a link to the answer sheets. + +The answer sheets are not kept in this repository. Every reader's whole output as its process +returned it, the exact prompt every reader received, the command the readers were started with, +and the instruction a delegated marker was given are posted in one or more comments on the pull +request, linked from the record and never edited, hidden or deleted after posting. + +The changelog line for the change names every run made on it, the passing run last. CI refuses a +specification change whose pull request adds no run file; it checks for at least one, and a pull +request that changes two documents adds a run for each. Each published version's changelog entry +carries the lines of the changes it contains, so a version's cohort record is the set of runs +those lines name. A version with no such line has not been accepted and MUST NOT be described as +published. The question sets and answer keys live under `acceptance/` in this repository so that +a run can be repeated and compared. + +`acceptance/ROUTED.md` lists every finding routed from a run, open until a change that fixes +its passage passes a cohort, or until the editor closes it with a recorded reason. The editor +reads that list before cutting a release. + +## Changes to this process + +The process files are this file, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md` and the checks that enforce them: `tools/changes.py`, the key-quotation checks +in `spec-qa.py`, the CI workflow and the pull request template. A change to any of them is a +change to this process, and so is a change to a threshold, a fixture, the wording of a question +or what a key item requires, as [Changing the test](#changing-the-test) says. They are not +specifications, and a change to them is not tested by a cohort: the cohort measures whether a +low-strength reader can follow a specification, and the process is followed by the editor. Such +a change names the runs or the case that showed the need for it, and goes through steps 1, 3 and +6 of `CONTRIBUTING.md`, with no scope, key or cohort step; its review reads every process file it +changes. It applies to every run whose readers start after it merges; a run made before then may +be made under it, following all of it at a commit named in the record before the readers start. +It is recorded by a line added at the top of [acceptance/CHANGES.md](acceptance/CHANGES.md), +naming its pull request, not in the changelog, which is for the three specifications. +`acceptance/ROUTED.md` and `acceptance/runs/` are records, not process files: a specification +pull request writes to them. diff --git a/CHANGELOG.md b/CHANGELOG.md index f5f8122..34686d0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -95,6 +95,107 @@ make a clause clearer and for a clarity rating; both questions were withdrawn be ## Toolchain specification (TOOLING-SPEC.md) +### Unreleased + +- Clarity, no obligation changed: four passages that cold readers of an unrelated change read as + the document pulling against itself, each listed in `acceptance/ROUTED.md`. Section [9](TOOLING-SPEC.md#9-conformance)'s + conformance sentence now says that section [8](TOOLING-SPEC.md#8-self-audit) bears on a shipping project's artefact grade + under clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately), not on its project grade, which is what clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) already said and the + answer key assumed. Clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) says that one identifier that does not resolve fails it and + that a blocking rule whose documentation is missing is graded as a defence with a gap, and + clause [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) says the same of a bundled defence, after readers took section [2](TOOLING-SPEC.md#2-terminology)'s Defence row to + put such a rule outside both clauses. Clause [9.2](TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance)'s duty to record a known gap in the + declaration falls only on a project or artefact that declares a version, and says so beside + "The declaration is optional", with the method's clause [3.2](SPEC.md#32-build-the-net) record applying either way. + Clause [4.3](TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)'s heading names both ways its body allows, disabling the route or a defence of + the toolchain's own. Clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) says every MUST of the detector specification's sections [4](DETECTOR-SPEC.md#4-authoring-rules) to [7](DETECTOR-SPEC.md#7-suppression) + applies and the six it lists are a summary, and its point 2 says the wrapped pair is what the + clause judges whilst the detector's own grade is a separate verdict. Prompted by runs 33 to 52 + and 55 on the defined-terms change, in which two or more readers quoted each passage. + [Run 57](acceptance/runs/057-toolchain-contradictions-run-1.md) failed on the first wording: every reader set clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)'s + "a defence with a gap" against the Defence row, so the sentence now says Defence there + reaches every blocking rule whether or not its documentation exists; clause [4.3](TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) says the + routes are the ones each detector offers, after two readers graded it Partial for routes the + fixture does not describe; section [1](TOOLING-SPEC.md#1-what-this-document-is-for) says the document defers to the detector specification + only in clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification); clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) says what the artefact grade counts; and section [9](TOOLING-SPEC.md#9-conformance)'s agent-support + sentence says section [7](TOOLING-SPEC.md#7-agent-context) recommends, after two readers set "additionally satisfies" against + that section's SHOULDs. [Run 58](acceptance/runs/058-toolchain-contradictions-run-2.md) failed + on four findings in the revised text, every keyed item five of five: four readers set clause + [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)'s "whether or not its documentation exists" against the Defence row again, so clauses + [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) and [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) now say that a blocking rule whose documentation is missing is not yet a + defence and is the failure each names; clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) says its six sentences are a summary and + not the test, and its Why names the method's clauses [3.1](SPEC.md#31-attribute-the-defect-to-a-class) to [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) rather than "six clauses"; + section [9](TOOLING-SPEC.md#9-conformance) says every project has one project grade and that section [8](TOOLING-SPEC.md#8-self-audit) bears on a shipping + project's separate artefact grade, clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) names the two levels, and the agent-support + sentence says which document's section [7](TOOLING-SPEC.md#7-agent-context) it means. + [Run 62](acceptance/runs/062-toolchain-contradictions-run-3.md) failed on the same clause + [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) sentence, every keyed item five of five: four readers could not settle whether the + clause covers its defences or every blocking rule, so its heading and its obligation sentence + now name both, a defence or a blocking rule it routes, and the body says why the blocking + rules are named. [Run 65](acceptance/runs/065-toolchain-contradictions-run-4.md) failed on it + a third time, three readers unable to square a clause on defences with rules that are not + yet one, so the heading and the obligation now name blocking rules alone, with one sentence + on why, and clause [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) drops its sentence on the same contrast; clause [9.1](TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) says which + document grades what, after two readers asked which section [7](TOOLING-SPEC.md#7-agent-context) the project grade rests on. + [Run 67](acceptance/runs/067-toolchain-contradictions-run-5.md) failed on clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)'s first + paragraph, three readers each stopping at a different sentence of it, and on section [1](TOOLING-SPEC.md#1-what-this-document-is-for)'s + "How to read this document" set against clause [4.4](TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes): the paragraph is rewritten in three, the + obligation first, then what is in scope, then the mechanism, with Remediation docs glossed in + section [2](TOOLING-SPEC.md#2-terminology)'s table; and section [1](TOOLING-SPEC.md#1-what-this-document-is-for) says that a toolchain can be graded from this document + alone because each clause that rests on the detector specification says in its own words what + that document requires. [Run 68](acceptance/runs/068-toolchain-contradictions-run-6.md) failed + on six findings in that text: readers set a URL scheme against "without network access", "MUST + ship with the project or with the Toolchain" against a third-party catalogue "not re-shipped + here", the sentence on why the clause names blocking rules against the Defence row and clause + [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) yet again, and clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification)'s "a summary, not the test" against section [1](TOOLING-SPEC.md#1-what-this-document-is-for). Clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) now + says only what it requires, of the project's own rules and bundled rules alike, with no + sentence on defences and no list of mechanisms, and its remediation docs may ship with the + detector whose rule they document; clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) and section [1](TOOLING-SPEC.md#1-what-this-document-is-for) both say the clause restates + the detector specification far enough to grade what a toolchain's evidence shows. + [Run 72](acceptance/runs/072-toolchain-contradictions-run-7.md) failed on four findings: four + readers set the Defence row and clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) against clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) once more, asking whether an + undocumented blocking rule is a defence and is listed, and two set section [1](TOOLING-SPEC.md#1-what-this-document-is-for)'s "graded + from this document alone" against clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification)'s "restated here or not". Section [2](TOOLING-SPEC.md#2-terminology) now says, + under its table, that a blocking rule whose documentation is missing is not yet a defence and + is what clauses [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access), [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) and [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) exist to catch; clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) says the listing covers every + blocking rule, documented or not, with a missing route showing; and section [1](TOOLING-SPEC.md#1-what-this-document-is-for) and clause + [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) both say a verdict from this document alone rests on what is restated and a grader with + the detector specification holds the detector to all of it. [Run + 76](acceptance/runs/076-toolchain-contradictions-run-8.md) failed on clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification)'s first + paragraph, all five readers unable to tell which of its two tests governs a grader without the + detector specification, and on section [2](TOOLING-SPEC.md#2-terminology)'s "These are additional.", which the new + paragraph had parted from its table. Clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) and section [1](TOOLING-SPEC.md#1-what-this-document-is-for) now give one test: a + grader checks the restated sentences and takes a verdict already given on the detector under + the detector specification as settled; the new paragraph sits at the end of section [2](TOOLING-SPEC.md#2-terminology), after + the Project record entry. [Run 81](acceptance/runs/081-toolchain-contradictions-run-9.md) + failed on clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), two readers setting "every Defence active" against the new sentence that the + listing covers every blocking rule, documented or not: a listing of defences said to cover + what is not one. Clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) now says such a rule is not yet a defence, that clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) fails it, + and that the listing can show it with the gap showing; section [2](TOOLING-SPEC.md#2-terminology)'s paragraph names + clauses [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) and [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) as the clauses that catch it. [Run 82](acceptance/runs/082-toolchain-contradictions-run-10.md) + failed on that clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) sentence again, two readers asking whether an empty route meets the MUST + and whether "can show" means such a rule need not appear, and on clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification)'s "a verdict + already given", two readers unable to tell which verdict or what a grader does without one. + Clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) says nothing of undocumented rules now, section [2](TOOLING-SPEC.md#2-terminology)'s closing paragraph being where + that is said; clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) and section [1](TOOLING-SPEC.md#1-what-this-document-is-for) name the verdict as one given under the detector + specification's conformance section and say that without one the restated sentences are the test. + [Run 83](acceptance/runs/083-toolchain-contradictions-run-11.md) failed on section [2](TOOLING-SPEC.md#2-terminology)'s + closing paragraph and on clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) set against it, three readers asking whether an + undocumented blocking rule has to appear in the listing. The paragraph now answers: nothing + the document asks of defences, the listing included, is asked of such a rule, and a toolchain + that runs one has failed clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) whether or not its listing shows it. [Run + 84](acceptance/runs/084-toolchain-contradictions-run-12.md) failed on clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), three + readers unable to tell whether "every one of them" or the restated sentences set the test, and + on the paragraph again, one reader still asking whether such a rule is listed and one unable + to place "they". Clause [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) now has one test, every MUST of the detector specification's authoring, reporting, resolving and suppression sections, with the + restated sentences as where to look first and a grader without the detector specification + checking only them and saying so; section [1](TOOLING-SPEC.md#1-what-this-document-is-for) says the same; the paragraph says in plain + sentences that clause [5.1](TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) does not require such a rule to be listed and that clauses [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) and + [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) reach it. [Run 85](acceptance/runs/085-toolchain-contradictions-run-13.md) failed on that + paragraph's sentence naming both clauses as reaching every blocking rule, two readers setting it against + clause [8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation)'s exclusion of a third-party detector's native catalogue; the paragraph names + clause [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) alone. Accepted by [run 86](acceptance/runs/086-toolchain-contradictions-run-14.md). + ### 0.2.0, 2026-09-08 Reissued the same day as 0.1.0 with sections [4](TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](TOOLING-SPEC.md#6-the-project-record) moved to the detector specification and @@ -120,10 +221,10 @@ since the clause says the listing MUST provide the route and [4.1](TOOLING-SPEC. again by a later clause; three readers of run 8 graded it so from the text. Findings applied across the runs: an eleven-term gloss and a reading note in sections [1](TOOLING-SPEC.md#1-what-this-document-is-for) and [2](TOOLING-SPEC.md#2-terminology); long sentences split; [4.1](TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) opens by listing the detector specification's MUSTs and its point 2 -leads with a three-case table of wrapped pairs, then the one question behind it; [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) says it -widens the detector document's reach to the project's own rules; [4.3](TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) loses its double negative; +leads with a three-case table of wrapped pairs, then the one question behind it; [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) says it +widens the detector document's reach to the project's own rules; [4.3](TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) loses its double negative; [5.2](TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) gives the derived-listing test; [6.2](TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one)'s heading carries the generic-justification rejection; -[8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) is consistent with [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) on third-party catalogues. Runs 1 to 7 also asked readers what would +[8.1](TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) is consistent with [4.2](TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) on third-party catalogues. Runs 1 to 7 also asked readers what would make a clause clearer and for a clarity rating; both questions were withdrawn before run 8, as `ACCEPTANCE.md` records. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fa1743e..ea5a525 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,7 +29,9 @@ review. It is a patch version when released. SHOULD or MAY in a numbered section or appendix. It needs everything a clarity edit needs, plus the five tests in SCOPE.md answered in the pull request, and it is at least a minor version when released. `tools/changes.py` tells the two apart mechanically and CI runs it on every pull -request. +request. A change to the acceptance process, as ACCEPTANCE.md defines one under "Changes to +this process", is neither: ACCEPTANCE.md says how it is made, and it is recorded in +`acceptance/CHANGES.md`, not in CHANGELOG.md. ## What a pull request contains @@ -55,20 +57,26 @@ survives, the change is yours. the editor sets the pre-release header. 2. **Scope.** For an obligation, the editor confirms each SCOPE.md test. A failed test closes the pull request with a row in `DECLINED.md`. -3. **Review and rewrite.** An independent review of the diff against the document, then the - editor's rewrite, pushed to the pull request branch where the contributor allows it or to an - editor's branch that supersedes it. +3. **Review and rewrite.** An independent review of the diff against the whole document, by a + reviewer who did not draft the change, then the editor's rewrite, pushed to the pull request + branch where the contributor allows it or to an editor's branch that supersedes it. The pull + request names the reviewer and links or contains the review. 4. **Keys.** Where the change touches a sentence an acceptance key quotes, or adds a load-bearing - obligation, the editor updates the key before the cohort runs, never after. -5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text. Its record is - committed under `acceptance/runs/`, the `Unreleased` changelog line names it, and CI refuses - a specification change that lacks one. For an obligation, the editor sets the document's - header to `-dev, unpublished` in the same commit if it still names a published - version. A wording change after the run means a fresh run. + obligation, the editor updates the key before the cohort runs, never after. The key's + required elements stay what its question asks for; `acceptance/README.md` says what a key + update may and may not do. +5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text, one cohort per + changed document. Its record is committed under `acceptance/runs/`, the `Unreleased` + changelog line names it, and CI refuses a specification change that lacks one. For an + obligation, the editor sets the document's header to `-dev, unpublished` in the + same commit if it still names a published version. A change to the document's text after the + run means a fresh run, as ACCEPTANCE.md says under "When it runs". A finding the + run routes to `acceptance/ROUTED.md` is not fixed in this pull request. 6. **Merge** into `next` by merge commit, and the branch is deleted. -Steps 2 to 5 are done by a person with a Claude Code session, not by CI, and they happen in -batches. A quiet pull request is not a declined one; a declined one says so. +Steps 2 to 5 are the editor's, not CI's, and they happen in batches. The editor may have an +agent draft, review or mark, as ACCEPTANCE.md allows for marking; every decision in these steps +stays the editor's. A quiet pull request is not a declined one; a declined one says so. ## Versions and releases diff --git a/TOOLING-SPEC.md b/TOOLING-SPEC.md index 209495a..8ed7aa6 100644 --- a/TOOLING-SPEC.md +++ b/TOOLING-SPEC.md @@ -39,7 +39,11 @@ building the mechanism first. A [Toolchain] that leaves the project to construct moved the obligation rather than met it. Where the project builds the means itself, those scripts are part of its [Toolchain] and are judged as such. -**How to read this document.** It is self-contained for grading a [Toolchain]. The +**How to read this document.** A [Toolchain] can be graded from this document alone, on the +evidence a project shows. Where a clause rests on the [detector specification](DETECTOR-SPEC.md), +as clauses 4.1, 4.2 and 4.4 do, it says what it holds the [Detector] to and restates the parts +most often failed; a grader without that document to hand checks what is restated and says so. +The table in section 2 glosses every term it borrows. Each clause states its obligation in its heading and first sentence, and the **Why** paragraph beneath is reasoning, not a further requirement. A link is there for depth and is not a prerequisite for the sentence that carries it. @@ -49,19 +53,20 @@ link is there for depth and is not a prerequisite for the sentence that carries Terms from the method specification and the [detector specification](DETECTOR-SPEC.md) carry over unchanged. Every capitalised term links to its definition; the ones this document leans on most are, in short: -| Term | In one line | -| -------------- | ----------------------------------------------------------------------------------- | -| [Detector] | A tool that reads code without executing it and reports what it finds | -| [Runner] | A tool that executes code, a test suite or a compilation, and reports what happened | -| [Rule] | One check a [Detector] evaluates | -| [Defence] | A [Rule] with its documentation, in force and [Blocking] | -| [Identifier] | The stable name printed with a finding that leads to its documentation | -| [Blocking] | Fails the run rather than issuing a [Warning] | -| [Exception] | A recorded, justified decision to leave an [Instance] unfixed or a [Rule] unapplied | -| [Practitioner] | Whoever is doing the work, a person or an [Agent] | -| [Owner] | The human who decides what the codebase may keep | -| [Bundled rule] | A [Rule] a [Detector] ships rather than one the project wrote | -| [Harness] | The route by which one [Rule] is run against supplied code on its own | +| Term | In one line | +| ------------------ | --------------------------------------------------------------------------------------------------- | +| [Detector] | A tool that reads code without executing it and reports what it finds | +| [Runner] | A tool that executes code, a test suite or a compilation, and reports what happened | +| [Rule] | One check a [Detector] evaluates | +| [Defence] | A [Rule] with its documentation, in force and [Blocking] | +| [Identifier] | The stable name printed with a finding that leads to its documentation | +| [Remediation docs] | The documentation an [Identifier] leads to: what the [Rule] is about, why, and how to fix a finding | +| [Blocking] | Fails the run rather than issuing a [Warning] | +| [Exception] | A recorded, justified decision to leave an [Instance] unfixed or a [Rule] unapplied | +| [Practitioner] | Whoever is doing the work, a person or an [Agent] | +| [Owner] | The human who decides what the codebase may keep | +| [Bundled rule] | A [Rule] a [Detector] ships rather than one the project wrote | +| [Harness] | The route by which one [Rule] is run against supplied code on its own | These are additional. @@ -81,6 +86,11 @@ The distinction between a [Bundled defence] and a project's own is significant t [Defence] is authored once and runs in codebases its author will never see, so everything a [Practitioner] needs in order to act on it has to travel with it. +Throughout this document, a [Blocking] [Rule] whose documentation is missing is not yet a +[Defence]. Clause 5.1 does not require it to be listed, because that clause lists [Defences]. +Clause 4.2 reaches every [Blocking] [Rule] a [Toolchain] runs, documented or not, and a +[Toolchain] that runs such a [Rule] fails clause 4.2. + ## 3. The division of responsibility Almost every requirement in section 8 of the method specification has two halves. Stating only one @@ -108,13 +118,18 @@ any row and the failure mode is visible too: either half alone leaves the [Pract ### 4.1 Every detector the toolchain routes a defence through MUST conform to the [detector specification](DETECTOR-SPEC.md) A [Toolchain] MUST NOT route a [Defence] through a [Detector] that does not -[Conform] to [the detector specification](DETECTOR-SPEC.md). That document's MUSTs, in brief, -are six. The [Detector] hosts bespoke [Rules]. It runs one [Rule] on its own against supplied +[Conform] to [the detector specification](DETECTOR-SPEC.md). That document's MUSTs are those of +its sections 4 to 7, and this clause holds the [Detector] to every one of them: a [Detector] +that fails any of them, after wrapping, fails this clause. The sentences that follow restate +the ones most often failed, so that a grader knows where to look first; they are not the whole +test, and a grader without that document to hand can check only them, and says so in the +verdict. The +[Detector] hosts bespoke [Rules]. It runs one [Rule] on its own against supplied code. It prints a stable [Identifier] with every finding. It runs locally, down to one file, with the result in the command's own output and nothing reportable only through a hosted service. It resolves every [Identifier] a [Bundled rule] can print to documentation shipped with it, offline. -It makes any inline [Suppression] route detectable or disableable. A [Detector] that fails any -one of those, after wrapping, fails this clause. Three points govern how that is met. +It makes any inline [Suppression] route detectable or disableable. Three points govern how that +is met. 1. **Wrapping is permitted.** The [Toolchain] MAY supply a mechanism the [Detector] lacks by wrapping it, with a [Harness] script or a @@ -133,9 +148,11 @@ one of those, after wrapping, fails this clause. Three points govern how that is Behind the table is one question. With the wrapping in place, does the [Detector] meet every MUST of sections 4 to 7 of the [detector specification](DETECTOR-SPEC.md), exercised as its clause 8.1 describes? If yes, this clause holds. If no, it fails, however - much else the [Detector] does well, and whether or not a clause below names the same gap - again. Wrapping can add a mechanism; it cannot excuse a gap. The [Detector]'s own verdict - under that document is unchanged by the wrapping. Satisfying most of that document is not + much else the [Detector] does well, and whether or not a later clause of this document, such + as clause 4.2 on resolution, names the same gap again. Wrapping can add a mechanism; it + cannot excuse a gap. What this clause judges is the + wrapped pair; the [Detector]'s own grade under that document, as an artefact on its own, is a + separate verdict that the wrapping does not change. Satisfying most of that document is not [Conformance] to it, any more than satisfying most of this one is. 3. **What cannot be wrapped.** A [Detector] that cannot host a bespoke [Rule] at all @@ -143,7 +160,7 @@ one of those, after wrapping, fails this clause. Three points govern how that is It MAY still run as one of the [Toolchain]'s checks, as a formatter or a [Runner] does. -**Why**: the six clauses of the method are carried out in a [Detector], and everything this +**Why**: the method's clauses 3.1 to 3.6 are carried out in a [Detector], and everything this document adds presumes those clauses can be followed there. A [Toolchain] that lists, records and resolves beautifully around a [Detector] in which no [Rule] can be written or proven has governed nothing. The wrapping is permitted because a [Practitioner] cannot tell where a @@ -151,26 +168,27 @@ mechanism lives, and the method does not care. It is bounded because a mechanism does not offer and the [Toolchain] does not add is a gap the project writes down under method clause 3.2. A gap recorded is not a gap closed. -### 4.2 The toolchain MUST resolve every identifier a defence it routes can print, from the installed copy, without network access +### 4.2 The toolchain MUST resolve every identifier a blocking rule it routes can print, from the installed copy, without network access + +The [Toolchain] MUST resolve every [Identifier] its [Blocking] [Rules] print to +[Remediation docs] on disk. That is every [Blocking] [Rule] the [Toolchain] runs, the project's +own and the bundled alike. One [Identifier] that does not resolve fails this clause, however +many others do. -This clause widens the [detector specification](DETECTOR-SPEC.md)'s reach: that document resolves -[Bundled rules] only, and this one covers every [Identifier] the project can see, the project's -own [Rules] included. The [Toolchain] MUST resolve every [Identifier] its [Defences] -print to [Remediation docs] on disk. The mechanism is keyed on the -[Identifier] exactly as printed, and it covers the project's own [Rules] and -every [Bundled defence]. Those [Remediation docs] MUST ship -with the project or with the [Toolchain], at a version tracked together. The +The resolving mechanism is keyed on the [Identifier] exactly as printed. Those +[Remediation docs] MUST ship with the project, with the [Toolchain] or with the [Detector] +whose [Rule] they document, at a version tracked together. The [Toolchain] MUST also give that documentation a place to live, so that a [Rule author] adding a [Rule] knows where its documentation goes. The documentation an [Identifier] resolves to MUST state the correct construction, not only the prohibition, specifically enough to act on, as method clause 8.4 requires. The [Toolchain] supplies the place and the [Rule author] the text. -This clause closes what the [detector specification](DETECTOR-SPEC.md) leaves open. That document holds a -[Detector] to on-disk resolution of its own [Bundled rules] and -no further, because it cannot know a project's documentation. A third-party [Detector]'s -native catalogue, which the [Toolchain] orchestrates without claiming as its own, is -resolved under that document's clause 6 and is not re-shipped here. +This clause closes what the [detector specification](DETECTOR-SPEC.md) leaves open: that +document holds a [Detector] to on-disk resolution of its own [Bundled rules] and no further, +because it cannot know a project's documentation, and this clause reaches the project's own +[Rules] as well. An [Identifier] a third-party [Detector] resolves under that document's clause +6 meets this clause, and one that nothing resolves fails it. **Why**: method specification clauses 3.6 and 8.3. The [Identifier] has to resolve for the reader, which for an [Agent] means mechanically and on disk, and the project's own @@ -179,7 +197,7 @@ reader, which for an [Agent] means mechanically and on disk, and the project's o has a [Message] that names a pattern without leading anywhere, which method clause 3.6 says does not [Conform]. -### 4.3 The toolchain MUST forbid, through a defence of its own, every suppression route that bypasses the project record +### 4.3 The toolchain MUST forbid every suppression route that bypasses the project record, by disabling it or by a defence of its own A [Conforming] [Toolchain] MUST disable every [Suppression] route that bypasses the [Project record], or MUST run a [Blocking] @@ -187,7 +205,10 @@ route that bypasses the [Project record], or MUST run a [Blocking] silenced through it without an entry appearing in the [Project record]: an inline [Suppression] comment, a per-line ignore and a silently generated [Baseline] all do. The obligation covers every such route in every [Detector] the [Toolchain] routes a -[Defence] through. Irreducible cases MUST be directed to the [Project record], +[Defence] through. The routes are the ones each [Detector] offers: where a [Detector]'s only +route is an ignore comment, this clause is met for that [Detector] once the comment is disabled +or a [Blocking] [Defence] fails on its use, and nothing is owed for a route it does not have. +Irreducible cases MUST be directed to the [Project record], where clause 6.2 requires a justification. The [detector specification](DETECTOR-SPEC.md) permits a [Detector] to ship such a route provided @@ -362,7 +383,8 @@ mistake rather than after it, which is why they are worth stating even as SHOULD The [Toolchain] MUST run an automated check that blocks its own release. The check covers every [Identifier] printed by a [Rule] the [Toolchain] authors or bundles -as its own [Defence], whatever kind of [Detector] carries it. Where a documentation +as its own, whatever kind of [Detector] carries it, and a missing page is what the check exists +to find. Where a documentation page covers a family of [Identifiers] by a pattern, as clause 6.5 of the [detector specification](DETECTOR-SPEC.md) allows, this audit MUST apply the pattern to every [Identifier] printed and confirm it lands on that page. A member added to the @@ -400,11 +422,15 @@ own [Defences] already run on its own source, because that is the only source th ## 9. Conformance **A project's [Toolchain] [Conforms]** if every MUST in sections 4 to 6 holds across the -assembled parts, wherever each part came from, and, where the [Toolchain] is one the project -ships, every MUST in section 8 as well. The project-level verdict is a single grade; which part of the -[Toolchain] satisfied each clause is evidence for that grade, not a second grade. +assembled parts, wherever each part came from; where the project also ships a [Detector] or a +[Toolchain], every MUST in section 8 bears on the artefact grade of clause 9.1 and not on this +project grade. The project-level verdict is a single grade; which part of the +[Toolchain] satisfied each clause is evidence for that grade, not a second grade. The artefact +a project ships is graded on its own, and that grade is the second level clause 9.1 names; a +project that ships nothing has only the one. -**A [Toolchain] [Conforms] with [Agent] support** if it additionally satisfies section 7. +**A [Toolchain] [Conforms] with [Agent] support** if, in addition, it does each thing section 7 +of this document recommends. Partial [Conformance] MUST NOT be described as [Conformance]. A [Toolchain] that satisfies most of this document is in a normal and respectable condition; it is not [Conforming], and describing it as such @@ -413,12 +439,15 @@ removes the only value the word has. ### 9.1 A project that ships a detector or a toolchain has two levels of conformance, graded separately As a project, it follows the method with its own assembled [Toolchain], like any other -project, and is graded against this document and section 7 of the method specification on that -basis. As an artefact, what it ships is graded for its consumers: a [Detector] against -[the detector specification](DETECTOR-SPEC.md), a [Toolchain] against this document as it stands -when installed into a [Consuming project] with nothing else built around it. The two -verdicts MUST be graded and declared separately, and neither implies the other. Section 8 bears on -the artefact grade only; clause 5.3 is its project-level counterpart. +project: that [Toolchain] is graded against this document, under section 9, and the project's +remediations against section 7 of the method specification, as any project's are. As an artefact, what it ships is graded for its consumers: a [Detector] against +[the detector specification](DETECTOR-SPEC.md), a [Toolchain] against this document, on what the +shipped [Toolchain] itself supplies once installed into a [Consuming project], counting nothing +the [Consuming project] adds around it. The two +verdicts MUST be graded and declared separately, and neither implies the other. The two levels +are the project grade of section 9, which every project has, and the artefact grade, which only +a project that ships has. Section 8 bears on the artefact grade only; clause 5.3 is its +project-level counterpart. **Why**: the two questions have different readers. A contributor to the artefact wants to know whether the project practises what it ships; a [Consuming project] wants to know what it @@ -433,15 +462,17 @@ record dependencies, the version of the method specification it follows and the document or the [detector specification](DETECTOR-SPEC.md) it [Conforms] to. A project that ships an artefact carries both levels of clause 9.1 in that declaration, each named separately. -The same declaration is where a gap is recorded once it is known. A project or artefact that has -learnt that it fails a MUST of the document it declares against MUST record that gap alongside the -version, in the same file or one it names. It may learn that from its own self-audit under section 8 -or from a [Practitioner]'s report under the method's clause 3.2. A declaration with a non-empty gap record is -a statement of where the project stands and is not a claim of [Conformance]. A mechanism gap -is by its nature one the [Toolchain] could not detect for itself, so the record is the only -place its [Owner] and its consumers can learn of it. - -The declaration is optional. A [Toolchain] assembled before this document existed, or by a +The same declaration is where a gap is recorded once it is known. A project or artefact that +declares a version and has learnt that it fails a MUST of the document it declares against MUST +record that gap alongside that version, in the same file or one it names. It may learn that from +its own self-audit under section 8 or from a [Practitioner]'s report under the method's clause +3.2. A declaration with a non-empty gap record is a statement of where the project stands and is +not a claim of [Conformance]. A mechanism gap is by its nature one the [Toolchain] could not +detect for itself, so the record is the only place its [Owner] and its consumers can learn of it. + +The declaration is optional, and so the duty above falls only on a project or artefact that has +one; a gap a [Practitioner] finds is recorded under the method's clause 3.2 in any case, whether +or not the project declares. A [Toolchain] assembled before this document existed, or by a project that has never read it, MAY be graded [Conforming] on evidence by anyone who exercises the clauses above against it. A verdict on any [Toolchain], declared or not, rests on that exercise and not on the claim. diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md new file mode 100644 index 0000000..daa3d44 --- /dev/null +++ b/acceptance/CHANGES.md @@ -0,0 +1,38 @@ +# Changes to the acceptance process + +Changes to the acceptance process, as [ACCEPTANCE.md](../ACCEPTANCE.md) defines one under +"Changes to this process", recorded here as it describes: a line is added at the top for each +change, naming its pull request. The process is not a specification and is not versioned, and a +change to it is not tested by a cohort. Changes to the three specifications are recorded in +[CHANGELOG.md](../CHANGELOG.md). + +- [Pull request #22](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/22): + the test is made to prove what it exists to prove, the clarity of the + document to a cold reader, after thirty-three runs on two changes (19 to 30, 31 to 33 and 35 + to 52) in which keyed answers reached five of five and the runs still failed. Five defects in + the process were found. Keys required elements the question never asked for (detector 2b's + "need not remove or forbid", detector 2c's "at a version tracked together", and on pull + request 14 method 2d's order of the kinds), so readers who answered the question as put were + marked wrong: now each key lists, per item, the elements an answer must state, one for + each part of the question, and the questions are reworded to ask for exactly those; anything + else in the key only catches a wrong assertion. A passage two readers found confusing stayed + open whilst any single reader of a later cohort named any part of its paragraph, so cleared + passages kept coming back: now a finding is settled run by run, raised by two readers of one + run and clear when the next names it by fewer than two. A finding landed on text the change + never touched, so a change's cohort acted as a review of the whole document and could not + converge: now a finding in text the change wrote or altered blocks the change, and a finding + elsewhere is routed to `acceptance/ROUTED.md` and to a change of its own, whilst the keyed + questions still prove the whole document on every run. Markers invented judgements the rules + left open and later runs cited them as precedent: now an open point is marked for the reader + and settled in this file or the key before the next run, and a run record states no judgement + the rules do not. The rule that any miss whose reasoning quoted the document was a finding + whatever the count is replaced by the key naming the wrong readings the text invites, which + count against the document on one reader, every other single miss being tolerated by the + all-but-one threshold. Question 2's instruction asks for the sentence or sentences relied on, + since three keys rest on two. The cohort, the packet, set-aside readers, the posted sheets, + the fuller record and the route for a process change are kept from the earlier proposal in + pull request 13, which this change supersedes. Relative to the rules on `next`, the + traceable-miss row is narrower and the confusion row is applied to the change rather than + the document; both are recorded here as the reason the earlier rules could not converge, and + neither changes a mark already given. Prompted by runs 19 to 30 on pull request 14 and runs + 31 to 33 and 35 to 52 on the defined-terms change. diff --git a/acceptance/README.md b/acceptance/README.md index 4cfb6ec..ec5c3b2 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -2,8 +2,10 @@ The question sets and answer keys that [ACCEPTANCE.md](../ACCEPTANCE.md) requires. One directory per specification document, each holding the fixed questions a cold reader answers and the key an -editor marks against. The fixture cases the application question uses live in `fixtures/`, and -the record of every run lives in `runs/`. +editor marks against. The fixture cases the application question uses live in `fixtures/`, the +record of every run lives in `runs/`, findings routed out of a run to a change of their own are +listed in [ROUTED.md](ROUTED.md), and changes to the acceptance process itself are recorded +in [CHANGES.md](CHANGES.md). | Document | Questions | Key | | ------------------ | ------------------------------------------------ | ------------------------------------ | @@ -14,15 +16,36 @@ the record of every run lives in `runs/`. ## How a run works 1. The editor identifies which documents the change touched. Only those are tested. -2. For each, five or more fresh low-strength readers each receive the document's current text, - its `QUESTIONS.md`, and the fixture the questions name. Nothing else: no key, no changelog, no - critique findings, no other reader's answers. -3. The editor marks each answer sheet against `KEY.md` and applies the pass criteria in - `ACCEPTANCE.md`. -4. The result is recorded as a file under [runs/](runs/), with cohort size, model, run number, - marks and the findings applied, and the change's `Unreleased` changelog line names that file. - A run that fails is followed by a change to the text, never to the questions, and a fresh - cohort. +2. For each, an odd number of fresh low-strength readers, at least five, each isolated as + `ACCEPTANCE.md` describes, receive the document's current text, its `QUESTIONS.md`, and the + fixture the questions name. Nothing else: no key, no changelog, no critique findings, no other + reader's answers. +3. The editor, or a separate marker the editor delegates to, marks each answer sheet against + `KEY.md` under the marking rules in `ACCEPTANCE.md` and applies its pass criteria. +4. The result is recorded as a file under [runs/](runs/), stating what `ACCEPTANCE.md` lists, + with the answer sheets posted on the pull request and linked from it, and the change's + `Unreleased` changelog line names that file. A run that fails is followed by a change to the + text its misses and findings name and a fresh cohort; a finding on text the change did not + write or alter is added to [ROUTED.md](ROUTED.md) instead. + +## How a key is written + +A key says, for each keyed item, exactly what is marked and nothing a marker has to guess: + +- **Question 1**: the obligations a bullet may name, the assertions that make a bullet wrong, + and the wrong readings the text invites, if any. +- **Question 2**: for each item, the elements an answer is required to state, numbered, one for + each part of the question; the assertions that make an answer wrong; and the sentence or + sentences of the document the answer rests on, quoted verbatim, which show the marker where + the elements come from and which `spec-qa.py` checks against the document; a reader's + quotation need not be that one. The required elements are what the question asks for, no + more: an element the question does not ask for is not required, however load-bearing, until + the question is reworded to ask for it. +- **Question 3**: for each item, the grade or grades accepted, any condition on the reasoning, + and the wrong readings the text invites, which count against the document when one reader + takes them. + +Everything else in a key is explanation for the marker and never marks an answer down. ## Keeping the keys honest @@ -30,8 +53,24 @@ A key quotes the document. When a clause is reworded, the key line that quotes i the same commit, and the change is one the acceptance run itself then tests. A key that no longer matches the document is a defect in this directory, not a reason to mark a reader wrong. +A key, a question set or a fixture changes by one of two routes, as `ACCEPTANCE.md` says under +"Changing the test", always before the readers of the next run start, and an open point a run +recorded is settled by whichever route it needs: + +- **In a specification pull request, under step 4 of `CONTRIBUTING.md`**: a quotation follows + the text it quotes; a keyed item is added for a new load-bearing obligation together with the + question that asks for it; or, where the change alters or removes the obligation an existing + item tests, that item and its question follow the text, which is the text changing and not + the test. The pull request's own changelog line records it. Such an update does not change + what an item requires of an obligation the change leaves as it was, nor the wording of a + question whose obligation is unchanged. +- **In a process change, with its own pull request**: what an existing item requires, the + wording of a question, a fixture, or the rules of marking. It is recorded in + [CHANGES.md](CHANGES.md) with its reason. + Questions are fixed so that runs can be compared across versions. Adding a keyed question is -allowed when a new clause is load-bearing; removing or softening one to make a run pass is not. +allowed when a new clause is load-bearing; removing or softening one to make a run pass is not; +rewording one is allowed so that it asks for what its key requires, and for no other reason. The reader may quote a passage it found confusing; it is never asked to rate the document, say what would make a passage clearer, or propose a change: the cohort confirms comprehension and does not design the text. diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md new file mode 100644 index 0000000..e53b671 --- /dev/null +++ b/acceptance/ROUTED.md @@ -0,0 +1,59 @@ +# Passages routed out of a cohort + +A finding is a passage two or more readers of one cohort named as confusing under question 4. +`ACCEPTANCE.md` routes here a finding that lies outside the text the change being tested wrote +or altered, rather than blocking the change whose cohort raised it, so that each change is +judged on what it changed and the document is still made clearer, one change at a time. The +editor reads this list before cutting a release. + +A finding is open until a change that reworks its passage passes a cohort, when the line names +that run and moves below the rule, or until the editor closes it with a reason recorded here. A +finding raised again by a later run is noted on its line, not listed twice. The status column +holds `open`, or the `editor/` branch or pull request number of the change that is +reworking the passage. The rows raised by runs 19 to 52 were raised under the earlier rules, on +untouched text of the change those runs tested, and are listed here so that they are not lost. + +## Open + +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | -------------------------------------------------- | +| `TOOLING-SPEC.md` | [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project), the heading and first paragraph | "SHOULD state its own defaults for anything the method leaves to the project" against clause [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one)'s "MUST NOT supply a default" for a justification | run 86 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), the first paragraph | "It MUST NOT supply a default, and MUST reject an Exception that omits one" against clause [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)'s defaults; named again by two readers of run 85 and two of run 86 | run 83 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) and [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Both clauses speak only of exceptions, where section 2's Project record entry and clause 6.1 cover calibrations and conventions too | run 81 | `open` | +| `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind; named again by four readers of run 81, and by two readers of run 84 as the Project record entry against clauses 6.1, 6.2 and 6.4; and by two readers of run 86, the entry and clause 6.1's first paragraph together | run 76 | `open` | +| `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | +| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68, 82 | `open` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | +| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | + +## Closed + +A row moves here when a change that reworks its passage passes a cohort, naming that run, or +when the editor closes it with a reason. + +| Document | Clause | Passage | Raised by | Closed by | +| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------ | +| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 20, 22, 24, 30, 36, 42, 45, 48, 51, 54, and 71 on a branch without the rework | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 20, 30, 39, 42, and 71 on a branch without the rework | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | runs 20, 54 | `editor/detector-route-rule`, run 66 | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on the two bounds | "Both bounds measure breadth within one level; the level itself is a separate question, answered against the report." | runs 19, 56, and 70 on a branch without the rework | `editor/defined-terms`, run 64 | +| `SPEC.md` | [3.5](../SPEC.md#35-enforce-permanently-and-block), the paragraph on a bundled defence | "For a Bundled defence, the project whose entry point demonstrates it is any project the Toolchain is genuinely installed into", and its substitutes sentence | runs 25, 29, 53, 59 | `editor/defined-terms`, run 64 | diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 132e6cb..dd559a8 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -1,77 +1,105 @@ # Answer key for DETECTOR-SPEC.md -Marked against the document at the version named in the changelog entry that records the run. -Quotations below are from the document; when a clause is reworded, update the quotation here in -the same commit. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, +update the quotation here in the same commit. `ACCEPTANCE.md` says how each item is marked; +`acceptance/README.md` says what each part of this key is for. ## 1. Restatement -A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting an obligation the document does not state, is a correct restatement. +A bullet is correct when it names one of these obligations. Four or more correct bullets, with +no bullet making an assertion listed below as wrong, is a correct restatement. - Support bespoke rules written by the project ([4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it)). - Provide a harness that runs a single rule against supplied code ([4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code)). -- Allow a stable identifier per rule and print it, unaltered, with every finding ([4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding)). +- Allow a stable identifier per rule and print it, unaltered, with every finding ([4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding)); a + bullet that says each rule carries a stable identifier names this obligation. - Be invocable locally by the practitioner, over a subset down to one file, with the result in - the command's own output and never only through a hosted service or CI-only mode ([5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally)). + the command's own output and never only through a hosted service or CI-only mode ([5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally)); + a bullet that says every finding, or each finding, must appear in that output names [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran), + whose summary may leave findings out, and is not marked wrong for the plural. - Resolve a printed identifier to documentation; for bundled rules from the installed copy, offline, with the documentation shipped and versioned with the rule ([6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) to [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together)). - Make any inline suppression route detectable or disableable ([7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)). Wrong if it asserts: the detector must enforce a project's suppression policy; the detector must declare a version to conform; the detector must list a project's active defences or produce an -agent summary (those belong to the toolchain specification). +agent summary (those belong to the toolchain specification). A bullet that states what +conformance means, or restates a SHOULD as advice, asserts no obligation and is neither correct +nor wrong. + +Wrong readings the text invites, which count against the document on one reader: none named. ## 2. Load-bearing statements -- **2a.** Conformance is satisfying every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression), judged on evidence; a detector - that predates the document can conform, and the declaration is optional. Key quotation, section - 8: "A Detector Conforms if it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression)." Supporting, [8.1](../../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps): "A - maintainer MAY declare the version of this document the detector conforms to". An answer that - says a declaration is required, or that partial satisfaction is conformance, is wrong. -- **2b.** It must make the route either disableable by configuration or detectable by a rule the - project can write or a mechanical check it documents; it is not required to remove or forbid - it. Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each one it MUST do at least one of two things, and either alone - satisfies this clause: make the route disableable by configuration; or make the route - detectable, by a Rule the project can write in the Detector itself or by a mechanical check the - Detector documents". An answer that says the detector must forbid the route, or must require a - reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD), is wrong. -- **2c.** With the rule, in the installed copy, at a version tracked together, and yes, offline. - Key quotations, [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access): "Resolution of a bundled rule's identifier MUST work from the installed - copy, without network access"; [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together): "A bundled rule's documentation MUST ship with the rule, at - a version tracked together". An answer naming the detector's website as an acceptable location - is wrong. +- **2a.** Required: (1) a detector conforms when it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression); + (2) yes, a detector that predates the document can conform. Wrong if it says a declaration is + required, or that partial satisfaction is conformance. Key quotation, section [8](../../DETECTOR-SPEC.md#8-conformance): "A Detector + Conforms if it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression)." Supporting, [8.1](../../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps): "A maintainer + MAY declare the version of this document the detector conforms to". The reason, that + conformance is judged on evidence and a declaration is optional, is explanation: an answer + that gives it, or gives it instead of a plain yes, states element (2). +- **2b.** Required: (1) the two things, both named: let the project switch the route off in its + configuration, or make the route detectable (by a rule the project can write, or by a + mechanical check the detector documents; naming either means of detection, or just + "detectable", states that alternative); an answer naming only one of the two things misses + this element; (2) yes, either one alone is enough. Wrong if it says the detector must remove or forbid the + route, or must require a reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD). Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each one it + MUST do at least one of two things, and either alone satisfies this clause: make the route + disableable by configuration; or make the route detectable, by a Rule the project can write in + the Detector itself or by a mechanical check the Detector documents". An answer need not say + that the detector is not required to remove or forbid the route; that follows from element (2) + and the question does not ask it. +- **2c.** Required: (1) with the rule, in the installed copy; (2) at a version tracked together + with the rule; (3) yes, it must be reachable without network access. Wrong if it names the + detector's website, or any location reached over the network, as acceptable. Key quotations, + [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access): "Resolution of a bundled rule's identifier MUST work from the installed copy, without + network access"; [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together): "A bundled rule's documentation MUST ship with the rule, at a version + tracked together". "Shipped with the rule", "in the installed package" or "on disk with the + detector" each state element (1); "versioned with the rule" or "at the same version" states + element (2). ## 3. Application: Lintomatic 4.2 -| Clause | Intended verdict | Reasoning the fixture supports | -| ------ | ---------------- | -------------------------------------------------------------------------------------------------------------------- | -| [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules via the `Rule` interface, first class in every report | -| [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `check --only ` runs one rule, bundled or project, with a distinguishing exit code | -| [4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `LM-NNNN` fixed at first release; project names taken as given, not rewritten; printed on every finding line | -| [5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | Local, no account, no server, no network | -| [5.2](../../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Yes | `check ` accepts a single file | -| [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | One line per finding on standard output whether or not the HTML report is enabled | -| [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | The hosted dashboard shows nothing the command line does not print | -| [6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Yes | `explain ` resolves a printed identifier; project rules resolve to their declared `doc` string | -| [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Yes | `explain` reads the installed package; there is no website | -| [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | No | Two of forty bundled rules print `No documentation declared`; every identifier a bundled rule can print MUST resolve | -| [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | The ignore comment is disableable with `ignore_comments = false` | - -Section [8](../../DETECTOR-SPEC.md#8-conformance) verdict: **does not conform**, because [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) fails and partial conformance MUST NOT be -described as conformance. A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Partial on the grounds that thirty-eight resolve -has read the clause differently and that counts against the document; a reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Yes -because they missed the last paragraph of the fixture has misread the fixture, not the document. - -Clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation): **does not change the verdict**. It is a SHOULD, and Lintomatic's failure to check its -own identifiers in CI is a missed SHOULD, not a further MUST failure; the MUST that fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). - -A reader who marks [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) No because the comment takes no reason has confused [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) with [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason), which is -SHOULD; that counts against the document if the reasoning quotes [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) or [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason). +| Clause | Accepted grade | What the fixture supports (no row carries a condition on the reasoning) | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules via the `Rule` interface, first class in every report | +| [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `check --only ` runs one rule, bundled or project, prints its findings as `check` does, and exits with a distinguishing code; the fixture says all three, so a grade given on the condition that `--only` prints as `check` does is the grade for this case | +| [4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `LM-NNNN` fixed at first release; project names taken as given, not rewritten; printed on every finding line | +| [5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | Local, no account, no server, no network | +| [5.2](../../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Yes | `check ` accepts a single file | +| [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | One line per finding on standard output whether or not the HTML report is enabled; a reader who notes that the HTML report need not be named for that reason is right | +| [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | The hosted dashboard shows nothing the command line does not print | +| [6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Yes | `explain ` resolves a printed identifier; project rules resolve to their declared `doc` string; the two undocumented bundled rules are [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together)'s failure | +| [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Yes | `explain` reads the installed package; there is no website | +| [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | No | Two of forty bundled rules print `No documentation declared`; every identifier a bundled rule can print MUST resolve | +| [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | The ignore comment is disableable with `ignore_comments = false` | + +No grade above carries a condition on the reasoning: the accepted grade is correct whatever the +reasoning, and a reader who adds a caveat about what the fixture does not say and still gives +the accepted grade is correct. + +The two answers the question then asks for, each with the condition its reason must meet: + +- **Conformance under section [8](../../DETECTOR-SPEC.md#8-conformance).** Accepted answer: does not conform. Condition: the reason + names [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), the two undocumented bundled rules, or partial conformance not counting. +- **Whether clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation) changes the verdict.** Accepted answer: it does not. Condition: the reason + says that it is a SHOULD, or that the MUST which fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). + +Wrong readings the text invites, which count against the document on one reader: + +- [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) graded Partial with reasoning that says most, or thirty-eight of forty, bundled rules + resolve, which has read the clause as admitting a proportion. +- [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) graded No with reasoning that says the comment takes no reason and quotes [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) or [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason), + which has read [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason)'s SHOULD into [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable). + +A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Yes because they missed the last paragraph of the fixture has misread the +fixture, which is the reader's miss. ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage named, as `ACCEPTANCE.md` defines naming and the same +passage. A passage named by two or more readers of the run is a finding: it blocks the change +where it lies in text the change wrote or altered, and is routed to `acceptance/ROUTED.md` +otherwise. What to change is the editor's decision; the reader is not asked and any suggestion +it volunteers is discarded unread. diff --git a/acceptance/detector/QUESTIONS.md b/acceptance/detector/QUESTIONS.md index 58d06d1..5c288f8 100644 --- a/acceptance/detector/QUESTIONS.md +++ b/acceptance/detector/QUESTIONS.md @@ -10,21 +10,22 @@ In your own words and in at most six bullets, what does this document require of ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. +Answer each part of each question in a sentence or two, then quote the sentence or sentences of +the document you are relying on. - 2a. What does it mean for a detector to conform, and can a detector that was written before this document existed, and has never heard of it, conform? - 2b. A detector ships an inline ignore comment. What must the detector do about that route for - it to conform? -- 2c. Where must the documentation for a bundled rule's identifier live, and must it be - reachable without network access? + it to conform, and is doing one of those things enough? +- 2c. Where must the documentation for a bundled rule's identifier live, at what version, and + must it be reachable without network access? ## 3. Application Grade Lintomatic 4.2, as described in the fixture, against every MUST clause in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression): [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it), [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), [4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding), [5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure), [5.2](../../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file), [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran), [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally), [6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation), [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access), [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable). For each give Yes, Partial or No and one line of reasoning. Then state whether Lintomatic conforms under section [8](../../DETECTOR-SPEC.md#8-conformance), and why. Finally, -state whether clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation) changes the verdict. +state whether clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation) changes the verdict, and why. ## 4. Confusion diff --git a/acceptance/fixtures/lintomatic.md b/acceptance/fixtures/lintomatic.md index f168d56..9d334fc 100644 --- a/acceptance/fixtures/lintomatic.md +++ b/acceptance/fixtures/lintomatic.md @@ -21,7 +21,8 @@ src/db.zig:88:3 proj.no-raw-sql Query built by string concatenation Fuller detail, including a snippet, is written to `.lintomatic/report.html`. The one-line form is printed whether or not the HTML report is enabled. -`lintomatic check --only ` runs a single rule. It accepts any rule name, +`lintomatic check --only ` runs a single rule and prints its findings as +`check` does. It accepts any rule name, bundled or project, and exits 1 when the rule fires, 0 when it does not, and 2 when the rule name is unknown. diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 3ca1a80..e2ffd60 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -1,13 +1,14 @@ # Answer key for SPEC.md -Marked against the document at the version named in the changelog entry that records the run. -Quotations below are from the document; when a clause is reworded, update the quotation here in -the same commit. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, +update the quotation here in the same commit. `ACCEPTANCE.md` says how each item is marked; +`acceptance/README.md` says what each part of this key is for. ## 1. Restatement -A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting an obligation the document does not state, is a correct restatement. +A bullet is correct when it names one of these obligations. Four or more correct bullets, with +no bullet making an assertion listed below as wrong, is a correct restatement. - Attribute the defect to a class, bounded on both sides, with the hazard stated ([3.1](../../SPEC.md#31-attribute-the-defect-to-a-class)). - Express the class as a rule in a detector that reads code, never a test ([3.2](../../SPEC.md#32-build-the-net)). @@ -18,62 +19,89 @@ bullet asserting an obligation the document does not state, is a correct restate with the rule ([3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation)). - Record a defect whose fix is deferred where the project's other decisions are enumerable, rather than only mentioning it (section [2](../../SPEC.md#2-when-the-method-applies)). +- Take a baseline, a suppression or an instance left unfixed to the owner, never on the + practitioner's own authority (section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom)). +- Fix the original defect conventionally, with a test that reproduces it, once the defence is in + place (sections [2](../../SPEC.md#2-when-the-method-applies) and [6](../../SPEC.md#6-relationship-to-test-driven-development)). +- Write down, for any code the rule is narrowed to exclude, why the hazard cannot arise there, + and take the exclusion to the owner where that sentence cannot be written ([3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire), Part B). Wrong if it asserts: the practitioner fixes the reported instance first and defends afterwards; -a test may serve as the detector; an agent may baseline the remaining instances. +a test may serve as the detector; an agent may baseline the remaining instances. A bullet that +says the verdict on a remediation rests on reproduction (section [7](../../SPEC.md#7-conformance)) describes the reviewer, +not the practitioner, and is neither correct nor wrong. + +Wrong readings the text invites, which count against the document on one reader: none named. ## 2. Load-bearing statements -- **2a.** Attribute the defect to a class, express it as a detector rule, prove the rule fires, - then sweep and fix every instance, in that order. Key quotations, section [3](../../SPEC.md#3-the-method): "Six clauses, in +- **2a.** Required: (1) the four steps before the fix: attribute the defect to a class, express + the class as a rule in a detector, prove the rule fires, and sweep the codebase; (2) in that + order, with the fix after the sweep. Wrong if it has the fix before the rule, or the sweep + before the proof, or a test as the detector. Key quotations, section [3](../../SPEC.md#3-the-method): "Six clauses, in order."; [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance): "The Practitioner MUST run the Rule across the entire codebase and record the total Instance count before fixing anything, and MUST then fix every Instance found rather - than only the one". An answer that has the fix before the rule, or the sweep before the proof, - is wrong. -- **2b.** The owner of the codebase, always a human; an agent may not. Key quotation, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): - "An Agent MUST NOT Baseline, suppress or knowingly leave an Instance unfixed on its own - authority, whatever the Instance count turns out to be." An answer that lets the practitioner - decide because the count was small, or because the project recorded no rule, is wrong (the - default owner paragraph in the terminology entry for Owner still requires a recorded decision). -- **2c.** All six clauses of section [3](../../SPEC.md#3-the-method) were followed for that defect, and the verdict rests on the - reviewer reproducing the red and green runs, not on the report. Key quotations, section [7](../../SPEC.md#7-conformance): "A - remediation Conforms if all six clauses of section [3](../../SPEC.md#3-the-method) were followed for that Defect, and the - verdict that they were rests on reproduction"; "A verdict on a remediation or a Defence MUST - rest on reproduction, not on the report". An answer that accepts the pull request description - as the verdict is wrong, and so is one that states the six clauses and says nothing of - reproduction. + than only the one". An answer that names the steps by their clause numbers in order states + both elements; one that says only that section [3](../../SPEC.md#3-the-method)'s six clauses are followed in order, + naming no step, does not state element (1). +- **2b.** Required: (1) the owner of the codebase, the human who holds that authority; (2) no, + an agent may not do so on its own authority. Wrong if it lets the practitioner or an agent + decide because the count was small, because the project recorded no owner, or at all. Key + quotation, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): "An Agent MUST NOT Baseline, suppress or knowingly leave an Instance + unfixed on its own authority, whatever the Instance count turns out to be." "The owner" names + the defined term and states that the decision is a human's; an answer need not add "a human" + to be correct. The default-owner sentences of the terminology entry for Owner still require a + recorded decision, so an answer naming whoever instructed the practitioner, for a project that + has recorded nobody, names the owner and is correct. +- **2c.** Required: (1) all six clauses of section [3](../../SPEC.md#3-the-method) were followed for that defect; (2) the verdict + rests on reproduction, the reviewer rerunning the defence, not on the report. Wrong if it + accepts the pull request description or the practitioner's report as the verdict, or says + that most clauses suffice. Key quotations, section [7](../../SPEC.md#7-conformance): "A remediation Conforms if all six clauses + of section [3](../../SPEC.md#3-the-method) were followed for that Defect, and the verdict that they were rests on + reproduction"; "A verdict on a remediation or a Defence MUST rest on reproduction, not on the + report". "Reproduction" or "reproducing the runs" states element (2); the red and green runs + need not be named. ## 3. Application: the ledger remediation -| Clause | Intended verdict | Reasoning the fixture supports | -| ----------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | Class, hazard and the next wider rule are named; Partial or No is accepted where the reader notes that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | -| [3.2](../../SPEC.md#32-build-the-net) | Yes | A rule in the project's static analyser, reading code, not a test | -| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | Commit B is red on the fixture and the originating instance and survives as its own commit | -| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Five findings recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority; Partial accepted where the reasoning names that suppression | -| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | The rule is permanent and blocking, but the suppression was not a recorded decision; [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | -| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule; Partial accepted where the reasoning quotes the versioning sentence | - -Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**, because [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) and [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) were not followed. The -**defence does not conform** either, because [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) fails; a reader who says the defence conforms -because the sweep is what failed has missed [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) and that counts against the document only if the -reasoning quotes the section [7](../../SPEC.md#7-conformance) definition of a conforming defence. - -Section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): the **inline ignore on `Report::render()`** was the owner's decision, not the -practitioner's. An answer naming instead the decision not to build the wider rule is half right: -that is also listed in section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) as the owner's to take, and an answer naming both is correct. -An answer naming neither is wrong. - -A reader who grades [3.5](../../SPEC.md#35-enforce-permanently-and-block) Yes because the rule fails the build has not applied the clause's opening -sentence on unrecorded suppression, and that counts against the document. A reader who grades -[3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) Yes because the URL resolves has read the versioning sentence of [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) -differently, which counts against the document; a reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the -fixture. A reader who attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing -passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as covering it, which is a fair reading and is not marked wrong. +| Clause | Accepted grades | Condition on the reasoning | What the fixture supports | +| ----------------------------------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | None | Class, hazard and the next wider rule are named; Partial or No is the reading that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | +| [3.2](../../SPEC.md#32-build-the-net) | Yes | None | A rule in the project's static analyser, reading code, not a test | +| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes, Partial or No | For Partial or No: attributes the inline ignore to this clause, a fair reading of Part B | Commit B is red on the fixture and the originating instance and survives as its own commit | +| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Mentions the instance left unfixed, the inline ignore, or `Report::render()` | Five findings were recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority | +| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | Mentions the suppression, the inline ignore, or that no recorded decision covers it | The rule is permanent and blocking, but the suppression was not a recorded decision. [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | +| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | Mentions the wiki, or that the documentation is not versioned or co-located with the rule | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule | + +The three answers the question then asks for, each with the condition its reason must meet: + +- **The remediation under section [7](../../SPEC.md#7-conformance).** Accepted answer: does not conform. Condition: the reason + names [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), [3.5](../../SPEC.md#35-enforce-permanently-and-block) or [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the clauses not followed; other clauses named alongside are not + marked. +- **The defence alone.** Accepted answer: does not conform. Condition: the reason names [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation), or + [3.5](../../SPEC.md#35-enforce-permanently-and-block)'s unrecorded suppression, which section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence also + requires to be absent; other clauses named alongside are not marked. +- **The decision section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) says was not the practitioner's.** Accepted answer: the inline ignore on + `Report::render()`, which was the owner's decision. Condition: none; any other decision named + alongside it is not marked. An answer naming only the decision to leave the wider rule unbuilt, + or naming none, is wrong. + +Wrong readings the text invites, which count against the document on one reader: + +- [3.5](../../SPEC.md#35-enforce-permanently-and-block) graded Yes with reasoning that says the rule blocks, or is permanent, and does not mention + the suppression. +- [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) graded Yes with reasoning that says the identifier resolves, or the URL works, and does not + mention versioning or co-location. +- The defence graded as conforming with reasoning that says only the sweep failed and that + quotes section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence. + +A reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the fixture, which is the reader's miss, whether or not +the inline ignore was attributed to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) instead. ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage named, as `ACCEPTANCE.md` defines naming and the same +passage. A passage named by two or more readers of the run is a finding: it blocks the change +where it lies in text the change wrote or altered, and is routed to `acceptance/ROUTED.md` +otherwise. What to change is the editor's decision; the reader is not asked and any suggestion +it volunteers is discarded unread. diff --git a/acceptance/method/QUESTIONS.md b/acceptance/method/QUESTIONS.md index 072a3c0..9a2f393 100644 --- a/acceptance/method/QUESTIONS.md +++ b/acceptance/method/QUESTIONS.md @@ -11,7 +11,8 @@ do when they are handed a defect? ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. +Answer each part of each question in a sentence or two, then quote the sentence or sentences of +the document you are relying on. - 2a. What does a practitioner do before fixing the defect they were handed, and in what order? - 2b. Who may adopt a baseline or suppress an instance, and may an agent do so on its own @@ -23,8 +24,8 @@ Answer each in one sentence, followed by the sentence of the document you are re Grade the ledger remediation, as described in the fixture, against each of the six clauses of section [3](../../SPEC.md#3-the-method): [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class), [3.2](../../SPEC.md#32-build-the-net), [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire), [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), [3.5](../../SPEC.md#35-enforce-permanently-and-block), [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation). For each give Yes, Partial or No and one line of -reasoning. Then state whether the remediation conforms under section [7](../../SPEC.md#7-conformance), whether the defence -alone conforms, and which decision in the fixture, if any, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) says was not the +reasoning. Then state whether the remediation conforms under section [7](../../SPEC.md#7-conformance) and why, whether the defence +alone conforms and why, and which decision in the fixture, if any, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) says was not the practitioner's to take. ## 4. Confusion diff --git a/acceptance/runs/057-toolchain-contradictions-run-1.md b/acceptance/runs/057-toolchain-contradictions-run-1.md new file mode 100644 index 0000000..fdd3244 --- /dev/null +++ b/acceptance/runs/057-toolchain-contradictions-run-1.md @@ -0,0 +1,148 @@ +# Run 57: TOOLING-SPEC.md, toolchain contradictions, first run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch + `editor/toolchain-contradictions`. The readers read commit 4d6ebbd. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. +- **Run within the pull request**: 1 +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `00f3043ad8fef7967b0915398bcfaca27d646f8112b0b6df4f9d7d75065eda60`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at 4d6ebbd, each after one + `===== FILE: =====` line; the marker rebuilt it from that commit and the hash matched. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. +- **Process change**: the run was made under the acceptance process change of pull request 22 + before it merged, following all of it at commit c31d49a, named before the readers started. +- **Verdict**: FAIL + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | ------------ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | miss (Part.) | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | miss (Part.) | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Partial (1, 2, 5), No (3, 4); +6.1 Yes (1, 4, 5), Partial (2, 3), both Partials mentioning that no place is given for +calibrations or conventions; 6.2 No (all five), each mentioning that nothing rejects a generic +justification. + +Question 1: correct bullets per reader 4, 5, 5, 5 and 5 (reader 1: 4.1, 4.2, 4.3 with 4.4 and +4.5, sections 5 and 6). No bullet asserts anything the key marks wrong. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote the section 1 sentence "It is whatever the project assembles to run its +checks through", which states element (2). + +### Misses + +- **Reader 1, 4.3, Partial.** The inline `// lintomatic-ignore` route is disabled, but "the + fixture is silent on any other route, such as a baseline, so 'every route' is not shown". The + reasoning does not quote the document (its only quotation is two words). +- **Reader 2, 4.3, Partial.** The inline route is disabled and exceptions go to the record, but + the fixture does not say whether Lintomatic has a baseline or another route, "and the clause + covers every route. I cannot confirm the list is complete." The reasoning does not quote the + document. + +The key accepts Yes only for 4.3 and names no wrong reading for it, so neither miss counts +against the document on its own; two misses on one item fail the threshold for questions 1 and +3\. Readers 3, 4 and 5 also noted the fixture's silence on baselines and graded Yes on what is +written. No miss matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..4d6ebbd adds, +rewords or deletes a sentence in the paragraph. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------- | ------------------- | +| 1 | Section 2 terminology table, [Defence] row: "A [Rule] with its documentation, in force and [Blocking]", set against clause 4.2's first paragraph | 1, 2, 3, 5 | no | +| 2 | Clause 4.2, first paragraph: "A [Blocking] [Rule] whose documentation is missing ... graded here as a [Defence] with a gap" (1, 2, 3, 5); "covers every [Identifier] the project can see" (4) | 1, 2, 3, 4, 5 | yes | +| 3 | Clause 4.2, second paragraph: "A third-party [Detector]'s native catalogue ... is resolved under that document's clause 6 and is not re-shipped here", set against the first paragraph | 3, 4 | no | +| 4 | Section 1, "How to read this document": "It is self-contained for grading a [Toolchain]" (2, set against clause 4.1's "every one of"); "The table in section 2 glosses every term it borrows" (3) | 2, 3 | no | +| 5 | Clause 9.1, first paragraph: "as it stands when installed into a [Consuming project] with nothing else built around it" (1); "is graded against this document and section 7 of the method specification" (5) | 1, 5 | no | +| 6 | Section 9, second paragraph: "A [Toolchain] [Conforms] with [Agent] support if it additionally satisfies section 7", section 7 being all SHOULDs | 3, 5 | no | +| 7 | Clause 4.1, first paragraph: requires "every one of" the detector specification's MUSTs in its sections 4 to 7 | 2 | yes | +| 8 | Section 9, first paragraph: "assembled parts, wherever each part came from" | 1 | yes | +| 9 | Clause 4.1, point 1: "is then what the [Practitioner] uses and what is judged" | 1 | no | +| 10 | Section 2 terminology, [Bundled defence] entry: "with its documentation, [Blocking], and routed through the [Toolchain]'s own entry point" | 4 | no | +| 11 | Clause 8.1, first paragraph: "a [Bundled defence] whose documentation is missing is still a [Bundled defence] for this clause" | 4 | yes | +| 12 | Section 3 table, [Identifier] resolution row, routing [Bundled rules] to the detector specification | 3 | no | +| 13 | Clause 4.3, first paragraph: "MUST disable every [Suppression] route ... or MUST run a [Blocking] [Defence] that fails on its use" | 3 | no | +| 14 | Clause 4.3, second paragraph: "This clause is where the project's decision is made and enforced, and the decision is no." | 3 | no | + +## Findings and where they lie + +- **Finding A, passage 2, clause 4.2 first paragraph** (five readers): text the change wrote. + Blocks. +- **Finding B, passage 1, section 2 [Defence] row** (four readers): every reader naming it sets + it against clause 4.2's first paragraph, which the change altered. Blocks. +- **Finding C, passage 3, clause 4.2 second paragraph** (readers 3 and 4): both set it against + clause 4.2's first paragraph, which the change altered. Blocks. +- **Finding D, passage 4, section 1 "How to read this document"** (readers 2 and 3): reader 2 + sets it against clause 4.1's first paragraph, which the change altered. Blocks. +- **Finding E, passage 5, clause 9.1 first paragraph** (readers 1 and 5): reader 1 sets it + against section 9's first paragraph, which the change altered. Blocks. +- **Finding F, passage 6, section 9 second paragraph** (readers 3 and 5): neither reader sets it + against a paragraph the change altered; both set it against section 7, which is unchanged. + Routed to `acceptance/ROUTED.md`; does not block. + +Passages 7 to 14 are each named by one reader: recorded, not findings. + +## Points the rules did not settle + +1. **Reader 5, clause 9.1.** The reader quotes "is graded against this document and section 7 of + the method specification" and says it could not resolve the reference because it had only + this document. That is close to the example of a remark that is not about a passage (another + document not supplied), but it quotes the document and names its clause. Marked by the + open-point rule: an open point on whether a reader named a passage counts the passage. It + therefore counts, and with reader 1 makes finding E. +2. **Question 1 bullets that state how conformance is measured rather than what the toolchain + must do**: reader 1's first and sixth ("judged on the whole toolchain"; "one project-level + grade"), reader 2's first, reader 4's first sentence ("a single conformance grade"), reader + 5's first and reader 3's sixth ("Self-audit (8) and the two-grade split (9.1) apply only to a + toolchain that a project ships"). They are on neither of the key's lists. Marked by the rule + for a bullet on neither list: not counted either way. Every reader has four or more correct + bullets without them, so no mark turns on it. For the key to settle. +3. **Section 7 described as "all SHOULDs"** (readers 3 and 5) is a description of a whole + section, which identifies no sentence or paragraph, so under the definition of naming it is + not a passage of its own; it is the reason both readers give for naming section 9's second + paragraph (passage 6). Had it counted, it would be a passage the change did not touch and + would be routed, not block. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item except + 4.3, which is three of five. Not met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: five (findings A + to E). Not met. +- Passages named by two or more readers elsewhere: one (finding F), routed to + `acceptance/ROUTED.md`. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125956319); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125955854); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125956709). diff --git a/acceptance/runs/058-toolchain-contradictions-run-2.md b/acceptance/runs/058-toolchain-contradictions-run-2.md new file mode 100644 index 0000000..7adaa89 --- /dev/null +++ b/acceptance/runs/058-toolchain-contradictions-run-2.md @@ -0,0 +1,150 @@ +# Run 58: TOOLING-SPEC.md, toolchain contradictions, second run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch + `editor/toolchain-contradictions`. The readers read commit 90a9936. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 57 in section 1 and clauses 4.1, 4.2, 4.3, 8.1, 9 and 9.1. +- **Run within the pull request**: 2 (run 57 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `43a220ebce6d3db005704ef829656671ecded332c3f3daf275b73db8abf0b825`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at 90a9936, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Partial (1, 2, 4, 5), No (3); +6.1 Yes (all five); 6.2 No (all five), each mentioning that nothing rejects a generic +justification. Every reader graded 4.3 Yes; readers 1 to 5 each noted that the fixture names no +route beyond the ignore comment. + +Question 1: correct bullets per reader 5, 5, 5, 5 and 5 (reader 1: 4.1; 4.2; 4.3; 5.1 to 5.3 +with 4.5; 6.1 to 6.3). No bullet asserts anything the key marks wrong. Bullets on neither list +are listed under the points the rules did not settle. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through", which states element (2); for 2b all five quote section 9's "The project-level +verdict is a single grade; ..."; for 2c all five quote clause 9.1's "The two verdicts MUST be +graded and declared separately, and neither implies the other." + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says there is no agent summary (all +five); S8 says Zigpipe ships no tooling (all five; reader 5 answers "No, to Zigpipe's project +grade" and places section 8 on zig-qa's artefact grade); Decl says the declaration is optional +(all five). + +### Misses + +None. No answer matches a wrong reading the key names: reader 5's 4.1 reasoning says Lintomatic +"meets most of the detector specification" but grades No. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..90a9936 adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------------------- | +| 1 | Clause 4.2, first paragraph: "Here [Defence] reaches every [Blocking] [Rule] the [Toolchain] runs, bundled or the project's own, whether or not its documentation exists" | 1, 3, 4, 5 | yes | +| 2 | Section 2 terminology table, [Defence] row: "A [Rule] with its documentation, in force and [Blocking]", set against clause 4.2's first paragraph | 1, 3, 4, 5 | no | +| 3 | Clause 4.1, first paragraph: "the six summarised here are the ones most often failed" (3, set against the Why paragraph's "six clauses of the method"); "That document's MUSTs are those of its sections 4 to 7, and every one of them applies; ... rests on them" (4) | 3, 4 | yes | +| 4 | Section 9, first paragraph: "The project-level verdict is a single grade" (1, set against the clause 9.1 heading); "every MUST in section 8 bears on its artefact grade under clause 9.1, not on this one", what "this one" refers to (2) | 1, 2 | yes | +| 5 | Clause 5.1, first paragraph: listing "every [Defence] active", set against clause 4.2 and the [Defence] row | 1 | no | +| 6 | Clause 9.1 heading: "has two levels of conformance", set against section 9's first paragraph | 1 | no | +| 7 | Clause 4.1, point 2: "whether or not a clause below names the same gap again" | 1 | yes | +| 8 | Clause 8.2, last paragraph: "has no release and no [Consuming project], so this section does not bear on its [Conformance]" | 2 | no | +| 9 | Clause 4.1, Why paragraph: "the six clauses of the method are carried out in a [Detector]" | 3 | no | +| 10 | Clause 6.2, last paragraph: "The [Toolchain] MUST reject a justification that could be pasted onto any [Exception] unchanged, by a check it documents", against the requirement to name hazard and scope | 3 | no | +| 11 | Section 2 terminology, [Bundled defence] entry: "one with its documentation" | 4 | no | +| 12 | Clause 8.1 heading: "if a bundled defence lacks resolvable documentation" | 4 | no | +| 13 | Clause 5.3, second paragraph: "its entry MAY be sourced from wherever it is enabled", what "its entry" refers to | 4 | no | +| 14 | Section 9, second paragraph: "does each thing section 7 recommends", which document's section 7 | 5 | yes | +| 15 | Clause 9.1, first paragraph: "section 7 of the method specification", set against section 9's second paragraph | 5 | yes | +| 16 | Clause 6.4: a SHOULD, and whether an unmet 6.4 affects any grade | 5 | no | + +## Findings and where they lie + +- **Finding A, passage 1, clause 4.2 first paragraph** (readers 1, 3, 4 and 5): the sentence + named was added by the change. Text the change wrote. Blocks. +- **Finding B, passage 2, section 2 [Defence] row** (readers 1, 3, 4 and 5): the row is + unchanged, but every reader naming it sets it against clause 4.2's first paragraph, which the + change altered. Blocks. +- **Finding C, passage 3, clause 4.1 first paragraph** (readers 3 and 4): the change rewrote the + sentences both readers quote. Text the change wrote. Blocks. +- **Finding D, passage 4, section 9 first paragraph** (readers 1 and 2): the change reworded the + paragraph's first sentence, which reader 2 quotes; reader 1 quotes its unchanged second + sentence, which lies in the same paragraph. Text the change altered. Blocks. + +Passages 5 to 16 are each named by one reader: recorded, not findings. Nothing is routed to +`acceptance/ROUTED.md` from this run. + +## Points the rules did not settle + +1. **Question 1 bullets on neither of the key's lists**: reader 1's first and reader 3's first + (the toolchain is judged as one assembled whole), reader 4's first ("must provide every + mechanism the method needs", the governing principle of section 1), reader 2's sixth and + reader 3's sixth (a shipped artefact graded separately, clause 9.1). Marked by the rule for a + bullet on neither list: not counted either way. Every reader has five correct bullets + without them, so no mark turns on it. For the key to settle. +2. **Reader 1's passage 5** gives clause 5.1 with the three-word quotation "every [Defence] + active"; **reader 5's passage 16** gives clause 6.4 with a description ("It is a SHOULD") + that identifies its heading rather than a quoted sentence. Whether each names a passage was + marked by the open-point rule: counted. Each is named by one reader only, so neither makes a + finding either way. +3. **Reader 1's passage 4** quotes section 9's second sentence, which the change did not reword, + in a paragraph whose first sentence it did. The rule places a finding by paragraph, so the + paragraph is the change's; had it been open, the open-point rule would have treated it as the + change's own text in any case. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit c31d49a, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: four (findings A to + D). Not met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A to D alone; every keyed item passed. + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125958931); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125957078); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125959274). diff --git a/acceptance/runs/062-toolchain-contradictions-run-3.md b/acceptance/runs/062-toolchain-contradictions-run-3.md new file mode 100644 index 0000000..4d083f1 --- /dev/null +++ b/acceptance/runs/062-toolchain-contradictions-run-3.md @@ -0,0 +1,149 @@ +# Run 62: TOOLING-SPEC.md, toolchain contradictions, third run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch + `editor/toolchain-contradictions`. The readers read commit 352cfcc. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 58 in clauses 4.1, 4.2, 8.1, 9 and 9.1. +- **Run within the pull request**: 3 (runs 57 and 58 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `7ceebd159dd90a77d1844c6f31d01d93fce2e3e6e38bf33e68f53044056c74aa`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at 352cfcc, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Partial (1, 4), No (2), Yes +(3, 5); 6.1 Yes (1, 4, 5), Partial (2, 3), each Partial mentioning that no location is given for +calibrations or conventions; 6.2 No (all five), each mentioning that nothing rejects a generic +justification. Every reader graded 4.3 Yes, noting that the fixture names no route beyond the +ignore comment. Reader 2 states that it assumed the two undocumented rules are Blocking; the +key reads a 4.2 No on that condition as a No, and the reader's other grades do not turn on it. + +Question 1: correct bullets per reader 6, 5, 4, 5 and 4 (reader 1: section 1's governing +principle; 4.1; 4.2; 4.3; 4.4 with 4.5; 5.1 to 5.2 with 6.2 and 6.3. Reader 3: 4.1; 4.2; 4.3 with +4.5; 5.1, 5.2, 6.2 and 6.3. Reader 5: 4.1; 4.2; 4.3 with 4.5; 5.1 to 5.3 with 6.1 and 6.2). No +bullet asserts anything the key marks wrong. The first bullet of readers 2 to 5 (the toolchain +judged as one assembled whole) states how conformance is measured, which the key counts neither +way. Bullets on neither list are listed under the points the rules did not settle. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through", which states element (2); for 2b all five quote section 9's "The project-level +verdict is a single grade; ..."; for 2c all five quote clause 9.1's "The two verdicts MUST be +graded and declared separately, and neither implies the other." Reader 1 states element (2) of +2c only in that quotation, which the Marking section counts as stated. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says section 7 is not met or there is no +agent summary (all five); S8 says Zigpipe ships no tooling (all five; readers 2, 3, 4 and 5 also +place section 8 on zig-qa's own artefact grade, reader 5 answering "No, not to Zigpipe's project +grade"); Decl says the declaration is optional, or that the verdict rests on the exercise (all +five). + +### Misses + +None. No answer matches a wrong reading the key names: reader 3's 4.1 reasoning says Lintomatic +"meets most of the detector specification" but grades No. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..352cfcc adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------------------- | +| 1 | Clause 4.2, first paragraph: "a [Blocking] [Rule] the [Toolchain] runs whose documentation is missing is not yet a \[Defence\]: that [Rule] is the failure this clause names, not a [Rule] outside its reach" (2, 3, 4, 5; 5 adds "and the clause reaches every [Blocking] [Rule] the [Toolchain] runs"); whether the clause's scope is set by "its [Defences]" or by every Blocking rule | 2, 3, 4, 5 | yes | +| 2 | Section 2 terminology table, [Defence] row: "section 2 defines a defence as a rule with its documentation" (3), "section 2 defines a Defence as having documentation" (5), each set against clause 4.2's first paragraph | 3, 5 | no | +| 3 | Clause 4.3, first paragraph: "Irreducible cases MUST be directed to the [Project record], where clause 6.2 requires a justification", "irreducible" not defined | 1 | yes | +| 4 | Section 2 terminology, [Project record] entry: "[Calibrations], [Exceptions] and conventions", set against clause 6.1 | 2 | no | +| 5 | Clause 6.1, first paragraph: "The [Toolchain] MUST define where the [Project record] lives and MUST load it", whether one location for all three kinds or only for exceptions | 2 | no | +| 6 | Clause 5.1, first paragraph: "every [Defence]", set against clause 4.2's first paragraph | 3 | no | +| 7 | Clause 6.2 heading: "reject a generic one" | 4 | no | +| 8 | Clause 6.2, last paragraph: "name the [Hazard] being accepted and the scope", the documented check of phrases, "cannot verify truth"; whether the check is the phrase list, a check that hazard and scope are named, or both | 4 | no | +| 9 | Clause 4.4, first paragraph: "so that clause 4.3 of the [detector specification] holds through the wrapping", set against section 1's "How to read" paragraph | 5 | no | +| 10 | Section 1, "How to read this document" paragraph: "self-contained for grading a [Toolchain], apart from clause 4.1", set against clause 4.4 | 5 | yes | + +Notes, not passages: reader 1 (question 3 preamble) and reader 4 (4.1 reasoning) remark that +the detector specification was not supplied; neither is under question 4. + +## Findings and where they lie + +- **Finding A, passage 1, clause 4.2 first paragraph** (readers 2, 3, 4 and 5): the sentences + named were added by the change. Text the change wrote. Blocks. +- **Finding B, passage 2, section 2 [Defence] row** (readers 3 and 5): the row is unchanged, but + both readers naming it set it against clause 4.2's first paragraph, which the change altered. + Blocks. + +Passages 3 to 10 are each named by one reader: recorded, not findings. Nothing is routed to +`acceptance/ROUTED.md` from this run. + +## Points the rules did not settle + +1. **Question 1 bullets on neither of the key's lists**: reader 3's sixth ("Agent-context + summary and automatic delivery are recommended (SHOULD)", with "a shipped artefact has a + separate grade with a self-audit") and reader 5's sixth ("partial conformance must not be + called conformance", with the single grade and the separate artefact grade). The SHOULD half + of reader 3's bullet and the section 9 partial-conformance MUST in reader 5's are on neither + list; the rest of each states how conformance is measured, which the key counts neither way. + Marked by the rule for a bullet on neither list: not counted either way. Readers 3 and 5 each + have four correct bullets without them, so no mark turns on it. For the key to settle. +2. **Passage 2**: readers 3 and 5 identify the [Defence] row by a section number and a + description, inside their remark on clause 4.2, not as a passage named on its own. Whether + that names the row was marked by the open-point rule: counted. Finding B rests on it; finding + A, and so the verdict, does not. +3. **Reader 4's 6.2 remark** names the heading and the last paragraph, each a paragraph of its + own, and **reader 2's 6.1 remark** names clause 6.1 and the [Project record] entry; each is + recorded as two passages. No other reader names any of them, so no finding turns on it. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit e6bdf40, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: two (findings A + and B). Not met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A and B alone; every keyed item passed. + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125960137); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125959666); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125960585). diff --git a/acceptance/runs/065-toolchain-contradictions-run-4.md b/acceptance/runs/065-toolchain-contradictions-run-4.md new file mode 100644 index 0000000..00d637f --- /dev/null +++ b/acceptance/runs/065-toolchain-contradictions-run-4.md @@ -0,0 +1,163 @@ +# Run 65: TOOLING-SPEC.md, toolchain contradictions, fourth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch + `editor/toolchain-contradictions`. The readers read commit dfd4b4d. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 62 in clause 4.2's heading and first paragraph. +- **Run within the pull request**: 4 (runs 57, 58 and 62 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `1186cb6938da03e55dfbec783676bac2414b667e1005d7cf831a67424e0ed90e`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at dfd4b4d, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (all five); 6.1 Yes (1, 4, +5), Partial (2, 3), each Partial mentioning that no location is given for calibrations or +conventions; 6.2 No (all five), each mentioning that nothing rejects generic text or a generic +justification. Reader 1 grades 6.1 Yes and adds that it would be Partial if calibrations and +conventions have no location; the grade in its table is Yes, and Yes and Partial are both +accepted. Readers 2 and 3 note that the fixture does not say the two undocumented rules are +Blocking; both grade 4.2 No, which the key accepts with no condition. + +Question 1: correct bullets per reader 6, 5, 4, 5 and 6 (reader 1: section 1's governing +principle; 4.1; 4.2; 4.3; 4.5 with 4.4; 5.1 to 5.3 with 6.1 to 6.3. Reader 2: 4.1; 4.2; 4.3; +4.4 and 4.5 with 5.1 to 5.3; 6.1 to 6.3. Reader 3: 4.1; 4.2; 4.3 with 4.5; 5.1 to 5.3 with 6.1 +to 6.3. Reader 4: 4.1; 4.2; 4.3 with 6.2; 4.5; 5.1, 5.2, 6.1 and 6.3. Reader 5: 4.1; 4.2; 4.3; +4.4; 4.5; sections 5 and 6). Reader 5's fourth and fifth bullets name different clauses of one +line of the key's list and each count. No bullet asserts anything the key marks wrong. The +first bullet of readers 2, 3 and 4 (the toolchain judged as one assembled whole) and reader 3's +sixth (one project grade, a separate artefact grade, the agent summary a SHOULD) state how +conformance is measured or call section 7 a SHOULD, which the key counts neither way; so does +reader 4's sixth, in calling the agent summary and the self-audit "SHOULD or conditional". A +bullet on neither list is listed under the points the rules did not settle. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through", which states element (2); for 2b all five quote section 9's "The project-level +verdict is a single grade; ..."; for 2c all five quote clause 9.1's "The two verdicts MUST be +graded and declared separately, and neither implies the other." No answer gives a grade per +part or a single grade for 2c. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says section 7 is not met or there is no +agent summary, and that the toolchain does not conform (all five); S8 says Zigpipe ships no +tooling (all five, each also placing zig-qa's grade on its own artefact); Decl says the +declaration is optional and the verdict rests on the exercise (all five). + +### Misses + +None. No answer matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..dfd4b4d adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------- | +| 1 | Clause 4.2, first paragraph: "a [Blocking] [Rule] whose documentation is missing is not yet a [Defence]" (2, 4, 5); the clause covers Blocking rules that by the definition are not Defences (2); whether clause 5.1 then covers an undocumented blocking rule (4, 5) | 2, 4, 5 | yes | +| 2 | Section 2 terminology table, [Defence] row: "A [Rule] with its documentation, in force and [Blocking]", each set against clause 4.2's first paragraph | 2, 4, 5 | no | +| 3 | Clause 5.1, first paragraph: "every [Defence] active in the project" (5), "clause 5.1's listing obligation" (4), each set against clause 4.2's first paragraph | 4, 5 | no | +| 4 | Clause 9.1, first paragraph: "graded against this document and section 7 of the method specification" (1, 3); which section 7 governs the project grade, set against section 1's "How to read" paragraph and section 9 (1) or section 9's first paragraph (3) | 1, 3 | yes | +| 5 | Clause 4.2 heading: "[Blocking] [Rules] it routes" (2), given with the clause number inside its remark on the first paragraph | 2 | yes | +| 6 | Section 1, "How to read this document" paragraph: self-contained "apart from 4.1", set against clause 9.1 | 1 | yes | +| 7 | Section 9, agent-support paragraph: "section 7 of this document", set against clause 9.1 | 1 | yes | +| 8 | Section 9, first paragraph: "defines the project grade by sections 4 to 6 of this document", set against clause 9.1 | 3 | yes | +| 9 | Clause 4.3, first paragraph: "A route bypasses the [Project record] when a finding can be silenced through it without an entry appearing in the [Project record]"; whether disabling an active rule in configuration is a route | 1 | yes | +| 10 | Section 2 terminology, [Project record] entry: "[Calibrations], [Exceptions] and conventions", set against clause 6.1 | 3 | no | +| 11 | Clause 6.1, first paragraph: "The [Toolchain] MUST define where the [Project record] lives and MUST load it"; whether the location covers all three kinds of judgement or only exceptions | 3 | no | +| 12 | Clause 4.1, point 2, paragraph after the table: "exercised as its clause 8.1 describes"; the reader could not follow the reference to the detector specification | 4 | yes | +| 13 | Clause 4.5, first paragraph: "MUST stop the levels below it from being treated as meaningful"; the direction of "below" | 4 | no | + +Notes, not passages: every reader remarks under question 3 that the detector specification, the +method specification or both were not supplied; none of those remarks is under question 4. + +## Findings and where they lie + +- **Finding A, passage 1, clause 4.2 first paragraph** (readers 2, 4 and 5): the sentence named + was added by the change. Text the change wrote. Blocks. +- **Finding B, passage 2, section 2 [Defence] row** (readers 2, 4 and 5): the row is unchanged, + but every reader naming it sets it against clause 4.2's first paragraph, which the change + altered. Blocks. +- **Finding C, passage 3, clause 5.1 first paragraph** (readers 4 and 5): the paragraph is + unchanged, but both readers set it against clause 4.2's first paragraph. Blocks. +- **Finding D, passage 4, clause 9.1 first paragraph** (readers 1 and 3): the sentence quoted is + unchanged, but the change rewords and adds sentences in the same paragraph, and both readers + set it against section 9 or section 1's "How to read" paragraph, which the change also altered. + Text the change altered. Blocks. + +Passages 5 to 13 are each named by one reader: recorded, not findings. Nothing is routed to +`acceptance/ROUTED.md` from this run. + +## Points the rules did not settle + +1. **A question 1 bullet on neither of the key's lists**: reader 3's sixth bullet says "default + values are SHOULDs", which is clause 6.4 and on neither list. Marked by the rule for a bullet + on neither list: not counted either way. Reader 3 has four correct bullets without it, so no + mark turns on it. For the key to settle. +2. **Passages set against another and named by description**: reader 4 identifies clause 5.1 as + "clause 5.1's listing obligation", and reader 3 identifies section 9's first paragraph by a + description, each inside a remark on another passage. Whether that names the paragraph was + marked by the open-point rule: counted. Finding C rests on reader 4's being counted; findings + A, B and D, and so the verdict, do not. +3. **Reader 2's short quotation "[Blocking] [Rules] it routes"** matches no run of the body and + is nearest the clause 4.2 heading. Whether it names the heading was marked by the open-point + rule: counted, as passage 5. No other reader names the heading, so no finding turns on it. +4. **Reader 4's remark on "exercised as its clause 8.1 describes"** quotes a passage with its + clause number, but its substance is that the detector specification was not supplied, which + the Marking section calls a note. Marked by the open-point rule: counted as a passage. Named + by reader 4 alone, so no finding turns on it. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 4081aa4, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: four (findings A, + B, C and D). Not met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A, B, C and D; every keyed item passed. + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125961653); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125961175); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125962019). diff --git a/acceptance/runs/067-toolchain-contradictions-run-5.md b/acceptance/runs/067-toolchain-contradictions-run-5.md new file mode 100644 index 0000000..1cc3e13 --- /dev/null +++ b/acceptance/runs/067-toolchain-contradictions-run-5.md @@ -0,0 +1,171 @@ +# Run 67: TOOLING-SPEC.md, toolchain contradictions, fifth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch + `editor/toolchain-contradictions`. The readers read commit 645d811. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 65 in clause 4.2's heading and first paragraph, clause 8.1's first paragraph + and clause 9.1's first paragraph. +- **Run within the pull request**: 5 (runs 57, 58, 62 and 65 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `efd52c615ca1e070ef6fc016a6d6eaeaac069cb9baf184ba36805631a0fabcf3`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at 645d811, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (1), Partial (2, 4, 5), No +(3); 6.1 Yes (1, 4, 5), Partial (2, 3), each Partial mentioning that no location is given for +calibrations or conventions; 6.2 No (all five), each mentioning that nothing rejects generic +text or a generic justification. Readers 1, 3, 4 and 5 note that the fixture does not say the +two undocumented rules are Blocking and grade 4.2 No on that assumption or inference, which the +key accepts with no condition. + +Question 1: correct bullets per reader 5, 5, 4, 5 and 5 (reader 1: 4.1; 4.2; 4.3 with 4.5; 5.1 +to 5.3; 6.1 to 6.3. Reader 2: 4.1; 4.2; 4.3; 4.4 with 4.5; 5.1 to 5.3 with 6.1 and 6.2. Reader +3: 4.1 with 4.2; 4.3 with 4.5; 5.1 to 5.3; 6.1 to 6.3. Reader 4: 4.1; 4.2; 4.3 with 4.5; 5.1 to +5.3; 6.1 to 6.3. Reader 5: 4.1; 4.2; 4.3 with 6.2; 4.4 with 4.5; 5.1 to 5.3 with 6.3 and 8.2). +Reader 5's fourth and sixth bullets name different clauses of one line of the key's list and +each count. No bullet asserts anything the key marks wrong. The first bullet of every reader +(the toolchain judged as one assembled whole, or one grade) states how conformance is measured, +and reader 3's sixth (a shipped artefact graded separately, section 7 SHOULDs, the declaration +optional) is of the kinds the key counts neither way. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through", which states element (2); for 2b all five quote section 9's "The project-level +verdict is a single grade; ..."; for 2c all five quote clause 9.1's "The two verdicts MUST be +graded and declared separately, and neither implies the other." No answer gives a grade per +part or a single grade for 2c. Reader 1's 2c says "two grades if it ships an artefact"; see the +points the rules did not settle. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says section 7 is not met or there is +no agent summary, and that the toolchain does not conform (all five); S8 says Zigpipe ships no +tooling (all five, readers 1, 2, 3 and 5 also placing zig-qa's grade on its own artefact); Decl +says the declaration is optional, not a condition, or that the verdict rests on the exercise +(all five). + +### Misses + +None. No answer matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..645d811 adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------- | +| 1 | Clause 4.2, first paragraph: "this one covers every [Identifier] the project can see, the project's own [Rules] included" and "covers the project's own [Rules] and every [Bundled defence]", set against the heading (1); "The mechanism is keyed on the [Identifier] exactly as printed", the mechanism not named first (3); "The [Toolchain] MUST resolve every [Identifier] its [Blocking] [Rules] print to [Remediation docs] on disk", the term not glossed in section 2 (5) | 1, 3, 5 | yes | +| 2 | Section 1, "How to read this document" paragraph: "self-contained for grading a [Toolchain], apart from clause 4.1", set against clause 4.4's citation of detector clauses 5.1 to 5.4 (4); described as where "the document says" the section 2 table "is how terms are defined for grading", set against clause 4.2 (5) | 4, 5 | yes | +| 3 | Clause 6.2 heading: "MUST carry a written justification that names the hazard and the scope" (3); "Clause 6.2's heading", "name the hazard and the scope" (4); each set against clause 6.2's fourth paragraph | 3, 4 | no | +| 4 | Clause 6.2, fourth paragraph: "by a check it documents" (3); "by a check it documents: 'needed for now', 'legacy', 'TODO' and their like" and "cannot verify truth" (4); whether the check must test that hazard and scope are named | 3, 4 | no | +| 5 | Clause 4.2 heading: "every [Identifier] a [Blocking] [Rule] it routes can print"; whether non-blocking rules are in scope | 1 | yes | +| 6 | Clause 6.2, first paragraph: "It MUST NOT supply a default", set against clause 6.4 | 1 | no | +| 7 | Clause 6.4 heading: "The [Toolchain] SHOULD state its own defaults for anything the method leaves to the project" | 1 | no | +| 8 | Clause 5.1, first paragraph: "MUST list every Defence active in the project" and "MUST provide the route to each one's full documentation"; whether an undocumented rule is listed | 2 | no | +| 9 | Section 2 terminology table, [Defence] row: "A Rule with its documentation, in force and Blocking", set against clause 5.1 | 2 | no | +| 10 | Clause 4.3, second paragraph: "This clause is where the project's decision is made and enforced, and the decision is no" | 2 | no | +| 11 | Clause 9.1, first paragraph: "MUST be graded and declared separately", set against clause 9.2 | 3 | yes | +| 12 | Clause 9.2, third paragraph: "The declaration is optional." | 3 | yes | +| 13 | Clause 6.1, first paragraph: "define where the [Project record] lives"; whether one path holds all three kinds of judgement | 3 | no | +| 14 | Section 2 terminology, [Project record] entry: the record covering calibrations, exceptions and conventions, set against clause 6.1 | 3 | no | +| 15 | Clause 4.4, first paragraph: "clauses 5.1 to 5.4 of the [detector specification]", set against section 1's "How to read" paragraph | 4 | no | +| 16 | Clause 4.5, first paragraph: "in the order method section 5 states"; the method specification not supplied | 4 | no | +| 17 | Clause 5.3, second paragraph: "Where such a [Defence] is not expressible in the [Toolchain]'s own [Detectors], its entry MAY be sourced from wherever it is enabled"; what "enabled" refers to | 5 | no | + +Notes, not passages: readers 2, 3 and 4 remark under question 3 that the detector specification +was not supplied; none of those remarks is under question 4. Reader 5's reference to "the +section 2 table" as a whole names no row of it. + +## Findings and where they lie + +- **Finding A, passage 1, clause 4.2 first paragraph** (readers 1, 3 and 5): the diff rewrites + this paragraph, adding the sentences on [Blocking] [Rules] and rewrapping the sentence reader 3 + quotes. Text the change wrote or altered. Blocks. +- **Finding B, passage 2, section 1 "How to read this document" paragraph** (readers 4 and 5, + reader 5 counted by the open-point rule, point 1 below): the diff adds "apart from clause 4.1, + which holds each [Detector] to the [detector specification](DETECTOR-SPEC.md) and summarises + that document's MUSTs" to this paragraph. Text the change wrote or altered. Blocks. +- **Finding C, passage 3, clause 6.2 heading** (readers 3 and 4): unchanged by the diff, and each + reader sets it only against clause 6.2's fourth paragraph, also unchanged. Elsewhere: to be + recorded in `acceptance/ROUTED.md` and routed; does not block. +- **Finding D, passage 4, clause 6.2 fourth paragraph** (readers 3 and 4): unchanged by the diff, + and set only against clause 6.2's heading. Elsewhere: to be recorded in `acceptance/ROUTED.md` + and routed; does not block. + +Passages 5 to 17 are each named by one reader: recorded, not findings. + +## Points the rules did not settle + +1. **Reader 5's description of section 1's "How to read" paragraph.** Reader 5 gives no clause + number for it, describing it as where "the document says" the section 2 table "is how terms + are defined for grading"; the nearest text is "The table in section 2 glosses every term it + borrows" in that paragraph, with section 2's opening paragraph the other candidate. Whether + that names the paragraph was marked by the open-point rule: counted. Finding B rests on it; + finding A, and so the verdict, does not. +2. **Remarks on another document not supplied, given as passages.** Reader 4's remarks on clauses + 4.4 and 4.5 and reader 5's on clause 4.2 each quote a passage with its clause number, but turn + in part on the detector or method specification not being supplied, which the Marking section + calls a note. Marked by the open-point rule: counted as passages. Finding A stands on readers 1 + and 3 without reader 5; finding B rests on reader 4's clause 4.4 remark being counted. +3. **Reader 1's 2c answer, "two grades if it ships an artefact".** The Marking section gives a + rule for a grade on a stated condition only under question 3. The question's company ships + the package, so read for its sense the answer states two for the case asked about, and its + quotation of clause 9.1 states element (2). Marked for the reader under the open-point rule: + correct. No other mark turns on it. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 9907492, named before the readers started. The process files at +645d811 are identical to those at 9907492. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: two (findings A + and B). Not met. +- Passages named by two or more readers elsewhere: two (findings C and D), for + `acceptance/ROUTED.md`; they do not block. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A and B; every keyed item passed. + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125962818); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125962391); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125963215). diff --git a/acceptance/runs/068-toolchain-contradictions-run-6.md b/acceptance/runs/068-toolchain-contradictions-run-6.md new file mode 100644 index 0000000..cbbbcf9 --- /dev/null +++ b/acceptance/runs/068-toolchain-contradictions-run-6.md @@ -0,0 +1,174 @@ +# Run 68: TOOLING-SPEC.md, toolchain contradictions, sixth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch + `editor/toolchain-contradictions`. The readers read commit 0080b8e. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 67 in clause 4.2 (rewritten in three paragraphs), section 2's table (a + Remediation docs row) and section 1's "How to read this document". +- **Run within the pull request**: 6 (runs 57, 58, 62, 65 and 67 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `dbce8540d4259779cdced1129c5fa71306bbe31597700618177c4a5f48662919`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at 0080b8e, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Partial (1), Yes (2, 3, 4, 5); +6.1 Yes (1, 2, 4, 5), Partial (3), the Partial mentioning that no location is given for +calibrations or conventions; 6.2 No (all five), each mentioning that nothing rejects generic +text or a generic justification. Readers 1, 2, 4 and 5 note that the fixture does not say the +two undocumented rules are Blocking and grade 4.2 No on that assumption or inference, which the +key accepts with no condition. + +Question 1: correct bullets per reader 6, 5, 6, 5 and 4 (reader 1: the governing principle; +4.1; 4.2; 4.3; 4.4 with 4.5; 5.1 to 5.3 with 6.1 to 6.3. Reader 2: 4.1; 4.2; 4.3; 4.4 with 4.5; +5.1 to 5.3 with 6.1 to 6.3. Reader 3: 4.1; 4.2; 4.3; 4.4 with 4.5; 5.1 to 5.3; 6.1 to 6.3. +Reader 4: 4.1 with 4.3; 4.2; 4.5; 5.1 to 5.3 with 6.3; 6.1 and 6.2. Reader 5: 4.1 with 4.2; 4.3; +4.4 with 4.5; 5.1 to 5.3 with 6.1 to 6.3). Reader 4's fourth and fifth bullets name different +clauses of one line of the key's list and each count. No bullet asserts anything the key marks +wrong. The first bullets of readers 2 and 5 (the toolchain judged as one assembled whole) state +how conformance is measured; reader 2's sixth calls the agent summary recommended; reader 4's +sixth (a declaration permitted, a project grade and a separate artefact grade) and reader 5's +sixth (one project grade, a shipped artefact graded separately, section 7 and the declaration +SHOULD or MAY, not conditions) are of the kinds the key counts neither way. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through", which states element (2); for 2b all five quote section 9's "The project-level +verdict is a single grade; ..."; for 2c all five quote clause 9.1's "The two verdicts MUST be +graded and declared separately, and neither implies the other." No answer gives a grade per +part or a single grade for 2c. Reader 1's 2c says "Two grades, if the company's project actually +ships the package"; see the points the rules did not settle. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says the toolchain does not conform +and that there is no agent summary (all five); S8 says Zigpipe ships no tooling (all five, +readers 1, 2 and 3 also placing zig-qa's grade on its own artefact); Decl says the declaration +is optional and the verdict rests on the exercise (all five). + +### Misses + +None. No answer matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..0080b8e adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | ------------------- | +| 1 | Clause 4.2, second paragraph: "The [Toolchain] resolves an [Identifier] by a mechanism of its own, a command, an index file or a URL scheme, keyed on the [Identifier] exactly as printed" and "MUST ship with the project or with the [Toolchain]", a URL scheme set against the heading (1, 5); "MUST ship with the project or with the [Toolchain]" set against the third paragraph (2) | 1, 2, 5 | yes | +| 2 | Clause 4.2 heading: "from the installed copy, without network access", set against the second paragraph's URL scheme | 1, 5 | yes | +| 3 | Clause 4.2, first paragraph: "names [Blocking] [Rules] rather than [Defences] because the failure it catches is a [Rule] that blocks without the documentation that would make it a [Defence]", set against the section 2 Defence row and clause 5.1 | 3, 5 | yes | +| 4 | Section 2 terminology table, [Defence] row: "A [Rule] with its documentation, in force and [Blocking]", set against clause 4.2's first paragraph (3, 5) and clause 5.1 (3, 4, 5) | 3, 4, 5 | no | +| 5 | Clause 5.1, first paragraph: "every [Defence] active in the project"; whether the two undocumented rules, which `zig-qa list` shows, are within it; set against the Defence row (3, 4, 5) and clause 4.2's first paragraph (3, 5) | 3, 4, 5 | no | +| 6 | Clause 4.1, first paragraph: "That document's MUSTs are those of its sections 4 to 7, and this clause holds the [Detector] to every one of them. ... they are a summary, not the test" (3); described as the clause taking its MUSTs from sections 4 to 7 of a document not supplied, graded from the summary (2) | 2, 3 | yes | +| 7 | Clause 6.2, fourth paragraph: "'needed for now', 'legacy', 'TODO' and their like", where the open phrase stops (2); "could be pasted onto any Exception unchanged, by a check it documents", set against the heading (5) | 2, 5 | no | +| 8 | Clause 4.2, third paragraph: "is not re-shipped here", set against the second paragraph | 2 | yes | +| 9 | Clause 6.1, first paragraph: "The [Toolchain] MUST define where the [Project record] lives and MUST load it", set against the Project record entry | 3 | no | +| 10 | Section 2 terminology, [Project record] entry: "[Calibrations], [Exceptions] and conventions", set against clause 6.1 | 3 | no | +| 11 | Clause 6.2 heading: "names the hazard and the scope", set against the fourth paragraph | 5 | no | +| 12 | Clause 4.3, second paragraph: "This clause is where the project's decision is made and enforced, and the decision is no", set against the Why paragraph | 5 | no | +| 13 | Clause 4.3, Why paragraph: suppression "an Owner decision", set against the second paragraph | 5 | no | +| 14 | Clause 4.1, point 2, "Behind the table" paragraph: its reference to the detector specification's "clause 8.1", that document not supplied | 5 | yes | + +Notes, not passages: readers 2, 3, 4 and 5 remark under question 3 that the detector +specification was not supplied; none of those remarks is under question 4. The section 2 table +was realigned by the diff for the new Remediation docs row; the Defence row's words, terms and +links are unchanged, so passage 4 is marked unchanged. + +## Findings and where they lie + +- **Finding A, passage 1, clause 4.2 second paragraph** (readers 1, 2 and 5): the diff writes + this paragraph's first sentence and rewraps the "MUST ship" sentence into it. Text the change + wrote or altered. Blocks. +- **Finding B, passage 2, clause 4.2 heading** (readers 1 and 5): the diff changes "a defence" + to "a blocking rule" in it. Text the change wrote or altered. Blocks. +- **Finding C, passage 3, clause 4.2 first paragraph** (readers 3 and 5): written by the diff. + Text the change wrote or altered. Blocks. +- **Finding D, passage 4, section 2 Defence row** (readers 3, 4 and 5): unchanged, but readers + 3 and 5 set it against clause 4.2's first paragraph, which the diff wrote. Text the change + wrote or altered, under the Marking section's rule on a paragraph a reader sets against one. + Blocks. +- **Finding E, passage 5, clause 5.1 first paragraph** (readers 3, 4 and 5): unchanged, but + readers 3 and 5 set it against clause 4.2's first paragraph. Text the change wrote or altered, + as for finding D. Blocks. +- **Finding F, passage 6, clause 4.1 first paragraph** (readers 2 and 3, counted by the + open-point rule, point 1 below): the diff rewrites the paragraph, adding "That document's + MUSTs are those of its sections 4 to 7" and "they are a summary, not the test". Text the + change wrote or altered. Blocks. +- **Finding G, passage 7, clause 6.2 fourth paragraph** (readers 2 and 5): unchanged by the + diff, and set only against clause 6.2's heading, also unchanged. Elsewhere: to be recorded in + `acceptance/ROUTED.md`, on the open clause 6.2 line run 67 raised, and routed; does not block. + +Passages 8 to 14 are each named by one reader: recorded, not findings. + +## Points the rules did not settle + +1. **Remarks on another document not supplied, given as passages.** Readers 2 and 3 name clause + 4.1's first paragraph under question 4, reader 3 by quotation and reader 2 by clause number + and a description identifying its "sections 4 to 7" sentence, but each turns in part on the + detector specification not being supplied, which the Marking section calls a note. Reader 5's + item on clause 4.1 point 2 is of the same kind. Marked by the open-point rule: counted as + passages. Finding F rests on readers 2 and 3 both being counted; findings A to E, and so the + verdict, do not. +2. **Reader 1's 2c answer, "Two grades, if the company's project actually ships the package".** + The Marking section gives a rule for a grade on a stated condition only under question 3. The + question's company publishes the package, so read for its sense the answer states two for the + case asked about, and its quotation of clause 9.1 states element (2). Marked for the reader + under the open-point rule: correct. No other mark turns on it. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit f3ec69e, named before the readers started. The process files at +0080b8e are identical to those at f3ec69e. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: six (findings A + to F). Not met. +- Passages named by two or more readers elsewhere: one (finding G), for `acceptance/ROUTED.md`; + it does not block. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A to F; every keyed item passed. + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125964042); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125963595); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125964458). diff --git a/acceptance/runs/072-toolchain-contradictions-run-7.md b/acceptance/runs/072-toolchain-contradictions-run-7.md new file mode 100644 index 0000000..e508d14 --- /dev/null +++ b/acceptance/runs/072-toolchain-contradictions-run-7.md @@ -0,0 +1,167 @@ +# Run 72: TOOLING-SPEC.md, toolchain contradictions, seventh run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch + `editor/toolchain-contradictions`. The readers read commit cb9b4e5. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 68 in clause 4.2 (which now states only what it requires, with no sentence on + defences and no list of mechanisms), clause 4.1's first paragraph and section 1's "How to read + this document". +- **Run within the pull request**: 7 (runs 57, 58, 62, 65, 67 and 68 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `6c9683df16a39ddd12169c438fe1f9c8a32050bd5346a81de4edb1c0d6990472`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at cb9b4e5, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`, which + the marker confirmed byte for byte. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Partial (1, 2), Yes (3, 4, 5); +6.1 Partial (1, 5), Yes (2, 3, 4), each Partial mentioning that the fixture gives no location for +calibrations or conventions; 6.2 No (all five), each mentioning that nothing rejects generic +text or a generic justification. Readers 1, 2, 3 and 4 note that the fixture does not say the +two undocumented rules are Blocking and grade 4.2 No on that assumption or inference, which the +key reads as a No. Readers 1, 4 and 5 grade 4.3 Yes on what the fixture writes, reader 1 by +assuming Lintomatic has no other route; see the points the rules did not settle. + +Question 1: correct bullets per reader 6, 5, 5, 6 and 5 (reader 1: 4.1; 4.2; 4.3; 4.5 with 4.4; +5.1 to 5.3 with 6.1 to 6.3; 8.1 and 8.2. Readers 2, 3 and 5: 4.1; 4.2; 4.3 with 4.5; 5.1 to 5.3; +6.1 to 6.3. Reader 4: the governing principle; 4.1; 4.2; 4.3 with 4.5 and 4.4; 5.1 to 5.3; 6.1 +to 6.3). No bullet asserts anything the key marks wrong. The first bullets of readers 2, 3 and 5 +(the toolchain judged as one assembled whole) and the first sentence of reader 1's sixth (the +project level and a shipped artefact graded separately) state how conformance is measured, which +the key counts neither way. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through", which states element (2); for 2b all five quote section 9's "The project-level +verdict is a single grade; ..."; for 2c all five quote clause 9.1's "The two verdicts MUST be +graded and declared separately, and neither implies the other." No answer gives a grade per +part or a single grade for 2c. Readers 2 and 4 add to 2b that Zigpipe, shipping nothing, has +only the one grade, which the key does not mark wrong. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says the toolchain does not conform +and that there is no agent summary (all five); S8 says Zigpipe ships no tooling (all five, +readers 1, 2, 4 and 5 also placing zig-qa's grade on its own artefact); Decl says the +declaration is optional and the verdict rests on the exercise (all five). + +### Misses + +None. No answer matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..cb9b4e5 adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------------------- | +| 1 | Section 2 terminology table, [Defence] row: "A [Rule] with its documentation, in force and [Blocking]", whether the two undocumented rules are Defences; set against clause 4.2's first paragraph (1) and clause 5.1 (1, 2, 3, 4) | 1, 2, 3, 4 | no | +| 2 | Clause 5.1, first paragraph: "every [Defence] active in the project" (1, 3, 4), "MUST provide the route to each one's full documentation" (2); set against the Defence row (1, 2, 3, 4) and, by reader 1, against clause 4.2's first paragraph, asking which of 4.2 and 5.1 governs the undocumented rules | 1, 2, 3, 4 | no | +| 3 | Section 1, "How to read this document": "A [Toolchain] can be graded from this document alone", set against clause 4.1's first paragraph | 2, 3 | yes | +| 4 | Clause 4.1, first paragraph: "that fails any one of that document's MUSTs, restated here or not, fails this clause" (2, 3) and "MUSTs are those of its sections 4 to 7" (2), set against section 1's "graded from this document alone" | 2, 3 | yes | +| 5 | Clause 4.2, first paragraph: "every [Identifier] its [Blocking] [Rules] print", set against the Defence row and clause 5.1 | 1 | yes | +| 6 | Clause 9.2, first paragraph: "A project or a shipped artefact MAY declare" against "A project that ships an artefact carries both levels of clause 9.1 in that declaration", read as unconditional | 1 | no | +| 7 | Clause 6.2, first paragraph: "MUST NOT supply a default", set against the clause 6.4 heading | 3 | no | +| 8 | Clause 6.4 heading: "SHOULD state its own defaults for anything the method leaves to the project", set against clause 6.2's first paragraph | 3 | no | +| 9 | Clause 6.2, fourth paragraph: "The justification MUST name the [Hazard] being accepted and the scope of the [Exception]. The [Toolchain] MUST reject a justification that could be pasted onto any [Exception] unchanged, by a check it documents", whether the check must test hazard and scope or only generic text | 4 | no | +| 10 | Clause 5.3, second paragraph: "Where such a [Defence] is not expressible in the [Toolchain]'s own [Detectors], its entry MAY be sourced from wherever it is enabled, provided the listing stays derived under clause 5.2 rather than hand-maintained", what "wherever it is enabled" covers | 5 | no | + +Notes, not passages: readers 2, 3 and 5 remark under question 3 that the detector specification +was not supplied; none of those remarks is under question 4. The section 2 table was realigned +by the diff for the Remediation docs row; the Defence row's words, terms and links are +unchanged, so passage 1 is marked unchanged. + +## Findings and where they lie + +- **Finding A, passage 1, section 2 Defence row** (readers 1, 2, 3 and 4): unchanged, but + reader 1 sets it against clause 4.2's first paragraph, which the diff rewrote. Text the change + wrote or altered, under the Marking section's rule on a paragraph a reader sets against one. + Blocks. +- **Finding B, passage 2, clause 5.1 first paragraph** (readers 1, 2, 3 and 4): unchanged, but + reader 1 sets it against clause 4.2's first paragraph, asking which of the two governs the + undocumented rules. Text the change wrote or altered, as for finding A. Blocks. +- **Finding C, passage 3, section 1 "How to read this document"** (readers 2 and 3): the diff + rewrites the paragraph's opening, adding "A [Toolchain] can be graded from this document + alone". Text the change wrote or altered. Blocks. +- **Finding D, passage 4, clause 4.1 first paragraph** (readers 2 and 3): the diff rewrites the + paragraph, adding "That document's MUSTs are those of its sections 4 to 7" and "restated here + or not". Text the change wrote or altered. Blocks. + +Passages 5 to 10 are each named by one reader: recorded, not findings. Passages 7 and 9 both lie +in clause 6.2 but in different paragraphs, so they are not the same passage. No finding lies +elsewhere, so nothing from this run goes to `acceptance/ROUTED.md`. + +## Points the rules did not settle + +1. **Clause 4.3 graded Yes on a stated assumption.** Reader 1 grades 4.3 Yes and says the + fixture does not say whether Lintomatic offers any other suppression route, "so this verdict + assumes it does not"; readers 4 and 5 say their Yes holds "only for what is written". The + Marking section reads a grade given on a stated condition as the grade for the case the + fixture presents only where the key says which case that is, and the key's 4.3 row does not + say whether Lintomatic's only route is the ignore comment. Marked for the reader under the + open-point rule: correct. Had the three been marked unanswered, 4.3 would have failed the + all-but-one threshold; the verdict below does not turn on it, since the run fails on the + findings. For the editor to settle in the key before the next run of this pull request. +2. **Readers 2 and 3 on section 1 against clause 4.1.** Each turns on grading 4.1 without the + detector specification, which the Marking section calls a note when the remark is not about a + passage. Both quote section 1's and clause 4.1's sentences and ask how they fit together, + which is a remark about those passages, so the marker counts them as passages; were that in + doubt, the open-point rule counts the passage. Findings C and D rest on it; findings A and B, + and so the verdict, do not. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit bc6ae14, named before the readers started. The process files at +cb9b4e5 (`ACCEPTANCE.md`, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md`, `tools/changes.py`, `spec-qa.py` and the CI workflow) are identical to those +at bc6ae14. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: four (findings A + to D). Not met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A to D; every keyed item passed. + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125965287); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125964789); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125965683). diff --git a/acceptance/runs/076-toolchain-contradictions-run-8.md b/acceptance/runs/076-toolchain-contradictions-run-8.md new file mode 100644 index 0000000..2dc28eb --- /dev/null +++ b/acceptance/runs/076-toolchain-contradictions-run-8.md @@ -0,0 +1,167 @@ +# Run 76: TOOLING-SPEC.md, toolchain contradictions, eighth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch + `editor/toolchain-contradictions`. The readers read commit fa79f8a. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 72: section 2 gains a paragraph under its table saying a blocking rule whose + documentation is missing is not yet a defence and is what clauses 4.2, 5.1 and 8.1 exist to + catch; clause 5.1 says the listing covers every blocking rule, documented or not, with a + missing route showing; section 1's "How to read this document" and clause 4.1's first + paragraph both say a verdict from this document alone rests on what is restated and a grader + with the detector specification holds the detector to all of it. The key's 4.3 row now says + which case the fixture presents. +- **Run within the pull request**: 8 (runs 57, 58, 62, 65, 67, 68 and 72 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `18f9a24ca270450e83434b252e766ca54bcc6fa744f855bcc76bc0c59613fe3a`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at fa79f8a, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`, which + the marker confirmed byte for byte. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (all five); 6.1 Yes (all +five); 6.2 No (all five), each mentioning that nothing rejects generic text. Readers 1, 3, 4 and +5 note that the fixture does not say the two undocumented rules are Blocking and grade 4.2 No on +that assumption, which the key's 4.2 row reads as a No. Readers 1, 2, 3, 4 and 5 grade 4.3 Yes +on the suppression route the fixture describes, several saying the fixture is silent on other +routes; the key's 4.3 row says the fixture presents one route, the ignore comment, so under the +Marking section a grade on that condition is the grade for this case. + +Question 1: correct clauses named per reader, distinct within the key's list: reader 1, the +governing principle; 4.1; 4.2; 4.3, 4.4 and 4.5; 5.1 to 5.3 with 6.1 to 6.3; section 8 as +shipped toolchains' self-audit duties. Readers 2 and 3: 4.1; 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 +with 6.1 to 6.3. Reader 4: 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3; 6.1 to 6.3. Reader 5: 4.1 and +4.2; 4.3 with 6.2; 4.4 and 4.5; 5.1 to 5.3 with 6.1 and 6.3. No bullet asserts anything the key +marks wrong. Every reader's first bullet (the toolchain judged as one assembled whole), reader +1's two levels of grading, reader 5's single project-level grade, and the remarks that section 7 +is recommended or a SHOULD (readers 1, 2, 3, 5) and that section 8 applies only to shipped +artefacts (reader 5) are counted neither way, as the key says. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through"; for 2b all five quote section 9's "The project-level verdict is a single grade; ..."; +for 2c all five quote clause 9.1's "The two verdicts MUST be graded and declared separately, and +neither implies the other." No answer gives a grade per part or a single grade for 2c. Reader 2 +adds to 2b that a second grade exists only where the project ships something, and reader 5 that +a failure in any part a clause touches fails that clause for the whole; the key marks neither +wrong. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says there is no agent summary (all +five), readers 3, 4 and 5 adding that the toolchain does not conform at all; S8 says Zigpipe +ships no tooling (all five), readers 2, 3, 4 and 5 placing section 8 on zig-qa's own artefact +grade; Decl says the declaration is optional and the verdict rests on the exercise (all five). + +### Misses + +None. No answer matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..fa79f8a (9e914fe +being the merge base with `next`) adds, rewords, moves or deletes a sentence in the paragraph, +or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------- | ------------------- | +| 1 | Clause 4.1, first paragraph: "The sentences that follow restate the ones most often failed, and a verdict from this document alone rests on them; a grader with that document to hand holds the [Detector] to every one of its MUSTs, restated here or not" (1, 2, 4, 5), and "rests on them" beside "holds the [Detector] to every one of them" (3); which governs without the detector specification | 1, 2, 3, 4, 5 | yes | +| 2 | Section 2, the paragraph "These are additional.", after the terminology table and before the added term entries: what it is additional to | 2, 5 | open; see below | +| 3 | Clause 6.1, first paragraph: "The [Toolchain] MUST define where the [Project record] lives and MUST load it", set against section 2's Project record entry ("[Calibrations], [Exceptions] and conventions"): one location for the whole record or one per kind | 3, 4 | no | +| 4 | Section 1, "How to read this document": "The table in section 2 glosses every term it borrows", where Suppression, Baseline, Agent, Conform, Warning and Fixture are not in the table | 1 | yes | +| 5 | Clause 4.3, first paragraph: "Irreducible cases MUST be directed to the [Project record], where clause 6.2 requires a justification": what "directed" requires of a toolchain | 3 | yes | + +Notes, not passages: readers 1, 3 and 4 remark under question 3 that the detector specification +was not supplied. Readers 1 to 5 tie the same remark to passage 1 under question 4, where it is +part of naming that passage. + +## Findings and where they lie + +- **Finding A, passage 1, clause 4.1 first paragraph** (all five readers): the diff rewrites the + paragraph, adding "That document's MUSTs are those of its sections 4 to 7", "a verdict from + this document alone rests on them" and "restated here or not". Text the change wrote or + altered. Blocks. +- **Finding B, passage 2, section 2 "These are additional."** (readers 2 and 5): the paragraph's + own words are unchanged, but the diff inserts the new paragraph "A [Blocking] [Rule] whose + documentation is missing is not yet a [Defence]. ..." between the terminology table and it, so + that it no longer follows the table it refers to. Whether that is text the change wrote or + altered is an open point (below); marked as the change's own text. Blocks. +- **Finding C, passage 3, clause 6.1 first paragraph** (readers 3 and 4): unchanged, and set + against section 2's Project record entry, which is also unchanged. Elsewhere: recorded in + `acceptance/ROUTED.md` as a finding against the document, routed to a change of its own, and + does not block. The marker writes only this record; the ROUTED row is for the editor to add. + +Passages 4 and 5 are each named by one reader, in paragraphs the diff changed: recorded, not +findings, applied at the editor's discretion. + +## Points the rules did not settle + +1. **Where finding B lies.** The Marking section counts a paragraph as the change's when the + diff adds, rewords, moves or deletes a sentence in it, or changes a word, term or link, or + when a reader sets it against such a paragraph. The diff does none of those to "These are + additional." itself; it inserts a paragraph immediately before it, between it and the table + whose terms it says the following entries are additional to. Reader 2 places it by "It + follows the terminology table", reader 5 by "after the terminology table and before the new + term definitions"; neither names the inserted paragraph. Whether a paragraph displaced from + its referent by an insertion has been moved, or is set against the inserted paragraph by a + reader who is confused by what it refers to, the rules do not say. The open-point rule treats + a finding whose location is open as the change's own text, so it is marked as blocking. + Run 72's ten passages did not include it, which is consistent with the insertion being its + cause. For the editor to settle in `ACCEPTANCE.md`, if it needs settling, before the next run + of this pull request. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 13b497b, named before the readers started. The process files at +fa79f8a (`ACCEPTANCE.md`, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md`, `tools/changes.py`, `spec-qa.py` and the CI workflows) are identical to those +at 13b497b. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: two (findings A + and B). Not met. +- Passages named by two or more readers elsewhere: one (finding C), routed to + `acceptance/ROUTED.md`; does not block. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A and B; every keyed item passed. + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125966495); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125966048); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125966891). diff --git a/acceptance/runs/081-toolchain-contradictions-run-9.md b/acceptance/runs/081-toolchain-contradictions-run-9.md new file mode 100644 index 0000000..e7533de --- /dev/null +++ b/acceptance/runs/081-toolchain-contradictions-run-9.md @@ -0,0 +1,183 @@ +# Run 81: TOOLING-SPEC.md, toolchain contradictions, ninth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch + `editor/toolchain-contradictions`. The readers read commit 0373a27. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 76: clause 4.1's first paragraph and section 1's "How to read this document" + now give one test, a grader checks the restated sentences and takes a verdict already given on + the detector under the detector specification as settled; and the paragraph on a blocking rule + whose documentation is missing, which sat under section 2's table and parted "These are + additional." from it, now ends section 2 after the Project record entry. +- **Run within the pull request**: 9 (runs 57, 58, 62, 65, 67, 68, 72 and 76 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `09c3a3ba5bcd441d513797fcd973e2d0873473ca867aa774ff94643902147e6d`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at 0373a27, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`, which + the marker confirmed byte for byte. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | miss | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (all five); 6.1 Partial +(readers 1, 2 and 5, each mentioning that the fixture names a place for exceptions and none for +calibrations or conventions) and Yes (readers 3 and 4); 6.2 No (all five), each mentioning that +nothing rejects generic text. Readers 1, 2, 3, 4 and 5 grade 4.2 No reading the two undocumented +rules as Blocking, which the key's 4.2 row reads as a No. All five grade 4.3 Yes on the one route +the fixture describes, which the key's 4.3 row says is the case the fixture presents. Readers 3, +4 and 5 give their 6.1 grade with a remark on what it would be in another case; see the points +below. + +Question 1: correct clauses named per reader, distinct within the key's list: reader 1, 4.1; +4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 with 6.1 to 6.3. Reader 2: 4.1; 4.2; 4.3; 5.1 to 5.3 with +6.1 and 6.2; 4.5. Reader 3: 4.1 with 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3; 6.1 to 6.3. Reader 4: +4.1; 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 with 6.1 to 6.3. Reader 5: 4.1; 4.2 with 4.3; 4.4 and +4.5; 5.1 to 5.3 with 6.1 and 6.3; 6.2 with 8.1 and 8.2. No bullet asserts anything the key marks +wrong. Every reader's first bullet (the toolchain judged as one assembled whole) and reader 2's +remark that section 7 is SHOULDs and the declaration optional are counted neither way, as the +key says. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through"; for 2b all five quote section 9's "The project-level verdict is a single grade; ..."; +for 2c all five quote clause 9.1's "The two verdicts MUST be graded and declared separately, and +neither implies the other." No answer gives a grade per part or a single grade for 2c. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says there is no agent summary (all +five), readers 3, 4 and 5 adding that the toolchain does not conform at all; S8 says Zigpipe +ships no tooling (all five), reader 1 adding that zig-qa's own artefact grade is not described +and giving no verdict on it; Decl says the declaration is optional and the verdict rests on the +exercise (all five). + +### Misses + +- **Reader 2, 6.3, Partial.** The key accepts Yes only. Reasoning: `zig-qa list --exceptions` + lists exceptions in the rule listing's format, but calibrations and conventions are not + described as enumerable, "so the same gap as 6.1 applies". The reasoning quotes nothing of the + document; it carries over the reader's 6.1 reading of section 2's Project record entry. The key + names no wrong reading for 6.3, so this is the reader's single miss on the item, tolerated by + the threshold. + +No answer matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..0373a27 (9e914fe +being the merge base with `next`) adds, rewords, moves or deletes a sentence in the paragraph, +changes a word, term or link in it, or parts it from what it refers to. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------ | +| 1 | Clause 5.1, first paragraph: "every [Defence] active in the project" with "The listing covers every [Blocking] [Rule] the [Toolchain] runs, documented or not"; a listing of Defences that covers what is not a Defence | 1, 5 | yes: the second sentence quoted is added by the diff | +| 2 | Clause 6.1, first paragraph: "The [Toolchain] MUST define where the [Project record] lives and MUST load it"; one location for the whole record or one per kind | 1, 2, 3, 4 | no | +| 3 | Section 2, the Project record entry: "[Calibrations], [Exceptions] and conventions" (1, 3), the whole entry quoted (2, 4), set against passage 2 | 1, 2, 3, 4 | no: its words are unchanged; the diff inserts a paragraph after it, but it refers to nothing that follows it | +| 4 | Clauses 6.2 and 6.3, described as speaking only of exceptions, set against passage 3 | 1, 2 | no; whether this names a passage is open, see below | +| 5 | Section 2, the Defence row: "A [Rule] with its documentation, in force and [Blocking]", set against passage 1 | 1 | no: the diff re-pads the table for an added row; the row's words are unchanged | +| 6 | Section 2, the paragraph "A [Blocking] [Rule] whose documentation is missing is not yet a [Defence]", set against passage 1 | 5 | yes: added by the diff | +| 7 | Clause 5.3, first paragraph: "Every [Defence] the project has written itself MUST appear in the listing clause 5.1 requires, meeting its content requirements in full"; what the content requirements are | 2 | no | +| 8 | Clause 4.1, first paragraph: "takes a verdict already given on the [Detector] under that document as settled"; whether to grade the detector itself or only the restated sentences, with no verdict given | 3 | yes: rewritten by the diff | +| 9 | Clause 9.2, first paragraph: "A project or a shipped artefact MAY declare" followed by "A project that ships an artefact carries both levels of clause 9.1 in that declaration"; an obligation inside an optional provision | 3 | no | + +Passage 8 is a passage and not a note: reader 3 says it was given no verdict on Lintomatic, but +its confusion is what the sentence asks a grader to do, not that another document was missing. +Reader 3 cites the next paragraph of 9.2 as confirming the declaration optional, which does not +set passage 9 against it. + +Notes, not passages: under question 3, readers 1 and 5 remark that the detector specification +was not supplied or not seen in full, and readers 2 and 4 that the fixture gives no verdict on +Lintomatic under it. + +## Findings and where they lie + +- **Finding A, passage 1, clause 5.1 first paragraph** (readers 1 and 5): the diff adds the + sentence "The listing covers every [Blocking] [Rule] the [Toolchain] runs, documented or not, + and a [Rule] whose route to documentation is missing is listed with that gap showing." Reader 5 + also sets the paragraph against section 2's added paragraph (passage 6). Text the change wrote + or altered. Blocks. +- **Finding B, passages 2 and 3, clause 6.1 first paragraph against section 2's Project record + entry** (readers 1, 2, 3 and 4): both paragraphs unchanged, and no reader naming them sets + either against a changed paragraph. Elsewhere: a finding against the document, recorded in + `acceptance/ROUTED.md` (its 6.1 row, raised by run 76, gains run 81), routed to a change of its + own, and does not block. The marker writes only this record; the ROUTED entry is for the + editor. +- **Finding C, passage 4, clauses 6.2 and 6.3** (readers 1 and 2), counted as named under the + open-point rule (below): unchanged, and set against the unchanged Project record entry. + Elsewhere: routed to `acceptance/ROUTED.md` with finding B, whose passage it belongs to; does + not block. + +Passages 5 to 9 are each named by one reader: recorded, not findings, applied at the editor's +discretion. + +## Points the rules did not settle + +1. **6.1 grades given with a remark on another case.** Reader 3 gives "Yes, with a caveat", that + it would be Partial if calibrations or conventions are recorded elsewhere and never read; + reader 4 gives Yes and says it would be Partial if those belong to the project record; reader + 5 gives Partial and says it becomes Yes if the project has neither. The Marking section reads + a grade given on a stated condition as the grade for the case the fixture presents only where + the key says which case that is, and the key's 6.1 row does not say it in those terms. + Judgement: each reader states one grade outright and the remark names what it would be in a + different case, so the stated grade is the grade; each is one the key accepts, and readers 1, + 2 and 5's Partial meets the key's condition. Had the remark made the grade conditional, the + open-point rule marks it for the reader. Marked for the readers. +2. **Whether passage 4 is named.** Readers 1 and 2 say clauses 6.1 to 6.3, or 6.2 and 6.3, + "mention only exceptions" or "speak only of exceptions", which describes the clauses and + identifies no one sentence or paragraph. The open-point rule counts the passage; it lies in + unchanged text and does not affect the verdict. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 389ef3f, named before the readers started. The process files at +0373a27 (`ACCEPTANCE.md`, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md`, `tools/changes.py`, `spec-qa.py` and the CI workflows) are identical to those +at 389ef3f. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item but 6.3, + where four of five. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: one (finding A). + Not met. +- Passages named by two or more readers elsewhere: findings B and C, routed to + `acceptance/ROUTED.md`; do not block. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on finding A; every keyed item passed. + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125967702); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125967247); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125968173). diff --git a/acceptance/runs/082-toolchain-contradictions-run-10.md b/acceptance/runs/082-toolchain-contradictions-run-10.md new file mode 100644 index 0000000..690c7a3 --- /dev/null +++ b/acceptance/runs/082-toolchain-contradictions-run-10.md @@ -0,0 +1,178 @@ +# Run 82: TOOLING-SPEC.md, toolchain contradictions, tenth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch + `editor/toolchain-contradictions`. The readers read commit de34774. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 81: clause 5.1 no longer says the listing covers every blocking rule, + documented or not; it says a blocking rule whose route to documentation is missing is not yet + a defence, that clause 4.2 fails it, and that the listing can show it with the gap showing; and + section 2's closing paragraph names clauses 4.2 and 8.1 as the clauses that catch such a rule. +- **Run within the pull request**: 10 (runs 57, 58, 62, 65, 67, 68, 72, 76 and 81 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `1fa098d5a7901d8211c09387ca7bffd0bcabcd05caba5565a489557ae5761028`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at de34774, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`, which + the marker confirmed byte for byte. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (all five); 6.1 Partial +(readers 2, 3 and 5, each mentioning that the fixture names a place for exceptions and none for +calibrations or conventions) and Yes (readers 1 and 4); 6.2 No (all five), each mentioning that +nothing rejects a generic justification or generic text. All five grade 4.2 No reading the two +undocumented rules as Blocking because `zig-qa run` stops when Lintomatic fails, which the key's +4.2 row reads as a No. All five grade 4.3 Yes on the one route the fixture describes, which the +key's 4.3 row says is the case the fixture presents. Readers 1, 2 and 3 cite the 5.1 sentence +that the listing "can show it with that gap showing" for their 5.1 Yes; reader 5 says the two +`(none)` entries are the gap clause 4.2 reports. + +Question 1: correct clauses named per reader within the first six bullets. Reader 1: 4.1 with +4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 with 6.3; 6.1 and 6.2. Reader 2: 4.1; 4.2; 4.3 with 4.5; +5.1 to 5.3 with 6.2 and 6.3. Reader 3: 4.1 with 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3; 6.1 to 6.3. +Reader 4 (seven bullets, the first six marked): 4.1; 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3. Reader +5: 4.1; 4.2; 4.3; 5.1 to 5.3 with 6.2 and 6.3; 4.5. No bullet asserts anything the key marks +wrong. Every reader's first bullet (the toolchain judged as one assembled whole), the remarks of +readers 1, 2, 3 and 5 that a shipped artefact has a second grade, and the remarks of readers 2 +and 3 that section 7 is SHOULDs, are counted neither way, as the key says. Reader 2's sixth +bullet also calls clause 6.4's defaults recommended; a bullet on a SHOULD asserts no obligation +and is not counted. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through"; for 2b all five quote section 9's "The project-level verdict is a single grade; ..."; +for 2c all five quote clause 9.1's "The two verdicts MUST be graded and declared separately, and +neither implies the other." No answer gives a grade per part or a single grade for 2c. Reader 1's +2b adds that Zigpipe ships nothing and so has only the one grade, quoting section 9; it asserts +nothing the key marks wrong. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says there is no agent summary (all +five), each adding that the toolchain does not conform at all; S8 says Zigpipe ships no tooling +(all five), readers 1, 2, 4 and 5 adding that zig-qa would carry its own artefact grade, which +none grades; Decl says the declaration is optional and the verdict rests on the exercise (all +five). + +### Misses + +None. No answer matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..de34774 (9e914fe +being the merge base with `next`) adds, rewords, moves or deletes a sentence in the paragraph, +changes a word, term or link in it, or parts it from what it refers to. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | ------------------------------------------------------------------------------------------------ | +| 1 | Clause 5.1, first paragraph: "It MUST provide the route to each one's full documentation" with "the listing can show it with that gap showing", whether an empty route meets the MUST (1); "MUST list every [Defence] active in the project" with "A [Blocking] [Rule] whose route to documentation is missing is not yet a [Defence] ... can show it", whether the undocumented rules must appear (3) | 1, 3 | yes: the second sentence each quotes is added by the diff | +| 2 | Clause 4.1, first paragraph: "takes a verdict already given on the [Detector] under that document as settled" (1, 2), with "a [Detector] that document finds not to [Conform] fails this clause, whether or not the sentences that follow show why" (2); which verdict is meant, and what a grader does when none has been given | 1, 2 | yes: rewritten by the diff | +| 3 | Clause 6.2, last paragraph: "by a check it documents: 'needed for now', 'legacy', 'TODO' and their like" against "could be pasted onto any [Exception] unchanged", which test governs (1); "name the [Hazard] ... and the scope" and "cannot verify truth", with the heading's "MUST reject a generic one", whether the check must test that hazard and scope are named (4) | 1, 4 | no: neither the paragraph nor the 6.2 heading is touched by the diff | +| 4 | Clause 4.3, second paragraph: "This clause is where the project's decision is made and enforced, and the decision is no"; what the "no" answers | 2 | no: the diff adds sentences to the first paragraph only and inserts no paragraph between them | +| 5 | Clause 4.4, first paragraph: "The [Toolchain]'s own entry point MUST meet clauses 5.1 to 5.4 of the [detector specification] for every [Defence] it routes"; the restatement covers three properties, so whether 5.4 is graded | 2 | no | +| 6 | Clause 6.1, first paragraph: "The [Toolchain] MUST define where the [Project record] lives and MUST load it", set against section 2's Project record entry; one location for all three kinds or not | 2 | no | +| 7 | Section 2: "These are additional."; what the terms are additional to | 2 | no: the diff re-pads the table above it and adds a row; the sentence and its place are unchanged | +| 8 | Section 2, the paragraph "A [Blocking] [Rule] whose documentation is missing is not yet a [Defence]"; it sits under the Project record entry, so which term it governs | 4 | yes: added by the diff | +| 9 | Clause 5.3, second paragraph: "Where such a [Defence] is not expressible in the [Toolchain]'s own [Detectors], its entry MAY be sourced from wherever it is enabled ..."; whether that reaches configuration outside the toolchain's own files | 5 | no | + +Notes, not passages: reader 4's remark on clause 9.1, "the project's remediations against +section 7 of the method specification", says it could not work out what that section requires +because the method specification was not in its materials; that is a remark that another document +was not supplied, and the Marking section records it as a note. Under question 3, readers 1, 2 +and 3 remark that no verdict on Lintomatic under the detector specification was given, and +reader 2 that it did not have the detector specification, when grading 4.1 and 4.4. + +## Findings and where they lie + +- **Finding A, passage 1, clause 5.1 first paragraph** (readers 1 and 3): the diff adds the + sentence "A [Blocking] [Rule] whose route to documentation is missing is not yet a \[Defence\]: + clause 4.2 fails it, and the listing can show it with that gap showing." Reader 1 sets it + against the paragraph's MUST on the route to documentation, reader 3 against its MUST to list + every Defence. Text the change wrote or altered. Blocks. +- **Finding B, passage 2, clause 4.1 first paragraph** (readers 1 and 2): the diff rewrites the + paragraph, adding the sentence each quotes, "A grader of this clause checks the sentences that + follow ... and takes a verdict already given on the [Detector] under that document as settled: + ...". Text the change wrote or altered. Blocks. Whether it is a passage rather than a note is + recorded below. +- **Finding C, passage 3, clause 6.2 last paragraph** (readers 1 and 4): unchanged, and neither + reader sets it against a changed paragraph; reader 4 sets it against the 6.2 heading, also + unchanged. Elsewhere: a finding against the document, recorded in `acceptance/ROUTED.md`, where + the open 6.2 row raised by runs 67 and 68 (whether the check must test that the hazard and the + scope are named) covers it and gains run 82; routed to a change of its own, and does not block. + The marker writes only this record; the ROUTED entry is for the editor. + +Passages 4 to 9 are each named by one reader: recorded, not findings, applied at the editor's +discretion. + +## Points the rules did not settle + +1. **Whether passage 2 is a passage or a note.** Readers 1 and 2 tie their confusion to the + fixture giving no verdict on Lintomatic under the detector specification, and the Marking + section makes a remark that another document was not supplied a note. Judgement: each reader's + confusion is what the sentence asks a grader to do, which verdict it means (1) and what to do + when none has been given (2), and that is about the passage, not about a missing document; so + it is a passage. Had it stayed open, the open-point rule counts the passage. Marked as named, + which makes finding B. +2. **6.1 grades given with a remark on another case.** Readers 2, 3 and 5 grade Partial and say + it would be Yes if the exceptions file is the whole project record (3 implicitly, 2 and 5 in + terms). The Marking section reads a grade given on a stated condition as the grade for the + case the fixture presents only where the key says which case that is, and the key's 6.1 row + does not. Judgement: each reader states Partial outright and the remark names a different + case, so Partial is the grade; it meets the key's condition. Had the remark made the grade + conditional, the open-point rule marks it for the reader. Marked for the readers. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 0da928d, named before the readers started. The process files at +de34774 (`ACCEPTANCE.md`, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md`, `tools/changes.py`, `spec-qa.py` and the CI workflows) are identical to those +at 0da928d. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: two (findings A + and B). Not met. +- Passages named by two or more readers elsewhere: finding C, routed to `acceptance/ROUTED.md`; + does not block. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A and B; every keyed item passed. + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125969059); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125968623); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125969436). diff --git a/acceptance/runs/083-toolchain-contradictions-run-11.md b/acceptance/runs/083-toolchain-contradictions-run-11.md new file mode 100644 index 0000000..6b21bac --- /dev/null +++ b/acceptance/runs/083-toolchain-contradictions-run-11.md @@ -0,0 +1,175 @@ +# Run 83: TOOLING-SPEC.md, toolchain contradictions, eleventh run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch + `editor/toolchain-contradictions`. The readers read commit c770722. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 82: clause 5.1 says nothing of undocumented rules any more; the paragraph on + a blocking rule whose documentation is missing now ends section 2, after the paragraph on the + distinction between a bundled defence and a project's own, and opens "Throughout this + document"; clause 4.1 and section 1 name the verdict a grader takes as one the detector already + holds under the detector specification's section 8, and clause 4.1 says that without one the + restated sentences are the test. +- **Run within the pull request**: 11 (runs 57, 58, 62, 65, 67, 68, 72, 76, 81 and 82 came + before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `1f981586432ce938dba63935e5be59cd7e1b36b1569fedc019b576877b34c06d`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at c770722, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`, which + the marker confirmed byte for byte. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (readers 1, 3 and 4) and +Partial (readers 2 and 5); 6.1 Partial (readers 1, 3, 4 and 5, each mentioning that the fixture +names a place for exceptions and none for calibrations or conventions) and Yes (reader 2); 6.2 +No (all five), each mentioning that nothing rejects a generic justification or generic text. +All five grade 4.1 No on the table's "Nothing for those two" row, and 4.2 No reading the two +undocumented rules as Blocking because `zig-qa run` stops when Lintomatic fails, which the key's +4.2 row reads as a No. Readers 3 and 4 give 5.1 Yes because section 2 says the two undocumented +rules are not yet Defences; readers 2 and 5 give Partial on the `(none)` routes. The key accepts +all three grades with no condition. + +Question 1: correct clauses named per reader within the first six bullets. Reader 1: 4.1 with +4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 with 6.3; 6.1 and 6.2. Reader 2: 4.1; 4.2 with 4.3; 4.4 and +4.5; 5.1 to 5.3 with 6.3; 6.1 and 6.2. Reader 3: 4.1 with 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 +with 6.1 to 6.3; 8.1 and 8.2. Reader 4: 4.1; 4.2; 4.3; 4.5 with 5.1 to 5.3; 6.1 and 6.2. +Reader 5: 4.1 with 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 with 6.3; 6.1 and 6.2. No bullet asserts +anything the key marks wrong. Every reader's first bullet (the toolchain judged as one assembled +whole), reader 3's remark that agent-facing summaries are SHOULDs, and reader 5's remark that a +shipped artefact gets a separate grade with section 8 applying only to it are counted neither +way, as the key says. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through"; for 2b all five quote section 9's "The project-level verdict is a single grade; ..."; +for 2c all five quote clause 9.1's "The two verdicts MUST be graded and declared separately, and +neither implies the other." No answer gives a grade per part or a single grade for 2c. Reader 2's +2b adds that a second grade arises only for a shipped artefact and Zigpipe ships none; it asserts +nothing the key marks wrong. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says the toolchain does not conform at +all and there is no agent summary (all five); S8 says Zigpipe ships no tooling to other projects +(all five); Decl says the declaration is optional and the verdict rests on the exercise or the +evidence (all five). + +### Misses + +- **Reader 5, clause 4.3: Partial** (key accepts Yes). Reasoning: the inline route is closed by + `ignore_comments = false` and exceptions go to `quality-exceptions.toml`, but the fixture does + not say whether Lintomatic has any other route, such as a silent baseline, so the reader cannot + confirm the clause covers every route. The grade is stated outright, not on a condition. The + reasoning quotes neither the document nor any sentence of clause 4.3; it is a reading of the + fixture's silence. The key names no wrong reading for 4.3, so this is a single miss on the item, + tolerated by the threshold (four of five correct). + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..c770722 (9e914fe +being the merge base with `next`) adds, rewords, moves or deletes a sentence in the paragraph, +changes a word, term or link in it, or parts it from what it refers to. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------- | +| 1 | Section 2, closing paragraph: "Throughout this document, a [Blocking] [Rule] whose documentation is missing is not yet a [Defence]. ..." (5 quotes it whole; 1 and 4 quote "a [Blocking] [Rule] whose documentation is missing is not yet a [Defence]"); each sets it against clause 5.1, whether undocumented blocking rules are listed | 1, 4, 5 | yes: added by the diff | +| 2 | Clause 5.1, first paragraph: "The [Toolchain] MUST list every [Defence] active in the project" (4, 5), and "every [Defence]" with clause number 5.1 and its listing requirement (1); set against passage 1 | 1, 4, 5 | no in itself; each reader sets it against passage 1, which the diff added | +| 3 | Clause 6.2, first paragraph: "It MUST NOT supply a default, and MUST reject an [Exception] that omits one"; set against clause 6.4's defaults, whether a justification is a delegated judgement a default may fill | 2, 3 | no; nor is either 6.4 paragraph the readers set it against | +| 4 | Clause 4.1, first paragraph: "Where it has not, the sentences that follow are the test" against "this clause holds the [Detector] to every one of them"; whether the restated sentences are the full test or a sample | 1 | yes: rewritten by the diff | +| 5 | Clause 6.2, last paragraph: "could be pasted onto any [Exception] unchanged, by a check it documents: 'needed for now', 'legacy', 'TODO' and their like"; which test the check applies | 1 | no | +| 6 | Clause 6.4, first paragraph: "Where the method specification delegates a judgement and the project has recorded nothing, a documented [Toolchain] default is what the [Practitioner] falls back to" | 2 | no | +| 7 | Clause 6.4 heading: "The [Toolchain] SHOULD state its own defaults for anything the method leaves to the project" | 3 | no | +| 8 | Clause 9.1, first paragraph: "The two verdicts MUST be graded and declared separately", set against 9.2's optional declaration | 3 | yes: the diff rewords sentences in the paragraph | +| 9 | Clause 9.2, third paragraph: "The declaration is optional", set against passage 8 | 3 | yes: the diff extends the sentence quoted | +| 10 | Section 2, the Consuming project entry: "A codebase that installs a [Detector] or a [Toolchain] shipped by somebody else", with the Bundled defence entry named by its term; whether the `zq.*` rules are bundled defences | 4 | no | +| 11 | Clause 6.1 heading: "MUST define a location for the project record", set against section 2's Project record entry; one location for all three kinds or not | 4 | no | + +Passages 6 and 7 are different paragraphs: the Marking section makes a heading a paragraph of its +own, so readers 2 and 3 do not name the same 6.4 passage. Passages 3 and 5 are different +paragraphs of 6.2. + +Notes, not passages: under question 3, readers 1, 3 and 5 remark that the detector specification +was not supplied and that they graded 4.1 (and, for reader 5, 4.4) on the restated sentences. + +## Findings and where they lie + +- **Finding A, passage 1, section 2 closing paragraph** (readers 1, 4 and 5): the diff adds the + paragraph. Text the change wrote or altered. Blocks. +- **Finding B, passage 2, clause 5.1 first paragraph** (readers 1, 4 and 5): the paragraph is + unchanged by the diff, but every reader naming it sets it against passage 1, which the diff + added: whether a Blocking Rule that section 2 says is "not yet a [Defence]" falls within the + duty to list "every [Defence] active in the project". A paragraph a reader naming the finding + sets against a changed paragraph is the change's own text under the Marking section. Blocks. +- **Finding C, passage 3, clause 6.2 first paragraph** (readers 2 and 3): unchanged, and both + readers set it against clause 6.4 (reader 2 its first paragraph, reader 3 its heading), also + unchanged. Elsewhere: a finding against the document, to be recorded in `acceptance/ROUTED.md`, + where no open row covers the default of 6.2 against the defaults of 6.4; routed to a change of + its own, and does not block. The marker writes only this record; the ROUTED entry is for the + editor. + +Passages 4 to 11 are each named by one reader: recorded, not findings, applied at the editor's +discretion. + +## Points the rules did not settle + +1. **Whether reader 4 names the Bundled defence entry.** Reader 4 quotes the Consuming project + entry and names "Bundled defence" only by its term in its item heading. Judgement: the term is + a description that identifies the entry, so it is named; had it stayed open, the open-point + rule counts the passage. No other reader names that entry, so it makes no finding either way. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit ae6faf2, named before the readers started. The process files at +c770722 (`ACCEPTANCE.md`, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md`, `tools/changes.py`, `spec-qa.py` and the CI workflows) are identical to those +at ae6faf2. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item but 4.3, + where four of five are correct. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: two (findings A + and B). Not met. +- Passages named by two or more readers elsewhere: finding C, routed to `acceptance/ROUTED.md`; + does not block. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A and B; every keyed item passed its threshold. + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125970205); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125969785); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125970613). diff --git a/acceptance/runs/084-toolchain-contradictions-run-12.md b/acceptance/runs/084-toolchain-contradictions-run-12.md new file mode 100644 index 0000000..1342b61 --- /dev/null +++ b/acceptance/runs/084-toolchain-contradictions-run-12.md @@ -0,0 +1,170 @@ +# Run 84: TOOLING-SPEC.md, toolchain contradictions, twelfth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch + `editor/toolchain-contradictions`. The readers read commit c2e3249. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 83 in section 2's closing paragraph alone: it now says that nothing this + document asks of defences, clause 5.1's listing included, is asked of a blocking rule whose + documentation is missing, and that a toolchain running one has failed clause 4.2 whether or + not its listing shows it. +- **Run within the pull request**: 12 (runs 57, 58, 62, 65, 67, 68, 72, 76, 81, 82 and 83 came + before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `eb3d2cc18c08b54960dd96080fcac5066e04ef556aaa983194c13e79bef930dc`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at c2e3249, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`, which + the marker confirmed byte for byte. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (all five), each saying the +two `(none)` rules are not Defences under section 2 and so outside 5.1 (readers 3, 4 and 5 add +that clause 4.2 catches them); 6.1 Yes (readers 1, 2, 3 and 5) and Partial (reader 4, mentioning +that section 2's Project record also holds calibrations and conventions and the fixture does not +say where those are, which meets the key's condition); 6.2 No (all five), each mentioning that +nothing rejects a generic justification or generic text. All five grade 4.1 No on the two +undocumented bundled rules with nothing supplied for them, and 4.2 No; readers 2, 3, 4 and 5 +read the two rules as Blocking because `zig-qa run` stops when Lintomatic fails, which the key's +4.2 row reads as a No, and reader 1 grades No outright. + +Question 1: correct clauses named per reader within the first six bullets. Reader 1: 4.1 with +4.2; 4.3 with 4.5; 4.4; 5.1 to 5.3; 6.1 to 6.3. Reader 2: 4.1; 4.2; 4.3 with 6.2; 4.4 and 4.5; +5.1 to 5.3 with 6.3. Reader 3: 4.1 with 4.2; 4.3; 4.5; 5.1 to 5.3; 6.1 to 6.3. Reader 4: 4.1 +with 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 with 6.3; 6.2. Reader 5: 4.1 with 4.2; 4.3 with 4.5; 4.4; +5.1 to 5.3 with 6.1 and 6.2; 6.3. Five correct bullets each. No bullet asserts anything the key +marks wrong. Every reader's first bullet (the toolchain judged as one assembled whole), reader +2's and reader 4's remarks that the agent summary is a SHOULD, reader 4's that a declaration is +optional, and readers 4's and 5's that a shipped artefact gets its own grade are counted neither +way, as the key says. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through"; for 2b all five quote section 9's "The project-level verdict is a single grade; ..."; +for 2c all five quote clause 9.1's "The two verdicts MUST be graded and declared separately, and +neither implies the other." No answer gives a grade per part or a single grade for 2c. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says the toolchain does not conform at +all, or that there is no summary, or both (all five); S8 says Zigpipe ships no tooling to other +projects (all five); Decl says the declaration is optional, or that the verdict rests on the +exercise of the clauses (all five). + +### Misses + +None. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..c2e3249 (9e914fe +being the merge base with `next`) adds, rewords, moves or deletes a sentence in the paragraph, +changes a word, term or link in it, or parts it from what it refers to. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------- | +| 1 | Clause 4.1, first paragraph: "Where it has not, the sentences that follow are the test" (2, 3), "A grader of this clause checks the sentences that follow" (3, 5), "this clause holds the [Detector] to every one of them" (3; 5 quotes "every one of them"); whether the restated sentences or every MUST of the detector specification is the test | 2, 3, 5 | yes: rewritten by the diff | +| 2 | Clause 4.1, point 2: "Does the [Detector] meet every MUST of sections 4 to 7 of the [detector specification](DETECTOR-SPEC.md), exercised as its clause 8.1 describes?" (2), "every MUST of sections 4 to 7 … exercised as its clause 8.1 describes" (5); set against passage 1 | 2, 5 | yes: the diff rewords sentences in the item | +| 3 | Section 2, closing paragraph: "not yet a [Defence]" and "clause 5.1's listing included", with the paragraph identified by its opening words, set against clause 5.1 (3); "where they reach a [Toolchain]'s [Rules] they reach every [Blocking] [Rule] it runs, documented or not", which clauses "they" refers to (5) | 3, 5 | yes: added by the diff, and the revision after run 83 | +| 4 | Section 2, the Project record entry: described as listing the project-record judgements, exceptions among them, set against clauses 6.2 and 6.4 (1); "[Calibrations], [Exceptions] and conventions", set against clause 6.1 (4) | 1, 4 | no | +| 5 | Clause 6.2, first paragraph: "MUST NOT supply a default", set against clause 6.4 | 1 | no | +| 6 | Clause 6.4 heading: "SHOULD state its own defaults for anything the method leaves to the project" | 1 | no | +| 7 | Clause 5.1, first paragraph: "MUST list every [Defence] active", set against passage 3 | 3 | no in itself; set against passage 3, which the diff added | +| 8 | Clause 4.3, first paragraph: "A route bypasses the [Project record] when a finding can be silenced through it without an entry appearing in the [Project record]"; whether disabling a whole rule in configuration is a suppression route | 4 | yes: the diff adds a sentence to the paragraph | +| 9 | Section 2, the Exception row of the table: "a recorded, justified decision to leave an [Instance] unfixed or a [Rule] unapplied", set against passage 8 | 4 | no | +| 10 | Clause 6.1, first paragraph: "The [Toolchain] MUST define where the [Project record] lives and MUST load it"; one location for all three kinds or not | 4 | no | +| 11 | Clause 9.1, first paragraph: "MUST be graded and declared separately", set against passage 12 | 5 | yes: the diff rewords sentences in the paragraph | +| 12 | Clause 9.2, third paragraph: "The declaration is optional", set against passage 11 | 5 | yes: the diff extends the sentence quoted | + +Notes, not passages: reader 4's third item, "exercised as its clause 8.1 describes" with clause +4.1, is a remark that the detector specification was not supplied, and so a note however much +of the document it quotes; under question 3, readers 1, 2, 3 and 4 remark the same of 4.1 or +4.4. Passages 5 and 6 are different paragraphs (a heading is a paragraph of its own). + +## Findings and where they lie + +- **Finding A, passage 1, clause 4.1 first paragraph** (readers 2, 3 and 5): the diff rewrites + the paragraph. Text the change wrote or altered. Blocks. +- **Finding B, passage 2, clause 4.1 point 2** (readers 2 and 5): the diff rewords the item's + sentences, and both readers set it against passage 1. Text the change wrote or altered. Blocks. +- **Finding C, passage 3, section 2 closing paragraph** (readers 3 and 5): the diff adds the + paragraph, and its revision after run 83 is the only change since that run. Text the change + wrote or altered. Blocks. +- **Finding D, passage 4, section 2's Project record entry** (readers 1 and 4): unchanged by the + diff, not parted from anything it refers to, and each reader sets it only against unchanged + paragraphs (reader 1 against 6.2 and 6.4, reader 4 against 6.1). Elsewhere: a finding against + the document, to be recorded in `acceptance/ROUTED.md`, where the open rows from runs 76 and 81 + on clause 6.1 and on 6.2 and 6.3 against this entry are the nearest; routed, and does not + block. The marker writes only this record; the ROUTED entry is for the editor. + +Passages 5 to 12 are each named by one reader: recorded, not findings, applied at the editor's +discretion. + +## Points the rules did not settle + +1. **Whether a paragraph cited under question 4 as context is named.** Readers 1 and 4 name the + Project record entry as what they set another clause against, not as itself confusing. + Judgement: the Marking section defines naming by quotation or by a clause number with an + identifying description, not by what the reader calls confusing, so it is named; had it + stayed open, the open-point rule counts the passage. Either way finding D lies elsewhere and + does not block. +2. **Whether reader 4 names clause 6.2 or 6.3** by "Clauses 6.2 and 6.3 speak only of + exceptions". Judgement: not named, because the description identifies a clause and no + sentence or paragraph within it, which the Marking section requires. Had it been counted, it + would at most join passage 5 in a finding on unchanged text set against unchanged text, which + is routed and does not block; the verdict is the same either way. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit a095fa7, named before the readers started. The process files at +c2e3249 (`ACCEPTANCE.md`, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md`, `tools/changes.py`, `spec-qa.py` and the CI workflows) are identical to those +at a095fa7. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: three (findings A, + B and C). Not met. +- Passages named by two or more readers elsewhere: finding D, routed to `acceptance/ROUTED.md`; + does not block. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A, B and C; every keyed item passed its threshold. + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125971464); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125971023); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125972035). diff --git a/acceptance/runs/085-toolchain-contradictions-run-13.md b/acceptance/runs/085-toolchain-contradictions-run-13.md new file mode 100644 index 0000000..0ab5d1f --- /dev/null +++ b/acceptance/runs/085-toolchain-contradictions-run-13.md @@ -0,0 +1,168 @@ +# Run 85: TOOLING-SPEC.md, toolchain contradictions, thirteenth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch + `editor/toolchain-contradictions`. The readers read commit 58924a8. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 84: clause 4.1's first paragraph now has one test, every MUST of the detector + specification's sections 4 to 7, with the restated sentences as where a grader looks first and + not the whole test, a grader without that document checking only them and saying so; section + 1's "How to read this document" says the same; section 2's closing paragraph says in plain + sentences that clause 5.1 does not require an undocumented blocking rule to be listed and that + clauses 4.2 and 8.1 reach it. +- **Run within the pull request**: 13 (runs 57, 58, 62, 65, 67, 68, 72, 76, 81, 82, 83 and 84 + came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `6d37cc625287ae0258a845657c48a96b7a53c698d998d6cde43c5c9e064a53ea`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at 58924a8, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`, which + the marker confirmed byte for byte. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (all five; readers 4 and 5 +add that the two `(none)` rules are not Defences under section 2 and so outside 5.1); 6.1 +Partial (readers 1 and 5, each mentioning that the Project record also holds calibrations and +conventions and the fixture gives no location for those, which meets the key's condition) and +Yes (readers 2, 3 and 4); 6.2 No (all five), each mentioning that nothing rejects a generic +justification or generic phrases. All five grade 4.1 No on the two undocumented bundled rules +with nothing supplied for them, each saying the detector specification was not supplied and +only the restated points were checked, which the key's 4.1 row accepts. All five grade 4.2 No; +readers 1, 2, 3 and 5 state that they read the two rules as Blocking because `zig-qa run` stops +when Lintomatic fails, which the key's 4.2 row reads as a No, and reader 4 states they are +Blocking for the same reason. + +Question 1: correct clauses named per reader within the first six bullets. Reader 1: 4.1 with +4.2; 4.3 with 4.4 and 4.5; 5.1 to 5.3; 6.1 to 6.3. Reader 2: 4.1; 4.2; 4.3 with 6.1 and 6.2; +4.4 and 4.5; 5.1, 5.2 and 6.3. Reader 3: 4.1 with 4.2; 4.3 with 4.4 and 4.5; 5.1 to 5.3; 6.1 to +6.3. Reader 4: the governing principle with 4.1 and 4.2; 4.3 with 4.5; 5.1 to 5.3 with 6.3; 6.1 +and 6.2. Reader 5: 4.1; 4.2; 4.3 with 6.2; 5.1, 5.2, 6.1 and 6.3; 4.4 and 4.5. Four or more +correct bullets each. No bullet asserts anything the key marks wrong. Every reader's first +bullet (the toolchain judged as one assembled whole), the remarks that section 7 or clause 6.4 +is a SHOULD or recommended (readers 1, 2, 3 and 5), that a declaration is optional or not a +condition (readers 1, 3 and 4), and that a shipped artefact gets its own grade (readers 1, 3, 4 +and 5) are counted neither way, as the key says. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through"; for 2b all five quote section 9's "The project-level verdict is a single grade; ..."; +for 2c all five quote clause 9.1's "The two verdicts MUST be graded and declared separately, and +neither implies the other." No answer gives a grade per part, or a single grade for 2c. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says the toolchain does not conform at +all, or that there is no summary, or both (all five); S8 says Zigpipe ships no tooling to other +projects (all five); Decl says the declaration is optional, or that the verdict rests on the +exercise of the clauses (all five). + +### Misses + +None. No answer matches a wrong reading the key names. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..58924a8 (9e914fe +being the merge base with `next`) adds, rewords, moves or deletes a sentence in the paragraph, +changes a word, term or link in it, or parts it from what it refers to. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------- | +| 1 | Section 2, closing paragraph: "Clauses 4.2 and 8.1 reach every [Blocking] [Rule] a [Toolchain] runs, documented or not" (3, 5, each set against passage 2); "a [Blocking] [Rule] whose documentation is missing is not yet a [Defence]. Clause 5.1 does not require it to be listed, because that clause lists [Defences]" (4, set against passage 3) | 3, 4, 5 | yes: added by the diff, and reworded in the revision after run 84 | +| 2 | Clause 8.1, first paragraph: "A third-party [Detector]'s native catalogue, which the [Toolchain] orchestrates without claiming as its own, is outside this audit", set against passage 1 | 3, 5 | yes: the diff rewords a sentence in the paragraph; set against passage 1 | +| 3 | Section 2, the Defence row of the table: "A [Rule] with its documentation, in force and [Blocking]", set against passage 1 | 4 | the row's words unchanged (the table is realigned); set against passage 1 | +| 4 | Clause 6.2, first paragraph: "It MUST NOT supply a default" (1, set against passage 5; 2, quoting the whole sentence, set against passage 6) | 1, 2 | no | +| 5 | Clause 6.4 heading: "The toolchain SHOULD state its own defaults for anything the method leaves to the project" | 1 | no | +| 6 | Clause 6.4, first paragraph: "a documented [Toolchain] default is what the [Practitioner] falls back to" | 2 | no | +| 7 | Section 3, the Identifier resolution row: "clause 4.2 here for the rest", set against passage 8 | 2 | no in itself; set against passage 8, which the diff wrote | +| 8 | Clause 4.2, first paragraph: "every [Blocking] [Rule] the [Toolchain] runs, the project's own and the bundled alike", set against passage 7 | 2 | yes: added by the diff | +| 9 | Clause 6.2, last paragraph: "That check cannot verify truth, and a [Toolchain]'s [Conformance] MUST NOT be read as having verified it" (what "it" refers to) and "MUST reject a justification that could be pasted onto any [Exception] unchanged, by a check it documents" (whether the phrases are examples or the full set) | 4 | no | + +Notes, not passages: none under question 4. Under question 3, every reader remarks that the +detector specification was not supplied; those remarks are not question 4 answers and are not +marked as passages. Passages 4, 5 and 6 are three paragraphs (a heading is a paragraph of its +own), so readers 1 and 2 name the same passage only in passage 4. + +## Findings and where they lie + +- **Finding A, passage 1, section 2 closing paragraph** (readers 3, 4 and 5): the diff adds the + paragraph and the revision after run 84 rewrote it. Text the change wrote or altered. Blocks. + Readers 3 and 5 read its claim that clause 8.1 reaches every Blocking Rule a Toolchain runs as + contradicting clause 8.1's exclusion of a third-party Detector's native catalogue. +- **Finding B, passage 2, clause 8.1 first paragraph** (readers 3 and 5): the diff rewords the + sentence on what the check covers ("as its own, whatever kind of [Detector] carries it, and a + missing page is what the check exists to find"), and both readers set the paragraph against + passage 1, which the diff wrote. Text the change wrote or altered. Blocks. +- **Finding C, passage 4, clause 6.2 first paragraph** (readers 1 and 2): unchanged by the diff, + not parted from anything it refers to, and each reader sets it only against unchanged text + (reader 1 against the clause 6.4 heading, reader 2 against clause 6.4's first paragraph). + Elsewhere: a finding against the document, to be recorded in `acceptance/ROUTED.md`, where the + open row from run 83 on this sentence against clause 6.4's defaults is the same passage; + routed, and does not block. The marker writes only this record; the ROUTED entry is for the + editor. + +Passages 3 and 5 to 9 are each named by one reader: recorded, not findings, applied at the +editor's discretion. + +## Points the rules did not settle + +1. **Whether reader 4's third item names passage 1 as confusing**, since the reader says it read + the paragraph and the Defence row as consistent after checking both. Judgement: the Marking + section defines naming by quotation or by a clause number with an identifying description, + not by what the reader calls confusing, so it is named. The verdict does not turn on it: + readers 3 and 5 alone make passage 1 a finding. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 852dcd8, named before the readers started. The process files at +58924a8 (`ACCEPTANCE.md`, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md`, `tools/changes.py`, `spec-qa.py` and the CI workflows) are identical to those +at 852dcd8. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: two (findings A + and B). Not met. +- Passages named by two or more readers elsewhere: finding C, routed to `acceptance/ROUTED.md`; + does not block. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A and B; every keyed item passed its threshold. + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125972905); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125972491); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125973283). diff --git a/acceptance/runs/086-toolchain-contradictions-run-14.md b/acceptance/runs/086-toolchain-contradictions-run-14.md new file mode 100644 index 0000000..84667ae --- /dev/null +++ b/acceptance/runs/086-toolchain-contradictions-run-14.md @@ -0,0 +1,187 @@ +# Run 86: TOOLING-SPEC.md, toolchain contradictions, fourteenth run + +- **Document**: TOOLING-SPEC.md +- **Pull request**: [25](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25), opened later from branch + `editor/toolchain-contradictions`. The readers read commit 6da7409. Change: section 9 against + clause 9.1 on section 8; clauses 4.2 and 8.1 on a blocking rule whose documentation is missing; + clause 9.2's gap duty against the optional declaration; clause 4.3's heading against its body; + clause 4.1's count of the detector specification's MUSTs and its point 2 on what is judged. + Revised after run 85 in section 2's closing paragraph alone: it now names clause 4.2 alone as + the clause that reaches every blocking rule a toolchain runs, documented or not; clause 8.1 is + no longer named there. +- **Run within the pull request**: 14 (runs 57, 58, 62, 65, 67, 68, 72, 76, 81, 82, 83, 84 and + 85 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `60c864ced29d5011fb73ca0f35139b8bfcc146303dd001952fb43a99bcaa910e`) is + `QUESTIONS.md`, `TOOLING-SPEC.md` and `fixtures/zigpipe.md` at 6da7409, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model, and each sheet is that output's `result`, which + the marker confirmed byte for byte. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the four answers (S9 conformance under section 9, AS agent support, S8 whether section 8 +applies, Decl whether the absence of a declaration affects the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 6.1 | 6.2 | 6.3 | S9 | AS | S8 | Decl | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | --- | --- | --- | ---- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | miss | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Grades given, for the items the key accepts more than one: 5.1 Yes (all five; readers 2 and 3 +add that the two `(none)` rules are not Defences under section 2 and so outside 5.1); 6.1 +Partial (readers 1, 4 and 5, each mentioning that the Project record also holds calibrations and +conventions and the fixture gives no location for those, which meets the key's condition; +reader 5 adds that it would be Yes if the record is read as exceptions only, and the Partial is +the grade it gives for the case) and Yes (readers 2 and 3); 6.2 No (all five), each mentioning +that nothing rejects a generic justification or generic phrasing. All five grade 4.1 No on the +two undocumented bundled rules with nothing supplied for them; readers 2, 3, 4 and 5 say the +detector specification was not supplied and only the restated points were checked, which the +key's 4.1 row accepts. All five grade 4.2 No; readers 1, 4 and 5 state that they read the two +rules as Blocking because they are enabled or because `zig-qa run` stops when Lintomatic fails, +which the key's 4.2 row reads as a No. + +Question 1: correct clauses named per reader within the first six bullets. Reader 1: 4.1 with +4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3; 6.1 to 6.3 (five bullets; its first bullet is not needed for +the count and is not decided). Reader 2: 4.1; 4.2; 4.3 with 6.1 and 6.2; 5.1, 5.2, 6.3 and 4.5. +Reader 3: 4.4 and 4.5; 4.1; 4.2; 4.3 with 6.1; 6.2; 5.1 to 5.3 with 6.3. Reader 4: 4.1; 4.2; 4.3 +with 4.5; 5.1 to 5.3 with 6.3; 6.1 and 6.2. Reader 5: 4.1; 4.2; 4.3; 4.4 and 4.5; 5.1 to 5.3 +with 6.3; 6.1 and 6.2. Four or more correct bullets each. No bullet asserts anything the key +marks wrong. The first bullets of readers 2 and 4 (the toolchain judged as one assembled +whole), the remarks that section 7 or the defaults are SHOULDs (readers 2, 4 and 5), that a +declaration is optional (reader 2) and that a shipping project has two levels (reader 2) are +counted neither way, as the key says. + +Question 2: every reader states both required elements of each item and quotes the document. +For 2a all five quote section 1, "It is whatever the project assembles to run its checks +through"; for 2b all five quote section 9's "The project-level verdict is a single grade; ..."; +for 2c all five quote clause 9.1's "The two verdicts MUST be graded and declared separately, and +neither implies the other." No answer gives a grade per part, or a single grade for 2c. + +Question 3: the four answers are given by every reader as the key accepts, with reasons meeting +its conditions: S9 names 4.1, 4.2 and 6.2 (all five); AS says the toolchain does not conform at +all, or that there is no summary, or both (all five); S8 says Zigpipe ships no tooling to other +projects, and readers 4 and 5 add that section 8 bears on zig-qa's artefact grade (all five); +Decl says the declaration is optional, or that the verdict rests on the exercise of the clauses +(all five). + +### Misses + +- **Reader 4, clause 6.3: Partial.** The key accepts Yes only, with no condition. The reader + grants that `zig-qa list --exceptions` lists exceptions in the same format as the rule listing, + and grades Partial because calibrations and conventions are not described and so cannot be + shown to be enumerable. The reasoning does not quote the document in that row; it rests on + section 2's Project record entry, which the reader cites by section in its 6.1 row. It matches + no wrong reading the key names. A single miss on the item, the reader's, tolerated by the + threshold: four of five correct. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..6da7409 (9e914fe +being the merge base with `next`) adds, rewords, moves or deletes a sentence in the paragraph, +changes a word, term or link in it, or parts it from what it refers to. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------- | +| 1 | Clause 6.1, first paragraph: "where the Project record lives" (1); "The [Toolchain] MUST define where the [Project record] lives and MUST load it. Not a documentation convention. A path the [Toolchain] loads." (5); each set against passage 2 | 1, 5 | no | +| 2 | Section 2, the Project record terminology entry: "Calibrations, Exceptions and conventions", given with "the definition in section 2" (1) and "Section 2 defines the project record as" (5); each set against passage 1 | 1, 5 | no; the paragraph the diff inserts in section 2 follows it and parts it from nothing it refers to | +| 3 | Clause 5.3, second paragraph: "its entry MAY be sourced from wherever it is enabled" (what "wherever" covers) | 1 | no | +| 4 | Clause 4.5, first paragraph: "A [Blocking] failure at the [Detector] level MUST stop the levels below it from being treated as meaningful, in the order method section 5 states." (what the levels below a detector are) | 2 | no | +| 5 | Clause 6.2, first paragraph: "It MUST NOT supply a default." (2, set against passage 6); "MUST NOT supply a default" (4, set against passage 6) | 2, 4 | no | +| 6 | Clause 6.4: "6.4, which SHOULDs a documented toolchain default" (2, a description); "SHOULD state its own defaults for anything the method leaves to the project" (4, the heading's words) | 2, 4 | no | +| 7 | Clause 4.1, first paragraph: "The sentences that follow restate the ones most often failed, so that a grader knows where to look first; they are not the whole test, and a grader without that document to hand can check only them, and says so in the verdict." (set against passage 8) | 3 | yes: the diff wrote the sentence | +| 8 | Clause 4.1, point 2: "does the [Detector] meet every MUST of sections 4 to 7 of the [detector specification], exercised as its clause 8.1 describes?" (set against passage 7) | 3 | yes: the diff rewords sentences in point 2 | + +Notes, not passages: reader 2's first item quotes clause 4.1 point 2, "with the wrapping in +place, does the [Detector] meet every MUST of sections 4 to 7 ... exercised as its clause 8.1 +describes?", and says only that it could not tell what the exercise is "because the detector +specification was not in what I was given". That is a remark that another document was not +supplied, which the Marking section's rule on question 4 records as a note and not a passage, +"however much of the document it quotes to locate the reference"; it is not counted with +reader 3's passage 8. Reader 2's second item also says method section 5 was not given, but +its remark that the clause does not define the levels below a detector is about the passage, so +passage 4 is counted. Under question 3, readers 2, 3, 4 and 5 remark that the detector +specification was not supplied; those are not question 4 answers. + +## Findings and where they lie + +- **Finding A, passage 1, clause 6.1 first paragraph** (readers 1 and 5): unchanged by the diff, + not parted from anything it refers to, and each reader sets it only against passage 2, which + is unchanged. Elsewhere: a finding against the document, recorded in `acceptance/ROUTED.md`, + where the open row on clause 6.1's first paragraph against section 2's Project record entry is + the same passage; routed, and does not block. +- **Finding B, passage 2, section 2's Project record entry** (readers 1 and 5): unchanged by the + diff; the paragraph the diff inserts at the end of section 2 comes after the entry and the + paragraph that follows it, and parts the entry from nothing it refers to; each reader sets it + only against passage 1. Elsewhere: routed to `acceptance/ROUTED.md` with finding A, whose open + row names both; does not block. +- **Finding C, passage 5, clause 6.2 first paragraph** (readers 2 and 4): unchanged by the diff, + not parted from anything it refers to, each reader setting it only against clause 6.4, which + is unchanged. Elsewhere: routed to `acceptance/ROUTED.md`, where the open row on this sentence + against clause 6.4's defaults is the same passage; does not block. +- **Finding D, passage 6, clause 6.4** (readers 2 and 4, counted as the same passage under the + open-point rule; see below): unchanged by the diff. Elsewhere: routed to + `acceptance/ROUTED.md` with finding C; does not block. + +The marker writes only this record; the ROUTED entries are for the editor. Passages 3, 4, 7 and +8 are each named by one reader: recorded, not findings, applied at the editor's discretion. +Passages 7 and 8 lie in text the change wrote, and are named by reader 3 alone. + +## Points the rules did not settle + +1. **Whether readers 2 and 4 name the same paragraph of clause 6.4.** Reader 4 quotes the + heading's words; reader 2 describes 6.4 as one "which SHOULDs a documented toolchain + default", which could identify the heading (its SHOULD) or the first paragraph ("a documented + [Toolchain] default"), and a heading is a paragraph of its own. Marked in the direction the + open-point rule gives: counted as the same passage, so finding D. The verdict does not turn on + it, since the passage lies outside the change either way. + +Considered and settled by a rule, so not open: reader 2's clause 4.1 item is a note under the +Marking section's question 4 rule, as above; and readers 1 and 5 each name both passage 1 and +passage 2, since each gives a clause or section number with words that identify each paragraph. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 568e323, named before the readers started. The process files at +6da7409 (`ACCEPTANCE.md`, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md`, `tools/changes.py`, `spec-qa.py` and the CI workflows and pull request +template under `.github/`) are identical to those at 568e323. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item but + 6.3, which is four of five. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: none. Met. +- Passages named by two or more readers elsewhere: findings A to D, routed to + `acceptance/ROUTED.md`; do not block. + +## Findings applied + +None at the time of marking. + +## Verdict + +PASS. Every keyed item met its threshold, no miss matches a wrong reading the key names, and +no finding lies in text the change wrote or altered. + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125974067); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125973613); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/25#issuecomment-6125974453). diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index 4994f74..3f7ad64 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -1,15 +1,22 @@ # Acceptance run records -One file per cohort run, `NNN-.md`, numbered in the one sequence the three documents -share. Runs 1 to 15 predate this directory and are summarised in the changelog entries for -1.0.0 and 1.0.1 of the method specification, 1.0.0 of the detector specification and 0.2.0 of -the toolchain specification; numbering here continues from 16. +One file per cohort run, `NNN--run-.md`, numbered in the one sequence the three +documents share, with `` the run number within its pull request. Runs 1 to 15 predate this +directory and are summarised in the changelog entries for 1.0.0 and 1.0.1 of the method +specification, 1.0.0 of the detector specification and 0.2.0 of the toolchain specification; +numbering here continues from 16. -A record states, in this order: the document; the pull request and the commit the readers were -given; the cohort size and model; the run number within that pull request; the marks per reader -per question against the key; the passages quoted as confusing and by how many readers; the -findings applied to the text as a result; and the verdict against the criteria in -`ACCEPTANCE.md`. Answer sheets are not pasted in full; the marks are the record. +A record states what `ACCEPTANCE.md` lists under "The record", in this order: the document; the +pull request and the commit the readers were given; the cohort size and model, with the model +version each reader reported; how the readers were isolated and the command they were started +with; any reader set aside and why; who marked and, for a delegated marker, how it was started; +the marks per reader per keyed item, with each miss and the reader's reasoning in brief; every +passage named under question 4, with the words quoted or the description given, its clause and +the readers naming it; each finding and whether it lies in text the change wrote or altered; any +point the rules did not settle; whether the run was made under a process change before it merged +and, if so, at which commit; the findings applied afterwards; the verdict; and a link to the +answer sheets posted on the pull request. Earlier records show the style only; where one differs +from this order, this file wins. Answer sheets are not kept here; the marks are the record. The `Unreleased` changelog line for the change names the record, and `tools/changes.py` refuses a specification change whose pull request adds no file here. diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 5a2602b..98df913 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -1,20 +1,21 @@ # Answer key for TOOLING-SPEC.md -Marked against the document at the version named in the changelog entry that records the run. -Quotations below are from the document; when a clause is reworded, update the quotation here in -the same commit. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, +update the quotation here in the same commit. `ACCEPTANCE.md` says how each item is marked; +`acceptance/README.md` says what each part of this key is for. ## 1. Restatement -A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting an obligation the document does not state, is a correct restatement. +A bullet is correct when it names one of these obligations. Four or more correct bullets, with +no bullet making an assertion listed below as wrong, is a correct restatement. - Route defences only through detectors that conform to the detector specification, wrapping permitted ([4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification)). - Resolve every identifier a routed defence can print, including the project's own, offline, - to documentation stating the correct construction ([4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access)). + to documentation stating the correct construction ([4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)). - Forbid, by disabling or by a blocking defence, every suppression route that bypasses the - project record ([4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record)). + project record ([4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)). - The toolchain's own entry point meets the detector reporting clauses and prints identifiers unaltered ([4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes)), and runs detectors before runners, stopping on a detector failure ([4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure)). - List the active defences, derived from configuration, including the project's own ([5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) to @@ -23,63 +24,91 @@ bullet asserting an obligation the document does not state, is a correct restate same way as the defences ([6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) to [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences)). - A shipped toolchain also fails its release on undocumented bundled defences and runs its own defences on its own source ([8.1](../../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation), [8.2](../../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source)). +- Make everything the method requires of a practitioner possible without the project building + the mechanism first (section [1](../../TOOLING-SPEC.md#1-what-this-document-is-for), the governing principle). +- Not describe partial conformance as conformance (section [9](../../TOOLING-SPEC.md#9-conformance)). Wrong if it asserts: the toolchain must be a single product; a declaration is required to -conform; the agent summary (section [7](../../TOOLING-SPEC.md#7-agent-context)) is a MUST. +conform; the agent summary (section [7](../../TOOLING-SPEC.md#7-agent-context)) is a MUST. A bullet that calls agent support or the +self-audit a further condition "where it applies", says that section [7](../../TOOLING-SPEC.md#7-agent-context) is recommended or a +SHOULD, or says a declaration is not a condition, asserts no obligation and is neither correct +nor wrong. So is a bullet that states how +conformance is measured rather than what is required: the toolchain judged as one assembled +whole at the project level (sections [1](../../TOOLING-SPEC.md#1-what-this-document-is-for) and [9](../../TOOLING-SPEC.md#9-conformance)), the project-level verdict a single grade +(section [9](../../TOOLING-SPEC.md#9-conformance)), or a shipped artefact graded separately ([9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately)). + +Wrong readings the text invites, which count against the document on one reader: none named. ## 2. Load-bearing statements -- **2a.** At the project level; the toolchain is whatever the project has assembled from - third-party, first-party and project-level parts. Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "A project's - Toolchain Conforms if every MUST in sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record) holds across the assembled parts, wherever - each part came from". An answer that says conformance is graded per tool, or that only a - packaged product can conform, is wrong. -- **2b.** One grade; which part satisfied each clause is evidence, not a grade of its own. Key - quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "The project-level verdict is a single grade; which part of the Toolchain - satisfied each clause is evidence for that grade, not a second grade." -- **2c.** Two, graded and declared separately, and neither implies the other. Key quotation, - [9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately): "The two verdicts MUST be graded and declared separately, and neither implies the other." - An answer giving one grade, or saying the artefact grade follows from the project grade, is - wrong. +- **2a.** Required: (1) at the project level; (2) the toolchain is whatever the project has + assembled to run its checks through, from third-party, first-party and project-level parts. + Wrong if it says conformance is graded per tool, or that only a packaged product can conform. + Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "A project's Toolchain Conforms if every MUST in sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record) + holds across the assembled parts, wherever each part came from". An answer that says everything the + project runs its checks through, or the assembled parts from wherever they came, or that lists + the three kinds of part, states element (2). +- **2b.** Required: (1) one grade; (2) which part satisfied each clause is evidence for that + grade, not a grade of its own. Wrong if it gives a grade per part. Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): + "The project-level verdict is a single grade; which part of the Toolchain satisfied each + clause is evidence for that grade, not a second grade." +- **2c.** Required: (1) two; (2) neither implies the other. Wrong if it gives one grade, or says + the artefact grade follows from the project grade or the reverse. That the two are graded and + declared separately is explanation; an answer that says it has stated element (1). Key quotation, [9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately): "The two verdicts MUST be graded and declared separately, and + neither implies the other." ## 3. Application: the Zigpipe project -| Clause | Intended verdict | Reasoning the fixture supports | -| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | Lintomatic fails detector [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) on two undocumented bundled rules and nothing in the wrapping closes that gap | -| [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)` | -| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | Ignore comments disabled project-wide; exceptions only through the record the toolchain reads | -| [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | `zig-qa run` is local, prints Lintomatic's output unchanged, one line per finding | -| [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | Lintomatic, then the tests, stopping on a Lintomatic failure | -| [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes, Partial or No | `zig-qa list` prints identifier, summary and documentation path without running the rules; Partial or No is accepted where the reasoning counts the two `(none)` routes, since 5.1 says the listing MUST provide the route and [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) point 2 says a gap may be named again by a later clause | -| [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | Derived from `lintomatic.toml`, not hand-maintained | -| [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | Yes | The `proj.*` rules appear in the same listing | -| [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes | `quality-exceptions.toml` is a defined path that `zig-qa run` loads | -| [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Partial or No | A justification is required and an empty one rejected, but nothing rejects a justification that could be pasted onto any entry | -| [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | `zig-qa list --exceptions` uses the same format and operation as the rule listing | - -Either Partial or No is accepted for [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one); the reasoning must mention the generic-justification -requirement. A Yes on [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) with reasoning that quotes only the first sentence of the clause is a -finding against the document. - -Section [9](../../TOOLING-SPEC.md#9-conformance) verdict: **does not conform**, because [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) and [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) fail; partial conformance MUST NOT be -described as conformance. **Not with agent support**, because section [7](../../TOOLING-SPEC.md#7-agent-context) is not met (no summary). -**Section [8](../../TOOLING-SPEC.md#8-self-audit) does not apply**, because Zigpipe ships nothing to other projects; a reader who applies -[8.1](../../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) to Zigpipe on the basis that Zigpipe's tooling includes a shipped package (zig-qa) has read -"where the Toolchain is one the project ships" (section [9](../../TOOLING-SPEC.md#9-conformance)) as reaching the consuming project, and that counts against the document. The -correct reading is that section [8](../../TOOLING-SPEC.md#8-self-audit) bears on zig-qa's own artefact grade, not on Zigpipe. - -Declaration: **does not affect the verdict**. Key quotation, [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance): "The declaration is optional." -A reader who says Zigpipe cannot be graded without one has read [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) wrongly. - -A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes on the grounds that Lintomatic "mostly" conforms has read either [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) -or the detector document's section [8](../../DETECTOR-SPEC.md#8-conformance) differently; it counts against this document only if the -reasoning quotes [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification). A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes because they missed the two undocumented rules -has misread the fixture. +| Clause | Accepted grades | Condition on the reasoning | What the fixture supports | +| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | None | Lintomatic fails detector [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) on two undocumented bundled rules and nothing in the wrapping closes that gap; a reader who notes that the detector specification was not supplied and grades from this document's summary of it is correct | +| [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) | No | None. A No given on the condition that the two bundled rules are blocking is a No: their firing stops `zig-qa run`, so they are | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)` | +| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) | Yes | None | The fixture names one suppression route of Lintomatic's, the inline ignore comment, disabled project-wide, with exceptions only through the record the toolchain reads; a grade given on the condition that the comment is Lintomatic's only route is the grade for this case | +| [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | None | `zig-qa run` is local, prints Lintomatic's output unchanged, one line per finding | +| [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | None | Lintomatic, then the tests, stopping on a Lintomatic failure | +| [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes, Partial or No | None | `zig-qa list` prints identifier, summary and documentation path without running the rules; Partial or No is the reading that counts the two `(none)` routes, since [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) says the listing MUST provide the route and [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) point 2 says a gap may be named again by a later clause | +| [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | None | Derived from `lintomatic.toml`, not hand-maintained | +| [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | Yes | None | The `proj.*` rules appear in the same listing | +| [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes or Partial | For Partial: mentions that the fixture names a place for exceptions and none for calibrations or conventions | `quality-exceptions.toml` is a defined path that `zig-qa run` loads; the project record entry also covers calibrations and conventions | +| [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Partial or No | Mentions that nothing rejects a generic justification, one that could be pasted onto any entry | A justification is required and an empty one rejected | +| [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | None | `zig-qa list --exceptions` uses the same format and operation as the rule listing | + +The four answers the question then asks for, each with the condition its reason must meet: + +- **Conformance under section [9](../../TOOLING-SPEC.md#9-conformance).** Accepted answer: does not conform. Condition: the reason + names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) or [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) among the failing clauses, or points to the grades the reader gave above, as "the + clauses graded No", where those grades have one of the three at No or Partial; other clauses + named alongside are not marked. +- **Conformance with agent support.** Accepted answer: no. Condition: the reason says that + section [7](../../TOOLING-SPEC.md#7-agent-context) is not met, or that there is no summary, or that the toolchain does not conform + under section [9](../../TOOLING-SPEC.md#9-conformance) at all. +- **Whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies.** Accepted answer: it does not. Condition: the reason says that + Zigpipe ships nothing to other projects, or that section [8](../../TOOLING-SPEC.md#8-self-audit) bears on zig-qa's own artefact + grade and not on Zigpipe. +- **Whether the absence of a declaration affects the verdict.** Accepted answer: it does not. + Condition: the reason says that the declaration is optional, or not a condition of + conformance, or that a verdict rests on the evidence. Key quotation, [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance): "The declaration is + optional." + +Wrong readings the text invites, which count against the document on one reader: + +- [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) graded Yes with reasoning that says a justification is required or an empty one rejected + and does not mention a generic justification, which has not read the sentence on one. +- Section [8](../../TOOLING-SPEC.md#8-self-audit) said to apply to Zigpipe with reasoning that says its tooling includes a shipped + package, which has read "where the project also ships a Detector or a Toolchain" (section [9](../../TOOLING-SPEC.md#9-conformance)) + as reaching the consuming project. +- [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) graded Yes with reasoning that says Lintomatic mostly or largely conforms and quotes [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), + which has read it as admitting a proportion. +- The declaration said to affect the verdict, with reasoning that Zigpipe cannot be graded without + one, which has read [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) as a condition. + +A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes because they missed the two undocumented rules has misread the +fixture, which is the reader's miss. ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage named, as `ACCEPTANCE.md` defines naming and the same +passage. A passage named by two or more readers of the run is a finding: it blocks the change +where it lies in text the change wrote or altered, and is routed to `acceptance/ROUTED.md` +otherwise. What to change is the editor's decision; the reader is not asked and any suggestion +it volunteers is discarded unread. diff --git a/acceptance/toolchain/QUESTIONS.md b/acceptance/toolchain/QUESTIONS.md index a4da266..fa45dd3 100644 --- a/acceptance/toolchain/QUESTIONS.md +++ b/acceptance/toolchain/QUESTIONS.md @@ -11,7 +11,8 @@ toolchain? ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. +Answer each part of each question in a sentence or two, then quote the sentence or sentences of +the document you are relying on. - 2a. At what level is toolchain conformance measured, and what is the toolchain? - 2b. A project's tooling is assembled from a third-party detector, a first-party package and @@ -23,10 +24,10 @@ Answer each in one sentence, followed by the sentence of the document you are re ## 3. Application Grade the Zigpipe project's toolchain, as described in the fixture, against every MUST clause in -sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record): [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record), [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure), [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration), [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones), [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences). For each give Yes, +sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record): [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access), [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own), [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure), [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration), [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones), [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences). For each give Yes, Partial or No and one line of reasoning. Then state whether the toolchain conforms under section -9, whether it conforms with agent support, and whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies to it. Finally, state -whether the absence of a declaration affects the verdict. +[9](../../TOOLING-SPEC.md#9-conformance) and why, whether it conforms with agent support and why, and whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies to it +and why. Finally, state whether the absence of a declaration affects the verdict, and why. ## 4. Confusion diff --git a/tools/php-qa-ci.md b/tools/php-qa-ci.md index d28c8b3..f241cb4 100644 --- a/tools/php-qa-ci.md +++ b/tools/php-qa-ci.md @@ -22,7 +22,7 @@ Every mechanism below was checked by running it in a checkout: `bin/qa`, `bin/qa For its own PHPStan bundle, every detector clause holds. A project's rules go under `rules:` in its `qaConfig/phpstan.neon` (detector [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it)). `bin/phpstan-rule phpqaci.nullCoalescingFalse src/ZzProbe.php` on a probe containing `?? false` printed `FIRED (1)` with the location and exited 1, and the same command for `phpqaci.nestedTernary` printed `did not fire` (detector [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code)). PHPStan prints the `phpqaci.*` identifier under every finding, and `RequireRuleIdentifierConstantRule`, on by default, rejects a magic-string identifier (detector [4.3](../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding), detector [4.4](../DETECTOR-SPEC.md#44-the-detector-should-enforce-43-with-a-rule-of-its-own)). `bin/qa -t phpstan -p ` runs one lane over one file locally with the result in the terminal, and the CI script runs the same `bin/qa` (detector [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to detector [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally)). `bin/rule-doc phpqaci.nestedTernary` resolves the identifier as printed to its rule, bundle, source and shipped page, offline, and the page carries a `The correct construction` section (detector [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) to detector [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) for the bundle). `RuleDocumentationTest` fails the build when a declared identifier resolves to no page, or to a page whose construction section is missing or only restates the summary (detector [6.4](../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation)). Inline `@phpstan-ignore` in every form is a finding of `ForbidInlinePhpstanIgnoreRule` (detector [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)), and behind it the default tier turns on PHPStan's `reportIgnoresWithoutComments`: a probe ignore with no reason printed both `phpqaci.inlinePhpstanIgnore` and `ignore.noComment (non-ignorable)` (detector [7.2](../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason)). -At toolchain level, every lane prints a stable identifier when it fails: a run of `bin/qa -t branchNamePolicy` on a `plan/*` branch ended with `🪪 phpqaci.branchNamePolicy (vendor/bin/rule-doc phpqaci.branchNamePolicy)`, and `EveryLaneNamesItselfTest` fails the build on a lane without one. `bin/rule-doc phpqaci.packageType` resolves a lane identifier to its page as it does a rule's, and `bin/rules .` prints no `no documentation page` row: all 69 indexed identifiers (`bin/rule-doc --list`) resolve to a page ([4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access)). `bin/rules .` lists every rule in the resolved neon chain, including the rules `phpstan/extension-installer` delivers, with the package each came from, and every lane with its identifier and documentation route; phase runners alone carry none ([5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) to [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones)). The record is the `ignoreErrors` block PHPStan itself loads ([6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself)); the `pij` lane reads it through every `includes:` file, and a probe baseline included from `qaConfig/phpstan.neon` with an uncommented entry failed it, naming the included file and line ([6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record)). A `phparkitect-baseline.json` is never read: on a probe project with one violation, PHPArkitect with a generated baseline printed `No violations detected`, and with `--skip-baseline`, which the lane always passes, reported the violation; the lane names the file when one is present ([4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record)). The same listing enumerates the record beside the defences, together with the deferred-defect and no-pattern record `qaConfig/defect-record.neon` (`defectRecord` in `--json`), which the same lane fails on when an entry does not read in full ([6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences)). `docs/defence-before-fix-defaults.md` states the toolchain's default for the owner, the project record, the sweep scope, fixtures and the method's three calibrations, linked from the agent block ([6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)). The entry point runs coding standards, linting and static analysis before tests and stops on a failure ([4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure)). `bin/rules --write-agent-summary` writes one line per active defence, identifier and page, into a marked region of the project's `CLAUDE.md`, refreshed on every install and update ([7.1](../TOOLING-SPEC.md#71-the-toolchain-should-generate-a-summary-of-the-active-defences-suitable-for-an-agents-context), [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically)). The release guard covers rules and lanes and requires a page stating a correct construction ([8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation)), and the self-check runs both bundled rule sets and the PHPArkitect tier on the package's own source ([8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source)). The manifest declares method 1.1.0 and toolchain 0.2.0 at both levels, with the gaps below recorded against their clauses, and `DefenceBeforeFixDeclarationTest` holds each recorded gap to a decision and a probe that fails when the gap closes ([9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance)). +At toolchain level, every lane prints a stable identifier when it fails: a run of `bin/qa -t branchNamePolicy` on a `plan/*` branch ended with `🪪 phpqaci.branchNamePolicy (vendor/bin/rule-doc phpqaci.branchNamePolicy)`, and `EveryLaneNamesItselfTest` fails the build on a lane without one. `bin/rule-doc phpqaci.packageType` resolves a lane identifier to its page as it does a rule's, and `bin/rules .` prints no `no documentation page` row: all 69 indexed identifiers (`bin/rule-doc --list`) resolve to a page ([4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access)). `bin/rules .` lists every rule in the resolved neon chain, including the rules `phpstan/extension-installer` delivers, with the package each came from, and every lane with its identifier and documentation route; phase runners alone carry none ([5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) to [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones)). The record is the `ignoreErrors` block PHPStan itself loads ([6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself)); the `pij` lane reads it through every `includes:` file, and a probe baseline included from `qaConfig/phpstan.neon` with an uncommented entry failed it, naming the included file and line ([6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)). A `phparkitect-baseline.json` is never read: on a probe project with one violation, PHPArkitect with a generated baseline printed `No violations detected`, and with `--skip-baseline`, which the lane always passes, reported the violation; the lane names the file when one is present ([4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own)). The same listing enumerates the record beside the defences, together with the deferred-defect and no-pattern record `qaConfig/defect-record.neon` (`defectRecord` in `--json`), which the same lane fails on when an entry does not read in full ([6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences)). `docs/defence-before-fix-defaults.md` states the toolchain's default for the owner, the project record, the sweep scope, fixtures and the method's three calibrations, linked from the agent block ([6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)). The entry point runs coding standards, linting and static analysis before tests and stops on a failure ([4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure)). `bin/rules --write-agent-summary` writes one line per active defence, identifier and page, into a marked region of the project's `CLAUDE.md`, refreshed on every install and update ([7.1](../TOOLING-SPEC.md#71-the-toolchain-should-generate-a-summary-of-the-active-defences-suitable-for-an-agents-context), [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically)). The release guard covers rules and lanes and requires a page stating a correct construction ([8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation)), and the self-check runs both bundled rule sets and the PHPArkitect tier on the package's own source ([8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source)). The manifest declares method 1.1.0 and toolchain 0.2.0 at both levels, with the gaps below recorded against their clauses, and `DefenceBeforeFixDeclarationTest` holds each recorded gap to a decision and a probe that fails when the gap closes ([9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance)). ## How it is not conformant @@ -47,40 +47,40 @@ php-qa-ci is a toolchain rather than a tool, so the question for each lane it ru Detector rows grade PHPStan as wrapped, with PHPArkitect as wrapped noted where it differs. -| Document | Clause | Result | Evidence | -| --------- | ------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------- | -| Detector | [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules under `rules:` in `qaConfig/phpstan.neon`; PHPArkitect project rules in `phparkitect.php` | -| Detector | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `bin/phpstan-rule ` fired red on a probe and stayed silent on another rule; PHPArkitect as wrapped has none | -| Detector | [4.3](../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `🪪 phpqaci.nullCoalescingFalse` printed under the finding, identifier in JSON mode too; PHPArkitect prints prose only | -| Detector | [4.4](../DETECTOR-SPEC.md#44-the-detector-should-enforce-43-with-a-rule-of-its-own) | Yes | `RequireRuleIdentifierConstantRule` in `rules-default.neon` | -| Detector | [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | Every command ran in the checkout with no service | -| Detector | [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Yes | `bin/qa -t phpstan -p src/ZzProbe.php` scanned that path alone; PHPArkitect's lane is non-path-supporting | -| Detector | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | Findings in the terminal, archived log named after them | -| Detector | [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | `ci.bash` runs `bin/qa` | -| Detector | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Yes | `bin/rule-doc phpqaci.nestedTernary` resolved as printed to its page; an unknown identifier exits 1 | -| Detector | [6.2](../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Partial | Bundle and lanes resolve offline, 69 of 69; `bin/rule-doc method.notFound` names phpstan.org, PHPStan's catalogue is online only | -| Detector | [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | Partial | Bundle and lane pages ship with the package; PHPStan's own rules ship none; PHPArkitect's tier rules have no page | -| Detector | [6.4](../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation) | Yes | `RuleDocumentationTest` fails the build on an identifier with no page or a page stating no correct construction | -| Detector | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | Inline ignores detectable by a bundled rule; PHPArkitect's baseline never read (`--skip-baseline` always passed) | -| Detector | [7.2](../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) | Yes | `reportIgnoresWithoutComments` on in the default tier; a probe printed `ignore.noComment (non-ignorable)` | -| Toolchain | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | PHPArkitect's tier rules carry no identifier, single-file run or resolver; PHPStan's native catalogue is online only; both declared | -| Toolchain | [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | Yes | `bin/rules .`: no `no documentation page` row; `bin/rule-doc phpqaci.packageType` resolves to its page | -| Toolchain | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | Inline ignore forbidden; `phparkitect-baseline.json` never read; an included PHPStan baseline fails the justification lane | -| Toolchain | [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | `bin/qa -t -p ` for every path-supporting lane | -| Toolchain | [4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | `bin/qa` runs coding standards, linting, static analysis, then tests, failing fast or reporting every failure before the success banner | -| Toolchain | [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes | Rules listed with identifier, summary, route and source package; lanes with identifier and page | -| Toolchain | [5.2](../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | Derived from the resolved neon chain and the lane registry | -| Toolchain | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | Yes | Contributor-only rules in the self-listing | -| Toolchain | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes | `ignoreErrors` in `qaConfig/phpstan.neon`, loaded by PHPStan | -| Toolchain | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Yes | `bin/qa -t pij` reads the whole include chain; an uncommented entry in an included file failed it | -| Toolchain | [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | `bin/rules .` lists the record, an included baseline's entries among it, and the deferred-defect record `qaConfig/defect-record.neon`, in text and JSON | -| Toolchain | [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project) | Yes | `docs/defence-before-fix-defaults.md` states the owner, record, sweep, fixture and calibration defaults | -| Toolchain | [7.1](../TOOLING-SPEC.md#71-the-toolchain-should-generate-a-summary-of-the-active-defences-suitable-for-an-agents-context) | Yes | `bin/rules --write-agent-summary` writes one line per active defence into the `CLAUDE.md` region | -| Toolchain | [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically) | Yes | Block written and refreshed on install and update | -| Toolchain | [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | Yes | `RuleDocumentationTest` covers rules and lanes and requires a page stating a correct construction | -| Toolchain | [8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source) | Yes | `SelfCheckRunsBundledRulesTest`; own `qaConfig/phpstan.neon` includes both bundles; `bin/qa -t arch` runs the tier on `src/` | -| Toolchain | [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Yes | Artefact and project graded separately above | -| Toolchain | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | Yes | `composer.json` `extra.defence-before-fix`: method 1.1.0, toolchain 0.2.0, two gaps; `project` with the same keys and one gap | +| Document | Clause | Result | Evidence | +| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Detector | [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules under `rules:` in `qaConfig/phpstan.neon`; PHPArkitect project rules in `phparkitect.php` | +| Detector | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `bin/phpstan-rule ` fired red on a probe and stayed silent on another rule; PHPArkitect as wrapped has none | +| Detector | [4.3](../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `🪪 phpqaci.nullCoalescingFalse` printed under the finding, identifier in JSON mode too; PHPArkitect prints prose only | +| Detector | [4.4](../DETECTOR-SPEC.md#44-the-detector-should-enforce-43-with-a-rule-of-its-own) | Yes | `RequireRuleIdentifierConstantRule` in `rules-default.neon` | +| Detector | [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | Every command ran in the checkout with no service | +| Detector | [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Yes | `bin/qa -t phpstan -p src/ZzProbe.php` scanned that path alone; PHPArkitect's lane is non-path-supporting | +| Detector | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | Findings in the terminal, archived log named after them | +| Detector | [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | `ci.bash` runs `bin/qa` | +| Detector | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Yes | `bin/rule-doc phpqaci.nestedTernary` resolved as printed to its page; an unknown identifier exits 1 | +| Detector | [6.2](../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Partial | Bundle and lanes resolve offline, 69 of 69; `bin/rule-doc method.notFound` names phpstan.org, PHPStan's catalogue is online only | +| Detector | [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | Partial | Bundle and lane pages ship with the package; PHPStan's own rules ship none; PHPArkitect's tier rules have no page | +| Detector | [6.4](../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation) | Yes | `RuleDocumentationTest` fails the build on an identifier with no page or a page stating no correct construction | +| Detector | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | Inline ignores detectable by a bundled rule; PHPArkitect's baseline never read (`--skip-baseline` always passed) | +| Detector | [7.2](../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) | Yes | `reportIgnoresWithoutComments` on in the default tier; a probe printed `ignore.noComment (non-ignorable)` | +| Toolchain | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | PHPArkitect's tier rules carry no identifier, single-file run or resolver; PHPStan's native catalogue is online only; both declared | +| Toolchain | [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) | Yes | `bin/rules .`: no `no documentation page` row; `bin/rule-doc phpqaci.packageType` resolves to its page | +| Toolchain | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) | Yes | Inline ignore forbidden; `phparkitect-baseline.json` never read; an included PHPStan baseline fails the justification lane | +| Toolchain | [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | `bin/qa -t -p ` for every path-supporting lane | +| Toolchain | [4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | `bin/qa` runs coding standards, linting, static analysis, then tests, failing fast or reporting every failure before the success banner | +| Toolchain | [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes | Rules listed with identifier, summary, route and source package; lanes with identifier and page | +| Toolchain | [5.2](../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | Derived from the resolved neon chain and the lane registry | +| Toolchain | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | Yes | Contributor-only rules in the self-listing | +| Toolchain | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes | `ignoreErrors` in `qaConfig/phpstan.neon`, loaded by PHPStan | +| Toolchain | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Yes | `bin/qa -t pij` reads the whole include chain; an uncommented entry in an included file failed it | +| Toolchain | [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | `bin/rules .` lists the record, an included baseline's entries among it, and the deferred-defect record `qaConfig/defect-record.neon`, in text and JSON | +| Toolchain | [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project) | Yes | `docs/defence-before-fix-defaults.md` states the owner, record, sweep, fixture and calibration defaults | +| Toolchain | [7.1](../TOOLING-SPEC.md#71-the-toolchain-should-generate-a-summary-of-the-active-defences-suitable-for-an-agents-context) | Yes | `bin/rules --write-agent-summary` writes one line per active defence into the `CLAUDE.md` region | +| Toolchain | [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically) | Yes | Block written and refreshed on install and update | +| Toolchain | [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | Yes | `RuleDocumentationTest` covers rules and lanes and requires a page stating a correct construction | +| Toolchain | [8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source) | Yes | `SelfCheckRunsBundledRulesTest`; own `qaConfig/phpstan.neon` includes both bundles; `bin/qa -t arch` runs the tier on `src/` | +| Toolchain | [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Yes | Artefact and project graded separately above | +| Toolchain | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | Yes | `composer.json` `extra.defence-before-fix`: method 1.1.0, toolchain 0.2.0, two gaps; `project` with the same keys and one gap | ## Notes for a practitioner diff --git a/tools/ts-qa-ci.md b/tools/ts-qa-ci.md index aed07db..aeeb1cd 100644 --- a/tools/ts-qa-ci.md +++ b/tools/ts-qa-ci.md @@ -39,56 +39,56 @@ The three conformance mechanisms cover the ESLint lanes in full and nothing else Bespoke rules under detector clause [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) enter through `tsQaConfig/eslint.config.js`, which is merged into the resolved flat config, so any ESLint plugin object the project writes is a first-class defence. Clause [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) is met by `ts-qa rule `, which runs the ordinary lint lane over one path and reports whether that one rule fired, exiting 1 with locations when it did; a fixture carrying an `eslint-disable` comment made `ts-qa/no-eslint-disable` fire and a clean rule stay silent on the same file. Identifiers are `ts-qa/`, chosen once in the plugin map and printed by ESLint with every finding, which is clause [4.3](../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding). Section [5](../DETECTOR-SPEC.md#5-reporting) is met by the entry point, its `-p ` subset flag and the `--llm` verdict table, which names the failing tool and the path to the full result. Clauses [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) to [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) are met for every bundled plugin rule by `ts-qa rule-doc `, which resolves the identifier exactly as printed to its section of the shipped `docs/cdd-rules.md` and prints it, offline, and a test in the suite walks every plugin rule to prove the page is there. That closes for the toolchain what ESLint alone fails, since ESLint's own rule pages are online only. -On the toolchain document, the listing under clauses [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) and [5.2](../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) is `ts-qa rules`, which walks the resolved configuration and prints each active defence's identifier, severity, a terse description and the on-disk route, then the project record beneath it in the same listing; the repository's own rule against its own source appears alongside the bundled ones. The project record under [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) is `tsQaConfig/tier-a-exemptions.json`, loaded by the toolchain itself; an entry without a justification throws, and so do "legacy" and "needed for now", each rejection naming the entry and asking for the hazard and the scope, which is [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) ([configuration](https://github.com/LongTermSupport/ts-qa-ci/blob/main/docs/configuration.md)). Inline suppression is forbidden under [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) by `ts-qa/no-eslint-disable`, which fires on every `eslint-disable`, `oxlint-disable` and `@ts-` directive form and points the reader at the record. Clause [4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) holds: phases 0 to 3 are detectors, phase 4 is the test runner, and a run that fails at phase 1 runs nothing after it. Clause [8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source) is the continuous integration workflow, which runs the toolchain on the package itself and fails on anything it cannot fix. The declaration under [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) is the `defenceBeforeFix` key in `package.json`, carrying both levels of clause [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) with their gaps recorded, so neither level claims conformance. +On the toolchain document, the listing under clauses [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) and [5.2](../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) is `ts-qa rules`, which walks the resolved configuration and prints each active defence's identifier, severity, a terse description and the on-disk route, then the project record beneath it in the same listing; the repository's own rule against its own source appears alongside the bundled ones. The project record under [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) is `tsQaConfig/tier-a-exemptions.json`, loaded by the toolchain itself; an entry without a justification throws, and so do "legacy" and "needed for now", each rejection naming the entry and asking for the hazard and the scope, which is [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) ([configuration](https://github.com/LongTermSupport/ts-qa-ci/blob/main/docs/configuration.md)). Inline suppression is forbidden under [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) by `ts-qa/no-eslint-disable`, which fires on every `eslint-disable`, `oxlint-disable` and `@ts-` directive form and points the reader at the record. Clause [4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) holds: phases 0 to 3 are detectors, phase 4 is the test runner, and a run that fails at phase 1 runs nothing after it. Clause [8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source) is the continuous integration workflow, which runs the toolchain on the package itself and fails on anything it cannot fix. The declaration under [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) is the `defenceBeforeFix` key in `package.json`, carrying both levels of clause [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) with their gaps recorded, so neither level claims conformance. ## How it is not conformant Three causes, each verified through the entry point. -**The dependency-cruiser lane is outside the three commands.** The shipped `no-circular` defence prints its identifier with a two-file cycle and fails the run, but `ts-qa rule-doc no-circular` exits 2 as unknown, `ts-qa rules` does not list it, the lane ignores `-p`, and the release self-audit walks plugin rules only. Through the wrapping, dependency-cruiser fails detector clauses [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) and [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation), which fails toolchain [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), and the toolchain fails [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) and [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) on the same fact. `ts-qa rule no-circular src` compounds it by answering "did not fire" with exit 0 for an identifier the harness cannot see. +**The dependency-cruiser lane is outside the three commands.** The shipped `no-circular` defence prints its identifier with a two-file cycle and fails the run, but `ts-qa rule-doc no-circular` exits 2 as unknown, `ts-qa rules` does not list it, the lane ignores `-p`, and the release self-audit walks plugin rules only. Through the wrapping, dependency-cruiser fails detector clauses [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) and [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation), which fails toolchain [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), and the toolchain fails [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access), [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) and [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) on the same fact. `ts-qa rule no-circular src` compounds it by answering "did not fire" with exit 0 for an identifier the harness cannot see. -**ESLint's bulk suppressions file is an open route.** A file containing the literal `PLACEHOLDER` fails the lint lane. With an `eslint-suppressions.json` in the project root naming that file and `ts-qa/no-placeholder`, the same run passes with every phase clean and the project record unchanged. Nothing disables the file or fails on its presence, so toolchain [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) fails, though every inline form is banned. +**ESLint's bulk suppressions file is an open route.** A file containing the literal `PLACEHOLDER` fails the lint lane. With an `eslint-suppressions.json` in the project root naming that file and `ts-qa/no-placeholder`, the same run passes with every phase clean and the project record unchanged. Nothing disables the file or fails on its presence, so toolchain [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) fails, though every inline form is banned. -**Resolution of a core rule stops at a URL.** The always-on as/enum ban is routed through ESLint's `no-restricted-syntax`, and `ts-qa rule-doc no-restricted-syntax` resolves to a one-line description and eslint.org, not to the toolchain's own shipped page for the ban. Detector [6.2](../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) and [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) fail for that identifier and toolchain [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) with them. +**Resolution of a core rule stops at a URL.** The always-on as/enum ban is routed through ESLint's `no-restricted-syntax`, and `ts-qa rule-doc no-restricted-syntax` resolves to a one-line description and eslint.org, not to the toolchain's own shipped page for the ban. Detector [6.2](../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) and [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) fail for that identifier and toolchain [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) with them. The derived listing also exposed a defect: `ts-qa/no-naive-datetime-template` is declared Tier A in the source and the documentation and is absent from the shipped generic configuration, so it is neither listed nor enforced anywhere, which fails [8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source) as a shipped rule inactive on the package itself. At project level, two of the bundled rules, `no-duplicate-section-ids` and `no-placeholder`, have no fixture test, so the red proof method clause [3.3](../SPEC.md#33-prove-the-net-by-making-the-rule-fire) requires is not in the repository for them. Two SHOULDs remain open from the earlier audit: detector [4.4](../DETECTOR-SPEC.md#44-the-detector-should-enforce-43-with-a-rule-of-its-own), a rule over the rules, and toolchain [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically), since `deploy-skills` delivers static files rather than the generated listing. ## Clause by clause -| Document | Clause | Result | Evidence | -| --------- | ------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | -| Detector | [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | `tsQaConfig/eslint.config.js` merged into the resolved config; a project dependency-cruiser config replaces the shipped one | -| Detector | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Partial | `ts-qa rule ts-qa/no-eslint-disable src/probe.ts` exited 1 with the location and a silent rule exited 0; no route for a dependency-cruiser rule | -| Detector | [4.3](../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `ts-qa/` chosen in the plugin map and printed unaltered; `no-circular` printed with its cycle | -| Detector | [4.4](../DETECTOR-SPEC.md#44-the-detector-should-enforce-43-with-a-rule-of-its-own) | No | No rule over the rules | -| Detector | [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | `npx ts-qa`, no infrastructure | -| Detector | [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Partial | `-p ` and the single-path `rule` command for the ESLint lane; the dependency-cruiser lane ignores `-p` | -| Detector | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | `--llm` verdict table names the failing tool and the path to the full result | -| Detector | [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | CI runs the same command | -| Detector | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Partial | `rule-doc ts-qa/no-eslint-disable` printed the shipped section; `rule-doc no-circular` exited 2 as unknown | -| Detector | [6.2](../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Partial | `docs/` ships and the printed route is a local path for plugin rules; a core rule resolves to eslint.org only | -| Detector | [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | Partial | Every plugin rule has a section in `docs/cdd-rules.md`, pinned by a test; no page for `no-circular`, none shipped for core rules | -| Detector | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | Directives detected by `ts-qa/no-eslint-disable`; the known-violations baseline needs a flag the toolchain never passes | -| Detector | [7.2](../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) | No | A directive's reason is optional and a baseline entry carries none | -| Toolchain | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | dependency-cruiser plus the wrapping fails detector [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) and [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) and a defence is routed through it | -| Toolchain | [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | `no-circular` unresolvable; `no-restricted-syntax` resolves to an upstream URL, not the shipped as/enum ban page | -| Toolchain | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | No | A `PLACEHOLDER` finding failed the lint lane; with `eslint-suppressions.json` naming it the run passed, all phases clean | -| Toolchain | [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | No | Entry point meets detector [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) for the ESLint lane; no subset for the dependency-cruiser lane | -| Toolchain | [4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | Detectors in phases 0 to 3, tests in 4; a failure at phase 1 ran nothing after it | -| Toolchain | [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | No | `ts-qa rules` printed eighteen defences with identifier, description and route; `no-circular` absent | -| Toolchain | [5.2](../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | The listing walks the resolved ESLint configuration, and showed a declared Tier A rule that the configuration never enables | -| Toolchain | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | No | The repository's own plugin rule is listed; its dependency-cruiser defence is not | -| Toolchain | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes | `tsQaConfig/tier-a-exemptions.json`, loaded by the toolchain on every run | -| Toolchain | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Yes | Missing, "legacy" and "needed for now" justifications each rejected with the entry named | -| Toolchain | [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | The project record is printed by `ts-qa rules` beneath the defences | -| Toolchain | [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project) | Yes | The always-on tier and its reasoning are documented in `docs/cdd-rules.md` and `docs/coding-standards.md` | -| Toolchain | [7.1](../TOOLING-SPEC.md#71-the-toolchain-should-generate-a-summary-of-the-active-defences-suitable-for-an-agents-context) | Partial | One line per defence with identifier and route, phrased as a description | -| Toolchain | [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically) | Partial | `deploy-skills` delivers static skills, not the generated summary | -| Toolchain | [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | No | The self-audit test walks plugin rules only; `no-circular` is outside it | -| Toolchain | [8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source) | No | CI runs the toolchain on itself, but `ts-qa/no-naive-datetime-template` ships and is active nowhere | -| Toolchain | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | Yes | `package.json` `defenceBeforeFix`: method 1.0.0, toolchain 0.2.0 at both levels, with the gaps above in `knownGaps` | -| Method 7 | [3.3](../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Partial | Twenty-five bundled rules have a fixture test; `no-duplicate-section-ids` and `no-placeholder` have none | -| Method 7 | record | Yes | Two justified, scoped exemptions, printed on every run and by `rules` | +| Document | Clause | Result | Evidence | +| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Detector | [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | `tsQaConfig/eslint.config.js` merged into the resolved config; a project dependency-cruiser config replaces the shipped one | +| Detector | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Partial | `ts-qa rule ts-qa/no-eslint-disable src/probe.ts` exited 1 with the location and a silent rule exited 0; no route for a dependency-cruiser rule | +| Detector | [4.3](../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `ts-qa/` chosen in the plugin map and printed unaltered; `no-circular` printed with its cycle | +| Detector | [4.4](../DETECTOR-SPEC.md#44-the-detector-should-enforce-43-with-a-rule-of-its-own) | No | No rule over the rules | +| Detector | [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | `npx ts-qa`, no infrastructure | +| Detector | [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Partial | `-p ` and the single-path `rule` command for the ESLint lane; the dependency-cruiser lane ignores `-p` | +| Detector | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | `--llm` verdict table names the failing tool and the path to the full result | +| Detector | [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | CI runs the same command | +| Detector | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Partial | `rule-doc ts-qa/no-eslint-disable` printed the shipped section; `rule-doc no-circular` exited 2 as unknown | +| Detector | [6.2](../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Partial | `docs/` ships and the printed route is a local path for plugin rules; a core rule resolves to eslint.org only | +| Detector | [6.3](../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | Partial | Every plugin rule has a section in `docs/cdd-rules.md`, pinned by a test; no page for `no-circular`, none shipped for core rules | +| Detector | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | Directives detected by `ts-qa/no-eslint-disable`; the known-violations baseline needs a flag the toolchain never passes | +| Detector | [7.2](../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) | No | A directive's reason is optional and a baseline entry carries none | +| Toolchain | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | dependency-cruiser plus the wrapping fails detector [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), [5.2](../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) and [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) and a defence is routed through it | +| Toolchain | [4.2](../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-blocking-rule-it-routes-can-print-from-the-installed-copy-without-network-access) | No | `no-circular` unresolvable; `no-restricted-syntax` resolves to an upstream URL, not the shipped as/enum ban page | +| Toolchain | [4.3](../TOOLING-SPEC.md#43-the-toolchain-must-forbid-every-suppression-route-that-bypasses-the-project-record-by-disabling-it-or-by-a-defence-of-its-own) | No | A `PLACEHOLDER` finding failed the lint lane; with `eslint-suppressions.json` naming it the run passed, all phases clean | +| Toolchain | [4.4](../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | No | Entry point meets detector [5.1](../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to [5.4](../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) for the ESLint lane; no subset for the dependency-cruiser lane | +| Toolchain | [4.5](../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | Detectors in phases 0 to 3, tests in 4; a failure at phase 1 ran nothing after it | +| Toolchain | [5.1](../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | No | `ts-qa rules` printed eighteen defences with identifier, description and route; `no-circular` absent | +| Toolchain | [5.2](../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | The listing walks the resolved ESLint configuration, and showed a declared Tier A rule that the configuration never enables | +| Toolchain | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | No | The repository's own plugin rule is listed; its dependency-cruiser defence is not | +| Toolchain | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes | `tsQaConfig/tier-a-exemptions.json`, loaded by the toolchain on every run | +| Toolchain | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Yes | Missing, "legacy" and "needed for now" justifications each rejected with the entry named | +| Toolchain | [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | The project record is printed by `ts-qa rules` beneath the defences | +| Toolchain | [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project) | Yes | The always-on tier and its reasoning are documented in `docs/cdd-rules.md` and `docs/coding-standards.md` | +| Toolchain | [7.1](../TOOLING-SPEC.md#71-the-toolchain-should-generate-a-summary-of-the-active-defences-suitable-for-an-agents-context) | Partial | One line per defence with identifier and route, phrased as a description | +| Toolchain | [7.2](../TOOLING-SPEC.md#72-the-toolchain-should-deliver-that-summary-into-the-project-automatically) | Partial | `deploy-skills` delivers static skills, not the generated summary | +| Toolchain | [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | No | The self-audit test walks plugin rules only; `no-circular` is outside it | +| Toolchain | [8.2](../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source) | No | CI runs the toolchain on itself, but `ts-qa/no-naive-datetime-template` ships and is active nowhere | +| Toolchain | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | Yes | `package.json` `defenceBeforeFix`: method 1.0.0, toolchain 0.2.0 at both levels, with the gaps above in `knownGaps` | +| Method 7 | [3.3](../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Partial | Twenty-five bundled rules have a fixture test; `no-duplicate-section-ids` and `no-placeholder` have none | +| Method 7 | record | Yes | Two justified, scoped exemptions, printed on every run and by `rules` | ## Notes for a practitioner