diff --git a/ACCEPTANCE.md b/ACCEPTANCE.md index 31c8df0..77eac8f 100644 --- a/ACCEPTANCE.md +++ b/ACCEPTANCE.md @@ -6,49 +6,70 @@ document that only a strong reader can follow is not a specification, and the re matter most, coding agents meeting the method in a failing pipeline, are often the weakest. The test is run with a low-strength model so that clarity is measured at the floor, not the ceiling. +The test proves one thing: that a capable reader, reading the document cold and once, takes from +it what the document exists to say, and is not confused by the text the change put there. The +keyed questions prove the document; the confusion question proves the change. Everything below +serves that, and a rule that stops it proving that is a defect in this file, fixed as +[Changes to this process](#changes-to-this-process) describes. + ## The cohort -- **At least five independent readers**, each a fresh instance of a low-strength model (the - smallest current model in its family, for example Haiku), with no shared context and no - access to the change history, the critique findings or each other. -- Each reader receives **only the changed document or documents** and the fixed question set - below. Nothing else is read first. +- **An odd number of independent readers, at least five**, each a fresh instance of a + low-strength model (the smallest current model in its family, for example Haiku), with no + shared context and no access to the change history, the critique findings or each other. + +- **Each reader is a separate process with no tools and no project context, only its packet**: + started from an empty directory outside any repository, with no user, project or local settings + files, no `CLAUDE.md`, skills, plugins, hooks or MCP servers, and no saved session, so that no + project instruction reaches it. The model tool's own fixed system prompt is not project + context. For example, with the packet on standard input: + + ``` + claude -p --safe-mode --model haiku --tools "" --setting-sources "" --strict-mcp-config --no-session-persistence --output-format json < packet.md + ``` + + A reader is set aside, and its answers do not count, when its answer text is empty, when its + process reports an error, or when the model version it reports is not the cohort's model, the + one version the editor names for the run before its readers start. A run + with a reader set aside is void: it takes a run number and has a record, and a fresh cohort + runs. No reader is set aside for any other cause. + +- Each reader receives **only its packet**: the document's question set, the changed document + and the fixture the questions name, each in full in the reader's prompt, the question set + first. Each file is preceded by exactly one line, `===== FILE: =====`, where `` is + the name the question set gives the file (so `fixtures/remediation-ledger.md`, and + `QUESTIONS.md` for the question set itself). The question set carries the instruction to the + reader, and nothing else is added. Every reader in a run receives the same packet. + - Readers are run **after** every stronger review has been applied. They are the last gate, not the first draft. ## The question set -Written before the run, with an **answer key** for every keyed question drawn verbatim from the -document, so that marking is mechanical. Each reader answers all of them. +Written before the run, with an **answer key** for every keyed question, so that marking is +mechanical. The key states, for each keyed item, exactly what an answer must contain and what +makes it wrong; the question asks for exactly that and nothing the key does not expect. A key +asks for nothing the question does not. Each reader answers all of them. 1. **Restatement.** In their own words, what the document asks of its subject, in at most six - bullets. Keyed against the list of MUST clauses: a bullet is correct when it names an - obligation the document actually states. - -2. **Load-bearing statements.** Three questions, fixed per document, each asking for one - sentence and a quotation: + bullets. Keyed against the key's list of the document's obligations: a bullet is correct when + it names one of them. - - `SPEC.md`: what a practitioner does before fixing a defect, in order; who may baseline or - suppress; what "conforms" means for a remediation. - - `DETECTOR-SPEC.md`: what "conforms" means and whether a tool that predates the document can - conform; what a detector must do about an inline suppression route; where a bundled rule's - documentation must live. - - `TOOLING-SPEC.md`: where conformance is measured; how a grade is assigned when parts come - from different sources; how many grades a project that ships tooling has. +2. **Load-bearing statements.** Three or more questions, fixed per document and listed in its + question set under `acceptance/`, each asking for a short answer and the sentence or + sentences of the document it rests on, quoted. Each has one or more parts, and the key lists + the element an answer must state for each part. An obligation is load-bearing, and gets a + question, when a reader who misreads it would act against the method; the editor decides, and + the pull request that adds the question says why. 3. **Application.** A fixture case supplied with the questions, graded clause by clause with - Yes, Partial or No and one line of reasoning. Keyed against the intended verdicts; a divergence - counts against the document only when the reader's reasoning shows the clause was read - differently, not when the fixture was misread. + Yes, Partial or No and one line of reasoning, then the verdicts the question set asks for, + each with its reason. Keyed against the accepted grades and answers, with the readings the + key names as ones the text invites counting against the document. 4. **Confusion.** Any passage the reader found confusing on its one reading, quoted with its clause number, and nothing else: not what would make it clearer, not a proposed change. -The cohort settles one thing: is the document clear, coherent and comprehensive. A reader getting -everything right after one reading proves it; a reader getting confused fails it. The document -is designed by its editor to resolve the comprehension failures the cohort shows; the cohort is -the acceptance layer, not the designer. - ## What the cohort is not asked The cohort may point at what it found confusing. It is never asked to rate the document, to say @@ -58,61 +79,197 @@ not there to design the text, and a document shaped by what its weakest reader s a document designed by that reader. The signals this test takes from the cohort are a keyed answer missed, a wrong verdict reasoned from the document, and a passage pointed at; what to do about any of them is the editor's decision. An earlier version of this test asked the reader -what would make a clause clearer and for a clarity rating; both were removed on the author's +what would make a clause clearer and for a clarity rating; both were removed on the editor's instruction for that reason, and neither is reinstated. +## Marking + +The editor marks each answer sheet against the document's key, or delegates the marking to a +separate marker that has seen neither the drafting nor the review. A delegated marker receives +the document, its question set, fixture and key at the commit the readers read, this file, the +answer sheets, and the facts the record needs that the sheets cannot show. Its marks are final +unless the editor shows that a rule here was misapplied, and a re-mark says which rule and why. + +Every mark is given by a rule in this file or in the key, as follows. Where neither settles a +mark, the marker records the point in the run record and marks it as the open-point rule below +says, and the editor settles it in this file or in the key, as +[Changing the test](#changing-the-test) allows, before the next run. A run record states no +judgement that is not written here or in the key; a judgement an earlier run recorded is not a +rule. + +- **Keyed items.** Question 1 as a whole is one keyed item, and so is each item of question 2 + and each answer of question 3 for which the key gives an accepted grade or answer. +- **Reading an answer.** An answer is read for its sense, not its wording: a paraphrase, a + synonym, or a defined term used in its defined sense states what it means. A reader quotes the + document when it gives, in quotation marks or a block quote, a run of six or more consecutive + words of the document in the document's order, verbatim apart from letter case, quotation-mark + or apostrophe style, and markdown link or emphasis markup; a quotation that drops or changes a + word quotes the document by its longest such run. +- **Question 1** is correct when at least four bullets each name an obligation the key lists and + no bullet asserts one the key marks wrong. Only the first six bullets are marked. Several + bullets that name different clauses of one line of the key's list each count; two that name + the same clause count once. A bullet asserting an obligation that is in neither list is not + counted either way, and is recorded as an open point for the key. +- **Question 2.** An answer is the reader's sentences and its quotation read together. It is + correct when it states every element the key lists as required for that item, in the + document's words or the reader's own, asserts nothing the key marks wrong, and quotes the + document. An element stated only in the quotation is stated. Anything in the key beyond the + required elements and the wrong assertions is explanation for the marker, and an answer is + never marked down for leaving it out. +- **Question 3.** Each item is correct when its grade or answer is one the key accepts and, + where the key sets a condition on the reasoning for it, the reasoning meets it. A grade or + answer the key accepts with no condition is correct whatever the reasoning. A grade given on a + stated condition is read as the grade for the case the fixture presents, where the key says + which case that is; otherwise the item is unanswered. +- **A miss that counts against the document.** The key names, for question 1 and for each item + of question 3, the wrong readings the text invites, each as a grade or answer together with + what the reasoning states or leaves out. A miss that matches one blocks whatever the count, + since one reader in five taking that reading means one agent in five acting on it, and it + blocks wherever the text lies, because the keyed questions are what prove the document. Every + other single miss under questions 1 and 3 is the reader's and is tolerated by the threshold + below; a second miss on the same item fails the threshold, and no judgement is needed. +- **An unanswered question.** Each keyed item left unanswered is a miss. Question 4 left + unanswered counts as no passage named. +- **Question 4.** A reader names a passage when it quotes the document, as defined above, or + gives a clause number with a description or a shorter quotation that identifies a sentence or + paragraph. A quotation whose words occur in more than one paragraph, given with no clause + number, names the paragraph where they first occur. A remark that is not about a passage of + the document, such as that another document was not supplied or that a term is defined only + in another document, is recorded as a note and is not a passage, however much of the document + it quotes to locate the reference. Two readers name the same passage when the words or sentences they name lie in + the same paragraph; a heading, a list item, a table row and a terminology entry are each a + paragraph of their own. A passage named by two or more readers of the run is a finding. +- **Where a finding lies.** The record says, for each finding, whether it lies in text the change + wrote or altered: a paragraph in which the pull request, read against its merge base with the + branch it targets, adds, rewords, moves or deletes a sentence, or changes a word, a term or a + link; a paragraph the pull request parts from what it refers to, by inserting a paragraph + between them or moving either; or a paragraph that any reader naming the finding sets against + such a paragraph. A finding there blocks the change. A finding anywhere else is a finding against the document, not + against the change: it is recorded in [acceptance/ROUTED.md](acceptance/ROUTED.md), routed to a + change of its own, and does not block. The change is judged on what it changed; the document + is proved by the keyed questions, which every run answers over the whole of it. +- **Open points.** An open point in marking an answer is marked for the reader, except that a + question 1 bullet on neither list is not counted, as above. An open point on whether a reader + named a passage, or whether two readers named the same one, counts the passage. An open point + on where a finding lies treats it as the change's own text. + ## Pass criteria -The change passes when all of the following hold on a single run: +The change passes when all of the following hold on a single run. The thresholds apply to the +readers whose answers count; with five readers, "every reader" is five of five and "all but one" +is four of five. -| Criterion | Threshold | -| ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | -| Load-bearing statements (question 2) | **5 of 5** readers correct on every one | -| Restatement and application (questions 1 and 3) | **At least 4 of 5** readers correct on each keyed item | -| A wrong answer traceable to the text | **None.** A miss whose reasoning quotes the document is a finding, whatever the count | -| A passage found confusing (question 4), by two or more | **None outstanding.** It is a finding and blocks until the text is changed and a fresh cohort no longer names it | -| Anything from one reader only | Recorded; applied at the editor's discretion; does not block | +| Criterion | Threshold | +| ------------------------------------------------------------------------------------- | -------------------------------------------------------------- | +| Load-bearing statements (question 2) | **Every** reader correct on every item | +| Restatement and application (questions 1 and 3) | **All but one** reader correct on each keyed item | +| A miss the key says counts against the document | **None**, whatever the count | +| A passage found confusing by two or more readers, in text the change wrote or altered | **None.** It is a finding and blocks until the text is changed | +| A passage found confusing by two or more readers, elsewhere | Recorded in `acceptance/ROUTED.md` and routed; does not block | +| Anything else from one reader only | Recorded; applied at the editor's discretion; does not block | The asymmetry is deliberate. A load-bearing statement is the sentence the whole document exists to make, and one reader in five getting it wrong means one agent in five will act on the wrong reading; unanimity is the only acceptable result there. Everywhere else a single divergent reader -is more likely to be the reader than the text, so four of five is tolerated, but a miss that the -reader can justify from the document is always the document's fault. +is more likely to be the reader than the text, so one such reader is tolerated, unless the key +says the text invites that reading. A finding is settled run by run: a passage two readers named +in one run has no standing in the next, which is judged on its own readers, and a passage the +fresh cohort names by fewer than two is clear. + +## Changing the test + +Nothing in this test is changed to let a particular run through. A threshold, a question, a +key, a fixture or a rule here is changed only before a run's readers start, never changes a mark +already given, and is changed by one of two routes, each recorded with its reason: + +- In the specification pull request that changes the text, under step 4 of `CONTRIBUTING.md`: + a key's quotation follows the text it quotes; a key gains an item, with the question that asks + for it, for a new load-bearing obligation; and where the change alters or removes the + obligation an existing item tests, the item and its question follow the text, since that is + the text changing and not the test. `acceptance/README.md` says what that may and may not do. +- By a change to this process, with its own pull request: a threshold, a fixture, a rule here, + what an existing key item requires of an unchanged obligation, or the wording of a question. -## The bar only moves up +An open point a run recorded is settled by whichever route it needs, before the next run of the +pull request whose run recorded it; a run of another pull request is not held up by it. -No criterion in this document is ever relaxed to let a change through. If the cohort is confused, -the test is working and the text is wrong; the fix is to the text, never to the threshold, the -question, the fixture or the key. A threshold may be raised, a keyed question may be added, and a -key may be corrected where it misquoted the document; none of those makes a failing run pass. The -one time a threshold was relaxed, after three runs, the relaxation was reversed on the author's -instruction and is recorded here so that it is not tried again. Removing the clarity rating and -the request for suggestions was not a relaxation: it changed what the cohort is asked, not how -strictly an answer is marked. +A question is reworded only so that it asks for what its key requires, and a key's required +elements are never more than its question asks for. A run made after such a change is a fresh +run under the new rules, not a re-mark of an old one. The test for a change to a rule is what +the rule measured: a rule that measured something other than whether a cold reader could follow +the text is a defect in this file and is corrected, and the correction says what the rule +measured instead; a rule that measured that, and found the text wanting, is not touched, and the +text is fixed. The one time a threshold was relaxed to let a change through, after three runs, +the relaxation was reversed on the editor's instruction and is recorded here so that it is not +tried again. Removing the clarity rating and the request for suggestions was not that: it +changed what the cohort is asked, not how strictly an answer is marked. ## After a failure -Fix the text, never the questions, and run a fresh cohort. Readers are not reused across runs. -The number of runs it took is recorded; a document that needed three runs is telling its editor -something. +Fix the text the run's misses and findings name and run a fresh cohort; readers are never reused. +A failed keyed item is fixed in the text that carries the obligation, wherever it lies, since the +keyed questions prove the document as a whole, whoever wrote the sentence; where that fix would +change an obligation, it is routed to a change of its own, and this change waits for it. A +finding on text the change wrote or altered is fixed there. A finding elsewhere goes to +`acceptance/ROUTED.md` and is not fixed in this change. The number of runs it took is recorded; +a document that needed three runs is telling its editor something. ## When it runs -Once per pull request, on the final text, after the editor's rewrite and any key update. It is -the last gate before the merge to `next`, so that every change on `next` is one the cohort has -accepted, and the release branch needs no cohort of its own unless it changes specification text. -A wording change after the run means a fresh run; readers are never reused. +On the final text of each pull request, after the editor's rewrite and any key update, one cohort +per changed document. It is the last gate before the merge to `next`, so that every change on +`next` is one the cohort has accepted, and the release branch needs no cohort of its own unless it +changes specification text. A fresh cohort runs after every change to the changed document's +text that follows a run, and after every void run; a text that failed is never run again +unchanged, and a commit that adds the run record or the changelog line changes no document. A version number or date that `PUBLISHING.md` or step 5 of `CONTRIBUTING.md` has the +editor set is not such a change, and neither is a merge of `next` into the branch that brings in +text another cohort accepted and alters no sentence this change wrote. ## The record -Each run has a file under `acceptance/runs/`, named `NNN-.md` with `NNN` the next number in -the one sequence shared by the three documents, stating: the document, the pull request and the -commit read, the cohort size and model, the run number in that pull request, the marks per -reader per question, the findings applied, and whether it passed. The changelog line for the -change names the file. CI refuses a specification change whose pull request adds no run file; -it checks for at least one, and a pull request that changes two documents adds a run for each. - -Each published version's changelog entry carries the lines of the changes it contains, so a -version's cohort record is the set of runs those lines name. A version with no such line has not -been accepted and MUST NOT be described as published. The question sets and answer keys live -under `acceptance/` in this repository so that a run can be repeated and compared. +Each run has a file under `acceptance/runs/`, named `NNN--run-.md` with `NNN` the next +number in the one sequence shared by the three documents and `` the run number within the pull +request, stating: the document; the pull request and the commit read; the cohort size and model, +with the model version each reader reported; how the readers were isolated and the command they +were started with; any reader set aside and why; who marked and, for a delegated marker, how it +was started; the marks per reader per keyed item, with each miss and the reader's reasoning in +brief; every passage named under question 4, with the words quoted or the description given, its +clause and the readers naming it; each finding and whether it lies in text the change wrote or +altered; any point the rules did not settle; whether the run was made under a process change +before it merged and, if so, at which commit; the findings applied afterwards; the verdict; and +a link to the answer sheets. + +The answer sheets are not kept in this repository. Every reader's whole output as its process +returned it, the exact prompt every reader received, the command the readers were started with, +and the instruction a delegated marker was given are posted in one or more comments on the pull +request, linked from the record and never edited, hidden or deleted after posting. + +The changelog line for the change names every run made on it, the passing run last. CI refuses a +specification change whose pull request adds no run file; it checks for at least one, and a pull +request that changes two documents adds a run for each. Each published version's changelog entry +carries the lines of the changes it contains, so a version's cohort record is the set of runs +those lines name. A version with no such line has not been accepted and MUST NOT be described as +published. The question sets and answer keys live under `acceptance/` in this repository so that +a run can be repeated and compared. + +`acceptance/ROUTED.md` lists every finding routed from a run, open until a change that fixes +its passage passes a cohort, or until the editor closes it with a recorded reason. The editor +reads that list before cutting a release. + +## Changes to this process + +The process files are this file, `acceptance/README.md`, `acceptance/runs/README.md`, +`CONTRIBUTING.md` and the checks that enforce them: `tools/changes.py`, the key-quotation checks +in `spec-qa.py`, the CI workflow and the pull request template. A change to any of them is a +change to this process, and so is a change to a threshold, a fixture, the wording of a question +or what a key item requires, as [Changing the test](#changing-the-test) says. They are not +specifications, and a change to them is not tested by a cohort: the cohort measures whether a +low-strength reader can follow a specification, and the process is followed by the editor. Such +a change names the runs or the case that showed the need for it, and goes through steps 1, 3 and +6 of `CONTRIBUTING.md`, with no scope, key or cohort step; its review reads every process file it +changes. It applies to every run whose readers start after it merges; a run made before then may +be made under it, following all of it at a commit named in the record before the readers start. +It is recorded by a line added at the top of [acceptance/CHANGES.md](acceptance/CHANGES.md), +naming its pull request, not in the changelog, which is for the three specifications. +`acceptance/ROUTED.md` and `acceptance/runs/` are records, not process files: a specification +pull request writes to them. diff --git a/CHANGELOG.md b/CHANGELOG.md index f5f8122..a5bd530 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,37 @@ version entry with no cohort line has not been accepted and is not published. ## Method specification (SPEC.md) +### Unreleased + +- New obligation, clause [3.2](SPEC.md#32-build-the-net): a rule the practitioner writes MUST be written in the strongest + kind of detector that can express it, semantic, else syntactic, else textual, judged by the + configuration it runs in; enabling an existing rule or tightening its thresholds is not writing + one, and giving an existing rule a pattern of the project's own, or changing the pattern of one + the project wrote, is. A weaker kind is allowed only on two grounds, that the stronger cannot + express the rule or that the language has no detector of that kind which accepts bespoke + rules; a stronger configuration of a detector the project runs counts, and one only a hosted + service or licence tier provides does not. The remediation docs of a rule the practitioner + writes MUST name its detector and kind, and where the kind is weaker than semantic the grounds + MUST be recorded there and in the description of the commit that adds or moves the rule or + first records them, naming the stronger detectors considered. The practitioner judges the + grounds; a project MAY require the owner's sign-off by recording that requirement where it + records its calibrations, which section [4](SPEC.md#4-authority-which-decisions-belong-to-whom) and the section [3](SPEC.md#3-the-method) map note. Clause [3.6](SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation)'s + list of what the remediation docs state defers to what clause [3.2](SPEC.md#32-build-the-net) adds. Section + [1](SPEC.md#1-terminology) defines textual, syntactic and semantic by whether a rule can ask the type of any + expression and what any name refers to across the whole program, naming mypy as semantic only + with `check_untyped_defs`. Section [2](SPEC.md#2-when-the-method-applies) and clause [3.2](SPEC.md#32-build-the-net) send a class that no extensible + detector of a kind the language has can express within clause [3.1](SPEC.md#31-attribute-the-defect-to-a-class)'s bounds to a bespoke + detector or a toolchain gap, not to "no pattern exists". Section [7](SPEC.md#7-conformance)'s conforming defence + carries the kind, and Appendix A carries the requirement. Prompted by project rules hosted in + ast-grep, where the language already had a semantic host, that decided by names: they reported + unrelated calls of the same name, and clause [3.2](SPEC.md#32-build-the-net) said any detector qualified whatever its + category. The method question set gains load-bearing statement 2d. Runs 19, 21, 23, 25, 27 + and 29 were made on an earlier text of this change under the earlier acceptance rules: runs + 19 and 29 failed on statement 2d, and every run failed on passages the change had not + touched, as `acceptance/CHANGES.md` records; the change was then reduced to the text above, + and the passages those runs named in untouched text are listed in `acceptance/ROUTED.md` for + changes of their own. Accepted by [run 70](acceptance/runs/070-semantic-detectors-run-13.md). + ### 1.1.0, 2026-10-02 - New obligation, section [2](SPEC.md#2-when-the-method-applies): a defect whose fix is deferred MUST be recorded where the project's @@ -68,6 +99,66 @@ pass it. ## Detector specification (DETECTOR-SPEC.md) +### Unreleased + +- New obligation, clause [8.2](DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades): a verdict on a detector MUST name the kind, semantic, syntactic + or textual, of each configuration it grades. It is not a level of conformance and does not change the verdict; it is + stated because method clause [3.2](SPEC.md#32-build-the-net) now requires a rule to be written in the strongest kind + of detector that can express it, and a verdict of conforming alone presents a textual detector + as an equal host. Section [9](DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) still adds no obligation to the method a practitioner + follows and names this one as binding whoever gives a verdict on a detector, a practitioner + included when they give one, and section [2](DETECTOR-SPEC.md#2-terminology)'s gloss carries the three terms and the + message. Runs 20, 22, 24, 26, 28 and 30 were made on an earlier text of this change under the + earlier acceptance rules and failed on passages the change had not touched, as + `acceptance/CHANGES.md` records; the passages those runs named in untouched text are listed + in `acceptance/ROUTED.md` for changes of their own. [Run + 71](acceptance/runs/071-semantic-detectors-run-14.md), under the rules of pull request 22, + failed on this clause's first paragraph, which two readers could not place, and on misses in + clauses [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) and [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) in text the change had not touched; the paragraph then gained a sentence + naming the configuration whose kind is named, and [run + 74](acceptance/runs/074-semantic-detectors-run-15.md) failed on that sentence, three readers + unable to tell which configuration it meant. The sentence is withdrawn: the verdict names the + kind of each configuration it grades. Accepted by [run + 79](acceptance/runs/079-semantic-detectors-run-16.md). +- Clarity, no obligation changed: four passages that cold readers of an unrelated change read as + the document pulling against itself or could not follow, each listed in `acceptance/ROUTED.md`. + Clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)'s detectable route is now a bespoke rule the project writes for the detector under + clause [4.1](DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it), after six runs read "a Rule the project can write in the Detector itself" as the + project editing the detector's own source, and a new paragraph says what detectable means and + what each of the two ways is. Clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps)'s duty to record a known gap falls only on a detector + that declares a version, and says so beside "The declaration is optional", with the method's + clause [3.2](SPEC.md#32-build-the-net) record applying either way. Clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) says what the remainder is, and that + there is none where the output prints every finding. Clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) says it asks for the + mechanism and not for every page, a missing page failing clause [6.3](DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), and that a + documentation path printed in the message is not enough on its own, after three readers of + run 54 set that sentence against the mechanism. Prompted by runs 32 to 54 on the defined-terms + change, in which two or more readers quoted each passage. + [Run 60](acceptance/runs/060-detector-route-rule-run-1.md) failed on two findings in the + revised text, every keyed item within threshold: two readers could not find Message defined, + so section [2](DETECTOR-SPEC.md#2-terminology)'s table gains a Message row; and two readers stopped at clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps)'s + "anyone who exercises the clauses above", which now says anyone may grade a detector by + checking it against each clause of sections [4](DETECTOR-SPEC.md#4-authoring-rules) to [7](DETECTOR-SPEC.md#7-suppression). + [Run 63](acceptance/runs/063-detector-route-rule-run-2.md) failed on one finding in the revised + text, every keyed item five of five: two readers needed two readings of clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran)'s + sentence on a report that only adds detail, which is now a sentence of its own with its subject + stated; three readers named clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation)'s paragraph on a project's own rule, which this + change does not touch and `acceptance/ROUTED.md` now lists. + [Run 66](acceptance/runs/066-detector-route-rule-run-3.md) passed, every keyed item five of + five, its one finding being that same clause [6.1](DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) paragraph, named again by three readers, + which stays in `acceptance/ROUTED.md` for a change of its own. Run 69, a cohort of the + defined-terms change on this document with this change merged in, then had two readers set + clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran)'s "MUST carry both a usable summary and the location of the remainder" against its + "need not name it", and two readers stop at clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps)'s gap sentence and its MAY: clause + [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) now owes the location only where the summary leaves a finding out, and clause [8.1](DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) says + where a practitioner's record of a gap lives and that a detector without a declaration can + still be found to conform. [Run 73](acceptance/runs/073-detector-route-rule-run-4.md) failed + on clause [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran)'s "usable summary" and on clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)'s second paragraph; clause + [5.3](DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) now says what a usable summary is and what the remainder is, and clause [7.1](DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) states + the two ways a route is detectable in one paragraph. [Run + 77](acceptance/runs/077-detector-route-rule-run-5.md) failed on the remainder sentence alone, + four readers unable to tell what "is not one" referred to; it is now three plain sentences. + Accepted by [run 78](acceptance/runs/078-detector-route-rule-run-6.md). + ### 1.0.0, 2026-09-08 First version, split out of toolchain specification 0.1.0 on the same day. States what a detector, diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fa1743e..ea5a525 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,7 +29,9 @@ review. It is a patch version when released. SHOULD or MAY in a numbered section or appendix. It needs everything a clarity edit needs, plus the five tests in SCOPE.md answered in the pull request, and it is at least a minor version when released. `tools/changes.py` tells the two apart mechanically and CI runs it on every pull -request. +request. A change to the acceptance process, as ACCEPTANCE.md defines one under "Changes to +this process", is neither: ACCEPTANCE.md says how it is made, and it is recorded in +`acceptance/CHANGES.md`, not in CHANGELOG.md. ## What a pull request contains @@ -55,20 +57,26 @@ survives, the change is yours. the editor sets the pre-release header. 2. **Scope.** For an obligation, the editor confirms each SCOPE.md test. A failed test closes the pull request with a row in `DECLINED.md`. -3. **Review and rewrite.** An independent review of the diff against the document, then the - editor's rewrite, pushed to the pull request branch where the contributor allows it or to an - editor's branch that supersedes it. +3. **Review and rewrite.** An independent review of the diff against the whole document, by a + reviewer who did not draft the change, then the editor's rewrite, pushed to the pull request + branch where the contributor allows it or to an editor's branch that supersedes it. The pull + request names the reviewer and links or contains the review. 4. **Keys.** Where the change touches a sentence an acceptance key quotes, or adds a load-bearing - obligation, the editor updates the key before the cohort runs, never after. -5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text. Its record is - committed under `acceptance/runs/`, the `Unreleased` changelog line names it, and CI refuses - a specification change that lacks one. For an obligation, the editor sets the document's - header to `-dev, unpublished` in the same commit if it still names a published - version. A wording change after the run means a fresh run. + obligation, the editor updates the key before the cohort runs, never after. The key's + required elements stay what its question asks for; `acceptance/README.md` says what a key + update may and may not do. +5. **The cohort.** The acceptance test in ACCEPTANCE.md runs on the final text, one cohort per + changed document. Its record is committed under `acceptance/runs/`, the `Unreleased` + changelog line names it, and CI refuses a specification change that lacks one. For an + obligation, the editor sets the document's header to `-dev, unpublished` in the + same commit if it still names a published version. A change to the document's text after the + run means a fresh run, as ACCEPTANCE.md says under "When it runs". A finding the + run routes to `acceptance/ROUTED.md` is not fixed in this pull request. 6. **Merge** into `next` by merge commit, and the branch is deleted. -Steps 2 to 5 are done by a person with a Claude Code session, not by CI, and they happen in -batches. A quiet pull request is not a declined one; a declined one says so. +Steps 2 to 5 are the editor's, not CI's, and they happen in batches. The editor may have an +agent draft, review or mark, as ACCEPTANCE.md allows for marking; every decision in these steps +stays the editor's. A quiet pull request is not a declined one; a declined one says so. ## Versions and releases diff --git a/DETECTOR-SPEC.md b/DETECTOR-SPEC.md index cdc6c9e..37b1582 100644 --- a/DETECTOR-SPEC.md +++ b/DETECTOR-SPEC.md @@ -1,6 +1,6 @@ # Defence Before Fix: Detector Specification -**Version**: 1.0.0, published 2026-09-08 +**Version**: 1.1.0-dev, unpublished **Companion to**: [the method specification](SPEC.md), version 1.1.0, and [the toolchain specification](TOOLING-SPEC.md), version 0.2.0 **Author**: [Joseph Edmonds](https://ltscommerce.dev), [Edmonds Commerce](https://edmondscommerce.co.uk) **Coined**: 22 February 2026, in [the original article](https://ltscommerce.dev/articles/defence-before-fix-static-analysis) @@ -35,15 +35,17 @@ construct the means has moved the obligation rather than met it. Terms from the method specification carry over unchanged. Every capitalised term links to its definition; the ones this document leans on most are, in short: -| Term | In one line | -| -------------- | ------------------------------------------------------------------------------ | -| [Detector] | A tool that reads code without executing it and reports what it finds | -| [Rule] | One check a [Detector] evaluates | -| [Identifier] | The stable name printed with a finding that leads to its documentation | -| [Practitioner] | Whoever is doing the work, a person or an [Agent] | -| [Toolchain] | Everything a project assembles to run its checks through, [Detectors] included | -| [Suppression] | Making a finding go away without fixing it | -| [Baseline] | A recorded set of existing findings a [Rule] is told to ignore | +| Term | In one line | +| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| [Detector] | A tool that reads code without executing it and reports what it finds | +| [Semantic], [Syntactic], [Textual] | What a [Rule] can ask in a configuration: the type of any expression and what any name refers to, program-wide; a parse tree short of that; or text | +| [Rule] | One check a [Detector] evaluates | +| [Message] | The text a [Rule] prints when it fires | +| [Identifier] | The stable name printed with a finding that leads to its documentation | +| [Practitioner] | Whoever is doing the work, a person or an [Agent] | +| [Toolchain] | Everything a project assembles to run its checks through, [Detectors] included | +| [Suppression] | Making a finding go away without fixing it | +| [Baseline] | A recorded set of existing findings a [Rule] is told to ignore | These are additional. @@ -157,7 +159,11 @@ any. ### 5.3 The result MUST reach the practitioner in the output of the command they ran Where the [Detector] writes fuller detail elsewhere, the invoked command's own output MUST -carry both a usable summary and the location of the remainder. +carry a usable summary and, where the summary leaves any finding out, the location of the +remainder. A summary is usable when each finding in it carries where it is and its +[Identifier]. The remainder is the findings the summary leaves out. Where the output prints +every finding, there is no remainder. A fuller report written elsewhere, such as a page with a +code snippet for each finding, is not a remainder, and the output need not name it. **Why**: method specification clause 8.2. A report the [Practitioner] has to go and find is a report that arrives after the decision it was meant to inform. @@ -183,13 +189,16 @@ Two cases, by who wrote the [Rule]: | The project's own | The project | Print the [Identifier] unaltered; the lookup is the [Toolchain]'s to give | In both, the mechanism is keyed on **the [Identifier] exactly as printed**. A command, an index -file or a URL are all acceptable forms. Each case in turn: +file or a URL are all acceptable forms. This clause asks for the mechanism, not for every page: +where the mechanism works but a [Bundled rule]'s [Identifier] has no documentation to resolve +to, that fails clause 6.3, not this clause. Each case in turn: **For a [Bundled rule]**, the [Detector] supplies the documentation and the lookup, and clauses 6.2 and 6.3 say where. The mechanism MUST resolve an [Identifier] presented alone, because the reader who needs it most has the [Identifier] from a log, a -ticket or a colleague and not the [Message]. A [Message] that carries its own -documentation path resolves that [Message], not the [Identifier]. +ticket or a colleague and not the [Message]. A documentation path printed in the [Message] is +therefore not enough on its own: it helps only a reader who has the [Message], and the +[Identifier] alone still leads nowhere. **For a project's own [Rule]**, the [Detector] cannot know the documentation, so what it owes is the half it can give: the [Identifier] printed unaltered under clause 4.3, and no @@ -236,7 +245,7 @@ An automated check over every [Identifier] a [Bundled rule] can print, applying family list or pattern of clause 6.5 where one is used, that blocks the release when any of them lands on no page. -**Why**: clause 6.1 is the clause most easily believed to be satisfied whilst being broken, because +**Why**: clause 6.3 is the clause most easily believed to be satisfied whilst being broken, because the documentation is written by the same person who wrote the [Rule] and its absence is invisible from the inside. Where the [Detector] is shipped inside a [Toolchain], the [toolchain specification](TOOLING-SPEC.md)'s self-audit makes this check a MUST for the [Toolchain]; a @@ -269,9 +278,16 @@ nothing to run. An inline ignore comment, a per-line directive and a generated [Baseline] are all such routes. The [Detector] MAY ship them. For each one it MUST do at least one of two things, and either alone satisfies this clause: make the route disableable by configuration; or make the route -detectable, by a [Rule] the project can write in the [Detector] itself or by a mechanical check -the [Detector] documents. Either lets a project which decides to forbid the route enforce that -decision. A route that can be neither switched off nor seen does not [Conform]. +detectable, by a bespoke [Rule] the project writes for the [Detector] under clause 4.1 or by a +mechanical check the [Detector] documents. Either lets a project which decides to forbid the +route enforce that decision. A route that can be neither switched off nor seen does not [Conform]. + +A route is detectable when the project can find every use of it, in either of two ways: by a +bespoke [Rule] the project registers under clause 4.1 that matches the route wherever it is +used, which the [Detector] runs like any other bespoke [Rule] and need not ship itself; or by a +mechanical check the [Detector] documents, such as a text search for the route's spelling. The +first way needs the [Detector]'s [Rules] to be able to match the route's text, so a [Detector] +whose [Rules] never see comments offers only the second for an ignore comment. **Why**: the method specification's position is that [Suppression] is an [Owner] decision under its clause 3.4 and section 4, and an [Owner] cannot decide something they are @@ -310,17 +326,19 @@ for a [Detector]. The declaration is machine-readable, in whatever form the [Detector]'s ecosystem uses to record dependencies. The same declaration is where a gap against this document is recorded once it is -known: a [Detector] that has learnt, from its own checks or from a [Practitioner]'s -report under the method's clause 3.2, that it fails a MUST in sections 4 to 7 MUST record that gap -alongside the version it declares, in the same file or one it names. A declaration with a non-empty -gap record is a statement of where the [Detector] stands and is not a claim of +known: a [Detector] that declares a version and has learnt, from its own checks or from a +[Practitioner]'s report under the method's clause 3.2, that it fails a MUST in sections 4 to 7 +MUST record that gap alongside that version, in the same file or one it names. A declaration +with a non-empty gap record is a statement of where the [Detector] stands and is not a claim of [Conformance]. -The declaration is optional. It is how a maintainer claims [Conformance]; it is not a -condition of it. A [Detector] that predates this document, or whose maintainer has never read -it, MAY be graded [Conforming] on evidence by anyone who exercises the clauses above against -it, and a verdict on any [Detector], declared or not, rests on that exercise and not on the -claim. A declaration tells the reader what the maintainer believes and what they know to be missing; +The declaration is optional, and so the duty above falls only on a [Detector] that has one; a +gap a [Practitioner] finds is recorded in their own project's record under the method's clause +3.2 in any case. The declaration is how a maintainer claims [Conformance]; it is not a condition +of it, and a [Detector] without one can still be found to [Conform]. A [Detector] that predates +this document, or whose maintainer has never read it, MAY be graded [Conforming] on evidence: +anyone may grade it, by checking it against each clause of sections 4 to 7, and a verdict on +any [Detector], declared or not, rests on that check and not on the claim. A declaration tells the reader what the maintainer believes and what they know to be missing; the reader still runs the [Harness]. The shape is illustrative rather than prescribed. In a Composer manifest: @@ -348,9 +366,32 @@ a specific installed artefact, checkable by anyone, including mechanically, and never [Conform] until its maintainer had heard of this document, which grades the maintainer's reading rather than the [Detector]. +### 8.2 A verdict on a detector MUST name its kind in each configuration it grades + +A verdict on a [Detector] MUST name its kind, [Semantic], [Syntactic] or [Textual], in each +configuration the verdict grades. A configuration here is one way of setting the [Detector] up to +run, and one [Detector] can have several: ESLint, for example, is [Syntactic] run without type +information and [Semantic] run with it. The kind is what a [Rule] can ask in that configuration, +as section 2 glosses it. + +This clause binds whoever gives the verdict, anyone grading a [Detector] under clause 8.1 +included, and asks nothing of the [Detector] itself; a maintainer who gives a verdict on their +own [Detector] is bound by it like anyone else. A declaration under clause 8.1 is the +maintainer's claim, not a verdict, and this clause asks nothing of it. A [Detector] is judged on +sections 4 to 7, as section 8 says, and the kind is not a level of [Conformance] and does not +change the verdict: a [Textual] [Detector] can [Conform], and a [Semantic] one can fail to. A +verdict that leaves the kind out has not met this clause, but what it finds about [Conformance] +still stands. + +**Why**: the method specification's clause 3.2 requires a [Rule] to be written in the strongest +kind of [Detector] that can express it, and a verdict of [Conforming] alone presents a [Textual] +[Detector] as an equal host. + ## 9. Relationship to the other specifications -This document adds no obligations to a [Practitioner] and relaxes none. Every clause here +This document adds no obligation to the method a [Practitioner] follows, and relaxes none. +Clause 8.2 adds one on whoever gives a verdict on a [Detector], a [Practitioner] included when +they give one: to name its kind. Every clause here exists to make a clause of the method specification achievable with a [Detector] in hand. Where this document and the method specification disagree, the method specification governs. It @@ -396,5 +437,8 @@ does not have to. [rule]: SPEC.md#rule [rule author]: #rule-author [rules]: SPEC.md#rule +[semantic]: SPEC.md#semantic [suppression]: SPEC.md#suppression +[syntactic]: SPEC.md#syntactic +[textual]: SPEC.md#textual [toolchain]: SPEC.md#toolchain diff --git a/PRIMER.md b/PRIMER.md index edb1f15..732ee0e 100644 --- a/PRIMER.md +++ b/PRIMER.md @@ -28,7 +28,8 @@ it is an instance of coalescing an absent value into a falsy one, which is a pat patterns can be detected mechanically. Then write the rule that detects that pattern. A static analysis rule, a custom lint rule, a -check in whatever tool your project already runs on every commit. Something that reads the code +check in a tool your project already runs on every commit or your language's ecosystem offers, +one that sees types where your language has one. Something that reads the code rather than running it, so it can be pointed at every file you have rather than only the paths your tests happen to exercise. diff --git a/README.md b/README.md index e43ba91..1f1bbd6 100644 --- a/README.md +++ b/README.md @@ -11,8 +11,8 @@ Six clauses, in order: 1. **Attribute the defect to a class.** Not "what went wrong here" but "what kind of thing is this an instance of", within a lower bound and an upper bound that are both checkable. -2. **Build the net.** A rule that reads code, in whatever detector the toolchain offers, bespoke - where nothing off the shelf will take it. +2. **Build the net.** A rule that reads code, in the strongest kind of detector that can express + it, bespoke where nothing off the shelf will take it. 3. **Prove the net by making the rule fire.** Red before green, on the originating instance, committed before the fix. 4. **Sweep the codebase, then fix every instance.** The count is the finding; the fix addresses diff --git a/SPEC.md b/SPEC.md index 54ddeb2..1795fbd 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,6 +1,6 @@ # Defence Before Fix: Method Specification -**Version**: 1.1.0, published 2026-10-02 +**Version**: 1.2.0-dev, unpublished **Companion to**: [the detector specification](DETECTOR-SPEC.md), version 1.0.0, and [the toolchain specification](TOOLING-SPEC.md), version 0.2.0 **Author**: [Joseph Edmonds](https://ltscommerce.dev), [Edmonds Commerce](https://edmondscommerce.co.uk) **Coined**: 22 February 2026, in [the original article](https://ltscommerce.dev/articles/defence-before-fix-static-analysis) @@ -29,7 +29,7 @@ US spelling: **Defense Before Fix**. Abbreviated [DBF] throughout. ## Status of this document -This is version 1.1.0 of the specification. It is normative: section 3 defines the method, +This is version 1.2.0-dev of the specification. It is normative: section 3 defines the method, section 4 states who decides what, and section 7 defines what [Conformance] means and who may claim it. @@ -70,12 +70,24 @@ The harm a [Class] causes, which is **not necessarily a failure**. It may be a f #### Detector -A tool that reads code without executing it and reports occurrences of a pattern. A [Detector] is not a [Runner]. What a [Detector] must offer so that a [Practitioner] can write, prove, run and resolve a [Rule] in it is specified in the [detector specification](DETECTOR-SPEC.md). +A tool that reads code without executing it and reports occurrences of a pattern. A [Detector] is not a [Runner]. By what it can see in the configuration it runs in, a [Detector] is [Textual], [Syntactic] or [Semantic], and clause 3.2 says which kind a [Rule] is written in. What a [Detector] must offer so that a [Practitioner] can write, prove, run and resolve a [Rule] in it is specified in the [detector specification](DETECTOR-SPEC.md). #### Runner A tool that **executes** code and reports what happened: a test [Runner], a compilation, a benchmark, a smoke check. A [Runner] answers questions about behaviour at one moment, on the paths it happened to exercise; a [Detector] answers questions about the text of the code, everywhere it exists. Detection under this method MUST come from a [Detector], for that reason. +#### Textual + +Of a [Detector]: one in which a [Rule] can ask only how the code is spelt, as grep does. It sees the text and nothing of its structure, so it cannot tell a call from a comment that names it. + +#### Syntactic + +Of a [Detector]: one in which a [Rule] can ask about the parse tree of the code, and at most some of the types of values, as ast-grep does. It sees the structure, but cannot always tell apart two calls of the same name on values of different types. + +#### Semantic + +Of a [Detector]: one in which a [Rule] can ask the type of any expression and what any name refers to, across the whole program, as the language's compiler or type checker resolves them, as in PHPStan, mypy with `check_untyped_defs`, Clippy, Dylint and ESLint with typescript-eslint's type information. One that infers types only in part, or only within one file, is [Syntactic]. The same [Detector] can be [Syntactic] in one configuration and [Semantic] in another: ESLint without type information is [Syntactic]. A [Semantic] [Detector] need not accept bespoke [Rules]: Clippy does not, and in Rust a project's own [Semantic] [Rules] are written for Dylint. + #### Rule One pattern definition within a [Detector]. @@ -189,7 +201,8 @@ them judge whether the attempt was one. There are three ways out of the method and they are recorded differently. No pattern exists: the [Defect] is out of scope under this section, and only the sentence above is recorded. A pattern exists but the -language has no extensible [Detector] and a bespoke one is not practical: a [Toolchain] gap +language has no extensible [Detector] of a kind that can express the [Rule] within clause 3.1's +bounds, and a bespoke one is not practical: a [Toolchain] gap under clause 3.2. A pattern exists, a [Detector] exists, but the [Toolchain] lacks a mechanism the [toolchain specification](TOOLING-SPEC.md) requires: a mechanism gap under clause 3.2, and the [Rule] is still built. The attempt that separates the first @@ -215,14 +228,14 @@ the [Defence] it implies is ever reached. Six clauses, in order. In brief, before the detail: -| Clause | The [Practitioner] MUST | The record shows | Goes to the [Owner] under section 4 | -| ------ | ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | -| 3.1 | Name the [Class] the [Defect] belongs to, bounded both ways, after an independent search | The [Class], the [Hazard] sentence, the two search techniques, the next wider [Rule] not built, whether a [Runner] check pins the reported behaviour | Whether the next wider [Rule] is built, and whether the [Rule] stays narrower than the search showed | -| 3.2 | Express the [Class] as a [Rule] in a [Detector], never a test | The [Rule], and any [Toolchain] gap that stopped a bespoke one | Whether a [Class] that could be defended is left undefended | -| 3.3 | Make the [Rule] fire, on the [Instance] or a [Fixture], in a commit of its own | The red run, and the sentence behind every [Narrowing] | Any exclusion whose sentence cannot be written: that is [Suppression] | -| 3.4 | [Sweep] the whole codebase, record the count, then fix every [Instance] | The count, corroborated, and what was fixed by hand or by pattern | Any [Instance] left unfixed, and any [Baseline] | -| 3.5 | Make the [Rule] permanent and [Blocking] in the project's own checks | The green run through the project's entry point, and the recorded decision behind any [Suppression] | Removing or disabling the [Rule], and any [Suppression] added to it | -| 3.6 | Print a terse [Message] with a stable [Identifier] that resolves to documentation versioned with the [Rule] | The [Remediation docs] | Nothing | +| Clause | The [Practitioner] MUST | The record shows | Goes to the [Owner] under section 4 | +| ------ | ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| 3.1 | Name the [Class] the [Defect] belongs to, bounded both ways, after an independent search | The [Class], the [Hazard] sentence, the two search techniques, the next wider [Rule] not built, whether a [Runner] check pins the reported behaviour | Whether the next wider [Rule] is built, and whether the [Rule] stays narrower than the search showed | +| 3.2 | Express the [Class] as a [Rule] in the strongest kind of [Detector] that can express it, never a test | The [Rule], its [Detector] and kind, why it is in a weaker kind than [Semantic] where it is, and any [Toolchain] gap that stopped a bespoke one | Whether a [Class] that could be defended is left undefended, and a weaker kind where the project requires the [Owner]'s sign-off | +| 3.3 | Make the [Rule] fire, on the [Instance] or a [Fixture], in a commit of its own | The red run, and the sentence behind every [Narrowing] | Any exclusion whose sentence cannot be written: that is [Suppression] | +| 3.4 | [Sweep] the whole codebase, record the count, then fix every [Instance] | The count, corroborated, and what was fixed by hand or by pattern | Any [Instance] left unfixed, and any [Baseline] | +| 3.5 | Make the [Rule] permanent and [Blocking] in the project's own checks | The green run through the project's entry point, and the recorded decision behind any [Suppression] | Removing or disabling the [Rule], and any [Suppression] added to it | +| 3.6 | Print a terse [Message] with a stable [Identifier] that resolves to documentation versioned with the [Rule] | The [Remediation docs] | Nothing | The fourth column is what an [Agent] or other [Practitioner] MUST NOT decide alone, clause by clause; section 4 states the full list and governs where the two differ. @@ -396,7 +409,46 @@ Any tool that reads code and reports pattern matches qualifies, whatever its cat walkers, architectural fitness checks, and [Detectors] that inspect a change before it lands. These are examples rather than a permitted list, and a [Detector] is not disqualified for being unfamiliar. -Where the affected language has no extensible [Detector] available, the [Practitioner] MAY build a +**Qualifying is not the same as being the right host.** A [Rule] the [Practitioner] writes MUST +be written in the strongest kind of [Detector] that can express it: a [Semantic] one, else a +[Syntactic] one, else a [Textual] one. A weaker kind is allowed only where the stronger kind +cannot express the [Rule] within clause 3.1's bounds, or where the language has no [Detector] +of the stronger kind that accepts bespoke [Rules]; those are the two grounds. For example, the +stronger kind cannot express a [Rule] that reads a file the [Semantic] [Detector] does not +parse. The kind is that of the configuration the [Rule] runs in. Enabling an existing [Rule], or tightening its thresholds, is +not writing one, and the off-the-shelf paragraph above covers it; giving an existing [Rule] a +pattern of the project's own, or changing the pattern of a [Rule] the project wrote, is writing +one. + +For the second ground, the language has a [Detector] of the stronger kind when the project +already runs one or the language's own [Detector] ecosystem offers one. A stronger configuration +of a [Detector] the project runs counts; a [Detector] or configuration that only a hosted service or a licence tier +provides does not. A [Detector] accepts bespoke [Rules] when the project can add its own [Rule] +to it through a documented extension point, such as a plugin interface; it need not meet the rest +of the [detector specification](DETECTOR-SPEC.md). + +On the [Remediation docs] of a [Rule] they write, the [Practitioner] MUST name the [Detector] and +its kind. Where that kind is weaker than [Semantic], the [Practitioner] MUST also record the +grounds in two places, both of them: on the [Rule]'s [Remediation docs], and in the description +of a commit. That commit is the one that adds the [Rule] or moves it to another [Detector]; for +a [Rule] already in place when its grounds are first written down, it is the commit that writes +them. What is recorded, in both places, is which of the two grounds holds at each +step down the order [Semantic], [Syntactic], [Textual], and why, naming the [Detectors] of the +stronger kind that were considered. The [Practitioner] judges whether the grounds hold, as +section 4 gives them the choice of [Detector]. A project MAY require its [Owner]'s sign-off on +a weaker kind as well, by recording that requirement where it records its [Calibrations]; where +it has, the [Practitioner] MUST record the sign-off with the grounds, and where the project has +recorded no such requirement none is needed. + +**A worked example.** Take a [Rule] against calling a method named `context` on one type. A +[Detector] that cannot see types reports every call named `context`, whatever type it is called +on, which is a [False positive] clause 3.1 does not tolerate; restricting it by the name of the +variable the call is made on misses the same call on a variable named otherwise, which is an +[Instance] the [Rule] never sees. A [Semantic] [Detector] decides by the type, so the bounds of +clause 3.1 can be held there. + +Where the affected language has no extensible [Detector] available of a kind that can express the +[Rule] within clause 3.1's bounds, the [Practitioner] MAY build a bespoke one: a program of the project's own that reads code and reports matches of the [Class], run through the project's own entry point like any other [Detector]. This is a last resort rather than a default, because a [Detector] the ecosystem maintains is cheaper to keep and easier for the next @@ -415,7 +467,8 @@ then the only requirement is that somebody deciding what to invest in the [Toolc it. A gap recorded where nobody looks has been forgotten with extra steps. There are two kinds of gap and they lead to different work. The language having no extensible -[Detector] and no practical bespoke one is the gap above, and the [Class] waits for one. The project's [Toolchain] +[Detector] of a kind that can express the [Rule] within clause 3.1's bounds, and no practical +bespoke one, is the gap above, and the [Class] waits for one. The project's [Toolchain] lacking a mechanism the [toolchain specification](TOOLING-SPEC.md) requires, such as a proving harness or an [Identifier] resolver, is a mechanism gap, and it does not put the [Class] out of reach: the [Practitioner] builds the [Rule], uses the substitutes clauses 3.3 and 3.6 already allow, and records @@ -711,7 +764,8 @@ pressure, which is the condition under which the original [Defect] was written. The failure [Message] MUST be terse, and it MUST carry a stable [Identifier] that resolves to [Remediation docs] shipped with the project. -That documentation MUST state three things: what the [Rule] is about, why it exists, and how to +That documentation MUST state three things, besides what clause 3.2 adds for a [Rule] the +[Practitioner] writes: what the [Rule] is about, why it exists, and how to fix a violation correctly using the project's preferred approach. A [Practitioner] who finds that an existing [Rule]'s printed [Identifier] does not resolve records @@ -767,7 +821,8 @@ they do not. **Decisions the [Practitioner] takes alone**, without seeking approval: - What [Class] the [Defect] belongs to, and where its bounds sit (clause 3.1). -- Which [Detector] to use and how to express the [Rule] in it (clause 3.2). +- Which [Detector] to use and how to express the [Rule] in it (clause 3.2), unless the project + requires the [Owner]'s sign-off on a weaker kind of [Detector], as clause 3.2 allows. - How to prove the [Rule], and what [Fixture] to prove it against (clause 3.3). - What the correct behaviour is at each [Instance], and therefore what each fix should be (clause 3.4). @@ -790,7 +845,8 @@ they do not. than that the [Hazard] is absent from what it would add. The [Instances] the search found are fixed regardless; what stays with the [Owner] is the [Class] left partly undefended. -The dividing line is that the [Practitioner] decides **how the [Defence] is built** and the [Owner] +The dividing line is that the [Practitioner] decides **how the [Defence] is built**, save a weaker +kind of [Detector] where the project requires the [Owner]'s sign-off on it, and the [Owner] decides **what the codebase is permitted to keep**. An [Agent] MUST NOT [Baseline], suppress or knowingly leave an [Instance] unfixed on its own authority, whatever the [Instance] count turns out to be. @@ -894,9 +950,12 @@ evidence is gone and the opportunity closes with it. the verdict that they were rests on reproduction, as this section closes by requiring, not on the report. -**A [Defence] [Conforms]** if it satisfies clauses 3.1, 3.2, 3.3, 3.5 and 3.6: it is drawn to a -[Class] within both bounds and not to the reported [Instance], it is evaluated by reading code, it -was proven to fire against either the originating [Instance] or [Fixture] code, it fails rather +**A [Defence] [Conforms]** if it satisfies each of clauses 3.1, 3.2, 3.3, 3.5 and 3.6 in full. +In brief: it is drawn to a +[Class] within both bounds and not to the reported [Instance], it is evaluated by reading code, in +the kind of [Detector] clause 3.2 requires and with the [Detector], its kind and any grounds +recorded as that clause requires, it was proven to fire against either the originating +[Instance] or [Fixture] code, it fails rather than warns, and its [Message] is terse and resolves to [Remediation docs] versioned alongside it. A [Defence] can [Conform] whilst the [Sweep] is incomplete; what that describes is a [Conforming] [Defence] over a codebase with known [Instances], which section 4 requires to be @@ -1036,7 +1095,7 @@ re-open it. ## 9. Citation -> Edmonds, Joseph. *Defence Before Fix*, version 1.1.0. First published 22 February 2026. +> Edmonds, Joseph. *Defence Before Fix*, version 1.2.0-dev. First published 22 February 2026. > ## Appendix A: Instructing an agent @@ -1051,8 +1110,18 @@ This appendix restates sections 2 to 4; where the two differ, the sections gover > for it, say so and fix the [Defect] conventionally. > > Write a custom [Rule] that detects the [Class], in a tool that reads code rather than running it. -> Draw it so that it catches more than the single [Instance] you started from, but never so broadly -> that it matches code which does not carry the [Hazard]. +> Put it in a [Semantic] [Detector], one in which a [Rule] can ask the type of any expression and +> what any name refers to, across the whole program; failing that, one that sees syntax; and only +> then one that sees text. Use a weaker one only where the stronger one cannot express the [Rule], +> or the language has none that takes custom [Rules]: none the project runs, in any configuration +> it can use without a hosted service or a licence tier, and none its ecosystem offers without +> one either. Name the [Detector] and its kind in the [Rule]'s documentation. Where you use a weaker one, +> write down which of those two grounds holds, and why, naming the stronger [Detectors] you +> considered, there and in the description of the commit that adds the [Rule], moves it to another +> [Detector], or first records the grounds, and where the project requires its [Owner]'s sign-off +> on a weaker one, get it and record it with the grounds. Draw +> the [Rule] so that it catches more than the single [Instance] you started from, but never so +> broadly that it matches code which does not carry the [Hazard]. > > Do not trust the [Rule] as your only way of finding [Instances]. Search independently as well, by > text search and by reading the code, and check the [Rule] catches what you found by hand. Make that @@ -1137,6 +1206,7 @@ This appendix restates sections 2 to 4; where the two differ, the sections gover [escalation]: #escalation [exception]: #exception [exceptions]: #exception +[false positive]: #false-positive [false positives]: #false-positive [fixture]: #fixture [fixtures]: #fixture @@ -1155,9 +1225,12 @@ This appendix restates sections 2 to 4; where the two differ, the sections gover [rules]: #rule [runner]: #runner [runners]: #runner +[semantic]: #semantic [suppress]: #suppression [suppression]: #suppression [suppressions]: #suppression [sweep]: #sweep +[syntactic]: #syntactic +[textual]: #textual [toolchain]: #toolchain [warning]: #warning diff --git a/acceptance/CHANGES.md b/acceptance/CHANGES.md new file mode 100644 index 0000000..daa3d44 --- /dev/null +++ b/acceptance/CHANGES.md @@ -0,0 +1,38 @@ +# Changes to the acceptance process + +Changes to the acceptance process, as [ACCEPTANCE.md](../ACCEPTANCE.md) defines one under +"Changes to this process", recorded here as it describes: a line is added at the top for each +change, naming its pull request. The process is not a specification and is not versioned, and a +change to it is not tested by a cohort. Changes to the three specifications are recorded in +[CHANGELOG.md](../CHANGELOG.md). + +- [Pull request #22](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/22): + the test is made to prove what it exists to prove, the clarity of the + document to a cold reader, after thirty-three runs on two changes (19 to 30, 31 to 33 and 35 + to 52) in which keyed answers reached five of five and the runs still failed. Five defects in + the process were found. Keys required elements the question never asked for (detector 2b's + "need not remove or forbid", detector 2c's "at a version tracked together", and on pull + request 14 method 2d's order of the kinds), so readers who answered the question as put were + marked wrong: now each key lists, per item, the elements an answer must state, one for + each part of the question, and the questions are reworded to ask for exactly those; anything + else in the key only catches a wrong assertion. A passage two readers found confusing stayed + open whilst any single reader of a later cohort named any part of its paragraph, so cleared + passages kept coming back: now a finding is settled run by run, raised by two readers of one + run and clear when the next names it by fewer than two. A finding landed on text the change + never touched, so a change's cohort acted as a review of the whole document and could not + converge: now a finding in text the change wrote or altered blocks the change, and a finding + elsewhere is routed to `acceptance/ROUTED.md` and to a change of its own, whilst the keyed + questions still prove the whole document on every run. Markers invented judgements the rules + left open and later runs cited them as precedent: now an open point is marked for the reader + and settled in this file or the key before the next run, and a run record states no judgement + the rules do not. The rule that any miss whose reasoning quoted the document was a finding + whatever the count is replaced by the key naming the wrong readings the text invites, which + count against the document on one reader, every other single miss being tolerated by the + all-but-one threshold. Question 2's instruction asks for the sentence or sentences relied on, + since three keys rest on two. The cohort, the packet, set-aside readers, the posted sheets, + the fuller record and the route for a process change are kept from the earlier proposal in + pull request 13, which this change supersedes. Relative to the rules on `next`, the + traceable-miss row is narrower and the confusion row is applied to the change rather than + the document; both are recorded here as the reason the earlier rules could not converge, and + neither changes a mark already given. Prompted by runs 19 to 30 on pull request 14 and runs + 31 to 33 and 35 to 52 on the defined-terms change. diff --git a/acceptance/README.md b/acceptance/README.md index 4cfb6ec..ec5c3b2 100644 --- a/acceptance/README.md +++ b/acceptance/README.md @@ -2,8 +2,10 @@ The question sets and answer keys that [ACCEPTANCE.md](../ACCEPTANCE.md) requires. One directory per specification document, each holding the fixed questions a cold reader answers and the key an -editor marks against. The fixture cases the application question uses live in `fixtures/`, and -the record of every run lives in `runs/`. +editor marks against. The fixture cases the application question uses live in `fixtures/`, the +record of every run lives in `runs/`, findings routed out of a run to a change of their own are +listed in [ROUTED.md](ROUTED.md), and changes to the acceptance process itself are recorded +in [CHANGES.md](CHANGES.md). | Document | Questions | Key | | ------------------ | ------------------------------------------------ | ------------------------------------ | @@ -14,15 +16,36 @@ the record of every run lives in `runs/`. ## How a run works 1. The editor identifies which documents the change touched. Only those are tested. -2. For each, five or more fresh low-strength readers each receive the document's current text, - its `QUESTIONS.md`, and the fixture the questions name. Nothing else: no key, no changelog, no - critique findings, no other reader's answers. -3. The editor marks each answer sheet against `KEY.md` and applies the pass criteria in - `ACCEPTANCE.md`. -4. The result is recorded as a file under [runs/](runs/), with cohort size, model, run number, - marks and the findings applied, and the change's `Unreleased` changelog line names that file. - A run that fails is followed by a change to the text, never to the questions, and a fresh - cohort. +2. For each, an odd number of fresh low-strength readers, at least five, each isolated as + `ACCEPTANCE.md` describes, receive the document's current text, its `QUESTIONS.md`, and the + fixture the questions name. Nothing else: no key, no changelog, no critique findings, no other + reader's answers. +3. The editor, or a separate marker the editor delegates to, marks each answer sheet against + `KEY.md` under the marking rules in `ACCEPTANCE.md` and applies its pass criteria. +4. The result is recorded as a file under [runs/](runs/), stating what `ACCEPTANCE.md` lists, + with the answer sheets posted on the pull request and linked from it, and the change's + `Unreleased` changelog line names that file. A run that fails is followed by a change to the + text its misses and findings name and a fresh cohort; a finding on text the change did not + write or alter is added to [ROUTED.md](ROUTED.md) instead. + +## How a key is written + +A key says, for each keyed item, exactly what is marked and nothing a marker has to guess: + +- **Question 1**: the obligations a bullet may name, the assertions that make a bullet wrong, + and the wrong readings the text invites, if any. +- **Question 2**: for each item, the elements an answer is required to state, numbered, one for + each part of the question; the assertions that make an answer wrong; and the sentence or + sentences of the document the answer rests on, quoted verbatim, which show the marker where + the elements come from and which `spec-qa.py` checks against the document; a reader's + quotation need not be that one. The required elements are what the question asks for, no + more: an element the question does not ask for is not required, however load-bearing, until + the question is reworded to ask for it. +- **Question 3**: for each item, the grade or grades accepted, any condition on the reasoning, + and the wrong readings the text invites, which count against the document when one reader + takes them. + +Everything else in a key is explanation for the marker and never marks an answer down. ## Keeping the keys honest @@ -30,8 +53,24 @@ A key quotes the document. When a clause is reworded, the key line that quotes i the same commit, and the change is one the acceptance run itself then tests. A key that no longer matches the document is a defect in this directory, not a reason to mark a reader wrong. +A key, a question set or a fixture changes by one of two routes, as `ACCEPTANCE.md` says under +"Changing the test", always before the readers of the next run start, and an open point a run +recorded is settled by whichever route it needs: + +- **In a specification pull request, under step 4 of `CONTRIBUTING.md`**: a quotation follows + the text it quotes; a keyed item is added for a new load-bearing obligation together with the + question that asks for it; or, where the change alters or removes the obligation an existing + item tests, that item and its question follow the text, which is the text changing and not + the test. The pull request's own changelog line records it. Such an update does not change + what an item requires of an obligation the change leaves as it was, nor the wording of a + question whose obligation is unchanged. +- **In a process change, with its own pull request**: what an existing item requires, the + wording of a question, a fixture, or the rules of marking. It is recorded in + [CHANGES.md](CHANGES.md) with its reason. + Questions are fixed so that runs can be compared across versions. Adding a keyed question is -allowed when a new clause is load-bearing; removing or softening one to make a run pass is not. +allowed when a new clause is load-bearing; removing or softening one to make a run pass is not; +rewording one is allowed so that it asks for what its key requires, and for no other reason. The reader may quote a passage it found confusing; it is never asked to rate the document, say what would make a passage clearer, or propose a change: the cohort confirms comprehension and does not design the text. diff --git a/acceptance/ROUTED.md b/acceptance/ROUTED.md new file mode 100644 index 0000000..e53b671 --- /dev/null +++ b/acceptance/ROUTED.md @@ -0,0 +1,59 @@ +# Passages routed out of a cohort + +A finding is a passage two or more readers of one cohort named as confusing under question 4. +`ACCEPTANCE.md` routes here a finding that lies outside the text the change being tested wrote +or altered, rather than blocking the change whose cohort raised it, so that each change is +judged on what it changed and the document is still made clearer, one change at a time. The +editor reads this list before cutting a release. + +A finding is open until a change that reworks its passage passes a cohort, when the line names +that run and moves below the rule, or until the editor closes it with a reason recorded here. A +finding raised again by a later run is noted on its line, not listed twice. The status column +holds `open`, or the `editor/` branch or pull request number of the change that is +reworking the passage. The rows raised by runs 19 to 52 were raised under the earlier rules, on +untouched text of the change those runs tested, and are listed here so that they are not lost. + +## Open + +| Document | Clause | Passage | Raised by | Status | +| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | -------------------------------------------------- | +| `TOOLING-SPEC.md` | [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project), the heading and first paragraph | "SHOULD state its own defaults for anything the method leaves to the project" against clause [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one)'s "MUST NOT supply a default" for a justification | run 86 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), the first paragraph | "It MUST NOT supply a default, and MUST reject an Exception that omits one" against clause [6.4](../TOOLING-SPEC.md#64-the-toolchain-should-state-its-own-defaults-for-anything-the-method-leaves-to-the-project)'s defaults; named again by two readers of run 85 and two of run 86 | run 83 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) and [6.3](../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Both clauses speak only of exceptions, where section 2's Project record entry and clause 6.1 cover calibrations and conventions too | run 81 | `open` | +| `TOOLING-SPEC.md` | [6.1](../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), the first paragraph | "MUST define where the Project record lives" against section 2's Project record entry (calibrations, exceptions and conventions): one location or one per kind; named again by four readers of run 81, and by two readers of run 84 as the Project record entry against clauses 6.1, 6.2 and 6.4; and by two readers of run 86, the entry and clause 6.1's first paragraph together | run 76 | `open` | +| `DETECTOR-SPEC.md` | [4.2](../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), the first paragraph | "a Detector failing 4.1 is graded here on its Bundled rules": whether it limits or waives anything | run 78 | `editor/defined-terms`, which removes the sentence | +| `SPEC.md` | [Appendix A](../SPEC.md#appendix-a-instructing-an-agent), the closing paragraph | "Only then fix the original Defect in the normal way, with a test that reproduces it" | run 70 | `open` | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "MUST carry both a usable summary and the location of the remainder" against "the output need not name it" | runs 69, 75 | `editor/detector-route-rule` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) and [9.1](../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately) | Section 9 counts section 8 in a shipping project's project grade; clause 9.1 says section 8 bears on the artefact grade only | runs 43, 46, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [2](../TOOLING-SPEC.md#2-terminology), [4.2](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through), [8.1](../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) | The Defence row, "with its documentation, in force and Blocking", against clauses 4.2 and 8.1, which exist to catch a Blocking rule whose documentation is missing | runs 33, 40, 43, 46, 49, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9.2](../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) | "MUST record that gap alongside the version" against "The declaration is optional." | runs 33, 52, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.3](../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) | The heading, "through a defence of its own", against the body's "MUST disable ... or MUST run a Blocking Defence" | runs 33, 55 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [4.1](../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | "That document's MUSTs, in brief, are six", where the detector specification's sections 4 to 7 carry eleven | runs 40, 46 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [9](../TOOLING-SPEC.md#9-conformance) | "A Toolchain Conforms with Agent support if, in addition, it satisfies section 7", where section 7 holds only SHOULDs | run 57 | `editor/toolchain-contradictions` | +| `TOOLING-SPEC.md` | [5.3](../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | "Where such a Defence is not expressible in the Toolchain's own Detectors, its entry MAY be sourced from wherever it is enabled": what "such a Defence" and "its entry" refer to | run 52 | `open` | +| `TOOLING-SPEC.md` | [6.2](../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | The heading, "the toolchain MUST reject a generic one", and the paragraph "The Toolchain MUST reject a justification that could be pasted onto any Exception unchanged, by a check it documents": whether the check must test that the hazard and the scope are named | runs 67, 68, 82 | `open` | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a project's own Rule" paragraph: its URL sentence, "as method clause 3.6 allows", "a documentation path printed alongside a shorter Identifier", and "The lookup for such a Rule is an obligation on the project's assembled Toolchain" against section 1's governing principle | runs 28, 63, 66 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Owner entry | The default Owner sentences: whether whoever instructed the Practitioner may take section 4's decisions, "Always a human", and the bracketed verb "Suppress" among nouns | runs 23, 41, 47, 50, 56, 64 | `open` | +| `SPEC.md` | [1](../SPEC.md#1-terminology), the Narrowing entry | "Narrowing that excludes code which does carry the Hazard is a Suppression" | run 23 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom) | "the Rule MAY merge at full width with that case recorded as a known Instance awaiting the Owner" against "provided the Rule does not become Blocking until every Instance is fixed" | runs 50, 70 | `open`; a method question for the editor | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the lead-in to the owner's decisions | "Decisions that MUST go to whoever owns the codebase", against the Owner entry | run 21 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the lower-bound list item | "That rejection is tested as a Narrowing is under clause 3.3", and "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt" | runs 23, 56, 59, 64, 70 | `open` | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on independent techniques | "Two techniques are independent when they would miss different things", and that a clean run of the Rule is never one of the two | run 25 | `open` | +| `SPEC.md` | [3](../SPEC.md#3-the-method), the table's 3.1 row | The fourth column, "Whether the next wider Rule is built, and whether the Rule stays narrower than the search showed" | run 21 | `open` | +| `SPEC.md` | [2](../SPEC.md#2-when-the-method-applies), "Attempt rather than pre-judge" | "naming at least two independent techniques tried at expressing the Rule, in the vocabulary clause 3.1 uses for a search" | runs 19, 29, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [4](../SPEC.md#4-authority-which-decisions-belong-to-whom), the list item on leaving the next wider Rule unbuilt | "The absence of an Instance is the Practitioner's reason not to widen unasked", read against clause 3.1's lower bound | runs 21, 53, 59, 61, 64, 70 | `open` | +| `SPEC.md` | [3.4](../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), the "Fix them all." paragraph | "the scale at which that becomes true is tremendous" | runs 19, 61 | `open` | + +## Closed + +A row moves here when a change that reworks its passage passes a cohort, naming that run, or +when the editor closes it with a reason. + +| Document | Clause | Passage | Raised by | Closed by | +| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------ | +| `DETECTOR-SPEC.md` | [7.1](../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | "by a Rule the project can write in the Detector itself", read as the project editing the detector's own source, against clause 4.1's bespoke rules | runs 20, 22, 24, 30, 36, 42, 45, 48, 51, 54, and 71 on a branch without the rework | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [8.1](../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps) | "MUST record that gap alongside the version it declares" against "The declaration is optional." | runs 20, 30, 39, 42, and 71 on a branch without the rework | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [5.3](../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | "the location of the remainder": what the remainder is when the summary prints every finding | run 32 | `editor/detector-route-rule`, run 66 | +| `DETECTOR-SPEC.md` | [6.1](../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | The "For a Bundled rule" paragraph's sentence on a Message that carries its own documentation path, against the clause's mechanism keyed on the Identifier | runs 20, 54 | `editor/detector-route-rule`, run 66 | +| `SPEC.md` | [3.1](../SPEC.md#31-attribute-the-defect-to-a-class), the paragraph on the two bounds | "Both bounds measure breadth within one level; the level itself is a separate question, answered against the report." | runs 19, 56, and 70 on a branch without the rework | `editor/defined-terms`, run 64 | +| `SPEC.md` | [3.5](../SPEC.md#35-enforce-permanently-and-block), the paragraph on a bundled defence | "For a Bundled defence, the project whose entry point demonstrates it is any project the Toolchain is genuinely installed into", and its substitutes sentence | runs 25, 29, 53, 59 | `editor/defined-terms`, run 64 | diff --git a/acceptance/detector/KEY.md b/acceptance/detector/KEY.md index 132e6cb..adf0b7f 100644 --- a/acceptance/detector/KEY.md +++ b/acceptance/detector/KEY.md @@ -1,77 +1,115 @@ # Answer key for DETECTOR-SPEC.md -Marked against the document at the version named in the changelog entry that records the run. -Quotations below are from the document; when a clause is reworded, update the quotation here in -the same commit. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, +update the quotation here in the same commit. `ACCEPTANCE.md` says how each item is marked; +`acceptance/README.md` says what each part of this key is for. ## 1. Restatement -A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting an obligation the document does not state, is a correct restatement. +A bullet is correct when it names one of these obligations. Four or more correct bullets, with +no bullet making an assertion listed below as wrong, is a correct restatement. - Support bespoke rules written by the project ([4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it)). - Provide a harness that runs a single rule against supplied code ([4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code)). -- Allow a stable identifier per rule and print it, unaltered, with every finding ([4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding)). +- Allow a stable identifier per rule and print it, unaltered, with every finding ([4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding)); a + bullet that says each rule carries a stable identifier names this obligation. - Be invocable locally by the practitioner, over a subset down to one file, with the result in - the command's own output and never only through a hosted service or CI-only mode ([5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally)). + the command's own output and never only through a hosted service or CI-only mode ([5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) to [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally)); + a bullet that says every finding, or each finding, must appear in that output names [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran), + whose summary may leave findings out, and is not marked wrong for the plural. - Resolve a printed identifier to documentation; for bundled rules from the installed copy, offline, with the documentation shipped and versioned with the rule ([6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) to [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together)). - Make any inline suppression route detectable or disableable ([7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable)). +- A verdict on the detector names its kind, semantic, syntactic or textual, in each + configuration it grades ([8.2](../../DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades)); the obligation is on whoever gives the verdict, as section + [9](../../DETECTOR-SPEC.md#9-relationship-to-the-other-specifications) says, and a bullet that says a verdict, or a grading, names the kind names it. -Wrong if it asserts: the detector must enforce a project's suppression policy; the detector must -declare a version to conform; the detector must list a project's active defences or produce an -agent summary (those belong to the toolchain specification). +Wrong if it asserts: the detector must be semantic, or of any one kind, to conform; the detector +must itself state or declare its kind, which clause [8.2](../../DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades) asks of the verdict and not of the +detector; the detector must enforce a project's suppression policy; the detector must declare a +version to conform; the detector must list a project's active defences or produce an +agent summary (those belong to the toolchain specification). A bullet that states what +conformance means, or restates a SHOULD as advice, asserts no obligation and is neither correct +nor wrong. + +Wrong readings the text invites, which count against the document on one reader: none named. ## 2. Load-bearing statements -- **2a.** Conformance is satisfying every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression), judged on evidence; a detector - that predates the document can conform, and the declaration is optional. Key quotation, section - 8: "A Detector Conforms if it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression)." Supporting, [8.1](../../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps): "A - maintainer MAY declare the version of this document the detector conforms to". An answer that - says a declaration is required, or that partial satisfaction is conformance, is wrong. -- **2b.** It must make the route either disableable by configuration or detectable by a rule the - project can write or a mechanical check it documents; it is not required to remove or forbid - it. Key quotation, [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each one it MUST do at least one of two things, and either alone - satisfies this clause: make the route disableable by configuration; or make the route - detectable, by a Rule the project can write in the Detector itself or by a mechanical check the - Detector documents". An answer that says the detector must forbid the route, or must require a - reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD), is wrong. -- **2c.** With the rule, in the installed copy, at a version tracked together, and yes, offline. - Key quotations, [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access): "Resolution of a bundled rule's identifier MUST work from the installed - copy, without network access"; [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together): "A bundled rule's documentation MUST ship with the rule, at - a version tracked together". An answer naming the detector's website as an acceptable location - is wrong. +- **2a.** Required: (1) a detector conforms when it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression); + (2) yes, a detector that predates the document can conform. Wrong if it says a declaration is + required, or that partial satisfaction is conformance. Key quotation, section [8](../../DETECTOR-SPEC.md#8-conformance): "A Detector + Conforms if it satisfies every MUST in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression)." Supporting, [8.1](../../DETECTOR-SPEC.md#81-a-maintainer-may-declare-the-version-of-this-document-the-detector-conforms-to-and-the-declaration-records-known-gaps): "A maintainer + MAY declare the version of this document the detector conforms to". The reason, that + conformance is judged on evidence and a declaration is optional, is explanation: an answer + that gives it, or gives it instead of a plain yes, states element (2). +- **2b.** Required: (1) the two things, both named: let the project switch the route off in its + configuration, or make the route detectable (by a bespoke rule the project writes for the + detector, or by a mechanical check the detector documents; naming either means of detection, + or just "detectable", states that alternative); an answer naming only one of the two things + misses this element; (2) yes, either one alone is enough. Wrong if it says the detector must + remove or forbid the route, or must require a reason ([7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason) is SHOULD). Key quotation, + [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable): "For each one it MUST do at least one of two things, and either alone satisfies this + clause: make the route disableable by configuration; or make the route detectable, by a + bespoke Rule the project writes for the Detector under clause [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) or by a mechanical check the + Detector documents". An answer need not say + that the detector is not required to remove or forbid the route; that follows from element (2) + and the question does not ask it. +- **2c.** Required: (1) with the rule, in the installed copy; (2) at a version tracked together + with the rule; (3) yes, it must be reachable without network access. Wrong if it names the + detector's website, or any location reached over the network, as acceptable. Key quotations, + [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access): "Resolution of a bundled rule's identifier MUST work from the installed copy, without + network access"; [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together): "A bundled rule's documentation MUST ship with the rule, at a version + tracked together". "Shipped with the rule", "in the installed package" or "on disk with the + detector" each state element (1); "versioned with the rule" or "at the same version" states + element (2). ## 3. Application: Lintomatic 4.2 -| Clause | Intended verdict | Reasoning the fixture supports | -| ------ | ---------------- | -------------------------------------------------------------------------------------------------------------------- | -| [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules via the `Rule` interface, first class in every report | -| [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `check --only ` runs one rule, bundled or project, with a distinguishing exit code | -| [4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `LM-NNNN` fixed at first release; project names taken as given, not rewritten; printed on every finding line | -| [5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | Local, no account, no server, no network | -| [5.2](../../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Yes | `check ` accepts a single file | -| [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | One line per finding on standard output whether or not the HTML report is enabled | -| [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | The hosted dashboard shows nothing the command line does not print | -| [6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Yes | `explain ` resolves a printed identifier; project rules resolve to their declared `doc` string | -| [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Yes | `explain` reads the installed package; there is no website | -| [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | No | Two of forty bundled rules print `No documentation declared`; every identifier a bundled rule can print MUST resolve | -| [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | The ignore comment is disableable with `ignore_comments = false` | - -Section [8](../../DETECTOR-SPEC.md#8-conformance) verdict: **does not conform**, because [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) fails and partial conformance MUST NOT be -described as conformance. A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Partial on the grounds that thirty-eight resolve -has read the clause differently and that counts against the document; a reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Yes -because they missed the last paragraph of the fixture has misread the fixture, not the document. - -Clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation): **does not change the verdict**. It is a SHOULD, and Lintomatic's failure to check its -own identifiers in CI is a missed SHOULD, not a further MUST failure; the MUST that fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). - -A reader who marks [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) No because the comment takes no reason has confused [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) with [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason), which is -SHOULD; that counts against the document if the reasoning quotes [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) or [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason). +| Clause | Accepted grade | What the fixture supports (no row carries a condition on the reasoning) | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it) | Yes | Project rules via the `Rule` interface, first class in every report | +| [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code) | Yes | `check --only ` runs one rule, bundled or project, prints its findings as `check` does, and exits with a distinguishing code; the fixture says all three, so a grade given on the condition that `--only` prints as `check` does is the grade for this case | +| [4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding) | Yes | `LM-NNNN` fixed at first release; project names taken as given, not rewritten; printed on every finding line | +| [5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure) | Yes | Local, no account, no server, no network | +| [5.2](../../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file) | Yes | `check ` accepts a single file | +| [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran) | Yes | One line per finding on standard output whether or not the HTML report is enabled; a reader who notes that the HTML report need not be named for that reason is right | +| [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally) | Yes | The hosted dashboard shows nothing the command line does not print | +| [6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation) | Yes | `explain ` resolves a printed identifier; project rules resolve to their declared `doc` string; the two undocumented bundled rules are [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together)'s failure | +| [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access) | Yes | `explain` reads the installed package; there is no website | +| [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) | No | Two of forty bundled rules print `No documentation declared`; every identifier a bundled rule can print MUST resolve | +| [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) | Yes | The ignore comment is disableable with `ignore_comments = false` | + +No grade above carries a condition on the reasoning: the accepted grade is correct whatever the +reasoning, and a reader who adds a caveat about what the fixture does not say and still gives +the accepted grade is correct. The fixture does not say what Lintomatic can see: a reader who +names its kind as unknown, or names none, is not marked on it. + +The two answers the question then asks for, each with the condition its reason must meet: + +- **Conformance under section [8](../../DETECTOR-SPEC.md#8-conformance).** Accepted answer: does not conform. Condition: the reason + names [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), the two undocumented bundled rules, or partial conformance not counting. +- **Whether clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation) changes the verdict.** Accepted answer: it does not. Condition: the reason + says that it is a SHOULD, or that the MUST which fails is [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together). + +Wrong readings the text invites, which count against the document on one reader: + +- [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) graded Partial with reasoning that says most, or thirty-eight of forty, bundled rules + resolve, which has read the clause as admitting a proportion. +- [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) graded No with reasoning that says the comment takes no reason and quotes [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable) or [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason), + which has read [7.2](../../DETECTOR-SPEC.md#72-an-inline-suppression-route-should-require-a-written-reason)'s SHOULD into [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable). +- The section [8](../../DETECTOR-SPEC.md#8-conformance) verdict made to turn on Lintomatic's kind, with reasoning that a detector of some + kind cannot conform or that the kind is a level of conformance, which has read [8.2](../../DETECTOR-SPEC.md#82-a-verdict-on-a-detector-must-name-its-kind-in-each-configuration-it-grades) as + changing the verdict: "the kind is not a level of Conformance and does not change the verdict". + +A reader who grades [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) Yes because they missed the last paragraph of the fixture has misread the +fixture, which is the reader's miss. ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage named, as `ACCEPTANCE.md` defines naming and the same +passage. A passage named by two or more readers of the run is a finding: it blocks the change +where it lies in text the change wrote or altered, and is routed to `acceptance/ROUTED.md` +otherwise. What to change is the editor's decision; the reader is not asked and any suggestion +it volunteers is discarded unread. diff --git a/acceptance/detector/QUESTIONS.md b/acceptance/detector/QUESTIONS.md index 58d06d1..5c288f8 100644 --- a/acceptance/detector/QUESTIONS.md +++ b/acceptance/detector/QUESTIONS.md @@ -10,21 +10,22 @@ In your own words and in at most six bullets, what does this document require of ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. +Answer each part of each question in a sentence or two, then quote the sentence or sentences of +the document you are relying on. - 2a. What does it mean for a detector to conform, and can a detector that was written before this document existed, and has never heard of it, conform? - 2b. A detector ships an inline ignore comment. What must the detector do about that route for - it to conform? -- 2c. Where must the documentation for a bundled rule's identifier live, and must it be - reachable without network access? + it to conform, and is doing one of those things enough? +- 2c. Where must the documentation for a bundled rule's identifier live, at what version, and + must it be reachable without network access? ## 3. Application Grade Lintomatic 4.2, as described in the fixture, against every MUST clause in sections [4](../../DETECTOR-SPEC.md#4-authoring-rules) to [7](../../DETECTOR-SPEC.md#7-suppression): [4.1](../../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-written-by-the-project-that-runs-it), [4.2](../../DETECTOR-SPEC.md#42-the-detector-must-provide-a-harness-that-runs-a-single-rule-against-supplied-code), [4.3](../../DETECTOR-SPEC.md#43-the-detector-must-allow-a-rule-to-carry-a-stable-identifier-and-must-print-it-with-every-finding), [5.1](../../DETECTOR-SPEC.md#51-the-detector-must-be-invocable-by-the-practitioner-locally-with-no-infrastructure), [5.2](../../DETECTOR-SPEC.md#52-the-detector-must-support-invocation-over-a-subset-at-minimum-a-single-file), [5.3](../../DETECTOR-SPEC.md#53-the-result-must-reach-the-practitioner-in-the-output-of-the-command-they-ran), [5.4](../../DETECTOR-SPEC.md#54-a-finding-must-not-be-reportable-only-through-a-hosted-service-licence-tier-or-ci-only-mode-the-practitioner-cannot-invoke-locally), [6.1](../../DETECTOR-SPEC.md#61-the-detector-must-provide-a-mechanism-that-resolves-a-printed-identifier-to-its-documentation), [6.2](../../DETECTOR-SPEC.md#62-resolution-of-a-bundled-rules-identifier-must-work-from-the-installed-copy-without-network-access), [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together), [7.1](../../DETECTOR-SPEC.md#71-a-detector-may-offer-an-inline-suppression-route-but-must-make-it-detectable-or-disableable). For each give Yes, Partial or No and one line of reasoning. Then state whether Lintomatic conforms under section [8](../../DETECTOR-SPEC.md#8-conformance), and why. Finally, -state whether clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation) changes the verdict. +state whether clause [6.4](../../DETECTOR-SPEC.md#64-the-detector-should-fail-its-own-release-if-a-bundled-rule-lacks-resolvable-documentation) changes the verdict, and why. ## 4. Confusion diff --git a/acceptance/fixtures/lintomatic.md b/acceptance/fixtures/lintomatic.md index f168d56..9d334fc 100644 --- a/acceptance/fixtures/lintomatic.md +++ b/acceptance/fixtures/lintomatic.md @@ -21,7 +21,8 @@ src/db.zig:88:3 proj.no-raw-sql Query built by string concatenation Fuller detail, including a snippet, is written to `.lintomatic/report.html`. The one-line form is printed whether or not the HTML report is enabled. -`lintomatic check --only ` runs a single rule. It accepts any rule name, +`lintomatic check --only ` runs a single rule and prints its findings as +`check` does. It accepts any rule name, bundled or project, and exits 1 when the rule fires, 0 when it does not, and 2 when the rule name is unknown. diff --git a/acceptance/method/KEY.md b/acceptance/method/KEY.md index 3ca1a80..29b64f6 100644 --- a/acceptance/method/KEY.md +++ b/acceptance/method/KEY.md @@ -1,16 +1,23 @@ # Answer key for SPEC.md -Marked against the document at the version named in the changelog entry that records the run. -Quotations below are from the document; when a clause is reworded, update the quotation here in -the same commit. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, +update the quotation here in the same commit. `ACCEPTANCE.md` says how each item is marked; +`acceptance/README.md` says what each part of this key is for. ## 1. Restatement -A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting an obligation the document does not state, is a correct restatement. +A bullet is correct when it names one of these obligations. Four or more correct bullets, with +no bullet making an assertion listed below as wrong, is a correct restatement. - Attribute the defect to a class, bounded on both sides, with the hazard stated ([3.1](../../SPEC.md#31-attribute-the-defect-to-a-class)). - Express the class as a rule in a detector that reads code, never a test ([3.2](../../SPEC.md#32-build-the-net)). +- Write the rule in the strongest kind of detector that can express it, semantic, else + syntactic, else textual, name the detector and its kind on the rule's remediation docs, and + record the grounds where it is in a weaker kind ([3.2](../../SPEC.md#32-build-the-net)). This line and the one above are + separate obligations of the same clause, and a bullet naming each counts once for each. A + bullet that says a rule goes in a semantic detector, without saying that a weaker kind is + never allowed, names this obligation. - Prove the rule fires, against the instance or a fixture, in a commit of its own ([3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire)). - Sweep the whole codebase, record the count, then fix every instance ([3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance)). - Make the rule permanent and blocking in the project's checks ([3.5](../../SPEC.md#35-enforce-permanently-and-block)). @@ -18,62 +25,114 @@ bullet asserting an obligation the document does not state, is a correct restate with the rule ([3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation)). - Record a defect whose fix is deferred where the project's other decisions are enumerable, rather than only mentioning it (section [2](../../SPEC.md#2-when-the-method-applies)). +- Take a baseline, a suppression or an instance left unfixed to the owner, never on the + practitioner's own authority (section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom)). +- Fix the original defect conventionally, with a test that reproduces it, once the defence is in + place (sections [2](../../SPEC.md#2-when-the-method-applies) and [6](../../SPEC.md#6-relationship-to-test-driven-development)). +- Write down, for any code the rule is narrowed to exclude, why the hazard cannot arise there, + and take the exclusion to the owner where that sentence cannot be written ([3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire), Part B). Wrong if it asserts: the practitioner fixes the reported instance first and defends afterwards; -a test may serve as the detector; an agent may baseline the remaining instances. +a test may serve as the detector; an agent may baseline the remaining instances; only a semantic +detector may host a rule, or no weaker kind is ever allowed. A bullet that +says the verdict on a remediation rests on reproduction (section [7](../../SPEC.md#7-conformance)) describes the reviewer, +not the practitioner, and is neither correct nor wrong. + +Wrong readings the text invites, which count against the document on one reader: none named. ## 2. Load-bearing statements -- **2a.** Attribute the defect to a class, express it as a detector rule, prove the rule fires, - then sweep and fix every instance, in that order. Key quotations, section [3](../../SPEC.md#3-the-method): "Six clauses, in +- **2a.** Required: (1) the four steps before the fix: attribute the defect to a class, express + the class as a rule in a detector, prove the rule fires, and sweep the codebase; (2) in that + order, with the fix after the sweep. Wrong if it has the fix before the rule, or the sweep + before the proof, or a test as the detector. Key quotations, section [3](../../SPEC.md#3-the-method): "Six clauses, in order."; [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance): "The Practitioner MUST run the Rule across the entire codebase and record the total Instance count before fixing anything, and MUST then fix every Instance found rather - than only the one". An answer that has the fix before the rule, or the sweep before the proof, - is wrong. -- **2b.** The owner of the codebase, always a human; an agent may not. Key quotation, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): - "An Agent MUST NOT Baseline, suppress or knowingly leave an Instance unfixed on its own - authority, whatever the Instance count turns out to be." An answer that lets the practitioner - decide because the count was small, or because the project recorded no rule, is wrong (the - default owner paragraph in the terminology entry for Owner still requires a recorded decision). -- **2c.** All six clauses of section [3](../../SPEC.md#3-the-method) were followed for that defect, and the verdict rests on the - reviewer reproducing the red and green runs, not on the report. Key quotations, section [7](../../SPEC.md#7-conformance): "A - remediation Conforms if all six clauses of section [3](../../SPEC.md#3-the-method) were followed for that Defect, and the - verdict that they were rests on reproduction"; "A verdict on a remediation or a Defence MUST - rest on reproduction, not on the report". An answer that accepts the pull request description - as the verdict is wrong, and so is one that states the six clauses and says nothing of - reproduction. + than only the one". An answer that names the steps by their clause numbers in order states + both elements; one that says only that section [3](../../SPEC.md#3-the-method)'s six clauses are followed in order, + naming no step, does not state element (1). +- **2b.** Required: (1) the owner of the codebase, the human who holds that authority; (2) no, + an agent may not do so on its own authority. Wrong if it lets the practitioner or an agent + decide because the count was small, because the project recorded no owner, or at all. Key + quotation, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): "An Agent MUST NOT Baseline, suppress or knowingly leave an Instance + unfixed on its own authority, whatever the Instance count turns out to be." "The owner" names + the defined term and states that the decision is a human's; an answer need not add "a human" + to be correct. The default-owner sentences of the terminology entry for Owner still require a + recorded decision, so an answer naming whoever instructed the practitioner, for a project that + has recorded nobody, names the owner and is correct. +- **2c.** Required: (1) all six clauses of section [3](../../SPEC.md#3-the-method) were followed for that defect; (2) the verdict + rests on reproduction, the reviewer rerunning the defence, not on the report. Wrong if it + accepts the pull request description or the practitioner's report as the verdict, or says + that most clauses suffice. Key quotations, section [7](../../SPEC.md#7-conformance): "A remediation Conforms if all six clauses + of section [3](../../SPEC.md#3-the-method) were followed for that Defect, and the verdict that they were rests on + reproduction"; "A verdict on a remediation or a Defence MUST rest on reproduction, not on the + report". "Reproduction" or "reproducing the runs" states element (2); the red and green runs + need not be named. +- **2d.** Required: (1) the strongest kind of detector that can express the rule, in the order + semantic, else syntactic, else textual; (2) the two grounds for a weaker kind, both: the + stronger kind cannot express the rule, or the language has no detector of the stronger kind + that accepts bespoke rules; (3) the grounds, which of the two holds and why, recorded in both + places, on the rule's remediation docs and in the description of a commit. Wrong if it says + any detector is as good as another, that a weaker kind may be used for a reason other than + the two grounds, that a weaker kind is never allowed, that the owner must always approve a + weaker kind (the practitioner judges it unless the project requires the owner's sign-off), or + that the two places are alternatives. Key quotations, [3.2](../../SPEC.md#32-build-the-net): "A Rule the Practitioner writes MUST be + written in the strongest kind of Detector that can express it: a Semantic one, else a + Syntactic one, else a Textual one."; "A weaker kind is allowed only where the stronger kind + cannot express the Rule within clause [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class)'s bounds, or where the language has no Detector of + the stronger kind that accepts bespoke Rules; those are the two grounds."; "Where that kind is weaker than Semantic, + the Practitioner MUST also record the grounds in two places, both of them: on the Rule's + Remediation docs, and in the description of a commit." Supporting: "On the Remediation docs of + a Rule they write, the Practitioner MUST name the Detector and its kind." Naming the stronger + detectors considered, and which commit it is, is explanation: an answer that gives it, or + leaves it out, is not marked on it. An answer that names the kinds in order without the words + "strongest" or "that can express it" states element (1). ## 3. Application: the ledger remediation -| Clause | Intended verdict | Reasoning the fixture supports | -| ----------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | Class, hazard and the next wider rule are named; Partial or No is accepted where the reader notes that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | -| [3.2](../../SPEC.md#32-build-the-net) | Yes | A rule in the project's static analyser, reading code, not a test | -| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes | Commit B is red on the fixture and the originating instance and survives as its own commit | -| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Five findings recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority; Partial accepted where the reasoning names that suppression | -| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | The rule is permanent and blocking, but the suppression was not a recorded decision; [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | -| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule; Partial accepted where the reasoning quotes the versioning sentence | - -Section [7](../../SPEC.md#7-conformance): the **remediation does not conform**, because [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) and [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) were not followed. The -**defence does not conform** either, because [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) fails; a reader who says the defence conforms -because the sweep is what failed has missed [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) and that counts against the document only if the -reasoning quotes the section [7](../../SPEC.md#7-conformance) definition of a conforming defence. - -Section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom): the **inline ignore on `Report::render()`** was the owner's decision, not the -practitioner's. An answer naming instead the decision not to build the wider rule is half right: -that is also listed in section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) as the owner's to take, and an answer naming both is correct. -An answer naming neither is wrong. - -A reader who grades [3.5](../../SPEC.md#35-enforce-permanently-and-block) Yes because the rule fails the build has not applied the clause's opening -sentence on unrecorded suppression, and that counts against the document. A reader who grades -[3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) Yes because the URL resolves has read the versioning sentence of [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) -differently, which counts against the document; a reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the -fixture. A reader who attributes the suppression to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) rather than [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) has read the narrowing -passage of [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) as covering it, which is a fair reading and is not marked wrong. +| Clause | Accepted grades | Condition on the reasoning | What the fixture supports | +| ----------------------------------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) | Yes, Partial or No | None | Class, hazard and the next wider rule are named; Partial or No is the reading that no independent search technique is recorded, which step 2 and the record [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class) requires both call for | +| [3.2](../../SPEC.md#32-build-the-net) | Partial or No | None | A rule in the project's static analyser, reading code, not a test, but the wiki page of commit D names neither the detector nor its kind; [3.2](../../SPEC.md#32-build-the-net): "On the Remediation docs of a Rule they write, the Practitioner MUST name the Detector and its kind." The fixture does not say what the analyser can see, so no grounds for a weaker kind are shown to be owed | +| [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) | Yes, Partial or No | For Partial or No: attributes the inline ignore to this clause, a fair reading of Part B | Commit B is red on the fixture and the originating instance and survives as its own commit | +| [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) | No or Partial | Mentions the instance left unfixed, the inline ignore, or `Report::render()` | Five findings were recorded, but one instance was left unfixed behind an inline ignore on the agent's own authority | +| [3.5](../../SPEC.md#35-enforce-permanently-and-block) | Partial or No | Mentions the suppression, the inline ignore, or that no recorded decision covers it | The rule is permanent and blocking, but the suppression was not a recorded decision. [3.5](../../SPEC.md#35-enforce-permanently-and-block): "neither does a Rule whose green run depends on a Suppression that no recorded decision covers" | +| [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) | No or Partial | Mentions the wiki, or that the documentation is not versioned or co-located with the rule | The message is terse and carries an identifier, but the documentation lives on a wiki, not versioned with the rule | + +The three answers the question then asks for, each with the condition its reason must meet: + +- **The remediation under section [7](../../SPEC.md#7-conformance).** Accepted answer: does not conform. Condition: the reason + names [3.2](../../SPEC.md#32-build-the-net), [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), [3.5](../../SPEC.md#35-enforce-permanently-and-block) or [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) among the clauses not followed; other clauses named alongside are not + marked. +- **The defence alone.** Accepted answer: does not conform. Condition: the reason names [3.2](../../SPEC.md#32-build-the-net), + [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation), or [3.5](../../SPEC.md#35-enforce-permanently-and-block)'s unrecorded suppression, which section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence + also requires to be absent; other clauses named alongside are not marked. +- **The decision section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) says was not the practitioner's.** Accepted answer: the inline ignore on + `Report::render()`, which was the owner's decision. Condition: none; any other decision named + alongside it is not marked. An answer naming only the decision to leave the wider rule unbuilt, + or naming none, is wrong. + +Wrong readings the text invites, which count against the document on one reader: + +- [3.5](../../SPEC.md#35-enforce-permanently-and-block) graded Yes with reasoning that says the rule blocks, or is permanent, and does not mention + the suppression. +- [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation) graded Yes with reasoning that says the identifier resolves, or the URL works, and does not + mention versioning or co-location. +- The defence graded as conforming with reasoning that says only the sweep failed and that + quotes section [7](../../SPEC.md#7-conformance)'s definition of a conforming defence. +- [3.2](../../SPEC.md#32-build-the-net) graded Yes with reasoning that says the rule is in a detector that reads code, or not a + test, and does not mention the detector's kind or the naming of the detector on the + remediation docs, which has read [3.2](../../SPEC.md#32-build-the-net) without its naming sentence. A reader who grades [3.2](../../SPEC.md#32-build-the-net) + Partial or No only because no grounds for a weaker kind are recorded has the grade on a + ground the fixture does not show, and is correct on the grade. + +A reader who grades [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance) Yes has misread the fixture, which is the reader's miss, whether or not +the inline ignore was attributed to [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire) instead. ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage named, as `ACCEPTANCE.md` defines naming and the same +passage. A passage named by two or more readers of the run is a finding: it blocks the change +where it lies in text the change wrote or altered, and is routed to `acceptance/ROUTED.md` +otherwise. What to change is the editor's decision; the reader is not asked and any suggestion +it volunteers is discarded unread. diff --git a/acceptance/method/QUESTIONS.md b/acceptance/method/QUESTIONS.md index 072a3c0..ba0efaa 100644 --- a/acceptance/method/QUESTIONS.md +++ b/acceptance/method/QUESTIONS.md @@ -11,20 +11,24 @@ do when they are handed a defect? ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. +Answer each part of each question in a sentence or two, then quote the sentence or sentences of +the document you are relying on. - 2a. What does a practitioner do before fixing the defect they were handed, and in what order? - 2b. Who may adopt a baseline or suppress an instance, and may an agent do so on its own authority? - 2c. What does it mean for a remediation to conform, and on what must a reviewer's verdict on it rest? +- 2d. In which kind of detector must a rule the practitioner writes be written, and in what order + are the kinds tried; on what grounds may a weaker kind be used; and what must the practitioner + record then, and where? ## 3. Application Grade the ledger remediation, as described in the fixture, against each of the six clauses of section [3](../../SPEC.md#3-the-method): [3.1](../../SPEC.md#31-attribute-the-defect-to-a-class), [3.2](../../SPEC.md#32-build-the-net), [3.3](../../SPEC.md#33-prove-the-net-by-making-the-rule-fire), [3.4](../../SPEC.md#34-sweep-the-codebase-then-fix-every-instance), [3.5](../../SPEC.md#35-enforce-permanently-and-block), [3.6](../../SPEC.md#36-make-the-failure-message-terse-and-point-it-at-real-documentation). For each give Yes, Partial or No and one line of -reasoning. Then state whether the remediation conforms under section [7](../../SPEC.md#7-conformance), whether the defence -alone conforms, and which decision in the fixture, if any, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) says was not the +reasoning. Then state whether the remediation conforms under section [7](../../SPEC.md#7-conformance) and why, whether the defence +alone conforms and why, and which decision in the fixture, if any, section [4](../../SPEC.md#4-authority-which-decisions-belong-to-whom) says was not the practitioner's to take. ## 4. Confusion diff --git a/acceptance/runs/019-semantic-detectors-run-1.md b/acceptance/runs/019-semantic-detectors-run-1.md new file mode 100644 index 0000000..d2ff6e7 --- /dev/null +++ b/acceptance/runs/019-semantic-detectors-run-1.md @@ -0,0 +1,190 @@ +# Run 19: SPEC.md, semantic detectors, first run + +- **Document**: SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + branch `editor/semantic-detectors` at 5b6093f, the commit the readers were given +- **Cohort**: five readers, model alias `haiku`; each reader reported `claude-haiku-5-5` +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input; no tools, no settings, no project context. + Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt` +- **Managed settings**: the machine applies no managed (policy) settings to the readers; no + managed settings file is present +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. Every sheet has answer text, no process reports an error + (`is_error` false, subtype `success`), and every reader reported the cohort's model. The editor + reports no reader set aside for a cause the sheets cannot show. +- **Run within the pull request**: 1, counting void runs; runs in the pull request so far: 1 +- **Marker**: delegated, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with the + marking instruction and its agent definition, having seen neither the drafting nor the review. + It read `ACCEPTANCE.md` and `acceptance/runs/README.md` in this worktree, which differ from + commit a695010 only by this pull request's question-set line for 2d, and the document, question + set, key and fixture at 5b6093f. The packet's three files match that commit, apart from one + blank line between the end of each file and the next file's marker line. + +## Marks + +`ok` is correct, `miss` is a miss; `T` marks a miss traceable to the text; `J1` and `J2` mark a +miss that rests on the judgement of that name below. + +| Reader | Q1 | 2a | 2b | 2c | 2d | 3.1 | 3.2 | 3.3 | 3.4 | 3.5 | 3.6 | Remediation | Defence | Section 4 | Q4 passages | +| ------ | ------- | --------- | --------- | --------- | ------ | ------- | ------- | -------------- | --- | --- | --- | ----------- | ------- | --------- | ----------- | +| 1 | correct | miss T J2 | miss T J1 | miss T J2 | miss T | Partial | Partial | Partial miss T | No | No | No | ok | ok | ok | two | +| 2 | correct | miss T | miss T J1 | miss T J2 | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 3 | correct | miss T | ok | miss T | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 4 | correct | miss T J2 | miss T J2 | miss T | miss T | No | Partial | Partial miss T | No | No | No | ok | ok | ok | three | +| 5 | correct | miss T J2 | ok | miss T J2 | miss T | Partial | Partial | Partial miss T | No | No | No | ok | ok | ok | three | + +Every grade under 3.1, 3.2 and 3.4 to 3.6 is one the key accepts, and every 3.1 grade of Partial +or No notes that no independent search technique is recorded. + +### Question 1 + +Five of five correct. Every sheet has at least four bullets naming obligations the key lists, and +no bullet asserts an obligation the document does not state. Reader 1's "neither a single +instance nor every use of a language feature" is supported by section 7 ("drawn to a [Class] +within both bounds and not to the reported [Instance]"); reader 4's search "chosen and run before +the rule exists" is supported by clause 3.1, which has the search recorded before clause 3.2 +begins. + +### Question 2 + +Every reader quoted words of the document for every item, so every miss here is traceable to the +text. + +- **2a**, none of five. The sentence states the order only partly or not at all: + - Reader 1 states class, defence built, rule proven, sweep, every instance fixed, in order; it + says the rule "is proven", not that it fires, and "the defence" rather than a detector rule + (J2). + - Reader 2: "attribute the defect to a class and build and prove the defence"; no sweep and no + fixing of every instance. + - Reader 3: "The Defence is built and proven first, then the Defect is fixed"; no class + attribution, no sweep, no fixing of every instance. + - Reader 4: "then sweep, then fix"; the fix is not stated as every instance (J2). + - Reader 5: "sweeps for every instance before fixing anything, then fixes the defect"; the fix + is stated as the defect, not every instance (J2). Its clause 3.4 quotation after the first + quotation is not part of the sentence (J1). +- **2b**, two of five (readers 3 and 5, who say the owner "is always a human" in the sentence). + Readers 1 and 2 say the owner is "Always a human" only after the quotation (J1); reader 4 does + not say it (J2: "the owner" is not read as carrying its definition). All five quote the key's + section 4 sentence and say an agent may not. +- **2c**, none of five. No sentence says the reviewer reproduces the red and green runs; each says + only that the verdict rests on "reproduction" (J2). Readers 3 and 4 also leave "not on the + report" out of the sentence: reader 3 has it only in its quotation, and reader 4's sentence + ends "The verdict rests on reproduction:" before the quotation. No reader accepts the pull + request description as the verdict. +- **2d**, none of five. No sentence states the second ground for a weaker kind, that the language + has no detector of the stronger kind that accepts bespoke rules: + - Reader 1: "a weaker kind allowed only where the stronger cannot"; one ground; no order of + kinds and no place for the record in the sentence (J2 for the order). + - Reader 2: "A weaker kind is allowed only where the stronger cannot express it"; one ground, + though its quotation gives both; no order and no place (J2 for the order). + - Reader 3: "only where the stronger one cannot express the Rule"; one ground; no place in the + sentence, only in the quotation. + - Reader 4: the sentence before the quotation states only the strongest kind; it states no + ground at all, and the record and its places come after the quotation (J1). + - Reader 5: "only when a stronger one cannot express the rule", then "which of the two + reasons holds and why" without stating the second; places only after the quotation (J1). + Whether (iii) needs a place named is a judgement (J3); it changes no mark, since every sheet + already misses (ii). + +### Question 3 + +- **3.3**, two of five (readers 2 and 3, Yes). Readers 1, 4 and 5 grade Partial because the + inline ignore on `Report::render()` is a Suppression and not a Narrowing under clause 3.3's + Part B, and also fail 3.4 on it. The key excuses a reader who attributes the suppression to 3.3 + "rather than" 3.4; these readers attribute it to both, which that sentence does not cover, so + each is a miss. Each reasoning states what the document requires (reader 1: "a suppression, + with no searched sentence, and it was not referred"; reader 4: "That makes it a Suppression + under 3.3, not a Narrowing"; reader 5: "its sentence does not address the hazard"), so each is + traceable. +- **Remediation**: five of five say it does not conform, each naming at least one failing clause + the key accepts. +- **Defence**: five of five say it does not conform, each naming 3.6. +- **Section 4**: five of five name the inline ignore on `Report::render()`; readers 1, 2, 4 and 5 also + mention the unbuilt wider rule as a second or arguable decision, which the key accepts alongside + it. + +### Judgements made in marking + +Each takes the stricter reading, the one that records more misses, as `ACCEPTANCE.md` requires +where its rules leave a judgement open. + +- **J1. The answer's sentence is what precedes its quotation.** Question 2 asks for one sentence + followed by a quotation, and a mark turns on "its sentence". Text the reader gives before the + quotation is read as the sentence however it is punctuated, since the questions ask two or + three things and nothing picks out one full stop; text after the quotation is outside the form + the question asks for and is not read as part of the sentence. Affects 2b (readers 1, 2), 2d + (readers 4, 5) and 2a (reader 5). +- **J2. An element of the key's answer is stated only when the sentence says it.** A less + specific word does not state the more specific element ("fix" is not "fix every instance", + "reproduction" is not "reproducing the red and green runs", "proven" is not "fires", "the + strongest kind" is not the order semantic, syntactic, textual), and a defined term is not read + as carrying its definition into the sentence ("the owner" is not "always a human", "the + defence" is not "a detector rule"). Affects 2a (readers 1, 4, 5), 2b (reader 4), 2c (readers + 1, 2, 5) and the order in 2d (readers 1, 2, 4). Runs 17 and 18 accepted "reproduction" for 2c; + those records are examples of style, not of marking. +- **J3. 2d's element (iii) needs a place named.** The key says naming one of the two places "is + enough", read as requiring at least one. No mark turns on it. +- **The 3.3 excuse is read as written**, covering a reader who attributes the suppression to 3.3 + rather than 3.4, not one who attributes it to both. +- **Not treated as open: the remediation verdict.** The key's section 7 line gives "does not + conform" as the intended answer, and its "because 3.2, 3.4 and 3.6" is reasoning, not a list a + reader must reproduce: it leaves out 3.5, which the key also has failing, and where the key + wants a clause named, for the defence, it says so. + +## Passages quoted as confusing + +Same passage as `ACCEPTANCE.md` defines it: sentences in common, or the same paragraph. + +| Passage | Clause | Readers | Words quoted, verbatim | +| ------------------------------------------------------------------- | -------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| The level paragraph, "Both bounds measure breadth within one level" | 3.1 | three: 1, 2, 3 | 1 and 3: "Both bounds measure breadth within one level; the level itself is a separate question, answered against the report." 2: "Where the [Defect] was reported as a behaviour, ... the [Class] a [Detector] can read usually sits one level below that \[Hazard\]: the mechanism of this [Instance], not the failure the report opens with." The bold sentence and the "Where the [Defect]" sentence are one paragraph. | +| The extension-points sentence | 3.2 | three: 3, 4, 5 | "a [Rule] whose construct its extension points cannot reach is one that [Detector] cannot express, and one its kind cannot express where no other [Detector] of that kind reaches it." | +| The search vocabulary in "Attempt rather than pre-judge" | 2 | two: 2, 5 | "naming at least two independent techniques tried at expressing the [Rule], in the vocabulary clause 3.1 uses for a search" | +| The scale at which a baseline is allowed | 3.4 | two: 4, 5 | "and the scale at which that becomes true is tremendous." | +| The default owner | 1, Owner | one: 1 | "Where the project has recorded nobody, the [Owner] is whoever instructed the [Practitioner], and [Escalation] goes there." | +| The limit on the wider-rule bullet | 4 | one: 4 | "This does not reopen a [Class] whose bounds a [Hazard] sentence has settled, and it reaches no further than the one wider [Rule] already on the record." | + +Of the four passages named by two or more, only the clause 3.2 sentence is text this pull request +adds; the other three predate it. The criterion does not distinguish, so all four are findings. +Reader 2's remarks on what it could not tell are not suggestions and are not recorded as +passages. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in 87536c5, 66301b9 and 3b848c7 +(key quotations kept verbatim in 85110c1 and ba8a0bd), after an independent review of each round of +edits. SPEC.md: clause 3.2 sets out the two grounds for a weaker kind as a numbered pair and says +the extension-points rule in two plain sentences; clause 3.3 Part B says an instance left behind an +inline ignore is a Suppression, governed by 3.4, 3.5 and section 4, not a Narrowing; clause 3.1 says +what a class's level is; section 2 says what independent techniques are; clause 3.4 says plainly +that a baseline is rare. Separately, pull request 13 changed how question 2 is marked (sentence and +quotation read together, each element for its gist) after this run, so the misses that rested on +judgements J1 and J2 needed no text change. A fresh cohort follows. + +## Process change + +Made under a process change before it merged: yes, pull request 13 at commit a695010, named +before the readers started. + +## Verdict + +**FAIL.** Against the criteria in `ACCEPTANCE.md`: + +- Load-bearing statements: not met. 2a none of five, 2b two of five, 2c none of five, 2d none of + five. Without J1 and J2 the result is still a fail: 2d none of five on the second ground, 2a + three of five (readers 2 and 3 miss), 2c three of five (readers 3 and 4 miss). +- Restatement and application: not met on 3.3, two of five; every other keyed item five of five. +- Misses traceable to the text: every miss on question 2, and 3.3 for readers 1, 4 and 5; each is + a finding. +- Passages found confusing by two or more: four, each a finding that blocks until the text is + changed and a fresh cohort no longer names it: clause 3.1's level paragraph, clause 3.2's + extension-points sentence, section 2's search-vocabulary sentence and clause 3.4's + "tremendous" sentence. +- Single-reader notes, recorded: section 1's default owner (reader 1); section 4's limit on the + wider-rule bullet (reader 4). + +Answer sheets, the reader prompt, the command and the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090071898), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090072435), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090072816), [comment 4](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090073231). diff --git a/acceptance/runs/020-semantic-detectors-run-2.md b/acceptance/runs/020-semantic-detectors-run-2.md new file mode 100644 index 0000000..83e9795 --- /dev/null +++ b/acceptance/runs/020-semantic-detectors-run-2.md @@ -0,0 +1,191 @@ +# Run 20: DETECTOR-SPEC.md, semantic detectors, second run in pull request 14 + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + editor's branch `editor/semantic-detectors` at 5b6093f; the question set, key and fixture read + at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input, with no tools, no settings, no project context + and no saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is the question set, the document and the fixture at + 5b6093f, in that order, each preceded by one `===== FILE: =====` line and nothing else + added. +- **Managed (policy) settings**: none; the machine applies no managed settings and no managed + settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. The editor reports none set aside for a cause the sheets cannot + show; every sheet has answer text, no process reports an error (`is_error` false, subtype + `success`), and every reported model is the cohort's model +- **Run within the pull request**: 2, counting void runs; run 19, on SPEC.md, was made at the same + time and does not bear on this document +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review +- **Verdict**: FAIL + +## Marks + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | Section 8 | Kind | 6.4 | Q4 passages | +| ------ | ------- | ---- | ---- | ---- | --- | --- | --- | --- | --- | ---- | --- | ---- | --- | --- | --- | --------- | ---------- | --- | ----------- | +| 1 | correct | miss | miss | miss | ok | ok | ok | ok | ok | miss | ok | miss | ok | ok | ok | ok | not marked | ok | 2 | +| 2 | correct | miss | miss | ok | ok | ok | ok | ok | ok | miss | ok | miss | ok | ok | ok | ok | not marked | ok | 1 | +| 3 | correct | ok | miss | miss | ok | ok | ok | ok | ok | miss | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 4 | correct | miss | miss | miss | ok | ok | ok | ok | ok | miss | ok | miss | ok | ok | ok | ok | not marked | ok | 4 | +| 5 | correct | miss | miss | miss | ok | ok | ok | ok | ok | miss | ok | miss | ok | ok | ok | ok | not marked | ok | 2 | + +Every miss above is traceable to the text: each one's reasoning quotes the document or states +what a clause requires. No question was left unanswered. + +### Question 1 + +All five restatements name four or more obligations from the key's list (4.1, 4.2, 4.3, 5.1 to +5.4, 6.1 to 6.3, 7.1), and none asserts an obligation the document does not state. Correct, five +of five. + +### Question 2 + +The key's answers state these things, and the key says none of them may be left out: + +- 2a: (a) Conformance is satisfying every MUST in sections 4 to 7; (b) judged on evidence; (c) a + detector that predates the document can conform; (d) the declaration is optional. +- 2b: (a) make the route disableable by configuration, or (b) detectable by a Rule the project can + write or a mechanical check the Detector documents; (c) the detector is not required to remove or + forbid it. +- 2c: (a) with the rule; (b) in the installed copy; (c) at a version tracked together; (d) yes, + reachable offline. + +| Reader | 2a | 2b | 2c | +| ------ | -------------------------------------------- | -------------------------------------------------------------------- | -------------------------------------- | +| 1 | miss: (b) only in the quotation | miss: (c) not stated; the means of detection only in the quotation | miss: (a) not stated (judgement 3) | +| 2 | miss: (b) and (d) only in the quotation | miss: (c) not stated; the means of detection only in the quotation | ok: all four, supported by 6.2 and 6.3 | +| 3 | ok: all four, supported by the 8.1 quotation | miss: (c) not stated; the means of detection only in the quotation | miss: (c) not stated | +| 4 | miss: (d) not stated (judgement 2) | miss: (c) not stated; the quotation states no key item (judgement 4) | miss: (b) not stated (judgement 3) | +| 5 | miss: (d) not stated (judgement 2) | miss: (c) not stated; the quotation states no key item (judgement 4) | miss: (c) not stated | + +Each miss is traceable: every reader's answer quotes the document in support. + +### Question 3 + +The key's intended verdicts are Yes for every clause but 6.3, which is No; the section 8 verdict +is "does not conform", because 6.3 fails; 6.4 does not change the verdict. + +- **5.3, all five readers Partial against Yes.** Each grades 5.3 Partial because the fixture does + not say the command's output gives the location of `.lintomatic/report.html`. Readers 2, 4 and 5 + state the requirement outright ("the clause requires both halves"; "which the clause requires + alongside it"; "5.3 requires it to"); reader 1 uses the clause's words "usable summary"; reader + 3 reports the fixture's silence in the row graded against 5.3 (judgement 5). All five are + traceable to the text. +- **6.1, readers 1, 2, 4 and 5 Partial against Yes.** Each holds that the two bundled identifiers + printing `No documentation declared` do not resolve to documentation, which each attributes to + 6.1 ("their identifiers resolve to nothing"; "which is not documentation"; "which resolves to no + documentation"). Traceable. Reader 3 graded Yes. +- **6.3**: No from all five, each citing that one unresolved identifier fails the clause. No reader + graded it Partial on the grounds that thirty-eight resolve. +- **Section 8**: all five say Lintomatic does not conform and name 6.3 as a failure; readers 1, 2, + 4 and 5 also count their 5.3 and 6.1 Partials, and reader 3 its 5.3 Partial, which repeats those + misses but does not change the verdict marked here. +- **Kind**: all five say the fixture does not let them name the kind and name none; reader 4 says + outright that this does not change the verdict. No reader made the verdict turn on the kind, so + none is marked on it. +- **6.4**: all five say it does not change the verdict, because it is a SHOULD. + +### Judgements made in marking + +Each takes the stricter reading, the one recording more misses or findings, as `ACCEPTANCE.md` +requires where its rules leave a mark open. + +1. **What counts as "its sentence" under question 2.** The rule asks that the answer's sentence + state each thing the key's answer states and that a quotation support it, so a key item that + appears only inside the reader's quotation is not stated by the sentence. Where a reader wrote + two sentences of answer before or between its quotations, both are taken as the answer. +2. **2a (d), readers 4 and 5.** Reader 4 says a detector "can be graded on evidence without its + maintainer knowing the document", and reader 5 that it is "judged on evidence rather than a + declaration". Each implies that no declaration is needed; neither says the declaration is + optional. Marked not stated. +3. **2c (a) and (b), readers 1 and 4.** Reader 1 says the documentation "must ship in the + installed package, at a version tracked with the rule", which does not say it ships with the + rule; reader 4 says it "must ship on disk with the rule", which does not say the installed + copy. Each marked not stated. +4. **2b quotations, readers 4 and 5.** Both quote only "For each one it MUST do at least one of + two things, and either alone satisfies this clause", which names neither thing and does not say + that forbidding is not required, so it does not by itself state a key item. A further reason + for the miss, already a miss on (c). For readers 1 to 4, "make it detectable" without its + means is also taken as short of (b). +5. **5.3, reader 3.** Its reasoning reports only the fixture's silence on the report's location, + but it is a grade against clause 5.3 that the clause is partly unmet, and naming the clause a + case fails attributes a requirement to it. Traceable. +6. **Question 4, a clause named as the context of a confusion.** Readers 1, 3 and 4 explain their + 7.1 confusion by naming clause 4.1 ("Clause 4.1 has the project writing its own rules"; "does + not fit 4.1's authoring route"; "given 4.1 describes project rules as separate"). Each is taken + as naming clause 4.1, which covers every sentence of it. Likewise reader 3's "given that the + declaration is optional" is taken as naming the 8.1 sentence "The declaration is optional.", + and reader 4's "the table in section 2" as naming that table. +7. **Question 4, single words quoted.** Reader 5's "optional" and "not a condition of" are + quotations under question 4, and each covers the 8.1 sentence "The declaration is optional." + or "It is how a maintainer claims Conformance; it is not a condition of it.", the only places + those words occur. + +No re-mark. + +## Passages quoted as confusing + +Grouped by paragraph, as `ACCEPTANCE.md` defines the same passage. + +| Clause and paragraph | Readers | Words quoted, verbatim | Blocks | +| --------------------------------------------------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------ | +| 7.1, first paragraph, the sentence beginning "For each one it MUST" | 1, 2, 3, 4, 5 | 1, 2: "make the route detectable, by a [Rule] the project can write in the [Detector] itself or by a mechanical check the [Detector] documents." 3, 4, 5: "by a [Rule] the project can write in the [Detector] itself." / "make the route detectable, by a [Rule] the project can write in the [Detector] itself" / "make the route detectable, by a [Rule] the project can write in the [Detector] itself." | Yes | +| 8.1, first paragraph, the sentence beginning "The same declaration" | 3, 4, 5 | 3, 4: "a [Detector] that has learnt, from its own checks or from a [Practitioner]'s report under the method's clause 3.2, that it fails a MUST in sections 4 to 7 MUST record that gap alongside the version it declares" (reader 3 with a closing full stop). 5: "a [Detector] that has learnt … that it fails a MUST in sections 4 to 7 MUST record that gap alongside the version it declares." | Yes | +| 8.1, second paragraph, "The declaration is optional." and the next sentence | 3, 4, 5 | 4: "The declaration is optional." 5: "optional"; "not a condition of". 3, by description: "given that the declaration is optional" (judgements 6 and 7) | Yes | +| 6.1, the paragraph beginning "For a [Bundled rule]", its last sentence | 1, 4 | 1, 4: "A [Message] that carries its own documentation path resolves that [Message], not the [Identifier]." | Yes | +| 4.1, named by clause number | 1, 3, 4 | No quotation; named as the context of the 7.1 confusion (judgement 6) | Yes | +| 6.1, the paragraph beginning "For a project's own [Rule]" | 4 | "Where the [Identifier] is itself a URL, as method clause 3.6 allows" | No | +| 1, second paragraph, the sentence beginning "The two are separable" | 2 | "installed into projects its maintainer will never see" | No | +| 2, the terminology table | 4 | No quotation; "the table in section 2", named as not defining "Message" (judgement 6) | No | + +Of these, only the section 2 table lies in text this pull request changed; the passages that block +all lie in text it kept. + +## Against the criteria + +- **Load-bearing statements (every reader on every one)**: not met. 2a one of five, 2b none of + five, 2c one of five. +- **Restatement and application (all but one on each keyed item)**: not met. Question 1 five of + five; 5.3 none of five; 6.1 one of five; every other clause, the section 8 verdict and 6.4 five + of five. +- **A miss traceable to the text (none)**: not met. Twenty-two, listed per reader above: 2a by + readers 1, 2, 4 and 5; 2b by all five; 2c by readers 1, 3, 4 and 5; 5.3 by all five; 6.1 by + readers 1, 2, 4 and 5. Each is a finding. +- **A passage found confusing by two or more (none outstanding)**: not met. Five passages are + named by two or more readers, as the table above shows. +- **From one reader only**: the 6.1 paragraph on a project's own rule, the section 1 sentence and + the section 2 table, recorded. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in 87536c5, 66301b9 and 3b848c7 +(key quotations kept verbatim in 85110c1 and ba8a0bd), after an independent review of each round of +edits. DETECTOR-SPEC.md: 5.3 says what the command's own output must carry, and that where its +summary leaves a finding out it must say where those findings can be read (an obligation change, +recorded in the changelog, so the text says what the key's intended Yes assumed); 6.1 asks for the +resolution mechanism, and missing bundled documentation fails 6.3, with 6.4's Why aligned; 7.1 says +the project writes the detecting rule in the detector itself and the detector need not remove or +forbid the route; 8.1 says what a gap is and that conformance is judged on evidence, with or without +a declaration. Separately, pull request 13 changed how question 2 is marked after this run, so the +misses that rested on judgements 1 to 4 needed no text change. A fresh cohort follows. + +## Process change + +Made under the process change of pull request 13 before it merged, at commit a695010, named +before the readers started; `ACCEPTANCE.md` and `acceptance/runs/README.md` in this worktree +differ from a695010 only by this pull request's question-set line adding question 2d for SPEC.md, +which does not bear on this document. + +## Verdict + +FAIL on every blocking criterion. Answer sheets, the readers' prompt and command, and the +marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090073628), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090074094), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090074397). diff --git a/acceptance/runs/021-semantic-detectors-run-3.md b/acceptance/runs/021-semantic-detectors-run-3.md new file mode 100644 index 0000000..7535f10 --- /dev/null +++ b/acceptance/runs/021-semantic-detectors-run-3.md @@ -0,0 +1,241 @@ +# Run 21: SPEC.md, semantic detectors, third run in pull request 14 + +- **Document**: SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + branch `editor/semantic-detectors` at 3aefdd1, the commit the readers were given; the question + set, key and fixture read at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input; no tools, no settings, no project context and no + saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is the question set, the document and the fixture at + 3aefdd1, in that order, each preceded by one `===== FILE: =====` line, with nothing + else added apart from one blank line between the end of each file and the next file's marker + line. +- **Managed (policy) settings**: none; the machine applies no managed settings to the readers and + no managed settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. Every sheet has answer text, no process reports an error + (`is_error` false, subtype `success`), and every reported model is the cohort's model. The + editor reports no reader set aside for a cause the sheets cannot show. +- **Run within the pull request**: 3, counting void runs; runs in the pull request so far: 3 (19, + 20 and this one). Run 22, on DETECTOR-SPEC.md, is made at the same time and does not bear on + this document. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review. It read `ACCEPTANCE.md` and + `acceptance/runs/README.md` in this worktree, which differ from commit 204f95b only by this pull + request's question-set line adding 2d for SPEC.md; the document, question set, key and fixture + at 3aefdd1; runs 19 and 20; and, for the fourth pass-criteria row, SPEC.md at 5b6093f, the + document run 19 read. + +## Marks + +`ok` is correct, `miss` is a miss; `T` marks a miss traceable to the text. Judgements J1 to J8 +are set out below. + +| Reader | Q1 | 2a | 2b | 2c | 2d | 3.1 | 3.2 | 3.3 | 3.4 | 3.5 | 3.6 | Remediation | Defence | Section 4 | Q4 passages | +| ------ | ------- | ------ | --- | --- | ------ | ------- | ------- | -------------- | --- | --- | --- | ----------- | ------- | --------- | ----------- | +| 1 | correct | miss T | ok | ok | miss T | Partial | Partial | Yes ok | No | No | No | ok | ok | ok | four | +| 2 | correct | ok | ok | ok | miss T | Partial | Partial | Partial miss T | No | No | No | ok | ok | ok | four | +| 3 | correct | miss T | ok | ok | miss T | Partial | Partial | Partial miss T | No | No | No | ok | ok | ok | two | +| 4 | correct | miss T | ok | ok | miss T | Partial | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 5 | correct | ok | ok | ok | miss T | Partial | Partial | Yes ok | No | No | No | ok | ok | ok | three | + +Every grade under 3.1, 3.2 and 3.4 to 3.6 is one the key accepts, and every 3.1 grade of Partial +notes that no independent search technique is recorded. + +### Question 1 + +Five of five correct. Every sheet has at least four bullets naming obligations the key lists, and +no bullet asserts an obligation the document does not state. Reader 3's "Write a custom rule" is +Appendix A's own wording ("Write a custom [Rule] that detects the [Class]"), and the "only then" +of readers 1 and 4, that the original defect is fixed last with a reproducing test, is Appendix +A's closing sentence. + +### Question 2 + +Every reader quoted words of the document for every item, so every miss here is traceable to the +text. + +- **2a**, two of five (readers 2 and 5). Every reader quotes the section 2 sentence "When it can, + building the [Defence] is not optional and MUST be attempted before the [Defect] is fixed.", which + states no order and no sweep, so each answer stands or falls on its sentence. + - Reader 1: "attributes it to a class, builds and proves a rule for that class, then sweeps and + counts before fixing (clauses 3.1 to 3.4). The original defect is fixed last"; the fix of + every instance is not stated, and naming clause 3.4 does not state it. + - Reader 2: class, "builds and proves the net", "sweeps and fixes every instance", in order; + correct (J2). + - Reader 3: "builds the defence before fixing the defect"; no class, no proof, no sweep, no fix + of every instance. + - Reader 4: "attributes the defect to a class and attempts the rule before fixing the defect, in + the order of section 3"; no proof, no sweep, no fix of every instance. + - Reader 5: class, rule, proof, "sweep and fix every instance", in order; correct. +- **2b**, five of five. Readers 1, 2 and 3 say the owner is a human; readers 4 and 5 say "the + owner of the codebase" and "the owner" (J4). All five quote the key's section 4 sentence and say + an agent may not. +- **2c**, five of five. Readers 1, 4 and 5 state the reviewer's rerun of the red and green runs, + reader 4 in its quotation of section 7's closing paragraph; readers 2 and 3 state reproduction, + not the report (J3). No reader accepts the pull request description as the verdict. +- **2d**, none of five. No answer, sentence and quotation read together, states the two grounds + on which a weaker kind is allowed: each says only that there are two. + - Reader 1: "A weaker kind is allowed only on one of two grounds"; its quotations give the order + and the record with both places, not the grounds. + - Reader 2: "A weaker kind is allowed only on one of two grounds"; no place for the record of + the grounds, only for naming the detector and kind (J5). + - Reader 3: "A weaker kind is allowed only on two grounds"; its second quotation stops before + the places (J5). + - Reader 4: "only on one of two grounds", quoting "A weaker kind is allowed on one of two + grounds only:"; nor does its answer state the order semantic, syntactic, textual, and it names + no place (J5). + - Reader 5: "only on one of two grounds"; no place (J5). + +### Question 3 + +- **3.3**, three of five (readers 1, 4 and 5, Yes). Reader 1's caveat, that the description does + not show how the fixture was excluded from the enforced run, leaves its grade at Yes. + - Reader 2 grades Partial because "Part B is breached: the `Report::render()` inline ignore is + a suppression, not a narrowing", and also grades 3.4 No on it. Its reasoning states what the + document requires of Part B, so the miss is traceable. Part B now opens by saying such an + ignore is a [Suppression] "and clauses 3.4 and 3.5 and section 4 govern it, not this clause"; + the key's excuse does not cover a reader who attributes it to both (J6). + - Reader 3 grades Partial because "the enforced run reports the fixture as one of its five + findings, and Part C says the enforced run 'does not report it'". It quotes clause 3.3's + sentence on a [Rule]'s [Fixtures], so the miss is traceable. +- **Remediation**: five of five say it does not conform, each naming at least one failing clause + the key accepts. +- **Defence**: five of five say it does not conform, each naming 3.6. +- **Section 4**: five of five name the inline ignore on `Report::render()`. Readers 1, 2, 4 and 5 + also raise the unbuilt wider rule as a second or less clear decision, which the key accepts + alongside it. + +### Judgements made in marking + +Where the rules leave a judgement open, the stricter reading, the one that records more misses or +more passages, is taken. Where a rule settles the point, the record says which. + +- **J1. The sentence.** Where a reader wrote more than one sentence before its quotation, all of + that text is read with the quotation as the answer. No mark turns on it: in each such case the + quotation states what the later sentences add, or the answer misses an element no sentence + states. +- **J2. "The net" (reader 2, 2a).** Settled by the rule that an element may be stated in the + document's words: "Build the net" and "Prove the net by making the rule fire" are the headings + of clauses 3.2 and 3.3, and the net is the [Rule] in a [Detector]. Read otherwise, reader 2 + would add one traceable miss on 2a; the verdict would not change. +- **J3. "Reproduction" (readers 2 and 3, 2c).** Settled by the rule that each element is read + for its gist, with sentence and quotation together: both quotations are section 7's sentence + that the verdict "rests on reproduction, as this section closes by requiring, not on the + report", and the closing paragraph it points to is the red and green reruns. The key's own + condition for a wrong answer is one that omits reproduction. +- **J4. "The owner" (readers 4 and 5, 2b).** Settled by the rule that a defined term used in its + defined sense states what it means: [Owner] is "Always a human", and section 4 and the key + both call it the owner of the codebase. Reader 4's question 4 doubt whether "whoever owns the + codebase" is the defined term is not read into its 2b answer, which is its sentence and + quotation; that doubt is recorded as a passage below. +- **J5. 2d's element (iii) needs a place named.** The key says naming one of the two places "is + enough", read as requiring at least one, as run 19 read it. No mark turns on it: every sheet + already misses (ii). +- **J6. The 3.3 excuse is read as written**, covering a reader who attributes the suppression to + 3.3 rather than 3.4, not one who attributes it to both, as run 19 read it. +- **J7. Reader 1's "its owner list".** Reader 1's first passage quotes table row 3.1 and section + 4's first practitioner bullet, and describes section 4's owner list as covering "only the wider + rule left unbuilt because no instance exists". The description is read as naming the whole + owner list, its lead-in included, so it covers the wider-rule bullet and the line "Decisions + that MUST go to whoever owns the codebase:". The narrower reading, the wider-rule bullet alone, + would leave the lead-in to reader 4 alone and remove one finding below. Reader 1's other + descriptions are read the same way: "the two grounds" covers both items of clause 3.2's list, + and "Clause 3.4, first paragraph against the Fixture paragraph" covers 3.4's opening paragraph + as well as the scope paragraph it quotes. Reader 3's "on the second ground" covers the second + item of that list. +- **J8. Clause labels.** A clause number given with a quotation, as question 4 asks, is read as + the quotation's label, not as naming the whole clause. Settled by the rule on quoting: naming a + clause number is not quoting, and question 4 asks for a quotation "with its clause number". + +## Passages quoted as confusing + +Same passage as `ACCEPTANCE.md` defines it: sentences in common, or the same paragraph; a list +item or table row is its own paragraph. + +| Passage | Clause | Readers | Words quoted, verbatim, or the description used | +| --------------------------------------------------------------------- | ------------------------ | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| The two grounds and the paragraph after them, "For the second ground" | 3.2 | four: 1, 2, 3, 5 | 1: "Where no [Detector] of that kind can reach it, the kind cannot express the [Rule], and the first ground holds." and "the two grounds" (J7). 2: "naming the [Detectors] of the stronger kind that were considered, on the [Rule]'s [Remediation docs] and in the description of the commit that adds the [Rule], moves it to another [Detector], or first records the grounds." 3: "The language has such a [Detector] when the project already runs it or the language's own [Detector] ecosystem offers it, the same check section 2 asks for before a [Toolchain] gap is recorded." and "the second ground" (J7). 5: "Where those extension points cannot reach the construct a [Rule] is about, that [Detector] cannot express the [Rule]."; "the stronger kind cannot express the [Rule]" (the first ground's list item); "A stronger configuration of a [Detector] the project runs is one; a [Detector] or configuration of the stronger kind available only through a hosted service or a licence tier is not." Every sentence quoted lies in the one paragraph that begins "For the second ground"; the first list item is named by readers 1 and 5, the second by readers 1 and 3 | +| The bullet on leaving the next wider rule unbuilt | 4 | three: 1, 2, 5 | 2: "Leaving unbuilt the next wider [Rule] that clause 3.1 required the [Practitioner] to name, where it was rejected because no [Instance] of it exists today rather than because the [Hazard] cannot arise there." 5: "it reaches no further than the one wider [Rule] already on the record." 1: by description, "its owner list covers only the wider rule left unbuilt because no instance exists" (J7) | +| Table row 3.1, the fourth column | 3 (given as 3.1 by both) | two: 1, 2 | 1: "Whether the next wider [Rule] is built". 2: "Whether the next wider [Rule] is built, and whether the [Rule] stays narrower than the search showed" | +| The lead-in to the owner's decisions | 4 | two: 1, 4 | 4: "**Decisions that MUST go to whoever owns the codebase:**", set beside the Owner entry (below). 1: by description, "its owner list" (J7) | +| The Owner entry | 1, Owner | one: 4 | "Whoever holds authority over a project's [Exceptions] and [Calibrations]. Always a human." (quoted after "**Owner**: ") | +| The practitioner's first decision | 4 | one: 1 | "What [Class] the [Defect] belongs to, and where its bounds sit (clause 3.1)" | +| The lower bound, the exact value or name | 3.1 | one: 1 | "A [Rule] drawn to the exact value or name found in the [Defect] is the same [Narrowing] however it is spelt." | +| The clean run of the rule and saturation | 3.1 | one: 3 | "A clean run of the [Rule] is therefore not evidence of saturation either: "the [Rule] found nothing the reading had missed" is one technique and a [Rule] run, not two techniques, and the stopping criterion cannot be applied to it." | +| What is in scope of the sweep, and a rule's own fixtures | 3.4 | one: 1 | "Whether vendored dependencies, generated code and test [Fixtures] are in scope is the project's decision" and "A [Rule]'s own [Fixtures] under clause 3.3 are never [Instances]."; by description, also 3.4's first paragraph (J7) | +| The three ways out of the method | 2 | one: 4 | "There are three ways out of the method and they are recorded differently." and "a mechanism gap under clause 3.2, and the [Rule] is still built" | +| What a conforming remediation is | 7 | one: 2 | "A remediation [Conforms] if all six clauses of section 3 were followed for that [Defect], and the verdict that they were rests on reproduction, as this section closes by requiring, not on the report." | + +Readers' accounts of what they could not tell are recorded with the passage they attach to and +are not suggestions; none of the sheets proposes a change. + +**Earlier passages (the fourth row).** Run 19 found four passages named by two or more readers. +As the paragraphs read in the changed document: + +- Clause 3.2's extension-points sentence lay in the paragraph that now begins "For the second + ground". Readers 1, 2, 3 and 5 of this cohort name sentences of that paragraph, so that + finding is not cleared. +- Clause 3.1's level paragraph, now "The bounds say how broad the [Class] is", is named by no + reader. Cleared. +- Section 2's "Attempt rather than pre-judge" paragraph is named by no reader; reader 4's section + 2 passage lies in the next paragraph, "There are three ways out". Cleared. +- Clause 3.4's "Fix them all" paragraph is named by no reader; reader 1's clause 3.4 passage lies + in the scope paragraph and, by description, the opening paragraph. Cleared. + +No sentence kept from one of those paragraphs is named elsewhere in the document. + +**Editor's note on the marker's instruction.** The instruction for this run asked for the stricter +reading wherever the rules leave a judgement open, which the process at 204f95b had replaced with +the judgement of a careful reader. The verdict does not turn on it: the findings that fail the run +rest on rules, not on those judgements. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in 72f1679 and the commit after +it that settles a review of those edits (key quotations kept verbatim in c5bfd10). SPEC.md: clause +3.2 states both grounds for a weaker kind in the one sentence that allows it, and its paragraph on +the second ground is short and plain; section 2 names the order (attribute, build, prove, sweep, +then fix every instance) where it says the defence comes before the fix; clause 3.3 Part B says an +ignore left over an instance that carries the hazard is not this clause's matter, and Part C says +which run it means; section 4's bullet on the next wider rule, the map's row 3.1 and section 4's +lead-in say plainly what the decision is and who takes it. A fresh cohort follows. + +## Process change + +Made under a process change before it merged: yes, pull request 13 at commit 204f95b, named +before the readers started. + +## Verdict + +**FAIL.** Against the criteria in `ACCEPTANCE.md`: + +- Load-bearing statements: not met. 2a two of five (readers 1, 3 and 4 miss), 2b five of five, + 2c five of five, 2d none of five: no reader states the two grounds for a weaker kind. +- Restatement and application: not met on 3.3, three of five (readers 2 and 3 miss); question 1 + and every other keyed item five of five. +- Misses traceable to the text, each a finding: 2a for readers 1, 3 and 4, each relying on the + section 2 sentence "When it can, building the [Defence] is not optional and MUST be attempted + before the [Defect] is fixed."; 2d for all five readers, on clause 3.2's grounds for a weaker + kind; 3.3 for reader 2, on Part B and the inline ignore; 3.3 for reader 3, on Part C's + "the enforced run does not report it". +- Passages found confusing by two or more, each a finding that blocks until the text is changed + and a fresh cohort no longer names it: clause 3.2's two grounds and the "For the second ground" + paragraph (four readers), which is also run 19's extension-points finding, not cleared; + section 4's bullet on leaving the next wider rule unbuilt (three); the fourth column of table + row 3.1 in section 3 (two); section 4's lead-in "Decisions that MUST go to whoever owns the + codebase:" (two, on J7). +- Single-reader notes, recorded: the Owner entry (reader 4); section 4's first practitioner + bullet (reader 1); clause 3.1's lower-bound sentence on the exact value or name (reader 1); + clause 3.1's clean-run sentence (reader 3); clause 3.4's scope paragraph and opening paragraph + (reader 1); section 2's three ways out (reader 4); section 7's conforming remediation (reader + 2). + +Answer sheets, the reader prompt, the command and the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090464782), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090465417), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090465825), [comment 4](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090466147). diff --git a/acceptance/runs/022-semantic-detectors-run-4.md b/acceptance/runs/022-semantic-detectors-run-4.md new file mode 100644 index 0000000..e8262c1 --- /dev/null +++ b/acceptance/runs/022-semantic-detectors-run-4.md @@ -0,0 +1,217 @@ +# Run 22: DETECTOR-SPEC.md, semantic detectors, fourth run in pull request 14 + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + editor's branch `editor/semantic-detectors` at 3aefdd1; the question set, key and fixture read + at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input, with no tools, no settings, no project context + and no saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is the question set, the document and the fixture at + 3aefdd1, in that order, each preceded by one `===== FILE: =====` line, with the files + separated by one blank line and nothing else added (judgement 11) +- **Managed (policy) settings**: none; the machine applies no managed settings and no managed + settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. The editor reports none set aside for a cause the sheets cannot + show; every sheet has answer text, no process reports an error (`is_error` false, subtype + `success`), and every reported model is the cohort's model +- **Run within the pull request**: 4, counting void runs; run 21, on SPEC.md, was made at the same + time and does not bear on this document +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review + +## Marks + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | Section 8 | Kind | 6.4 | Q4 passages | +| ------ | ------- | ---- | ---- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --------- | ---------- | --- | ----------- | +| 1 | miss | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 3 | +| 2 | correct | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 3 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 3 | +| 4 | miss | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 5 | miss | miss | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 3 | + +Every miss under question 2 is traceable to the text: each answer quotes the document in support. +The three misses under question 1 are the readers' own: the bullet each turns on quotes nothing. +No question was left unanswered. + +### Question 1 + +Every restatement names four or more obligations from the key's list. Readers 2 and 3 assert no +obligation the document does not state: correct. Readers 1, 4 and 5 each have a bullet that +requires every finding in the command's own output: reader 1, "the command's own output carries +every finding"; reader 4, "put every finding in the command's own output"; reader 5, "put each +finding and its identifier in the command's own output". Clause 5.3 requires a usable summary +and, where the summary leaves a finding out, a statement of where the rest can be read; printing +every finding is one way to meet it, not the obligation. Each is marked a bullet asserting an +obligation the document does not state, so incorrect (judgement 2). Correct, two of five. + +### Question 2 + +The key's answers state these elements: + +- 2a: (a) Conformance is satisfying every MUST in sections 4 to 7; (b) judged on evidence; (c) a + detector that predates the document can conform; (d) the declaration is optional. +- 2b: (a) make the route disableable by configuration, or (b) detectable by a Rule the project can + write or a mechanical check the Detector documents; (c) the detector is not required to remove or + forbid it. +- 2c: (a) with the rule; (b) in the installed copy; (c) at a version tracked together; (d) yes, + reachable offline. + +Each answer is read as its sentence and its quotation together, for the gist of each element. + +| Reader | 2a | 2b | 2c | +| ------ | ------------------------------------------------------------------------------------------- | ---------------------------------- | ------------------------------------------------------------------- | +| 1 | miss: (d) not stated (judgement 3) | miss: (c) not stated (judgement 4) | ok: all four, (c) from the 6.3 quotation | +| 2 | miss: (d) not stated (judgement 3); (a) as "the evidence against the MUSTs", not every MUST | miss: (c) not stated (judgement 4) | ok: all four, (c) from the 6.3 quotation | +| 3 | ok: all four; (d) as "rests on evidence rather than on the detector's claim" | miss: (c) not stated (judgement 4) | ok: all four, from the sentence and the 6.2 and 6.3 headings quoted | +| 4 | ok: all four; (d) as "judged on evidence rather than on any claim it makes" | miss: (c) not stated (judgement 4) | ok: all four | +| 5 | miss: (d) not stated (judgement 3) | miss: (c) not stated (judgement 4) | miss: (a) and (c) not stated; only 6.2 quoted (judgement 6) | + +No answer asserts anything the document does not state or that the key says makes an answer +wrong: none says a declaration is required, that partial satisfaction is conformance, that the +detector must forbid the route or require a reason, or that a website will do. + +### Question 3 + +The key's intended verdicts are Yes for every clause but 6.3, which is No; the section 8 verdict +is "does not conform", because 6.3 fails; 6.4 does not change the verdict. + +- **Every clause**: all five readers give the intended verdict on every clause, 4.1 to 7.1. On + 6.3 each cites that one unresolved identifier fails the clause; no reader grades it Partial on + the grounds that thirty-eight resolve. On 5.3 and 6.1, the clauses run 20 failed on, all five + grade Yes. +- **Section 8**: all five say Lintomatic does not conform, because 6.3 fails. +- **Kind**: all five say the fixture does not let them name the kind and name none, so none is + marked on it. Reader 1 calls its verdict "incomplete on that point" but does not make the + verdict turn on the kind; readers 3 and 5 say outright that it does not change the verdict. +- **6.4**: all five say it does not change the verdict, because it is a SHOULD. Reader 1 adds that + whether 6.4 applies as a MUST depends on whether Lintomatic ships inside a Toolchain, which does + not alter its answer. + +### Judgements made in marking + +01. **How an open judgement is made.** `ACCEPTANCE.md` at 204f95b says an open judgement is made + as a careful reader of the document would; the editor's instruction asks for the stricter + reading wherever the rules leave a judgement open. Each judgement below is made as a careful + reader would, and where a careful reader could take either reading, the stricter one, the one + recording more misses or findings, is taken and the judgement says so. +02. **Question 1, "every finding" in the output, readers 1, 4 and 5.** The heading of 5.3 speaks + of "the result", which a reader may take as every finding, but its body requires only a + summary and a pointer to what the summary leaves out. A careful reader could read these bullets + as a compression of 5.3 or as an obligation it does not state; the stricter reading is taken, + and each is marked incorrect. +03. **2a (d), the declaration optional.** The question's second part asks whether a detector that + has never heard of the document can conform; the key answers it with both (c) and (d), so (d) + is an element that answers a part of the question. Its gist is that conformance does not depend + on a declaration or claim. Readers 3 and 4 say the grade rests on evidence rather than on the + detector's claim, which the document calls the declaration's purpose ("it is how they claim + [Conformance]"), and so state it. Readers 2 and 5 say only that the detector is judged on + evidence; reader 1 says it is judged "not on whether its maintainer has heard of the document". + Each of the three, with the 8.1 quotation all five give, implies that no declaration is needed + without saying so; a careful reader could go either way, and the stricter reading is taken. +04. **2b (c), not required to remove or forbid.** The question asks what the detector must do + about the route, and the key's answer includes what it need not do, so (c) answers the + question. All five say either of the two things alone is enough. That entails that nothing + further is required but does not say that the route may stay; a careful reader could go either + way, and the stricter reading is taken. Marked not stated, as run 20 marked it. +05. **2b (b), the means of detection.** Clause 7.1 defines "Detectable" in its own text, and each + reader uses "detectable", or reader 4 "findable", in that sense, so (b) is stated: a defined + term used in its defined sense states what it means. It decides no mark, since every 2b answer + misses (c). +06. **2c, reader 5.** Its sentence says the documentation "lives in the installed package" and its + only quotation is 6.2. Neither says the documentation ships with the rule nor that it is + versioned with it. Element (c) answers where the documentation must live, as part of the key's + answer to that part; not stated. Whether "in the installed package" carries (a) is open, and + the stricter reading is taken. +07. **Question 4, a clause number inside a quotation.** Readers 1 to 4 quote the 7.1 sentence that + contains "of the kind clause 4.1 requires it to host". The clause number is part of the quoted + run, so it covers the 7.1 sentence; it is not a passage named by clause number and does not + cover clause 4.1. +08. **Question 4, reader 4's "in the [Detector]".** Taken as one run, these words occur in the 7.1 + sentence and, on the stricter reading, in the 6.3 Why sentence "If its documentation lives only + in the [Detector]'s repository or on its website, ...", where the following word is + "[Detector]'s". The quotation is taken to cover both. The second blocks nothing. +09. **Question 4, a passage quoted only to set another against it.** Reader 1 quotes the 8.1 + sentence "A declaration with a non-empty gap record ..." as what it read the other 8.1 sentence + against. A quotation under question 4 counts whatever it is quoted for, so it is recorded. +10. **The fourth row's "earlier passage".** Read as a passage named by two or more readers in an + earlier run of this document in this pull request, which is run 20 at 5b6093f. Run 20's + single-reader passages, the 6.1 paragraph on a project's own rule, the section 1 sentence and + the section 2 table, are covered by no passage this cohort names, so the stricter reading would + change nothing. +11. **The packet's blank lines.** The prompt joins the three files with one blank line between + each file's end and the next file's header. A blank line adds nothing for a reader to read, + every reader received the same packet, and a packet's form is not a cause in `ACCEPTANCE.md` + for setting a reader aside, so no reader is set aside for it. + +No re-mark. + +## Passages quoted as confusing + +Grouped by paragraph, as `ACCEPTANCE.md` defines the same passage. A passage blocks when two or +more readers name it, or when it covers a sentence of a paragraph in which one of run 20's +blocking passages lay, as that paragraph reads at 3aefdd1. + +| Clause and paragraph | Readers | Words quoted, verbatim | Blocks | +| ------------------------------------------------------------------------------------------ | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | +| 7.1, first paragraph, the sentence beginning "Detectable means" | 1, 2, 3, 4 | 1, 2, 4: "Detectable means the project can find every use of the route, either with a [Rule] the project writes itself in the [Detector], of the kind clause 4.1 requires it to host, or with a mechanical check the [Detector] documents." 3: "either with a [Rule] the project writes itself in the [Detector], of the kind clause 4.1 requires it to host, or with a mechanical check the [Detector] documents." 4 also: "in the [Detector]" (judgement 8) | Yes: four readers, and run 20's 7.1 paragraph | +| 8.1, first paragraph, its last sentence | 1 | "A declaration with a non-empty gap record is a statement of where the [Detector] stands and is not a claim of [Conformance]." (judgement 9) | Yes: run 20's 8.1 first paragraph | +| 8.1, second paragraph, the sentence beginning "The declaration is for" | 1 | "The declaration is for the maintainer: it is how they claim [Conformance] and record the gaps they know of." | Yes: run 20's 8.1 second paragraph | +| 6.1, the paragraph beginning "For a [Bundled rule]" | 3 | "A documentation path printed inside the [Message] does not meet this." | Yes: run 20's 6.1 paragraph on a bundled rule | +| 6.1, the paragraph beginning "In both", the sentence beginning "Where the mechanism works" | 5 | "Where the mechanism works but a [Bundled rule]'s [Identifier] has no documentation to resolve to, that fails clause 6.3, not this clause." | No | +| 5.3, its paragraph, the sentence beginning "Output that prints" | 3 | "Output that prints every finding so meets this clause." | No | +| 6.4, the Why paragraph | 5 | "the [toolchain specification]'s self-audit makes this check a MUST for the [Toolchain]." | No | +| 8, the paragraph beginning "A verdict on a [Detector] MUST also name" | 5 | "the kind of each configuration it grades"; "the one the [Detector] has without any hosted service or licence tier" | No | +| 6.3, the Why paragraph, the sentence beginning "If its documentation lives" | 4 | "in the [Detector]" (judgement 8) | No | + +No reader names clause 4.1, the fifth of run 20's blocking passages (judgement 7). + +## Against the criteria + +- **Load-bearing statements (every reader on every one)**: not met. 2a three of five correct + (readers 3 and 4), 2b none of five, 2c four of five (all but reader 5). +- **Restatement and application (all but one on each keyed item)**: not met. Question 1 two of + five; every clause of question 3, the section 8 verdict and 6.4 five of five. +- **A miss traceable to the text (none)**: not met. Nine, each a finding: 2a by readers 1, 2 and + 5; 2b by all five; 2c by reader 5. +- **A passage found confusing by two or more (none outstanding)**: not met. The 7.1 sentence + defining "Detectable" is named by four readers; and run 20's blocking passages in 7.1's first + paragraph, 8.1's first and second paragraphs and 6.1's paragraph on a bundled rule are still + outstanding, since this cohort names passages that cover sentences of each of those paragraphs. +- **From one reader only**: the 6.1 sentence on a mechanism that works without documentation, the + 5.3 sentence on output that prints every finding, the 6.4 Why sentence, the section 8 paragraph + on kind and the 6.3 Why sentence, recorded. + +**Editor's note on the marker's instruction.** The instruction for this run asked for the stricter +reading wherever the rules leave a judgement open, which the process at 204f95b had replaced with +the judgement of a careful reader. The verdict does not turn on it: the findings that fail the run +rest on rules, not on those judgements. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in 72f1679 and the commit after +it that settles a review of those edits (key quotations kept verbatim in c5bfd10). DETECTOR-SPEC.md: +7.1 says in its first sentence that the detector need not remove or forbid the route, and defines +detectable in short sentences; 8.1's two paragraphs and 6.1's sentence on a documentation path +inside a Message are shorter and plainer; section 8 says conformance is judged on the evidence and a +declaration is optional; 5.3 says the output need not print every finding if it says where the rest +can be read. A fresh cohort follows. + +## Process change + +Made under the process change of pull request 13 before it merged, at commit 204f95b, named +before the readers started; `ACCEPTANCE.md` and `acceptance/runs/README.md` in this worktree +differ from 204f95b only by this pull request's question-set line adding question 2d for SPEC.md, +which does not bear on this document. + +## Verdict + +FAIL on every blocking criterion. Answer sheets, the readers' prompt and command, and the +marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090466450), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090466829), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090467103). diff --git a/acceptance/runs/023-semantic-detectors-run-5.md b/acceptance/runs/023-semantic-detectors-run-5.md new file mode 100644 index 0000000..1c9ddd6 --- /dev/null +++ b/acceptance/runs/023-semantic-detectors-run-5.md @@ -0,0 +1,238 @@ +# Run 23: SPEC.md, semantic detectors, fifth run in pull request 14 + +- **Document**: SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + branch `editor/semantic-detectors` at 0b8d699, the commit the readers were given; the question + set, key and fixture read at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input; no tools, no settings, no project context and no + saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is byte for byte the question set, the document and the + fixture at 0b8d699, in that order, each preceded by one `===== FILE: =====` line, with + nothing else added apart from one blank line between the end of each file and the next file's + marker line. +- **Managed (policy) settings**: none; the machine applies no managed settings to the readers and + no managed settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. Every sheet has answer text, no process reports an error + (`is_error` false, subtype `success`), and every reported model is the cohort's model. The + editor reports no reader set aside for a cause the sheets cannot show. +- **Run within the pull request**: 5, counting void runs; runs in the pull request so far: 5 (19, + 20, 21, 22 and this one). Run 24, on DETECTOR-SPEC.md, is made at the same time and does not + bear on this document. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review. It read `ACCEPTANCE.md` and + `acceptance/runs/README.md` in this worktree, which differ from commit 204f95b only by this pull + request's question-set line adding 2d for SPEC.md; the document, question set, key and fixture + at 0b8d699; run 21 in full and run 19 in part; and, for the fourth pass-criteria row, the change + to SPEC.md between 3aefdd1, the document run 21 read, and 0b8d699. It also looked at the key's + commit history, which a delegated marker is not given; no mark rests on it (J2 is reasoned from + the key's text at 0b8d699). + +## Marks + +`ok` is correct, `miss` is a miss; `T` marks a miss traceable to the text. Judgements J1 to J8 +are set out below. + +| Reader | Q1 | 2a | 2b | 2c | 2d | 3.1 | 3.2 | 3.3 | 3.4 | 3.5 | 3.6 | Remediation | Defence | Section 4 | Q4 passages | +| ------ | ------- | --- | --- | --- | ------ | ------- | ------- | ------ | --- | --- | --- | ----------- | ------- | --------- | ----------- | +| 1 | correct | ok | ok | ok | miss T | No | No | Yes ok | No | No | No | ok | ok | ok | six | +| 2 | correct | ok | ok | ok | miss T | Partial | Partial | Yes ok | No | No | No | ok | ok | ok | three | +| 3 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | three | +| 4 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 5 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | three | + +Every grade under 3.1, 3.2 and 3.4 to 3.6 is one the key accepts. No reader grades 3.2, 3.5 or +3.6 Yes, so none of the key's cases that count against the document arises. + +### Question 1 + +Five of five correct. Every sheet has at least four bullets naming obligations the key lists +(class and hazard, the strongest kind of detector, the proof in a commit of its own, the sweep and +the fix of every instance, permanent and blocking, the terse message with an identifier resolving +to versioned documentation), and no bullet asserts an obligation the document does not state. +Reader 5's second bullet, recording one sentence naming two techniques tried where no rule can be +written, is section 2's "Attempt rather than pre-judge" paragraph; the closing "fix the original +defect with a reproducing test" of readers 2, 3 and 5 is section 6 and Appendix A's last sentence. + +### Question 2 + +Every reader quoted words of the document for every item, so every miss here is traceable to the +text. + +- **2a**, five of five. Each states class, rule, proof, sweep and the fix of every instance, in + that order, and quotes section 2's sentence "The order is section 3's: attribute the [Defect] to + a [Class], build the [Rule], prove that it fires, run the [Sweep], then fix every [Instance]." +- **2b**, five of five. Readers 1, 2, 3 and 4 say the owner is a human; reader 5 says "Only the + Owner" (J4). All five quote the key's section 4 sentence and say an agent may not. +- **2c**, five of five. Each states all six clauses and a verdict resting on reproduction, not on + the report, in its sentence or its quotation of section 7; reader 2's second quotation gives the + red and green reruns. No reader accepts the pull request description as the verdict. +- **2d**, none of five. The key's three elements are (i) the strongest kind, in the order + semantic, syntactic, textual; (ii) both grounds for a weaker kind; (iii) the reason recorded, + with at least one place named and the places not given as alternatives (J2). + - Reader 1: (i) in its first quotation. (ii) missed: "A weaker kind is allowed 'on two grounds + only'" says there are two and states neither, and its second quotation, "which of the two + grounds above holds and why, naming the [Detectors] of the stronger kind that were + considered", does not state them either. (iii) missed: that quotation stops before the places. + - Reader 2: (i) and (ii) stated, the second quotation giving both grounds. (iii) missed: "the + practitioner must record which ground holds and why", with a quotation ending at "were + considered...", names no place (J2). + - Reader 3: (i) and (iii) stated, its second quotation naming the remediation docs "and" the + commit description. (ii) missed: "A weaker kind is allowed only on one of two grounds", and no + quotation states them. + - Reader 4: (i) and (ii) stated. (iii) missed: "The record must include 'for each step down from + [Semantic], which of the two grounds above holds and why, naming the [Detectors] of the + stronger kind that were considered'" names no place (J2). + - Reader 5: (ii) and (iii) stated, the places given with "and". (i) missed: "A Rule must be + written in the strongest detector kind that can express it", and neither quotation names + semantic, syntactic or textual (J3). + +### Question 3 + +Five of five on every keyed item. + +- **3.3**: five of five Yes. Each says the inline ignore is a [Suppression] not judged under this + clause, as Part B's opening now says; none attributes it to 3.3. Reader 5's caveat that the + fixture does not show a fresh-checkout reproduction leaves its grade at Yes. +- **Remediation**: five of five say it does not conform, each naming at least one failing clause + the key accepts. +- **Defence**: five of five say it does not conform, each naming 3.6, and readers 1, 2, 3 and 4 + naming 3.2 as well. +- **Section 4**: five of five name the inline ignore on `Report::render()`, and each also names + leaving the next wider rule unbuilt, which the key accepts alongside it. + +### Judgements made in marking + +Made as a careful reader of the document would, as `ACCEPTANCE.md` requires. Where a rule settles +the point, the record says which. + +- **J1. The sentence.** Where a reader wrote more than one sentence before or around its + quotations, all of that text is read with the quotations as the answer. No mark turns on it. +- **J2. 2d's element (iii) needs a place named.** The key says of (iii) that "naming one of the + two places is enough" and that presenting them as alternatives is wrong; "enough" sets a + minimum, and a key that let an answer name no place would have no minimum to state and no + alternatives to forbid. Runs 19 and 21 read it the same way. Readers 2 and 4 miss on this + alone; read otherwise, both would be correct on 2d and the 2d miss would rest on readers 1, 3 + and 5. The verdict does not turn on it. +- **J3. 2d's element (i) needs the order (reader 5).** The key's element is "the strongest kind + that can express the rule, in the order semantic, syntactic, textual". "The strongest detector + kind that can express it" names no kind, and the question asks which kind. Read otherwise, + reader 5 would be correct on 2d; the verdict would not change. +- **J4. "The Owner" (reader 5, 2b).** Settled by the rule that a defined term used in its defined + sense states what it means: [Owner] is "Always a human". +- **J5. Every passage under question 4 counts, including one set beside the main passage.** + Settled in part by the rule that "a quotation under question 4, of any length" covers what its + words overlap, and that a passage named "by clause number or description" covers what it names. + Where a reader says it was confused by two passages read together, a careful reader takes both + as the confusing passage, since the confusion lies in neither alone. So reader 1's description + of the Narrowing entry, its "Fix them all" and its section 4 sentence, and reader 3's + description of the Semantic entry's "need not accept bespoke [Rules]" sentence, are recorded as + passages. The finding on the Narrowing entry turns on this: read only as context, reader 1's + description would leave the entry to reader 4 alone. No other finding turns on it. +- **J6. Elided quotations.** A quotation with an ellipsis is read as one quotation per unbroken + run of words, each in the document's order (reader 5's 3.3 passage, reader 2's 3.5 passage). + Settled by the rule on quoting: consecutive words of any length. +- **J7. Clause labels.** A clause number given with a quotation, as question 4 asks, is read as + the quotation's label, not as naming the whole clause, as run 21 read it. Reader 5's "Clause + 3.3, Part B" label on "the [Practitioner]'s to decide", which lies in the paragraph introducing + the three parts, does not extend the passage to all of Part B. +- **J8. The earlier 3.2 passage, as it reads in the changed document.** Run 21's passage lay in + clause 3.2's two numbered grounds and the paragraph that began "For the second ground". In the + changed document the grounds are one sentence at the end of the paragraph beginning + "**Qualifying is not the same as being the right host.**", and the "For the second ground" + paragraph is split: its sentences on when the language has a detector of the stronger kind and + when one accepts bespoke rules, reworded, form the paragraph beginning "The language has a + [Detector] of the stronger kind", and its record sentences, kept, form the next paragraph, + beginning "On the [Remediation docs] of a [Rule] they write". All three are read as that + earlier paragraph and list as they now read. + +## Passages quoted as confusing + +Same passage as `ACCEPTANCE.md` defines it: sentences in common, or the same paragraph; a list +item, table row or terminology entry is its own paragraph. + +| Passage | Clause | Readers | Words quoted, verbatim, or the description used | +| ---------------------------------------------------------------------------- | ------------ | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| The lower bound, the exact value or name and the rejection of the wider rule | 3.1 | three: 1, 2, 4 | 1 and 4: "A [Rule] drawn to the exact value or name found in the [Defect] is the same [Narrowing] however it is spelt." 2: "That rejection is tested as a [Narrowing] is under clause 3.3: a sentence stating why the [Hazard] cannot arise in the code the wider [Rule] would add, confirmed by search, with uncertainty routed upwards under section 4." All in the lower-bound list item | +| The Owner entry | 1, Owner | two: 1, 5 | 1: "Where the project has recorded nobody, the [Owner] is whoever instructed the [Practitioner]." 5: "Always a human." and "Where the project has recorded nobody, the [Owner] is whoever instructed the [Practitioner]." | +| The Narrowing entry | 1, Narrowing | two: 1, 4 | 4: "[Narrowing] that excludes code which does carry the [Hazard] is a [Suppression]." 1: by description, "The terminology section defines [Narrowing] as reducing a rule's scope to stop reporting code without the hazard" (J5) | +| When the language has a detector of the stronger kind | 3.2 | one: 3 | "The language has a [Detector] of the stronger kind when the project already runs one, or the language's own [Detector] ecosystem offers one." Run 21's open passage (J8) | +| The two grounds for a weaker kind | 3.2 | one: 3 | "no [Detector] of the stronger kind that accepts bespoke [Rules]", in the sentence that allows a weaker kind on two grounds. Run 21's open passage (J8) | +| A semantic detector need not accept bespoke rules | 1, Semantic | one: 3 | By description, "which the terminology says may not accept bespoke rules" (J5) | +| Where a gap is recorded | 3.2 | one: 2 | "A gap is recorded in the project record where the [Toolchain] defines one, as the [toolchain specification](TOOLING-SPEC.md) requires it to; only where it does not is the location the project's choice, and then the only requirement is that somebody deciding what to invest in the [Toolchain] would find it." | +| The entry point for a bundled defence | 3.5 | one: 2 | "For a [Bundled defence], the project whose entry point demonstrates it is any project the [Toolchain] is genuinely installed into, the [Consuming project] included", and the terms "Bundled defence" and "Consuming project", which occur only in that sentence | +| Narrowings enumerable and reported to the owner | 3.3, Part B | one: 5 | "Every [Narrowing] MUST be enumerable by the same means as the project's [Exceptions]" and "a [Narrowing] that excludes more code than the [Rule] still covers MUST be reported to the [Owner] as if it were a [Suppression]" (J6) | +| The three parts of clause 3.3 | 3.3 | one: 5 | "the [Practitioner]'s to decide" (J5, J7) | +| A defence conforming whilst the sweep is incomplete | 7 | one: 1 | "A [Defence] can [Conform] whilst the [Sweep] is incomplete." | +| Fix them all | 3.4 | one: 1 | "Fix them all" (J5) | +| The dividing line, an agent's authority | 4 | one: 1 | "An [Agent] MUST NOT [Baseline], suppress or knowingly leave an [Instance] unfixed" (J5) | + +Readers' accounts of what they could not tell are recorded with the passage they attach to and are +not suggestions; none of the sheets proposes a change. + +**Earlier passages (the fourth row).** Run 21 found four passages named by two or more readers, +the open ones. As the paragraphs read in the changed document: + +- Clause 3.2's two grounds and the "For the second ground" paragraph (J8): reader 3 names a + sentence of the paragraph beginning "The language has a [Detector] of the stronger kind", and + words of the sentence that now states the two grounds. Not cleared. +- Section 4's bullet on leaving the next wider rule unbuilt: named by no reader. Reader 2's 3.1 + passage is in clause 3.1's lower-bound list item, not the bullet. Cleared. +- The fourth column of table row 3.1 in section 3: named by no reader. Cleared. +- Section 4's lead-in, now "**Decisions that MUST go to the [Owner]**, which the [Practitioner] + never takes alone:": named by no reader. Reader 1's section 4 sentence lies in the paragraph + beginning "The dividing line", not the lead-in. Cleared. + +No sentence kept from one of the three cleared paragraphs is named elsewhere in the document. +Reader 1's "Fix them all" lies in the clause 3.4 paragraph run 19 named and run 21 cleared; it is +not an open passage and is recorded above as one reader's. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in b172fd6 and the Owner fix +after its review (key quotations kept verbatim in 296aad5). SPEC.md: clause 3.2's record sentence +names both grounds itself, and its sentence on when the language has a detector of the stronger kind +is plain; clause 3.1's lower-bound item is in short, plain sentences; section 1's Owner entry says +plainly who the default Owner is, the first human up the chain of instructions, and its Narrowing +entry says plainly when a narrowing is a suppression. A fresh cohort follows. + +## Process change + +Made under a process change before it merged: yes, pull request 13 at commit 204f95b, named +before the readers started. + +## Verdict + +**FAIL.** Against the criteria in `ACCEPTANCE.md`: + +- Load-bearing statements: not met. 2a, 2b and 2c five of five; 2d none of five. Readers 1 and 3 + state neither ground for a weaker kind, only that there are two; readers 1, 2 and 4 name no + place for the record (J2); reader 5 does not give the order semantic, syntactic, textual (J3). +- Restatement and application: met. Question 1 and every keyed item of question 3 five of five. +- Misses traceable to the text, each a finding: 2d for all five readers, on clause 3.2's + paragraph beginning "**Qualifying is not the same as being the right host.**" and the record + sentence "Where that kind is weaker than [Semantic], the [Practitioner] MUST also record, for + each step down from [Semantic], which of the two grounds above holds and why, naming the + [Detectors] of the stronger kind that were considered, on the [Rule]'s [Remediation docs] and in + the description of the commit that adds the [Rule], moves it to another [Detector], or first + records the grounds.", which every reader quotes. +- Passages found confusing by two or more, each a finding that blocks until the text is changed + and a fresh cohort no longer names it: clause 3.1's lower-bound list item (three readers); the + Owner entry in section 1 (two); the Narrowing entry in section 1 (two, on J5). Run 21's clause + 3.2 passage is not cleared: reader 3 names the paragraph beginning "The language has a + [Detector] of the stronger kind" and the sentence stating the two grounds (J8). Run 21's other + three passages are cleared. +- Single-reader notes, recorded: the Semantic entry (reader 3); clause 3.2's paragraph on where a + gap is recorded (reader 2); clause 3.5's bundled defence sentence (reader 2); clause 3.3's + paragraph on narrowings enumerable and reported to the owner, and its three-parts paragraph + (reader 5); section 7's conforming defence over an incomplete sweep (reader 1); clause 3.4's + "Fix them all" (reader 1); section 4's "An [Agent] MUST NOT" sentence (reader 1). + +Answer sheets, the reader prompt, the command and the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090684297), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090684613), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090684941), [comment 4](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090685201). diff --git a/acceptance/runs/024-semantic-detectors-run-6.md b/acceptance/runs/024-semantic-detectors-run-6.md new file mode 100644 index 0000000..96f845e --- /dev/null +++ b/acceptance/runs/024-semantic-detectors-run-6.md @@ -0,0 +1,226 @@ +# Run 24: DETECTOR-SPEC.md, semantic detectors, sixth run in pull request 14 + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + editor's branch `editor/semantic-detectors` at 0b8d699; the question set, key and fixture read + at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input, with no tools, no settings, no project context + and no saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is the question set, the document and the fixture at + 0b8d699, in that order, each preceded by one `===== FILE: =====` line, with the files + separated by one blank line and nothing else added, the same form as run 22's packet +- **Managed (policy) settings**: none; the machine applies no managed settings and no managed + settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. The editor reports none set aside for a cause the sheets cannot + show; every sheet has answer text, no process reports an error (`is_error` false, subtype + `success`), and every reported model is the cohort's model +- **Run within the pull request**: 6, counting void runs; run 23, on SPEC.md, was made at the same + time and does not bear on this document +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review + +## Marks + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | Section 8 | Kind | 6.4 | Q4 entries | +| ------ | ------- | ---- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --------- | ---------- | --- | ---------- | +| 1 | correct | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 2 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 3 | miss | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 4 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 5 | correct | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | + +"Q4 entries" counts the passages each reader lists under question 4; the words each quotes are in +the table of passages below. Every miss under question 2 is traceable to the text: each answer +quotes the document in support. The miss under question 1 is the reader's own: the bullet it turns +on quotes nothing. No question was left unanswered. + +### Question 1 + +Every restatement names four or more obligations from the key's list. Readers 1, 2, 4 and 5 assert +no obligation the document does not state: correct. Reader 2's "A reason is SHOULD-required" +states 7.2 as the SHOULD it is, and reader 4's "A detector conforms when it meets every MUST in +sections 4 to 7" states section 8's definition, not an obligation; neither is wrong. Reader 5's +"the command's own output must carry a usable summary with each finding's location and +identifier" follows 5.3's own words (judgement 2). Reader 3's fourth bullet says "each finding must +appear in the output of the command the practitioner ran"; clause 5.3 says "The summary need not +list every finding", so the bullet asserts an obligation the document does not state: incorrect +(judgement 2). Correct, four of five. + +### Question 2 + +The key's answers state these elements: + +- 2a: (a) Conformance is satisfying every MUST in sections 4 to 7; (b) judged on evidence; (c) a + detector that predates the document can conform; (d) the declaration is optional. +- 2b: (a) make the route disableable by configuration, or (b) detectable by a Rule the project can + write or a mechanical check the Detector documents; (c) the detector is not required to remove or + forbid it. +- 2c: (a) with the rule; (b) in the installed copy; (c) at a version tracked together; (d) yes, + reachable offline. + +Each answer is read as its sentence and its quotation together, for the gist of each element. + +| Reader | 2a | 2b | 2c | +| ------ | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------- | +| 1 | miss: (d) not stated; (b) from the 8.1 quotation (judgement 3) | miss: (c) not stated (judgement 4) | ok: all four, (c) from the 6.3 quotation | +| 2 | ok: all four; (d) from the section 8 quotation, "a declaration under clause 8.1 is optional" | miss: (c) not stated (judgement 4) | ok: all four, (c) from the 6.3 quotation | +| 3 | miss: (d) not stated; (b) from the 8.1 quotation (judgement 3) | miss: (c) not stated (judgement 4); (b) by the defined term (judgement 5) | ok: all four, (c) from the 6.3 quotation | +| 4 | ok: all four; (d) from the section 8 quotation | miss: (c) not stated (judgement 4); (b) as "findable by the project" | ok: all four | +| 5 | miss: (d) not stated (judgement 3) | miss: (c) not stated (judgement 4) | ok: all four, from the sentence and the 6.3 and 6.2 quotations | + +No answer asserts anything the document does not state or that the key says makes an answer +wrong: none says a declaration is required, that partial satisfaction is conformance, that the +detector must forbid the route or require a reason, or that a website will do. + +### Question 3 + +The key's intended verdicts are Yes for every clause but 6.3, which is No; the section 8 verdict +is "does not conform", because 6.3 fails; 6.4 does not change the verdict. + +- **Every clause**: all five readers give the intended verdict on every clause, 4.1 to 7.1. On + 6.3 each cites that one unresolved identifier fails the clause; no reader grades it Partial on + the grounds that thirty-eight resolve. Reader 2 calls the fixture ambiguous on whether the two + rules are bundled, and grades No either way. +- **Section 8**: all five say Lintomatic does not conform, because 6.3 fails. +- **Kind**: all five say the fixture does not let them name the kind and name none, so none is + marked on it. Readers 2 and 5 call that part of the verdict incomplete or undetermined but do not + make the verdict turn on it; readers 3 and 4 say outright that the kind does not change the + verdict. +- **6.4**: all five say it does not change the verdict, because it is a SHOULD. Reader 1 adds that a + maintainer declaring a version would record 6.3 and 6.4 under `known-gaps`, although 8.1 defines a + gap as a failed MUST; reader 3 adds that 6.4 becomes a MUST only inside a Toolchain. Neither aside + alters the answer marked (judgement 6). + +### Judgements made in marking + +01. **How an open judgement is made.** As `ACCEPTANCE.md` at 204f95b says, and as the editor's + instruction asks: as a careful reader of the document would, with the reason stated. Where a + rule settles a mark, the rule is applied as written, however the result looks. +02. **Question 1, the output bullets of readers 3 and 5.** Clause 5.3 at 0b8d699 says "The summary + need not list every finding". Reader 3's "each finding must appear in the output" contradicts + that sentence, and a careful reader takes it as an obligation the document does not state. + Reader 5's "a usable summary with each finding's location and identifier" repeats the clause's + "a usable summary, each finding in it with where it is and its [Identifier]", and a careful + reader takes "each finding" there as each finding in the summary, as the clause does. +03. **2a (d), the declaration optional, readers 1, 3 and 5.** As run 22 held, (d) answers the + question's second part, whether a detector that has never heard of the document can conform. + Each of the three quotes the 8.1 sentence that a detector whose maintainer has never read the + document "MAY be graded [Conforming] on evidence"; none says that a declaration is optional or + not needed. That no declaration is needed follows from the quotation but is not said by it. + `ACCEPTANCE.md` names the ways an element is stated: the document's words, a paraphrase, a + synonym, a defined term in its defined sense. An element reached only by inference from another + is not among them, and a careful reader of that rule does not add it; this keeps marking + mechanical, as the question set section asks. Marked not stated. Readers 2 and 4 quote section + 8's "a declaration under clause 8.1 is optional", and so state it. +04. **2b (c), not required to remove or forbid, all five.** As run 22 held, (c) answers what the + detector must do about the route, from the side of what it need not do. Every reader says, or + quotes, that either of the two things alone satisfies the clause. That entails that removal or + forbidding is not required, but does not say it, and 7.1's first sentence, "The [Detector] does + not have to remove or forbid an inline [Suppression] route", which says it, is quoted by none. + By the same reasoning as judgement 3, marked not stated. +05. **2b (b), the means of detection.** As in run 22: clause 7.1 defines "Detectable" in its own + text, and each reader uses "detectable", or reader 4 "findable by the project", in that sense, + so (b) is stated. It decides no mark, since every 2b answer misses (c). +06. **Asides outside the keyed answer.** Reader 1's statement that 6.4 would be recorded as a known + gap, and reader 3's on when 6.4 is a MUST, are not keyed items; each sheet's answer to whether + 6.4 changes the verdict is the key's, and is marked so. +07. **Question 4, single words and short runs quoted.** Reader 3 quotes "configuration", "in + configuration" and "configuration file"; reader 4 quotes "credits"; reader 5 quotes "MAY". + `ACCEPTANCE.md` says a quotation "of any length" covers every sentence an occurrence of the + quoted words overlaps, "at every place that run occurs", with letter case ignored. The rule + settles it: "configuration" covers eight sentences, among them 4.1's first and the 7.1 sentence + "For each one it MUST do at least one of two things ..."; "MAY" covers nine, among them the 7.1 + sentence "The [Detector] MAY ship them." and the 8.1 sentence beginning "A [Detector] that + predates". Each place is listed in the table below. +08. **Question 4, a heading.** A heading counts as one sentence; it is taken as its own paragraph, + not part of the paragraph below it, since it is a separate block of the document. Reader 5's + "MAY" covers the headings of 6.5, 7.1 and 8.1; none of those decides anything, since the + paragraphs below 7.1 and 8.1 that block are covered by other sentences. +09. **Question 4, reader 4's closing remark.** "Several clauses also cite method-specification + clauses (3.2, 3.4, 3.6, 8.1–8.3) and the toolchain specification, which were not in the packet." + It names clauses of the method specification, not of this document, and says nothing was + confusing; a careful reader takes it as a remark about the packet, not a passage named. Taken + otherwise it would add nothing that changes the verdict. +10. **The open passages, and clause 4.1.** The passages runs 20 and 22 found confusing by two or + more, and not since cleared, are taken as: 7.1's first paragraph; 8.1's first paragraph; 8.1's + second paragraph; and 6.1's paragraph beginning "For a [Bundled rule]". Clause 4.1, named by + three readers of run 20, is taken as cleared by run 22: the document's text changed after run + 20 and no reader of run 22 named a passage covering a sentence of 4.1. A careful reader takes + "until the text is changed" as the document's text, since the same row speaks of "the changed + document". Read the other way, 4.1 would still be open and outstanding, since reader 3's + "configuration" covers its first sentence; the verdict does not turn on it. + +No re-mark. + +## Passages quoted as confusing + +Grouped by paragraph, as `ACCEPTANCE.md` defines the same passage. A passage blocks when two or +more readers name it, or when it covers a sentence of a paragraph in which one of the open +passages lay (judgement 10), as that paragraph reads at 0b8d699. + +| Clause and paragraph | Readers | Words quoted, verbatim | Blocks | +| ----------------------------------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------- | +| 8, the paragraph beginning "A verdict on a [Detector] MUST also name", its first and second sentences | 1, 3, 4, 5 | 1, 4, 5: "The kind named is the one the [Detector] has without any hosted service or licence tier, as clause 5.4 credits no finding reported only there." 4 also: "credits". 3: "A verdict on a [Detector] MUST also name the kind of each configuration it grades"; and "configuration" (judgement 7) | Yes: four readers | +| 7.1, first paragraph, the sentences "The [Detector] MAY ship them." and "For each one it MUST ..." | 3, 5 | 3: "configuration", covering "make the route disableable by configuration". 5: "MAY", covering "The [Detector] MAY ship them." (judgement 7) | Yes: two readers, and the open 7.1 paragraph of runs 20 and 22 | +| 8.1, second paragraph, the sentence beginning "A [Detector] that predates" | 5 | "A [Detector] that predates this document, or whose maintainer has never read it, MAY be graded [Conforming] on evidence by anyone who exercises the clauses above against it, and a verdict on any [Detector], declared or not, rests on that exercise and not on the claim." Also "MAY" | Yes: the open 8.1 second paragraph of runs 20 and 22 | +| 4.1, first paragraph, "Configuration of an existing [Rule] set is not sufficient." | 3 | "configuration" (judgement 7) | No (judgement 10) | +| 8, second paragraph, its first sentence | 2 | "Partial [Conformance] MUST NOT be described as [Conformance]." | No | +| 8, third paragraph, its first sentence | 2 | "This document defines one level," | No | +| 4.2, first paragraph, its second sentence | 3 | "a [Detector] failing 4.1 is graded here on its [Bundled rules]"; "every [Rule] the [Detector] runs, bundled or bespoke" | No | +| 4.3, first paragraph, the sentences ending "configuration file." and "in configuration is not." | 3 | "configuration"; "in configuration"; "configuration file" (judgement 7) | No | +| 4.3, the table rows for `proj.no-raw-sql` and `rule 7` | 3 | "configuration"; "in configuration"; "configuration file" (judgement 7) | No | +| 2, the terminology table, the row for [Semantic], [Syntactic], [Textual] | 3 | "configuration" (judgement 7) | No | +| 6.4, the Why paragraph | 4 | "the [toolchain specification](TOOLING-SPEC.md)'s self-audit makes this check a MUST for the [Toolchain]" | No | +| 1, first paragraph; 2, the definition of [Rule author]; 6.3, first paragraph; 6.5, first paragraph | 5 | "MAY", each by an occurrence of "MAY" or "May" (judgement 7) | No | +| The headings of 6.5, 7.1 and 8.1 | 5 | "MAY" (judgements 7 and 8) | No | + +Of the open passages, 8.1's first paragraph and 6.1's paragraph beginning "For a [Bundled rule]" +are cleared: the text of each changed after run 22, and no reader of this cohort names a passage +covering a sentence of either. 7.1's first paragraph and 8.1's second paragraph are still +outstanding. The section 8 paragraph on kind was also named by one reader of run 22. + +## Against the criteria + +- **Load-bearing statements (every reader on every one)**: not met. 2a two of five correct + (readers 2 and 4), 2b none of five, 2c five of five. +- **Restatement and application (all but one on each keyed item)**: met. Question 1 four of five; + every clause of question 3, the section 8 verdict and 6.4 five of five. +- **A miss traceable to the text (none)**: not met. Eight, each a finding: 2a by readers 1, 3 and + 5; 2b by all five. +- **A passage found confusing by two or more (none outstanding)**: not met. The section 8 + paragraph on kind is named by four readers, and 7.1's first paragraph by two; and the open + passages in 7.1's first paragraph and 8.1's second paragraph are still outstanding, since this + cohort names passages covering sentences of each. +- **From one reader only**: the section 8 sentences on partial conformance and on one level, the + 4.1, 4.2 and 4.3 passages, the section 2 table row, the 6.4 Why paragraph and the other places + reader 5's "MAY" covers, recorded. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in b172fd6 (key quotations kept +verbatim in 296aad5). DETECTOR-SPEC.md: 8.1's sentence that a detector may be graded on evidence +says the declaration is optional; 7.1's sentence that either means alone satisfies the clause says +neither requires the detector to remove or forbid the route; section 8's paragraph on naming the +kind, 7.1's first paragraph and 8.1's second paragraph are in short, plain sentences. Pull request +13 now limits a question 4 quotation to the place the reader names, so this run's one-word +quotations would no longer reach other passages. A fresh cohort follows. + +## Process change + +Made under the process change of pull request 13 before it merged, at commit 204f95b, named +before the readers started; `ACCEPTANCE.md` and `acceptance/runs/README.md` in this worktree +differ from 204f95b only by this pull request's question-set line adding question 2d for SPEC.md, +which does not bear on this document. + +## Verdict + +FAIL on the first, third and fourth criteria. Answer sheets, the readers' prompt and command, and +the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090685442), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090685662), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090685883). diff --git a/acceptance/runs/025-semantic-detectors-run-7.md b/acceptance/runs/025-semantic-detectors-run-7.md new file mode 100644 index 0000000..95ac89f --- /dev/null +++ b/acceptance/runs/025-semantic-detectors-run-7.md @@ -0,0 +1,269 @@ +# Run 25: SPEC.md, semantic detectors, seventh run in pull request 14 + +- **Document**: SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + branch `editor/semantic-detectors` at dfb9b68, the commit the readers were given; the question + set, key and fixture read at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input; no tools, no settings, no project context and no + saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is byte for byte the question set, the document and the + fixture at dfb9b68, in that order, each preceded by one `===== FILE: =====` line, with + nothing else added apart from one blank line between the end of each file and the next file's + marker line. +- **Managed (policy) settings**: none; the machine applies no managed settings to the readers and + no managed settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. Every sheet has answer text, no process reports an error + (`is_error` false, subtype `success`), and every reported model is the cohort's model. The + editor reports no reader set aside for a cause the sheets cannot show. +- **Run within the pull request**: 7, counting void runs; runs in the pull request so far, + counting void runs: 7, as the editor supplied. Run 26 is made at the same time and does not bear + on this document. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review. It read `ACCEPTANCE.md` and + `acceptance/runs/README.md` in this worktree, which differ from commit 598908b only by this pull + request's question-set line adding 2d for SPEC.md; the document, question set, key and fixture + at dfb9b68; runs 21 and 23 in full, run 19's verdict, and the headers of runs 20, 22 and 24, + which are on DETECTOR-SPEC.md; and, for the fourth pass-criteria row, the change to SPEC.md + between 0b8d699, the document run 23 read, and dfb9b68. + +## Marks + +`ok` is correct, `miss` is a miss; `T` marks a miss traceable to the text. Judgements J1 to J8 +are set out below. + +| Reader | Q1 | 2a | 2b | 2c | 2d | 3.1 | 3.2 | 3.3 | 3.4 | 3.5 | 3.6 | Remediation | Defence | Section 4 | Q4 passages | +| ------ | ------- | --- | --- | --- | ------ | --- | ------- | ------ | --- | --- | -------------- | ----------- | ------- | --------- | ----------- | +| 1 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 2 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | Partial miss T | ok | ok | ok | three | +| 3 | correct | ok | ok | ok | ok | No | Partial | Yes ok | No | No | Partial miss T | ok | ok | ok | two | +| 4 | correct | ok | ok | ok | ok | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 5 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | four | + +Every grade under 3.1, 3.2, 3.4 and 3.5, and every 3.6 grade of No, is one the key accepts; every +3.1 grade of No notes that no independent search technique is recorded. No reader grades 3.2, 3.5 +or 3.6 Yes, so none of the key's cases that count against the document arises. The two 3.6 grades +of Partial are marked under J5. + +### Question 1 + +Five of five correct. Every sheet has at least four bullets naming obligations the key lists (the +class bounded both ways with its hazard, the strongest kind of detector and never a test, the +proof in a commit of its own, the sweep with its count and the fix of every instance, permanent +and blocking, the terse message with an identifier resolving to versioned documentation, the +record of a deferred defect), and no bullet asserts an obligation the document does not state. +Reader 5's "strongest kind of detector", naming no fallback, is correct as the key says of a +bullet naming the strongest kind. The closing "fix the original defect with a reproduction test" +of readers 2 and 5 is section 2 and section 6. + +### Question 2 + +Every reader quoted words of the document for every item, so every miss here is traceable to the +text. + +- **2a**, five of five. Each states class, rule, proof, sweep and the fix of every instance, in + that order, and quotes section 2's sentence "The order is section 3's: attribute the [Defect] to + a [Class], build the [Rule], prove that it fires, run the [Sweep], then fix every [Instance]." +- **2b**, five of five. Each says only the owner may and an agent may not, and quotes the key's + section 4 sentence. Readers 1, 3, 4 and 5 say the owner is a human; reader 2 says "Only the + Owner" (J4). +- **2c**, five of five. Each states all six clauses and a verdict resting on reproduction, not on + the report, in its sentence or its quotation of section 7; readers 1, 4 and 5 quote the red and + green reruns. No reader accepts the pull request description as the verdict. +- **2d**, two of five (readers 3 and 4). The key's three elements are (i) the strongest kind, in + the order semantic, syntactic, textual; (ii) both grounds for a weaker kind; (iii) the reason + recorded, with at least one place named and the places not given as alternatives (J2). + - Reader 1: (i) in its first quotation; (iii) stated, its second quotation giving the + remediation docs "and" the commit description. (ii) missed: "A weaker kind is allowed only on + two grounds" says there are two and states neither, and its quotation of the record sentence, + "... which of the two grounds holds ... and why", elides the parenthesis that names them (J6). + - Reader 2: as reader 1. (i) and (iii) stated; (ii) missed: "only on one of two grounds", with + the same record quotation eliding the grounds (J6). + - Reader 3: (i) in its sentence and quotation, (ii) in its quotation of the "two grounds only" + sentence, (iii) with both places joined by "and". Correct. + - Reader 4: (i), (ii) and (iii) in its three quotations, the places joined by "and". Correct. + - Reader 5: (ii) stated in its quotation of the "two grounds only" sentence. (i) missed: "the + strongest kind of detector that can express it", and no quotation names the order semantic, + syntactic, textual (J3). (iii) missed: its record quotation, "MUST also record, ... which of + the two grounds holds ... and why", elides the places, and its sentence names none (J2, J6). + +### Question 3 + +- **3.1**: five of five No, each noting that no two independent search techniques are recorded. +- **3.2**: five of five Partial, each saying the rule reads code and is not a test, and each that + the detector or its kind is not named. Readers 2, 3 and 5 add that no grounds for a weaker kind + are recorded, which the key does not mark wrong. +- **3.3**: five of five Yes. Readers 2, 3, 4 and 5 say the inline ignore is a [Suppression] judged + under 3.4 and 3.5, not here; none attributes it to 3.3. +- **3.4** and **3.5**: five of five No, each naming the inline ignore on `Report::render()`, under + 3.5 as uncovered by any recorded decision. +- **3.6**, three of five (readers 1, 4 and 5, No). Readers 2 and 3 grade Partial, which the key + accepts only "where the reasoning quotes the versioning sentence" (J5). + - Reader 2: "But it sits on a company wiki, not shipped or versioned with the rule." This + paraphrases the requirement and quotes nothing. The reasoning states what the document + requires, so the miss is traceable; reader 2 also names clause 3.6's external URL sentence + under question 4. + - Reader 3: 'they are not "shipped with the project" or "versioned alongside the [Rule]" as the + clause requires.' The first is verbatim from 3.6's opening sentence, which requires shipping, + not versioning; the second is not the document's words, which are "versions it alongside the + [Rule]" in 3.6 and "versioned alongside it" in section 7. The reasoning quotes the document, + so the miss is traceable. +- **Remediation**: five of five say it does not conform, each naming failing clauses the key + accepts. +- **Defence**: five of five say it does not conform, each naming 3.6 as not met and readers 2, 3 + and 4 naming 3.2 as well. Readers 1, 3, 4 and 5 also count 3.5 against the defence, on the + uncovered suppression; the key does not mark that wrong. +- **Section 4**: five of five name the inline ignore on `Report::render()`, and each also names + leaving the next wider rule unbuilt, which the key accepts alongside it. + +### Judgements made in marking + +Made as a careful reader of the document would, as `ACCEPTANCE.md` requires. Where a rule settles +the point, the record says which. + +- **J1. The sentence.** Where a reader wrote more than one sentence before or around its + quotations, all of that text is read with the quotations as the answer. No mark turns on it. +- **J2. 2d's element (iii) needs a place named.** The key says of (iii) that "naming one of the + two places is enough" and that presenting them as alternatives is wrong; "enough" sets a + minimum, and a key that let an answer name no place would have no minimum to state and no + alternatives to forbid. Runs 19, 21 and 23 read it the same way. Reader 5's (iii) turns on it; + reader 5 misses (i) as well (J3). +- **J3. 2d's element (i) needs the order (reader 5).** The key's element is "the strongest kind + that can express the rule, in the order semantic, syntactic, textual", and the question asks + which kind. "The strongest kind of detector that can express it" names no kind, and "weaker than + [Semantic]" in its record quotation names the top of the order, not the order. Run 23 read it + the same way. Read otherwise under both J2 and J3, reader 5 would be correct on 2d; 2d would + still fail on readers 1 and 2, whose misses on (ii) rest on no judgement. +- **J4. "The Owner" (reader 2, 2b).** Settled by the rule that a defined term used in its defined + sense states what it means: the [Owner] "is always a human". +- **J5. 3.6 Partial and "the versioning sentence" (readers 2 and 3).** The key accepts Partial + "where the reasoning quotes the versioning sentence", and `ACCEPTANCE.md` defines quoting: the + document's consecutive words, verbatim apart from letter case, quotation-mark style and markup; + a paraphrase is not quoting. Which sentence is the versioning sentence the key leaves open. It + is taken in the wider sense, either of 3.6's sentences requiring the documentation to be + versioned with the rule: "A [Rule] and its documentation are one artefact and MUST be versioned + as one." and "An external URL [Conforms] only where the project controls it and versions it + alongside the [Rule].", the second because the key's case of a reader who grades Yes "because + the URL resolves" is a misreading of it. Neither reader quotes either sentence, so both miss on + 3.6, and both reasonings state or quote what the document requires, so both misses are + traceable. Read leniently, taking reader 3's "versioned alongside the [Rule]" as a quotation of + the URL sentence despite the changed word, reader 3 would be correct, 3.6 would be four of five + and the second criterion met; reader 2's traceable miss would stand, and the verdict would not + change. +- **J6. Elided quotations.** A quotation with an ellipsis is read as one quotation per unbroken + run of words, each in the document's order, and states only what those runs state (readers 1, + 2 and 5 under 2d; reader 4's section 8 passage). Settled by the rule on quoting: consecutive + words of any length. +- **J7. Clause labels.** A clause number given with a quotation, as question 4 asks, is read as + the quotation's label, not as naming the whole clause, as runs 21 and 23 read it. Reader 1's + "clause 3.3, Part B" label does not extend its passage to all of Part B. +- **J8. Passages set beside the main one count.** Where a reader says it was confused by two + passages read together, a careful reader takes both as the confusing passage, since the + confusion lies in neither alone, as run 23 read it. So reader 2's first two items each cover + both passages they set side by side, reader 4's first item covers all three of its quotations, + and reader 5's fourth item covers both its quotations. Reader 5's first item quotes section 2 + and describes clause 3.1 as defining "independence for searches", not knowing whether that test + applies to techniques for writing a rule; the confusion lies in how the two read together, so + the description is read as naming clause 3.1's definition, the sentence "Two techniques are + independent when they would miss different things." A finding turns on this: read as context + only, clause 3.1's paragraph beginning with that sentence would be reader 2's alone. The verdict + would not change. + +## Passages quoted as confusing + +Same passage as `ACCEPTANCE.md` defines it: sentences in common, or the same paragraph; a list +item, table row or terminology entry is its own paragraph. + +| Passage | Clause | Readers | Words quoted, verbatim, or the description used | +| ----------------------------------------------------------------- | ----------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Which ground holds when extension points cannot reach a construct | 3.2 | four: 1, 3, 4, 5 | All four: "If a [Detector] passes both but its extension points cannot reach the construct a [Rule] is about, that [Detector] cannot express the [Rule]. If no [Detector] of the stronger kind can reach it, the stronger kind cannot express the \[Rule\]: that is the first ground, not the second." The last two sentences of the paragraph beginning "The second ground asks two questions". Run 23's open passage (below) | +| The entry point for a bundled defence | 3.5 | two: 3, 5 | Both: "For a [Bundled defence], the project whose entry point demonstrates it is any project the [Toolchain] is genuinely installed into, the [Consuming project] included", reader 3 to "since that is where the [Rule] will be enforced.", reader 5 ending in an ellipsis; both add that "Bundled defence" and "Consuming project" are not defined in SPEC.md | +| Two techniques are independent; a rule run is not one of them | 3.1 | two: 2, 5 | 2: "a clean run of the [Rule]", "not evidence of saturation" and "is never one of the two", set beside clause 3.3's "A single execution of the [Rule] answers both" (J8). 5: by description, "Clause 3.1 defines independence for searches", set beside section 2 (J8). All in the paragraph beginning "Two techniques are independent when they would miss different things." | +| Narrowings reported to the owner as if suppressions | 3.3, Part B | one: 1 | "a [Narrowing] that excludes more code than the [Rule] still covers MUST be reported to the [Owner] as if it were a [Suppression]." (J7) | +| What a defence must satisfy of clause 3.5 | 3.5 and 7 | one: 2 | Section 7's "it fails rather than warns", in the paragraph beginning "A [Defence] [Conforms]", and clause 3.5's "does not [Conform]", in its opening paragraph (J8) | +| Proving and sweeping in one run | 3.3, Part A | one: 2 | "A single execution of the [Rule] answers both", set beside clause 3.1 (above, J8) | +| An external URL | 3.6 | one: 2 | "An external URL [Conforms] only where the project controls it and versions it alongside the [Rule]." | +| Check order and conformance | 5, 7 and 8, Scope | one: 4 | 5: "a project whose checks run in another order is a project that does not [Conform]". 7: "Nothing here constrains how the project runs its checks or what it does when they fail". 8: "How a project chooses to run its quality checks" and "is the project's own business and no part of this specification." (J6, J8) | +| Independent techniques for expressing a rule | 2 | one: 5 | "at least two independent techniques tried at expressing the [Rule]" and "Two techniques are independent when they would miss different things, as clause 3.1 says of a search.", in the "Attempt rather than pre-judge" paragraph; set beside clause 3.1 (above, J8) | +| A toolchain gap and the toolchain's claim | 3.6 and 3.2 | one: 5 | 3.6: "it is a [Toolchain] gap and blocks the [Toolchain]'s claim under section 7 until fixed." 3.2: "is recorded against the [Toolchain]'s claim under section 7 as well" (J8) | + +Readers' accounts of what they could not tell are recorded with the passage they attach to and are +not suggestions; none of the sheets proposes a change. + +**Earlier passages (the fourth row).** The open passages are those earlier runs of this document +found confusing by two or more and did not clear: run 23's three, and run 21's clause 3.2 passage, +which run 23 did not clear (run 19's other three and run 21's other three were cleared in runs 21 +and 23). As the paragraphs read in the changed document: + +- Clause 3.2's two grounds, run 21's passage, which run 23 read as the grounds sentence of the + paragraph beginning "**Qualifying is not the same as being the right host.**", the paragraph + then beginning "The language has a [Detector] of the stronger kind", and the record paragraph + beginning "On the [Remediation docs] of a [Rule] they write". The second of those now begins + "The second ground asks two questions"; readers 1, 3, 4 and 5 name its last two sentences. Not + cleared. +- Clause 3.1's lower-bound list item: named by no reader. Reader 2's and reader 5's clause 3.1 + passage lies in the paragraph beginning "Two techniques are independent", not the list item. + Cleared. +- The Owner entry in section 1: named by no reader. Cleared. +- The Narrowing entry in section 1: named by no reader. Reader 1's narrowing passage is clause + 3.3 Part B's sentence, which does not occur in the entry. Cleared. + +No sentence kept from one of the three cleared paragraphs is named elsewhere in the document. +Reader 5's section 2 passage lies in the "Attempt rather than pre-judge" paragraph that run 19 +named and run 21 cleared; it is not an open passage and is recorded above as one reader's. Reader +3 and reader 5's clause 3.5 passage was named by one reader in run 23 and is a new finding here. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in 0b2d190 and the commit after +it that settles its review (key quotations kept verbatim in d07c177 and that commit). SPEC.md: +clause 3.2's record sentence states both grounds in its main clause, and its paragraph on the second +ground is four short sentences; clause 3.5 says what a bundled defence is in plain words; clause +3.1's paragraph on independent techniques is shorter; clause 3.6's versioning requirement is one +short sentence. A fresh cohort follows. + +## Process change + +Made under a process change before it merged: yes, pull request 13 at commit 598908b, named +before the readers started. + +## Verdict + +**FAIL.** Against the criteria in `ACCEPTANCE.md`: + +- Load-bearing statements: not met. 2a, 2b and 2c five of five; 2d two of five. Readers 1 and 2 + state neither ground for a weaker kind, only that there are two; reader 5 does not give the + order semantic, syntactic, textual (J3) and names no place for the record (J2). +- Restatement and application: not met on 3.6, three of five (readers 2 and 3 grade Partial + without quoting the versioning sentence, J5); question 1 and every other keyed item five of + five. +- Misses traceable to the text, each a finding: 2d for readers 1, 2 and 5, each quoting clause + 3.2's record sentence "Where that kind is weaker than [Semantic], the [Practitioner] MUST also + record, on the [Rule]'s [Remediation docs] and in the description of the commit that adds the + [Rule], moves it to another [Detector], or first records the grounds, for each step down the + order [Semantic], [Syntactic], [Textual], which of the two grounds holds (the stronger kind + cannot express the [Rule], or the language has no [Detector] of the stronger kind that accepts + bespoke [Rules]) and why, naming the [Detectors] of the stronger kind that were considered." + with the grounds (readers 1 and 2) or the places (reader 5) elided; 3.6 for readers 2 and 3, + on clause 3.6's requirement that the documentation be shipped and versioned with the rule. +- Passages found confusing by two or more, each a finding that blocks until the text is changed + and a fresh cohort no longer names it: clause 3.2's paragraph beginning "The second ground asks + two questions", its last two sentences (four readers), which is also run 21's open passage, not + cleared; clause 3.5's sentence on the entry point for a [Bundled defence] (two); clause 3.1's + paragraph beginning "Two techniques are independent when they would miss different things." + (two, on J8). Run 23's three passages are cleared. +- Single-reader notes, recorded: clause 3.3 Part B's sentence on narrowings reported to the owner + (reader 1); clause 3.5's opening paragraph against section 7's defence paragraph, clause 3.3 + Part A's proving-and-sweeping paragraph, and clause 3.6's external URL sentence (reader 2); + sections 5, 7 and 8 on check order and conformance (reader 4); section 2's "Attempt rather than + pre-judge" paragraph, and clause 3.6's unresolved identifier paragraph against clause 3.2's + paragraph on the two kinds of gap (reader 5). + +Answer sheets, the reader prompt, the command and the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090845841), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090846386), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090846766), [comment 4](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090847033). diff --git a/acceptance/runs/026-semantic-detectors-run-8.md b/acceptance/runs/026-semantic-detectors-run-8.md new file mode 100644 index 0000000..85b7515 --- /dev/null +++ b/acceptance/runs/026-semantic-detectors-run-8.md @@ -0,0 +1,243 @@ +# Run 26: DETECTOR-SPEC.md, semantic detectors, eighth run in pull request 14 + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + editor's branch `editor/semantic-detectors` at dfb9b68; the question set, key and fixture read + at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input, with no tools, no settings, no project context + and no saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is byte for byte the question set, the document and the + fixture at dfb9b68, in that order, each preceded by one `===== FILE: =====` line, with the + files separated by one blank line and nothing else added, the same form as the packets of runs + 22 and 24 +- **Managed (policy) settings**: none; the machine applies no managed settings and no managed + settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. The editor reports none set aside for a cause the sheets cannot + show; every sheet has answer text, no process reports an error (`is_error` false, subtype + `success`), and every reported model is the cohort's model +- **Run within the pull request**: 8, counting void runs; run 25, made at the same time, does not + bear on this document +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review + +## Marks + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | Section 8 | Kind | 6.4 | Q4 entries | +| ------ | ------- | --- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --------- | ---------- | --- | ---------- | +| 1 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 2 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 3 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 4 | miss | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 5 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | + +"Q4 entries" counts the entries each reader lists under question 4; the words each quotes are in +the table of passages below. Every miss under question 2 is traceable to the text: each answer +quotes the document in support. The miss under question 1 is the reader's own: the bullet it turns +on quotes nothing. No question was left unanswered. + +### Question 1 + +Every restatement names four or more obligations from the key's list. Readers 1, 2, 3 and 5 assert +no obligation the document does not state: correct. Reader 3's last bullet, that "a verdict must +name the rule kind (Semantic, Syntactic, or Textual)", is the obligation the key marks correct on +a verdict (judgement 3). Readers 2 and 5 say findings must appear in, or be put in, the command's +own output, without "each" or "every" (judgement 2). Reader 4's sixth bullet says "The SHOULDs add +a reason requirement for suppressions, a release check for undocumented bundled rules, and a +conformance declaration that records known gaps"; the declaration is a MAY (8.1's heading, and "a +declaration is optional"), so the bullet asserts a recommendation the document does not make: +incorrect (judgement 4). Correct, four of five. + +### Question 2 + +The key's answers state these elements: + +- 2a: (a) Conformance is satisfying every MUST in sections 4 to 7; (b) judged on evidence; (c) a + detector that predates the document can conform; (d) the declaration is optional. +- 2b: (a) make the route disableable by configuration, or (b) detectable by a Rule the project can + write or a mechanical check the Detector documents; (c) the detector is not required to remove or + forbid it. +- 2c: (a) with the rule; (b) in the installed copy; (c) at a version tracked together; (d) yes, + reachable offline. + +Each answer is read as its sentence and its quotation together, for the gist of each element. + +| Reader | 2a | 2b | 2c | +| ------ | --------------------------------------------------------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------- | +| 1 | ok: all four; (b) and (d) from the 8.1 quotation | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6) | ok: all four; (a) and (c) from the 6.3 quotation | +| 2 | ok: all four; (b) and (d) from the 8.1 quotation | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6) | ok: all four; (b) from the 6.2 quotation, (c) from the 6.3 quotation | +| 3 | ok: all four; (d) from the 8.1 quotation | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6) | ok: all four; (a) and (c) from the 6.3 quotation | +| 4 | ok: all four; (d) from the section 8 and 8.1 quotations | miss: (c) not stated (judgement 5) | ok: all four | +| 5 | ok: all four, in the sentence, which says "a declaration is optional" | miss: (c) not stated (judgement 5) | ok: all four; (b) from the 6.2 quotation | + +No answer asserts anything the document does not state or that the key says makes an answer +wrong: none says a declaration is required, that partial satisfaction is conformance, that the +detector must forbid the route or require a reason, or that a website will do. Every quotation is +verbatim apart from letter case, link and emphasis markup and a closing punctuation mark; the 2c +quotations of readers 1, 2, 3, 4 and 5 are the headings of 6.2 and 6.3. + +### Question 3 + +The key's intended verdicts are Yes for every clause but 6.3, which is No; the section 8 verdict +is "does not conform", because 6.3 fails; 6.4 does not change the verdict. + +- **Every clause**: all five readers give the intended verdict on every clause, 4.1 to 7.1. On + 6.3 each cites that one unresolved identifier fails the clause, or quotes 6.3's "One + [Identifier] that does not resolve fails this clause, however many others do."; no reader grades + it Partial on the grounds that thirty-eight resolve. Readers 1, 2, 4 and 5 add caveats on 4.2 or + 4.3 about what the fixture does not say, and grade Yes. +- **Section 8**: all five say Lintomatic does not conform, because 6.3 fails. +- **Kind**: all five say the fixture does not let them name the kind and name none, so none is + marked on it. Readers 1, 3 and 5 call the verdict incomplete on that point and reader 2 says a + complete verdict would need it; none makes the verdict turn on it, and readers 3 and 5 say + outright that it does not change the verdict. +- **6.4**: all five say it does not change the verdict, because it is a SHOULD. Readers 3, 4 and 5 + add asides on what a declaration under 8.1 would record (judgement 7). + +### Judgements made in marking + +01. **How an open judgement is made.** As `ACCEPTANCE.md` at 598908b says: as a careful reader of + the document would, with the reason stated. Where a rule settles a mark, it is applied as + written. +02. **Question 1, "Findings" in the output, readers 2 and 5.** Reader 2 says "Findings must appear + in the output of the command the practitioner ran"; reader 5 that the detector must "put + findings in the command's own output". Run 24 marked incorrect a bullet saying "each finding + must appear", because 5.3 says "The summary need not list every finding". A plain plural + without "each" or "every" says what 5.3's heading says, "The result MUST reach the + practitioner in the output of the command they ran", and a careful reader takes it as that + heading, not as a claim about every finding. Correct. +03. **Question 1, reader 3's "rule kind".** Section 8 puts the naming of the kind on the verdict, + for each configuration graded; the terminology table defines the three kinds as "What a [Rule] + can ask in a configuration". The key marks correct a bullet saying a verdict must state the + detector's kind. "Rule kind" names the same three kinds by what a rule can ask, which the + document's own definition does, and a careful reader takes the bullet as the key's obligation + on the verdict. Correct. +04. **Question 1, reader 4's "SHOULDs".** The document defines its key words under RFC 2119, where + SHOULD recommends and MAY permits, and it says in 8.1 and section 8 that a declaration is + optional. A careful reader of a document that draws that line takes "The SHOULDs add ... a + conformance declaration" as stating a recommendation to declare that the document does not + make, and so an obligation the document does not state. Read the other way, as a loose list of + the non-MUST provisions, the bullet would be correct; Question 1 would then be five of five, and + neither reading changes whether the second criterion holds or adds a traceable miss, since the + bullet quotes nothing. +05. **2b (c), not required to remove or forbid, all five.** The question asks what the detector + must do about the route; the key's answer includes what it need not do, so (c) answers a part + of the question. Readers 1, 2 and 3 say the detector must do one of the two things; readers 4 + and 5 add "Either one is enough". That entails that nothing more is required, but it does not + say that removing or forbidding the route is not required: "Either one is enough" paraphrases + the first half of 7.1's sentence "Either alone satisfies this clause, and neither requires the + [Detector] to remove or forbid the route", not its second half, and no reader quotes that half + or 7.1's first sentence, "The [Detector] does not have to remove or forbid an inline + [Suppression] route." `ACCEPTANCE.md` names the ways an element is stated: the document's words, + a paraphrase, a synonym, a defined term in its defined sense. An element reached only by + inference from another is not among them, and a careful reader of that rule does not add it. + Marked not stated, as runs 22 and 24 marked the same element. The verdict does not turn on this + judgement alone: the fourth criterion fails whichever way it is made. +06. **2b (b), the means of detection.** Clause 7.1 defines "Detectable" in its own text. Readers 1, + 2 and 3 use "detectable" in that sense, and so state (b), as runs 22 and 24 held; reader 5's + "make every use of the route detectable" uses the definition's own words. It decides no mark, + since every 2b answer misses (c). +07. **Asides outside the keyed answer.** Reader 3's statement that 6.4 would not appear in + `known-gaps`, and readers 4's and 5's that a declaration would record the 6.3 gap, are not keyed + items; each sheet's answer to whether 6.4 changes the verdict is the key's, and is marked so. +08. **Question 4, a clause number inside a quotation, and context in the reader's explanation.** + Reader 4's quotation contains "as clause 4.1 lets it", and its explanation speaks of "a rule + written under 4.1"; reader 1's explanation of its section 8 entry speaks of "section 8's + conformance test"; reader 3's quotation contains "a [Detector] failing 4.1". Each reader heads + its entry with the clause it found confusing (7.1, 8 and 4.2) and quotes the words; a careful + reader takes the other clause as the context of the confusion, not a further passage named. + Read the other way, each would add a passage named by one reader only, in a paragraph no + earlier passage of this document left open (clause 4.1 was cleared by run 22, as run 24 + records), and nothing that blocks. +09. **Question 4, reader 5's quoted terms.** Reader 5 heads its entry "Section 2, the terminology + paragraph" and quotes "Every capitalised term links to its definition," and then the terms + "[Semantic], [Syntactic] or [Textual]" and "[Conform]" as the ones it found defined only by + links. `ACCEPTANCE.md` says a quotation covers what an occurrence of its words overlaps "at the + place the reader names it", and reaches every place the run occurs only "where the reader names + no place". Reader 5 names section 2's terminology paragraph for the whole entry, and neither + term's run occurs there (the terminology table, a separate paragraph, reads "[Semantic], + [Syntactic], [Textual]"). A careful reader takes the entry as the section 2 sentence, with the + terms as the examples the reader gives of it, so the term quotations cover no further sentence. + Read the other way, "[Semantic], [Syntactic] or [Textual]" would cover the first sentence of + section 8's paragraph on kind, and "[Conform]" the sentences of 5.4, 7.1's first paragraph, + section 8's paragraph on kind, 8.1's Why paragraph and section 9 where that word occurs; that + would make reader 5 a second reader of section 8's paragraph on kind and of 7.1's first + paragraph, both of which block already, and the verdict does not turn on it. +10. **Question 4, a reader that resolved its confusion.** Reader 2 says of its 6.1 and 6.3 passage + "I resolved this with 6.1's own pointer". Question 4 asks for any passage found confusing on + the one reading, and the reader names it as such "On my first read", so it is recorded. +11. **Question 4, closing punctuation.** Reader 1's "... it grades." ends with a full stop where + the document has a comma, and reader 5's "... to its definition," and reader 2's "... not for + every page," end with a comma where the document has a semicolon or a full stop. The words are + consecutive and verbatim; the punctuation mark that follows them is not a word, so each is a + quotation. +12. **The open passages.** The passages earlier runs of this document found confusing by two or + more, and not since cleared, are those run 24 left outstanding or added: 7.1's first paragraph + (runs 20, 22 and 24), 8.1's second paragraph (runs 20, 22 and 24) and section 8's paragraph + beginning "A verdict on a [Detector] MUST also name" (run 24, four readers). The text of each + changed after run 24, in b172fd6, and each is taken as it reads at dfb9b68. + +No re-mark. + +## Passages quoted as confusing + +Grouped by paragraph, as `ACCEPTANCE.md` defines the same passage. A passage blocks when two or +more readers name it, or when it covers a sentence of a paragraph in which one of the open +passages lay (judgement 12), as that paragraph reads at dfb9b68. + +| Clause and paragraph | Readers | Words quoted, verbatim | Blocks | +| ---------------------------------------------------------------------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | +| 8, the paragraph beginning "A verdict on a [Detector] MUST also name", its first sentence | 1 | "A verdict on a [Detector] MUST also name the kind of each configuration it grades." (judgement 11) | Yes: the open section 8 paragraph of run 24 | +| 7.1, first paragraph, the sentences "Detectable means ...", "There are two ways." and "One is a [Rule] ..." | 4 | "Detectable means the project can find every use of the route. There are two ways. One is a [Rule] the project writes and registers with the [Detector], as clause 4.1 lets it." (judgement 8) | Yes: the open 7.1 paragraph of runs 20, 22 and 24 | +| 4.3, first paragraph, the sentence beginning "The [Detector] MUST print it, unaltered" | 1 | "in its default output and in every machine-readable format it offers." | No | +| 4.2, first paragraph, its second sentence | 3 | "This applies to every [Rule] the [Detector] runs, bundled or bespoke; a [Detector] failing 4.1 is graded here on its [Bundled rules]." (judgement 8) | No; one reader of run 24 also named it | +| 6.1, the paragraph beginning "In both", the sentences "This clause asks for the mechanism ..." and "Where the mechanism works ..." | 2 | "This clause asks for the mechanism, not for every page," and "Where the mechanism works but a [Bundled rule]'s [Identifier] has no documentation to resolve to, that fails clause 6.3, not this clause." (judgements 10 and 11) | No; one reader of run 22 also named the second sentence | +| 6.3, first paragraph, its first sentence | 2 | "Every [Identifier] a [Bundled rule] can print MUST resolve, under clause 6.1, to a page in that shipped documentation." (judgement 10) | No | +| 2, first paragraph, its second sentence | 5 | "Every capitalised term links to its definition," and the terms "[Semantic], [Syntactic] or [Textual]" and "[Conform]" (judgements 9 and 11) | No | + +No passage is named by two or more readers of this cohort, on the reading of judgement 9. Of the +open passages, 8.1's second paragraph is cleared: its text changed after run 24 and no reader of +this cohort names a passage covering a sentence of it, on either reading of judgement 9. 7.1's +first paragraph and section 8's paragraph on kind are still outstanding. + +## Against the criteria + +- **Load-bearing statements (every reader on every one)**: not met. 2a five of five, 2b none of + five, 2c five of five. +- **Restatement and application (all but one on each keyed item)**: met. Question 1 four of five; + every clause of question 3, the section 8 verdict and 6.4 five of five. +- **A miss traceable to the text (none)**: not met. Five, each a finding: 2b by all five readers, + each quoting 7.1's sentence "For each route it ships, it MUST do at least one of two things: ...", + none stating that the detector need not remove or forbid the route. +- **A passage found confusing by two or more (none outstanding)**: not met. The open passages in + 7.1's first paragraph (reader 4) and section 8's paragraph on kind (reader 1) are still named by + this cohort. +- **From one reader only**: the 4.3 sentence on machine-readable formats, the 4.2 sentence on a + detector failing 4.1, the 6.1 sentences on the mechanism and a missing page, the 6.3 sentence on + every identifier resolving, and the section 2 sentence on capitalised terms, recorded. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in 0b2d190 and the commit after +it that settles its review (key quotations kept verbatim in d07c177 and that commit). DETECTOR- +SPEC.md: clause 7.1's MUST sentence says the detector need not remove or forbid the route, and its +first paragraph is plainer; section 8's paragraph on the kind is shorter. A fresh cohort follows. + +## Process change + +Made under the process change of pull request 13 before it merged, at commit 598908b, named +before the readers started; `ACCEPTANCE.md` and `acceptance/runs/README.md` in this worktree +differ from 598908b only by this pull request's question-set line adding question 2d for SPEC.md, +which does not bear on this document. + +## Verdict + +FAIL on the first, third and fourth criteria. Answer sheets, the readers' prompt and command, and +the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090847649), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090848005), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090848260). diff --git a/acceptance/runs/027-semantic-detectors-run-9.md b/acceptance/runs/027-semantic-detectors-run-9.md new file mode 100644 index 0000000..54eb638 --- /dev/null +++ b/acceptance/runs/027-semantic-detectors-run-9.md @@ -0,0 +1,266 @@ +# Run 27: SPEC.md, semantic detectors, ninth run in pull request 14 + +- **Document**: SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + branch `editor/semantic-detectors` at 1db0c67, the commit the readers were given; the question + set, key and fixture read at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input; no tools, no settings, no project context and no + saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is byte for byte the question set, the document and the + fixture at 1db0c67, in that order, each preceded by one `===== FILE: =====` line, with + nothing else added apart from one blank line between the end of each file and the next file's + marker line. +- **Managed (policy) settings**: none; the machine applies no managed settings to the readers and + no managed settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. Every sheet has answer text, no process reports an error + (`is_error` false, subtype `success`), and every reported model is the cohort's model. The + editor reports no reader set aside for a cause the sheets cannot show. +- **Run within the pull request**: 9, counting void runs; runs in the pull request so far, + counting void runs: 9, as the editor supplied. Run 28 is made at the same time and does not bear + on this document. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review. It read `ACCEPTANCE.md` and + `acceptance/runs/README.md` in this worktree, which differ from commit 598908b only by this pull + request's question-set line adding 2d for SPEC.md; the document, question set, key and fixture + at 1db0c67; run 25 in full, the passages and verdicts of runs 19, 21 and 23, and the headers of + runs 20, 22, 24 and 26, which are on DETECTOR-SPEC.md; and, for the fourth pass-criteria row, + the change to SPEC.md between dfb9b68, the document run 25 read, and 1db0c67. + +## Marks + +`ok` is correct, `miss` is a miss; `T` marks a miss traceable to the text. Judgements J1 to J8 +are set out below. + +| Reader | Q1 | 2a | 2b | 2c | 2d | 3.1 | 3.2 | 3.3 | 3.4 | 3.5 | 3.6 | Remediation | Defence | Section 4 | Q4 passages | +| ------ | ------- | --- | --- | --- | ------ | ------- | ------- | ------ | --- | --- | ------- | ----------- | ------- | --------- | ----------- | +| 1 | correct | ok | ok | ok | miss T | Partial | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 2 | correct | ok | ok | ok | ok | No | No | Yes ok | No | No | No | ok | ok | ok | two | +| 3 | correct | ok | ok | ok | miss T | No | No | Yes ok | No | No | No | ok | ok | ok | two | +| 4 | correct | ok | ok | ok | ok | Partial | Partial | Yes ok | No | No | Partial | ok | ok | ok | two | +| 5 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | + +Every grade under 3.1, 3.2, 3.4, 3.5 and 3.6 is one the key accepts; reader 4's 3.6 Partial +quotes the versioning sentence (J7). No reader grades 3.2, 3.5 or 3.6 Yes, so none of the key's +cases that count against the document arises. + +### Question 1 + +Five of five correct. Every sheet has at least four bullets naming obligations the key lists (the +class bounded both ways with its hazard, the strongest kind of detector and never a test, the +proof in a commit of its own, the sweep with its count and the fix of every instance, permanent +and blocking, the terse message with an identifier resolving to versioned documentation, the +record of a deferred defect), and no bullet asserts an obligation the document does not state. +The section 2 bullets of readers 1, 2 and 4, on building the defence first or attempting rather than +pre-judging, state section 2's obligations (J6 for reader 4). The closing "fix the original defect with a reproduction test" +of readers 1, 3 and 5 is section 2 and section 6. Reader 3's line after its six bullets, on what +belongs to the Owner, states section 4 and is not a seventh bullet; no mark turns on it. + +### Question 2 + +Every reader quoted words of the document for every item, so every miss here is traceable to the +text. + +- **2a**, five of five. Each states class, rule, proof, sweep and the fix of every instance, in + that order, and quotes section 2's sentence "The order is section 3's: attribute the [Defect] to + a [Class], build the [Rule], prove that it fires, run the [Sweep], then fix every [Instance]." +- **2b**, five of five. Each says only the owner may and an agent may not, and quotes the key's + section 4 sentence. Every reader says the owner is a human, readers 4 and 5 quoting "The [Owner] + is always a human." +- **2c**, five of five. Each states all six clauses and a verdict resting on reproduction, and + each quotes section 7's "MUST rest on reproduction, not on the report"; no reader accepts the + pull request description as the verdict. +- **2d**, two of five (readers 2 and 4). The key's three elements are (i) the strongest kind, in + the order semantic, syntactic, textual; (ii) both grounds for a weaker kind; (iii) the reason + recorded, with at least one place named and the places not given as alternatives (J2). + - Reader 1: (i) and (ii) in its two quotations, the strongest-kind sentence and the sentence + stating the two grounds. (iii) missed: "the practitioner must record which ground holds and + why" names no place, and neither quotation names one (J2). + - Reader 2: (i) and (ii) in its quotations, (iii) in its sentence, "in the rule's remediation + docs and in the commit description", with its quotation of "naming the [Detectors] of the + stronger kind that were considered". Correct. + - Reader 3: (i) in its quotation. (ii) missed: "A weaker kind is allowed only on two grounds" + says there are two and states neither, and its only quotation is the strongest-kind sentence. + (iii) missed: "record which ground holds and why" names no place (J2). + - Reader 4: (i), (ii) and (iii) in its three quotations, the third giving the places joined by + "and" (J3). Correct. + - Reader 5: (i) in its quotation. (ii) missed: "only on one of two grounds" states neither. + (iii) missed: "record which ground holds at each step down, with the stronger detectors + considered" names no place (J2). + +### Question 3 + +- **3.1**: readers 2, 3 and 5 No, readers 1 and 4 Partial, each noting that no two independent + search techniques are recorded. +- **3.2**: readers 1, 4 and 5 Partial, readers 2 and 3 No, each saying the detector or its kind is + not named; readers 1, 4 and 5 say the rule is not a test, and every reader adds that no grounds + for a weaker kind are recorded, which the key does not mark wrong. +- **3.3**: five of five Yes. Readers 1, 3, 4 and 5 say the inline ignore is not judged here; none + attributes it to 3.3. +- **3.4** and **3.5**: five of five No, each naming the inline ignore on `Report::render()`, under + 3.5 as uncovered by any recorded decision. +- **3.6**: five of five. Readers 1, 2, 3 and 5 grade No; reader 4 grades Partial and quotes "A + [Rule]'s documentation MUST be versioned with the [Rule]." (J7). +- **Remediation**: five of five say it does not conform, each naming 3.4 among the failing clauses + (J8). +- **Defence**: five of five say it does not conform, each naming 3.6 as not met and 3.2 as well, + readers 3 and 4 saying 3.2 cannot be shown or confirmed and reader 5 that it is only partial. Every reader also counts 3.1 or 3.5 against the defence; the key does + not mark that wrong. +- **Section 4**: five of five name the inline ignore on `Report::render()`, and each also names + leaving the next wider rule unbuilt, which the key accepts alongside it. + +### Judgements made in marking + +Made as a careful reader of the document would, as `ACCEPTANCE.md` requires. Where a rule settles +the point, the record says which. + +- **J1. The sentence.** Where a reader wrote more than one sentence before or around its + quotations, all of that text is read with the quotations as the answer. No mark turns on it. +- **J2. 2d's element (iii) needs a place named.** The key says of (iii) that "naming one of the + two places is enough" and that an answer presenting them as alternatives is wrong; "enough" sets + a minimum, and a key that let an answer name no place would have no minimum to state and no + alternatives to forbid. Runs 19, 21, 23 and 25 read it the same way. Reader 1's miss on 2d turns + on it; readers 3 and 5 miss (ii) as well. Read otherwise, reader 1 would be correct and 2d three + of five; 2d would still fail on readers 3 and 5, whose misses on (ii) rest on no judgement, and + the verdict would not change. +- **J3. Elided quotations.** A quotation with an ellipsis is read as one quotation per unbroken + run of words, each in the document's order, and states only what those runs state (reader 4's + third 2d quotation; reader 5's clause 3.2 passage). Settled by the rule on quoting: consecutive + words of any length. Every run in reader 4's quotation is verbatim. +- **J4. Clause labels.** A clause number given with a quotation, as question 4 asks, is read as + the quotation's label, not as naming the whole clause, as runs 21, 23 and 25 read it. Reader 2's + "Section 3.3, Part B" and reader 3's "Clause 3.3, Part B" labels do not extend their passages to + all of Part B, so the two are not the same passage: reader 2's lies in the paragraph beginning + "A [Narrowing] reduces the [Practitioner]'s own work" and the table's first row, reader 3's in + the paragraph beginning "**An inline ignore comment, an ignore entry or a [Baseline]**". +- **J5. Passages set beside the main one count.** Where a reader says it was confused by two + passages read together, a careful reader takes both as the confusing passage, since the + confusion lies in neither alone, as runs 23 and 25 read it. So reader 3's section 7 item covers, + by its description "the recorded-decision condition that clause 3.5 itself sets", the sentences + of clause 3.5's opening paragraph that set it; reader 4's clause 3.5 item covers section 7's + "it fails rather than warns"; reader 2's Part B item covers the table's first row, which gives + the [Narrowing] to the [Practitioner]; reader 4's clause 3.1 item covers both step 4 and the + paragraph beginning "**What this clause leaves on the record**". Reader 2's section 2 item says + that "In 3.1 the same phrase refers to ways of searching for instances" and that it could not + tell "whether the two uses were meant to share the same test of independence"; the test of + independence is clause 3.1's sentence "Two techniques are independent when they would miss + different things.", so the description is read as naming that sentence, as run 25 read reader + 5's like description. A finding turns on this: clause 3.1's paragraph beginning with that + sentence, run 25's open passage, is not cleared on it. Read as context only, that paragraph is + named by no reader and is cleared; the verdict would not change. +- **J6. "Two independent search techniques" (reader 4, question 1).** Reader 4's second bullet + says the out-of-scope conclusion is stated "in one sentence naming two independent search + techniques"; section 2's sentence asks for "at least two independent techniques tried at + expressing the [Rule]". Read for its gist, the bullet names section 2's obligation to state that + conclusion in one sentence naming two independent techniques and to record it, an obligation + the document states, with the techniques mislabelled; it is marked correct. Read as asserting a + different obligation, it would be the reader's miss, since the bullet quotes nothing, question 1 + would be four of five and the second criterion still met. Reader 2 names the same sentence under + question 4. +- **J7. 3.6 Partial and "the versioning sentence" (reader 4).** The key accepts Partial "where the + reasoning quotes the versioning sentence". Clause 3.6's versioning requirement is now the one + sentence "**A [Rule]'s documentation MUST be versioned with the [Rule].**", which reader 4 quotes + verbatim; it is the versioning sentence on either sense run 25 considered, so no mark turns on + which. +- **J8. The remediation item.** The key's answer is that the remediation does not conform, giving + 3.2, 3.4 and 3.6 as the reason. A reader saying it does not conform and naming among the failing + clauses one the key names is correct, as runs 23 and 25 marked it; every reader names 3.4. Reader + 4 names 3.1, 3.4 and 3.5, having graded 3.2 and 3.6 Partial; it asserts nothing the key marks + wrong. + +## Passages quoted as confusing + +Same passage as `ACCEPTANCE.md` defines it: sentences in common, or the same paragraph; a list +item, table row or terminology entry is its own paragraph. + +| Passage | Clause | Readers | Words quoted, verbatim, or the description used | +| ------------------------------------------------------------- | ----------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Where the grounds for a weaker kind are recorded | 3.2 | two: 1, 5 | 1: "on the [Rule]'s [Remediation docs] and in the description of the commit that adds the [Rule], moves it to another [Detector], or first records the grounds." 5: "...naming the [Detectors] of the stronger kind that were considered, on the [Rule]'s [Remediation docs] and in the description of the commit that adds the [Rule], moves it to another [Detector], or first records the grounds." Both in the paragraph beginning "On the [Remediation docs] of a [Rule] they write". Run 21's open passage (below) | +| What a defence must satisfy of clause 3.5 | 3.5 and 7 | two: 3, 4 | 3: "A Defence Conforms if it satisfies clauses 3.1, 3.2, 3.3, 3.5 and 3.6: … it fails rather than warns, …", in section 7's paragraph beginning "**A [Defence] [Conforms]**", set beside, by description, "the recorded-decision condition that clause 3.5 itself sets" (J3, J5). 4: "neither does a [Rule] whose green run depends on a [Suppression] that no recorded decision covers", in clause 3.5's opening paragraph, set beside section 7's "it fails rather than warns" (J5) | +| Independent techniques for expressing a rule and for a search | 2 and 3.1 | one: 2 | "stated in one sentence, naming at least two independent techniques tried at expressing the [Rule]", in section 2's "Attempt rather than pre-judge" paragraph, set beside, by description, clause 3.1's "same test of independence" (J5). Run 25's open clause 3.1 passage (below) | +| Narrowings enumerated and reported as if suppressions | 3.3, Part B | one: 2 | "Every [Narrowing] MUST be enumerable by the same means as the project's [Exceptions]..." and "a [Narrowing] that excludes more code than the [Rule] still covers MUST be reported to the [Owner] as if it were a [Suppression].", in the paragraph beginning "A [Narrowing] reduces the [Practitioner]'s own work", set beside, by description, the table's [Narrowing] row (J3, J4, J5) | +| An inline ignore is not judged under clause 3.3 | 3.3, Part B | one: 3 | "An inline ignore comment, an ignore entry or a Baseline left over an Instance that carries the Hazard is not judged under this clause. Leaving an Instance that carries the Hazard in place behind one is not a Narrowing, whatever note accompanies it. It is a Suppression of the Instance. Clauses 3.4 and 3.5 and section 4 govern it, and it does not fail this clause." (J4) | +| Leaving the next wider rule unbuilt | 4 | one: 1 | "It is a reason not to widen unasked, not authority to leave the wider [Rule] out, and recording it as a known gap does not make it the [Practitioner]'s decision.", in section 4's list item beginning "Leaving unbuilt the next wider [Rule]" | +| Step 4 and the record of the next wider rule | 3.1 | one: 4 | "Where the [Class] stays at the reported [Instance], name the next wider [Rule] and why it was not built.", step 4 of the five steps, set beside "the next wider [Rule] that was not built, with the reason", in the paragraph beginning "**What this clause leaves on the record**" (J5) | +| The default owner's decisions recorded | 1, Owner | one: 5 | "The default [Owner] is usually the person under the same deadline as the [Practitioner], so a decision they take under section 4 to keep an [Instance] unfixed, [Suppress], [Baseline] or remove a [Rule] MUST be recorded in the project record with its justification." | + +Readers' accounts of what they could not tell are recorded with the passage they attach to and are +not suggestions; none of the sheets proposes a change. + +**Earlier passages (the fourth row).** The open passages are those earlier runs of this document +found confusing by two or more and did not clear: run 25's three, of which one is run 21's clause +3.2 passage, not cleared by runs 23 or 25. Every other passage of runs 19, 21 and 23 was cleared +in runs 21, 23 and 25. As the paragraphs read in the changed document: + +- Clause 3.2's grounds and record, run 21's passage, which run 23 read as the grounds sentence of + the paragraph beginning "**Qualifying is not the same as being the right host.**", the paragraph + now beginning "For the second ground", and the record paragraph beginning "On the \[Remediation + docs\] of a [Rule] they write", whose record sentences run 23 found kept from run 21's paragraph. + Readers 1 and 5 name the last part of the record paragraph's second sentence. Not cleared. +- Clause 3.5's sentence on the entry point for a [Bundled defence], now the paragraph beginning "A + [Bundled defence] is a [Defence] that a shipped [Toolchain] enables by default": named by no + reader. Readers 3 and 4 name clause 3.5's opening paragraph, not this one. Cleared. +- Clause 3.1's paragraph beginning "Two techniques are independent when they would miss different + things.": named by reader 2 by description (J5). Not cleared. + +No sentence kept from the cleared paragraph is named elsewhere in the document. Three passages +named here by one reader were once open and have since been cleared: section 4's list item on +leaving the next wider rule unbuilt (run 21, three readers; cleared in run 23), named by reader +1; the Owner entry in section 1 (run 23, two readers; cleared in run 25), named by reader 5; and +section 2's "Attempt rather than pre-judge" paragraph (run 19; cleared in run 21), named by reader +2\. Each is recorded as one reader's. The section 7 and clause 3.5 passage named here by readers 3 +and 4 was named by one reader, reader 2, in run 25, and is a new finding here. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in c6ee1a8 and the commit after +it that settles its review (key quotations kept verbatim in 224579f and that commit). The editor +first read every sheet, in runs 19 to 27, that named each passage, and fixed the cause the sheets +show. SPEC.md: clause 3.2 states the two grounds in their own sentence after the strongest-kind +sentence and names the two places early in the record sentence; section 7 states that a defence +conforms only where clause 3.5 is met in full, including its recorded-decision condition; clause 3.1 +uses "technique" in one sense only and says when a search per spelling counts. A fresh cohort +follows. + +## Process change + +Made under a process change before it merged: yes, pull request 13 at commit 598908b, named +before the readers started. + +## Verdict + +**FAIL.** Against the criteria in `ACCEPTANCE.md`: + +- Load-bearing statements: not met. 2a, 2b and 2c five of five; 2d two of five. Readers 3 and 5 + state neither ground for a weaker kind, only that there are two; readers 1, 3 and 5 name no + place for the record (J2). +- Restatement and application: met. Question 1 and every keyed item of question 3 five of five. +- Misses traceable to the text, each a finding: 2d for readers 1, 3 and 5, each quoting clause + 3.2's sentence "A [Rule] the [Practitioner] writes MUST be written in the strongest kind of + [Detector] that can express it: a [Semantic] one, else a [Syntactic] one, else a [Textual] + one."; readers 3 and 5 on the grounds for a weaker kind, which they say are two without stating + them, and readers 1, 3 and 5 on the record of those grounds, which they give without the place + it is recorded (reader 1's on J2). +- Passages found confusing by two or more, each a finding that blocks until the text is changed + and a fresh cohort no longer names it: clause 3.2's record paragraph beginning "On the + [Remediation docs] of a [Rule] they write", the places named at the end of its second sentence + (two readers), which is also run 21's open passage, not cleared; section 7's paragraph beginning + "**A [Defence] [Conforms]**", its "it fails rather than warns", against clause 3.5's opening + paragraph on a [Suppression] no recorded decision covers (two). Run 25's clause 3.1 paragraph + beginning "Two techniques are independent when they would miss different things." is not + cleared (one reader, on J5). Run 25's clause 3.5 passage on the [Bundled defence] is cleared. +- Single-reader notes, recorded: section 4's list item on leaving the next wider rule unbuilt + (reader 1); section 2's "Attempt rather than pre-judge" paragraph, and clause 3.3 Part B's + paragraph on narrowings enumerated and reported as if suppressions with the table's [Narrowing] + row (reader 2); clause 3.3 Part B's paragraph on an inline ignore (reader 3); clause 3.1's step + 4 against its record paragraph (reader 4); the Owner entry's sentence on the default owner's + decisions (reader 5). + +Answer sheets, the reader prompt, the command and the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090998641), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090998842), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090999356), [comment 4](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6090999753). diff --git a/acceptance/runs/028-semantic-detectors-run-10.md b/acceptance/runs/028-semantic-detectors-run-10.md new file mode 100644 index 0000000..6404294 --- /dev/null +++ b/acceptance/runs/028-semantic-detectors-run-10.md @@ -0,0 +1,253 @@ +# Run 28: DETECTOR-SPEC.md, semantic detectors, tenth run in pull request 14 + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + editor's branch `editor/semantic-detectors` at 1db0c67; the question set, key and fixture read + at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input, with no tools, no settings, no project context + and no saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is byte for byte the question set, the document and the + fixture at 1db0c67, in that order, each preceded by one `===== FILE: =====` line, with the + files separated by one blank line and nothing else added, the same form as the packets of runs + 22, 24 and 26 +- **Managed (policy) settings**: none; the machine applies no managed settings and no managed + settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. The editor reports none set aside for a cause the sheets cannot + show; every sheet has answer text, no process reports an error (`is_error` false, subtype + `success`), and every reported model is the cohort's model +- **Run within the pull request**: 10, counting void runs; run 27, made at the same time, does not + bear on this document +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review + +## Marks + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | Section 8 | Kind | 6.4 | Q4 entries | +| ------ | ------- | ---- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --------- | ---------- | --- | ---------- | +| 1 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 2 | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 3 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 4 | correct | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | + +"Q4 entries" counts the entries each reader lists under question 4; the words each quotes are in +the table of passages below. Every miss under question 2 is traceable to the text: each answer +quotes the document in support. The miss under question 1 is the reader's own: the bullet it turns +on quotes nothing. No question was left unanswered. + +### Question 1 + +Every restatement names four or more obligations from the key's list. Readers 1, 3, 4 and 5 assert +no obligation the document does not state: correct. Reader 3's bullet that "a verdict must name the +detector's kind, Semantic, Syntactic or Textual (section 8)" is the obligation the key marks correct +on a verdict, and its bullet that "Inline suppressions should require a written reason (7.2, +SHOULD)" states a recommendation the document makes, as a recommendation (judgement 3). Reader 4's +first bullet and the last sentence of reader 2's sixth state what conformance is; they name no +obligation and assert none the document does not state. Readers 1 and 5 say a printed identifier +must resolve to its documentation, in the generality of 6.1's heading, as the key's own bullet does +(judgement 4). Reader 2's fourth bullet says "every finding must appear in the output of the command +the practitioner ran"; 5.3 says "The summary need not list every finding", so the bullet asserts an +obligation the document does not state: incorrect (judgement 2). Correct, four of five. + +### Question 2 + +The key's answers state these elements: + +- 2a: (a) Conformance is satisfying every MUST in sections 4 to 7; (b) judged on evidence; (c) a + detector that predates the document can conform; (d) the declaration is optional. +- 2b: (a) make the route disableable by configuration, or (b) detectable by a Rule the project can + write or a mechanical check the Detector documents; (c) the detector is not required to remove or + forbid it. +- 2c: (a) with the rule; (b) in the installed copy; (c) at a version tracked together; (d) yes, + reachable offline. + +Each answer is read as its sentence and its quotation together, for the gist of each element. + +| Reader | 2a | 2b | 2c | +| ------ | -------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | -------------------------------------------------------------------- | +| 1 | ok: all four; (d) from the sentence, "not on a declaration", and the 8.1 quotation | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6) | ok: all four; (a) and (c) from the 6.3 quotation | +| 2 | ok: all four; (a) from the section 8 quotation, (b) from the 8.1 quotation | ok: all three; (c) from the 7.1 quotation, "it need not remove or forbid the route" | ok: all four; (b) from the 6.2 quotation, (c) from the 6.3 quotation | +| 3 | ok: all four | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6) | ok: all four; (c) from the 6.3 quotation | +| 4 | miss: (a) not stated, "judged on evidence against the MUSTs" (judgement 7); (d) stated | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6) | ok: all four | +| 5 | ok: all four | ok: all three; (c) from the 7.1 quotation; (b) as "make every use of it detectable" | ok: all four; (b) from the 6.2 quotation | + +No answer asserts anything the document does not state or that the key says makes an answer +wrong: none says a declaration is required, that partial satisfaction is conformance, that the +detector must forbid the route or require a reason, or that a website will do. Every quotation is +verbatim apart from letter case, link and emphasis markup and a closing punctuation mark; reader 4's +8.1 quotation stops before the colon of its sentence, and reader 1's 2a quotation is the whole of +8.1's sentence. + +### Question 3 + +The key's intended verdicts are Yes for every clause but 6.3, which is No; the section 8 verdict +is "does not conform", because 6.3 fails; 6.4 does not change the verdict. + +- **Every clause**: all five readers give the intended verdict on every clause, 4.1 to 7.1. On + 6.3 each cites that one unresolved identifier fails the clause, readers 1, 2 and 3 quoting 6.3's + "One [Identifier] that does not resolve fails this clause"; no reader grades it Partial on the + grounds that thirty-eight resolve. Readers 1, 2, 3 and 4 add caveats on 4.2 or 4.3 about what the + fixture does not say, and grade Yes. +- **Section 8**: all five say Lintomatic does not conform, because 6.3 fails. +- **Kind**: all five say the fixture does not let them name the kind and name none, so none is + marked on it. Reader 3 calls the verdict incomplete on that duty and reader 4 says it cannot + complete the verdict from the fixture; none makes the verdict turn on the kind, and readers 1, 2, + 3 and 5 say outright that it does not change the verdict or the outcome. +- **6.4**: all five say it does not change the verdict, because it is a SHOULD. Readers 2 and 3 add + that 7.2 is an unmet SHOULD, reader 2 that a declaration would record the 6.3 gap, and reader 1 + that the toolchain MUST of 6.4's Why paragraph does not apply (judgement 8). + +### Judgements made in marking + +01. **How an open judgement is made.** As `ACCEPTANCE.md` at 598908b says: as a careful reader of + the document would, with the reason stated. Where a rule settles a mark, it is applied as + written. +02. **Question 1, reader 2's "every finding must appear".** Runs 22 and 24 marked incorrect bullets + saying every finding, or each finding, must appear in the output, because 5.3 says "The summary + need not list every finding" and "If it leaves any finding out, the output MUST also say where + the rest can be read". Run 26 marked correct a plain plural, "findings", as 5.3's heading. + Reader 2's "every finding" is the first case, not the second: a careful reader takes it as a + claim about each finding, which 5.3 contradicts. Incorrect. The bullet quotes nothing, so the + miss is the reader's. +03. **Question 1, reader 3's two further statements.** The kind bullet is the obligation the key + marks correct on a verdict, as run 26 held for the same words. The 7.2 statement is labelled a + SHOULD and is what 7.2 says, "The [Detector] SHOULD reject, or be configurable to reject, an + inline [Suppression] that carries no reason"; unlike run 26's reader 4, who called a MAY a + SHOULD, it asserts nothing the document does not state. Correct. +04. **Question 1, "a printed identifier must resolve", readers 1 and 5.** For a project's own rule + 6.1 asks only that the identifier be printed unaltered. The bullets speak at the level of 6.1's + heading, "The detector MUST provide a mechanism that resolves a printed identifier to its + documentation", and the key's bullet is in the same words, so a careful reader takes them as + that obligation. Correct. +05. **2b (c), not required to remove or forbid, readers 1, 3 and 4.** As runs 22, 24 and 26 held, + the key's answer includes what the detector need not do, so (c) answers a part of the question. + Readers 1, 3 and 4 say the detector must do one of the two things. Each quotes 7.1 from "it MUST + do at least one of two things", which follows the words "it need not remove or forbid the + route"; reader 1 adds in its sentence that either alone is enough, and readers 3 and 4 quote on + to "Either alone satisfies this clause." That either alone suffices entails that nothing more is required, but + does not say that removing or forbidding the route is not required, and an element reached only + by inference from another is not among the ways `ACCEPTANCE.md` names of stating it. Marked not + stated, as the earlier runs marked the same element. Readers 2 and 5 quote the clause from "For + each route it ships, it need not remove or forbid the route", and so state it. +06. **2b (b), the means of detection.** Clause 7.1 defines when a route is detectable in its own + text; readers 1, 3 and 4 use "detectable" in that sense and so state (b), as runs 22, 24 and 26 + held. It decides no mark, since each of those answers misses (c). +07. **2a (a), reader 4's "judged on evidence against the MUSTs".** The question asks what it means to + conform. The key's element is that conformance is satisfying every MUST in sections 4 to 7, and + it says an answer that partial satisfaction is conformance is wrong, so the element's gist is the + threshold, every MUST, not only what is measured. Reader 4's sentence says what conformance is + judged against, which is as true of a detector that fails one MUST as of one that meets them all, + and its quotations, section 8's second sentence and 8.1's sentence on a detector that predates + the document, do not carry "every MUST in sections 4 to 7" either. A careful reader does not + take "judged against the MUSTs" as saying all of them must be met. Not stated: a miss, and since + the answer quotes the document, traceable. Read the other way, the answer would be correct and + the third criterion would lose one finding; the verdict does not turn on this judgement, since + 2b fails the first and third criteria by three readers whichever way it is made. +08. **Asides outside the keyed answer.** Readers 2's and 3's notes on 7.2, reader 2's on what a + declaration would record, and reader 1's on the toolchain MUST of 6.4's Why paragraph are not + keyed items; each sheet's answer to whether 6.4 changes the verdict is the key's, and is marked + so. +09. **Question 4, reader 4's second entry, named by description.** Reader 4 writes "Clauses 6.1, 6.4, + 7.1 and section 9 refer to clauses in `TOOLING-SPEC.md` and `SPEC.md`, which I was not given", + with no quotation. `ACCEPTANCE.md` says a passage named without a quotation, by clause number or + description, covers every sentence of what it names. A careful reader takes what this entry + names as the references in those clauses to the other two specifications, not every sentence of + each clause: the clause numbers say where the references are, and the description says which + sentences confused the reader. So taken, it covers, in 6.1, the sentences of the paragraph + beginning "For a project's own [Rule]" that cite method clause 3.6 and clause 4.2 of the + toolchain specification, and the Why paragraph's first sentence; in 6.4, the Why paragraph's + sentence on the toolchain specification's self-audit; in 7.1, the Why paragraph's sentences + citing method clause 3.4 and section 4 and clause 4.3 of the toolchain specification; and in + section 9, the sentences that refer to the method or toolchain specification. Read the other + way, as every sentence of each clause, it would also cover 7.1's first paragraph, which blocks + already, and 6.1's heading, which reader 2 also names, making that heading a further passage + named by two readers; the fourth criterion fails either way, and the verdict does not turn on it. +10. **Question 4, a passage quoted only to set another against it.** Reader 1 quotes 8.1's "A + [Detector] that predates this document ... MAY be graded [Conforming] on evidence," as what it + read the 8.1 sentence on a declaration with no gap against. As run 22 held, a quotation under + question 4 counts whatever it is quoted for; each run of words either side of the ellipsis lies + in 8.1's second paragraph, at the place the reader names, so it is recorded there. +11. **Question 4, closing punctuation.** Reader 1's "... (TOOLING-SPEC.md)." ends with a full stop + where the document has a comma; reader 2's "... MUST name its kind," and reader 1's "... on + evidence," end with a comma where the document has a comma or a colon. The words are consecutive + and verbatim; the punctuation mark that follows them is not a word, so each is a quotation, as + run 26 held. +12. **The open passages.** The passages earlier runs of this document found confusing by two or + more, and not since cleared, are those run 26 left outstanding: 7.1's first paragraph (runs 20, + 22, 24 and 26) and section 8's paragraph on the kind, which at 1db0c67 begins "A verdict on a + [Detector] MUST name its kind" (runs 24 and 26). The text of each changed after run 26, in + 0b2d190 and e783b64, and each is taken as it reads at 1db0c67. 8.1's first and second paragraphs, + 6.1's paragraph beginning "For a [Bundled rule]" and clause 4.1 were cleared by runs 22, 24 and + 26, and do not reopen when one reader of a later cohort names them. + +No re-mark. + +## Passages quoted as confusing + +Grouped by paragraph, as `ACCEPTANCE.md` defines the same passage. A passage blocks when two or +more readers name it, or when it covers a sentence of a paragraph in which one of the open +passages lay (judgement 12), as that paragraph reads at 1db0c67. + +| Clause and paragraph | Readers | Words quoted, verbatim | Blocks | +| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | +| 8, the paragraph beginning "A verdict on a [Detector] MUST name its kind", its first, second and fifth sentences | 2, 3, 4, 5 | 2: "A verdict on a [Detector] MUST name its kind," and "the kind is not a level of [Conformance] and does not change the verdict." 3: "each configuration the verdict grades" and "configuration". 4: "A verdict on a [Detector] MUST name its kind" and "The kind is not a level of [Conformance] and does not change the verdict." 5: "A verdict on a [Detector] MUST name its kind", "This duty is on whoever gives the verdict, not on the [Detector]" and "the kind is not a level of [Conformance] and does not change the verdict." (judgement 11) | Yes: four readers, and the open section 8 paragraph of runs 24 and 26 | +| 7.1, first paragraph, the sentence beginning "A route is detectable when" | 3 | "A route is detectable when the project can find every use of it, either with a [Rule] of the project's own that the [Detector] runs, or with a mechanical check the [Detector] documents." | Yes: the open 7.1 paragraph of runs 20, 22, 24 and 26 | +| 6.1, the paragraph beginning "For a project's own [Rule]", its first sentence, its sentence beginning "Where the [Identifier] is itself a URL" and its last sentence | 1, 2; 4 by description | 1: "The lookup for such a Rule is an obligation on the project's assembled Toolchain, under clause 4.2 of the [toolchain specification](TOOLING-SPEC.md)." 2: "what it owes is the half it can give: the [Identifier] printed unaltered under clause 4.3, and no transformation of it." 4: no quotation, the references to the other specifications (judgements 9 and 11) | Yes: two readers, three on judgement 9 | +| 6.1, the heading | 2 | "The detector MUST provide a mechanism that resolves a printed identifier to its documentation," (judgement 11) | No; on the other reading of judgement 9, reader 4 too | +| 8.1, first paragraph, the sentence "A declaration that records no gap claims [Conformance]." | 1 | "A declaration that records no gap claims [Conformance]." | No; cleared by run 24 (judgement 12) | +| 8.1, second paragraph, its first sentence | 1 | "A [Detector] that predates this document" and "MAY be graded [Conforming] on evidence," (judgements 10 and 11) | No; cleared by run 26 (judgement 12) | +| 3, the table row "Resolution of a project's own [Rule]" | 5 | "The documentation and, with the [Toolchain], the lookup." | No | +| 6.1, the Why paragraph, its first sentence; 6.4, the Why paragraph, its sentence on the toolchain specification's self-audit; 7.1, the Why paragraph, its first and last sentences; 9, the sentences that refer to the method or toolchain specification | 4 | No quotation; "Clauses 6.1, 6.4, 7.1 and section 9 refer to clauses in `TOOLING-SPEC.md` and `SPEC.md`" (judgement 9) | No | + +Section 8's paragraph on the kind is named by four readers of this cohort, and 6.1's paragraph on a +project's own rule by two, three on judgement 9. Of the open passages, neither is cleared: this +cohort names passages covering sentences of 7.1's first paragraph and of section 8's paragraph on +the kind, as each reads at 1db0c67. + +## Findings applied + +Applied to the text, never to a question, threshold, fixture or key, in c6ee1a8 and the commit after +it that settles its review (key quotations kept verbatim in 224579f and that commit). The editor +first read every sheet, in runs 20 to 28, that named each passage, and fixed the cause the sheets +show. DETECTOR-SPEC.md: clause 7.1 says after the two things too that neither requires removing or +forbidding the route, and says what the detector does when the project detects a route with its own +rule; section 8's paragraph on the kind uses "configuration" in one sense and says the kind does not +change the conformance verdict; clause 6.1's paragraph on a project's own rule no longer leans on +cross-references a reader of this document alone cannot follow. A fresh cohort follows. + +## Process change + +Made under the process change of pull request 13 before it merged, at commit 598908b, named +before the readers started; `ACCEPTANCE.md` and `acceptance/runs/README.md` in this worktree +differ from 598908b only by this pull request's question-set line adding question 2d for SPEC.md, +which does not bear on this document. + +## Verdict + +- **Load-bearing statements (every reader on every one)**: not met. 2a four of five (all but + reader 4), 2b two of five (readers 2 and 5), 2c five of five. +- **Restatement and application (all but one on each keyed item)**: met. Question 1 four of five; + every clause of question 3, the section 8 verdict and 6.4 five of five. +- **A miss traceable to the text (none)**: not met. Four, each a finding: 2b by readers 1, 3 and 4, + each quoting 7.1 from "it MUST do at least one of two things", none stating that the detector + need not remove or forbid the route; and 2a by reader 4, quoting section 8's "It is judged on the + evidence, and a declaration under clause 8.1 is optional." and not stating that conformance is + satisfying every MUST in sections 4 to 7 (judgement 7). +- **A passage found confusing by two or more (none outstanding)**: not met. Section 8's paragraph on + the kind is named by readers 2, 3, 4 and 5, and 6.1's paragraph beginning "For a project's own + [Rule]" by readers 1 and 2 (and 4, judgement 9); the open passages in 7.1's first paragraph + (reader 3) and section 8's paragraph on the kind are still outstanding. +- **From one reader only**: 6.1's heading, 8.1's first and second paragraphs, the section 3 table + row on a project's own rule, and the references reader 4 names in 6.1, 6.4, 7.1 and section 9, + recorded. + +FAIL on the first, third and fourth criteria. Answer sheets, the readers' prompt and command, and +the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091000028), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091000433), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091000975). diff --git a/acceptance/runs/029-semantic-detectors-run-11.md b/acceptance/runs/029-semantic-detectors-run-11.md new file mode 100644 index 0000000..3917fc6 --- /dev/null +++ b/acceptance/runs/029-semantic-detectors-run-11.md @@ -0,0 +1,267 @@ +# Run 29: SPEC.md, semantic detectors, eleventh run in pull request 14 + +- **Document**: SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + branch `editor/semantic-detectors` at db47301, the commit the readers were given; the question + set, key and fixture read at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input; no tools, no settings, no project context and no + saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is byte for byte the question set, the document and the + fixture at db47301, in that order, each preceded by one `===== FILE: =====` line, with + nothing else added apart from one blank line between the end of each file and the next file's + marker line. +- **Managed (policy) settings**: none; the machine applies no managed settings to the readers and + no managed settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. Every sheet has answer text, no process reports an error + (`is_error` false, subtype `success`), and every reported model is the cohort's model. The + editor reports no reader set aside for a cause the sheets cannot show. +- **Run within the pull request**: 11, counting void runs; runs in the pull request so far, + counting void runs: 11, as the editor supplied. Run 30 is made at the same time and does not + bear on this document. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review. It read `ACCEPTANCE.md` and + `acceptance/runs/README.md` in this worktree, which differ from commit 598908b only by this pull + request's question-set line adding 2d for SPEC.md; the document, question set, key and fixture + at db47301; run 27 in full, the passages, judgements and verdict of run 25, the passage history + of runs 19, 21 and 23, and the headers of runs 20, 22, 24, 26 and 28, which are on + DETECTOR-SPEC.md; and, for the fourth pass-criteria row, the change to SPEC.md between 1db0c67, + the document run 27 read, and db47301. + +## Marks + +`ok` is correct, `miss` is a miss; `T` marks a miss traceable to the text. Judgements J1 to J8 +are set out below. + +| Reader | Q1 | 2a | 2b | 2c | 2d | 3.1 | 3.2 | 3.3 | 3.4 | 3.5 | 3.6 | Remediation | Defence | Section 4 | Q4 passages | +| ------ | ------- | --- | ------ | --- | ------ | ------- | ------- | ------ | --- | --- | --- | ----------- | ------- | --------- | ----------- | +| 1 | correct | ok | ok | ok | ok | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 2 | correct | ok | ok | ok | miss T | Partial | Partial | Yes ok | No | No | No | ok | ok | ok | three | +| 3 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | four | +| 4 | correct | ok | ok | ok | miss T | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | +| 5 | correct | ok | miss T | ok | ok | No | Partial | Yes ok | No | No | No | ok | ok | ok | two | + +Every grade under 3.1, 3.2, 3.4, 3.5 and 3.6 is one the key accepts. No reader grades 3.2, 3.5 or +3.6 Yes, so none of the key's cases that count against the document arises. + +### Question 1 + +Five of five correct. Every sheet has at least four bullets naming obligations the key lists (the +class bounded both ways with its hazard, the strongest kind of detector and never a test, the +proof in a commit of its own, the sweep with its count and the fix of every instance, permanent +and blocking, the terse message with an identifier resolving to versioned documentation, the +record of a deferred defect), and no bullet asserts an obligation the document does not state. +The bullets of readers 1, 2, 4 and 5 on attempting rather than pre-judging, or on recording a +no-pattern conclusion in one sentence naming two techniques, state section 2's obligations; reader +5's closing "fix the original defect conventionally with a reproduction test" is section 2 and +section 6. + +### Question 2 + +Every reader quoted words of the document for every item, so every miss here is traceable to the +text. + +- **2a**, five of five. Each states class, rule, proof, sweep and the fix of every instance, in + that order, and quotes section 2's sentence "The order is section 3's: attribute the [Defect] to + a [Class], build the [Rule], prove that it fires, run the [Sweep], then fix every [Instance]." +- **2b**, four of five. Readers 1 to 4 say only the owner may and an agent may not, and quote the + key's section 4 sentence; readers 1, 3 and 4 also quote "The [Owner] is always a human." Reader + 5 says "Only a human may adopt a baseline or suppress an instance", quoting the section 4 + sentence and, as an addition, "The [Owner] is always a human."; it does not say the owner is + who may (J4). Miss, traceable. +- **2c**, five of five. Each states all six clauses and a verdict resting on reproduction, and + each quotes section 7's definition of a conforming remediation or its "MUST rest on + reproduction, not on the report"; no reader accepts the pull request description as the + verdict. Reader 5's withdrawn fragment before its quotation is read under J1. +- **2d**, two of five (readers 1 and 5). The key's three elements are (i) the strongest kind, in + the order semantic, syntactic, textual; (ii) both grounds for a weaker kind; (iii) the reason + recorded, with at least one place named and the places not given as alternatives (J2). + - Reader 1: (i) in its quotation of the strongest-kind sentence; (ii) in its sentence, both + grounds; (iii) in its sentence and its quotation of the record sentence, both places joined + by "and". Correct. + - Reader 2: (i) in its sentence and quotation; (iii) in its quotation of the record sentence. + (ii) missed: "A weaker kind is allowed only on one of two grounds" states neither ground, and + neither quotation states one. Miss, traceable. + - Reader 3: (ii) in its quotation of "A weaker kind is allowed only where ... those are the two + grounds."; (iii) in its quotation of the record sentence. (i) missed: "the strongest kind of + detector that can express it" names no kind, and "weaker than [Semantic]" names the top of the + order, not the order (J3). Miss, traceable. + - Reader 4: (ii) in its quotation of the grounds sentence; (iii) in its sentence, "in the rule's + documentation and in the commit". (i) missed as for reader 3: no kind and no order in its + sentence or quotation (J3). Miss, traceable. + - Reader 5: (i), (ii) and (iii) in its three quotations, the strongest-kind sentence, the + grounds sentence and the record sentence. Correct. + +### Question 3 + +- **3.1**: readers 1, 3, 4 and 5 No, reader 2 Partial, each noting that no two independent search + techniques are recorded. +- **3.2**: five of five Partial, each saying the detector or its kind is not named on the + documentation and that the rule is not a test; every reader adds that no grounds for a weaker + kind are recorded, which the key does not mark wrong. +- **3.3**: five of five Yes. Readers 1, 2, 4 and 5 say the inline ignore is not judged here; none + attributes it to 3.3. +- **3.4** and **3.5**: five of five No, each naming the inline ignore on `Report::render()`, under + 3.5 as uncovered by any recorded decision. +- **3.6**: five of five No, each on the wiki documentation not versioned with the rule. +- **Remediation**: five of five say it does not conform, each naming 3.4 among the failing clauses + (J7). +- **Defence**: five of five say it does not conform, each naming 3.6 as failing, reader 3 by its + ground, "its documentation be versioned alongside it" (J8); readers 1, 2 and 5 also name 3.2 as + not met in full. Every reader also counts 3.5, and most 3.1, against the defence; the key does + not mark that wrong. +- **Section 4**: five of five name the inline ignore on `Report::render()`, and each also names + leaving the next wider rule unbuilt, which the key accepts alongside it. + +### Judgements made in marking + +Made as a careful reader of the document would, as `ACCEPTANCE.md` requires. Where a rule settles +the point, the record says which. + +- **J1. The sentence.** Where a reader wrote more than one sentence before or around its + quotations, all of that text is read with the quotations as the answer. Reader 5's 2c opens with + a fragment it withdraws in the same line ("... is not the quote here"); the withdrawn fragment + is not read as part of the answer. No mark turns on it. +- **J2. 2d's element (iii) needs a place named**, as runs 19 to 27 read the key's "naming one of + the two places is enough". Every reader names both places this run, so no mark turns on it. +- **J3. 2d's element (i) needs the order (readers 3 and 4).** The key's element is "the strongest + kind that can express the rule, in the order semantic, syntactic, textual", and the question + asks which kind. "The strongest kind of detector that can express it" names no kind, and + "weaker than [Semantic]" in a record quotation names the top of the order, not the order. Runs + 23 and 25 read it the same way. Both readers give the order under question 1, but each keyed + item is marked on its own answer. Read otherwise, readers 3 and 4 would be correct on 2d; 2d + would still fail on reader 2, whose miss on (ii) rests on no judgement, and the verdict would not + change. +- **J4. "Only a human" (reader 5, 2b).** The key's answer is the owner of the codebase, always a + human, and an answer that omits the owner is wrong. Reader 5's sentence names no one but "a + human", which on its face includes a human practitioner, to whom section 4 gives none of these + decisions; its added quotation "The [Owner] is always a human." says the owner is human, not + that the owner is who may. Read together, the answer does not state that the decision is the + owner's, so it omits the owner. Read otherwise, as naming the owner through the quotation it + gives in support, 2b would be five of five and the verdict would not change, since 2d fails on + reader 2 without a judgement. +- **J5. Passages a reader quotes or sets beside its main one count.** Where a reader's question 4 + item quotes words of the document at a named place, or says it was confused by two passages read + together, a careful reader takes each as a passage it names, since the confusion lies in neither + alone, as runs 23, 25 and 27 read it. So reader 1's section 2 item also covers clause 3.2's + paragraph beginning "There are two kinds of gap", where its quoted "mechanism gap" and "the gap + above" occur in clause 3.2, the first in the same sentence reader 3 quotes; reader 2's section 2 + item, which says "clause 3.1 uses it for searching for instances", covers clause 3.1's sentence + "In this clause a technique is a way of searching the codebase for [Instances].", as run 27 read + that reader's like description; reader 3's and reader 4's items each cover both passages they + quote. Two findings turn on this. Read as context only, reader 1's clause 3.2 passage is one + reader's, not two, and run 25's clause 3.1 passage is cleared; the verdict would not change. +- **J6. Run 25's clause 3.1 paragraph as it reads in the changed document.** The paragraph that + began "Two techniques are independent when they would miss different things." is now three + paragraphs, beginning "In this clause a technique is a way of searching", "A technique counts + only if" and "The record MUST name both techniques", the last kept verbatim from it. All three + are taken as that paragraph as it reads now. No mark turns on it: reader 2's description names a + sentence of the first. +- **J7. The remediation item.** The key's answer is that the remediation does not conform, giving + 3.2, 3.4 and 3.6 as the reason. A reader saying it does not conform and naming among the failing + clauses one the key names is correct, as runs 23, 25 and 27 marked it; every reader names 3.4, + and every reader 3.6. +- **J8. The defence item (reader 3).** The key marks correct a reader who names 3.2 or 3.6. Reader + 3 names 3.5 by number and 3.6 by its requirement, that the documentation "be versioned alongside + it", having graded 3.6 No on that ground; it names 3.6's failure, and is correct. + +## Passages quoted as confusing + +Same passage as `ACCEPTANCE.md` defines it: sentences in common, or the same paragraph; a list +item, table row or terminology entry is its own paragraph. + +| Passage | Clause | Readers | Words quoted, verbatim, or the description used | +| ---------------------------------------------------------- | --------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Independent techniques for expressing a rule | 2 | two: 2, 5 | 2: "A technique here is a way of trying to write the [Rule]. Two such techniques are independent when they would miss different things, which is the same test clause 3.1 applies to ways of searching for [Instances]." 5: "The conclusion that no pattern exists MUST be stated in one sentence, naming at least two independent techniques tried at expressing the [Rule], and recorded where the project's other decisions are enumerable under clause 8.7, not only with the fix." Both in the "Attempt rather than pre-judge" paragraph | +| The entry point and substitutes for a bundled defence | 3.5 | two: 2, 3 | 2: "A mechanism gap that stops the [Toolchain] running its own entry point on its own source moves the demonstration to such a project; it does not weaken it." 3: "The substitutes clause 3.2 allows, for proving under clause 3.3 and for resolution under clause 3.6, do not extend to this clause." Both in the paragraph beginning "A [Bundled defence] is a [Defence] that a shipped [Toolchain] enables by default"; reader 3 sets it beside the clause 3.2 passage below | +| The two kinds of gap | 3.2 | two: 1, 3 | 3: "uses the substitutes clauses 3.3 and 3.6 already allow". 1: "mechanism gap" and "the gap above", attributed to clause 3.2 (J5). Both in the paragraph beginning "There are two kinds of gap and they lead to different work" | +| The three ways out, a toolchain gap | 2 | one: 1 | "A pattern exists but the language has no extensible [Detector] of a kind that can express the [Rule] within clause 3.1's bounds, and a bespoke one is not practical: a [Toolchain] gap under clause 3.2.", in the paragraph beginning "There are three ways out of the method", set beside clause 3.2's paragraph above (J5) | +| A technique in clause 3.1 | 3.1 | one: 2 | By description, "clause 3.1 uses it for searching for instances", set beside its section 2 passage above: the sentence "In this clause a technique is a way of searching the codebase for [Instances]." (J5, J6). Run 25's open passage (below) | +| An assumed calibration against escalating doubt | 3 and 3.1 | one: 3 | "**Where the project has recorded nothing, the [Practitioner] is not blocked.** Proceed on the examples below, **state the [Calibration] assumed**", in section 3's paragraph of that name, set beside "Where the [Practitioner] is not sure, the decision goes to the [Owner] under section 4.", in clause 3.1's lower-bound list item | +| Merging at full width against not blocking until every fix | 4 | one: 4 | "the [Rule] MAY merge at full width with that case recorded as a known [Instance] awaiting the [Owner]", in the paragraph beginning "**What to do whilst waiting.**", set beside "provided the [Rule] does not become [Blocking] until every [Instance] is fixed", in the paragraph beginning "A remediation MAY be delivered across several changes" | +| Leaving the next wider rule unbuilt | 4 | one: 5 | "It is a reason not to widen unasked, not authority to leave the wider [Rule] out, and recording it as a known gap does not make it the [Practitioner]'s decision.", in section 4's list item beginning "Leaving unbuilt the next wider [Rule]" | + +Readers' accounts of what they could not tell are recorded with the passage they attach to and are +not suggestions; none of the sheets proposes a change. + +**Earlier passages (the fourth row).** The open passages are those earlier runs of this document +found confusing by two or more and did not clear: run 27's two new passages and run 25's clause +3.1 passage, which run 27 did not clear; one of run 27's two is run 21's clause 3.2 passage, +which runs 23, 25 and 27 did not clear. Every other passage of runs 19, 21, 23 and 25 was cleared +in runs 21, 23, 25 and 27. As the paragraphs read in the changed document: + +- Clause 3.2's grounds and record, run 21's passage: the paragraph beginning "**Qualifying is not + the same as being the right host.**", the paragraph beginning "For the second ground", and the + record paragraph beginning "On the [Remediation docs] of a [Rule] they write". Named by no + reader. Readers 1 and 3 name clause 3.2's paragraph beginning "There are two kinds of gap", + which is not one of these. Cleared. +- Section 7's paragraph beginning "**A [Defence] [Conforms]**", with clause 3.5's opening + paragraph, run 27's passage: named by no reader. Readers 2 and 3 name clause 3.5's \[Bundled + defence\] paragraph, not its opening paragraph. Cleared. +- Clause 3.1's paragraph that began "Two techniques are independent when they would miss + different things.", run 25's passage, now the three paragraphs beginning "In this clause a + technique is a way of searching", "A technique counts only if" and "The record MUST name both + techniques" (J6): named by reader 2 by description (J5). Not cleared. + +No sentence kept from a cleared paragraph is named elsewhere in the document. Four passages named +here were once open and have since been cleared: section 2's "Attempt rather than pre-judge" +paragraph (run 19, two readers; cleared in run 21), named here by readers 2 and 5; clause 3.5's +[Bundled defence] paragraph (run 25, two readers; cleared in run 27), named here by readers 2 and +3; clause 3.1's lower-bound list item (run 23, three readers; cleared in run 25), named here by +reader 3; and section 4's list item on leaving the next wider rule unbuilt (run 21, three readers; +cleared in run 23), named here by reader 5. The first two are named by two readers and are new +findings here; the last two are recorded as one reader's. Clause 3.2's paragraph on the two kinds +of gap, named by one reader in run 25, is named by two here and is a new finding. + +## Findings applied + +None yet. After six cohorts on this pull request the editor has stopped to put a decision to the +owner before changing the text again: the misses that recur in every run (question 2d's order and +grounds, and on the detector document question 2b's "need not remove or forbid") turn on elements of +the keys that the questions do not ask for directly, and on the fourth row keeping a passage open +while any single reader of a fresh cohort names it. Neither can be changed by the editor, since a +question, a key and a threshold may not be changed to let a run pass. + +## Process change + +Made under a process change before it merged: yes, pull request 13 at commit 598908b, named +before the readers started. + +## Verdict + +**FAIL.** Against the criteria in `ACCEPTANCE.md`: + +- Load-bearing statements: not met. 2a and 2c five of five; 2b four of five (reader 5, on J4); 2d + two of five. Reader 2 states neither ground for a weaker kind, only that there are two; readers + 3 and 4 do not give the order semantic, syntactic, textual (J3). +- Restatement and application: met. Question 1 and every keyed item of question 3 five of five. +- Misses traceable to the text, each a finding: 2d for reader 2, quoting clause 3.2's + strongest-kind and record sentences and saying the grounds are two without stating them; 2d for + readers 3 and 4, quoting clause 3.2's sentence "A weaker kind is allowed only where the stronger + kind cannot express the [Rule], or where the language has no [Detector] of the stronger kind + that accepts bespoke [Rules]; those are the two grounds." and giving the strongest kind without + the order (on J3); 2b for reader 5, quoting section 4's sentence on an agent's authority and + saying only "a human" may, without the owner (on J4). +- Passages found confusing by two or more, each a finding that blocks until the text is changed + and a fresh cohort no longer names it: section 2's "Attempt rather than pre-judge" paragraph, on + techniques tried at expressing a rule (two readers); clause 3.5's paragraph beginning "A + [Bundled defence] is a [Defence] that a shipped [Toolchain] enables by default", on the moved + demonstration and the substitutes that do not extend to it (two); clause 3.2's paragraph + beginning "There are two kinds of gap and they lead to different work" (two, one on J5). Run + 25's clause 3.1 paragraph on independent techniques is not cleared (one reader, on J5). Run 21's + clause 3.2 passage on the grounds and their record, and run 27's section 7 and clause 3.5 + passage, are cleared. +- Single-reader notes, recorded: section 2's "There are three ways out of the method" paragraph + (reader 1); section 3's "Where the project has recorded nothing" paragraph against clause 3.1's + lower-bound list item (reader 3); section 4's "What to do whilst waiting" paragraph against its + paragraph on delivering a remediation across several changes (reader 4); section 4's list item + on leaving the next wider rule unbuilt (reader 5). + +Answer sheets, the reader prompt, the command and the marker's instruction: posted on pull request 14 in posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091186417), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091186696), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091187011), [comment 4](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091187459). diff --git a/acceptance/runs/030-semantic-detectors-run-12.md b/acceptance/runs/030-semantic-detectors-run-12.md new file mode 100644 index 0000000..7005da5 --- /dev/null +++ b/acceptance/runs/030-semantic-detectors-run-12.md @@ -0,0 +1,268 @@ +# Run 30: DETECTOR-SPEC.md, semantic detectors, twelfth run in pull request 14 + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [14](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14), + editor's branch `editor/semantic-detectors` at db47301; the question set, key and fixture read + at the same commit +- **Cohort**: five readers, model alias `haiku`; every reader reported `claude-haiku-5-5`, as the + editor supplied in the cohort's model list and as each reader's own JSON output shows +- **Isolation**, as the editor supplied it: each reader a separate `claude -p` process started + through the `ccy-claude` wrapper, which only reattaches the session's credential and passes + arguments through verbatim, in a fresh empty directory made by `mktemp -d` outside any + repository, with the packet on standard input, with no tools, no settings, no project context + and no saved session. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The marker checked that the prompt is byte for byte the question set, the document and the + fixture at db47301, in that order, each preceded by one `===== FILE: =====` line, with the + files separated by one blank line and nothing else added, the same form as the packets of runs + 22, 24, 26 and 28 +- **Managed (policy) settings**: none; the machine applies no managed settings and no managed + settings file is present, as the editor supplied +- **The rest of the isolation**: attested by the editor +- **Readers set aside**: none. The editor reports none set aside for a cause the sheets cannot + show; every sheet has answer text, no process reports an error (`is_error` false, subtype + `success`), and every reported model is the cohort's model +- **Run within the pull request**: 12, counting void runs; run 29, made at the same time, does not + bear on this document +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) dispatched with + the editor's instruction and its agent definition, `.claude/agents/dbf-spec-editor.md`, having + seen neither the drafting nor the review + +## Marks + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | Section 8 | Kind | 6.4 | Q4 entries | +| ------ | ------- | --- | ---- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --------- | ---------- | --- | ---------- | +| 1 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 2 | miss | ok | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 3 | correct | ok | miss | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 1 | +| 4 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 4 | +| 5 | correct | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | not marked | ok | 2 | + +"Q4 entries" counts the entries each reader lists under question 4; the words each quotes are in +the table of passages below. Every miss under question 2 is traceable to the text: each answer's +sentence states what the document requires, and each quotes the document in support. The miss +under question 1 is the reader's own: the bullet it turns on quotes nothing. No question was left +unanswered. + +### Question 1 + +Every restatement names four or more obligations from the key's list. Readers 1, 3, 4 and 5 assert +no obligation the document does not state: correct. Reader 1's sixth bullet and reader 5's sixth +name the duty to name the kind on a verdict, which the key marks correct; reader 3's sixth and +reader 5's sixth also say what conformance means, which names no obligation and asserts none the +document does not state. Readers 1, 3 and 5 say a printed identifier must resolve to its +documentation, in the generality of 6.1's heading, as run 28 held (judgement 4). Readers 1, 2, 3, 4 +and 5 say that each rule carries, needs or has a stable identifier (judgement 3). Reader 3's "put +the results in the output of the command the user ran" and reader 5's "findings must appear in the +output of the command the practitioner ran" are plain plurals, as run 26 held. Reader 2's fifth +bullet says "Each finding, with its location and identifier, must appear in the output of the +command the practitioner ran (5.3)"; 5.3 says "The summary need not list every finding", so the +bullet asserts an obligation the document does not state: incorrect (judgement 2). Correct, four of +five. + +### Question 2 + +The key's answers state these elements: + +- 2a: (a) Conformance is satisfying every MUST in sections 4 to 7; (b) judged on evidence; (c) a + detector that predates the document can conform; (d) the declaration is optional. +- 2b: (a) make the route disableable by configuration, or (b) detectable by a Rule the project can + write or a mechanical check the Detector documents; (c) the detector is not required to remove or + forbid it. +- 2c: (a) with the rule; (b) in the installed copy; (c) at a version tracked together; (d) yes, + reachable offline. + +Each answer is read as its sentence and its quotation together, for the gist of each element. + +| Reader | 2a | 2b | 2c | +| ------ | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | +| 1 | ok: all four; (b) and (d) from the 8.1 quotation | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6) | ok: all four; (a) and (c) from the 6.3 quotation | +| 2 | ok: all four; (b) and (d) from the 8.1 quotation | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6); its quotation drops a word (judgement 8) | miss: (c) not stated; only 6.2 quoted (judgement 9) | +| 3 | ok: all four; (b) and (d) from the section 8 quotation | miss: (c) not stated (judgement 5); (b) as "make every use of it detectable" | miss: (c) not stated; only 6.2 quoted (judgement 9) | +| 4 | ok: all four; (d) as "without a declaration", (b) as "the verdict rests on checking the detector" | miss: (c) not stated (judgement 5); (b) by the defined term (judgement 6) | ok: all four; (a) and (c) from the 6.3 quotation | +| 5 | ok: all four; (b) and (d) from the 8.1 quotation | miss: (c) stated only as "does not have to remove the route" (judgement 7); (b) by the defined term | ok: all four; (a) and (c) from the 6.3 quotation, (b) from the 6.2 quotation | + +No answer asserts anything the document does not state or that the key says makes an answer +wrong: none says a declaration is required, that partial satisfaction is conformance, that the +detector must forbid the route or require a reason, or that a website will do. Every quotation but +reader 2's under 2b is verbatim apart from letter case, link and emphasis markup and a closing +punctuation mark. + +### Question 3 + +The key's intended verdicts are Yes for every clause but 6.3, which is No; the section 8 verdict +is "does not conform", because 6.3 fails; 6.4 does not change the verdict. + +- **Every clause**: all five readers give the intended verdict on every clause, 4.1 to 7.1. On + 6.3 each cites that one unresolved identifier fails the clause, all five quoting 6.3's "One + [Identifier] that does not resolve fails this clause, however many others do"; no reader grades + it Partial on the grounds that thirty-eight resolve. Readers 1 to 5 add caveats on 4.2 or 4.3 + about what the fixture does not say, and grade Yes. +- **Section 8**: all five say Lintomatic does not conform, because 6.3 fails. +- **Kind**: all five say the fixture does not let them name the kind and name none, so none is + marked on it. Readers 2, 3 and 5 call the verdict short of that duty; none makes the verdict turn + on the kind, and all five say it does not change the verdict or the conformance finding. +- **6.4**: all five say it does not change the verdict, because it is a SHOULD. Reader 3 adds + that 7.2 is an unmet SHOULD and that a declaration would record the 6.3 gap (judgement 10). + +### Judgements made in marking + +01. **How an open judgement is made.** As `ACCEPTANCE.md` at 598908b says: as a careful reader of + the document would, with the reason stated. Where a rule settles a mark, it is applied as + written. +02. **Question 1, reader 2's "Each finding ... must appear".** Runs 22, 24 and 28 marked incorrect + bullets saying every finding, or each finding, must appear in the output, because 5.3 says "The + summary need not list every finding" and "If it leaves any finding out, the output MUST also + say where the rest can be read". Reader 2's bullet is the same claim about each finding. + Incorrect. The bullet quotes nothing, so the miss is the reader's. Reader 2 also gives seven + bullets where the question asks for at most six; the key does not make that wrong, and the mark + does not turn on it. +03. **Question 1, "each rule carries a stable identifier".** Clause 4.3's heading asks the detector + to "allow a rule to carry a stable identifier", and its body says "The [Detector] MUST print it, + unaltered, alongside every finding the [Rule] reports". A detector can print an identifier with + every finding of every rule only if each rule carries one, and the key's own bullet joins the + two. A careful reader takes these bullets as 4.3, not as a further obligation. Correct. Read the + other way, readers 1, 3, 4 and 5 would all miss question 1 and the second criterion would fail + too; the verdict is FAIL either way. +04. **Question 1, "a printed identifier must resolve".** As run 28 held: the bullets speak at the + level of 6.1's heading, "The detector MUST provide a mechanism that resolves a printed + identifier to its documentation", and the key's bullet is in the same words. Correct. +05. **2b (c), not required to remove or forbid, readers 1 to 4.** As runs 22, 24, 26 and 28 held, + the key's answer includes what the detector need not do, so (c) answers a part of the question. + Readers 1, 2, 3 and 4 say the detector must do one of the two things, readers 1, 2 and 4 adding + that one alone is enough. Each quotes 7.1 from "it MUST do at least one of two things" or, in + reader 2's case, from "but MUST do"; none quotes the words before it, "it need not remove or + forbid the route", nor the words after "Either alone satisfies this clause", "neither requires + the [Detector] to remove or forbid the route". That one alone suffices entails that nothing more + is required, but an element reached only by inference from another is not among the ways + `ACCEPTANCE.md` names of stating it. Not stated. +06. **2b (b), the means of detection.** Clause 7.1 defines when a route is detectable in its own + text; readers 1, 2, 4 and 5 use "detectable" in that sense and so state (b), as runs 22 to 28 + held. It decides no mark. +07. **2b (c), reader 5's "It does not have to remove the route."** The element is that the + detector is not required to remove or forbid the route, and 7.1 says so twice in those words. + Removing a route and forbidding it are different things: the first takes the route out of the + detector, the second stops its use whilst it stays; and the answer the key names as wrong is + that the detector "must forbid the route". Reader 5 states the first half and leaves the second + open, and its quotation, from "but it MUST do at least one of two things", does not carry it. + A careful reader does not take half of the element as its gist. Not stated: a miss, traceable, + since the sentence states what the document requires. Read the other way, reader 5 would be + correct on 2b and the third criterion would lose one finding; the verdict does not turn on this + judgement, since 2b fails the first and third criteria by four readers whichever way it is made. +08. **2b, reader 2's quotation.** Reader 2 quotes "but MUST do at least one of two things: ...", + where the document has "but it MUST do". The words from "MUST do" to "detectable" are a run of + the document's consecutive words, verbatim; the quotation as given is not, since it drops a + word. The mark does not turn on it: the answer misses (c) either way, and the miss is traceable + because its sentence states, in the reader's own words, what 7.1 requires. +09. **2c (c), at a version tracked together, readers 2 and 3.** As run 22 held for a reader whose + only quotation was 6.2, element (c) answers where the documentation must live, as part of the + key's answer to that part. Reader 2's sentence says the documentation "must ship with the + installed rule and be reachable offline", reader 3's that it "must ship with the installed rule, + and it must be reachable without network access", and each quotes only 6.2. Each states (a), + (b) and (d); neither says the documentation is versioned with the rule. Not stated: a miss, and + traceable, since each sentence states what the document requires and each quotes 6.2. +10. **Asides outside the keyed answer.** Reader 3's notes on 7.2 and on what a declaration would + record, and the remarks of readers 1, 2, 4 and 5 on Lintomatic missing 6.4, are not keyed + items; each sheet's answer to whether 6.4 changes the verdict is the key's, and is marked so. +11. **Question 4, reader 4's entry on documents not given.** Reader 4 writes "Several terms and + clauses are defined in documents I was not given, for example" and quotes "Terms from the method + specification carry over unchanged" and "the toolchain specification's self-audit" in 6.4. The + entry quotes, and its description names no place in the document: it says what the reader + lacked, and its quotations are the places it names. A careful reader takes the passages as the + two quoted, in section 2's first paragraph and in 6.4's Why paragraph. Read the other way, as + every sentence that uses a term or cites a clause of the other specifications, it would cover + sentences of nearly every paragraph, among them 6.1's paragraph beginning "For a project's own + [Rule]", which would then not be cleared; the fourth criterion fails either way, and the + verdict does not turn on it. +12. **Question 4, closing punctuation.** Reader 4's "... that [Conformance] is judged on." ends with + a full stop where the document has a colon; reader 5's "... in sections 4 to 7." ends with a + full stop where the sentence goes on. The words are consecutive and verbatim; the punctuation + mark that follows them is not a word, so each is a quotation, as runs 26 and 28 held. +13. **Question 4, a passage quoted only to set another against it.** Reader 5 quotes "a declaration + is optional" as what it read 8.1's sentence on a declaration with no gap beside. As runs 22 and + 28 held, a quotation under question 4 counts whatever it is quoted for. The run occurs only in + 8.1's second paragraph, at the place the reader names, and is recorded there. +14. **The open passages, and paragraphs the change split.** The passages earlier runs of this + document found confusing by two or more, and not since cleared, are those run 28 left + outstanding or added: 7.1's first paragraph (runs 20 to 28), section 8's paragraph beginning + "A verdict on a [Detector] MUST name its kind" (runs 24 to 28), and 6.1's paragraph beginning + "For a project's own [Rule]" (run 28, two readers). At 1db0c67, which run 28 read, 7.1's first + paragraph and section 8's paragraph on the kind were each one paragraph; between 1db0c67 and + db47301 the change split each in two, 7.1 at "A route is detectable when", section 8 at "This + MUST is on whoever gives the verdict". `ACCEPTANCE.md` asks whether a reader names a sentence of + the paragraph an earlier passage lay in "as that paragraph reads in the changed document"; a + careful reader takes a split paragraph to read, in the changed document, as the two paragraphs + the change made of it, not as the first of them alone. So taken, a sentence of 7.1's second + paragraph, or of section 8's paragraph beginning "This MUST", covers the open passage. Read the + other way, the open 7.1 passage would be cleared, but 7.1's second paragraph and section 8's + paragraph beginning "This MUST" are each named by two or more readers of this cohort, and + reader 5 names the kept first sentence of section 8's paragraph on the kind, so the fourth + criterion fails either way. 8.1's first and second paragraphs, 6.1's paragraph beginning "For a + [Bundled rule]" and clause 4.1 were cleared by runs 22, 24 and 26, and do not reopen when one + reader of a later cohort names them; 8.1's first paragraph is named by two readers of this + cohort, which is a new finding of this run, not a reopening. + +No re-mark. + +## Passages quoted as confusing + +Grouped by paragraph, as `ACCEPTANCE.md` defines the same passage. A passage blocks when two or +more readers name it, or when it covers a sentence of a paragraph in which one of the open +passages lay (judgement 14), as that paragraph reads at db47301. + +| Clause and paragraph | Readers | Words quoted, verbatim | Blocks | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | +| 7.1, second paragraph, its second and third sentences, "The first is a [Rule] ..." and "That way works only where ..." | 1, 2, 4 | 1: "That way works only where the [Detector] lets such a [Rule] see every use of the route" 2: "The first is a [Rule] the project writes for itself, under clause 4.1, and the [Detector] runs. That way works only where the [Detector] lets such a [Rule] see every use of the route, and the [Detector] need not ship the [Rule] itself." 4: "That way works only where the [Detector] lets such a [Rule] see every use of the route, and the [Detector] need not ship the [Rule] itself." | Yes: three readers, and the open 7.1 paragraph of runs 20 to 28 (judgement 14) | +| 8, the paragraph beginning "This MUST is on whoever gives the verdict", its first sentence; and the paragraph beginning "A verdict on a [Detector] MUST name its kind", its first sentence | 4, 5 | 4: "This MUST is on whoever gives the verdict, not on the [Detector], so it is not one of the MUSTs in sections 4 to 7 that [Conformance] is judged on." 5: "A verdict on a [Detector] MUST name its kind, [Semantic], [Syntactic] or [Textual], in each configuration the verdict grades." and "This MUST is on whoever gives the verdict, not on the [Detector], so it is not one of the MUSTs in sections 4 to 7." (judgement 12) | Yes: two readers, and the open section 8 paragraph of runs 24 to 28 (judgement 14) | +| 8.1, first paragraph, its third sentence, "A gap is a MUST ...", and its last but one, "A declaration that records no gap claims [Conformance]." | 4, 5 | 4: "A gap is a MUST in sections 4 to 7 that the [Detector] fails." 5: "A declaration that records no gap claims [Conformance]." | Yes: two readers, in the same paragraph | +| 8.1, second paragraph, its first sentence | 5 | "a declaration is optional" (judgement 13) | No; cleared by run 26 (judgement 14) | +| 4.3, first paragraph, the sentence beginning "The [Detector] MUST print it, unaltered" | 3 | "The [Detector] MUST print it, unaltered, alongside every finding the [Rule] reports, in its default output and in every machine-readable format it offers." | No; one reader of run 26 also named it | +| 2, first paragraph, its first sentence | 4 | "Terms from the method specification carry over unchanged" (judgement 11) | No | +| 6.4, the Why paragraph, its sentence beginning "Where the [Detector] is shipped inside a [Toolchain]" | 4 | "the toolchain specification's self-audit" (judgement 11) | No | + +7.1's second paragraph is named by three readers of this cohort, section 8's paragraph beginning +"This MUST is on whoever gives the verdict" by two, and 8.1's first paragraph by two. Of the open +passages, 6.1's paragraph beginning "For a project's own [Rule]" is cleared: its text changed after +run 28 and no reader of this cohort names a passage covering a sentence of it, on the reading of +judgement 11. 7.1's first paragraph and section 8's paragraph on the kind, as each reads at +db47301, are still outstanding (judgement 14). + +## Findings applied + +None yet. After six cohorts on this pull request the editor has stopped to put a decision to the +owner before changing the text again: the misses that recur in every run (question 2d's order and +grounds, and on the detector document question 2b's "need not remove or forbid") turn on elements of +the keys that the questions do not ask for directly, and on the fourth row keeping a passage open +while any single reader of a fresh cohort names it. Neither can be changed by the editor, since a +question, a key and a threshold may not be changed to let a run pass. + +## Process change + +Made under the process change of pull request 13 before it merged, at commit 598908b, named +before the readers started; `ACCEPTANCE.md` and `acceptance/runs/README.md` in this worktree +differ from 598908b only by this pull request's question-set line adding question 2d for SPEC.md, +which does not bear on this document. + +## Verdict + +- **Load-bearing statements (every reader on every one)**: not met. 2a five of five, 2b none of + five, 2c three of five (readers 1, 4 and 5). +- **Restatement and application (all but one on each keyed item)**: met. Question 1 four of five; + every clause of question 3, the section 8 verdict and 6.4 five of five. +- **A miss traceable to the text (none)**: not met. Seven, each a finding: 2b by readers 1, 2, 3 + and 4, each quoting 7.1 from "it MUST do at least one of two things" (reader 2 from "but MUST + do"), none stating that the detector need not remove or forbid the route; 2b by reader 5, saying + only that the detector "does not have to remove the route" (judgement 7); and 2c by readers 2 and + 3, each quoting only 6.2 and not stating that the documentation ships at a version tracked + together with the rule (judgement 9). +- **A passage found confusing by two or more (none outstanding)**: not met. 7.1's second paragraph + is named by readers 1, 2 and 4; section 8's paragraph beginning "This MUST is on whoever gives the + verdict" by readers 4 and 5; 8.1's first paragraph by readers 4 and 5. The open passages in 7.1's + first paragraph and section 8's paragraph on the kind are still outstanding (judgement 14); the + open passage in 6.1's paragraph on a project's own rule is cleared. +- **From one reader only**: 8.1's second paragraph, the 4.3 sentence on machine-readable formats, + section 2's first sentence and 6.4's Why sentence on the toolchain specification's self-audit, + recorded. + +FAIL on the first, third and fourth criteria. Answer sheets, the readers' prompt and command, and +the marker's instruction: posted on pull request 14 in [comment 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091188050), [comment 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091188377), [comment 3](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6091188758). diff --git a/acceptance/runs/060-detector-route-rule-run-1.md b/acceptance/runs/060-detector-route-rule-run-1.md new file mode 100644 index 0000000..1b11d48 --- /dev/null +++ b/acceptance/runs/060-detector-route-rule-run-1.md @@ -0,0 +1,141 @@ +# Run 60: DETECTOR-SPEC.md, detector route rule, first run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [23](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23), opened after the run from branch + `editor/detector-route-rule`. The readers read commit ed30e69. Change: clause 7.1's detectable + route is a bespoke rule the project writes for the detector under clause 4.1, with a paragraph + on what detectable means; clause 8.1's gap duty falls only on a detector that declares a + version; clause 5.3 says what the remainder is; clause 6.1 says it asks for the mechanism, a + missing page failing clause 6.3, and that a path in the message is not enough on its own. +- **Run within the pull request**: 1. +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `d3b016b18288e3589df60de97f746a31179924dbdd5faa339c055fbf95d89125`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at ed30e69, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | --- | --- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | miss | ok | ok | + +Question 1: correct bullets per reader 6, 6, 6, 6 and 6, one for each line of the key's list +(4.1; 4.2; 4.3; 5.1 to 5.4; 6.1 to 6.3; 7.1). Reader 1's third bullet adds 4.4 as a SHOULD and +its sixth adds what conformance means beside 7.1; both are, by the key, neither correct nor +wrong, and each bullet still names its obligation. Reader 2's fourth bullet says every finding +must reach the output, which the key says names 5.3 and is not wrong. No bullet asserts anything +the key marks wrong. + +Question 2: every reader states every required element of each item and quotes the document. +For 2a all five quote section 8's "A Detector Conforms if it satisfies every MUST in sections 4 +to 7" or clause 8.1's "A Detector that predates this document, or whose maintainer has never +read it, MAY be graded Conforming on evidence"; for 2b all five quote clause 7.1's "For each one +it MUST do at least one of two things, and either alone satisfies this clause", and name both +things and that either is enough; for 2c all five quote the headings or first sentences of 6.2 +and 6.3. Readers 1 and 2, under 2b, add that the detectable route is closed to a detector whose +rules never see comments, so that it needs the disable route or a documented mechanical check; +that is not among the key's wrong assertions for 2b, and the answers are correct by the rule. + +Question 3: readers 1 to 4 give every accepted grade; reader 5 gives every accepted grade on the +ten rows it answers. S8: all five say Lintomatic does not conform, naming 6.3. 6.4: all five say +it does not change the verdict, because it is a SHOULD. + +### Misses + +- **Reader 5, question 3, clause 7.1**: unanswered. Its table ends at 6.3 and no grade for 7.1 + appears anywhere on the sheet, so the item is a miss under "An unanswered question". There is + no reasoning, so nothing quotes the document; it is not a reading the key names (7.1 graded + No on 7.2's reason). The other four readers are correct on the item, so the threshold of all + but one is met. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..ed30e69 adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | ------------------- | +| 1 | Clause 6.1, "For a Bundled rule" paragraph: "A documentation path printed in the [Message] is therefore not enough on its own"; both readers say [Message] is not defined in the document they were given | 1, 5 | yes | +| 2 | Clause 8.1, second paragraph: "MAY be graded [Conforming] on evidence by anyone who exercises the clauses above against it" (5, whether the MAY is a permission or a possibility); "by anyone who exercises the clauses above against it" (3, whether "the clauses above" are sections 4 to 7 or earlier clauses of section 8) | 3, 5 | yes | +| 3 | Clause 6.1, "For a project's own Rule" paragraph: "Where the [Identifier] is itself a URL, as method clause 3.6 allows"; the reader could not check method clause 3.6 | 1 | no | +| 4 | Clause 6.1 heading: "The Detector MUST provide a mechanism that resolves a printed identifier to its documentation", set against passage 5 | 2 | no | +| 5 | Clause 6.1 table, project's own row: "Print the Identifier unaltered; the lookup is the Toolchain's to give", set against passage 4 | 2 | no | +| 6 | Clause 5.3, first paragraph: "a report that only adds detail to those same findings, such as a code snippet, need not be named in the output" | 2 | yes | +| 7 | Clause 8.1, first paragraph: "a [Detector] that declares a version and has learnt ... that it fails a MUST in sections 4 to 7 MUST record that gap alongside that version", set against section 8's definition and the optional declaration | 3 | yes | +| 8 | Clause 4.3, first paragraph: "in its default output and in every machine-readable format it offers"; "machine-readable" is not defined | 4 | no | + +## Findings and where they lie + +- **Finding A, passage 1, clause 6.1 "For a Bundled rule" paragraph** (readers 1 and 5): the + sentence both readers quote was written by the change, replacing "A [Message] that carries its + own documentation path resolves that [Message], not the [Identifier]." Text the change wrote. + Blocks. +- **Finding B, passage 2, clause 8.1 second paragraph** (readers 3 and 5): the sentence both + readers quote is not itself reworded, but the change rewrote the paragraph's first sentence + ("The declaration is optional, and so the duty above falls only on a [Detector] that has one; + ...") and added its second. The rule places a finding by paragraph, so the paragraph is the + change's. Blocks. + +Passages 3 to 8 are each named by one reader: recorded, not findings. Nothing is routed to +`acceptance/ROUTED.md` from this run. + +## Points the rules did not settle + +1. **Passage 1, a term defined in a document not supplied.** Readers 1 and 5 each quote the + sentence with its clause number, which `ACCEPTANCE.md` defines as naming a passage, and the + confusion each gives is that [Message] is defined in the method specification, which they + were not given. The rule that a remark about another document not being supplied is a note, + not a passage, covers a remark "not about a passage of the document"; these remarks are about + a quoted passage, so they were read as naming it. Had it been open, the open-point rule on + whether a reader named a passage counts the passage in any case. Finding A turns on this + point. Reader 1's passage 3 (method clause 3.6) is the same case, counted the same way, and + is named by one reader only. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit e6bdf40, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item but + 7.1, where four of five are correct (reader 5 left it unanswered). Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: two (findings A + and B). Not met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on findings A and B alone; every keyed item passed its threshold. + +## Answer sheets + +Posted on the pull request in three comments: +[the command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124578731), +[the answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124580496) and +[the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124582261). diff --git a/acceptance/runs/063-detector-route-rule-run-2.md b/acceptance/runs/063-detector-route-rule-run-2.md new file mode 100644 index 0000000..61604ee --- /dev/null +++ b/acceptance/runs/063-detector-route-rule-run-2.md @@ -0,0 +1,147 @@ +# Run 63: DETECTOR-SPEC.md, detector route rule, second run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [23](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23), opened after the run from branch + `editor/detector-route-rule`. The readers read commit 4cbecad. Change: clause 7.1's detectable + route is a bespoke rule the project writes for the detector under clause 4.1, with a paragraph + on what detectable means; clause 8.1's gap duty falls only on a detector that declares a + version; clause 5.3 says what the remainder is; clause 6.1 says it asks for the mechanism, a + missing page failing clause 6.3, and that a path in the message is not enough on its own. + Revised after run 60 in section 2's table (a Message row) and clause 8.1's second paragraph. +- **Run within the pull request**: 2 (run 60 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `30b41f341b26e48b3c8bf1b1d55a25283ffd43c835328eaf5cd9ec021d265e09`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at 4cbecad, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Question 1: correct bullets per reader 6, 6, 6, 6 and 6, one for each line of the key's list +(4.1; 4.2; 4.3; 5.1 to 5.4; 6.1 to 6.3; 7.1). Reader 3's third bullet adds 4.4 as a SHOULD; +readers 2 and 3 add to their sixth bullet that a written reason is a SHOULD; reader 5's sixth +bullet adds the SHOULDs of 7.2, 4.4 and 6.4. Each restates a SHOULD as advice, which the key +says is neither correct nor wrong, and each bullet still names its obligation. No bullet asserts +anything the key marks wrong. + +Question 2: every reader states every required element of each item and quotes the document. +For 2a all five state that conformance is every MUST in sections 4 to 7 and that a detector +predating the document can conform, quoting section 8's definition or clause 8.1's "A Detector +that predates this document, or whose maintainer has never read it, MAY be graded Conforming on +evidence" (reader 1's quotation of section 8 drops "[Detector]", and quotes the document by the +nine-word run "if it satisfies every MUST in sections 4 to 7"; its 8.1 quotation is verbatim). +For 2b all five name both things and that either is enough, quoting clause 7.1's "For each one +it MUST do at least one of two things, and either alone satisfies this clause". Readers 2, 3 +and 5 add that the bespoke-rule way of detection is closed to a detector whose rules never see +comments; reader 2 goes on to say that for an ignore comment the configuration switch or a +documented mechanical check is what remains. Neither is among the key's wrong assertions for +2b (removing or forbidding the route, or requiring a reason), and the answers are correct by the +rule. For 2c all five state the installed copy, the version tracked together and no network +access, quoting the headings of 6.2 and 6.3. + +Question 3: all five give every accepted grade. Reader 2 grades 4.3 "Yes, with a caveat" about +machine-readable formats; the key accepts Yes with no condition on the reasoning and says a +caveat with the accepted grade is correct. S8: all five say Lintomatic does not conform, naming +6.3. 6.4: all five say it does not change the verdict, because it is a SHOULD. + +### Misses + +None. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..4cbecad adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------- | +| 1 | Clause 5.3, first paragraph: "a report that only adds detail to those same findings, such as a code snippet, need not be named in the output" (1, took two readings to see whether the exemption covers the HTML report; 2, the subject of "need not be named" took two readings to settle) | 1, 2 | yes | +| 2 | Clause 6.1, "For a project's own Rule" paragraph: "Where the [Identifier] is itself a URL, as method clause 3.6 allows, the [Detector] prints that URL and has done what this clause asks" (1 and 4, could not check method clause 3.6); "The lookup for such a [Rule] is an obligation on the project's assembled [Toolchain]" (5, set against passages 7 and 8) | 1, 4, 5 | no | +| 3 | Clause 4.3, first paragraph: "in its default output and in every machine-readable format it offers"; "machine-readable" not said, so whether the HTML report is covered | 2 | no | +| 4 | Clause 6.4, Why paragraph: "the [toolchain specification]'s self-audit makes this check a MUST for the [Toolchain]"; the reader was not given that document | 2 | yes | +| 5 | Clause 8.1, second paragraph: "A Detector that predates this document, or whose maintainer has never read it, MAY be graded Conforming on evidence"; whether the MAY is a permission to the grader or an eligibility of the detector, settled by the next sentence | 3 | yes | +| 6 | Clause 7.1, second paragraph: "That way is open only where the [Detector]'s [Rules] can see the route: a [Detector] whose [Rules] never see comments cannot make an ignore comment detectable this way"; what it means for a rule to "see" a comment | 4 | yes | +| 7 | Section 1, the governing principle paragraph: "a [Conforming] [Detector] MUST make that possible without the project building the mechanism first", set against passages 2 and 8 | 5 | no | +| 8 | Clause 6.1 table, project's own row: "the lookup is the [Toolchain]'s to give", set against passage 7 | 5 | no | + +Notes, not passages: reader 5 says the fixture's `explain` paragraph and its last paragraph +disagree about which rules print `No documentation declared`; that is about the fixture, not +the document. + +## Findings and where they lie + +- **Finding A, passage 1, clause 5.3 first paragraph** (readers 1 and 2): the sentence both + readers quote was added by the change, with "The remainder is the findings the summary leaves + out." Text the change wrote. Blocks. +- **Finding B, passage 2, clause 6.1 "For a project's own Rule" paragraph** (readers 1, 4 and + 5): the diff does not touch this paragraph, and no reader naming it sets it against a + paragraph the diff changed (reader 5 sets it against section 1's governing principle and the + 6.1 table's project row, both unchanged). Not the change's text. Routed to + `acceptance/ROUTED.md`; does not block. + +Passages 3 to 8 are each named by one reader: recorded, not findings. Passages 4, 5 and 6 lie +in changed paragraphs but are single. + +## Points the rules did not settle + +1. **Finding B, readers 1 and 4: a passage named for a reference to a document not supplied.** + Both quote the URL sentence with its clause number, which `ACCEPTANCE.md` defines as naming a + passage, and the confusion each gives is that method clause 3.6 is in a document they were + not given. The rule that a remark that another document was not supplied is a note covers a + remark "not about a passage of the document"; these remarks are attached to a quoted + passage. Marked as the open-point rule says: an open point on whether a reader named a + passage counts the passage. Without readers 1 and 4, reader 5 alone names the paragraph and + there is no finding B. Finding A does not turn on this point. Reader 2's passage 4 is the + same case, single either way. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 4081aa4, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: one (finding A). + Not met. +- Passages named by two or more readers elsewhere: one (finding B), routed to + `acceptance/ROUTED.md`. + +## Findings applied + +None at the time of marking; for the editor, under "After a failure" in `ACCEPTANCE.md`. + +## Verdict + +FAIL, on finding A alone; every keyed item passed its threshold. + +## Answer sheets + +Posted on the pull request in three comments: +[the command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124582736), +[the answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124583990) and +[the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124585481). diff --git a/acceptance/runs/066-detector-route-rule-run-3.md b/acceptance/runs/066-detector-route-rule-run-3.md new file mode 100644 index 0000000..72cc618 --- /dev/null +++ b/acceptance/runs/066-detector-route-rule-run-3.md @@ -0,0 +1,148 @@ +# Run 66: DETECTOR-SPEC.md, detector route rule, third run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [23](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23), opened after the run from branch + `editor/detector-route-rule`. The readers read commit 44fdd9e. Change: clause 7.1's detectable + route is a bespoke rule the project writes for the detector under clause 4.1, with a paragraph + on what detectable means; clause 8.1's gap duty falls only on a detector that declares a + version; clause 5.3 says what the remainder is; clause 6.1 says it asks for the mechanism, a + missing page failing clause 6.3, and that a path in the message is not enough on its own; + section 2's table gains a Message row. Revised after run 63 in clause 5.3's sentence on a + fuller report elsewhere. +- **Run within the pull request**: 3 (runs 60 and 63 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `5f991fd47d3b8d092464ca41bfdfd9f5707647f46e3b274929a0c5a44f5a50c4`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at 44fdd9e, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Question 1: correct bullets per reader 6, 6, 6, 6 and 6, one for each line of the key's list +(4.1; 4.2; 4.3; 5.1 to 5.4; 6.1 to 6.3; 7.1). Readers 2 and 3 add to their sixth bullet that a +detector conforms when it meets every MUST in sections 4 to 7, which the key says states what +conformance means and is neither correct nor wrong; the bullet still names 7.1. Reader 4's +fourth bullet says every finding goes in the command's output, which the key says names 5.3 and +is not marked wrong for the plural. Reader 5's fifth bullet adds that a project rule's +identifier is printed unaltered, which is 4.3 restated inside the 6.1 to 6.3 bullet. No bullet +asserts anything the key marks wrong. + +Question 2: every reader states every required element of each item and quotes the document. +For 2a all five state that conformance is every MUST in sections 4 to 7 and that a detector +predating the document can conform, quoting section 8's definition or clause 8.1's "A Detector +that predates this document, or whose maintainer has never read it, MAY be graded Conforming on +evidence". For 2b all five name both things, disableable by configuration or detectable, and +that either is enough, quoting clause 7.1's "For each one it MUST do at least one of two +things, and either alone satisfies this clause". Readers 1, 3, 4 and 5 add that the bespoke-rule +way of detection is closed to a detector whose rules never see comments, which is the +document's own sentence and is not among the key's wrong assertions for 2b (removing or +forbidding the route, or requiring a reason). For 2c readers 1 to 4 state the installed copy, a +version tracked together and no network access, quoting the 6.2 and 6.3 headings; reader 5 +states all three in its own words (citing the headings without quotation marks) and quotes +clause 6.3's "Every [Identifier] a [Bundled rule] can print MUST resolve, under clause 6.1, to a +page in that shipped documentation", which quotes the document. + +Question 3: all five give every accepted grade. Readers 1 to 5 add caveats to 4.2 (the fixture +does not say `--only` prints finding lines) and 4.3 (no machine-readable format described); +reader 5 adds to 4.2 "If it does not, this is No", and reader 3 adds to its 6.3 No that the +grade would need revisiting if its reading of the fixture were wrong. Each gives the accepted +grade as its grade for the fixture as written, and the key says a reader who adds a caveat about +what the fixture does not say and still gives the accepted grade is correct. S8: all five say +Lintomatic does not conform, naming 6.3. 6.4: all five say it does not change the verdict, +because it is a SHOULD. + +### Misses + +None. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..44fdd9e adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------- | +| 1 | Clause 6.1, "For a project's own Rule" paragraph: "Where the [Identifier] is itself a URL, as method clause 3.6 allows, the [Detector] prints that URL and has done what this clause asks" and "The lookup for such a [Rule] is an obligation on the project's assembled [Toolchain], under clause 4.2 of the [toolchain specification]" (1, could not check the documents cited); "What this clause forbids is a documentation path printed alongside a shorter [Identifier] that cannot be looked up on its own" (3 and 4, what "shorter" is measured against) | 1, 3, 4 | no | +| 2 | Clause 4.3, first paragraph: "A prefix the [Detector] adds from the invoking directory or the [Rule]'s location is derived from the file path. A namespace the [Rule author] assigns once in configuration is not." (no reason given) | 2 | no | +| 3 | Clause 6.3, the "dangling one" paragraph: "The failure to guard against is not the absent document but the **dangling one**", set against the clause's first paragraph, "Every Identifier a Bundled rule can print MUST resolve, under clause 6.1, to a page in that shipped documentation"; whether the MUST covers absent documentation | 3 | no | +| 4 | Section 2, "These are additional."; what the terms are additional to | 3 | no | +| 5 | Clause 7.1, second paragraph: "That way is open only where the [Detector]'s [Rules] can see the route: a [Detector] whose [Rules] never see comments cannot make an ignore comment detectable this way"; what a bespoke rule can see | 4 | yes | +| 6 | Clause 7.2, first paragraph: "The [Detector] SHOULD NOT supply a default one, including each entry of a generated [Baseline]"; what "one" refers back to | 5 | no | +| 7 | Clause 8.1, second paragraph: "a gap a [Practitioner] finds is recorded under the method's clause 3.2 in any case", set against the same paragraph's "falls only on a [Detector] that has one"; where the record is kept with no declaration | 5 | yes | + +Notes, not passages: none. + +## Findings and where they lie + +- **Finding A, passage 1, clause 6.1 "For a project's own Rule" paragraph** (readers 1, 3 and + 4): the diff 9e914fe..44fdd9e changes the paragraph before it ("In both, ...") and the "For a + Bundled rule" paragraph, but not this one; no reader naming it sets it against a paragraph the + diff changed (reader 1 points at documents not supplied, readers 3 and 4 at the comparison in + "shorter"). Not the change's text. Routed to `acceptance/ROUTED.md`, where the paragraph's row + raised by run 63 is noted as raised again by run 66; does not block. + +Passages 2 to 7 are each named by one reader: recorded, not findings. Passages 5 and 7 lie in +changed paragraphs but are single. + +## Points the rules did not settle + +1. **Finding A, reader 1: a passage named for a reference to a document not supplied.** Reader 1 + quotes two sentences of the paragraph with the clause number, which `ACCEPTANCE.md` defines + as naming a passage, and its confusion is that method clause 3.6 and the toolchain + specification were not given. The rule that a remark that another document was not supplied + is a note covers a remark "not about a passage of the document"; this remark is attached to + quoted passages. Marked as the open-point rule says: an open point on whether a reader named + a passage counts the passage. The finding does not turn on this point, since readers 3 and 4 + name the same paragraph, and the finding is routed either way. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 9907492, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: none. Met. +- Passages named by two or more readers elsewhere: one (finding A), routed to + `acceptance/ROUTED.md`. + +## Findings applied + +None at the time of marking; finding A is for `acceptance/ROUTED.md`. + +## Verdict + +PASS. Every keyed item passed its threshold, no miss counts against the document, and the one +finding lies outside the text the change wrote or altered. + +## Answer sheets + +Posted on the pull request in three comments: +[the command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124587014), +[the answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124587505) and +[the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23#issuecomment-6124587875). diff --git a/acceptance/runs/070-semantic-detectors-run-13.md b/acceptance/runs/070-semantic-detectors-run-13.md new file mode 100644 index 0000000..154a021 --- /dev/null +++ b/acceptance/runs/070-semantic-detectors-run-13.md @@ -0,0 +1,234 @@ +# Run 70: SPEC.md, semantic detectors, thirteenth run + +- **Document**: SPEC.md + +- **Pull request**: 14, open on branch `editor/semantic-detectors`. The readers were given commit + befe52c. The change, read against its merge base with `next`, 9e914fe, adds to clause 3.2 the + obligation that a [Rule] the [Practitioner] writes is written in the strongest kind of + [Detector] that can express it, with the two grounds for a weaker kind, the naming of the + [Detector] and its kind on the [Remediation docs] and the recording of the grounds on the + [Remediation docs] and in a commit description, the owner's sign-off where the project requires + it, and a worked example; adds the Textual, Syntactic and Semantic terminology entries and a + sentence to the Detector entry; rewrites the section 3 map's 3.2 row; adds "of a kind that can + express the [Rule] within clause 3.1's bounds" to section 2's three-ways-out paragraph and to + clause 3.2's bespoke-detector and two-kinds-of-gap paragraphs; adds clause 3.6's deferral to + what clause 3.2 adds; adds the sign-off proviso to section 4's first list and dividing-line + paragraph; rewrites section 7's conforming-[Defence] sentence; extends Appendix A's "Write a + custom [Rule]" paragraph; adds three link definitions; and sets the version to 1.2.0-dev. The + question set gains item 2d for the new obligation; the key gains 2d and the 3.2 lines that go + with it. + +- **Cohort**: five readers, none reused from any earlier run; model `claude-haiku-5-5`, the + cohort's model named for the run before its readers started. Every reader reported + `claude-haiku-5-5`. + +- **Isolation**, as the editor supplied it and attests: each reader a separate `claude -p` + process, started through the `ccy-claude` wrapper (which only reattaches the session's + credential), in a fresh empty directory made by `mktemp -d` outside any repository, with the + packet on standard input; no tools, no settings, no project context, no saved session, and no + managed settings on the machine. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt's sha256 is `659a8d793ec9f5e94261703c695c94c17f3045a72f6988899c5734ac7ce59f9c`, + checked by the marker against the file; the editor states it is byte for byte + `acceptance/method/QUESTIONS.md`, `SPEC.md` and `acceptance/fixtures/remediation-ledger.md` at + befe52c, in that order, each after one `===== FILE: =====` line, one blank line between + files. + +- **Readers set aside**: none. Every answer text is non-empty, every reader's JSON has `is_error` + false and subtype `success`, and every reader reported the cohort's model. + +- **Run within the pull request**: 13. Runs 19 to 30 came before, on both documents, under the + earlier acceptance rules and on an earlier, larger text of the change; runs 19, 21, 23, 25, 27 + and 29 read this document. + +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + editor's marking instruction and its agent definition, having seen neither the drafting nor + the review. It marked against `ACCEPTANCE.md` and `acceptance/method/KEY.md` at befe52c, by the + Marking section and the key alone. No judgement was taken from the records of runs 19 to 30; + run 64 was read for the shape of a record only. + +## Marks + +| Reader | Q1 | 2a | 2b | 2c | 2d | 3.1 | 3.2 | 3.3 | 3.4 | 3.5 | 3.6 | Remediation | Defence | Section 4 | +| ------ | ------- | --- | --- | --- | --- | ------- | ------- | ------- | --- | --- | --- | ----------- | ------- | --------- | +| 1 | correct | ok | ok | ok | ok | Partial | Partial | Yes | No | No | No | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | No | Partial | Yes | No | No | No | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | No | Partial | Yes | No | No | No | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | Partial | Partial | Partial | No | No | No | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | Partial | Partial | miss | No | No | No | ok | ok | ok | + +One miss in the run: reader 5 on 3.3, graded Partial without the key's condition. Every other +cell gives the reader's grade, accepted, or "ok" for a correct answer. No answer matches a wrong +reading the key names. + +### Question 1 + +Each reader has at least four bullets naming obligations the key lists and none the key marks +wrong. + +- Reader 1: 3.1 (two bullets, counted once), 3.2's detector and its strongest kind with 3.3, + 3.4 with section 4, 3.5 with 3.6, the original defect fixed with a reproduction test and the + deferred defect recorded. +- Reader 2: 3.1 (two bullets, counted once), 3.2's strongest kind, 3.3, 3.4 with section 4, 3.5 + with 3.6. Its sixth bullet's "A verdict on the result rests on reproduction" describes the + reviewer, which the key says is neither correct nor wrong. +- Reader 3: 3.1 (two bullets, counted once), 3.2's strongest kind and never a test, 3.3, 3.4 with + section 4, 3.5 with 3.6 and the reproduction test. +- Reader 4: 3.1 (two bullets, counted once), 3.2's strongest kind, 3.3, 3.4 with section 4, 3.5 + with 3.6 and the reproduction test. +- Reader 5: 3.1 (three bullets, counted once), 3.2's strongest kind with 3.6's identifier, 3.3, + 3.4 with section 4, 3.5 with 3.6. + +No reader has the fix first, a test as the detector, an agent baselining, or a weaker kind never +allowed; readers 1, 2, 3 and 5 give the order semantic, syntactic, textual, which the key counts +as naming the strongest-kind line. Readers 3, 4 and 5 also assert, inside a bullet that names +clause 3.3, that an exclusion needs a written sentence or goes to the owner; see the points below. + +### Question 2 + +- **2a**, five of five. Each states the four steps, attribution, the rule, its proof and the + sweep, in that order with the fixing after the sweep, and each quotes clause 3.4's "The + [Practitioner] MUST run the [Rule] across the entire codebase and record the total [Instance] + count before fixing anything" (reader 4 from "record the total"). Reader 2 adds that the + document "does not place those steps in one consistent sequence", citing Appendix A's fix of + the original defect last; that is not an assertion the key marks wrong, and its own sequence + has the fix after the sweep. +- **2b**, five of five. Each names the owner, a human, and says an agent may not; each quotes + section 4's "An [Agent] MUST NOT [Baseline], suppress or knowingly leave an [Instance] unfixed + on its own authority, whatever the [Instance] count turns out to be." +- **2c**, five of five. Each states that all six clauses were followed and that the verdict rests + on reproduction. Readers 1, 2, 3 and 5 quote section 7's conforming-remediation sentence; + reader 4 states it in its own words and quotes section 7's "A verdict on a remediation or a + [Defence] MUST rest on reproduction, not on the report". +- **2d**, five of five. Each states the order semantic, else syntactic, else textual, quoting + clause 3.2's "A [Rule] the [Practitioner] writes MUST be written in the strongest kind of + [Detector] that can express it". Each quotes the two-grounds sentence, which states both + grounds; reader 3 also gives them in its own words. Each states that the grounds are + recorded in both places, the [Remediation docs] and a commit description: readers 1 and 3 in + their own words, readers 2 and 4 by "two places" with the quotation "on the [Rule]'s + [Remediation docs], and in the description of a commit", reader 5 by that quotation. No reader + says the places are alternatives, that a weaker kind is never allowed, or that the owner must + always approve; reader 1 says sign-off "applies only where the project requires it". + +### Question 3 + +- **3.1**: three Partial (readers 1, 4, 5), two No (readers 2, 3); the key accepts any grade. +- **3.2**: five Partial, each on the detector or its kind not named. No reader graded Yes. +- **3.3**: three Yes (readers 1, 2, 3), accepted without condition. Reader 4 Partial, attributing + the inline ignore on `Report::render` to Part B: correct. **Reader 5 Partial: a miss.** Its + reasoning is that the enforced run reports the [Fixture], which "3.3 says it must not", and + that the [Fixture]'s status after commit C is not explained; it does not attribute the inline + ignore to this clause, the key's condition for Partial or No. The reasoning paraphrases clause + 3.3 Part C ("the enforced run does not report it") and gives no quotation of the document. The + key names no wrong reading for 3.3, so this is a single miss, tolerated by the threshold. +- **3.4**: five No, each naming `Report::render()` left under the inline ignore. +- **3.5**: five No, each on the inline ignore no recorded decision covers. +- **3.6**: five No, each naming the wiki and the documentation not versioned with the rule. +- **Remediation**: five say it does not conform, each naming 3.4, 3.5 and 3.6 among the clauses + not followed. +- **Defence**: five say it does not conform, each naming 3.5's uncovered suppression and 3.6. +- **Section 4**: five name the inline ignore on `Report::render()`. All five also name, as a + second decision, leaving the wider rule unbuilt; not marked. + +## Passages named under question 4 + +| Passage | Clause | Readers | Text | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------ | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| "Attempt rather than pre-judge": "in the vocabulary clause 3.1 uses for a search" (reader 1); "naming at least two independent techniques tried at expressing the [Rule], in the vocabulary clause 3.1 uses for a search" (readers 2, 4) | 2 | three: 1, 2, 4 | Untouched. Readers 2 and 4 set it against clause 3.1's search techniques, untouched | +| The two-bounds paragraph: "Both bounds measure breadth within one level; the level itself is a separate question, answered against the report." (readers 1, 5) | 3.1 | two: 1, 5 | Untouched. Reader 5 sets it against the lower and upper bounds, untouched | +| The lower-bound list item: "That rejection is tested as a [Narrowing] is under clause 3.3" (reader 1); "A Rule drawn to the exact value or name found in the Defect is the same Narrowing however it is spelt." (reader 3) | 3.1 | two: 1, 3 | Untouched. Reader 1 sets it against clause 3.3's test, reader 3 against the Narrowing entry; both untouched | +| "A remediation MAY be delivered across several changes": "provided the [Rule] does not become [Blocking] until every [Instance] is fixed" (reader 2); "does not become [Blocking] until every [Instance] is fixed" (reader 4) | 4 | two: 2, 4 | Untouched. Reader 2 sets it against "What to do whilst waiting", reader 4 against Appendix A's "Make the [Rule] a permanent part" paragraph; both untouched | +| Appendix A's closing paragraph: "Only then fix the original [Defect]" (reader 2, against clause 3.4 and section 2); "Only then fix the original Defect." (reader 3, against clause 3.3's "the fix is committed after it"); "Only then fix the original [Defect] in the normal way, with a test that reproduces it." (reader 4) | App A | three: 2, 3, 4 | Untouched; the change altered only Appendix A's "Write a custom [Rule]" paragraph, which no reader names. The paragraphs set against it are untouched | +| Section 4's list item on leaving the next wider [Rule] unbuilt: "The absence of an Instance is the Practitioner's reason not to widen unasked." (reader 3); described as "§4 reserves that decision to the owner", set against clause 3.1's worked example (reader 2) | 4 | two: 3, and 2 (open point) | Untouched | +| "What to do whilst waiting": "the [Practitioner] … leaves the [Rule] unmerged rather than merged in a weakened form" and "the [Rule] MAY merge at full width with that case recorded as a known [Instance] awaiting the [Owner]" | 4 | one: 2 | Untouched | +| Clause 3.1's worked example: "was not built because most of the codebase's catches are legitimate and the [Owner] has not decided" | 3.1 | one: 2 | Untouched | +| Section 2's "When it can" paragraph: "before the [Defect] is fixed", and clause 3.4's opening sentence: "fix every [Instance]", both set against Appendix A's closing paragraph | 2, 3.4 | one: 2 | Untouched | +| Clause 3.2's recording paragraph: "which of the two grounds holds at each step down the order [Semantic], [Syntactic], [Textual], and why" | 3.2 | one: 2 | Written by the change | +| Clause 3.3 Part A, "The proof MUST survive as a commit of its own": "the fix is committed after it" | 3.3 | one: 3 | Untouched | +| Clause 3.4, "Fix them all.": "the scale at which that becomes true is tremendous." | 3.4 | one: 3 | Untouched | +| Appendix A's "Make the [Rule] a permanent part of the project's quality checks, failing rather than [Warning]." paragraph | App A | one: 4 | Untouched | +| Section 4's lead-in "Decisions that MUST go to whoever owns the codebase:", set against the Owner entry, "Whoever holds authority over a project's [Exceptions] and [Calibrations]" | 4, 1 | one: 5 | Untouched, and the Owner entry untouched | +| Clause 3.3 Part B, "A [Narrowing] reduces the [Practitioner]'s own work": "a [Narrowing] that excludes more code than the [Rule] still covers MUST be reported to the [Owner] as if it were a [Suppression]." | 3.3 | one: 5 | Untouched | + +Notes, not passages: none. Suggestions volunteered: none. + +## Findings and where they lie + +1. **Section 2, "Attempt rather than pre-judge"**, readers 1, 2 and 4. Untouched; the change's + words in section 2 are in the next paragraph, "There are three ways out", which no reader + names. Readers 2 and 4 set it against clause 3.1's search techniques, untouched. Routed to + `acceptance/ROUTED.md` (its existing row, raised again); does not block. +2. **Clause 3.1, the two-bounds paragraph**, readers 1 and 5. Untouched by this change, and set + against the bounds list items, untouched. Its row is closed by `editor/defined-terms` run 64, + whose rewrite is not in this branch; routed to `acceptance/ROUTED.md`, noted on that row as + raised again on the text before that rewrite; does not block. +3. **Clause 3.1, the lower-bound list item**, readers 1 and 3. Untouched; set against clause + 3.3's test and the Narrowing entry, both untouched. Routed to `acceptance/ROUTED.md` (its + existing row, raised again); does not block. +4. **Section 4, "A remediation MAY be delivered across several changes"**, readers 2 and 4. + Untouched; set against "What to do whilst waiting" and Appendix A's "Make the [Rule] a + permanent part" paragraph, both untouched. Routed to `acceptance/ROUTED.md`, on the existing + section 4 row that sets "MAY merge at full width" against this sentence; does not block. +5. **Appendix A, "Only then fix the original [Defect] in the normal way"**, readers 2, 3 and 4. + Untouched: each paragraph of Appendix A's block quote is a paragraph of its own, and the change + altered only "Write a custom [Rule]". Set against section 2's "When it can", clause 3.4's + opening sentence, clause 3.3 Part A's "The proof MUST survive" and section 4's several-changes + paragraph, all untouched. Routed to `acceptance/ROUTED.md` as a new row; does not block. +6. **Section 4, the list item on leaving the next wider [Rule] unbuilt**, reader 3, and reader 2 + by the open-point rule. Untouched; reader 2 sets it against clause 3.1's worked example, + untouched. Routed to `acceptance/ROUTED.md` (its existing row, raised again); does not block. + +No finding lies in text the change wrote or altered. Of the passages from one reader only, one +lies there: clause 3.2's recording paragraph, "at each step down the order" (reader 2). No reader +names the terminology entries for Textual, Syntactic or Semantic, the section 3 map's 3.2 row, +clause 3.2's other new paragraphs, clause 3.6's deferral, section 4's proviso or section 7's +conforming-[Defence] sentence. + +## Points the rules did not settle + +- **Whether reader 2 names section 4's next-wider list item.** Reader 2 quotes clause 3.1's worked + example and sets it against "§4 reserves that decision to the owner", a clause number with a + description that identifies the list item on leaving the next wider [Rule] unbuilt. Marked by + the open-point rule as naming the passage, which makes finding 6. It lies in untouched text + either way, so the verdict is the same. +- **Question 1 bullets on [Narrowing].** Readers 3, 4 and 5 each assert, in a bullet that also + names clause 3.3's proof, that an exclusion or narrowing needs a written sentence, or goes to + the owner (clause 3.3 Part B). That obligation is in neither of the key's lists. Not counted + either way; each bullet counts for its clause 3.3 content, and each reader has four or more + correct bullets without it. An open point for the key. +- **Considered, and settled by the rules.** The marker's agent definition says a miss whose + reasoning quotes the document is a finding whatever the count; `ACCEPTANCE.md` has no such + rule, and the marker marked by `ACCEPTANCE.md`. The one miss, reader 5 on 3.3, quotes nothing, + so the point does not arise in this run. + +## Process change + +Made under the process change of pull request 22 before it merged, at commit f3ec69e, named +before the readers started. f3ec69e is on `editor/acceptance-coherent`, one commit after +5852a85, which this branch merged; it changes only `acceptance/ROUTED.md`. `ACCEPTANCE.md`, +`acceptance/README.md`, `acceptance/runs/README.md` and `CONTRIBUTING.md` are identical at +f3ec69e and befe52c. + +## Findings applied + +None at the time of marking; what to change is the editor's decision. + +## Against the criteria + +- Load-bearing statements, every reader correct on every item: met; 2a, 2b, 2c and 2d five of + five. +- Restatement and application, all but one reader correct on each keyed item: met; 3.3 four of + five, every other item five of five. +- A miss the key says counts against the document: none. +- A passage found confusing by two or more readers, in text the change wrote or altered: none. +- A passage found confusing by two or more readers, elsewhere: six, section 2's "Attempt rather + than pre-judge", clause 3.1's two-bounds paragraph and lower-bound list item, section 4's + several-changes paragraph and next-wider list item, and Appendix A's closing paragraph, routed + to `acceptance/ROUTED.md`. +- From one reader only: nine passages, recorded. + +**Verdict: PASS.** + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125136762); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125136402); the prompt, [part 1](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125137201) and [part 2](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125137748). diff --git a/acceptance/runs/071-semantic-detectors-run-14.md b/acceptance/runs/071-semantic-detectors-run-14.md new file mode 100644 index 0000000..e7d4322 --- /dev/null +++ b/acceptance/runs/071-semantic-detectors-run-14.md @@ -0,0 +1,170 @@ +# Run 71: DETECTOR-SPEC.md, semantic detectors, fourteenth run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: 14, open on branch `editor/semantic-detectors`. The readers read commit + befe52c. Change: a new clause 8.2, that a verdict on a detector names its kind, semantic, + syntactic or textual, in each configuration it grades; section 9's sentence on that + obligation; section 2's table rows for the three kinds and for Message. +- **Run within the pull request**: 14 (runs 19 to 30 and run 70 came before; runs 20, 22, 24, + 26, 28 and 30 read this document on an earlier, larger text of the change under the earlier + acceptance rules). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `d45ab8c1ff1dc2ecabd1bad36eca32c3b0858327c5079cd4b6aea2abcb4e5cc3`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at befe52c, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model; each sheet is that output's `result`. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | --- | ---- | --- | --- | --- | --- | --- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | miss | ok | miss | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | miss | ok | miss | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | miss | ok | miss | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | miss | ok | miss | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | miss | ok | ok | ok | ok | ok | ok | ok | + +Question 1: every reader's first six bullets name, one each, the key's lines for 4.1; 4.2; 4.3; +5.1 to 5.4; 6.1 to 6.3; and 7.1. Reader 2's sixth bullet adds 7.2 as a SHOULD and what +conformance means, and reader 4's sixth adds what conformance means; the key says such a +statement is neither correct nor wrong, and both bullets still name 7.1. Reader 3's seventh +bullet, on conformance, is not marked, since only the first six are. No reader names the 8.2 +line, which the count does not need. No bullet asserts anything the key marks wrong. + +Question 2: every reader states every required element of each item and quotes the document. +2a: all five state that conformance is every MUST in sections 4 to 7 and that a detector +predating the document can conform, quoting clause 8.1's "A Detector that predates this +document, or whose maintainer has never read it, MAY be graded Conforming on evidence" +(readers 2, 3 and 5 also quote section 8's definition). 2b: all five name both things, +disableable by configuration or detectable, and that either alone is enough, quoting clause +7.1's "For each one it MUST do at least one of two things, and either alone satisfies this +clause". 2c: all five state that the documentation ships with the rule or lives in the installed +copy, at a version tracked together, and resolves without network access, quoting the 6.2 and +6.3 headings; readers 2, 3 and 5 add that a website does not satisfy it, which is not among the +key's wrong assertions. + +Question 3: every reader gives the accepted grade on 4.1, 4.2, 4.3, 5.1, 5.2, 5.4, 6.2, 6.3 and +7.1, several with caveats about what the fixture does not say (4.2's output, 4.3's +machine-readable formats), which the key accepts. S8: all five say Lintomatic does not conform, +naming 6.3; all five note that the verdict cannot name Lintomatic's kind because the fixture +does not say it, and that this does not alter the conformance finding, which the key does not +mark. 6.4: all five say it does not change the verdict, because it is a SHOULD. + +### Misses + +The key's wrong readings for question 3 are 6.3 graded Partial on a proportion, 7.1 graded No +on the missing reason, and the section 8 verdict made to turn on the kind. No miss matches one. +None of the reasoning below quotes the document as `ACCEPTANCE.md` defines quoting (a run of six +or more consecutive words); each paraphrases it or quotes the fixture. + +- **5.3, readers 1 to 5, Partial (key: Yes).** Every reader reasons that the one-line output is a + usable summary but that the fixture never says the command's output gives the location of + `.lintomatic/report.html`, which they read clause 5.3's second sentence ("both a usable summary + and the location of the remainder") as requiring. Reader 1 adds, under question 4, that what + makes a summary "usable" decided its grade. Five of five: the item fails its threshold. +- **6.1, readers 1 to 4, Partial (key: Yes).** Each reasons that `explain` resolves an identifier + presented alone, but that the two bundled rules printing `No documentation declared` do not + resolve to documentation, so the mechanism fails for them; reader 4 says it counts the same + defect under 6.1 and 6.3. Reader 5 gives Yes. Four of five: the item fails its threshold. + +Both items fail as keyed items, which `ACCEPTANCE.md` says are fixed in the text that carries +the obligation wherever it lies; clauses 5.3 and 6.1 are not text the diff 9e914fe..befe52c +changes. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..befe52c adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------- | +| 1 | Clause 8.1, first paragraph: "a [Detector] that has learnt, from its own checks or from a [Practitioner]'s report under the method's clause 3.2, that it fails a MUST in sections 4 to 7 MUST record that gap alongside the version it declares" (readers 4 and 5; reader 1 a shorter quotation of the same sentence); whether the duty binds a detector that makes no declaration, set against passage 2 by readers 1 and 4, and against "the version it declares" within the sentence by reader 5 | 1, 4, 5 | no | +| 2 | Clause 8.1, second paragraph: "The declaration is optional" (readers 1 and 4) and "it is not a condition of it" (reader 4), set against passage 1 | 1, 4 | no | +| 3 | Clause 7.1, first paragraph: "or make the route detectable, by a [Rule] the project can write in the [Detector] itself" (readers 2 and 3, reader 3 with the rest of the sentence); whether that rule is clause 4.1's bespoke rule, added to configuration, or a change to the detector's own codebase | 2, 3 | no | +| 4 | Clause 8.2, first paragraph: "the same configurations the method specification's clause 3.2 counts when it asks for the strongest kind" (readers 2 and 3); which configurations are meant, the method specification not having been supplied. Reader 1 names "the method specification's clause 3.2" with clause 8.2 among terms it read as ordinary English for want of `SPEC.md` (point 2 below) | 1, 2, 3 | yes | +| 5 | Clause 5.3, first paragraph: "the invoked command's own output MUST carry both a usable summary and the location of the remainder"; what makes a summary usable | 1 | no | +| 6 | Section 2, introductory paragraph: "Terms from the method specification carry over unchanged", with [Class] and [Owner] pointing to `SPEC.md`, which reader 1 was not given (point 2 below) | 1 | no | + +Notes, not passages: none beyond what point 2 below counts as passages. + +## Findings and where they lie + +- **Finding A, passage 1, clause 8.1 first paragraph** (readers 1, 4 and 5): not changed by the + diff, and the paragraph the readers set it against (passage 2) is not changed either. Routed + to `acceptance/ROUTED.md`; does not block. +- **Finding B, passage 2, clause 8.1 second paragraph** (readers 1 and 4): not changed by the + diff, and set only against passage 1. Routed to `acceptance/ROUTED.md`; does not block. +- **Finding C, passage 3, clause 7.1 first paragraph** (readers 2 and 3): not changed by the diff, + and set against no changed paragraph. Routed to `acceptance/ROUTED.md`; does not block. +- **Finding D, passage 4, clause 8.2 first paragraph** (readers 2 and 3, and reader 1 under point + 2): the diff adds the whole of clause 8.2. Text the change wrote; blocks. + +`acceptance/ROUTED.md` at befe52c lists, under Closed, rows on the passages of findings A and B +(clause 8.1, "MUST record that gap alongside the version it declares" against "The declaration +is optional.") and of finding C (clause 7.1, "by a Rule the project can write in the Detector +itself"), closed by run 66 on `editor/detector-route-rule`, whose rework of those clauses is not +in the text these readers read. Where the routed findings are entered is the editor's decision. + +## Points the rules did not settle + +1. **Passage 4, readers 2 and 3: a passage or a note.** Each quotes a sentence of clause 8.2 with + the clause number and says it could not tell which configurations the sentence means, + because the method specification was not supplied. `ACCEPTANCE.md` makes a remark that + another document was not supplied a note only where it is "not about a passage of the + document"; these remarks are about what a quoted sentence means. Marked as passages, which + is also what the open-point rule says: an open point on whether a reader named a passage + counts the passage. Finding D turns on this point. +2. **Reader 1's third item: a passage or a note.** Its subject is that `SPEC.md` was not given, + which reads as a note, but it quotes section 2's "Terms from the method specification carry + over unchanged" (a run of eight words) and names "the method specification's clause 3.2" + with clause 8.2, whose first occurrence in 8.2 is the first paragraph. Counted, by the + open-point rule, as naming section 2's introductory paragraph (passage 6) and clause 8.2's + first paragraph (passage 4). Finding D does not turn on this point, since readers 2 and 3 + name the same paragraph. +3. **Readers 1 and 4 naming two paragraphs in one item.** Each quotes sentences from both the + first and the second paragraph of clause 8.1 and sets them against each other. Counted as + naming both paragraphs, by the open-point rule. Finding A stands on readers 4 and 5 without + reader 1; finding B depends on this point, and is routed either way. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit f3ec69e, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item except + 5.3 (none of five) and 6.1 (one of five). Not met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: one (finding D, + clause 8.2 first paragraph). Not met. +- Passages named by two or more readers elsewhere: three (findings A, B and C), for + `acceptance/ROUTED.md`. + +## Findings applied + +None at the time of marking. + +## Verdict + +FAIL. Question 3's items 5.3 and 6.1 fail the all-but-one threshold, and finding D lies in +clause 8.2's first paragraph, which the change wrote. + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125138630); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125138207); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125139111). diff --git a/acceptance/runs/073-detector-route-rule-run-4.md b/acceptance/runs/073-detector-route-rule-run-4.md new file mode 100644 index 0000000..2f070f2 --- /dev/null +++ b/acceptance/runs/073-detector-route-rule-run-4.md @@ -0,0 +1,151 @@ +# Run 73: DETECTOR-SPEC.md, detector route rule, fourth run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [23](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23), from branch + `editor/detector-route-rule`. The readers read commit d67174c. Change: clause 7.1's detectable + route is a bespoke rule the project writes for the detector under clause 4.1, with a paragraph + on what detectable means; clause 8.1's gap duty falls only on a detector that declares a + version; clause 5.3 says what the remainder is; clause 6.1 says it asks for the mechanism, a + missing page failing clause 6.3, and that a path in the message is not enough on its own; + section 2's table gains a Message row. Revised after run 69 (a cohort of another change with + this one merged in) in clause 5.3's obligation sentence, the location of the remainder owed + only where the summary leaves a finding out, and in clause 8.1's second paragraph, where a + practitioner's record of a gap lives and that a detector without a declaration can still be + found to conform; so this is a fresh run. +- **Run within the pull request**: 4 (runs 60, 63 and 66 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `0df0b01c02c5d2fd83b37d218f88785980384c3a0ecadce92abf7088211c9249`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at d67174c, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Question 1: correct bullets per reader 6, 6, 6, 6 and 6. Readers 1, 2, 3 and 5 give one bullet +per line of the key's list (4.1; 4.2; 4.3; 5.1 to 5.4; 6.1 to 6.3; 7.1). Reader 4 splits 5.1 to +5.3 from 5.4, which are different clauses of one line and each count, and joins 6.1 to 6.3 with +7.1 in its sixth bullet. Reader 2's fourth bullet says every finding must reach the command's +output, which the key says names 5.3 and is not marked wrong for the plural; its fifth bullet +says every identifier the detector can print must have a page, which names the 6.1 to 6.3 line +and is not among the key's wrong assertions. Reader 5 adds, after its six bullets, a line on +what conformance means; it is not among the first six bullets and states no obligation. No +bullet asserts anything the key marks wrong. + +Question 2: every reader states every required element of each item and quotes the document. +For 2a all five state that conformance is every MUST in sections 4 to 7 and that a detector +predating the document can conform; readers 1 to 4 quote section 8's definition, and all five +quote clause 8.1's "A Detector that predates this document, or whose maintainer has never read +it, MAY be graded Conforming on evidence" (reader 5 also its "a Detector without one can still +be found to Conform"). For 2b all five name both things, disableable by configuration or +detectable, and that either alone is enough, quoting clause 7.1's "For each one it MUST do at +least one of two things, and either alone satisfies this clause". Readers 2, 3 and 5 add that +the bespoke-rule way is closed to a detector whose rules never see comments, the document's own +sentence, which is not among the key's wrong assertions for 2b. For 2c all five state that the +documentation ships with the rule (in the installed copy), at a version tracked together, and +is reachable without network access, quoting the 6.2 and 6.3 headings; reader 4 adds clause +6.2's sentence that a website catalogue is the wrong place, which agrees with the key. + +Question 3: readers 1, 2, 4 and 5 give every accepted grade, each with caveats on 4.2 (the +fixture does not say `--only` prints finding lines) and 4.3 (no machine-readable format +described); the key says a caveat with the accepted grade is correct. S8: all five say +Lintomatic does not conform, naming 6.3. 6.4: all five say it does not change the verdict, +because it is a SHOULD. + +### Misses + +- **Reader 3, clause 4.2: Partial** (accepted: Yes). Reasoning: `--only` runs one rule and exits + 1 when it fires, but the clause requires the harness to report "what it printed" and the + fixture never says `--only` prints the finding line; "it becomes Yes if `--only` output + includes the finding". The reasoning quotes three words of the document, below the six that + `ACCEPTANCE.md` defines as quoting it, so it does not quote the document. The key names no + wrong reading for 4.2. A single miss, the reader's, tolerated by the threshold. Reader 3's + question 4 note puts the close call on the fixture, not the document. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..d67174c adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | ------------------- | +| 1 | Clause 8.1, first paragraph: "The same declaration is where a gap against this document is recorded once it is known: a [Detector] that declares a version and has learnt… that it fails a MUST in sections 4 to 7 MUST record that gap…"; whether a MUST in section 8 counts toward conformance | 1 | yes | +| 2 | Clause 5.3, first paragraph: "the invoked command's own output MUST carry a usable summary" (2, what makes a summary usable); "The remainder is the findings the summary leaves out." (5, whether a fuller HTML report is a remainder, settled only by the sentences after) | 2, 5 | yes | +| 3 | Clause 7.1, second paragraph: "a [Detector] whose [Rules] never see comments cannot make an ignore comment detectable this way" (2, whether a property of the detector's API or of its rules); "That way is open only where the [Detector]'s [Rules] can see the route: …" (4, whether the route was open to Lintomatic) | 2, 4 | yes | +| 4 | Section 1, "The governing principle" paragraph: "a [Conforming] [Detector] MUST make that possible without the project building the mechanism first"; an unnumbered MUST outside sections 4 to 7, and whether it bears on conformance | 4 | no | +| 5 | Clause 4.2, first paragraph: "a [Detector] failing 4.1 is graded here on its [Bundled rules]", set beside the sentence before it that the clause applies to every rule | 5 | no | + +Notes, not passages: reader 3 wrote "none", adding that its 4.2 close call came from the fixture +not stating what `--only` prints, not from the document. + +## Findings and where they lie + +- **Finding A, passage 2, clause 5.3 first paragraph** (readers 2 and 5): the diff 9e914fe..d67174c + rewords this paragraph's obligation sentence and adds its three following sentences, so it is + text the change wrote or altered. Reader 2's words ("usable summary") are in the part of the + sentence the change kept, but the rule is by paragraph. Blocks. +- **Finding B, passage 3, clause 7.1 second paragraph** (readers 2 and 4): the diff adds this + paragraph. Text the change wrote. Blocks. + +Passages 1, 4 and 5 are each named by one reader: recorded, not findings. Passage 1 lies in a +changed paragraph but is single. Nothing is routed to `acceptance/ROUTED.md` from this run. + +## Points the rules did not settle + +1. **Reader 3's 4.2 grade carries a condition for another grade.** `ACCEPTANCE.md` reads a grade + given on a stated condition as the grade for the case the fixture presents where the key says + which case that is, and otherwise as unanswered; the key says no row carries a condition. + Reader 3 gives Partial as its grade and states the condition only for Yes. Marked as Partial, + the grade given; read either way (Partial, or unanswered) the item is a miss, so the mark does + not turn on the point. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit bc6ae14, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item but + 4.2, which is four of five. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: two (findings A and + B). Not met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking; findings A and B are on the change's own text, to be fixed there +before a fresh cohort. + +## Verdict + +FAIL. Every keyed item passed its threshold and no miss counts against the document, but +findings A (clause 5.3, first paragraph) and B (clause 7.1, second paragraph) lie in text the +change wrote or altered, and each blocks. + +## Answer sheets + +Posted on the pull request: LINK PLACEHOLDER. diff --git a/acceptance/runs/074-semantic-detectors-run-15.md b/acceptance/runs/074-semantic-detectors-run-15.md new file mode 100644 index 0000000..adcd679 --- /dev/null +++ b/acceptance/runs/074-semantic-detectors-run-15.md @@ -0,0 +1,145 @@ +# Run 74: DETECTOR-SPEC.md, semantic detectors, fifteenth run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: 14, open on branch `editor/semantic-detectors`. The readers read commit + f7e55e1. Change: a new clause 8.2, that a verdict on a detector names its kind, semantic, + syntactic or textual, in each configuration it grades; section 9's sentence on that + obligation; section 2's table row for the three kinds. Since run 71 the branch has merged + `editor/detector-route-rule` (pull request 23), whose rework of clauses 5.3, 6.1, 7.1 and 8.1 + and whose Message row in section 2 are that change's text, not this one's; clause 8.2's + sentence on which configuration's kind is named was reworded after run 71. +- **Run within the pull request**: 15 (runs 19 to 30, 70 and 71 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `003d030ade886b2008da3abc90e75d8c62be8f02f79c3b4aec05818379bcc550`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at f7e55e1, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model; each sheet is that output's `result`. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Question 1: readers 1, 3 and 4 name, one bullet each, the key's lines for 4.1; 4.2; 4.3; 5.1 to +5.4; 6.1 to 6.3; and 7.1. Reader 2 names the same six, its sixth bullet adding what conformance +means, which the key says is neither correct nor wrong; its fourth says every finding must +appear in the command's output, which the key says is not marked wrong for the plural. Reader 5 +names 4.1 with 4.2, 4.3, 5.1 to 5.4, 6.1 to 6.3 and 7.1, and its sixth bullet says any verdict +on a detector must name its kind, which names the 8.2 line and does not put the duty on the +detector. No bullet asserts anything the key marks wrong. + +Question 2: every reader states every required element of each item and quotes the document. +2a: all five state that conformance is every MUST in sections 4 to 7 and that a detector +predating the document can conform, quoting section 8's definition and clause 8.1's "A Detector +that predates this document, or whose maintainer has never read it, MAY be graded Conforming on +evidence". 2b: all five name both things, disableable by configuration or detectable, and that +either alone is enough, quoting clause 7.1's "For each one it MUST do at least one of two +things, and either alone satisfies this clause" (reader 2 with an ellipsis inside the sentence, +reader 3 ending in an ellipsis; each run before the ellipsis is well over six words). 2c: all +five state that the documentation ships with the rule in the installed copy, at a version +tracked together, and resolves without network access, quoting the 6.2 and 6.3 headings (reader +4 also 6.3's first sentence); readers 3 and 5 add that a website does not satisfy it, which is +not among the key's wrong assertions. + +Question 3: every reader gives the accepted grade on all eleven clauses, several with caveats on +what the fixture does not say (what `--only` prints under 4.2, machine-readable formats under +4.3), which the key accepts since no row carries a condition. S8: all five say Lintomatic does +not conform, naming 6.3 or the two undocumented bundled rules; all five say the verdict cannot +name Lintomatic's kind because the fixture does not say it, and that this leaves the +conformance finding standing, which the key does not mark. 6.4: all five say it does not change +the verdict, because it is a SHOULD. + +### Misses + +None. No answer matches a wrong reading the key names for question 1 or question 3. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "this change's text" means a paragraph the diff +d67174c..f7e55e1 adds, rewords, moves, deletes a sentence from, or changes a word, term or link +in. The diff 9e914fe..f7e55e1 shows this change and the merged detector change together. + +| # | Passage (clause, words or description) | Readers | This change's text | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------ | +| 1 | Clause 8.2, first paragraph: "The kind named is the one the [Detector] has in a configuration that needs no hosted service and no licence tier." Reader 2: whether a verdict grading a hosted configuration names the local kind or the hosted one; reader 3: whether it means the strongest kind available locally or the kind of whichever local configuration is named; reader 4 gives no reason | 2, 3, 4 | yes | +| 2 | Clause 6.1, paragraph "For a project's own Rule": "Where the [Identifier] is itself a URL, as method clause 3.6 allows, the [Detector] prints that URL and has done what this clause asks."; the method specification was not supplied (point 1 below) | 1 | no | +| 3 | Clause 8.1, first paragraph: "the method's clause 3.2", given with clause 8.1 as another reference to an unsupplied document (point 1 below) | 1 | no | +| 4 | Clause 6.3, first paragraph: "One [Identifier] that does not resolve fails this clause, however many others do. A page per [Identifier] satisfies this; a page per family of [Identifiers] MAY be used instead, under clause 6.5."; whether a prose-only family page meets 6.3's MUST | 5 | no | +| 5 | Clause 6.5, first paragraph: "Such a page SHOULD let the check clause 6.4 asks for confirm that each member resolves to it."; set against passage 4 | 5 | no | +| 6 | Clause 6.5, table row "Says in prose that everything under `LM-01` is documented here": "No: a person can infer it, a check cannot"; set against passage 4 | 5 | no | +| 7 | Clause 6.1, paragraph "In both, the mechanism is keyed on the Identifier exactly as printed", described, not quoted: "6.1 requires a lookup keyed on the identifier exactly as printed"; set against passage 4 (point 2 below) | 5 | no | + +Notes, not passages: reader 1's general remark that it took references to the method and +toolchain specifications on trust, not having been given them. + +## Findings and where they lie + +- **Finding A, passage 1, clause 8.2 first paragraph** (readers 2, 3 and 4): the diff + d67174c..f7e55e1 adds the whole of clause 8.2, and the sentence the readers quote was reworded + after run 71. Text the change wrote; blocks. + +No other passage is named by two or more readers, so nothing is routed to +`acceptance/ROUTED.md` from this run. Passages 2 to 7 are each one reader's and are recorded +only. + +## Points the rules did not settle + +1. **Reader 1's question 4: passages or a note.** Its subject is that the method specification + was not given, which reads as a note, but it quotes a sentence of clause 6.1 (passage 2) and + names "the method's clause 3.2" with clause 8.1, a four-word quotation that occurs in both + paragraphs of 8.1. Counted, by the open-point rule (an open point on whether a reader named a + passage counts the passage), as naming clause 6.1's paragraph and clause 8.1's first + paragraph, where the words first occur; the general remark is recorded as a note. No finding + turns on this point: no other reader names either paragraph. +2. **Reader 5's description of clause 6.1.** It gives a clause number and a description, not a + quotation, that identifies the sentence "In both, the mechanism is keyed on the Identifier + exactly as printed"; counted as a passage by the open-point rule. It is not the paragraph + reader 1 names in clause 6.1, so no finding turns on it. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit bc6ae14, named before the readers started. `ACCEPTANCE.md`, +`acceptance/README.md` and `acceptance/runs/README.md` are identical at bc6ae14 and f7e55e1. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. Met. +- Passages named by two or more readers in text the change wrote or altered: one (finding A, + clause 8.2 first paragraph). Not met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking. + +## Verdict + +FAIL. Every keyed item passes, but finding A, named by readers 2, 3 and 4, lies in the first +paragraph of clause 8.2, which the change wrote: the sentence "The kind named is the one the +Detector has in a configuration that needs no hosted service and no licence tier." + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125140101); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125139595); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125140560). diff --git a/acceptance/runs/077-detector-route-rule-run-5.md b/acceptance/runs/077-detector-route-rule-run-5.md new file mode 100644 index 0000000..7d1852f --- /dev/null +++ b/acceptance/runs/077-detector-route-rule-run-5.md @@ -0,0 +1,154 @@ +# Run 77: DETECTOR-SPEC.md, detector route rule, fifth run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [23](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23), from branch + `editor/detector-route-rule`. The readers read commit dbdf841. Change: clause 7.1's detectable + route is a bespoke rule the project writes for the detector under clause 4.1, with a paragraph + on what detectable means; clause 8.1's gap duty falls only on a detector that declares a + version; clause 5.3 says what the remainder is; clause 6.1 says it asks for the mechanism, a + missing page failing clause 6.3, and that a path in the message is not enough on its own; + section 2's table gains a Message row. Revised after run 73 in the two paragraphs its findings + named: clause 5.3's paragraph now defines a usable summary (each finding carries where it is + and its Identifier) and the remainder (the findings the summary leaves out; none where the + output prints every finding; a fuller report written elsewhere is not one and need not be + named), and clause 7.1's second paragraph now states the two ways a route is detectable in one + paragraph (a bespoke Rule registered under clause 4.1, or a mechanical check the Detector + documents); so this is a fresh run. +- **Run within the pull request**: 5 (runs 60, 63, 66 and 73 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `2a96d5b0c6339f5402c9883018d35c0353c42dfad581c94d9b80a8b428882bc9`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at dbdf841, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Question 1: correct bullets among the first six, per reader, 6, 6, 6, 5 and 6. Readers 1 and 5 +give one bullet per line of the key's list. Reader 2 joins 4.1 and 4.2 in its first bullet and +splits 5.1 to 5.3 from 5.4. Reader 3 gives seven bullets; only the first six are marked (4.1, +4.2, 4.3, 5.1 to 5.3, 5.4, 6.1 to 6.3), and its seventh, on 7.1, is not marked. Reader 4 gives +five bullets, each naming a line of the key's list. Readers 1, 2 and 4 say the command's output +carries each finding with its location and identifier, which names 5.3 and is not marked wrong +for the plural. No bullet asserts anything the key marks wrong. + +Question 2: every reader states every required element of each item and quotes the document. +For 2a all five state that conformance is every MUST in sections 4 to 7 and that a detector +predating the document can conform, quoting section 8's definition and clause 8.1's "A Detector +that predates this document, or whose maintainer has never read it, MAY be graded Conforming on +evidence"; readers 1 and 4 also quote "a Detector without one can still be found to Conform". +For 2b all five name both things, disableable by configuration or detectable, and that either +alone is enough, quoting clause 7.1's "For each one it MUST do at least one of two things, and +either alone satisfies this clause" (reader 2 with an ellipsis; its longest verbatim run is well +over six words). Reader 4 adds that a detector whose rules never see comments has only the +documented mechanical check, the document's own sentence, which is not among the key's wrong +assertions for 2b. For 2c all five state that the documentation ships with the rule in the +installed copy, at a version tracked together, and is reachable without network access, +quoting the 6.2 and 6.3 headings; readers 3, 4 and 5 add clause 6.3's "Every Identifier a +Bundled rule can print MUST resolve", which agrees with the key. + +Question 3: readers 1, 2, 3 and 5 give every accepted grade, several with caveats on 4.2 (the +fixture does not say `--only` prints finding lines) and 4.3 (no machine-readable format +described); the key says a caveat with the accepted grade is correct. Readers 1, 3 and 5 state +that they read the two undocumented rules as bundled; reader 3 adds what would follow if they +were project rules, but gives No as its grade without condition. All five grade 5.3 Yes, every +finding being printed so no remainder needs naming, and 6.1 Yes. S8: all five say Lintomatic +does not conform, naming 6.3. 6.4: all five say it does not change the verdict, because it is a +SHOULD. + +### Misses + +- **Reader 4, clause 4.2: Partial** (accepted: Yes). Reasoning: `--only` runs a single rule and + exits 1 when it fires, which covers "whether the Rule fired", but the fixture never says a + single-rule run prints its findings, so "what it printed" is not established. The reasoning + quotes four and three words of clause 4.2, below the six that `ACCEPTANCE.md` defines as + quoting the document, so it does not quote the document; it puts the gap on the fixture's + silence. The key names no wrong reading for 4.2. A single miss, the reader's, tolerated by the + threshold. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..dbdf841 adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ------------------- | +| 1 | Clause 5.3, first paragraph: "The remainder is the findings the summary leaves out: where the output prints every finding there is none, and a fuller report written elsewhere, such as a page with a code snippet for each finding, is not one and need not be named."; what "is not one" refers to (the remainder or the summary), and the double negative | 2, 3, 4, 5 | yes | +| 2 | Clause 7.1, second paragraph: "The first way needs the [Detector]'s [Rules] to be able to match the route's text, so a [Detector] whose [Rules] never see comments offers only the second for an ignore comment."; how the project's bespoke rule relates to the detector's matching, given "need not ship itself" | 1 | yes | +| 3 | Clause 6.4, Why paragraph: "Where the [Detector] is shipped inside a [Toolchain], the [toolchain specification]'s self-audit makes this check a MUST for the [Toolchain]; a [Detector] released on its own is asked for it as a SHOULD."; whether a detector inside a toolchain keeps the SHOULD | 1 | yes | +| 4 | Clause 6.1, "For a project's own [Rule]" paragraph: "The lookup for such a [Rule] is an obligation on the project's assembled [Toolchain], under clause 4.2 of the [toolchain specification]"; the referenced clause could not be checked | 3 | no | +| 5 | Clause 7.1, Why paragraph: "the method specification's position is that [Suppression] is an [Owner] decision under its clause 3.4 and section 4"; it depends on definitions in SPEC.md | 3 | no | + +Notes, not passages: reader 1's third item is about the fixture's `explain` paragraph and its +last paragraph (whether the two rules printing `No documentation declared` are bundled), not a +passage of the document. Reader 2 notes that defined terms link to SPEC.md, which was not +supplied: a remark about another document. + +## Findings and where they lie + +- **Finding A, passage 1, clause 5.3 first paragraph** (readers 2, 3, 4 and 5): the diff + 9e914fe..dbdf841 rewords this paragraph's obligation sentence and adds the sentences after it, + including the one all four readers quote. Text the change wrote. Blocks. + +Passages 2 to 5 are each named by one reader: recorded, not findings. Passages 2 and 3 lie in +changed paragraphs but are single. Nothing is routed to `acceptance/ROUTED.md` from this run. + +## Points the rules did not settle + +1. **Reader 3's passages 4 and 5.** Each quotes six or more words of the document with its + clause number, which `ACCEPTANCE.md` counts as naming a passage, but the confusion stated is + that SPEC.md or TOOLING-SPEC.md was not supplied, which it records as a note and not a + passage. Under the open-point rule, an open point on whether a reader named a passage counts + the passage, so both are listed as passages. Each is named by reader 3 alone, so the verdict + does not turn on the point. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit bc6ae14, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item but + 4.2, which is four of five. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: one (finding A). + Not met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking; finding A is on the change's own text, to be fixed there before a +fresh cohort. + +## Verdict + +FAIL. Every keyed item passed its threshold and no miss counts against the document, but +finding A (clause 5.3, first paragraph, the sentence defining the remainder, named by four of +five readers) lies in text the change wrote, and blocks. + +## Answer sheets + +Posted on the pull request: LINK PLACEHOLDER. diff --git a/acceptance/runs/078-detector-route-rule-run-6.md b/acceptance/runs/078-detector-route-rule-run-6.md new file mode 100644 index 0000000..a2ebfda --- /dev/null +++ b/acceptance/runs/078-detector-route-rule-run-6.md @@ -0,0 +1,177 @@ +# Run 78: DETECTOR-SPEC.md, detector route rule, sixth run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: [23](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/23), from branch + `editor/detector-route-rule`. The readers read commit ac14996. Change: clause 7.1's detectable + route is a bespoke rule the project writes for the detector under clause 4.1, with a paragraph + on what detectable means; clause 8.1's gap duty falls only on a detector that declares a + version; clause 5.3 says what the remainder is; clause 6.1 says it asks for the mechanism, a + missing page failing clause 6.3, and that a path in the message is not enough on its own; + section 2's table gains a Message row. Revised after run 77 in the one paragraph its finding + named: clause 5.3's remainder sentence is now three plain sentences (the remainder is the + findings the summary leaves out; where the output prints every finding there is no remainder; + a fuller report written elsewhere is not a remainder and the output need not name it); so this + is a fresh run. +- **Run within the pull request**: 6 (runs 60, 63, 66, 73 and 77 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `9d52a2f0c2c15cb1f4d1b7d628fc403560534fa35cdde941e26efe717846f35c`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at ac14996, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 1 | correct | ok | ok | ok | ok | miss | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Question 1: every reader has six correct bullets among the first six, one per line of the key's +list (4.1, 4.2, 4.3, 5.1 to 5.4, 6.1 to 6.3, 7.1). Reader 1 gives seven bullets; its seventh, on +section 8, is not marked. Readers 3 and 5 also restate SHOULDs (4.4, 6.4, 7.2) as SHOULDs, and +reader 4's sixth bullet adds what conformance means; the key says neither is correct nor wrong. +Reader 4 says the output carries a pointer to any findings the summary leaves out, which names +5.3. No bullet asserts anything the key marks wrong. + +Question 2: every reader states every required element of each item and quotes the document. +For 2a all five state that conformance is every MUST in sections 4 to 7 and that a detector +predating the document can conform, quoting clause 8.1's "A Detector that predates this +document, or whose maintainer has never read it, MAY be graded Conforming on evidence"; readers +1, 3 and 4 also quote section 8's definition, reader 2 its "satisfies every MUST in sections 4 +to 7". For 2b all five name both things, disableable by configuration or detectable, and that +either alone is enough, quoting clause 7.1's "For each one it MUST do at least one of two +things, and either alone satisfies this clause"; readers 2, 4 and 5 add that 7.2's reason is a +SHOULD, and reader 3 that a detector whose rules never see comments has only the mechanical +check, the document's own sentence; neither is among the key's wrong assertions. For 2c all five +state that the documentation ships with the rule (reader 2 "inside the installed rule package"), +at a version tracked together or the same version, and reachable without network access, +quoting clause 6.2's heading and clause 6.3's heading or its "Every Identifier a Bundled rule can +print MUST resolve" sentence. None names a website as acceptable; reader 4 quotes 6.2's Why +saying it is not. + +Question 3: readers 2 to 5 give every accepted grade; readers 2, 3 and 4 add caveats on 4.2 (the +fixture does not say `--only` prints finding lines) and 4.3 (no machine-readable format +described), and the key says a caveat with the accepted grade is correct. Reader 3 states that +it read the two undocumented rules as bundled and gives No for 6.3 without condition. All five +grade 5.3 Yes because every finding is printed so there is no remainder, and 6.1 Yes; readers 2, +3 and 4 say the two missing pages fail 6.3, not 6.1. No reader grades 6.3 Partial or 7.1 No. S8: +all five say Lintomatic does not conform, naming 6.3 (reader 1 adds its own 4.2 Partial as a +second reason). 6.4: all five say it does not change the verdict, because it is a SHOULD. + +### Misses + +- **Reader 1, clause 4.2: Partial** (accepted: Yes). Reasoning: `--only` exits 1 when the rule + fires, so whether it fired is reported, but the fixture never says `--only` prints what the + rule printed, which 4.2 requires. The quotation in the reasoning is from the fixture ("exits 1 + when the rule fires, 0 when it does not"); clause 4.2 is paraphrased, not quoted, so the + reasoning does not quote the document. It puts the gap on the fixture's silence. The key names + no wrong reading for 4.2. A single miss, the reader's, tolerated by the threshold. + +## Question 4: passages named + +Paragraphs as `ACCEPTANCE.md` defines them; "changed" means the diff 9e914fe..ac14996 adds, +rewords, moves or deletes a sentence in the paragraph, or changes a word, term or link in it. +That diff changes section 2's Message row, clause 5.3's first paragraph, clause 6.1's "In both" +and "For a Bundled rule" paragraphs, clause 6.4's Why paragraph, clause 7.1's first and second +paragraphs, and clause 8.1's first and second paragraphs. + +| # | Passage (clause, words or description) | Readers | Changed by the diff | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------- | +| 1 | Clause 4.2, first paragraph: "a [Detector] failing 4.1 is graded here on its [Bundled rules]."; reader 2: the sentence's purpose unclear until read as applying 4.2 to bundled rules alone | 1, 2 | no | +| 2 | Clause 6.1, the table row for a project's own Rule: "Print the [Identifier] unaltered; the lookup is the [Toolchain]'s to give", set against the clause heading's "MUST provide a mechanism that resolves a printed identifier to its documentation" | 1 | no | +| 3 | Clause 6.1, the heading, which reader 1 sets against passage 2; counted as named by reader 1 under the open-point rule | 1 | no | +| 4 | Clause 8.1, the JSON example: `"method": "1.0.1"`, against the header's method specification version 1.1.0; what the field means | 2 | no | +| 5 | Clause 8.1, second paragraph: "A [Detector] that predates this document, or whose maintainer has never read it, MAY be graded [Conforming] on evidence: anyone may grade it, ... rests on that check and not on the claim."; how every detector is graded, under a heading about the optional declaration | 3 | yes | +| 6 | Clause 4.3, first paragraph: "The [Detector] MUST print it, unaltered, alongside every finding the [Rule] reports, in its default output and in every machine-readable format it offers."; whether the HTML report is a machine-readable format, set against 5.3's "fuller report" | 5 | no | +| 7 | Clause 5.3, first paragraph, which reader 5 cites ("5.3 calls the HTML a 'fuller report'") in passage 6; counted as named by reader 5 under the open-point rule | 5 | yes | + +Notes, not passages: reader 2's items on clause 6.1 ("Where the [Identifier] is itself a URL, as +method clause 3.6 allows": the method specification was not provided, so the cross-reference +could not be checked) and on clause 7.1's Why paragraph ([Owner], [Class] and [Hazard] are +defined only in the method specification, which was not given); reader 4's single item, the same +6.1 sentence and the references to method clauses 3.3 (4.2's Why) and 3.2 (8.1), with *Class* +and *Owner*, all because the method specification was not provided. Each is a remark that +another document was not supplied, which `ACCEPTANCE.md` records as a note and not a passage; +see the first point below. + +## Findings and where they lie + +- **Finding A, passage 1, clause 4.2 first paragraph** (readers 1 and 2): the diff + 9e914fe..ac14996 does not touch clause 4.2, and neither reader sets the paragraph against a + paragraph the change wrote or altered. A finding against the document, not the change: to be + recorded in `acceptance/ROUTED.md` as a new open row and routed to a change of its own. Does + not block. + +Passages 2 to 7 are each named by one reader: recorded, not findings. Passages 5 and 7 lie in +changed paragraphs but are single. + +## Points the rules did not settle + +1. **Remarks that the method specification was not supplied, made by quoting a sentence.** + Readers 2 and 4 each quote six or more words of clause 6.1's "For a project's own Rule" + paragraph, with the clause number, and say only that the method specification was not + provided, so the cross-reference could not be checked. `ACCEPTANCE.md` both says a quotation + with a clause number names a passage and says a remark "such as that another document was not + supplied" is a note and not a passage. Run 77 recorded this as an open point and counted the + passages; `ACCEPTANCE.md` and the key have not changed on it since, so it was not settled + before this run as "Changing the test" asks. This marker holds that the rule settles it: the + stated confusion in each item is exactly the rule's own example of a note, and the quotation + only locates the reference whose target was not supplied; so these are notes, and no open + point arises. The verdict turns on this. Were it held open and the passages counted, readers 2 + and 4 would form a finding on clause 6.1's "For a project's own Rule" paragraph, unchanged + and routed (it is already open in `acceptance/ROUTED.md`, runs 28, 63, 66); and reader 4's + "8.1's 'the method's clause 3.2'", whose words occur in both of clause 8.1's paragraphs, would + leave open whether reader 4 and reader 3 (passage 5) named the same changed paragraph, which + the open-point rule counts, so that finding would block. The editor settles the point in + `ACCEPTANCE.md` before any further run. +2. **Reader 1's passage on clause 6.1** sets the table row against the heading; whether it names + one passage or both is not settled, so both are counted (passages 2 and 3). Reader 5's + citation of clause 5.3 is counted the same way (passage 7). No other reader names any of + these paragraphs, so the verdict does not turn on either. + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 13b497b, named before the readers started. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item but + 4.2, which is four of five. Met. +- A miss the key says counts against the document: none. +- Passages named by two or more readers in text the change wrote or altered: none, on the + marking of point 1. Met. +- Passages named by two or more readers elsewhere: one (finding A, clause 4.2), routed to + `acceptance/ROUTED.md`; does not block. + +## Findings applied + +None at the time of marking; finding A is to be added to `acceptance/ROUTED.md` by the editor. + +## Verdict + +PASS, on the marking of point 1. Every keyed item meets its threshold, no miss counts against +the document, and the one finding, clause 4.2's first paragraph, lies outside the text the change +wrote and is routed. + +## Answer sheets + +Posted on the pull request: LINK PLACEHOLDER. diff --git a/acceptance/runs/079-semantic-detectors-run-16.md b/acceptance/runs/079-semantic-detectors-run-16.md new file mode 100644 index 0000000..f17d355 --- /dev/null +++ b/acceptance/runs/079-semantic-detectors-run-16.md @@ -0,0 +1,150 @@ +# Run 79: DETECTOR-SPEC.md, semantic detectors, sixteenth run + +- **Document**: DETECTOR-SPEC.md +- **Pull request**: 14, open on branch `editor/semantic-detectors`. The readers read commit + 23f5bf0. Change: a new clause 8.2, that a verdict on a detector names its kind, semantic, + syntactic or textual, in each configuration it grades; section 9's sentence on that + obligation; section 2's table row for the three kinds. The branch carries + `editor/detector-route-rule` (pull request 23) at its commit d67174c, whose rework of clauses + 5.3, 6.1, 7.1 and 8.1 and whose Message row in section 2 are that change's text, not this + one's. Since run 74, clause 8.2's sentence on which configuration's kind is named (one needing + no hosted service and no licence tier) has been removed; the clause's first paragraph now ends + at "as section 2 glosses it". +- **Run within the pull request**: 16 (runs 19 to 30, 70, 71 and 74 came before). +- **Cohort**: five readers, none reused from any earlier run. Model named for the run before the + readers started: `claude-haiku-5-5`. Every reader reported `claude-haiku-5-5`. +- **Isolation**: each reader a separate `claude -p` process, started through the `ccy-claude` + wrapper (which only reattaches the session's credential) in a fresh empty directory made by + `mktemp -d` outside any repository, with the packet on standard input; no tools, no settings, + no project context, no saved session; the machine applies no managed settings. The editor + attests the rest of the isolation. Command: + `ccy-claude -p --model haiku --safe-mode --tools '' --setting-sources '' --strict-mcp-config --no-session-persistence --output-format json < prompt.txt`. + The prompt (sha256 `7b94f116b3f875170cdfd9fd61f845b3cf7f233723bbd749dd38e93adb6a1fd9`) is + `QUESTIONS.md`, `DETECTOR-SPEC.md` and `fixtures/lintomatic.md` at 23f5bf0, each after one + `===== FILE: =====` line, one blank line between files; the marker rebuilt it from that + commit and it matched byte for byte. +- **Readers set aside**: none. Every reader's output has `is_error` false, subtype `success`, + non-empty answer text and the cohort's model; each sheet is that output's `result`. +- **Marked by**: a delegated marker, a fresh `dbf-spec-editor` agent (Claude Opus) given the + marking instruction and its agent definition, having seen neither the drafting nor the review. + +## Marks + +Readers are numbered 1 to 5 as their sheets are. Question 3 columns: the eleven clauses, then +the two answers (S8 conformance under section 8, 6.4 whether clause 6.4 changes the verdict). + +| Reader | Q1 | 2a | 2b | 2c | 4.1 | 4.2 | 4.3 | 5.1 | 5.2 | 5.3 | 5.4 | 6.1 | 6.2 | 6.3 | 7.1 | S8 | 6.4 | +| ------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 1 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 2 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 3 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 4 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | +| 5 | correct | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | ok | + +Question 1: readers 1, 2 and 4 name, one bullet each, the key's lines for 4.1; 4.2; 4.3; 5.1 to +5.4; 6.1 to 6.3; and 7.1. Reader 3 names the same six, its sixth bullet adding what conformance +means, which the key says is neither correct nor wrong. Reader 5's first bullet states what +conformance means (neither correct nor wrong); its others name 4.1, 4.2, 4.3 (as "every rule has +a stable identifier", which the key says names 4.3), 5.1 to 5.4, and, in its sixth bullet, both +6.1 to 6.3 and 7.1 (point 2 below). No reader names the 8.2 line, and no bullet asserts anything +the key marks wrong: none says the detector must be of one kind or must state its own kind. + +Question 2: every reader states every required element of each item and quotes the document. +2a: all five state that conformance is every MUST in sections 4 to 7 and that a detector +predating the document can conform; readers 1, 2, 3 and 5 quote section 8's definition, and all +five quote clause 8.1's "A Detector that predates this document, or whose maintainer has never +read it, MAY be graded Conforming on evidence". 2b: all five name both things, disableable by +configuration or detectable, and say either alone is enough, quoting clause 7.1's "For each one +it MUST do at least one of two things, and either alone satisfies this clause" (readers 1, 3 and +4 with an ellipsis or ending in one; the run before it is well over six words). Readers 2 and 5 +add that a written reason is a SHOULD under 7.2, which is not a wrong assertion. 2c: all five +state that the documentation ships with the rule in the installed copy, at a version tracked +together, and resolves without network access, quoting the 6.2 and 6.3 headings (reader 2 also +6.3's first sentence, reader 3 also 6.5's sentence on the installed file); reader 1 adds that a +website does not satisfy it, which is not among the key's wrong assertions. + +Question 3: every reader gives the accepted grade on all eleven clauses, several with caveats on +what the fixture does not say (what `--only` prints under 4.2, machine-readable formats under +4.3), which the key accepts since no row carries a condition. Reader 2's 4.2 is Yes with the +remark that it would be Partial if `--only` printed nothing; the grade given is Yes, and the key +says a caveat on what the fixture does not say leaves the accepted grade correct. S8: all five +say Lintomatic does not conform, naming 6.3 or the two undocumented bundled rules; all five say +the verdict cannot name Lintomatic's kind because the fixture does not say it, and that the +conformance finding stands, which the key does not mark. None makes the verdict turn on the +kind. 6.4: all five say it does not change the verdict, because it is a SHOULD. + +### Misses + +None. No answer matches a wrong reading the key names for question 1 or question 3. + +## Question 4: passages named + +Reader 1 wrote "none". Paragraphs as `ACCEPTANCE.md` defines them; "this change's text" means a +paragraph the diff d67174c..23f5bf0 adds, rewords, moves, deletes a sentence from, or changes a +word, term or link in. The diff 9e914fe..23f5bf0 shows this change and the merged detector +change together. + +| # | Passage (clause, words or description) | Readers | This change's text | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------ | +| 1 | Clause 6.4, **Why** paragraph: "Where the [Detector] is shipped inside a [Toolchain], the [toolchain specification]'s self-audit makes this check a MUST for the [Toolchain]; a [Detector] released on its own is asked for it as a SHOULD because ..."; the "because" reads as a reason for a MUST as well as for a SHOULD | 2 | no | +| 2 | Clause 7.1, **Why** paragraph: "is an [Owner] decision under its clause 3.4 and section 4"; Owner is defined in the unsupplied method specification (point 1 below) | 3 | no | +| 3 | Clause 4.1, **Why** paragraph: "the [Classes] that matter most to a project"; Class is defined in the unsupplied method specification (point 1 below) | 3 | no | +| 4 | Clause 7.1, second paragraph ("A route is detectable when ..."): "That way is open only where the [Detector]'s [Rules] can see the route: a [Detector] whose [Rules] never see comments cannot make an ignore comment detectable this way.", read with "the [Detector] ships no such [Rule] itself, and need not"; how the detector's MUST is discharged when the project does the work | 4 | no | +| 5 | Clause 4.3, first paragraph: "A prefix the [Detector] adds from the invoking directory or the [Rule]'s location is derived from the file path. A namespace the [Rule author] assigns once in configuration is not."; whether a detector-added prefix not derived from location is allowed | 4 | no | +| 6 | Clause 6.1, heading: "The [Detector] MUST provide a mechanism that resolves a printed identifier to its documentation"; whether the MUST applies to project rules at all | 5 | no | +| 7 | Clause 6.1, paragraph "For a project's own Rule": "what it owes is the half it can give: the [Identifier] printed unaltered under clause 4.3"; set against passage 6 | 5 | no | + +Notes, not passages: reader 2's remark that the fixture does not say whether Lintomatic sits +inside a toolchain, so it read it as standalone. + +Passages 2 and 4 are both in clause 7.1 but in different paragraphs (the **Why** paragraph and +the second paragraph), so they are not the same passage; passages 6 and 7 are both reader 5's. + +## Findings and where they lie + +None. No passage is named by two or more readers, so nothing blocks and nothing is routed to +`acceptance/ROUTED.md` from this run. Passages 1 to 7 are each one reader's and are recorded +only; none lies in text this change wrote or altered (clauses 6.1 and 7.1 are the detector +change's text at d67174c, and clauses 4.1, 4.3 and 6.4 are unchanged by d67174c..23f5bf0). +No reader named clause 8.2, section 9 or section 2's kinds row. + +## Points the rules did not settle + +1. **Reader 3's question 4: passages or notes.** Each item's subject is a term defined only in + the method specification, which was not supplied, which reads as a note; but each quotes six + or more consecutive words of the document with a clause number. Counted, by the open-point + rule (an open point on whether a reader named a passage counts the passage), as naming + clause 7.1's **Why** paragraph and clause 4.1's **Why** paragraph. No finding turns on this + point: no other reader names either paragraph. +2. **Reader 5's sixth question 1 bullet names two lines of the key's list** (6.1 to 6.3, and + 7.1). The rules say how bullets naming one line count, not how one bullet naming two lines + counts. Marked for the reader, by the open-point rule; no mark turns on it, since reader 5's + bullets name at least four lines of the key without it (4.1, 4.2, 4.3, 5.1 to 5.4). + +## Process change + +The run was made under the acceptance process change of pull request 22 before it merged, +following all of it at commit 13b497b, named before the readers started. `ACCEPTANCE.md`, +`acceptance/README.md`, `acceptance/runs/README.md` and `CONTRIBUTING.md` are identical at +13b497b and 23f5bf0. + +## Against the criteria + +- Load-bearing statements: five of five on 2a, 2b and 2c. Met. +- Restatement and application: five of five on question 1 and on every question 3 item. Met. +- A miss the key says counts against the document: none. Met. +- Passages named by two or more readers in text the change wrote or altered: none. Met. +- Passages named by two or more readers elsewhere: none. + +## Findings applied + +None at the time of marking. + +## Verdict + +PASS. Every reader is correct on every keyed item, no miss counts against the document, and no +passage is named by two or more readers. + +## Answer sheets + +Posted on the pull request: [answer sheets](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125344415); [command, models, marker instruction and agent definition](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125343899); [the prompt](https://github.com/Defence-Before-Fix/defence-before-fix.github.io/pull/14#issuecomment-6125344785). diff --git a/acceptance/runs/README.md b/acceptance/runs/README.md index 4994f74..3f7ad64 100644 --- a/acceptance/runs/README.md +++ b/acceptance/runs/README.md @@ -1,15 +1,22 @@ # Acceptance run records -One file per cohort run, `NNN-.md`, numbered in the one sequence the three documents -share. Runs 1 to 15 predate this directory and are summarised in the changelog entries for -1.0.0 and 1.0.1 of the method specification, 1.0.0 of the detector specification and 0.2.0 of -the toolchain specification; numbering here continues from 16. +One file per cohort run, `NNN--run-.md`, numbered in the one sequence the three +documents share, with `` the run number within its pull request. Runs 1 to 15 predate this +directory and are summarised in the changelog entries for 1.0.0 and 1.0.1 of the method +specification, 1.0.0 of the detector specification and 0.2.0 of the toolchain specification; +numbering here continues from 16. -A record states, in this order: the document; the pull request and the commit the readers were -given; the cohort size and model; the run number within that pull request; the marks per reader -per question against the key; the passages quoted as confusing and by how many readers; the -findings applied to the text as a result; and the verdict against the criteria in -`ACCEPTANCE.md`. Answer sheets are not pasted in full; the marks are the record. +A record states what `ACCEPTANCE.md` lists under "The record", in this order: the document; the +pull request and the commit the readers were given; the cohort size and model, with the model +version each reader reported; how the readers were isolated and the command they were started +with; any reader set aside and why; who marked and, for a delegated marker, how it was started; +the marks per reader per keyed item, with each miss and the reader's reasoning in brief; every +passage named under question 4, with the words quoted or the description given, its clause and +the readers naming it; each finding and whether it lies in text the change wrote or altered; any +point the rules did not settle; whether the run was made under a process change before it merged +and, if so, at which commit; the findings applied afterwards; the verdict; and a link to the +answer sheets posted on the pull request. Earlier records show the style only; where one differs +from this order, this file wins. Answer sheets are not kept here; the marks are the record. The `Unreleased` changelog line for the change names the record, and `tools/changes.py` refuses a specification change whose pull request adds no file here. diff --git a/acceptance/toolchain/KEY.md b/acceptance/toolchain/KEY.md index 5a2602b..e098c90 100644 --- a/acceptance/toolchain/KEY.md +++ b/acceptance/toolchain/KEY.md @@ -1,13 +1,14 @@ # Answer key for TOOLING-SPEC.md -Marked against the document at the version named in the changelog entry that records the run. -Quotations below are from the document; when a clause is reworded, update the quotation here in -the same commit. +Marked against the document at the commit the readers read, with this key, its question set and +fixture at that same commit. Quotations below are from the document; when a clause is reworded, +update the quotation here in the same commit. `ACCEPTANCE.md` says how each item is marked; +`acceptance/README.md` says what each part of this key is for. ## 1. Restatement -A bullet is correct when it names one of these obligations. Four or more correct bullets, and no -bullet asserting an obligation the document does not state, is a correct restatement. +A bullet is correct when it names one of these obligations. Four or more correct bullets, with +no bullet making an assertion listed below as wrong, is a correct restatement. - Route defences only through detectors that conform to the detector specification, wrapping permitted ([4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification)). @@ -23,63 +24,91 @@ bullet asserting an obligation the document does not state, is a correct restate same way as the defences ([6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) to [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences)). - A shipped toolchain also fails its release on undocumented bundled defences and runs its own defences on its own source ([8.1](../../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation), [8.2](../../TOOLING-SPEC.md#82-a-shipped-toolchain-must-run-its-own-bundled-defences-on-its-own-source)). +- Make everything the method requires of a practitioner possible without the project building + the mechanism first (section [1](../../TOOLING-SPEC.md#1-what-this-document-is-for), the governing principle). +- Not describe partial conformance as conformance (section [9](../../TOOLING-SPEC.md#9-conformance)). Wrong if it asserts: the toolchain must be a single product; a declaration is required to -conform; the agent summary (section [7](../../TOOLING-SPEC.md#7-agent-context)) is a MUST. +conform; the agent summary (section [7](../../TOOLING-SPEC.md#7-agent-context)) is a MUST. A bullet that calls agent support or the +self-audit a further condition "where it applies", says that section [7](../../TOOLING-SPEC.md#7-agent-context) is recommended or a +SHOULD, or says a declaration is not a condition, asserts no obligation and is neither correct +nor wrong. So is a bullet that states how +conformance is measured rather than what is required: the toolchain judged as one assembled +whole at the project level (sections [1](../../TOOLING-SPEC.md#1-what-this-document-is-for) and [9](../../TOOLING-SPEC.md#9-conformance)), the project-level verdict a single grade +(section [9](../../TOOLING-SPEC.md#9-conformance)), or a shipped artefact graded separately ([9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately)). + +Wrong readings the text invites, which count against the document on one reader: none named. ## 2. Load-bearing statements -- **2a.** At the project level; the toolchain is whatever the project has assembled from - third-party, first-party and project-level parts. Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "A project's - Toolchain Conforms if every MUST in sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record) holds across the assembled parts, wherever - each part came from". An answer that says conformance is graded per tool, or that only a - packaged product can conform, is wrong. -- **2b.** One grade; which part satisfied each clause is evidence, not a grade of its own. Key - quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "The project-level verdict is a single grade; which part of the Toolchain - satisfied each clause is evidence for that grade, not a second grade." -- **2c.** Two, graded and declared separately, and neither implies the other. Key quotation, - [9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately): "The two verdicts MUST be graded and declared separately, and neither implies the other." - An answer giving one grade, or saying the artefact grade follows from the project grade, is - wrong. +- **2a.** Required: (1) at the project level; (2) the toolchain is whatever the project has + assembled to run its checks through, from third-party, first-party and project-level parts. + Wrong if it says conformance is graded per tool, or that only a packaged product can conform. + Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): "A project's Toolchain Conforms if every MUST in sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record) + holds across the assembled parts, wherever each part came from". An answer that says everything the + project runs its checks through, or the assembled parts from wherever they came, or that lists + the three kinds of part, states element (2). +- **2b.** Required: (1) one grade; (2) which part satisfied each clause is evidence for that + grade, not a grade of its own. Wrong if it gives a grade per part. Key quotation, section [9](../../TOOLING-SPEC.md#9-conformance): + "The project-level verdict is a single grade; which part of the Toolchain satisfied each + clause is evidence for that grade, not a second grade." +- **2c.** Required: (1) two; (2) neither implies the other. Wrong if it gives one grade, or says + the artefact grade follows from the project grade or the reverse. That the two are graded and + declared separately is explanation; an answer that says it has stated element (1). Key quotation, [9.1](../../TOOLING-SPEC.md#91-a-project-that-ships-a-detector-or-a-toolchain-has-two-levels-of-conformance-graded-separately): "The two verdicts MUST be graded and declared separately, and + neither implies the other." ## 3. Application: the Zigpipe project -| Clause | Intended verdict | Reasoning the fixture supports | -| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | Lintomatic fails detector [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) on two undocumented bundled rules and nothing in the wrapping closes that gap | -| [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)` | -| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | Ignore comments disabled project-wide; exceptions only through the record the toolchain reads | -| [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | `zig-qa run` is local, prints Lintomatic's output unchanged, one line per finding | -| [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | Lintomatic, then the tests, stopping on a Lintomatic failure | -| [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes, Partial or No | `zig-qa list` prints identifier, summary and documentation path without running the rules; Partial or No is accepted where the reasoning counts the two `(none)` routes, since 5.1 says the listing MUST provide the route and [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) point 2 says a gap may be named again by a later clause | -| [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | Derived from `lintomatic.toml`, not hand-maintained | -| [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | Yes | The `proj.*` rules appear in the same listing | -| [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes | `quality-exceptions.toml` is a defined path that `zig-qa run` loads | -| [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Partial or No | A justification is required and an empty one rejected, but nothing rejects a justification that could be pasted onto any entry | -| [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | `zig-qa list --exceptions` uses the same format and operation as the rule listing | - -Either Partial or No is accepted for [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one); the reasoning must mention the generic-justification -requirement. A Yes on [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) with reasoning that quotes only the first sentence of the clause is a -finding against the document. - -Section [9](../../TOOLING-SPEC.md#9-conformance) verdict: **does not conform**, because [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) and [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) fail; partial conformance MUST NOT be -described as conformance. **Not with agent support**, because section [7](../../TOOLING-SPEC.md#7-agent-context) is not met (no summary). -**Section [8](../../TOOLING-SPEC.md#8-self-audit) does not apply**, because Zigpipe ships nothing to other projects; a reader who applies -[8.1](../../TOOLING-SPEC.md#81-a-shipped-toolchain-must-fail-its-own-release-if-a-bundled-defence-lacks-resolvable-documentation) to Zigpipe on the basis that Zigpipe's tooling includes a shipped package (zig-qa) has read -"where the Toolchain is one the project ships" (section [9](../../TOOLING-SPEC.md#9-conformance)) as reaching the consuming project, and that counts against the document. The -correct reading is that section [8](../../TOOLING-SPEC.md#8-self-audit) bears on zig-qa's own artefact grade, not on Zigpipe. - -Declaration: **does not affect the verdict**. Key quotation, [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance): "The declaration is optional." -A reader who says Zigpipe cannot be graded without one has read [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) wrongly. - -A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes on the grounds that Lintomatic "mostly" conforms has read either [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) -or the detector document's section [8](../../DETECTOR-SPEC.md#8-conformance) differently; it counts against this document only if the -reasoning quotes [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification). A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes because they missed the two undocumented rules -has misread the fixture. +| Clause | Accepted grades | Condition on the reasoning | What the fixture supports | +| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) | No | None | Lintomatic fails detector [6.3](../../DETECTOR-SPEC.md#63-a-bundled-rules-documentation-must-ship-with-the-rule-at-a-version-tracked-together) on two undocumented bundled rules and nothing in the wrapping closes that gap; a reader who notes that the detector specification was not supplied and grades from this document's summary of it is correct | +| [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) | No | None. A No given on the condition that the two bundled rules are blocking is a No: their firing stops `zig-qa run`, so they are | `zq.*` and `proj.*` resolve offline, but the two enabled bundled rules resolve to `(none)` | +| [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record) | Yes | None | The fixture names one suppression route of Lintomatic's, the inline ignore comment, disabled project-wide, with exceptions only through the record the toolchain reads; a grade given on the condition that the comment is Lintomatic's only route is the grade for this case | +| [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes) | Yes | None | `zig-qa run` is local, prints Lintomatic's output unchanged, one line per finding | +| [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure) | Yes | None | Lintomatic, then the tests, stopping on a Lintomatic failure | +| [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) | Yes, Partial or No | None | `zig-qa list` prints identifier, summary and documentation path without running the rules; Partial or No is the reading that counts the two `(none)` routes, since [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them) says the listing MUST provide the route and [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) point 2 says a gap may be named again by a later clause | +| [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration) | Yes | None | Derived from `lintomatic.toml`, not hand-maintained | +| [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones) | Yes | None | The `proj.*` rules appear in the same listing | +| [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself) | Yes or Partial | For Partial: mentions that the fixture names a place for exceptions and none for calibrations or conventions | `quality-exceptions.toml` is a defined path that `zig-qa run` loads; the project record entry also covers calibrations and conventions | +| [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) | Partial or No | Mentions that nothing rejects a generic justification, one that could be pasted onto any entry | A justification is required and an empty one rejected | +| [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences) | Yes | None | `zig-qa list --exceptions` uses the same format and operation as the rule listing | + +The four answers the question then asks for, each with the condition its reason must meet: + +- **Conformance under section [9](../../TOOLING-SPEC.md#9-conformance).** Accepted answer: does not conform. Condition: the reason + names [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access) or [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) among the failing clauses, or points to the grades the reader gave above, as "the + clauses graded No", where those grades have one of the three at No or Partial; other clauses + named alongside are not marked. +- **Conformance with agent support.** Accepted answer: no. Condition: the reason says that + section [7](../../TOOLING-SPEC.md#7-agent-context) is not met, or that there is no summary, or that the toolchain does not conform + under section [9](../../TOOLING-SPEC.md#9-conformance) at all. +- **Whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies.** Accepted answer: it does not. Condition: the reason says that + Zigpipe ships nothing to other projects, or that section [8](../../TOOLING-SPEC.md#8-self-audit) bears on zig-qa's own artefact + grade and not on Zigpipe. +- **Whether the absence of a declaration affects the verdict.** Accepted answer: it does not. + Condition: the reason says that the declaration is optional, or not a condition of + conformance, or that a verdict rests on the evidence. Key quotation, [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance): "The declaration is + optional." + +Wrong readings the text invites, which count against the document on one reader: + +- [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one) graded Yes with reasoning that says a justification is required or an empty one rejected + and does not mention a generic justification, which has not read the sentence on one. +- Section [8](../../TOOLING-SPEC.md#8-self-audit) said to apply to Zigpipe with reasoning that says its tooling includes a shipped + package, which has read "where the Toolchain is one the project ships" (section [9](../../TOOLING-SPEC.md#9-conformance)) as + reaching the consuming project. +- [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) graded Yes with reasoning that says Lintomatic mostly or largely conforms and quotes [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), + which has read it as admitting a proportion. +- The declaration said to affect the verdict, with reasoning that Zigpipe cannot be graded without + one, which has read [9.2](../../TOOLING-SPEC.md#92-the-declaration-is-the-claim-and-the-known-gap-record-not-a-condition-of-conformance) as a condition. + +A reader who grades [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification) Yes because they missed the two undocumented rules has misread the +fixture, which is the reader's miss. ## 4. Confusion -No keyed answer. Record every passage quoted. Two or more readers quoting the same passage is a -finding that blocks until the text is changed and a fresh cohort no longer quotes it. What to -change is the editor's decision; the reader is not asked and any suggestion it volunteers is -discarded unread. +No keyed answer. Record every passage named, as `ACCEPTANCE.md` defines naming and the same +passage. A passage named by two or more readers of the run is a finding: it blocks the change +where it lies in text the change wrote or altered, and is routed to `acceptance/ROUTED.md` +otherwise. What to change is the editor's decision; the reader is not asked and any suggestion +it volunteers is discarded unread. diff --git a/acceptance/toolchain/QUESTIONS.md b/acceptance/toolchain/QUESTIONS.md index a4da266..d4e5cd7 100644 --- a/acceptance/toolchain/QUESTIONS.md +++ b/acceptance/toolchain/QUESTIONS.md @@ -11,7 +11,8 @@ toolchain? ## 2. Load-bearing statements -Answer each in one sentence, followed by the sentence of the document you are relying on, quoted. +Answer each part of each question in a sentence or two, then quote the sentence or sentences of +the document you are relying on. - 2a. At what level is toolchain conformance measured, and what is the toolchain? - 2b. A project's tooling is assembled from a third-party detector, a first-party package and @@ -25,8 +26,8 @@ Answer each in one sentence, followed by the sentence of the document you are re Grade the Zigpipe project's toolchain, as described in the fixture, against every MUST clause in sections [4](../../TOOLING-SPEC.md#4-the-detectors-a-toolchain-routes-defences-through) to [6](../../TOOLING-SPEC.md#6-the-project-record): [4.1](../../TOOLING-SPEC.md#41-every-detector-the-toolchain-routes-a-defence-through-must-conform-to-the-detector-specification), [4.2](../../TOOLING-SPEC.md#42-the-toolchain-must-resolve-every-identifier-a-defence-it-routes-can-print-from-the-installed-copy-without-network-access), [4.3](../../TOOLING-SPEC.md#43-the-toolchain-must-forbid-through-a-defence-of-its-own-every-suppression-route-that-bypasses-the-project-record), [4.4](../../TOOLING-SPEC.md#44-the-toolchains-own-invocation-must-satisfy-the-detector-specifications-reporting-clauses-for-every-defence-it-routes), [4.5](../../TOOLING-SPEC.md#45-the-toolchains-entry-point-must-run-detectors-before-runners-and-must-stop-on-a-detector-failure), [5.1](../../TOOLING-SPEC.md#51-the-toolchain-must-be-able-to-list-the-defences-active-in-a-project-without-triggering-them), [5.2](../../TOOLING-SPEC.md#52-the-listing-must-be-derived-from-the-active-configuration), [5.3](../../TOOLING-SPEC.md#53-a-projects-own-defences-must-appear-in-the-listing-alongside-bundled-ones), [6.1](../../TOOLING-SPEC.md#61-the-toolchain-must-define-a-location-for-the-project-record-and-must-read-it-itself), [6.2](../../TOOLING-SPEC.md#62-every-exception-in-the-project-record-must-carry-a-written-justification-that-names-the-hazard-and-the-scope-and-the-toolchain-must-reject-a-generic-one), [6.3](../../TOOLING-SPEC.md#63-the-project-record-must-be-enumerable-by-the-same-means-as-the-defences). For each give Yes, Partial or No and one line of reasoning. Then state whether the toolchain conforms under section -9, whether it conforms with agent support, and whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies to it. Finally, state -whether the absence of a declaration affects the verdict. +[9](../../TOOLING-SPEC.md#9-conformance) and why, whether it conforms with agent support and why, and whether section [8](../../TOOLING-SPEC.md#8-self-audit) applies to it +and why. Finally, state whether the absence of a declaration affects the verdict, and why. ## 4. Confusion diff --git a/defence-before-fix-project-prompt.md b/defence-before-fix-project-prompt.md index ab7f4ce..f34cbe1 100644 --- a/defence-before-fix-project-prompt.md +++ b/defence-before-fix-project-prompt.md @@ -13,8 +13,18 @@ it. Do not decide in advance whether that is possible, attempt it. If you cannot for it, say so and fix the Defect conventionally. Write a custom Rule that detects the Class, in a tool that reads code rather than running it. -Draw it so that it catches more than the single Instance you started from, but never so broadly -that it matches code which does not carry the Hazard. +Put it in a Semantic Detector, one in which a Rule can ask the type of any expression and +what any name refers to, across the whole program; failing that, one that sees syntax; and only +then one that sees text. Use a weaker one only where the stronger one cannot express the Rule, +or the language has none that takes custom Rules: none the project runs, in any configuration +it can use without a hosted service or a licence tier, and none its ecosystem offers without +one either. Name the Detector and its kind in the Rule's documentation. Where you use a weaker one, +write down which of those two grounds holds, and why, naming the stronger Detectors you +considered, there and in the description of the commit that adds the Rule, moves it to another +Detector, or first records the grounds, and where the project requires its Owner's sign-off +on a weaker one, get it and record it with the grounds. Draw +the Rule so that it catches more than the single Instance you started from, but never so +broadly that it matches code which does not carry the Hazard. Do not trust the Rule as your only way of finding Instances. Search independently as well, by text search and by reading the code, and check the Rule catches what you found by hand. Make that diff --git a/index.md b/index.md index 39ce92d..354eab0 100644 --- a/index.md +++ b/index.md @@ -22,8 +22,8 @@ version. 1. **Attribute the defect to a class.** Not "what went wrong here" but "what kind of thing is this an instance of", within a lower bound and an upper bound that are both checkable. -2. **Build the net.** A rule that reads code, in whatever detector the toolchain offers, bespoke - where nothing off the shelf will take it. +2. **Build the net.** A rule that reads code, in the strongest kind of detector that can express + it, bespoke where nothing off the shelf will take it. 3. **Prove the net by making the rule fire.** Red before green, on the originating instance, committed before the fix. 4. **Sweep the codebase, then fix every instance.** The count is the finding; the fix addresses diff --git a/package.json b/package.json index 740ca4d..62845c6 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@longtermsupport/defence-before-fix", - "version": "1.1.0", + "version": "1.2.0-dev", "description": "Defence Before Fix: the method and toolchain specifications. Documentation only.", "license": "CC-BY-4.0", "author": "Joseph Edmonds (https://ltscommerce.dev)", diff --git a/tools/mypy.md b/tools/mypy.md index 0170163..713c2bf 100644 --- a/tools/mypy.md +++ b/tools/mypy.md @@ -43,4 +43,4 @@ Clause [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-writ ## Notes for a practitioner -Treat mypy as a detector for type-shaped classes only: where the class is "this value can be None here", tightening the configuration is a legitimate off-the-shelf rule under method clause [3.2](../SPEC.md#32-build-the-net), proven by watching the count go red. For any other class, host the defence in Pylint or a Flake8 plugin instead, and enable `warn_unused_ignores` plus a check in a second detector on new `# type: ignore` comments so the inline route stays visible. +Treat mypy as a detector for type-shaped classes: where the class is "this value can be None here", tightening the configuration is a legitimate off-the-shelf rule under method clause [3.2](../SPEC.md#32-build-the-net), proven by watching the count go red. For any other class, a mypy plugin, with `check_untyped_defs` enabled, can host a rule its hooks reach, and method clause [3.2](../SPEC.md#32-build-the-net) asks for that semantic host before a weaker kind; a rule the hooks cannot reach goes to a weaker detector, with the reason the clause requires recorded. Enable `warn_unused_ignores` plus a check in a second detector on new `# type: ignore` comments so the inline route stays visible. diff --git a/tools/pyright.md b/tools/pyright.md index fed4533..5685db3 100644 --- a/tools/pyright.md +++ b/tools/pyright.md @@ -43,4 +43,4 @@ Clause [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-writ ## Notes for a practitioner -Pyright can carry an off-the-shelf defence where the class is a type hazard one of its named rules already detects, and raising that rule to `error` in `pyrightconfig.json` is a recorded, proven change. It cannot host anything bespoke, so keep a second detector such as Pylint for the project's own classes, set `enableTypeIgnoreComments` to false and `reportUnnecessaryTypeIgnoreComment` to error, and have the second detector flag `# pyright: ignore` since Pyright will not. +Pyright can carry an off-the-shelf defence where the class is a type hazard one of its named rules already detects, and raising that rule to `error` in `pyrightconfig.json` is a recorded, proven change. It cannot host anything bespoke, so keep a second detector for the project's own classes, chosen as method clause [3.2](../SPEC.md#32-build-the-net) requires (a mypy plugin where its hooks can express the rule, a weaker kind such as Pylint only with the reason recorded), set `enableTypeIgnoreComments` to false and `reportUnnecessaryTypeIgnoreComment` to error, and have the second detector flag `# pyright: ignore` since Pyright will not. diff --git a/tools/ruff.md b/tools/ruff.md index f875bbb..b011ac1 100644 --- a/tools/ruff.md +++ b/tools/ruff.md @@ -43,4 +43,4 @@ Clause [4.1](../DETECTOR-SPEC.md#41-the-detector-must-support-bespoke-rules-writ ## Notes for a practitioner -Ruff cannot host a bespoke defence, so use it for the industry's known hazards and pair it with a second detector that accepts project rules, such as Pylint or a Flake8 plugin, for the class that the method turns up. Enable `PGH004` and `RUF100`, and forbid the specific `# noqa: CODE` form in the configuration of that second detector or in review policy, since Ruff can neither switch it off nor see it. +Ruff cannot host a bespoke defence, so use it for the industry's known hazards and pair it with a second detector that accepts project rules for the class that the method turns up, chosen as method clause [3.2](../SPEC.md#32-build-the-net) requires: a mypy plugin where its hooks can express the rule, and a weaker kind such as Pylint or a Flake8 plugin only with the reason recorded. Enable `PGH004` and `RUF100`, and forbid the specific `# noqa: CODE` form in the configuration of that second detector or in review policy, since Ruff can neither switch it off nor see it. diff --git a/tools/semgrep.md b/tools/semgrep.md index 30de714..ec54f37 100644 --- a/tools/semgrep.md +++ b/tools/semgrep.md @@ -12,7 +12,7 @@ checked: 2026-09-08, version 1.176.0 {% include tool-grades.html %} -Semgrep is a pattern-based static analyser whose rules are written in YAML using the syntax of the target language itself, across some thirty languages. The open-source command line tool runs locally; the Semgrep AppSec Platform adds a hosted registry, Pro rules, cross-file analysis and triage. In a pipeline it is the usual choice for a project-specific rule that a language linter cannot express cheaply, and its rule-testing harness is one of the best in the field, which makes it a natural detector for the method. +Semgrep is a pattern-based static analyser whose rules are written in YAML using the syntax of the target language itself, across some thirty languages. The open-source command line tool runs locally; the Semgrep AppSec Platform adds a hosted registry, Pro rules, cross-file analysis and triage. In a pipeline it can host a project-specific rule that a language linter cannot express, where method clause [3.2](../SPEC.md#32-build-the-net), which asks for the strongest kind of detector that can express the rule, allows it, and its rule-testing harness is one of the best in the field, which makes it a natural detector for the method. ## How it is conformant