Skip to content

Acceptance process: the test proves the clarity of the change, with keys that ask what the question asks - #22

Open
LTSCommerce wants to merge 19 commits into
nextfrom
editor/acceptance-coherent
Open

LTSCommerce wants to merge 19 commits into
nextfrom
editor/acceptance-coherent

Conversation

@LTSCommerce

@LTSCommerce LTSCommerce commented Oct 12, 2026 •

Copy link
Copy Markdown
Member

What this changes

The acceptance process, so that the cohort test proves what it exists to prove, the clarity of the document to a cold reader, and nothing else. ACCEPTANCE.md is rewritten around five fixes, with CONTRIBUTING.md, acceptance/README.md, acceptance/runs/README.md, the three question sets and the three keys brought into line, and two new files, acceptance/CHANGES.md (the record of process changes) and acceptance/ROUTED.md (findings routed out of a run to a change of their own).

  1. Keys ask what the question asks. Each key lists, per load-bearing item, the numbered elements an answer must state, one for each part of the question, and the assertions that make an answer wrong; everything else in the key is explanation and never marks an answer down. The questions are reworded to ask for exactly those elements (question 2's instruction asks for "the sentence or sentences"; detector 2b asks whether one of the two things is enough; detector 2c asks at what version; the question 3 verdict items ask "and why"). Detector 2b no longer requires "need not remove or forbid", which the question never asked and 28 of 30 readers were marked down for.
  2. A confusion finding is settled run by run. A passage named by two or more readers of one run is a finding; the next run is judged on its own readers, and a passage fewer than two of them name is clear. Nothing is carried from run to run except the record.
  3. A change is judged on what it changed. A finding in a paragraph the pull request added, reworded, moved, deleted from or changed a term or link in (read against its merge base), or that a reader sets against such a paragraph, blocks the change. A finding anywhere else is routed to acceptance/ROUTED.md and a change of its own, and does not block. The keyed questions still cover the whole document on every run, so the document is still proved; a keyed miss blocks wherever its text lies.
  4. Marking judgements are written into the rules. The Marking section defines a keyed item, quoting (a six-word verbatim run), reading for sense, question 1 counting, a correct question 2 answer, a question 3 grade with its condition, a wrong reading the text invites (stated in the key as a grade plus what the reasoning says or omits, blocking on one reader), an unanswered item, naming a passage, the same passage, where a finding lies, and the direction every open point takes. A run record states no judgement the rules do not; an open point is settled in the rules or the key before that pull request's next run.
  5. The "traceable miss" row goes. Any miss whose reasoning quoted the document was a finding whatever the count, which made the all-but-one tolerance meaningless. Now a single miss under questions 1 and 3 counts against the document only where the key names that reading; everything else is the reader's, and a second miss on the same item fails the threshold without a judgement.

Kept from pull request #13, which this supersedes: the odd cohort of at least five, the isolated reader process and its example command, the packet form, set-aside readers and void runs, the sheets posted on the pull request, the fuller run record, the route for a process change and acceptance/CHANGES.md.

The observed case

Thirty-three runs on two changes (19 to 30 on pull request #14; 31 to 33 and 35 to 52 on the defined-terms change) in which every keyed answer reached five of five and the runs still failed: on key elements the questions never asked for, on passages the change never touched, on flags that one later reader kept open, and on judgements J1 to J12 that the markers invented and later runs cited as precedent. Runs 050 to 052 got worse after a sixth rewording, not better. The run records on those branches are the evidence.

Kind of change

  • Clarity edit
  • Obligation

Neither: a change to the acceptance process, recorded in acceptance/CHANGES.md and not tested by a cohort, as ACCEPTANCE.md says under "Changes to this process". tools/changes.py --base origin/next and spec-qa.py are clean; the unit tests pass.

Where it is looser than the rules on next, and why

Two rows are narrower than next's, and the pull request says so rather than hiding it. The traceable-miss row (point 5) measured whether a reader had quoted the document, not whether the text was unclear. The confusion row (point 3) measured the whole document on every change, so that a one-line rename was reviewed sixty paragraphs at a time by five readers and could not converge. Neither change alters a mark already given; every run under the new rules is a fresh run. ACCEPTANCE.md under "Changing the test" now states the test for touching a rule: a rule that measured something other than whether a cold reader could follow the text is corrected, with what it measured recorded; a rule that measured that is left alone and the text is fixed.

Review

Three independent reviews by a reviewer that did not draft the change, each applied in its own commit (review 1: 19 findings; review 2: 20; review 3: 9, none blocking). acceptance/CHANGES.md carries the line for this change; its pull request number is filled in by the next commit.

Not done here, by design

No cohort run: a process change is not tested by one. Merging is the editor's. The two held changes and the two changes routed out of their runs were rerun under these rules at a commit named in each record, as "Changes to this process" allows, and each passed: pull request #14 (runs 70 and 79), #24 (runs 64, 80 and 87), #23 (run 78) and #25 (run 86).

Settled during those runs

Open points the run records raised were settled here before the next run of the pull request that raised them, as "Changing the test" requires, each in its own commit: a remark that another document was not supplied, or that a term is defined only there, is a note however much of the document it quotes to locate the reference; a paragraph the pull request parts from what it refers to, by inserting a paragraph between them or moving either, is the change's text; the toolchain key's 4.3 row and the detector key's 4.2 row say which case the fixture presents, so that a grade given on a stated condition is the grade for that case; the Lintomatic fixture says check --only prints its findings as check does; and acceptance/ROUTED.md carries every finding the runs made in text no change had touched.

🤖 Generated with Claude Code

LTSCommerce and others added 19 commits October 12, 2026 11:34
…eys that ask what the question asks

Keys list the required elements per part of each question, questions ask for exactly
those, a confusion finding is settled run by run and blocks only in text the change
wrote or altered (others go to acceptance/ROUTED.md), single misses count against the
document only where the key names the reading, and open marking points are settled in
the rules before the next run rather than invented per run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
One route for each kind of change to the test (quotation and new item in the specification
pull request, everything else a process change); ROUTED.md and runs/ are records, not process
files; question 1 defined once; open points marked in a stated direction; quoting needs a
six-word run; a finding lies in the change's text when any sentence of its paragraph changed;
wrong readings stated as grade plus reasoning; keys ask only what their questions ask (detector
2a, toolchain 2c); the question 3 tables separate the condition from the explanation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Question 1 defined once (first six bullets, same clause counts once, a bullet on neither list
not counted); the verdict items of question 3 ask for their reason and the keys state each
as an accepted answer with a condition; a finding lies in the change's text also where the
readers set it against that text, read against the merge base; the cohort's model is the
version the editor names; a keyed miss whose fix would change an obligation is routed and
waited for; the question set is the one list of load-bearing questions; ROUTED.md defines its
status column and links its clauses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two routes for changing the test, with a changed or removed obligation's key item following
the text in the specification pull request; an open point holds up only its own pull request;
a fresh run follows a change to the document, not to the record; detector 2b names both
things; the section 7 and section 9 reasons accept every failing clause; the restatement rule
reads plainly; ROUTED.md defines a finding as ACCEPTANCE.md does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The method key lists fixing the original defect with a reproducing test among the
obligations, says a bullet on the verdict resting on reproduction describes the reviewer, and
says that an answer to 2a naming no step does not state the steps. The toolchain key lists the
governing principle, says a bullet on how conformance is measured is neither correct nor
wrong, and accepts a section 9 reason that points to the grades given. acceptance/ROUTED.md
gains the findings runs 53 to 59 routed, and the runs that raised existing rows again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The toolchain key counts a bullet on not describing partial conformance as conformance, and
says a bullet calling section 7 recommended or a SHOULD asserts no obligation.
acceptance/ROUTED.md gains clause 3.4's "tremendous" sentence from run 61 and notes that run
on the two rows it raised again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
acceptance/ROUTED.md gains the detector's clause 6.1 paragraph on a project's own rule, from
run 63, and notes run 64 on the four method rows it raised again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ows from run 67

acceptance/ROUTED.md gains a Closed table, with the detector's clauses 7.1, 8.1, 5.3 and 6.1
rows closed by editor/detector-route-rule's run 66, notes run 66 on the clause 6.1 row on a
project's own rule, and gains the toolchain's clause 6.2 heading and documented-check
paragraph from run 67.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… run 64

acceptance/ROUTED.md gains the passages runs 19 to 30 on the semantic-detectors change named
in text that change no longer touches (the Narrowing entry, section 4's lead-in to the
owner's decisions, clause 3.1's paragraph on independent techniques and the section 3 table's
3.1 row), notes those runs on the rows they raised again, and closes the method's two-bounds
paragraph and bundled-defence paragraph by editor/defined-terms's run 64.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s 68 to 71

The method key lists the sentence behind a narrowing, and the owner's decision where it cannot
be written, among the obligations, after three readers of run 70 restated it.
acceptance/ROUTED.md gains Appendix A's closing paragraph from run 70 and the detector's
clause 5.3 from run 69, and notes runs 68, 70 and 71 on the rows they raised again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ED 5.3 row adds run 75

Run 72's open point: three readers graded clause 4.3 on the condition that the ignore comment
is Lintomatic's only suppression route, and the key did not say which case the fixture
presents. The 4.3 row now says so, and such a grade is the grade for this case.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…parted paragraph is the change's text

Settles the open points of runs 76, 77, 78 and 80. ACCEPTANCE.md says a remark that another
document was not supplied, or that a term is defined only there, is a note however much of the
document it quotes to locate the reference; and that a paragraph the pull request parts from
what it refers to, by inserting a paragraph between them or moving either, is the change's
text. The Lintomatic fixture says `check --only` prints its findings as `check` does, and the
detector key's 4.2 row says so and which case a conditional grade is for. ROUTED gains the
toolchain 6.1 Project record sentence (run 76) and the detector 4.2 sentence on a detector
failing 4.1 (run 78, taken by editor/defined-terms); the 4.3 row links the section, whose
heading the toolchain change rewrites.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… 6.2 and 6.3 speak only of exceptions

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…, from run 83

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…es 6.1, 6.2 and 6.4

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… 6.4 again

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…inst 6.2 new

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant