Skip to content

docs: plan for closing the 26 open evidence markers - #38

Merged
TheMeinerLP merged 1 commit into
mainfrom
docs/evidence-plan
Aug 2, 2026
Merged

docs: plan for closing the 26 open evidence markers#38
TheMeinerLP merged 1 commit into
mainfrom
docs/evidence-plan

Conversation

@TheMeinerLP

Copy link
Copy Markdown
Contributor

A worklist for the TODO(maintainer) markers in the wiki, written so a fresh session can execute it
without the context of the one that produced it.

Why it is phased rather than a list

The 26 markers are not one kind of problem:

Phase What How many
1 Closes with one CI benchmark run each 5
2 Arithmetic that contradicts the rows beside it — no run will fix 4
3 Two pages disagree; the source decides 2
4 Needs a -prof gc run 2
5 Unrecoverable — the correct outcome is an explicit statement 4 groups
6 Decisions for a maintainer, not for an agent 5

Phase 1 first because the evidence is merely missing and the tool now exists. Phase 2 next because
those numbers are wrong: 97 % and 63 % against published rows that give 98.0 % and 65.3 %, and a
ratio column that does not follow from its own operands. A reader who checks those finds the error,
and every other figure on the page loses credit with it.

The constraints are the interesting part

Five rules, each learned by breaking it in the round that produced this plan:

  • After changing a page, grep its name and read every hit in context. Anchored links redirect
    mechanically; the danger is the sentence without an anchor whose prose asserts what the page
    holds. Eighteen survived one split, on nine pages, four of which were on no change list. This
    failed five times in one session and no tool reports it.
  • Never publish a single-fork measurement. A one-fork run showed an ordering two forks did not
    confirm.
  • Bind every count to a commit. The benchmark class count went 14 → 16 → 17 in one day and broke
    seven sentences across three files. A count in a heading is worse — prefer removing it.
  • Never compare figures from different machines. A ratio is no more portable than an absolute
    number: 1.35× on a CI runner against 1.13× on the machine behind the published tables.
  • A checker that has never failed proves nothing. Inject bad targets first.

Note on the line numbers

Verified against the wiki at 662e4db — all fourteen spot-checked references hit their marker. They
will drift, the plan says so, and it names grep -rn 'TODO(maintainer)' as the authority instead.
Following its own rule about counts bound to a commit.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QzEdy5fN5JKGxJo8gwtNeu

The wiki carries 26 TODO(maintainer) markers. They are not one kind of problem
and should not be worked as one: a third close with a benchmark run the CI
workflow now makes cheap, a third are arithmetic that no run will fix, and the
rest are decisions or are unrecoverable and have to say so.

Ordered so effort goes where it buys the most evidence. Phase 1 is five
markers that need one workflow run each. Phase 2 is numbers that contradict
the rows printed beside them — worse than missing evidence, because a reader
who checks them finds the error and every other figure loses credit.

The global constraints are the five things this project learned by breaking
them: grep for a page name after changing it and read every hit in context;
never publish a single-fork measurement; bind every count to a commit; never
compare figures from different machines; and make a checker fail before
trusting it.

Line numbers are bound to 662e4db and will drift. The plan says so and names
the grep that is the authority instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QzEdy5fN5JKGxJo8gwtNeu
@TheMeinerLP
TheMeinerLP requested a review from a team as a code owner August 2, 2026 18:09
@TheMeinerLP
TheMeinerLP merged commit ebca9ad into main Aug 2, 2026
5 checks passed
@TheMeinerLP
TheMeinerLP deleted the docs/evidence-plan branch August 3, 2026 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant