From d7c006ae45ba12209d1d3598d3892828407015ac Mon Sep 17 00:00:00 2001 From: Tomohito YABU Date: Fri, 14 Aug 2026 22:56:38 +0900 Subject: [PATCH 1/7] =?UTF-8?q?=F0=9F=93=9D=20docs:=20=E3=83=AC=E3=83=93?= =?UTF-8?q?=E3=83=A5=E3=82=A2=E3=83=BC=E5=AE=9B=E3=81=A6=E3=82=B3=E3=83=A1?= =?UTF-8?q?=E3=83=B3=E3=83=88=E3=81=AE=E7=AF=80=E3=81=A8=E3=80=81=E3=81=9D?= =?UTF-8?q?=E3=81=AE=E7=AF=80=E5=8D=98=E4=BD=8D=E5=88=A4=E5=AE=9A=E3=81=8C?= =?UTF-8?q?=E8=90=BD=E3=81=A1=E3=81=9F=E8=A8=98=E9=8C=B2=20(#35)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit rule 側は発火条件のみ（+764B, 11行）。ブロック単位で判定すること、 実際に見つかるのは誤った tier ではなく長さの方であること、の2点。 doc 側に depth を置く。時制で判別する案を169ブロックで検証して 17件が要判定・4件が誤答、A ブロック全体の約7%に誤爆し、誤爆先が 体系的に最も長く load-bearing なブロックだった、という記録。 § "Claims you author" の「guard には陰性対照が要る」の worked example になっている。 重複した数値は分割レビューでどのシャードからも見えないので、 レビューエージェントではなく repo 側 grep に回す。 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01UzC2UHsj9cRrQVzrRaHW8T --- docs/claim-verification.md | 40 ++++++++++++++++++++++++++++++++++++- rules/knowledge-layering.md | 11 ++++++++++ 2 files changed, 50 insertions(+), 1 deletion(-) diff --git a/docs/claim-verification.md b/docs/claim-verification.md index 2996cf6..48a1b24 100644 --- a/docs/claim-verification.md +++ b/docs/claim-verification.md @@ -29,7 +29,8 @@ Authored at implementation *or review-fix* time, and executed by nobody. Four sh expressible as a `Verify by` lookup: - **Why-comment on a mechanism** → delete the mechanism and run the tests. Green means the claim - is false, or the tests never covered it. + is false, or the tests never covered it. Its *destination* is a separate question — see + § "A comment written for the reviewer" below. - **A detector / guard / gate** → construct the thing it claims to catch and confirm it fires. A guard's success case proves nothing; only a negative control does. Scope it to the claim it defends: a check narrower than that claim (a files-only loop behind a files-and-directories @@ -48,6 +49,43 @@ expressible as a `Verify by` lookup: written in sequence also read as a **partition** — state the overlap, or the reader does the arithmetic wrong, in the direction that understates residue. +## A comment written for the reviewer + +Backs `rules/knowledge-layering.md` § "Anti-pattern: a comment written for the reviewer". A comment +whose only content is what *this* change did is addressed to a reviewer, who reads the PR body +anyway, rather than to the next editor, who does not. + +**Do not key the rule on wording.** The tempting discriminator is tense — "must stay identical to X" +constrains, "was left identical to X" reports — and it was measured over 169 comment blocks from two +model generations. It needed to decide 17 of them and got 4 wrong, because: + +- It reads the sentence's grammatical head, not its payload. "…lives in `LeafIcon.swift`, which owns + the default this file used to apply" is a stale move record with a present-tense main clause; + "…live in `Foo+Bar.swift` to keep this file under the length budget" is a live navigational + breadcrumb with identical grammar. +- It cannot see duplication at all — a measured figure copy-pasted to a second package is the defect, + and every word of it is a legitimate present-tense fact. +- Backward-looking clauses are frequently load-bearing: a forward rule reading "any key added *after + that* bumps the version" is unparseable once the history clause it refers to is gone. + +The unit of deletion is the **block**, so a per-clause flag on a block-level artifact misfires — in +that corpus, on ~7% of all load-bearing blocks, and systematically on the longest ones, which are +the comments whose loss causes the mistakes the convention exists to prevent. + +**The form that survived the negative control**: flag only when *every* sentence in the block is a +backward-looking report, or when the block restates a figure that has a canonical site elsewhere. On +the same corpus that caught every true instance with no false positives. + +The duplicated-figure shape needs a **repo-side grep**, not a review agent: a review that splits a +large diff by file or axis gives no shard sight of all the sites, so the property is invisible by +construction. Frame the grep as *new code must not add hits*, with the existing count recorded as an +acknowledged baseline — the *reframe* disposition, not a must-return-zero. + +**Length is the commoner defect.** Across the same corpus one generation wrote ~45% more comment +lines per block, and ~50% more blocks per commit, at an unchanged A/B/C/D distribution — the same +content, longer. That is compressible with no information loss, and it is a cheaper and far safer +correction than any rule that proposes deleting a category of content. + ## Reading a probe's outcome **It gets misread in both directions.** Assert that the mutation's anchor matched — a `replace` that diff --git a/rules/knowledge-layering.md b/rules/knowledge-layering.md index db8d30e..207e4f4 100644 --- a/rules/knowledge-layering.md +++ b/rules/knowledge-layering.md @@ -69,6 +69,17 @@ git ls-files -z --cached --others --exclude-standard \ | xargs -0 grep -nHE 'memory `[a-z_]+\.md`' ``` +## Anti-pattern: a comment written for the reviewer + +Same misfiling, one tier down. A comment is read by the **next editor**, so a block saying *only* +what this change did — provenance, the diff's own argument, a figure a canonical site already +states — belongs in the PR body or an ADR. Act on the **whole block**, never a clause: a +backward-looking sentence is routinely what makes the forward rule intelligible. And expect +**length**, not misfiling, to be the defect you actually find — current models state the same +content markedly longer. Measured false-flag rate of the per-clause form, and why a duplicated +figure needs a repo-side grep rather than a review agent: +`~/.claude/kit-docs/claim-verification.md` § "A comment written for the reviewer". + ## Verify before you lock it One discipline, three moments where a claim becomes load-bearing and nobody downstream will check From 54047aa0cc3eef61f1d06769f39d23e27a3d6cc0 Mon Sep 17 00:00:00 2001 From: Tomohito YABU Date: Fri, 14 Aug 2026 23:02:05 +0900 Subject: [PATCH 2/7] =?UTF-8?q?=F0=9F=93=9D=20docs:=20=C2=A7=20"A=20commen?= =?UTF-8?q?t=20written=20for=20the=20reviewer"=20=E3=82=92=E7=B7=A0?= =?UTF-8?q?=E3=82=81=E3=82=8B=20(#35)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 事実主張はすべて保持したままの推敲。169ブロック / 要判定17件 / 誤答4件、 A ブロックの ~7%、~45% と ~50% はいずれも不変。 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01UzC2UHsj9cRrQVzrRaHW8T --- docs/claim-verification.md | 47 ++++++++++++++++++-------------------- 1 file changed, 22 insertions(+), 25 deletions(-) diff --git a/docs/claim-verification.md b/docs/claim-verification.md index 48a1b24..e19d1e2 100644 --- a/docs/claim-verification.md +++ b/docs/claim-verification.md @@ -29,8 +29,7 @@ Authored at implementation *or review-fix* time, and executed by nobody. Four sh expressible as a `Verify by` lookup: - **Why-comment on a mechanism** → delete the mechanism and run the tests. Green means the claim - is false, or the tests never covered it. Its *destination* is a separate question — see - § "A comment written for the reviewer" below. + is false, or the tests never covered it. Its *destination* is a separate question (§ below). - **A detector / guard / gate** → construct the thing it claims to catch and confirm it fires. A guard's success case proves nothing; only a negative control does. Scope it to the claim it defends: a check narrower than that claim (a files-only loop behind a files-and-directories @@ -52,39 +51,37 @@ expressible as a `Verify by` lookup: ## A comment written for the reviewer Backs `rules/knowledge-layering.md` § "Anti-pattern: a comment written for the reviewer". A comment -whose only content is what *this* change did is addressed to a reviewer, who reads the PR body -anyway, rather than to the next editor, who does not. +whose only content is what *this* change did addresses the reviewer, not the next editor. **Do not key the rule on wording.** The tempting discriminator is tense — "must stay identical to X" -constrains, "was left identical to X" reports — and it was measured over 169 comment blocks from two -model generations. It needed to decide 17 of them and got 4 wrong, because: +constrains, "was left identical to X" reports. Measured over 169 comment blocks from two model +generations, it had to decide 17 and got 4 wrong, because: - It reads the sentence's grammatical head, not its payload. "…lives in `LeafIcon.swift`, which owns - the default this file used to apply" is a stale move record with a present-tense main clause; - "…live in `Foo+Bar.swift` to keep this file under the length budget" is a live navigational - breadcrumb with identical grammar. -- It cannot see duplication at all — a measured figure copy-pasted to a second package is the defect, - and every word of it is a legitimate present-tense fact. -- Backward-looking clauses are frequently load-bearing: a forward rule reading "any key added *after - that* bumps the version" is unparseable once the history clause it refers to is gone. - -The unit of deletion is the **block**, so a per-clause flag on a block-level artifact misfires — in -that corpus, on ~7% of all load-bearing blocks, and systematically on the longest ones, which are -the comments whose loss causes the mistakes the convention exists to prevent. + the default this file used to apply" is a stale move record; "…live in `Foo+Bar.swift` to keep + this file under the length budget" is a live navigational breadcrumb — identical grammar. +- It cannot see duplication: a measured figure copy-pasted to a second package is the defect, and + every word of it is a legitimate present-tense fact. +- Backward-looking clauses are frequently load-bearing — a forward rule reading "any key added + *after that* bumps the version" is unparseable once the history clause is gone. + +The unit of deletion is the **block**, so a per-clause flag misfires — on ~7% of load-bearing blocks +in that corpus, and systematically on the longest, whose loss causes exactly the mistakes the +convention prevents. **The form that survived the negative control**: flag only when *every* sentence in the block is a -backward-looking report, or when the block restates a figure that has a canonical site elsewhere. On -the same corpus that caught every true instance with no false positives. +backward-looking report, or when the block restates a figure with a canonical site elsewhere — every +true instance caught, no false positives. The duplicated-figure shape needs a **repo-side grep**, not a review agent: a review that splits a large diff by file or axis gives no shard sight of all the sites, so the property is invisible by -construction. Frame the grep as *new code must not add hits*, with the existing count recorded as an -acknowledged baseline — the *reframe* disposition, not a must-return-zero. +construction. Frame it as *new code must not add hits*, existing count recorded as an acknowledged +baseline — the *reframe* disposition, not a must-return-zero. -**Length is the commoner defect.** Across the same corpus one generation wrote ~45% more comment -lines per block, and ~50% more blocks per commit, at an unchanged A/B/C/D distribution — the same -content, longer. That is compressible with no information loss, and it is a cheaper and far safer -correction than any rule that proposes deleting a category of content. +**Length is the commoner defect.** In that corpus one generation wrote ~45% more comment lines per +block and ~50% more blocks per commit at an unchanged A/B/C/D distribution — same content, longer, +so compressible with no information loss. A cheaper and safer correction than deleting a category of +content. ## Reading a probe's outcome From 044ff2a2ccbe8a0588ea28b5d5489c1da3d2dfec Mon Sep 17 00:00:00 2001 From: Tomohito YABU Date: Fri, 14 Aug 2026 23:03:58 +0900 Subject: [PATCH 3/7] =?UTF-8?q?=F0=9F=93=9D=20docs:=20=E5=AE=9F=E8=A1=8C?= =?UTF-8?q?=E5=8F=AF=E8=83=BD=E3=81=AA=E5=BD=A2=E3=82=92=E5=85=88=E9=A0=AD?= =?UTF-8?q?=E3=81=AB=E5=87=BA=E3=81=97=E3=80=81=E6=99=82=E5=88=B6=E6=A1=88?= =?UTF-8?q?=E3=82=92=E5=8F=8D=E9=9D=A2=E6=95=99=E5=B8=AB=E3=81=B8=E9=99=8D?= =?UTF-8?q?=E6=A0=BC=E3=81=99=E3=82=8B=20(#35)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 「何をすべきか」（ブロック単位の判定）が節の末尾にあり、採用しなかった 時制案の説明が先に来ていた。読み手が最初に当たるものを入れ替える。 rule 側は指し先の要約を短縮。数値はいずれも不変 （169ブロック / 要判定17件 / 誤答4件 / ~7% / ~45% / ~50%）。 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01UzC2UHsj9cRrQVzrRaHW8T --- docs/claim-verification.md | 50 +++++++++++++++++-------------------- rules/knowledge-layering.md | 3 +-- 2 files changed, 24 insertions(+), 29 deletions(-) diff --git a/docs/claim-verification.md b/docs/claim-verification.md index e19d1e2..aeb7b76 100644 --- a/docs/claim-verification.md +++ b/docs/claim-verification.md @@ -53,35 +53,31 @@ expressible as a `Verify by` lookup: Backs `rules/knowledge-layering.md` § "Anti-pattern: a comment written for the reviewer". A comment whose only content is what *this* change did addresses the reviewer, not the next editor. -**Do not key the rule on wording.** The tempting discriminator is tense — "must stay identical to X" -constrains, "was left identical to X" reports. Measured over 169 comment blocks from two model -generations, it had to decide 17 and got 4 wrong, because: - -- It reads the sentence's grammatical head, not its payload. "…lives in `LeafIcon.swift`, which owns - the default this file used to apply" is a stale move record; "…live in `Foo+Bar.swift` to keep - this file under the length budget" is a live navigational breadcrumb — identical grammar. -- It cannot see duplication: a measured figure copy-pasted to a second package is the defect, and - every word of it is a legitimate present-tense fact. -- Backward-looking clauses are frequently load-bearing — a forward rule reading "any key added - *after that* bumps the version" is unparseable once the history clause is gone. - -The unit of deletion is the **block**, so a per-clause flag misfires — on ~7% of load-bearing blocks -in that corpus, and systematically on the longest, whose loss causes exactly the mistakes the -convention prevents. - -**The form that survived the negative control**: flag only when *every* sentence in the block is a -backward-looking report, or when the block restates a figure with a canonical site elsewhere — every -true instance caught, no false positives. - -The duplicated-figure shape needs a **repo-side grep**, not a review agent: a review that splits a -large diff by file or axis gives no shard sight of all the sites, so the property is invisible by -construction. Frame it as *new code must not add hits*, existing count recorded as an acknowledged -baseline — the *reframe* disposition, not a must-return-zero. +**The form that survived a negative control**: flag a block only when *every* sentence in it is a +backward-looking report, or when it restates a figure with a canonical site elsewhere. Over 169 +comment blocks from two model generations, that caught every true instance with no false positives. + +**Do not key it on wording instead.** Tense is the tempting discriminator — "must stay identical to +X" constrains, "was left identical to X" reports — and on that corpus it had to decide 17 blocks and +got 4 wrong. It reads the grammatical head, not the payload: + +- "…lives in `LeafIcon.swift`, which owns the default this file used to apply" (stale move record) + and "…live in `Foo+Bar.swift` to keep this file under the length budget" (live breadcrumb) have + identical grammar. +- Duplication is invisible to it — a figure copy-pasted to a second package is the defect, and every + word of it is a legitimate present-tense fact. +- It strips backward-looking clauses a forward rule depends on ("any key added *after that* bumps + the version"). + +Per-clause flagging misfires on ~7% of load-bearing blocks, worst on the longest. + +The duplicated-figure shape needs a **repo-side grep**, not a review agent — a review split by file +or axis gives no shard sight of all the sites. Frame it as *new code must not add hits*, existing +count as an acknowledged baseline (the *reframe* disposition). **Length is the commoner defect.** In that corpus one generation wrote ~45% more comment lines per -block and ~50% more blocks per commit at an unchanged A/B/C/D distribution — same content, longer, -so compressible with no information loss. A cheaper and safer correction than deleting a category of -content. +block and ~50% more blocks per commit at an unchanged A/B/C/D distribution — same content, longer. +Compressing it loses nothing, and is safer than any rule that deletes a category of content. ## Reading a probe's outcome diff --git a/rules/knowledge-layering.md b/rules/knowledge-layering.md index 207e4f4..240a84f 100644 --- a/rules/knowledge-layering.md +++ b/rules/knowledge-layering.md @@ -76,8 +76,7 @@ what this change did — provenance, the diff's own argument, a figure a canonic states — belongs in the PR body or an ADR. Act on the **whole block**, never a clause: a backward-looking sentence is routinely what makes the forward rule intelligible. And expect **length**, not misfiling, to be the defect you actually find — current models state the same -content markedly longer. Measured false-flag rate of the per-clause form, and why a duplicated -figure needs a repo-side grep rather than a review agent: +content markedly longer. False-flag rates and the duplicated-figure grep: `~/.claude/kit-docs/claim-verification.md` § "A comment written for the reviewer". ## Verify before you lock it From b77553e69755b10704aa28e6d954520d8a7386c7 Mon Sep 17 00:00:00 2001 From: Tomohito YABU Date: Fri, 14 Aug 2026 23:09:42 +0900 Subject: [PATCH 4/7] =?UTF-8?q?=F0=9F=93=9D=20docs:=20=E3=83=88=E3=83=AA?= =?UTF-8?q?=E3=82=AC=E3=83=BC=E3=81=AF=E3=80=8C=E4=B8=8D=E5=9C=A8=E3=80=8D?= =?UTF-8?q?=E3=81=A7=E6=9B=B8=E3=81=8D=E3=80=81=E5=84=AA=E5=85=88=E9=A0=86?= =?UTF-8?q?=E4=BD=8D=E3=82=92=E6=98=8E=E7=A4=BA=E3=81=99=E3=82=8B=20(#35)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit consumer 側でこの節から起こしたレビュー規則を、実在するコメント6件に 当てて検証した結果、2件の欠陥が出た。どちらも読み返しでは見えない。 「全文が報告なら fire」は監査できない — 何が決め手だったかをエージェントが 指せない。「durable な主張がどの文にも無ければ fire」なら救った文を引用 できる。前者は1文ブロックで黙って退化もする（節=ブロックになる）。 トリガーと「load-bearing な節を切るな」の安全弁を両方持つ規則は、どちらが 勝つかを書かないと、生きたポインタを持つブロックがトリガーを引いたときに 「ポインタを消す」か「修正不能な指摘」のどちらかになる。 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01UzC2UHsj9cRrQVzrRaHW8T --- docs/claim-verification.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/claim-verification.md b/docs/claim-verification.md index aeb7b76..6ad8249 100644 --- a/docs/claim-verification.md +++ b/docs/claim-verification.md @@ -71,6 +71,14 @@ got 4 wrong. It reads the grammatical head, not the payload: Per-clause flagging misfires on ~7% of load-bearing blocks, worst on the longest. +**Word the trigger as an absence, and state precedence.** Running the drafted review-agent bullet +over six real blocks caught two further defects. "Flag when *every* sentence merely reports" cannot +be audited — the agent cannot point at what convinced it — whereas "flag when *no* sentence states a +durable claim" makes the saving sentence citable; the first form also degenerates silently on a +one-sentence block, where clause and block coincide. And a rule carrying both a trigger and a "never +cut a load-bearing clause" safeguard must say which wins, or a block that fires the trigger while +holding a live pointer yields either a deleted pointer or an unactionable finding. + The duplicated-figure shape needs a **repo-side grep**, not a review agent — a review split by file or axis gives no shard sight of all the sites. Frame it as *new code must not add hits*, existing count as an acknowledged baseline (the *reframe* disposition). From 608af5afba7b88a442e66616a0f4e513545cf757 Mon Sep 17 00:00:00 2001 From: Tomohito YABU Date: Fri, 14 Aug 2026 23:18:17 +0900 Subject: [PATCH 5/7] =?UTF-8?q?=F0=9F=93=9D=20docs:=20=E9=95=B7=E3=81=95?= =?UTF-8?q?=E3=81=AB=E6=9B=B8=E3=81=8F=E5=81=B4=E3=81=AE=E6=89=8B=E9=A0=86?= =?UTF-8?q?=E3=82=92=E4=B8=8E=E3=81=88=E3=80=81~50%=20=E3=82=92=E5=AE=9F?= =?UTF-8?q?=E6=B8=AC=E3=81=AE=20~47%=20=E3=81=AB=E7=9B=B4=E3=81=99=20(#35)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 「長さの方が主な欠陥」と述べながら、規則は移設（削除）の手順しか持って いなかった。長さの是正は「短く書き直す」であって「消す」ではないので、 別の行為として書く。閾値は簡潔だった世代の実測値 5.5行から取る。 19 → 28ブロック/コミットは +47%。~50% は丸めすぎだった。 コーパスと世代層別の出典を #35 へ向ける。 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01UzC2UHsj9cRrQVzrRaHW8T --- docs/claim-verification.md | 12 +++++++++++- rules/knowledge-layering.md | 13 ++++++++----- 2 files changed, 19 insertions(+), 6 deletions(-) diff --git a/docs/claim-verification.md b/docs/claim-verification.md index 6ad8249..bbe0610 100644 --- a/docs/claim-verification.md +++ b/docs/claim-verification.md @@ -84,9 +84,19 @@ or axis gives no shard sight of all the sites. Frame it as *new code must not ad count as an acknowledged baseline (the *reframe* disposition). **Length is the commoner defect.** In that corpus one generation wrote ~45% more comment lines per -block and ~50% more blocks per commit at an unchanged A/B/C/D distribution — same content, longer. +block and ~47% more blocks per commit at an unchanged A/B/C/D distribution — same content, longer. Compressing it loses nothing, and is safer than any rule that deletes a category of content. +**The rewrite-once instruction the rule carries is a hypothesis, not a settled mechanism.** A +structural instruction is the only thing observed to shorten a draft (25%, against 14% for "compress +this"), but it worked as a *live* instruction, while the always-loaded rule sat in the same context +and did not fire — three verbose drafts were written under it. So ship it and re-measure: comment +lines per block should fall toward 5.5 at an unchanged A/B/C/D distribution. If it does not move, +the mechanism belongs at review or commit time and the always-loaded lines should come back out. + +Corpus, generation stratification (the commits' `Co-Authored-By` trailer) and every figure above: +#35. + ## Reading a probe's outcome **It gets misread in both directions.** Assert that the mutation's anchor matched — a `replace` that diff --git a/rules/knowledge-layering.md b/rules/knowledge-layering.md index 240a84f..bbb3a5c 100644 --- a/rules/knowledge-layering.md +++ b/rules/knowledge-layering.md @@ -73,11 +73,14 @@ git ls-files -z --cached --others --exclude-standard \ Same misfiling, one tier down. A comment is read by the **next editor**, so a block saying *only* what this change did — provenance, the diff's own argument, a figure a canonical site already -states — belongs in the PR body or an ADR. Act on the **whole block**, never a clause: a -backward-looking sentence is routinely what makes the forward rule intelligible. And expect -**length**, not misfiling, to be the defect you actually find — current models state the same -content markedly longer. False-flag rates and the duplicated-figure grep: -`~/.claude/kit-docs/claim-verification.md` § "A comment written for the reviewer". +states — belongs in the PR body or an ADR. **When flagging that**, move the whole block, never a +clause: a backward-looking sentence is routinely what makes the forward rule intelligible. But +expect **length**, not misfiling, to be the defect you actually find — current models state the +same content markedly longer, and the remedy there is rewriting the block shorter, which is a +different act from deleting one. So when *writing* one: a block past ~6 lines (the concise +generation sat at 5.5) gets one rewrite at half the length, and **the rewrite wins** unless it +dropped a forward-looking fact. False-flag rates, why the duplicated-figure shape needs a +repo-side grep: `~/.claude/kit-docs/claim-verification.md` § "A comment written for the reviewer". ## Verify before you lock it From f3e9470642aae30efd37e9f43647d361f265e872 Mon Sep 17 00:00:00 2001 From: Tomohito YABU Date: Fri, 14 Aug 2026 23:32:24 +0900 Subject: [PATCH 6/7] =?UTF-8?q?=E2=9C=A8=20feat:=20comment-density.py=20?= =?UTF-8?q?=E3=81=A8=E3=80=81=E3=82=B2=E3=83=BC=E3=83=883=E6=A1=88?= =?UTF-8?q?=E3=82=92=E5=8F=8D=E8=A8=BC=E3=81=97=E3=81=9F=E8=BC=83=E6=AD=A3?= =?UTF-8?q?=E8=A8=98=E9=8C=B2=20(#35)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 規則の長さ主張は測ってから道具なしで出荷していた。コホート中央値を `Co-Authored-By` trailer で層別して出す道具を足す。 **per-commit ゲートは載せない。** 簡潔な世代を対照に較正すると3案とも 落ちた。片方の閾値では冗長側の80-96%を捕らえる設定が簡潔側の52-79%を、 両方では誤爆12-22%まで下がる代わり検出が32-48%まで崩れる。規則自身の トリガー単位であるブロック単位では**両コホートの中央値が3.0行で同じ**。 較正の副産物として主張が2つ変わった。差は長さより**本数** （16→26ブロック/コミット +63% 対 4.3→5.9行/ブロック +37%）なので、 長さを狙う機構は自分の欠陥の小さい方を狙っている。閾値も平均ではなく 簡潔側の上位1割から取り直して ~10行にした。そして Fable 5 は両コホート より下（3.4行/ブロック）で、軸は世代ではなくモデルだった。 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01UzC2UHsj9cRrQVzrRaHW8T --- docs/claim-verification.md | 30 +++++-- rules/knowledge-layering.md | 22 ++--- scripts/comment-density.py | 170 ++++++++++++++++++++++++++++++++++++ 3 files changed, 206 insertions(+), 16 deletions(-) create mode 100755 scripts/comment-density.py diff --git a/docs/claim-verification.md b/docs/claim-verification.md index bbe0610..a4b7fa1 100644 --- a/docs/claim-verification.md +++ b/docs/claim-verification.md @@ -90,12 +90,30 @@ Compressing it loses nothing, and is safer than any rule that deletes a category **The rewrite-once instruction the rule carries is a hypothesis, not a settled mechanism.** A structural instruction is the only thing observed to shorten a draft (25%, against 14% for "compress this"), but it worked as a *live* instruction, while the always-loaded rule sat in the same context -and did not fire — three verbose drafts were written under it. So ship it and re-measure: comment -lines per block should fall toward 5.5 at an unchanged A/B/C/D distribution. If it does not move, -the mechanism belongs at review or commit time and the always-loaded lines should come back out. - -Corpus, generation stratification (the commits' `Co-Authored-By` trailer) and every figure above: -#35. +and did not fire — three verbose drafts were written under it. So ship it and re-measure with +`scripts/comment-density.py`: blocks per commit should fall toward the concise cohort's 16, on that +tool's counting rule. The other half of the measure — an unchanged A/B/C/D distribution — the tool +cannot see, so it still needs a hand pass. If neither moves, the mechanism belongs at review time +and the always-loaded lines come back out. + +**Three gate designs were built and refuted.** Calibrating each against the concise generation +killed it. Per-commit on either threshold: catching 80-96% of the verbose cohort also flagged 52-79% +of the concise one. Per-commit on both: false flags drop to 12-22% but detection collapses to +32-48%. Per-block — the unit the rule's own trigger uses — **the cohorts share a median block of 3.0 +lines**, and a ≥12-line trigger fires on 7.8% of concise blocks against 11.4% of verbose ones (at +≥10 lines, the rule's authoring trigger, 10.5% against 15.0% — a top tenth in both cohorts). A +gate at any of those rates teaches its reader to ignore it, so the tool ships as cohort measurement +only. + +What separates the cohorts is **count, not length**: 16 → 26 blocks per commit (+63%) against 4.3 → +5.9 lines per block (+37%), over a shared median. The verbose generation writes more comments rather +than much longer ones, so a length-keyed mechanism aims at the smaller half of its own defect — +hence the rule leading with the count. Fable 5 sits below both cohorts (3.4 lines per block, 32.1% +comment share): the axis is the model, not recency. + +Corpus, stratification (the commits' `Co-Authored-By` trailer), and the A/B/C/D and +45%/+47% +figures: #35. The calibration above is this session's and is *not* in that issue; it runs on the +tool's own counting rule, moves if `commit_stats` changes, and is re-runnable via `--dump`. ## Reading a probe's outcome diff --git a/rules/knowledge-layering.md b/rules/knowledge-layering.md index bbb3a5c..305c7fa 100644 --- a/rules/knowledge-layering.md +++ b/rules/knowledge-layering.md @@ -71,16 +71,18 @@ git ls-files -z --cached --others --exclude-standard \ ## Anti-pattern: a comment written for the reviewer -Same misfiling, one tier down. A comment is read by the **next editor**, so a block saying *only* -what this change did — provenance, the diff's own argument, a figure a canonical site already -states — belongs in the PR body or an ADR. **When flagging that**, move the whole block, never a -clause: a backward-looking sentence is routinely what makes the forward rule intelligible. But -expect **length**, not misfiling, to be the defect you actually find — current models state the -same content markedly longer, and the remedy there is rewriting the block shorter, which is a -different act from deleting one. So when *writing* one: a block past ~6 lines (the concise -generation sat at 5.5) gets one rewrite at half the length, and **the rewrite wins** unless it -dropped a forward-looking fact. False-flag rates, why the duplicated-figure shape needs a -repo-side grep: `~/.claude/kit-docs/claim-verification.md` § "A comment written for the reviewer". +Same misfiling, one tier down. A comment is read by the **next editor**, so a block where *no* +sentence states a durable claim — only provenance, the diff's own argument, a figure a canonical +site already states — belongs in the PR body or an ADR. **When flagging that**, move the whole +block, never a clause: a backward-looking sentence is routinely what makes the forward rule +intelligible. But expect **volume**, not misfiling, to be the defect you actually find; it tracks +the model, not recency, and its remedy is rewriting shorter — a different act from deleting. When +*writing*, watch the count before the length: the measured excess is ~60% more blocks against ~35% +longer ones, so the block not worth writing is commoner than the block worth shortening. Past ~10 +lines (the top tenth even for the concise baseline) rewrite once at half length, and **the rewrite +wins** unless it dropped a forward-looking fact. False-flag rates, the calibration that refuted a +commit-level gate, why the duplicated-figure shape needs a repo-side grep: +`~/.claude/kit-docs/claim-verification.md` § "A comment written for the reviewer". ## Verify before you lock it diff --git a/scripts/comment-density.py b/scripts/comment-density.py new file mode 100755 index 0000000..aa741d0 --- /dev/null +++ b/scripts/comment-density.py @@ -0,0 +1,170 @@ +#!/usr/bin/env python3 +"""comment-density.py — measure and gate comment volume in a diff. + +Backs `rules/knowledge-layering.md` § "Anti-pattern: a comment written for the +reviewer", whose volume claim was measured but shipped without a tool. It +reports cohort medians over a revision range, grouped by the `Co-Authored-By` +trailer. That covers the volume half of the section's success measure; the +"unchanged A/B/C/D distribution" half needs a hand pass — this tool classifies +nothing. + +**There is deliberately no gate.** Three designs were calibrated against a +generation known to be concise, and all three died: per-commit on either of two +thresholds (catching 80-96% of the verbose cohort cost 52-79% false flags), +per-commit on both (false flags down to 12-22%, but detection down to 32-48%), +and per-block (the two cohorts share a median block of 3.0 lines, so no +threshold separates them). A gate at those rates teaches its reader to ignore +it. Do not re-add one without re-running that control — `--dump` emits the +per-commit rows it needs, and the numbers are in `docs/claim-verification.md` +§ "A comment written for the reviewer". + +Counting rule (stated so the figure is reproducible, not so it is exact): only +lines whose first non-blank character opens a comment are counted, and a +"block" is a run of consecutive such lines *within one hunk*. It errs in both +directions — a trailing comment after code is missed, pushing the ratio down; +a full-line `//` inside a multiline string is counted, pushing it up — but the +first is far commoner, so it predominantly under-reports. + +Usage: + comment-density.py [--repo DIR] [--by-model] [--min-added N] +""" + +import argparse +import re +import statistics +import subprocess +import sys +from collections import defaultdict + +# Full-line comment openers by extension. Block-comment continuation lines +# (` * …`) count too — they are the same authored prose. +LINE_MARKERS = { + "//": {".swift", ".js", ".jsx", ".ts", ".tsx", ".go", ".java", ".kt", + ".c", ".h", ".cc", ".cpp", ".hpp", ".rs", ".scala", ".dart"}, + "#": {".py", ".rb", ".sh", ".bash", ".zsh", ".yml", ".yaml", ".toml"}, +} +BLOCK_OPEN = re.compile(r"^(/\*|\*|\"\"\"|'''|=begin)") + +TEST_PATH = re.compile(r"(^|/)(tests?|spec|__tests__)/|[Tt]ests?\.\w+$|_test\.\w+$") + + +def markers_for(path): + ext = path[path.rfind("."):] if "." in path else "" + return [m for m, exts in LINE_MARKERS.items() if ext in exts] + + +def is_comment(line, markers): + s = line.strip() + if not s: + return False + return any(s.startswith(m) for m in markers) or bool(BLOCK_OPEN.match(s)) + + +def git(repo, *args): + return subprocess.run(["git", "-C", repo, *args], + capture_output=True, text=True, check=True).stdout + + +def commit_stats(repo, sha, include_tests): + """Return (code_lines, comment_lines, block_sizes) over the commit's added lines.""" + diff = git(repo, "show", "--unified=0", "--format=", "--no-color", sha) + code = comment = 0 + blocks, run, markers = [], 0, [] + for line in diff.splitlines(): + # A hunk header ends any run: under --unified=0 the next hunk is a + # non-adjacent part of the file, so carrying `run` across one would + # merge unrelated comments into a single oversized pseudo-block. + if line.startswith("+++ b/") or line.startswith("@@"): + if run: + blocks.append(run) + run = 0 + if line.startswith("@@"): + continue + path = line[6:] + markers = [] if (not include_tests and TEST_PATH.search(path)) else markers_for(path) + continue + if not line.startswith("+") or line.startswith("+++") or not markers: + continue + body = line[1:] + if is_comment(body, markers): + comment += 1 + run += 1 + else: + if run: + blocks.append(run) + run = 0 + if body.strip(): + code += 1 + if run: + blocks.append(run) + return code, comment, blocks + + +def model_of(repo, sha): + trailer = git(repo, "show", "-s", + "--format=%(trailers:key=Co-Authored-By,valueonly)", sha) + m = re.search(r"(Claude [\w.]+ [\d.]+)", trailer) + return m.group(1) if m else "unknown" + + +def collect(repo, rev_range, include_tests, min_added): + shas = git(repo, "rev-list", "--no-merges", rev_range).split() + rows = [] + for sha in shas: + code, comment, blocks = commit_stats(repo, sha, include_tests) + if code + comment < min_added: + continue + rows.append({ + "sha": sha[:9], + "code": code, + "comment": comment, + "ratio": comment / (code + comment), + "blocks": len(blocks), + "per_block": statistics.mean(blocks) if blocks else 0.0, + "max_block": max(blocks) if blocks else 0, + }) + return rows + + +def summarize(label, rows): + if not rows: + print(f"{label:<24} (no commits met the threshold)") + return + med = lambda k: statistics.median(r[k] for r in rows) + print(f"{label:<24} n={len(rows):<4} ratio={med('ratio'):.1%} " + f"lines/block={med('per_block'):.1f} blocks/commit={med('blocks'):.0f}") + + +def main(): + p = argparse.ArgumentParser(add_help=True) + p.add_argument("range", help="git revision range, e.g. HEAD~200..HEAD") + p.add_argument("--repo", default=".") + p.add_argument("--by-model", action="store_true", + help="group by the Co-Authored-By trailer") + p.add_argument("--include-tests", action="store_true") + p.add_argument("--min-added", type=int, default=30) + p.add_argument("--dump", action="store_true", + help="one TSV row per commit, so the calibration in the doc " + "can be re-run without reimplementing this") + a = p.parse_args() + + rows = collect(a.repo, a.range, a.include_tests, a.min_added) + if a.dump: + print("sha\tmodel\tratio\tblocks\tper_block\tmax_block") + for r in rows: + print(f"{r['sha']}\t{model_of(a.repo, r['sha'])}\t{r['ratio']:.4f}\t" + f"{r['blocks']}\t{r['per_block']:.2f}\t{r['max_block']}") + return 0 + if not a.by_model: + summarize("all", rows) + return 0 + groups = defaultdict(list) + for r in rows: + groups[model_of(a.repo, r["sha"])].append(r) + for label in sorted(groups, key=lambda k: -len(groups[k])): + summarize(label, groups[label]) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From 88ee28bb31564f3a354b4003861d55b796eb5742 Mon Sep 17 00:00:00 2001 From: Tomohito YABU Date: Fri, 14 Aug 2026 23:37:10 +0900 Subject: [PATCH 7/7] =?UTF-8?q?=E2=9C=85=20test:=20parse=5Fadded=20?= =?UTF-8?q?=E3=82=92=E5=88=87=E3=82=8A=E5=87=BA=E3=81=97=E3=80=81=E3=83=8F?= =?UTF-8?q?=E3=83=B3=E3=82=AF=E5=A2=83=E7=95=8C=E3=81=AE=E8=87=AA=E5=B7=B1?= =?UTF-8?q?=E3=83=86=E3=82=B9=E3=83=88=E3=82=92=E7=A9=8D=E3=82=80=20(#35)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ゲート3案を落とした較正は、ブロック境界の欠陥が乗ったままの計器で 回していた（@@ でリセットせず、非隣接ハンクのコメントが1ブロックに融合）。 この形の欠陥はコホート集計には現れないので、既知の diff を当てる fixture が無い限り誰も捕まえられない。 git 呼び出しから parse_added を切り出し、5ケースの --self-test を積んだ。 負の対照として修正前のコードに当て、ハンク境界のケースのみが [2,2] を期待して [4] を返すことを確認済み。 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EXUZckBU1oD6hMoVCdy6iQ --- scripts/comment-density.py | 48 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-) diff --git a/scripts/comment-density.py b/scripts/comment-density.py index aa741d0..01d4c64 100755 --- a/scripts/comment-density.py +++ b/scripts/comment-density.py @@ -26,7 +26,8 @@ first is far commoner, so it predominantly under-reports. Usage: - comment-density.py [--repo DIR] [--by-model] [--min-added N] + comment-density.py [--repo DIR] [--by-model] [--dump] [--min-added N] + comment-density.py --self-test """ import argparse @@ -66,8 +67,17 @@ def git(repo, *args): def commit_stats(repo, sha, include_tests): - """Return (code_lines, comment_lines, block_sizes) over the commit's added lines.""" diff = git(repo, "show", "--unified=0", "--format=", "--no-color", sha) + return parse_added(diff, include_tests) + + +def parse_added(diff, include_tests): + """Return (code_lines, comment_lines, block_sizes) over a diff's added lines. + + Split out from the git call so `--self-test` can exercise it on a known + diff: the one defect found here was a block-boundary error, which no + cohort-level number would have exposed. + """ code = comment = 0 blocks, run, markers = [], 0, [] for line in diff.splitlines(): @@ -135,8 +145,42 @@ def summarize(label, rows): f"lines/block={med('per_block'):.1f} blocks/commit={med('blocks'):.0f}") +SELF_TEST = [ + # (name, diff, expected (code, comment, blocks)) + ("comments in separate hunks stay separate blocks", + "+++ b/a.swift\n@@ -1,0 +2,2 @@\n+// one\n+// two\n@@ -30,0 +34,2 @@\n+// three\n+// four\n", + (0, 4, [2, 2])), + ("code inside a hunk splits the block", + "+++ b/a.swift\n@@ -1,0 +2,3 @@\n+// one\n+let x = 1\n+// two\n", + (1, 2, [1, 1])), + ("a file switch closes the open block", + "+++ b/a.swift\n@@ -1,0 +2,1 @@\n+// one\n+++ b/b.swift\n@@ -1,0 +2,1 @@\n+// two\n", + (0, 2, [1, 1])), + ("test files are excluded", + "+++ b/Tests/aTests.swift\n@@ -1,0 +2,1 @@\n+// ignored\n", + (0, 0, [])), + ("an unknown extension contributes nothing", + "+++ b/README.md\n@@ -1,0 +2,1 @@\n+// not a known comment syntax here\n", + (0, 0, [])), +] + + +def self_test(): + failed = 0 + for name, diff, expected in SELF_TEST: + got = parse_added(diff, include_tests=False) + got = (got[0], got[1], got[2]) + if got != expected: + print(f"FAIL {name}\n expected {expected}\n got {got}") + failed += 1 + print(f"{'FAIL' if failed else 'PASS'} ({len(SELF_TEST) - failed}/{len(SELF_TEST)})") + return 1 if failed else 0 + + def main(): p = argparse.ArgumentParser(add_help=True) + if "--self-test" in sys.argv: + return self_test() p.add_argument("range", help="git revision range, e.g. HEAD~200..HEAD") p.add_argument("--repo", default=".") p.add_argument("--by-model", action="store_true",