📝 docs: 超過は長さでなく本数だった — 測定ツールとゲート3案の反証

docs/claim-verification.md

@@ -29,7 +29,7 @@ Authored at implementation *or review-fix* time, and executed by nobody. Four sh
 expressible as a `Verify by` lookup:
 
 - **Why-comment on a mechanism** → delete the mechanism and run the tests. Green means the claim
-  is false, or the tests never covered it.
+  is false, or the tests never covered it. Its *destination* is a separate question (§ below).
 - **A detector / guard / gate** → construct the thing it claims to catch and confirm it fires. A
   guard's success case proves nothing; only a negative control does. Scope it to the claim it
   defends: a check narrower than that claim (a files-only loop behind a files-and-directories
@@ -48,6 +48,73 @@ expressible as a `Verify by` lookup:
   written in sequence also read as a **partition** — state the overlap, or the reader does the
   arithmetic wrong, in the direction that understates residue.
 
+## A comment written for the reviewer
+
+Backs `rules/knowledge-layering.md` § "Anti-pattern: a comment written for the reviewer". A comment
+whose only content is what *this* change did addresses the reviewer, not the next editor.
+
+**The form that survived a negative control**: flag a block only when *every* sentence in it is a
+backward-looking report, or when it restates a figure with a canonical site elsewhere. Over 169
+comment blocks from two model generations, that caught every true instance with no false positives.
+
+**Do not key it on wording instead.** Tense is the tempting discriminator — "must stay identical to
+X" constrains, "was left identical to X" reports — and on that corpus it had to decide 17 blocks and
+got 4 wrong. It reads the grammatical head, not the payload:
+
+- "…lives in `LeafIcon.swift`, which owns the default this file used to apply" (stale move record)
+  and "…live in `Foo+Bar.swift` to keep this file under the length budget" (live breadcrumb) have
+  identical grammar.
+- Duplication is invisible to it — a figure copy-pasted to a second package is the defect, and every
+  word of it is a legitimate present-tense fact.
+- It strips backward-looking clauses a forward rule depends on ("any key added *after that* bumps
+  the version").
+
+Per-clause flagging misfires on ~7% of load-bearing blocks, worst on the longest.
+
+**Word the trigger as an absence, and state precedence.** Running the drafted review-agent bullet
+over six real blocks caught two further defects. "Flag when *every* sentence merely reports" cannot
+be audited — the agent cannot point at what convinced it — whereas "flag when *no* sentence states a
+durable claim" makes the saving sentence citable; the first form also degenerates silently on a
+one-sentence block, where clause and block coincide. And a rule carrying both a trigger and a "never
+cut a load-bearing clause" safeguard must say which wins, or a block that fires the trigger while
+holding a live pointer yields either a deleted pointer or an unactionable finding.
+
+The duplicated-figure shape needs a **repo-side grep**, not a review agent — a review split by file
+or axis gives no shard sight of all the sites. Frame it as *new code must not add hits*, existing
+count as an acknowledged baseline (the *reframe* disposition).
+
+**Length is the commoner defect.** In that corpus one generation wrote ~45% more comment lines per
+block and ~47% more blocks per commit at an unchanged A/B/C/D distribution — same content, longer.
+Compressing it loses nothing, and is safer than any rule that deletes a category of content.
+
+**The rewrite-once instruction the rule carries is a hypothesis, not a settled mechanism.** A
+structural instruction is the only thing observed to shorten a draft (25%, against 14% for "compress
+this"), but it worked as a *live* instruction, while the always-loaded rule sat in the same context
+and did not fire — three verbose drafts were written under it. So ship it and re-measure with
+`scripts/comment-density.py`: blocks per commit should fall toward the concise cohort's 16, on that
+tool's counting rule. The other half of the measure — an unchanged A/B/C/D distribution — the tool
+cannot see, so it still needs a hand pass. If neither moves, the mechanism belongs at review time
+and the always-loaded lines come back out.
+
+**Three gate designs were built and refuted.** Calibrating each against the concise generation
+killed it. Per-commit on either threshold: catching 80-96% of the verbose cohort also flagged 52-79%
+of the concise one. Per-commit on both: false flags drop to 12-22% but detection collapses to
+32-48%. Per-block — the unit the rule's own trigger uses — **the cohorts share a median block of 3.0
+lines**, and a ≥12-line trigger fires on 7.8% of concise blocks against 11.4% of verbose ones (at
+≥10 lines, the rule's authoring trigger, 10.5% against 15.0% — a top tenth in both cohorts). A
+gate at any of those rates teaches its reader to ignore it, so the tool ships as cohort measurement
+only.
+
+What separates the cohorts is **count, not length**: 16 → 26 blocks per commit (+63%) against 4.3 →
+5.9 lines per block (+37%), over a shared median. The verbose generation writes more comments rather
+than much longer ones, so a length-keyed mechanism aims at the smaller half of its own defect —
+hence the rule leading with the count. Fable 5 sits below both cohorts (3.4 lines per block, 32.1%
+comment share): the axis is the model, not recency.
+
+Corpus, stratification (the commits' `Co-Authored-By` trailer), and the A/B/C/D and +45%/+47%
+figures: #35. The calibration above is this session's and is *not* in that issue; it runs on the
+tool's own counting rule, moves if `commit_stats` changes, and is re-runnable via `--dump`.
+
 ## Reading a probe's outcome
 
 **It gets misread in both directions.** Assert that the mutation's anchor matched — a `replace` that

rules/knowledge-layering.md

@@ -69,6 +69,21 @@ git ls-files -z --cached --others --exclude-standard \
   | xargs -0 grep -nHE 'memory `[a-z_]+\.md`'
 ```
 
+## Anti-pattern: a comment written for the reviewer
+
+Same misfiling, one tier down. A comment is read by the **next editor**, so a block where *no*
+sentence states a durable claim — only provenance, the diff's own argument, a figure a canonical
+site already states — belongs in the PR body or an ADR. **When flagging that**, move the whole
+block, never a clause: a backward-looking sentence is routinely what makes the forward rule
+intelligible. But expect **volume**, not misfiling, to be the defect you actually find; it tracks
+the model, not recency, and its remedy is rewriting shorter — a different act from deleting. When
+*writing*, watch the count before the length: the measured excess is ~60% more blocks against ~35%
+longer ones, so the block not worth writing is commoner than the block worth shortening. Past ~10
+lines (the top tenth even for the concise baseline) rewrite once at half length, and **the rewrite
+wins** unless it dropped a forward-looking fact. False-flag rates, the calibration that refuted a
+commit-level gate, why the duplicated-figure shape needs a repo-side grep:
+`~/.claude/kit-docs/claim-verification.md` § "A comment written for the reviewer".
+
 ## Verify before you lock it
 
 One discipline, three moments where a claim becomes load-bearing and nobody downstream will check

scripts/comment-density.py

@@ -0,0 +1,214 @@
+#!/usr/bin/env python3
+"""comment-density.py — measure and gate comment volume in a diff.
+
+Backs `rules/knowledge-layering.md` § "Anti-pattern: a comment written for the
+reviewer", whose volume claim was measured but shipped without a tool. It
+reports cohort medians over a revision range, grouped by the `Co-Authored-By`
+trailer. That covers the volume half of the section's success measure; the
+"unchanged A/B/C/D distribution" half needs a hand pass — this tool classifies
+nothing.
+
+**There is deliberately no gate.** Three designs were calibrated against a
+generation known to be concise, and all three died: per-commit on either of two
+thresholds (catching 80-96% of the verbose cohort cost 52-79% false flags),
+per-commit on both (false flags down to 12-22%, but detection down to 32-48%),
+and per-block (the two cohorts share a median block of 3.0 lines, so no
+threshold separates them). A gate at those rates teaches its reader to ignore
+it. Do not re-add one without re-running that control — `--dump` emits the
+per-commit rows it needs, and the numbers are in `docs/claim-verification.md`
+§ "A comment written for the reviewer".
+
+Counting rule (stated so the figure is reproducible, not so it is exact): only
+lines whose first non-blank character opens a comment are counted, and a
+"block" is a run of consecutive such lines *within one hunk*. It errs in both
+directions — a trailing comment after code is missed, pushing the ratio down;
+a full-line `//` inside a multiline string is counted, pushing it up — but the
+first is far commoner, so it predominantly under-reports.
+
+Usage:
+  comment-density.py <range> [--repo DIR] [--by-model] [--dump] [--min-added N]
+  comment-density.py --self-test
+"""
+
+import argparse
+import re
+import statistics
+import subprocess
+import sys
+from collections import defaultdict
+
+# Full-line comment openers by extension. Block-comment continuation lines
+# (` * …`) count too — they are the same authored prose.
+LINE_MARKERS = {
+    "//": {".swift", ".js", ".jsx", ".ts", ".tsx", ".go", ".java", ".kt",
+           ".c", ".h", ".cc", ".cpp", ".hpp", ".rs", ".scala", ".dart"},
+    "#": {".py", ".rb", ".sh", ".bash", ".zsh", ".yml", ".yaml", ".toml"},
+}
+BLOCK_OPEN = re.compile(r"^(/\*|\*|\"\"\"|'''|=begin)")
+
+TEST_PATH = re.compile(r"(^|/)(tests?|spec|__tests__)/|[Tt]ests?\.\w+$|_test\.\w+$")
+
+
+def markers_for(path):
+    ext = path[path.rfind("."):] if "." in path else ""
+    return [m for m, exts in LINE_MARKERS.items() if ext in exts]
+
+
+def is_comment(line, markers):
+    s = line.strip()
+    if not s:
+        return False
+    return any(s.startswith(m) for m in markers) or bool(BLOCK_OPEN.match(s))
+
+
+def git(repo, *args):
+    return subprocess.run(["git", "-C", repo, *args],
+                          capture_output=True, text=True, check=True).stdout
+
+
+def commit_stats(repo, sha, include_tests):
+    diff = git(repo, "show", "--unified=0", "--format=", "--no-color", sha)
+    return parse_added(diff, include_tests)
+
+
+def parse_added(diff, include_tests):
+    """Return (code_lines, comment_lines, block_sizes) over a diff's added lines.
+
+    Split out from the git call so `--self-test` can exercise it on a known
+    diff: the one defect found here was a block-boundary error, which no
+    cohort-level number would have exposed.
+    """
+    code = comment = 0
+    blocks, run, markers = [], 0, []
+    for line in diff.splitlines():
+        # A hunk header ends any run: under --unified=0 the next hunk is a
+        # non-adjacent part of the file, so carrying `run` across one would
+        # merge unrelated comments into a single oversized pseudo-block.
+        if line.startswith("+++ b/") or line.startswith("@@"):
+            if run:
+                blocks.append(run)
+                run = 0
+            if line.startswith("@@"):
+                continue
+            path = line[6:]
+            markers = [] if (not include_tests and TEST_PATH.search(path)) else markers_for(path)
+            continue
+        if not line.startswith("+") or line.startswith("+++") or not markers:
+            continue
+        body = line[1:]
+        if is_comment(body, markers):
+            comment += 1
+            run += 1
+        else:
+            if run:
+                blocks.append(run)
+                run = 0
+            if body.strip():
+                code += 1
+    if run:
+        blocks.append(run)
+    return code, comment, blocks
+
+
+def model_of(repo, sha):
+    trailer = git(repo, "show", "-s",
+                  "--format=%(trailers:key=Co-Authored-By,valueonly)", sha)
+    m = re.search(r"(Claude [\w.]+ [\d.]+)", trailer)
+    return m.group(1) if m else "unknown"
+
+
+def collect(repo, rev_range, include_tests, min_added):
+    shas = git(repo, "rev-list", "--no-merges", rev_range).split()
+    rows = []
+    for sha in shas:
+        code, comment, blocks = commit_stats(repo, sha, include_tests)
+        if code + comment < min_added:
+            continue
+        rows.append({
+            "sha": sha[:9],
+            "code": code,
+            "comment": comment,
+            "ratio": comment / (code + comment),
+            "blocks": len(blocks),
+            "per_block": statistics.mean(blocks) if blocks else 0.0,
+            "max_block": max(blocks) if blocks else 0,
+        })
+    return rows
+
+
+def summarize(label, rows):
+    if not rows:
+        print(f"{label:<24} (no commits met the threshold)")
+        return
+    med = lambda k: statistics.median(r[k] for r in rows)
+    print(f"{label:<24} n={len(rows):<4} ratio={med('ratio'):.1%}  "
+          f"lines/block={med('per_block'):.1f}  blocks/commit={med('blocks'):.0f}")
+
+
+SELF_TEST = [
+    # (name, diff, expected (code, comment, blocks))
+    ("comments in separate hunks stay separate blocks",
+     "+++ b/a.swift\n@@ -1,0 +2,2 @@\n+// one\n+// two\n@@ -30,0 +34,2 @@\n+// three\n+// four\n",
+     (0, 4, [2, 2])),
+    ("code inside a hunk splits the block",
+     "+++ b/a.swift\n@@ -1,0 +2,3 @@\n+// one\n+let x = 1\n+// two\n",
+     (1, 2, [1, 1])),
+    ("a file switch closes the open block",
+     "+++ b/a.swift\n@@ -1,0 +2,1 @@\n+// one\n+++ b/b.swift\n@@ -1,0 +2,1 @@\n+// two\n",
+     (0, 2, [1, 1])),
+    ("test files are excluded",
+     "+++ b/Tests/aTests.swift\n@@ -1,0 +2,1 @@\n+// ignored\n",
+     (0, 0, [])),
+    ("an unknown extension contributes nothing",
+     "+++ b/README.md\n@@ -1,0 +2,1 @@\n+// not a known comment syntax here\n",
+     (0, 0, [])),
+]
+
+
+def self_test():
+    failed = 0
+    for name, diff, expected in SELF_TEST:
+        got = parse_added(diff, include_tests=False)
+        got = (got[0], got[1], got[2])
+        if got != expected:
+            print(f"FAIL {name}\n  expected {expected}\n  got      {got}")
+            failed += 1
+    print(f"{'FAIL' if failed else 'PASS'} ({len(SELF_TEST) - failed}/{len(SELF_TEST)})")
+    return 1 if failed else 0
+
+
+def main():
+    p = argparse.ArgumentParser(add_help=True)
+    if "--self-test" in sys.argv:
+        return self_test()
+    p.add_argument("range", help="git revision range, e.g. HEAD~200..HEAD")
+    p.add_argument("--repo", default=".")
+    p.add_argument("--by-model", action="store_true",
+                   help="group by the Co-Authored-By trailer")
+    p.add_argument("--include-tests", action="store_true")
+    p.add_argument("--min-added", type=int, default=30)
+    p.add_argument("--dump", action="store_true",
+                   help="one TSV row per commit, so the calibration in the doc "
+                        "can be re-run without reimplementing this")
+    a = p.parse_args()
+
+    rows = collect(a.repo, a.range, a.include_tests, a.min_added)
+    if a.dump:
+        print("sha\tmodel\tratio\tblocks\tper_block\tmax_block")
+        for r in rows:
+            print(f"{r['sha']}\t{model_of(a.repo, r['sha'])}\t{r['ratio']:.4f}\t"
+                  f"{r['blocks']}\t{r['per_block']:.2f}\t{r['max_block']}")
+        return 0
+    if not a.by_model:
+        summarize("all", rows)
+        return 0
+    groups = defaultdict(list)
+    for r in rows:
+        groups[model_of(a.repo, r["sha"])].append(r)
+    for label in sorted(groups, key=lambda k: -len(groups[k])):
+        summarize(label, groups[label])
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())