Retract the precision numbers: the baseline was given the rubric

A reader refused to believe the result — if plain Claude Code beats both
harnesses, why would anyone use either — and the disbelief was correct.

The two arms were not asked the same question. The baseline prompt said "report
ONLY genuine defects... do not report style, naming, typing, documentation, or
preference", so it stayed silent on all of that. The harness arms were given
their own /review, which is written to produce a thorough multi-section report,
and it duly reported things like "inconsistent parameter typing (nit)", "no tests
(low)" and "no call sites (info)".

None of those is a false claim about a defect. They are correct observations the
baseline had been instructed not to make, and the scoring counted every one as a
false positive. The precision column therefore compared a constrained prompt with
unconstrained reviews and scored the difference as error.

The extraction pass now applies one bar to every arm: keep only what the review
presents as an actual bug, drop anything it frames as a nit, a style or typing
preference, a missing test, a suggestion or an informational note. On a single
sanity run that takes vstack's false positives on the same fixtures from 5 to 2.

The retracted table stays in RESULTS.md with the reason attached, because
deleting a published result that turned out to be wrong is worse than leaving it
standing and saying why. A corrected 120-review run is in progress.

Worth recording that no check in this repository caught this. The falsifiability
suite proves each check can fail; it has nothing to say about a benchmark that
asks two arms different questions.

Author: itsvedantkumar

tests/evals/RESULTS.md

@@ -135,7 +135,17 @@ error bars. The fixtures are also small and Python-only; a defect class this set
 is a defect class this says nothing about.
 
 
-## Run of 2026-08-21 — 8 fixtures, 5 samples per arm (120 reviews)
+## RETRACTED — Run of 2026-08-21 — 8 fixtures, 5 samples per arm (120 reviews)
+
+> **These numbers are unfair to both harnesses and are retracted.** The scoring counted a
+> reviewer's nits, style notes and "no tests here" observations as false positives, while the
+> baseline had been explicitly instructed not to make such observations at all. So the precision
+> column compared a constrained prompt against unconstrained thorough reviews and scored the
+> difference as error. The table is kept because deleting a wrong result you already published is
+> worse than leaving it standing with the reason attached. A corrected run replaces it below.
+>
+> Caught by a reader refusing to believe the result, not by any check in this repository. The
+> question "why would anyone use these harnesses then" was the right one to ask.
 
 | arm | recall | false positives | precision |
 |---|---|---|---|

tests/evals/run-pathways.sh

@@ -167,7 +167,22 @@ extract_findings() {
   [ -n "$review" ] || { printf '[]'; return; }
   ed=$(mktemp -d "$ROOT/extract.XXXXXX")
   printf '%s' "$review" > "$ed/review.txt"
-  out=$( cd "$ed" && timeout 180 claude -p "Read review.txt. It is a code review. Convert every genuine defect it reports into this JSON array and output ONLY the array, no prose: [{\"line\": <integer>, \"category\": \"security|correctness|resource-leak\", \"summary\": \"<one sentence>\"}]. Use the line number the review gives for each defect. If it reports no defects, output []." \
+  # The extractor decides what counts as a defect claim, and it must apply the same bar to every
+  # arm. The first version asked only for "defects" and let the reviewer's own framing through,
+  # which made this benchmark unfair in a way that took an outsider's disbelief to catch.
+  #
+  # The `none` arm was told "do not report style, naming, typing, documentation, or preference",
+  # so it stayed silent on those. The harness arms were given their own /review, which is written
+  # to produce a thorough multi-section report — and it duly reported things like "inconsistent
+  # parameter typing (nit)", "no tests (low)" and "no call sites (info)". Those are not false
+  # claims about defects. They are correct observations that the baseline had been instructed not
+  # to make, and counting them as false positives scored the harnesses for answering a question
+  # nobody asked them to skip.
+  #
+  # So the extractor now drops anything the review itself frames as a nit, a suggestion, a
+  # missing test, a style or typing preference, or an informational note, and keeps only what it
+  # presents as an actual bug. Same instruction, same model, every arm.
+  out=$( cd "$ed" && timeout 180 claude -p "Read review.txt. It is a code review. Extract ONLY findings that the review presents as a genuine BUG, security problem, or resource-handling error in the code — something that would misbehave at runtime. EXCLUDE anything the review frames as a nit, style, naming, typing or annotation preference, a missing test, missing documentation, a suggestion, or an informational note, however it is labelled. Output ONLY a JSON array and no prose: [{\"line\": <integer>, \"category\": \"security|correctness|resource-leak\", \"summary\": \"<one sentence>\"}]. Use the line number the review gives. If it reports no genuine bug, output []." \
          --setting-sources=project --output-format=stream-json --verbose < /dev/null 2>/dev/null \
        | jq -rs '[.[]|select(.type=="assistant")|.message.content[]?|select(.type=="text")|.text]|join("")' 2>/dev/null )
   printf '%s' "$out" | grep -o '\[[^][]*\]' | tail -1