Skip to content

PRECISION-TUNING.md

Latest commit

 

History

History
113 lines (71 loc) · 6.28 KB

PRECISION-TUNING.md

File metadata and controls

113 lines (71 loc) · 6.28 KB
Raw

Review Precision Tuning

How to raise the code-reviewer's precision from the baseline 71% (recall 100%) toward 90%+, and the honest tradeoffs. All levers are default-off; nothing changes until you opt in.

The baseline and the goal

Validated baseline (50 real LLM calls, 10-bug gold standard): recall 100% / precision 71% / root-cause fix 100%. Precision 71% means ~4 false positives per 14 findings — review time wasted on noise. The goal is to cut false positives without losing the bugs you'd catch.

Precision and recall are a seesaw. Recall is 100% with margin, so there's room to trade. But the right target depends on the scenario:

Scenario Optimize for Suggested preset
CI gate (blocks merge) recall (don't miss bugs) low or balanced
Pre-PR self-review (suggestions) precision (less noise) strict
Critical system gate recall + auditability balanced + confidence gate

One-click presets

Set review.precision_preset in config/qa-loop.yaml. Explicit per-field config overrides the preset (preset only fills fields you didn't set).

review:
  precision_preset: balanced   # low | balanced | strict
Preset ② diff-anchor ③ verify ④ skeptic consensus_min Profile
low off off off 1 (union) recall-first, ~71% precision / 100% recall (default)
balanced loose (±3) on off 1 cuts pre-existing + speculative FPs, keeps recall
strict strict on on 2 (consensus) precision-max, lowest recall — self-review, not gates

Start with balanced for most teams. Move to strict only for suggestion-only flows where false positives are the pain. Use low for blocking gates where a missed bug costs more than noise.

The five levers (what each does, when to use)

① Confidence threshold calibration (measure, then set)

The reviewer self-rates confidence 0–100 and reports ≥80. But LLMs are over-confident, so 80 is a guess. Calibrate it from data:

cd loop-engine/experiment
python calibrate_threshold.py --target-precision 0.90

This runs the reviewer on the 10-seed gold standard with per-finding confidence (no internal filter), sweeps thresholds 60→95, prints a precision@threshold curve, and recommends the lowest threshold hitting your target. Cheapest lever. Try this first.

  • If a threshold hits 90% precision → use it, you're done (no need for ②③④).
  • If no threshold hits 90% (FPs are as confident as real bugs) → self-rated confidence is useless; switch to the structural levers ②③④.

② Diff anchoring (real-PR #1 lever)

review.diff_anchor: true runs git diff on the scope, parses added lines, and drops findings not anchored to a changed line. Kills "pre-existing issue flagged as new bug" — the top false-positive source in real PRs.

  • anchor_mode: loose (default): keep findings within ±anchor_context (3) lines of a change — tolerates bugs triggered by adjacent lines.
  • anchor_mode: strict: finding must land on a changed line.
  • No-ops when not in a git repo or no diff (won't over-kill).

Cannot be ablated on the gold standard (whole-file review, no diff) — measure it on a real merged PR: run with and without ②, compare FP counts. This is usually the single biggest real-world gain.

③ Finding verifier (independent refutation)

review.verify_findings: true — between Review and Fix, an independent finding-verifier agent tries to refute each raw finding (default kill). Drops false positives before they waste a fix. Needs a finding-verifier entry in agent_models (reuse the confidence-evaluator model). Bypasses self-rated confidence (independent second opinion).

④ Skeptic perspective + consensus

review.skeptic: true adds a generalist refutation-first perspective running in parallel with the 4 specialized perspectives. review.consensus_min: 2 requires ≥2 perspectives to report a finding for it to survive. Real bugs get reported by both the specialized perspective and the skeptic → survive. FPs reported by only one → dropped.

⑤ False-positive memory (cross-session)

Already on by default — findings the bug-fixer marks skipped are persisted to .qa-memory.json under known_false_positive and suppressed on future runs. Team-level accumulation: the same FP stops recurring. Humans can curate the file directly. Doesn't change single-run precision; reduces repeated noise.

How to measure the real gain

cd loop-engine/experiment

# 1. Can self-rated confidence alone hit 90%? (lever ①)
python calibrate_threshold.py --target-precision 0.90

# 2. How much do ③④ add over baseline? (structural levers)
python run_precision_ablation.py
# prints precision/recall/FP for: baseline / ③ verify / ④ consensus / ③+④

# 3. ② only matters on real PRs — run on a merged PR with review.diff_anchor: true

The ablation reuses the production _merge_findings / _apply_verdicts / _anchor_filter code, so its numbers reflect real behavior, not a reimplementation.

Recommended path

  1. Run ① (calibrate_threshold.py). If it hits 90% → set the threshold, done.
  2. If ① stalls → enable precision_preset: balanced (② + ③), re-measure with the ablation.
  3. For self-review flows wanting max precision → strict, accept the recall cost.
  4. For real PRs → ② is the big lever; measure on a merged PR.
  5. Tune over weeks → ⑤ accumulates; revisit thresholds as the team's FP signature evolves.

Honest notes

  • Don't chase 100% precision. LLM review has irreducible uncertainty; pushing too hard makes the reviewer "report nothing". 90% is a realistic, useful target.
  • Measure, don't assume. The levers' real impact depends on your model and codebase. The scripts exist so you don't have to guess.
  • strict is not for blocking gates. A missed Critical in a gate costs more than a few false positives. For gates, prefer low/balanced and let humans adjudicate.

See also: experiment/REPRODUCE.md (reproduce the validation), CHANGELOG.md (lever history).