Skip to content

REPRODUCE.md

Latest commit

 

History

History
97 lines (65 loc) · 6.52 KB

REPRODUCE.md

File metadata and controls

97 lines (65 loc) · 6.52 KB
Raw

Reproducing the validation experiment

The README/INTRODUCTION headlines (recall 100% / precision 71% vs 36% / root-cause fix 100%) come from a real LLM run, not a synthetic claim. This doc shows how to reproduce it with your own model access. Baseline numbers are in RESULT-REPORT.md.

What the experiment proves

Group Question it answers Script
A Does methodology injection (adversarial verification + confidence ≥80 + 9 security patterns) improve review quality over a bare prompt? run_group_a.py
B Can the bug-fixer (root-cause 5-step) actually fix bugs — including a hard race — and fix root cause not symptom? run_group_b.py

The gold standard is bug_seeds.py: 10 known bugs (easy/medium/hard, across security/correctness/boundary). Group B fixes 5 of them.

Prerequisites

You need access to a model that speaks the Anthropic messages API (a relay/gateway with x-api-key auth works; the original run used deepseek-v4 for review/fix and qwen3.7 as judge, both via relay). Two cost tiers:

  • A strong/"reasoning" model for review + fix (group A + B) — set as ANTHROPIC_DEFAULT_SONNET_MODEL
  • A cheaper model for the LLM-judge (group B) — set as ANTHROPIC_DEFAULT_HAIKU_MODEL

A reasoning model is not required — any Anthropic-API-compatible model works. If yours isn't a reasoning model, results still reproduce (the thinking-block handling in llm_call.py is a no-op for non-reasoning models).

Steps

cd loop-engine/experiment

# 1. Set model access (relay/gateway with x-api-key auth)
export ANTHROPIC_BASE_URL="https://your-relay.example.com"
export ANTHROPIC_AUTH_TOKEN="sk-..."
export ANTHROPIC_DEFAULT_SONNET_MODEL="deepseek-v4"   # review + fix (strong)
export ANTHROPIC_DEFAULT_HAIKU_MODEL="qwen3.7"        # judge (cheaper)

# 2. Smoke: one call works + auth is x-api-key (not Bearer)
python -c "from llm_call import make_call_fn; c=make_call_fn(); print(c('ping', None)[:80])"

# 3. Group A: bare vs enhanced reviewer (≈40 calls)
python run_group_a.py

# 4. Group B: fix + judge (≈10 calls)
python run_group_b.py

Total: ~50 calls, slowest single call ~30s (reasoning thinking).

Expected results (baseline to compare against)

Metric Bare Enhanced Decision gate Pass?
Recall 100% (10/10) 100% (10/10) ≥70% ✅
Findings reported 28 14 — enhanced −50%
False positives 18 4 — −78%
Precision 36% 71% ≤30% FP ✅ (+35pp)
Group B bug Difficulty Fixed correctly Root-cause fix
B01 SQL injection easy ✓ ✓ parameterized query
B04 null easy ✓ ✓ explicit entry guard
B06 discount logic medium ✓ ✓ add-not-subtract
B08 race hard ✓ ✓ double-checked locking (DCL)
B10 divide-by-zero easy ✓ ✓ denominator check

If your numbers are far off, see "Gotchas" below and RESULT-REPORT.md §四.

Gotchas (already solved in llm_call.py — listed so you know what to check)

These are the production-integration pitfalls the original run exposed. They are handled in llm_call.py; if you adapt it, preserve them:

  1. Reasoning-model thinking blocks — content[0] is a thinking block, not text; max_tokens gets eaten by thinking. Fix: parse only type == "text" blocks; max_tokens=4096.
  2. Chinese vs English keyword mismatch — the model may answer in Chinese ("空值") while expected is English ("null") → false miss. Fix: SYNONYMS map in run_group_a.py (extend if your model uses other phrasings).
  3. Relay auth header unknown — some relays want x-api-key, others Bearer. Fix: the Step-2 smoke confirms x-api-key works before the full run.

Reproducing without a model

You cannot reproduce the LLM numbers without model access — that's the point (it's a real run, not a mock). But you can validate the gold standard and parsing logic offline:

# bug seeds parse correctly
python -c "from bug_seeds import SEEDS; print(len(SEEDS), 'seeds'); [print(s.id, s.difficulty, s.expected_finding) for s in SEEDS]"

# JSON extraction is robust
python -c "from llm_call import extract_json_array; print(extract_json_array('\`\`\`json\n[{\"a\":1}]\n\`\`\`'))"

Improving review precision (calibration + false-positive memory)

The headline 71% precision is the enhanced reviewer at a self-reported confidence ≥80 gate. Two improvements are implemented:

  • Calibrate the threshold from data — calibrate_threshold.py runs the reviewer on the 10-seed gold standard with confidence per finding (no internal filter), sweeps thresholds 60→95, and prints a precision@threshold curve. It recommends the lowest threshold that hits your target precision (default 90%). Run: python calibrate_threshold.py --target-precision 0.90. Requires the same env vars as Group A.
  • Ablate the structural levers (③④) — run_precision_ablation.py measures the real precision gain of the finding-verifier (③) and skeptic-consensus (④) against a baseline enhanced reviewer, on the 10-seed gold standard. It reuses the production _merge_findings / _apply_verdicts so it tests the real logic. Conditions: baseline / ③ verify / ④ consensus / ③+④. Run: python run_precision_ablation.py. Same env vars as Group A.
  • Cross-session false-positive memory — qa_loop.py now persists findings the bug-fixer marked skipped into .qa-memory.json under known_false_positive, and the review phase suppresses those signatures on subsequent runs (same mechanism as known_fixed). Curation: humans can edit .qa-memory.json directly. Backward-compatible with old memory files.

Note: threshold calibration raises single-run precision; the false-positive memory reduces repeated false positives across runs (team-level accumulation), not single-run precision. The ablation script tells you whether ③/④ beat a self-reported confidence threshold (they bypass LLM over-confidence, so they help most in the "pessimistic" case where calibration alone fails).

Notes on variance

LLM runs are stochastic. Expect ±1 finding per seed across runs; the direction of the effect (enhanced ≪ bare false-positives, root-cause fixes) is stable. If recall drops below ~70% on a seed, that's a model-quality issue — try a stronger SONNET model, not a code change.