Live eval: --runs, --critic-timeout, and the scorer-tuning rule

CHANGELOG.md

@@ -8,6 +8,8 @@
 
 ### Added
 
+- Add `--runs N` and `--critic-timeout` to `./install.sh eval --live-llm`: repeat the corpus and report recall per run, its mean and range, clean-control findings per run and how often each defect was found; and give slow backends more than the fixed 170 s per critic call ([#34](https://github.com/forkrul/damascus_development/pull/34))
+- Document the scorer-tuning rule in `evals/program.md`: diagnosis terms added after reading replies come from a backend other than the one being reported, and the reported run uses a frozen scorer ([#34](https://github.com/forkrul/damascus_development/pull/34))
 - Add read-only review agents `temper-critic` (Read, Glob, Grep), `hone-critic` (plus Bash, for repros run in a throwaway worktree) and `review-judge`. Temper and hone now write each round's inputs to a bundle file and dispatch these agents on it, so critics can't edit the work under review and stay blind to earlier rounds and ratings. `tools:` is enforced only by harnesses that honour it; elsewhere the agents' no-edit rule still binds ([#33](https://github.com/forkrul/damascus_development/pull/33))
 
 ### Fixed

docs/reference.md

@@ -214,6 +214,7 @@ skill pages say exactly where each one is read.
 ./install.sh eval                                   # J_damascus, 0–100; exits 1 below 95
 ./install.sh eval --json --record                   # JSON report; append a row to evals/results.tsv
 ./install.sh eval --live-llm --backend <backend>    # also benchmark a real critic model
+./install.sh eval --live-llm --backend <backend> --runs 3 --critic-timeout 300   # repeat; report the spread
 ./install.sh eval --check-corpus                    # self-check the live benchmark's scorer and corpus
 ```
 
@@ -231,7 +232,11 @@ says what is wrong in words that don't occur in the prompt, so a model that echo
 input scores 0. The report gives recall, blocking findings on the clean control (false
 positives), the echo baseline, and any case where the backend failed. Every prompt and
 raw reply is saved under `.damascus/eval-runs/<time>-<backend>/` (`--out-dir` to
-override) so a score can be checked by hand. Treat one run as a rough reading: replies
-vary from run to run, and the scorer's term lists are still being tuned. A
+override) so a score can be checked by hand. Replies vary from run to run, so
+`--runs N` repeats the corpus and reports recall per run, its mean and range, and how
+often each defect was found; `--critic-timeout` (default 170 s) bounds each critic call,
+and a call that runs out is reported as an errored case, not a clean one. The scorer's
+term lists are still being tuned; a term added after reading replies must come from a
+backend other than the one being reported. A
 `--backend rule-auditor` run scores the regex checker, whose patterns were written
 against this corpus, so its result says nothing about models.

evals/program.md

@@ -33,7 +33,7 @@ Run the immutable evaluation harness from the repository root:
 3. **Scientific Protocol & Model-Agnosticism Invariants ($25\text{ pts}$):** Verifies BLOCKING-only capture-recapture stratification ($D_B, m_B$), vacuity falsification checks, M0 micro-temper, composable `.damascus/addenda/<stage>.md` hooks, `model: inherit` across all agents, and zero deprecated Pydantic v1 / SQLAlchemy 1.x / naive UTC patterns.
 4. **Karpathy Simplicity Regularizer ($15\text{ pts}$):** Rewards concise, local-model-friendly prompts (penalizing any `agents/*.md` file exceeding $150$ lines or `skills/*/SKILL.md` file exceeding $260$ lines).
 
-**Live critic benchmark (`--live-llm`, reported separately, not part of $J_{\text{damascus}}$).** Every case in `evals/corpus/seeded_cases.json`, the clean control included, goes to a real critic. A seeded defect counts as found only when one `[BLOCKING]` finding names it (an `anchors` term: its ID, task or symbol) **and** says what is wrong with a `diagnosis` term that occurs nowhere in the prompt. Words the critic could copy from its input earn nothing, so echoing the prompt scores 0. The report gives recall, BLOCKING findings on the clean control (false positives), other BLOCKING findings on seeded cases, and the echo baseline. Each prompt and raw reply is saved under `.damascus/eval-runs/` for audit. `--check-corpus` self-checks the scorer and corpus in CI: echo scores 0, every defect is matched by its own `description` and by no sibling's, and NIT lines never count. When adding a case, give every defect anchors and diagnosis terms that pass that check.
+**Live critic benchmark (`--live-llm`, reported separately, not part of $J_{\text{damascus}}$).** Every case in `evals/corpus/seeded_cases.json`, the clean control included, goes to a real critic. A seeded defect counts as found only when one `[BLOCKING]` finding names it (an `anchors` term: its ID, task or symbol) **and** says what is wrong with a `diagnosis` term that occurs nowhere in the prompt. Words the critic could copy from its input earn nothing, so echoing the prompt scores 0. The report gives recall, BLOCKING findings on the clean control (false positives), other BLOCKING findings on seeded cases, and the echo baseline. Each prompt and raw reply is saved under `.damascus/eval-runs/` for audit. `--check-corpus` self-checks the scorer and corpus in CI: echo scores 0, every defect is matched by its own `description` and by no sibling's, and NIT lines never count. When adding a case, give every defect anchors and diagnosis terms that pass that check. `--runs N` sends the corpus N times and reports recall per run, its mean and range, and how often each defect was found; `--critic-timeout` sets how long one critic call may take (default 170 s). **Tuning the scorer:** a diagnosis term added after reading replies must come from a backend you are *not* about to report; then rerun the reported backend with the scorer frozen and quote only that run. Terms taken from the replies being scored fit the scorer to that model.
 
 ---
 

evals/run_eval.py

@@ -560,14 +560,48 @@ def check_corpus() -> list[str]:
     return errors
 
 
-def run_live_llm_corpus_eval(backend: str = "auto", out_dir: Path | None = None) -> dict[str, Any]:
+def run_live_llm_corpus_eval(
+    backend: str = "auto", out_dir: Path | None = None, runs: int = 1, timeout_s: float = 170.0
+) -> dict[str, Any]:
     """Optional live LLM benchmark over every corpus case, including the clean control.
 
-    Each prompt and raw reply is saved under `out_dir` so the score can be audited.
+    Each prompt and raw reply is saved under `out_dir` so the score can be audited. With
+    `runs` > 1 the corpus is sent `runs` times (replies under `run-<k>/`) and the report
+    adds the spread: replies vary from run to run, so one run is a rough reading.
     """
-    seeded, clean = load_corpus_cases()
     ts = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
     out_dir = out_dir or LIVE_RUNS_DIR / f"{ts}-{backend}"
+    if runs <= 1:
+        return live_llm_corpus_run(backend, out_dir, timeout_s)
+    per_run = [live_llm_corpus_run(backend, out_dir / f"run-{k}", timeout_s) for k in range(1, runs + 1)]
+    recalls = [r["recall"] for r in per_run]
+    found_rate: dict[str, float] = {}
+    for r in per_run:
+        for c in r["cases"]:
+            for d in c["matched"]:
+                found_rate[d] = found_rate.get(d, 0) + 1 / runs
+    report = {
+        "backend": backend,
+        "runs": len(per_run),
+        "answered_by": per_run[0]["answered_by"] if len({r["answered_by"] for r in per_run}) == 1 else "mixed",
+        "total_defects": per_run[0]["total_defects"],
+        "matched_defects_per_run": [r["matched_defects"] for r in per_run],
+        "recall_mean": round(sum(recalls) / len(recalls), 2),
+        "recall_min": min(recalls),
+        "recall_max": max(recalls),
+        "clean_false_positives_per_run": [r["clean_false_positives"] for r in per_run],
+        "errored_cases_per_run": [r["errored_cases"] for r in per_run],
+        "defect_found_rate": {d: round(v, 2) for d, v in sorted(found_rate.items())},
+        "echo_baseline": per_run[0]["echo_baseline"],
+        "replies_dir": str(out_dir),
+    }
+    (out_dir / "report.json").write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
+    return report
+
+
+def live_llm_corpus_run(backend: str, out_dir: Path, timeout_s: float) -> dict[str, Any]:
+    """One pass over the corpus; see run_live_llm_corpus_eval."""
+    seeded, clean = load_corpus_cases()
     out_dir.mkdir(parents=True, exist_ok=True)
     cmd_tail = ["--no-fallback"] if backend not in ("auto", "rule-auditor") else []
     case_results: list[dict[str, Any]] = []
@@ -577,8 +611,8 @@ def run_live_llm_corpus_eval(backend: str = "auto", out_dir: Path | None = None)
         role = "critic-b" if "diff" in case else "critic-a"
         proc = run_cmd(
             [sys.executable, str(SCRIPTS_DIR / "damascus-agent-run"), "--backend", backend, "--role", role,
-             "--timeout", "170", "--prompt", prompt, *cmd_tail],
-            timeout_s=180.0,
+             "--timeout", str(timeout_s), "--prompt", prompt, *cmd_tail],
+            timeout_s=timeout_s + 10,
         )
         (out_dir / f"{case['id']}.prompt.txt").write_text(prompt, encoding="utf-8")
         (out_dir / f"{case['id']}.reply.txt").write_text(proc.stdout + (f"\n--- stderr ---\n{proc.stderr}" if proc.stderr else ""), encoding="utf-8")
@@ -685,6 +719,18 @@ def main(argv: list[str] | None = None) -> int:
         default=None,
         help="Where --live-llm saves each prompt and raw reply (default: .damascus/eval-runs/<UTC time>-<backend>/).",
     )
+    parser.add_argument(
+        "--runs",
+        type=int,
+        default=1,
+        help="With --live-llm, send the corpus this many times and report the spread (default: 1).",
+    )
+    parser.add_argument(
+        "--critic-timeout",
+        type=float,
+        default=170.0,
+        help="With --live-llm, seconds each critic call may take before it counts as an errored case (default: 170).",
+    )
     parser.add_argument(
         "--check-corpus",
         action="store_true",
@@ -724,7 +770,9 @@ def main(argv: list[str] | None = None) -> int:
     }
 
     if args.live_llm:
-        report["live_llm_corpus"] = run_live_llm_corpus_eval(args.backend, args.out_dir)
+        report["live_llm_corpus"] = run_live_llm_corpus_eval(
+            args.backend, args.out_dir, runs=args.runs, timeout_s=args.critic_timeout
+        )
 
     if args.record:
         append_results_tsv(
@@ -744,7 +792,23 @@ def main(argv: list[str] | None = None) -> int:
         print(f"=== Damascus Evaluation Harness (J_damascus = {total_score:.2f} / 100.00) ===")
         for line in report["notes"]:
             print(f"  {line}")
-        if args.live_llm:
+        if args.live_llm and "runs" in report["live_llm_corpus"]:
+            llm_res = report["live_llm_corpus"]
+            label = "LIVE-LLM" if llm_res["answered_by"] == "model" else "RULE-AUDITOR"
+            total = llm_res["total_defects"]
+            print(
+                f"  [{label}] {llm_res['runs']} runs — recall per run: "
+                + ", ".join(f"{m}/{total}" for m in llm_res["matched_defects_per_run"])
+                + f" (mean {llm_res['recall_mean'] * 100:.0f}%, min {llm_res['recall_min'] * 100:.0f}%, max {llm_res['recall_max'] * 100:.0f}%)"
+            )
+            print("    clean-control BLOCKING findings per run: " + ", ".join(map(str, llm_res["clean_false_positives_per_run"])))
+            for k, errs in enumerate(llm_res["errored_cases_per_run"], 1):
+                if errs:
+                    print(f"    [WARN] run {k}: backend failed on {', '.join(errs)}; that run's totals are incomplete")
+            rates = llm_res["defect_found_rate"]
+            print("    found in share of runs: " + ", ".join(f"{d} {r * 100:.0f}%" for d, r in rates.items()) + (" (others never)" if len(rates) < total else ""))
+            print(f"    prompts and raw replies: {llm_res['replies_dir']}/run-<k>/")
+        elif args.live_llm:
             llm_res = report["live_llm_corpus"]
             label = "LIVE-LLM" if llm_res["answered_by"] == "model" else "RULE-AUDITOR"
             note = "" if label == "LIVE-LLM" else " (deterministic pattern checks, not a model review)"