diff --git a/CHANGELOG.md b/CHANGELOG.md index 0b33a05..05e296b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,8 @@ ### Added +- Add `--runs N` and `--critic-timeout` to `./install.sh eval --live-llm`: repeat the corpus and report recall per run, its mean and range, clean-control findings per run and how often each defect was found; and give slow backends more than the fixed 170 s per critic call ([#34](https://github.com/forkrul/damascus_development/pull/34)) +- Document the scorer-tuning rule in `evals/program.md`: diagnosis terms added after reading replies come from a backend other than the one being reported, and the reported run uses a frozen scorer ([#34](https://github.com/forkrul/damascus_development/pull/34)) - Add read-only review agents `temper-critic` (Read, Glob, Grep), `hone-critic` (plus Bash, for repros run in a throwaway worktree) and `review-judge`. Temper and hone now write each round's inputs to a bundle file and dispatch these agents on it, so critics can't edit the work under review and stay blind to earlier rounds and ratings. `tools:` is enforced only by harnesses that honour it; elsewhere the agents' no-edit rule still binds ([#33](https://github.com/forkrul/damascus_development/pull/33)) ### Fixed diff --git a/docs/reference.md b/docs/reference.md index bcf1f09..6410288 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -214,6 +214,7 @@ skill pages say exactly where each one is read. ./install.sh eval # J_damascus, 0–100; exits 1 below 95 ./install.sh eval --json --record # JSON report; append a row to evals/results.tsv ./install.sh eval --live-llm --backend # also benchmark a real critic model +./install.sh eval --live-llm --backend --runs 3 --critic-timeout 300 # repeat; report the spread ./install.sh eval --check-corpus # self-check the live benchmark's scorer and corpus ``` @@ -231,7 +232,11 @@ says what is wrong in words that don't occur in the prompt, so a model that echo input scores 0. The report gives recall, blocking findings on the clean control (false positives), the echo baseline, and any case where the backend failed. Every prompt and raw reply is saved under `.damascus/eval-runs/