Repository navigation
Live eval: --runs, --critic-timeout, and the scorer-tuning rule - #34
Merged
Merged
Conversation
…-tuning rule Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-ups from #27.
--runs Nsends the corpus N times (replies underrun-<k>/) and reports recall per run with mean/min/max, clean-control BLOCKING findings per run, errored cases per run, and each defect's found-rate across runs. One run stays the default and keeps its report shape.--critic-timeout Sreplaces the fixed 170 s per critic call (opencode hit it about once per run in Make the live critic eval measure something #27). A call that times out is still reported as an errored case, never as clean.evals/program.mdanddocs/reference.md: diagnosis terms added after reading replies must come from a backend other than the one being reported, and the reported run uses a frozen scorer — the lesson from Make the live critic eval measure something #27, where synonyms taken from Claude's replies inflated Claude's rescored number.Tested with
--backend rule-auditor --runs 2(aggregation, per-run dirs, report) and a single run (unchanged output). Not re-run against live models here. Local: lint, sphinx, eval 100.00.🤖 Generated with Claude Code