Benchmark v2 — schema-first, multi-source, rubric-based judging - #162
Merged
Conversation
Complete rework of the benchmark system:
Data layout
- Split monolithic languages.yaml / reference_texts.yaml into per-entry YAMLs
under benchmark/data/{languages,reference_texts/<lang>}/
- 17 reference texts across 6 source languages (en, fr, de, es, ja, ko, zh-Hans)
- 45 target languages (added en for back-translation pairs)
- New benchmark/data/submissions/ for community contributions
Schemas
- JSON Schema files for submission, language, reference_text, model
- Strict validation enforced both locally (CLI) and in CI
CLI
- New --pairs flag (source:target) replacing en-source-only --languages
- New --no-evaluate flag to skip auto-judge for manual evaluation
- Poe added as a translation provider
- New cmd_submit and cmd_aggregate_submissions
Manual evaluation pipeline
- scripts/dump_for_evaluation.py — dumps a brief for a human/LLM judge
- scripts/apply_evaluations.py — patches a run with externally-produced scores
- scripts/submission_to_run.py — reconstruct a runnable JSON from a submission
for re-evaluation with a stronger judge later
Aggregation + wiki
- benchmark/aggregator.py merges multiple submissions on (model, text, lang)
via median; surfaces n_obs and verified/self-reported badges
- Wiki templates updated to display Obs and Verified columns
GitHub Actions
- validate-submission.yml — runs schema validation + replay sampling on PRs
touching benchmark/data/submissions/
- publish-wiki.yml — regenerates and pushes the wiki on merge to main
Documentation
- docs/JUDGE_RUBRIC.md — formal scoring rubric (v1) with anchored 1-10 scale,
penalty table, hard ceilings, and dispersion rule for cross-model judging
- docs/BENCHMARK_WORKFLOW.md — end-to-end procedure for maintainers
- docs/CONTRIBUTING_BENCHMARK.md — contributor guide
Notes
- Wiki workflow gated on push to main; nothing fires from this feature branch
- benchmark_results/ now gitignored (local artefacts only)
- .claude/ commands stay local; users opt-in via gitignore exception
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/archive_v1_wiki.py One-shot script that clones the wiki, renames every existing page with the `Archive-` prefix, rewrites internal cross-page links to keep the archived content browsable on its own, generates `Archive-Index.md`, then commits and pushes. Supports --dry-run, --no-push, --message, --wiki-url. home.md.j2 Top-of-home banner: "Methodology v2 — judge: Claude Opus 4.7 ... Previous benchmark archived under Archive-Index". Browse section includes a link to Archive-Index. docs/BENCHMARK_WORKFLOW.md New section "Switching the live wiki from v1 to v2" documenting the archive procedure (dry-run, push, cleanup behaviour). Confirms the publish-wiki workflow's cleanup globs leave Archive-* pages untouched. .gitignore /benchmark_results/ and .wiki_repo_archive/ excluded. Smoke-tested against the live wiki: 49 v1 pages would be archived (1 Home, 2 cross-cutting tables, 1 Benchmark-System, 19 Language-*, 26 Model-*). Link rewriter preserves external URLs and rewrites only page-name targets that match an archived page. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…-opus-4-7-rubric-v1)
45 translations evaluated:
- en→{zh-Hans, es, fr, vi}: 10 each
- ja→en, ko→en, ja→zh-Hans: 1 each
- zh-Hans→en: 2
Per-pair averages (overall, rubric v1):
ja→en 9.00 (excellent: kokoro)
zh-Hans→en 8.25 (lu_xun, hu_shi)
en→es 7.80
en→fr 7.70
en→vi 7.70
ja→zh-Hans 7.50
en→zh-Hans 7.40 (recurring untranslated-Latin / mixed-script issues)
ko→en 6.50 (Jongno hallucination on unsu_jouen_nal)
Global average: 7.67. Distribution well spread across 6.0-8.5 thanks to the
formal rubric (start at 10, deduct per detected issue, hard cap 9.0 without
human reference).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replay Report
|
| text | lang | status | details |
|---|---|---|---|
| kokoro | zh-Hans | ok | lang_detected=zh-cn, lang_ok=True, length_ok=True |
| study_scarlet | zh-Hans | suspicious | lang_detected=zh-cn, lang_ok=True, length_ok=False |
| dorian_gray | zh-Hans | suspicious | lang_detected=zh-cn, lang_ok=True, length_ok=False |
| study_scarlet | es | ok | lang_detected=es, lang_ok=True, length_ok=True |
hydropix
marked this pull request as ready for review
May 9, 2026 21:54
losskot
pushed a commit
to losskot/TranslateBooksWithLLMs
that referenced
this pull request
Jul 10, 2026
…opix#162) * feat(benchmark): v2 — schema-first, multi-source, rubric-based judging Complete rework of the benchmark system: Data layout - Split monolithic languages.yaml / reference_texts.yaml into per-entry YAMLs under benchmark/data/{languages,reference_texts/<lang>}/ - 17 reference texts across 6 source languages (en, fr, de, es, ja, ko, zh-Hans) - 45 target languages (added en for back-translation pairs) - New benchmark/data/submissions/ for community contributions Schemas - JSON Schema files for submission, language, reference_text, model - Strict validation enforced both locally (CLI) and in CI CLI - New --pairs flag (source:target) replacing en-source-only --languages - New --no-evaluate flag to skip auto-judge for manual evaluation - Poe added as a translation provider - New cmd_submit and cmd_aggregate_submissions Manual evaluation pipeline - scripts/dump_for_evaluation.py — dumps a brief for a human/LLM judge - scripts/apply_evaluations.py — patches a run with externally-produced scores - scripts/submission_to_run.py — reconstruct a runnable JSON from a submission for re-evaluation with a stronger judge later Aggregation + wiki - benchmark/aggregator.py merges multiple submissions on (model, text, lang) via median; surfaces n_obs and verified/self-reported badges - Wiki templates updated to display Obs and Verified columns GitHub Actions - validate-submission.yml — runs schema validation + replay sampling on PRs touching benchmark/data/submissions/ - publish-wiki.yml — regenerates and pushes the wiki on merge to main Documentation - docs/JUDGE_RUBRIC.md — formal scoring rubric (v1) with anchored 1-10 scale, penalty table, hard ceilings, and dispersion rule for cross-model judging - docs/BENCHMARK_WORKFLOW.md — end-to-end procedure for maintainers - docs/CONTRIBUTING_BENCHMARK.md — contributor guide Notes - Wiki workflow gated on push to main; nothing fires from this feature branch - benchmark_results/ now gitignored (local artefacts only) - .claude/ commands stay local; users opt-in via gitignore exception Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(benchmark): wiki archive path for v1 -> v2 transition scripts/archive_v1_wiki.py One-shot script that clones the wiki, renames every existing page with the `Archive-` prefix, rewrites internal cross-page links to keep the archived content browsable on its own, generates `Archive-Index.md`, then commits and pushes. Supports --dry-run, --no-push, --message, --wiki-url. home.md.j2 Top-of-home banner: "Methodology v2 — judge: Claude Opus 4.7 ... Previous benchmark archived under Archive-Index". Browse section includes a link to Archive-Index. docs/BENCHMARK_WORKFLOW.md New section "Switching the live wiki from v1 to v2" documenting the archive procedure (dry-run, push, cleanup behaviour). Confirms the publish-wiki workflow's cleanup globs leave Archive-* pages untouched. .gitignore /benchmark_results/ and .wiki_repo_archive/ excluded. Smoke-tested against the live wiki: 49 v1 pages would be archived (1 Home, 2 cross-cutting tables, 1 Benchmark-System, 19 Language-*, 26 Model-*). Link rewriter preserves external URLs and rewrites only page-name targets that match an archived page. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * submit(benchmark): gemma3:27b on the 8 canonical pairs (judge: claude-opus-4-7-rubric-v1) 45 translations evaluated: - en→{zh-Hans, es, fr, vi}: 10 each - ja→en, ko→en, ja→zh-Hans: 1 each - zh-Hans→en: 2 Per-pair averages (overall, rubric v1): ja→en 9.00 (excellent: kokoro) zh-Hans→en 8.25 (lu_xun, hu_shi) en→es 7.80 en→fr 7.70 en→vi 7.70 ja→zh-Hans 7.50 en→zh-Hans 7.40 (recurring untranslated-Latin / mixed-script issues) ko→en 6.50 (Jongno hallucination on unsu_jouen_nal) Global average: 7.67. Distribution well spread across 6.0-8.5 thanks to the formal rubric (start at 10, deduct per detected issue, hard cap 9.0 without human reference). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: hydropix <5154373+hydropix@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Complete rework of the benchmark system. Lives in parallel with v1 — the wiki workflow is gated on push to main, so this branch does not affect the live wiki.
What's in
Data layout
benchmark/data/benchmark/data/submissions/for community contributionsSchemas + validation
scripts/validate_submission.pyfor local + CI checksCLI
--pairsflag (source:target) replaces en-only--languages--no-evaluatefor manual judging workflowssubmitandaggregate-submissionscommandsManual evaluation pipeline
scripts/dump_for_evaluation.py— dumps a brief for a human/LLM judgescripts/apply_evaluations.py— patches a run with judge scoresscripts/submission_to_run.py— re-evaluate a historical submissionAggregation + wiki
benchmark/aggregator.pymerges submissions via median, exposesn_obsand verified/self-reported badgesGitHub Actions
validate-submission.yml— schema validation + replay sampling on PRpublish-wiki.yml— regenerates wiki on merge to mainDocumentation
docs/JUDGE_RUBRIC.md— formal scoring rubric v1 (anchored 1-10 scale, penalty table, hard ceilings)docs/BENCHMARK_WORKFLOW.md— end-to-end proceduredocs/CONTRIBUTING_BENCHMARK.md— contributor guideStatus
Draft. Iterating on:
Test plan
--pairsand--no-evaluateflags work end-to-endsubmission_to_run.pyreconstructs a re-evaluable run JSON from a submissionn_obs=2and median scores🤖 Generated with Claude Code