docs(benchmarks): re-score the public claim manifest with the current analyzer - #855
Merged
Merged
Conversation
… analyzer Owner-approved 2026-09-16 (decision recorded 2026-09-02). KO GPT-family catch 44% -> 52%; miss set byte-identical to the 48 reviewed hashes, so the step-1 blinded review carries over unchanged. All sibling-cell moves reported in rebaseline-latest.md; historical tables amended append-only.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Execute the owner-approved re-score of the public claim manifest with the current analyzer. The decision was recorded 2026-09-02 in
docs/research/ko-gpt-miss-review-step1-decision-20260902.md("left as an owner decision"); the owner took it on 2026-09-16. Deterministic only — no model calls, no network, no keys; no analyzer/threshold code changes.What changed
artifacts/rebaseline-2025/rebaseline-2026.scored.public.jsonl— regenerated vianode scripts/rebaseline-build-claim-manifest.mjs(default paths; private inputs staged locally, never committed). 800 records, claim-ready.docs/benchmarks/rebaseline-latest.{md,json}— regenerated vianpm run benchmark:rebaseline:report.ko-gpt-miss-review-step1-decision-20260902.mdand2026-rebaseline.md; historical tables untouched.Every changed figure (old → new)
Carryover verification (mandatory check — PASS)
Miss set of the ko gpt-family cell under the new score vs the 48 reviewed
text_hashes ofko-gpt-miss-review.v1.jsonl: identical sets — 48 vs 48, 0 in-new-not-reviewed, 0 in-reviewed-not-new. The blinded 2-reviewer step-1 evidence carries over unchanged; the 8precondition-violated:document-hotexclusions are exactly the newly caught rows (44% + 8 = 52%).Verification
node scripts/rebaseline-build-claim-manifest.mjsnpm run benchmark:rebaseline:reportnpm run release:checknpm run check:no-private-assetsnpm run lintRollback
Revert this PR; the previous manifest and reports return byte-identical.