docs(benchmark): Publish Opus 4.8 high comparison - #395
Merged
Conversation
Add fresh traced Claude SDK and Pi high-effort Opus 4.8 benchmark rows. Record the runtime cost analysis showing the Claude SDK cost gap comes from larger input context rather than extra tool fanout. Co-Authored-By: GPT-5 <noreply@anthropic.com>
Add the cross-model comparison between Opus 4.8 high and Sonnet 4.6 to the benchmark analysis data and page copy. Capture the evidence that Opus is cleaner but more selective on this corpus. Co-Authored-By: GPT-5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add fresh traced Opus 4.8 high benchmark rows for Claude SDK and Pi, then update the benchmark page with what the traces show. The page now explains both the runtime cost gap and why Opus 4.8 high trails Sonnet 4.6 on recall.
Traced Benchmark Rows
Adds complete traced corpus rows for
claude-opus-4-8through Claude SDK andanthropic/claude-opus-4-8through Pi, then supersedes the older Opus 4.8 rows.Runtime Analysis
Records sanitized trace aggregates showing Claude SDK cost comes from larger repeated input/cache context, while Pi does more turns and tool executions in the high-effort pair.
Cross-Model Analysis
Adds the 4.8-versus-4.6 explanation: Opus 4.8 high emits fewer and cleaner candidate findings, but exits investigations earlier and misses enough known corpus issues to trail Sonnet 4.6 on recall.