docs(benchmark): Publish Opus 4.8 high comparison

packages/docs/src/content/docs/benchmarking.mdx

@@ -145,27 +145,113 @@ but it did make the run materially more expensive.
 Pi runs without an explicit Warden `--effort` use Pi's default thinking level,
 which is currently medium.
 
-The Opus 4.7 and 4.8 Pi runs currently have an unusual shape. They complete
-cleanly, but many no-finding chunks are very short. In the traced Opus 4.8 Pi
-rerun, chunks with findings averaged 4.3 turns and 3.1k output tokens, while
-no-finding chunks averaged 1.8 turns and 824 output tokens. Eighty-one of 137
-no-finding chunks were one-turn scans, and 32 of 68 missed corpus entries were
-covered by one-turn no-finding scans. The Opus 4.7 run predates trace capture,
-but its low token use and short timings show a similar pattern.
-
-Treat this as a hypothesis, not a model diagnosis. Through Pi, the traces
-suggest many no-finding chunks terminate early. The miss pattern is consistent
-with under-exploration of cross-file authorization and data-boundary invariants,
-not a proven inability to reason about a found issue.
+## Opus 4.8 High: Claude SDK vs Pi
+
+The Opus 4.8 high-effort comparison now has a fresh traced pair. Both rows scan
+the same 156 analysis chunks, complete with zero failed chunks, use Warden's
+finding verifier, and have agent-verified scoring. Pi found 21 of 86 known
+corpus entries and emitted 24 total findings. The Claude SDK found 17 of 86 and
+emitted 17 total findings.
+
+The cost gap is still large, but the trace shape is different from Sonnet 4.6.
+Claude SDK records $79.56 total cost, including $61.08 for scan work. Pi
+records $21.31 total cost, including $17.39 for scan work. On scan work alone,
+Claude SDK cost is 3.5x Pi, input tokens are 4.35x Pi, cache reads are 3.82x
+Pi, and cache creation is 6.10x Pi. Output tokens do not explain the gap: Pi
+actually emitted slightly more scan output tokens than Claude SDK.
+
+The traces do not show Claude SDK doing more tool work. Claude SDK used 375
+turns and 219 tool executions. Pi used 426 turns and 371 tool executions. Pi
+also produced more final findings. The difference is that each Claude SDK turn
+carried much more input context: about 60.0k scan input tokens per turn versus
+12.1k for Pi.
+
+No-finding chunks show the same pattern. Claude SDK no-finding chunks averaged
+2.0 turns, 1.0 tool executions, 118.4k scan input tokens, and $0.35 scan cost.
+Pi no-finding chunks averaged 2.3 turns, 1.8 tool executions, 25.8k scan input
+tokens, and $0.09 scan cost. Finding chunks were similar on turns but not on
+context size: Claude SDK averaged 5.6 turns and $0.76 scan cost; Pi averaged
+5.5 turns and $0.22.
+
+Representative chunks make the point. On `project_rules.py:607-808`, both
+runtimes used one turn and no tools. Claude SDK used 48.4k scan input tokens
+and cost $0.18. Pi used 8.7k scan input tokens and cost $0.03. On
+`replay_counts.py:1-202`, both again used one turn and no tools. Claude SDK
+used 48.3k scan input tokens and cost $0.19. Pi used 8.7k scan input tokens
+and cost $0.04.
+
+The heavier files do not reverse the conclusion. Across
+`integrations/perforce/integration.py`, Claude SDK used 18 turns, 14 tool
+executions, 1.43M scan input tokens, and $2.66 scan cost, producing one final
+finding. Pi used 28 turns, 30 tool executions, 460k scan input tokens, and
+$1.11 scan cost, producing two final findings. Across
+`integrations/msteams/webhook.py`, Claude SDK used 15 turns, 11 tool
+executions, 1.40M scan input tokens, and $3.13 scan cost, producing no final
+finding. Pi used 17 turns, 13 tool executions, 260k scan input tokens, and
+$0.85 scan cost, producing one final finding.
+
+The practical read is that Opus 4.8 on Pi is not cheaper because it skips more
+work. In this high-effort pair, Pi does more turns and more tool executions,
+but each turn carries a much smaller input/cache footprint. Claude SDK's extra
+cost is mostly repeated context volume and verifier context volume, not
+additional tool fanout.
+
+### Why Opus 4.8 high scored below Sonnet 4.6
+
+Opus 4.8 high is also cleanly comparable with the Sonnet 4.6 rows. All four
+runs scan the same 156 chunks, complete with zero failed chunks, use Warden's
+finding verifier, and have agent-verified scoring. The lower score is not a
+coverage problem.
+
+The main difference is recall. On Pi, Sonnet 4.6 found 25 of 86 known corpus
+entries and emitted 32 findings. Opus 4.8 high found 21 of 86 and emitted 24
+findings. Through the Claude SDK, Sonnet 4.6 found 24 of 86 and emitted 32
+findings. Opus 4.8 high found 17 of 86 and emitted 17 findings.
+
+That is not because Opus 4.8 high was noisier. It emitted fewer findings that
+did not match the known corpus: 4 on Pi versus Sonnet's 7, and 2 through the
+Claude SDK versus Sonnet's 8. The tradeoff went the wrong way for this corpus:
+Opus 4.8 high produced a cleaner candidate set, but the set was too small.
+
+The traces support that read. Opus 4.8 high used fewer turns and produced less
+scan output than Sonnet 4.6 in both runtimes. On Pi, Opus 4.8 high used 426
+turns and 316k scan output tokens; Sonnet 4.6 used 628 turns and 365k scan
+output tokens. Through the Claude SDK, Opus 4.8 high used 375 turns and 308k
+scan output tokens; Sonnet 4.6 used 939 turns and 890k scan output tokens. The
+Claude SDK contrast is especially sharp: Opus 4.8 high ended 86 chunks in one
+turn, while Sonnet 4.6 did that on only 8 chunks.
+
+The matched corpus IDs also do not point to one narrow vulnerability category.
+On Pi, Opus 4.8 high recovered 15 of Sonnet's 25 known matches and found 6
+different known issues. Through the Claude SDK, it recovered 10 of Sonnet's 24
+known matches and found 7 different known issues. Across both runtimes, Sonnet
+4.6 found 31 unique known corpus entries, Opus 4.8 high found 23, and only 17
+overlap. Sonnet-only coverage includes invite-token validation, release
+threshold project scoping, OAuth token lifetime and replay checks, webhook
+freshness checks, identity unlinking, and frontend URL handling. Opus-only
+coverage includes OAuth userinfo validation, replay delete scope, preprod
+size-analysis access, relocation retry state leakage, and GitHub Actions output
+injection.
+
+The best supported conclusion is that Opus 4.8 high is more selective under
+this Warden prompt and corpus. It scans every chunk, and it does not fail more
+often. It simply exits many investigations earlier and reports fewer candidate
+issues. That improves apparent precision but misses enough known vulnerabilities
+to trail Sonnet 4.6 on recall.
+
+The older Opus 4.7 and default-medium Opus 4.8 Pi rows still have an unusual
+shape. They complete cleanly, but many no-finding chunks are very short. Treat
+that as historical context for default-runtime behavior, not as the current
+high-effort Opus 4.8 comparison.
 
 ## Current Takeaway
 
 In the current clean, agent-verified rows, GPT 5.5 on Pi found 41 of 86 with
 explicit high effort and 28 of 86 with explicit low effort. Sonnet 4.6 found 25
 of 86 on Pi and 24 of 86 through the Claude SDK. Opus 4.6 on Pi with explicit
-high effort found 23 of 86. Opus 4.8 found 18 of 86 on Pi at Pi's default level,
-14 of 86 on Pi with explicit high effort, and 17 of 86 through the Claude SDK.
-Opus 4.7 on Pi found 6 of 86 at Pi's default level.
+high effort found 23 of 86. Opus 4.8 with explicit high effort found 21 of 86
+on Pi and 17 of 86 through the Claude SDK. Opus 4.8 found 18 of 86 on Pi at
+Pi's default level. Opus 4.7 on Pi found 6 of 86 at Pi's default level.
 
 Use those numbers as a relative comparison for this corpus. They are not a
 general pass rate for Sentry.