Decision-quality eval: score how well each model chooses - #109
Open
shaumik wants to merge 10 commits into
Open
Conversation
…racle
Score how well a side chose, not just whether it won. A live battle stored
every turn's engine state, and the engine is a pure function of (state,
actions), so each turn re-simulates to recover the exact action played
(engine.LegalActions x ResolveTurn until the stored next state reproduces
byte-for-byte). Then a depth-limited expectimax oracle scores every legal
action from the identical fog-of-war view (ai.MakeView), and the value gap
between the played action and the best is the decision's regret.
- ai.ExpectimaxAgent.ScoreActions exposes the per-action maximin values
searchRoot already computes (additive; Decide unchanged), so callers can
measure regret rather than only agree/disagree.
- eval.ScoreDecisions walks stored turns, recovers each free choice on a side,
and returns per-decision {chosen, best, agree, regret, blunder} plus a count
of skipped faint/replacement turns (v1 scores clean turns; coverage reported,
not hidden).
- cmd/decision-eval is a spike over a battle JSON export (db-replay's shape).
Validated on real stored battles: recovery re-simulates exactly, and regret
correctly separates real mistakes from picking an equal-value alternative
(binary agreement counts equal moves as misses; regret scores them 0).
The naive recovery replayed one ResolveTurn and compared to the stored next state, so any turn that fainted a Pokémon (resolution stops in the replacement phase) couldn't be matched and was skipped — about half of all turns. settle now drives the resolved state through its forced replacements, searching the legal replacement picks (a cascade recurses), so faint turns reproduce the stored next state and are recovered like any other. On real battles this takes coverage from ~40-65% to ~100% of choosing turns. Test plays a full heuristic mirror that KOs and replaces, then asserts every choosing turn — faint turns included — re-derives its actions exactly.
Roll per-decision regret up to a per-model table: blunder rate (headline), median regret, match rate, win rate. AggregateByModel folds scored battles; decision-eval -manifest scores a batch and prints/JSON-emits the table; decision-report.sh joins bid= attribution to Postgres exports and drives it. Regret is heavy-tailed (missed lethal ~= winValue): median ignores the tail, mean is winsorized at a cap. Test pins the arithmetic and the tail handling. On the fresh attributed batch, blunder rate tracks win rate across all four models (Opus < Gemini < Sonnet < Haiku), so decision quality predicts the outcome win/loss alone hides.
…atch A stalled agent session reads forever at 0% CPU (Opus in particular walls out this way — a hung model/streaming connection). With no timeout, one such game blocked the whole attributed batch for 13+ hours. Wrap claude -p in a portable watchdog (macOS has no timeout(1)) that TERM/KILLs the session and its MCP child after POKEARENA_GAME_TIMEOUT seconds (default 1200), so the game lands as unfinished instead of hanging. Mirrors the agy harness's --print-timeout.
Adds a 'how well did each model choose?' table to the benchmark report: blunder rate (with a bar), median regret, oracle-match rate, and win rate for context, sorted cleanest-first. RunRecord carries DecisionQuality []ModelStats; bench-report loads it from decision-eval's JSON (-decision-quality flag) so the report stays offline and never re-scores. build-report.sh folds it in via POKEARENA_DQ_JSON. Also makes the baseline round-robin trace optional: with no trace, bench-report synthesizes a header from the dataset and renders the agentic arm alone — enough for a decision-quality report without the full ladder. Test pins the section's ordering, the cleanest tag, and self-containment.
…asured The decision-quality metric could only be run against battles a live batch had played into Postgres, with model attribution kept alongside in /tmp. That made it expensive to re-measure, untestable, and — once those files were gone — impossible to re-run at all. decision-sim plays deterministic policies and writes the same export shape the live path persists, so the pipeline runs from a checkout with no gateway, no database, and no API spend. eval.CaptureStored is the piece that makes it faithful: it stores state only after a turn has fully settled, replacements folded in, because a leaked mid-turn replace state would shift the pre/post pairing ScoreDecisions relies on and mis-attribute every decision after the first KO. What that bought, on library v2 (72 games, oracle depth 3): expectimax d2 win 44% blunder 3% median regret 0 expectimax d1 win 61% blunder 11% median regret 12 heuristic win 56% blunder 21% median regret 111 random win 0% blunder 39% median regret 192 Blunder rate is monotone in policy strength. That is the metric's soundness property and it had never been shown: validation to date covered data integrity (does recovery reproduce the stored state), after which the metric was pointed at four models whose true ordering nobody knows, so an inverted metric would have gone unnoticed. It is now a test, not a table. The same run exposes a limitation that was hiding in plain sight. Expectimax d2 blunders least and wins least -- it is not the strongest policy here, it is the one most similar to the oracle, which is expectimax d3 (68% match rate against the heuristic's 27%). The doc's fairness argument is about information: every policy decides from the identical ai.MakeView projection. That argument is correct and says nothing about algorithm, and algorithm turns out to matter. This qualifies the published headline directly: "Gemini blunders least but wins less than Opus" is structurally the same result, so it cannot be read as "reasons more cleanly" without ruling out proximity to the yardstick. A deeper oracle does not help -- it is a more expectimax-shaped one. As a cross-check, d2's 44% independently reproduces benchmark.md §6's v2 figure (42.9%, CI [36.8, 49.2]) from a separate code path, which is evidence the offline capture reproduces live-shaped battles rather than merely plausible ones. The four-model table itself is labelled v1-era and unrepeatable. It was measured before library v2 gave every pick a nature and an EV spread, and unlike the depth sweep it cannot be re-run: the battles and their attribution are gone, so reproducing it means paying for a fresh batch across four vendors. Leaving v1 results adjacent to v2 ones with nothing to distinguish them is the exact failure the v2 re-sweep was written up to avoid. Also calibrates BlunderThreshold=300 against real data for the first time: random's median regret is 192, below the bar, so it is a severe-tail cut rather than a sloppiness detector. Constant left alone; the doc now says what it means. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZ6bbo8VkaDrs47Yz8v84U
…out it
The previous commit flagged that scoring against an expectimax oracle conflates
playing well with searching the way the oracle searches, and left the size of
that effect unknown for want of a second oracle "which does not exist yet."
It did exist. The heuristic agent is the opposite family -- depth-0, no
lookahead, no opponent model -- and already scores every legal action internally
to pick its move. Exposing ScoreActions on it was ten lines. The error was
treating the oracle slot as "strongest available player," which admits no
candidates above expectimax d3; what the role actually needs is independence,
and a weaker judge from an unrelated family is far more informative than a
marginally stronger one from the same family.
Scoring the same 72 v2 battles against each:
vs expectimax d3 vs heuristic
expectimax d2 3% (best) 19% (3rd)
expectimax d1 11% (2nd) 13% (2nd)
heuristic 21% (3rd) 2% (best)
random 39% (worst) 22% (worst)
The three skilled policies rank in exactly opposite order; each judge crowns its
own family. Match rate states it plainly: the heuristic policy agrees with the
heuristic oracle 92% of the time and with expectimax 27% -- same player, same
games, same fog-of-war view.
So the bias is not a modest caveat, it is total. A single-oracle blunder rate
does not rank skilled policies; it reports proximity to the judge. What the
metric still supports is documented: a floor (both judges separate incompetent
from competent), within-family comparisons, and agreement across families --
which is the strongest available claim and requires two oracles.
This lands on the published headline. "Gemini blunders least but wins less than
Opus" is structurally identical to "expectimax d2 blunders least but wins least,"
where the cause is known to be kinship. No LLM is an expectimax, so the model
ordering is not shown to be wrong -- but the grounds for believing it are gone,
because the one time the ordering could be checked against an independent judge
it inverted. The doc now says that rather than the earlier softer version.
Second finding, from nearly shipping a broken test: at oracle depth 2 the
expectimax judge gives random 35% and heuristic 33%, a two-point margin -- it
cannot tell random play from competent play. At depth 3, 43% vs 15%. Depth 3 is
a floor below which the metric measures nothing, and that runs against the
intuition from benchmark.md §6, where expectimax wins fewer games as depth
rises. Playing well and judging well are different capabilities.
Tests assert what survives both judges (random is worst) and pin the family bias
itself, so a future change that makes a single oracle's blunder rate look like
an absolute score fails. The expectimax arm stays in decision-sim: a two-point
margin dressed up as a soundness property is worse than no test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZ6bbo8VkaDrs47Yz8v84U
shaumik
force-pushed
the
feat/decision-quality-eval
branch
2 times, most recently
from
August 12, 2026 02:37
2347315 to
baa0532
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Scores how well each model chose, not just whether it won. For every free
decision in a recorded live battle, we re-simulate the turn to recover the exact
action played, then ask a reference oracle what it would have played from the
identical fog-of-war view, and measure the regret — the value the choice
gave up. Rolled up per model: blunder rate, median regret, match rate, win rate.
The engine is a pure function of (state, actions) and deterministic from the
stored RNGState, so action recovery is exact and doubles as a validity check on
the recorded state.
The finding that changed: the oracle decides the answer
Scoring against a single expectimax oracle conflates playing well with
searching the way the oracle searches. The heuristic agent is the opposite
family — depth-0, no lookahead, no opponent model — and already scores every
legal action internally, so exposing
ScoreActionson it gives a second,independent judge for ~10 lines.
Scoring the same 72 offline battles (team library v2) against each:
The three skilled policies rank in exactly opposite order. Each judge crowns
its own family. Match rate says it plainly: the heuristic policy matches the
heuristic oracle's top pick 92% of the time and expectimax's 27% — same player,
same games, same fog-of-war view.
Only one finding survives both judges: random is worst.
So a single-oracle blunder rate does not rank skilled policies; it reports
proximity to the judge. What the metric still supports is now documented: a
floor, within-family comparisons, and agreement across families (the strongest
available claim, and it needs two oracles).
A second requirement, from the same data: at oracle depth 2 the expectimax
judge gives random 35% and heuristic 33% — a 2-point margin, i.e. it cannot tell
random play from competent play. At depth 3, 43% vs 15%. Depth 3 is a floor,
not a preference. This cuts against
benchmark.md§6, where expectimax winsfewer games as depth rises: playing well and judging well are different
capabilities.
The model table is now labelled v1-era and unrepeatable
The per-model numbers were measured before team library v2 gave every pick a
nature and an EV spread, so they describe a format that no longer ships. Unlike
the depth sweep, they cannot be re-run: the battles lived in a local
Postgres and their model attribution in
/tmp, and both are gone. Reproducingmeans paying for a fresh batch across four vendors.
They are kept, labelled, with the reason — leaving v1 results adjacent to v2
ones with nothing to distinguish them is the exact failure the v2 re-sweep was
written up to avoid.
Reproducing it no longer needs infrastructure
cmd/decision-simplays deterministic policies and writes the same export shapethe live path persists — no gateway, no database, no API spend, fixed seeds:
eval.CaptureStoredis the piece that makes this faithful: it stores state onlyafter a turn has fully settled, replacements folded in, because a leaked
mid-turn replace state would shift the pre/post pairing
ScoreDecisionsrelieson and mis-attribute every decision after the first KO.
As a cross-check, expectimax d2's 44% win rate here independently reproduces
benchmark.md§6's v2 figure (42.9%, 95% CI [36.8, 49.2]) from a separate codepath — evidence the offline capture reproduces genuinely live-shaped battles.
How it's built
internal/eval/decisionquality.go— action recovery (recoverActions+settlefor faint/replacement turns → ~full coverage) andScoreDecisions.internal/ai/expectimax.go—ScoreActionsexposes the per-action maximinvalues
Decidealready computes (additive;Decideunchanged).internal/ai/heuristic.go—ScoreActionsfor the second oracle family.internal/eval/decisioncapture.go—CaptureStored, offline battles in thelive storage shape.
internal/eval/decisionreport.go—AggregateByModel. Regret is heavy-tailed(a missed lethal ≈ winValue), so the median ignores the tail and the mean is
winsorized at a cap.
cmd/decision-eval—-inscores one battle;-manifestscores a batch;-oraclepicks the judge.cmd/decision-sim— offline batch generator.scripts/bench/play-live.sh— caps each live game's wall clock (run_cappedwatchdog, default 1200s) so a hung agent session can't wedge a batch.
aggregation, capture↔recovery round-tripping, the surviving ranking property,
and the family bias itself — so a future change that treats a single oracle's
blunder rate as an absolute score fails.
Follow-ups (not in this PR)
different families but the same author and era; a trained policy would test
whether they share blind spots.
What this PR originally claimed (superseded)
Why this no longer stands. "Blunders least, wins less" is exactly what
expectimax d2 produces against this oracle, and there the cause is kinship with
the judge rather than cleaner play. Swapping in a second judge reversed the
ranking of every skilled policy.
No language model is an expectimax, so this does not show the model ordering is
wrong — it shows there is no evidence it is right. The one time the ordering
could be checked against an independent judge, it inverted. Haiku being worst on
win rate and blunder rate is still consistent with genuinely weaker play; the
Opus/Gemini flip, which was the headline, needs both oracles to mean anything,
and the battles it was computed from no longer exist.