Agent identity, a repeatable benchmark runner, and a decision metric with no judge - #129
Open
shaumik wants to merge 15 commits into
Open
Agent identity, a repeatable benchmark runner, and a decision metric with no judge#129shaumik wants to merge 15 commits into
shaumik wants to merge 15 commits into
Conversation
A live_pvp battle is created before its players arrive: the creator POSTs
p1_name/p2_name, both trainer FKs bind then, and the slots fill later over
the join WebSocket. So the names on a battle were always the creator's
guesses about who would show up -- "Opponent", "AI", "Agent 2" -- and there
was no moment in the protocol at which a slot's actual occupant could say
otherwise. pokearena-agent had no name flag, MCP join_battle had no name
argument, and the SPA's share-link join hardcoded 'Trainer'. Elo has been
real since the beginning and has been accruing to placeholders.
This adds the missing moment. A joiner may declare a name on the play URL
(?name=), the gateway sanitizes it and rebinds that slot's trainer on the
battle row, and the name rides the attach message so the session labels the
slot in room frames and in the engine state it builds. Omitting it keeps the
creator's name, so every existing flow is unchanged.
The same missing field was blocking the benchmark from the other side: PR
109's handoff records that Postgres carries no model identity, so per-model
attribution had to be reconstructed from bid= mappings in /tmp -- and a full
attributed batch was re-run after those were wiped. One cause, two symptoms.
Scope, stated plainly: this makes identity expressible and attributable, not
verified. Any holder of a slot token may claim any name, including one
already on the board; claim-a-handle + secret is separate work, and shipping
half of it would produce a board that looks verified. README and live-pvp.md
§3 say so in those words, with impersonation listed under "not designing
against" rather than left unmentioned.
Two constraints that would be real bugs rather than known limitations:
a name is only accepted after the slot claim succeeds, so guessing a battle
id cannot rewrite its trainers; and RebindBattleTrainer refuses a battle that
already has a winner, so a late or replayed join cannot move a rating that
was computed against the previous trainer.
Accepting a name at attach turned Match.trainerName into a field written by
the action-pump goroutine and read by the coordinator -- benign in behavior,
a data race in the memory model. It gets its own mutex-guarded type following
the slotConns precedent. set("") is a no-op, or an anonymous re-attach after
a blip would blank a name the slot declared on its first one.
pokearena-agent --name defaults to the model id: an unnamed agent inherits
the placeholder, which is the behavior that caused all of this, so staying
anonymous is the wrong default for a bot. The SPA remembers its name in
localStorage so a returning player keeps their row.
Also corrects two stale rows in the README status table -- the leaderboard
has been visible in the SPA since the field-state work.
The rebind SQL is verified against a real Postgres (integration tag): the
column mapping and the settled-battle guard are both invisible to a unit
test of the surrounding Go.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZ6bbo8VkaDrs47Yz8v84U
…racle
Score how well a side chose, not just whether it won. A live battle stored
every turn's engine state, and the engine is a pure function of (state,
actions), so each turn re-simulates to recover the exact action played
(engine.LegalActions x ResolveTurn until the stored next state reproduces
byte-for-byte). Then a depth-limited expectimax oracle scores every legal
action from the identical fog-of-war view (ai.MakeView), and the value gap
between the played action and the best is the decision's regret.
- ai.ExpectimaxAgent.ScoreActions exposes the per-action maximin values
searchRoot already computes (additive; Decide unchanged), so callers can
measure regret rather than only agree/disagree.
- eval.ScoreDecisions walks stored turns, recovers each free choice on a side,
and returns per-decision {chosen, best, agree, regret, blunder} plus a count
of skipped faint/replacement turns (v1 scores clean turns; coverage reported,
not hidden).
- cmd/decision-eval is a spike over a battle JSON export (db-replay's shape).
Validated on real stored battles: recovery re-simulates exactly, and regret
correctly separates real mistakes from picking an equal-value alternative
(binary agreement counts equal moves as misses; regret scores them 0).
The naive recovery replayed one ResolveTurn and compared to the stored next state, so any turn that fainted a Pokémon (resolution stops in the replacement phase) couldn't be matched and was skipped — about half of all turns. settle now drives the resolved state through its forced replacements, searching the legal replacement picks (a cascade recurses), so faint turns reproduce the stored next state and are recovered like any other. On real battles this takes coverage from ~40-65% to ~100% of choosing turns. Test plays a full heuristic mirror that KOs and replaces, then asserts every choosing turn — faint turns included — re-derives its actions exactly.
Roll per-decision regret up to a per-model table: blunder rate (headline), median regret, match rate, win rate. AggregateByModel folds scored battles; decision-eval -manifest scores a batch and prints/JSON-emits the table; decision-report.sh joins bid= attribution to Postgres exports and drives it. Regret is heavy-tailed (missed lethal ~= winValue): median ignores the tail, mean is winsorized at a cap. Test pins the arithmetic and the tail handling. On the fresh attributed batch, blunder rate tracks win rate across all four models (Opus < Gemini < Sonnet < Haiku), so decision quality predicts the outcome win/loss alone hides.
…atch A stalled agent session reads forever at 0% CPU (Opus in particular walls out this way — a hung model/streaming connection). With no timeout, one such game blocked the whole attributed batch for 13+ hours. Wrap claude -p in a portable watchdog (macOS has no timeout(1)) that TERM/KILLs the session and its MCP child after POKEARENA_GAME_TIMEOUT seconds (default 1200), so the game lands as unfinished instead of hanging. Mirrors the agy harness's --print-timeout.
Adds a 'how well did each model choose?' table to the benchmark report: blunder rate (with a bar), median regret, oracle-match rate, and win rate for context, sorted cleanest-first. RunRecord carries DecisionQuality []ModelStats; bench-report loads it from decision-eval's JSON (-decision-quality flag) so the report stays offline and never re-scores. build-report.sh folds it in via POKEARENA_DQ_JSON. Also makes the baseline round-robin trace optional: with no trace, bench-report synthesizes a header from the dataset and renders the agentic arm alone — enough for a decision-quality report without the full ladder. Test pins the section's ordering, the cleanest tag, and self-containment.
…asured The decision-quality metric could only be run against battles a live batch had played into Postgres, with model attribution kept alongside in /tmp. That made it expensive to re-measure, untestable, and — once those files were gone — impossible to re-run at all. decision-sim plays deterministic policies and writes the same export shape the live path persists, so the pipeline runs from a checkout with no gateway, no database, and no API spend. eval.CaptureStored is the piece that makes it faithful: it stores state only after a turn has fully settled, replacements folded in, because a leaked mid-turn replace state would shift the pre/post pairing ScoreDecisions relies on and mis-attribute every decision after the first KO. What that bought, on library v2 (72 games, oracle depth 3): expectimax d2 win 44% blunder 3% median regret 0 expectimax d1 win 61% blunder 11% median regret 12 heuristic win 56% blunder 21% median regret 111 random win 0% blunder 39% median regret 192 Blunder rate is monotone in policy strength. That is the metric's soundness property and it had never been shown: validation to date covered data integrity (does recovery reproduce the stored state), after which the metric was pointed at four models whose true ordering nobody knows, so an inverted metric would have gone unnoticed. It is now a test, not a table. The same run exposes a limitation that was hiding in plain sight. Expectimax d2 blunders least and wins least -- it is not the strongest policy here, it is the one most similar to the oracle, which is expectimax d3 (68% match rate against the heuristic's 27%). The doc's fairness argument is about information: every policy decides from the identical ai.MakeView projection. That argument is correct and says nothing about algorithm, and algorithm turns out to matter. This qualifies the published headline directly: "Gemini blunders least but wins less than Opus" is structurally the same result, so it cannot be read as "reasons more cleanly" without ruling out proximity to the yardstick. A deeper oracle does not help -- it is a more expectimax-shaped one. As a cross-check, d2's 44% independently reproduces benchmark.md §6's v2 figure (42.9%, CI [36.8, 49.2]) from a separate code path, which is evidence the offline capture reproduces live-shaped battles rather than merely plausible ones. The four-model table itself is labelled v1-era and unrepeatable. It was measured before library v2 gave every pick a nature and an EV spread, and unlike the depth sweep it cannot be re-run: the battles and their attribution are gone, so reproducing it means paying for a fresh batch across four vendors. Leaving v1 results adjacent to v2 ones with nothing to distinguish them is the exact failure the v2 re-sweep was written up to avoid. Also calibrates BlunderThreshold=300 against real data for the first time: random's median regret is 192, below the bar, so it is a severe-tail cut rather than a sloppiness detector. Constant left alone; the doc now says what it means. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZ6bbo8VkaDrs47Yz8v84U
…out it
The previous commit flagged that scoring against an expectimax oracle conflates
playing well with searching the way the oracle searches, and left the size of
that effect unknown for want of a second oracle "which does not exist yet."
It did exist. The heuristic agent is the opposite family -- depth-0, no
lookahead, no opponent model -- and already scores every legal action internally
to pick its move. Exposing ScoreActions on it was ten lines. The error was
treating the oracle slot as "strongest available player," which admits no
candidates above expectimax d3; what the role actually needs is independence,
and a weaker judge from an unrelated family is far more informative than a
marginally stronger one from the same family.
Scoring the same 72 v2 battles against each:
vs expectimax d3 vs heuristic
expectimax d2 3% (best) 19% (3rd)
expectimax d1 11% (2nd) 13% (2nd)
heuristic 21% (3rd) 2% (best)
random 39% (worst) 22% (worst)
The three skilled policies rank in exactly opposite order; each judge crowns its
own family. Match rate states it plainly: the heuristic policy agrees with the
heuristic oracle 92% of the time and with expectimax 27% -- same player, same
games, same fog-of-war view.
So the bias is not a modest caveat, it is total. A single-oracle blunder rate
does not rank skilled policies; it reports proximity to the judge. What the
metric still supports is documented: a floor (both judges separate incompetent
from competent), within-family comparisons, and agreement across families --
which is the strongest available claim and requires two oracles.
This lands on the published headline. "Gemini blunders least but wins less than
Opus" is structurally identical to "expectimax d2 blunders least but wins least,"
where the cause is known to be kinship. No LLM is an expectimax, so the model
ordering is not shown to be wrong -- but the grounds for believing it are gone,
because the one time the ordering could be checked against an independent judge
it inverted. The doc now says that rather than the earlier softer version.
Second finding, from nearly shipping a broken test: at oracle depth 2 the
expectimax judge gives random 35% and heuristic 33%, a two-point margin -- it
cannot tell random play from competent play. At depth 3, 43% vs 15%. Depth 3 is
a floor below which the metric measures nothing, and that runs against the
intuition from benchmark.md §6, where expectimax wins fewer games as depth
rises. Playing well and judging well are different capabilities.
Tests assert what survives both judges (random is worst) and pin the family bias
itself, so a future change that makes a single oracle's blunder rate look like
an absolute score fails. The expectimax arm stays in decision-sim: a two-point
margin dressed up as a soundness property is worse than no test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZ6bbo8VkaDrs47Yz8v84U
Scoring choices against a reference agent measures agreement with that agent: two references of different families rank the same policies in opposite orders, so blunder rate cannot be published as a quality score. This is the other approach -- count only mistakes that are provably mistakes. Four categories, each a guaranteed no-op rather than a strong player's opinion: attacking into a type immunity, healing at full HP, inflicting a status the target already has, and boosting a stat already pinned at +6. No reference agent is consulted, so no choice of reference can flip the result. Two rules keep it defensible, and both are load-bearing. Certainty: a category is included only when the action cannot possibly accomplish anything, because a false positive costs far more here than a missed detection -- the whole value is that every count survives an audit. Knowability: a mistake counts only if the player could have known from the fog-of-war view it held, which is why ability-granted immunities (Levitate, Flash Fire) are not charged while type immunities are. Penalising unknowable information would make this a luck metric. It is a floor, not a full account of skill: a policy can score zero and still play badly. That is the trade -- a number that means exactly what it says. Validation is per category, not on a pooled rate, because the categories measure different failures and different policies fail differently. The sharp claim: the heuristic checks type effectiveness before attacking and records 0 immune attacks over 220 decisions, while random records 4 over 118. Two things the metric found immediately, both kept as findings rather than quietly fixed: The heuristic baseline has a real defect -- 27 boost-at-cap turns over six games, and it re-applies Thunder Wave to an already-paralysed target for six consecutive turns. It is NOT fixed here. The heuristic is the benchmark's reference opponent, so changing how it plays invalidates every number measured against it, the depth sweep in benchmark.md §6 above all. That is a decision with benchmark-wide consequences and should be made deliberately, not as a side effect of adding a metric. TestScoreVerifiable_HeuristicBoostsAtCap pins the finding and documents that it should be deleted when the bot is fixed. And a wrong assumption of mine, caught by the checker: Electric vs Ground is a true 0x immunity, so Thunderbolt into a Rhydon is a provable error, not the "resisted but not immune" control case I had written it as. The control now uses a genuine 0.5x matchup. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZ6bbo8VkaDrs47Yz8v84U
Running a benchmark meant invoking run-batch.sh once per model per team, each
call truncating its own results file, with the model-to-battle mapping living
in /tmp. Scaling it up meant remembering what had already been played. That is
how a previous batch was lost.
bench-run takes a config -- entrants, teams, games per team -- and plays the
whole matrix. Four properties, each answering a specific way a long run fails:
Resumable. The plan is a pure function of the config, and each game writes its
own result file named from its coordinates, so re-running the same command
skips what finished and plays the rest. There is no central ledger to lose and
no -resume flag to remember; a laptop that slept overnight is a non-event.
Scaling up is editing games_per_team and re-running.
Balanced under interruption. Games are interleaved across entrants, so stopping
halfway leaves every entrant with the same number of games rather than the first
complete and the last with none. In plan order a truncated run is not a
comparison at all; interleaved, a partial run is still usable.
Failures stay out of the dataset. A game whose harness errors or times out
writes no result file, so the next run retries it. A wrong CLI flag costs a
retry rather than a poisoned batch.
Attribution is a database fact. The entrant id travels as the battle's trainer
name -- which the join-identity work earlier in this branch made possible -- so
"which agent played this battle" is recorded in Postgres instead of inferred
from a scratch directory that may not survive the night.
The unit of identity is the entrant, deliberately not the model: a model reached
through Claude Code and the same model through a bare API harness are different
entrants, because the agent runtime around the model is part of what is being
measured. That makes both comparisons fall out of one dataset -- hold the model
fixed and vary the harness to compare runtimes, hold the harness fixed and vary
the model to compare models.
bench-report -bench-run prints the standings from the result files alone: no
gateway, no Postgres, no API access, so the table under a published claim can
always be re-derived from a directory small enough to commit beside it. Every
win rate carries a Wilson interval rather than offering it as an option,
because a win rate from a small run is the easiest number here to over-read.
Unfinished games are reported in their own column and excluded from the
denominator -- a harness that fails to finish games is a result worth seeing,
not something to hide inside a win rate by scoring it as a loss.
Validation rejects two configs that would silently corrupt a run: duplicate
entrant ids, and distinct ids that reduce to the same filename ("a/b" and
"a-b"), which would overwrite each other's results. The second was caught by
its own test.
Verified end to end against a stand-in harness: full matrix, re-run is a no-op,
resume after deleting results replays exactly the missing games, scaling 3->5
games per team keeps the 18 played and adds 12, and a wholly failing harness
writes zero result files.
The codex arm is wired but its flags are unconfirmed against a live install;
noted in the script, and a wrong flag fails loudly rather than degrading a game.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZ6bbo8VkaDrs47Yz8v84U
The verifiable-error metric added yesterday was built to judge contestants. The first thing it measured was our own reference opponent: 27 boost-at-cap turns across six games, and Thunder Wave re-applied to an already-paralysed target for six consecutive turns. Two distinct causes, and the second is the instructive one. The boost branch had no check at all -- it valued a self-boost at 55 while healthy without ever consulting me.Stages, so at +6, where the move cannot change anything, it still outscored most attacks. The status branch *did* check, and was commented "a status move is wasted on an already-statused foe". It returned 0. But a damaging move that deals no damage also scores 0, and Decide breaks ties toward the earliest legal action, so a dead status move in an early slot won the tie and was replayed every turn. The code held the right belief, correctly stated, and was still wrong, because 0 is not a neutral value in a scoring function whose other outputs bottom out there. Both now return a deadMoveScore well below any live option including a switch -- giving up a turn to reposition genuinely beats spending it on a guaranteed no-op. This is not the engine. Every one of these moves fails correctly in the engine, which is exactly why they were detectable: the metric looks for actions that provably cannot accomplish anything, and the engine's correct rejection is what makes "provably" true. Rules, replays, determinism and fairness are untouched. But the heuristic is the opponent every figure in benchmark.md §6 is measured against, so the document's only load-bearing table had to be re-measured rather than assumed still valid. 240 games per depth, same methodology: depth 1 54.4% -> 53.8% depth 2 42.9% -> 42.9% depth 3 42.1% -> 42.1% Depths 2 and 3 are identical to the exact game (103/240 and 101/240 both times); depth 1 moved by two games out of 240. The reason is visible once you look at which turns were wasted: they clustered in already-decided stall positions -- the six-turn Thunder Wave loop was on the wall team at turns 72-77, in a game long settled. A bot wasting turns in a position it has already won or lost does not change the result. I expected a larger move and braced to re-caveat §6. Re-running the whole sweep costs 22 minutes of deterministic offline compute and no money, which is a cheap way to turn that guess into a fact. The sweep itself is now committed as TestDepthSweep behind POKEARENA_DEPTH_SWEEP rather than left as a scratch file, and §6 carries the command -- the same mistake in miniature as the lost /tmp attribution, a published number whose derivation was not reproducible. Deliberately not fixed: the heuristic under-values Rest, which carries no effect block in the dataset and so never reaches the heal branch. The line drawn is correct a provable waste, do not change a strategy. A move that cannot possibly work is a defect in any sense; teaching the bot to use Rest well is a judgement about how the game should be played, changes the baseline's strength, and is a decision about what the benchmark measures rather than a bug fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZ6bbo8VkaDrs47Yz8v84U
golangci-lint could not run in the authoring environment (its build targets an older Go than this repo), so these landed unchecked. It is now runnable there via `go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.12.2` with GOTOOLCHAIN pinned, which is how the fixes below were verified — 0 issues. One of the findings is a genuine bug, not style. loadConfig ended with return c, json.Unmarshal(b, &c) and Go does not order the read of `c` against the call that fills it, so the function could return a zero-valued config while reporting no error. It happened to work in the end-to-end run; it was never guaranteed to. Split into an explicit unmarshal-then-return. The rest: - exec.Command -> exec.CommandContext (noctx), threading a context from run() that is canceled on teardown, so a straggling child process is killed rather than orphaned. Deliberately no timeout on it: the per-game wall clock lives in the runner script, which also kills the CLI's own children, and interrupt still lets in-flight games finish writing so no half-written result file is mistaken for a completed game. - fmt.Errorf %v -> %w so the underlying exec error stays unwrappable. - Unchecked fmt.Sscanf -> strconv.Atoi with the error handled. An unparseable winner= field now leaves the game at -1 (unfinished) rather than silently reading as a win for side 0 -- an under-counted result beats a fabricated one. - Two De Morgan simplifications flagged by staticcheck. - gofumpt on the depth-sweep test. - British spellings the repo's misspell linter rejects. Verified: build, vet, vet -tags=integration, golangci-lint v2.12.2, and the full suite under -race, all clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UZ6bbo8VkaDrs47Yz8v84U
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #109 — its eight commits are rebased in here, so close that one when this lands.
Three pieces of work plus a baseline fix the new metric turned up. Each commit stands alone; the summaries below are the why, the commit messages carry the detail.
1. A joining agent can say who it is
A
live_pvpbattle is created before its players arrive: the creator POSTsp1_name/p2_name, both trainer FKs bind then, and the slots fill later over the WebSocket. So the names on a battle were always the creator's guesses —"Opponent","AI","Agent 2"— and no point in the protocol let a slot's actual occupant say otherwise.pokearena-agenthad no name flag, MCPjoin_battlehad no name argument, the SPA's share-link join hardcoded'Trainer'.Elo has been real since the beginning and has been accruing to placeholders.
This adds the missing moment. A joiner declares a name on the play URL (
?name=), the gateway sanitizes it and rebinds that slot's trainer on the battle row, and the name rides the attach message so the session labels the slot in room frames and engine state. Omitting it keeps the creator's name, so every existing flow is unchanged.The same missing field was blocking the benchmark from the other side — #109's handoff records that Postgres carries no model identity, so per-model attribution lived in
/tmpand a full batch was re-run after it was wiped. One cause, two symptoms.Scope, stated plainly: this makes identity expressible and attributable, not verified. Any slot-token holder can claim any name. Claim-a-handle + secret is separate work, and shipping half of it would produce a board that only looks verified.
READMEandlive-pvp.md§3 say so, with impersonation listed under "not designing against."Two constraints that would be real bugs rather than known limitations: a name is only accepted after the slot claim succeeds, and
RebindBattleTrainerrefuses a battle that already has a winner, so a late or replayed join can't move a settled rating. The SQL is verified against a real Postgres under the integration tag.2. One resumable command for a whole benchmark run
Running a benchmark meant invoking
run-batch.shonce per model per team, each call truncating its own results file.bench-runtakes a config — entrants, teams, games per team — and plays the whole matrix.Four properties, each answering a specific way a long run fails:
-resumeflag to remember. Scaling up is editinggames_per_team.The unit of identity is the entrant, deliberately not the model: a model through Claude Code and the same model through a bare API harness are different entrants, because the agent runtime is part of what's measured. Both comparisons fall out of one dataset — hold the model fixed and vary the harness, or the reverse.
bench-report -bench-runprints standings from the result files alone: no gateway, no Postgres, no API access. Every win rate carries a Wilson interval rather than offering it as an option, because a win rate from a small run is the easiest number here to over-read. Unfinished games get their own column and are excluded from the denominator — a harness that fails to finish games is a result worth seeing, not something to hide inside a win rate.Verified end to end against a stand-in harness: full matrix, re-run is a no-op, resume after deleting results replays exactly the missing games, scaling 3→5 keeps the 18 played and adds 12, a wholly failing harness writes zero files.
3. Decision quality without a judge — and why one was needed
#109 scored choices against an expectimax oracle. Adding a second oracle from a different family (the depth-0 heuristic, ~10 lines to expose) and scoring the same 72 battles against each:
The three skilled policies rank in exactly opposite order. Each judge crowns its own family. Match rate says it plainly: the heuristic policy agrees with the heuristic oracle 92% of the time and with expectimax 27% — same player, same games, same fog-of-war view.
So a single-oracle blunder rate doesn't rank skilled policies; it reports proximity to the judge. The doc now says what the metric can support: a floor, within-family comparisons, and agreement across families. This lands on #109's original headline — "Gemini blunders least but wins less than Opus" is structurally identical to "expectimax d2 blunders least but wins least," where the cause is known to be kinship. The four-model table is labelled v1-era and unrepeatable.
The replacement counts only provable mistakes — attacking into a type immunity, healing at full HP, inflicting a status the target has, boosting at +6. No reference agent, so no choice of reference can flip it. Two rules keep it defensible: certainty (guaranteed no-ops only, never merely bad ideas) and knowability (only mistakes the player could have seen from its own view — hidden Levitate isn't charged, public types are).
Also: a depth-2 expectimax judge gives random 35% and heuristic 33%, a 2-point margin — it cannot tell random from competent. Depth 3 gives 43% vs 15%. Depth 3 is a floor, not a preference.
cmd/decision-simmakes all of it reproducible offline — no gateway, no database, no API spend.4. The baseline bug the new metric found
Pointed at contestants, it immediately indicted our own reference opponent: 27 boost-at-cap turns across six games, and Thunder Wave re-applied to an already-paralyzed target for six consecutive turns.
The boost branch had no ceiling check. The status branch did check and returned
0— but a damaging move dealing no damage also scores 0, and ties break toward the earliest slot, so a dead status move in slot one won the tie and replayed every turn. The code held the right belief, correctly commented, and was still wrong, because zero isn't neutral in a scoring function whose other outputs bottom out there.Not the engine. Every one of these moves fails correctly in the engine — that's exactly why they were detectable.
But the heuristic is the opponent every figure in
benchmark.md§6 is measured against, so the table was re-measured rather than assumed valid:Depths 2 and 3 identical to the exact game; depth 1 moved by two games out of 240. The wasted turns clustered in already-decided stall positions. The sweep is now committed as
TestDepthSweepbehindPOKEARENA_DEPTH_SWEEP=1so §6's numbers can be re-derived rather than trusted.Deliberately not fixed: the heuristic under-values Rest. That's a strategy change, not a provable-waste bug, and it would move the baseline's strength — a decision about what the benchmark measures, which should be made deliberately.
Verification
go build ./...,go vet ./...,go vet -tags=integration ./...— cleangolangci-lintv2.12.2 (the version CI pins) — 0 issues-race, the way CI runs itThe first CI run failed on lint, because the
golangci-lintinstalled in the authoring environment targets an older Go than this repo and refused to start. That is worked around — pin the toolchain and run the exact version CI uses:Worth calling out because one of the 13 findings was a real bug rather than style:
loadConfigendedreturn c, json.Unmarshal(b, &c), and Go does not order the read ofcagainst the call that fills it — so it could return a zero-valued config while reporting success, and a benchmark run would have silently planned nothing. Fixed, along withexec.Command→CommandContext(a straggling child is killed at teardown rather than orphaned), an uncheckedSscanf(an unparseablewinner=now reads as unfinished rather than a win for side 0),%v→%w, two De Morgan simplifications, and formatting.Docs
READMEstatus table (two stale rows corrected — the leaderboard has been visible in the SPA since the field-state work),live-pvp.md,mcp-protocol.md,benchmark.md§6,decision-quality.md,running-the-benchmark.md, plus three backlog entries.🤖 Generated with Claude Code