Benchmark design: turn the arena into a reproducible LLM decision-making benchmark (v1 battle track first)

[shaumik]

Goal

Turn PokéArena from an eval harness (arena + Elo plumbing, no published results)
into an actual benchmark with reproducible numbers: a deterministic, provider-agnostic
eval for long-horizon, hidden-information decision-making, measured on our own engine.

This issue is the design spec. It supersedes ad-hoc thinking; file sub-issues for the workstreams.

Framing — do NOT overclaim novelty

We are not "the first LLM Pokémon benchmark." Prior art is real and crowded:

• PokéLLMon (2024) — human-parity agent, plays the real Showdown ladder. (arXiv 2402.01118)
• PokéChamp (ICML 2025 spotlight) — 6v6 Gen 9 OU, minimax+LLM, ~1300–1500 Showdown Elo, released a 3M-game dataset. This is almost exactly "competitive 6v6 LLM benchmark." (arXiv 2503.04094)
• "Reasoning Under Pressure: LLMs in Competitive Pokémon Battles" (OpenReview HANazTe5Im)
• "LLMs as Pokémon Battle Agents" (arXiv 2512.17308); vjeux/llm-pokemon-battle.

Correct positioning: "a reproducible, deterministic eval for long-horizon hidden-information
decision-making." Our differentiator is NOT the domain — it's that we built our own
deterministic engine instead of wrapping Showdown, which buys two things nobody else structurally has:

1. Per-move regret vs a ground-truth optimal. We have expectimax + a damage calc, so we can
   score every LLM decision against the optimal move, not just "did it win." Low-variance,
   reproducible, per-decision signal. This is the flagship metric.
2. Variance-controlled mirror matches. Same seed, same team, both sides — RNG is byte-identical,
   the only variable is the policy. Showdown can't do this. Kills the "Pokémon is just luck" objection.

Core principle: never let two unknown policies vary at once

If a model builds its own team AND battles with it, a win is unattributable (better play or better team?).
Isolate. Hold one axis constant to measure the other.

Two orthogonal tracks

Track A — Battle (tactical decision-making) — SHIP FIRST

• Fix the team. Mirror matches, fixed seeds. Only the policy varies.
• Agent population: 2–3 LLMs (Opus 4.8, Sonnet, Haiku min; + GPT/Gemini/Llama if feasible)
  • expectimax (strong baseline) + heuristic + random (floors). We already have all five controllers.
• Metrics, increasing rigor:
  1. Win rate vs fixed baselines (random → heuristic → expectimax), with Wilson 95% CIs.
  2. Elo from round-robin among all agents (relative ranking only — see limitations).
  3. Per-move regret vs expectimax-optimal (the flagship metric): at each LLM turn, run
     expectimax for the optimal action + value gap, log it. Reproducible, doesn't need the game to finish.
• Also log per model: token cost, $/game, decision latency (capability-per-dollar).

Track B — Build (strategic design / metagame reasoning)

• Fix the battler. Each model builds N teams; a neutral fixed engine (expectimax) pilots
  every team against a fixed gauntlet (or round-robin among model-built teams). Win rate of the
  team, piloted by a constant battler = team quality. This "fixed battler pilots everyone's teams"
  trick is what isolates build skill.
• Scoring = outcome + cheap structural diagnostics (no games needed, reproducible, interpretable):
  type coverage, shared-weakness count (stacked weaknesses = bad), speed-tier spread,
  role coverage, and legality/validity rate (did it produce a legal team — surprisingly
  discriminating for LLMs).
• Failure analysis: do models stack weaknesses? over-index on famous Pokémon? build illegal sets?

Combined view

Present a 2×2 (builds well × plays well, and do they correlate?), but the measurement must
decompose into the two isolated tracks.

Versioning — the engine is a moving target (items now; EV/IV/nature next)

A benchmark whose substrate changes is meaningless without versioning. We already have the bones
(_provenance.json, --data-version).

• Pin every result to (engine version + data version + ruleset). Frame as a feature:
  reproducibility, not an apology.
• Freeze a v1 ruleset NOW and ship numbers on it. Today's engine (no items, fixed L50,
  neutral EV/IV/nature) is a clean controlled environment. Don't block v1 on items/EVs.
  When items land, define v2 and re-run. Treat engine versions as benchmark editions
  (like MMLU → MMLU-Pro). The v1→v2 delta is itself a result ("does item reasoning separate
  models more than raw battling?").
• Items/EV/IV/nature are almost entirely a BUILD-track problem (EV spreads = combinatorial
  explosion; in-battle they only add modest reasoning like "is it Choice-locked?"). Another reason
  to keep tracks separate. This gives a deliberate difficulty ladder against saturation:
  battling → +items → +EV/IV/nature → (later) doubles/tera. "The benchmark grows with the frontier."

Schema forward-compat — already correct, keep the discipline

engine.TeamPick (internal/engine/team_validation.go) already uses optional omitempty fields:
Item is optional (empty = holds nothing), Ability defaults to slot 0. EV/IV/nature must land
the same way — optional fields defaulting to neutral so old teams stay valid and new capability is
purely additive. Never mutate existing fields; only add optional siblings.

Hard questions to have crisp answers for

• "Why not Pokémon Showdown?" → Reproducibility + per-decision ground truth. We replay any game
  bit-for-bit and score every move vs optimal; Showdown can't.
• "Your Elo has no human anchor — what does 1500 mean?" → Honest: it's relative ranking within
  our agent pool, not a claim vs humans. Regret is the absolute metric. Optionally validate a slice
  of our engine against Showdown's sim to claim fidelity.

Honest limitations (own them, don't hide them)

• Custom engine → no external human population anchoring Elo (mitigate: relative Elo + regret).
• Simplified metagame: ~80 species, no items yet, fixed L50, no EV/IV/nature. It's a controlled
  decision environment, not "competitive Pokémon" in the Smogon sense.
• High variance domain → need enough games + CIs; use determinism (mirror + fixed seeds) to reduce it.

Proposed sequencing (v1 first)

1. Headless eval runner over existing internal/ai controllers + engine: run N games per pairing,
   no UI, log every decision + engine state at each decision point.
2. Metrics: per-move regret (expectimax optimal + gap), win rate + Elo, Wilson 95% CIs; log token
   cost + latency.
3. Run v1 (fixed teams, mirror, fixed seeds), ~100+ games/pairing. Get real numbers.
4. Analysis + failure taxonomy (panic-switching — PokéLLMon's named failure — type-matchup misses,
   bad sacks, wasted setup).
5. Package: short report (question, method, reproducibility, results table w/ CIs, regret curves,
   failure taxonomy, limitations).
6. Build track: stub interfaces now (neutral-battler harness + structural diagnostics), fill in
   once items/EVs land.

v1 battle-track results are the priority — they convert "harness" into "benchmark with numbers,"
and the build track depends on the battle engine as its constant-battler.

---

Design captured from a spec discussion. Battle track ships first; build track and v2 (items) follow.