Agent identity, a repeatable benchmark runner, and a decision metric with no judge

README.md

@@ -186,6 +186,7 @@ export ANTHROPIC_API_KEY=sk-ant-…
 | `--model` | `claude-haiku-4-5-20251001` | Anthropic model id. Use opus for stronger play at higher cost. |
 | `--turn-timeout` | `12s` | Per-turn LLM budget. The gateway default-actions the slot if exceeded. |
 | `--data-version` | `gen1-v1` | Must match the gateway's `DATA_VERSION` env. |
+| `--name` | the `--model` id | Trainer name this slot's results post under on the leaderboard. |
 
 ---
 
@@ -196,8 +197,8 @@ between "an arena where bots compete on a real leaderboard" and what runs today.
 
 | Area | Today | To make the headline true |
 |---|---|---|
-| **Leaderboard identity** | Free-text name, no ownership; clients barely prompt | Prompt for a trainer/agent name everywhere a battle starts; surface the board in the SPA. (Optional later: claim-a-handle + secret to stop impersonation.) |
-| **Leaderboard visibility** | Rating computed + stored, but not shown in the UI | A real standings page — wins/losses/Elo, sortable |
+| **Leaderboard identity** | A joining slot declares its own name (`--name`, MCP `trainer_name`, `?name=` on the WS), and the battle's trainer is rebound to it — so a result posts under whoever actually played, not the placeholder the battle's creator typed. Names are **self-reported and unverified** | Claim-a-handle + secret, so a name on the board can't be taken by anyone who wants it |
+| **Leaderboard visibility** | Standings page in the SPA (name + Elo), fed by `/api/leaderboard` | Sortable, with wins/losses alongside the rating — the store already returns them |
 | **Bot onboarding** | Two reference clients, MCP + CLI | A 5-minute "write your own bot" quickstart against a documented protocol |
 | **Provider coverage** | Benchmark (`cmd/bench`) runs Anthropic, OpenAI, Gemini, and local Ollama behind one `Client` interface, in `raw`/`cot` conditions; the live harness (`pokearena-agent`) is still Anthropic-only | Bring the remaining vendors to the live harness too |
 

backlog/2026-08-11T18-30-identity-on-join.md

@@ -0,0 +1,106 @@
+# Identity belongs to the joiner, not the creator
+
+The leaderboard has had working Elo since the beginning and has never
+meant anything. Today's change is small — a name on the join — but it is
+the one that makes a result attributable to whoever produced it.
+
+## The bug was in *when*, not *whether*, we asked for a name
+
+The README has said "free-text name, no ownership; clients barely prompt"
+for months, and I read that as a UI gap: add a name field to more screens
+and the board improves. That is not what was wrong.
+
+A `live_pvp` battle is created before its players arrive. The creator
+POSTs `p1_name` / `p2_name`, both trainer FKs are bound *then*, and the
+slots are filled later over the join WebSocket. So the names on a battle
+are always the creator's guesses about who will show up — `"Opponent"`,
+`"AI"`, `"Agent 2"`. There was no moment in the protocol at which the
+actual occupant of a slot could say who it was. `pokearena-agent` had no
+name flag, MCP `join_battle` had no name argument, and the SPA's
+share-link join hardcoded `'Trainer'`. Not one of them was under-prompting;
+there was nothing to prompt *into*.
+
+That reframes the fix. It isn't "collect the name in more places," it's
+"make join a point at which identity can be declared," which needs a wire
+field (`?name=` on the play URL), a message field (`LiveAction.Trainer`),
+and a write (`RebindBattleTrainer`). Once those exist the three clients
+are one line each.
+
+## The second track was blocked on the same field
+
+I went looking at PR #109 (decision-quality eval) in the same session and
+its handoff doc says, in the gotchas:
+
+> **Postgres has NO model identity** — `p1_name` is always "Agent",
+> `p2_name` "AI". Model attribution ONLY comes from the `bid=`→model
+> mapping in the run dirs. The previous mapping (`/tmp/pk-agentic`) was
+> wiped, which is why we re-ran.
+
+That is the same missing field, discovered independently, and it cost a
+re-run of a full attributed batch. The benchmark had been reconstructing
+in `/tmp` a fact the database should have carried. Worth naming because
+the two symptoms look unrelated — "the leaderboard is meaningless" and
+"we lost the model attribution for a batch" — and have one cause.
+
+## What I deliberately did not build
+
+The name is **self-reported**. Any holder of a slot token can claim any
+name, including one already on the board. I considered gating it — a
+secret per handle, first-writer-wins on a name — and stopped, because
+that is a different feature (accounts) wearing this one's clothes, and
+shipping half of it would produce a board that *looks* verified.
+
+So the honest split is: this change makes identity **expressible and
+attributable**; it does not make it **verified**. Both the README status
+row and `live-pvp.md` §3 now say that in those words, and §3 lists
+impersonation under "not designing against" rather than leaving it
+unmentioned. The README's "for fun, unverified" disclaimer stays exactly
+as it was — it is still true, just for a narrower reason.
+
+Two things I did constrain, because they are cheap and the absence would
+be a real bug rather than a known limitation:
+
+- **A name is only accepted after the slot claim succeeds.** Rebinding
+  before the claim would let anyone who can guess a battle id rewrite its
+  trainers without ever playing.
+- **`RebindBattleTrainer` refuses a battle with a winner.** Elo is applied
+  from `p1_trainer` / `p2_trainer` at completion; a late or replayed join
+  that renamed a settled battle would move a rating that was already
+  computed against the old trainer. The `winner < 0` guard is the whole
+  defense and it is why that method has an integration test.
+
+## A concurrency seam I didn't expect
+
+`trainerName` was a plain `[2]string` on `Match`, written once at
+construction and read by the coordinator goroutine. Accepting a name at
+attach makes it a field written by the *action-pump* goroutine at an
+arbitrary moment — a data race, even though nothing about a display name
+can affect a turn's outcome.
+
+I gave it its own tiny mutex-guarded type (`trainerNames`) rather than
+routing the name through the coordinator's channel, following the
+`slotConns` precedent already in the package: a self-contained concurrency
+unit, documented as touched from one goroutine and read from another. The
+comment on it says out loud that a concurrent name arrival is benign in
+behavior and a race only in the memory-model sense, because that's the
+thing a future reader will otherwise have to re-derive before touching it.
+
+`set("")` is a no-op rather than a store. Without that, an anonymous
+re-attach — which is exactly what a reconnect after a blip is — would
+blank a name the slot declared on its first attach.
+
+## Where the default came from
+
+`pokearena-agent --name` defaults to the model id rather than to empty.
+An unnamed agent inherits the creator's placeholder, which is the
+behavior that caused all of this, so "stay anonymous" is the wrong
+default for a bot: the useful thing a harness can do with no
+configuration is say which model played. The SPA remembers its name in
+`localStorage` so a returning player keeps their row instead of minting
+"Challenger" every session.
+
+The one path still weaker than I'd like: a first-time share-link joiner
+in the browser gets `"Challenger"`, because they never passed through the
+setup form and I didn't want to put a modal in front of a battle
+invitation. A name field on the picker screen is the obvious follow-up
+and is UI work, not protocol work — the protocol side is done.

backlog/2026-08-11T21-15-remeasuring-decision-quality.md

@@ -0,0 +1,195 @@
+# Re-measuring decision quality, and what wouldn't re-measure
+
+PR #109 had been open since 17 July and was 46 commits behind. The plan was
+"rebase it and re-measure on library v2." The rebase was the easy half; the
+re-measure turned out to be two different questions with two different answers,
+and separating them is most of what this round produced.
+
+## The rebase was free, which is worth noting
+
+Eight commits, 1367 lines, 46 commits of drift — and exactly one file
+overlapped with anything `main` had touched (`internal/ai/expectimax.go`, where
+main changed the replace-phase handling and the PR appended `ScoreActions`).
+Cherry-picked clean, built, and the whole suite went green with no edits.
+
+I had assumed a branch that stale would need real work, and it didn't, because
+the PR is almost entirely *additive* — a new package file, a new command, a new
+report section. That is a property of how it was scoped, not luck. Worth
+remembering the next time a branch's age is used as an argument for abandoning
+it: age is a proxy for conflict risk, and a purely additive branch barely
+accumulates any.
+
+## The published table can't be re-measured, and saying so is the deliverable
+
+`docs/decision-quality.md` carried a four-model table — Gemini / Opus / Sonnet /
+Haiku, blunder rate and median regret — measured in July on v1 neutral teams.
+Library v2 landed 4 August. So those numbers describe a format that no longer
+ships, which is precisely the failure the
+[v2 re-sweep entry](2026-08-04T17-51-resweeping-expectimax-on-v2.md) was written
+up to prevent, recurring in a different document three weeks later.
+
+The depth sweep could be re-measured because it runs offline. This one can't:
+the battles lived in a Postgres on the author's machine and their model
+attribution in `/tmp/pk-agentic-v2`, and the handoff doc records that `/tmp`
+being wiped had *already* forced one re-run. Both are gone now. Reproducing the
+table means paying for a fresh batch across four vendors — a decision about
+money, not an afternoon of compute.
+
+So the table is labelled v1-era and unrepeatable, with the reason, next to the
+v2 numbers rather than instead of them. Same treatment the pre-fix depth numbers
+got in August. The thing I want to avoid is a reader assuming one run produced
+everything on the page.
+
+## What *could* be measured, and the bias it exposed
+
+The metric itself had never been shown to rank correctly. It was validated on
+"does recovery reproduce the stored state" — a data-integrity property — and
+then pointed straight at four models, where nobody knows the true ordering. If
+the metric were subtly inverted, nothing in the pipeline would have said so.
+
+`cmd/decision-sim` closes that: it plays deterministic policies offline and
+writes the same export shape the live path persists, so the whole pipeline runs
+from a checkout with no gateway, no database, and no API spend. 72 games on v2:
+
+| policy | win rate | blunder rate | median regret |
+|---|---:|---:|---:|
+| expectimax d2 | 44% | 3% | 0 |
+| expectimax d1 | 61% | 11% | 12 |
+| heuristic | 56% | 21% | 111 |
+| random | 0% | 39% | 192 |
+
+Blunder rate is monotone in policy strength. Good — that's the soundness check,
+and it's now a test rather than a hope.
+
+But look at the first two rows. **Expectimax d2 blunders least and wins least.**
+It is not the best player in the table; it is the player most *similar to the
+oracle*, which is expectimax d3. Its 68% match rate against the heuristic's 27%
+is the same fact stated louder.
+
+That is a real limitation and it was hiding in plain sight. The doc's central
+fairness argument is that every policy is scored from `ai.MakeView` — the
+identical fog-of-war projection — so the oracle is "a better player looking at
+the same information, not an omniscient one." That argument is correct and it is
+about *information*. It says nothing about *algorithm*, and algorithm turns out
+to matter: agreement with an expectimax oracle partly measures being an
+expectimax.
+
+The uncomfortable part is what that does to the doc's headline finding. "Gemini
+blunders least but wins less than Opus" is structurally identical to "d2 blunders
+least but wins least," and in the case where I know the cause, the cause is
+kinship with the yardstick. That doesn't explain the model result away — no LLM
+is running expectimax — but it means the finding cannot be read as
+"Gemini reasons more cleanly" without an argument that its style isn't simply
+closer to the oracle's. Separating those needs a second oracle of a different
+family, and raising the depth won't do it: a deeper oracle is a *more*
+expectimax-shaped one.
+
+## A cross-check I didn't plan and am glad I ran
+
+Expectimax d2 came out at 44% against the heuristic. `docs/benchmark.md` §6 puts
+it at 42.9% [36.8, 49.2] on v2. Those are the same number, produced by two
+entirely separate code paths — the `bench` match runner there, `CaptureStored`
+plus a fresh export/score pipeline here.
+
+I ran it as a sanity check on my harness. It is better than that: it is evidence
+the offline capture reproduces live-shaped battles faithfully, which is the
+assumption the whole `decision-sim` path rests on. Without it, "the metric works
+offline" would have been an assertion about code I had just written.
+
+## Threshold calibration, briefly
+
+`BlunderThreshold = 300` was flagged in the original handoff as a first guess to
+calibrate once the data existed. On v2: random's *median* regret is 192, below
+the bar. So 300 is not "notices sloppiness" — it is a severe-tail cut, and even a
+policy choosing uniformly at random sits under it half the time. That is
+defensible for a headline metric, and it is exactly why median regret belongs in
+the table next to it rather than behind it. Left the constant alone; the number
+is fine once you know what it means, and now the doc says.
+
+## The ranking is now a test, not a table
+
+I first wrote this entry with "wire the ordering into CI" as the open item, then
+noticed the objection to that: the monotonicity above is the metric's *only*
+soundness property, and leaving it as a number in a document means it is checked
+whenever someone re-reads the document. It is the thing most likely to break
+silently if the oracle or the value function moves.
+
+`TestScoreDecisions_RanksAWorsePolicyAsBlunderingMore` pins it — random against
+heuristic, one team, a depth-2 oracle, 0.8 seconds. Coarse on purpose: the
+property is the *direction* of the ranking, and the widest available gap is the
+one least likely to flake. The four-policy sweep stays in `decision-sim` where
+it can afford the nine minutes.
+
+## The second oracle existed all along
+
+I closed the section above with "a second oracle of a different family would
+quantify the bias, and doesn't exist." That was wrong within the hour. The
+heuristic agent *is* a second family — depth-0, no lookahead, no opponent model
+— and it already scores every legal action internally to pick its own move.
+Exposing `ScoreActions` on it was ten lines.
+
+The lesson is not "look harder before declaring something missing," though that
+too. It is that I had been thinking of the oracle as *the strongest available
+player*, so the only candidates I considered were things stronger than
+expectimax d3 — of which there are none here. The requirement is not strength,
+it is **independence**. A weaker judge from a different family is far more
+informative than a marginally stronger one from the same family, because the
+question is whether two unrelated notions of "good move" agree.
+
+## What the second judge said
+
+Same 72 battles, scored twice:
+
+| policy | vs expectimax d3 | vs heuristic |
+|---|---:|---:|
+| expectimax d2 | 3% (best) | 19% (3rd) |
+| expectimax d1 | 11% (2nd) | 13% (2nd) |
+| heuristic | 21% (3rd) | 2% (best) |
+| random | 39% (worst) | 22% (worst) |
+
+The three skilled policies rank in **exactly opposite order**. Each judge crowns
+its own family. Match rate is the cleanest statement of it: the heuristic policy
+agrees with the heuristic oracle 92% of the time and with expectimax 27% — same
+player, same games, same fog-of-war projection.
+
+I expected the bias to be real and modest, something to note in a caveat. It is
+total. On this evidence a single-oracle blunder rate does not rank skilled
+policies at all; it reports proximity to the judge. The one finding that
+survives both is that random is worst.
+
+That is a much harsher result for the metric than I went looking for, and it
+lands directly on the doc's headline. "Gemini blunders least but wins less than
+Opus" is structurally identical to "expectimax d2 blunders least but wins least,"
+and in the case where the cause is knowable, the cause is kinship. No LLM is an
+expectimax, so this doesn't prove the model ordering wrong — it removes the
+grounds for believing it. The one time the ordering could be checked against an
+independent judge, it inverted.
+
+## The judge has to be strong enough to be a judge
+
+A second finding from the same runs, and one I nearly shipped a broken test
+over. I wrote the soundness test with a depth-2 oracle because depth 3 was slow,
+and it failed — random scored *better* than the heuristic. Not a flake: at depth
+2 the expectimax judge gives random 35% and heuristic 33%, a two-point margin.
+At depth 3, 43% vs 15%.
+
+So a depth-2 oracle cannot distinguish random play from competent play. Depth 3
+isn't a preference, it's the floor below which the metric measures nothing.
+
+The part worth keeping: this runs *against* the intuition from
+[§6](../docs/benchmark.md), where expectimax wins fewer games as depth rises.
+Playing well and judging well are different capabilities, and the depth that
+hurts one is required by the other. I would have assumed the depth sweep's
+conclusion transferred. It doesn't.
+
+The test now asserts the wide-margin, fast half (random vs heuristic under the
+heuristic judge, 22% vs 3%, one second) and documents why the expectimax arm
+lives in `decision-sim` instead. A two-point margin dressed up as a soundness
+property would have been worse than no test.
+
+## Still open
+
+A third oracle that isn't hand-built. Expectimax and the heuristic are different
+families but the same author and the same era of thinking about this game; they
+could share blind spots. A trained policy would be the real test. Nothing in the
+repo is close to that today.