Make PokéArena discoverable: license, module path, releases, Python package - #155
Open
shaumik wants to merge 12 commits into
Open
Make PokéArena discoverable: license, module path, releases, Python package#155shaumik wants to merge 12 commits into
shaumik wants to merge 12 commits into
Conversation
The module was named "pokearena", which is not a resolvable import path. That made "go install github.com/shaumik/PokeArena/cmd/...@latest" fail and kept the repo off pkg.go.dev entirely — the most natural way to obtain the MCP server and the bench harness did not work at all. Rename the module and rewrite the 402 import lines that referenced it. The root package keeps its name (pokearena), so cmd/pokearena-agent aliases the import rather than relying on a basename that no longer matches. The repo also had no LICENSE, which means all rights reserved: unusable in published research, unusable commercially, and rejected on sight by the registries and awesome-lists we want to be listed in. MIT, with the trademark position the README already states kept alongside it.
The README opened on the arena and its leaderboard — which the same README then admits is unverified, keyed on free-text names with no ownership. We were leading with the weakest asset on the page. The strongest one was buried in benchmark.md section 2: the engine is ours rather than a Showdown wrapper, which buys variance-controlled mirror matches. Same seed, same team, both sides, byte-identical RNG stream, so the only free variable is the policy. That is the claim no Showdown-wrapping harness can structurally make, so it is now the headline. The first runnable block was 'docker compose up' — Postgres, RabbitMQ, Redis and four services. An agent evaluating the repo in a sandbox cannot run that and silently gives up. cmd/bench needs none of it, so it goes first; the full arena keeps its section further down. Add AGENTS.md for the agent doing the discovering: the no-infra path, what does and does not need services, the MCP tool surface, and the observation and action shapes. The heading is now 'Connect your agent (Pv-Agent)', which repairs the anchor ARCHITECTURE.md and mcp-protocol.md were already linking to.
cmd/bench is the project's zero-setup entry point — the one thing a
reader can run in a minute with no services and no API key. But it
loaded the dex with LoadDex("data") and defaulted -teams to
data/benchmark-teams.json, both relative to the working directory. So
'go run github.com/shaumik/PokeArena/cmd/bench@latest' died at dex load
from the module cache, and the front door only ever opened from inside
a clone.
Widen the module-root embed to cover benchmark-teams.json,
_provenance.json and model-pricing.json, and fall back to it when the
dataset directory is absent. An explicit -data that does not exist is
still a hard error; the user named a directory and we should not
quietly ignore them.
The library, provenance and pricing loaders grow fs.FS variants with
the disk versions as thin wrappers, matching what LoadDex/LoadDexFS
already did.
Embedded and on-disk runs were diffed over 1801 trace rows of a
heuristic-vs-expectimax run: byte-identical, so the fallback cannot
fork the benchmark from the copy in data/.
Three discovery surfaces the repo had no answer for.
CITATION.cff gives GitHub its 'Cite this repository' button and gives a
researcher's tooling something machine-readable to pick up. CFF 1.2.0,
validated against the published schema.
.claude-plugin/ registers the existing MCP server so Claude Code can
install PokeArena in one step, and lets the repo serve as its own
marketplace. The launcher resolves a release binary, then PATH, then
falls back to 'go run' from the module root, and keeps every diagnostic
on stderr so stdout stays a clean JSON-RPC stream. The bundled skill is
written off docs/mcp-protocol.md and the real handlers rather than the
README's prose: the phase gate, {ready:false} as an ordinary timeout,
act's optimistic write, and forfeit-on-leave.
docs/deeper-search-played-worse.md pulls benchmark.md section 6 out
into something a stranger will read: expectimax searching deeper and
playing worse, traced to a fog-of-war reconstruction that scored a KO
of the visible active as winning the whole game. It keeps the part that
makes it honest — the corrected model is weaker than the buggy one,
because the phantom KO had been inducing helpful aggression — and it
keeps the doc's caveat that the d2/d3 tail is one run's suggestion, not
a result. Every figure was diffed against section 6; none is new.
New command, same convention as the other root-dropped build artifacts: 'go build ./cmd/pokearena-env' leaves a 5.9M binary in the repo root and it should not be committable.
The repo had no tags and no releases, which is not just untidy: the official MCP Registry only accepts artifacts from GitHub or GitLab releases, so PokeArena was structurally ineligible for the place MCP clients actually browse. Nobody could find it there because it could not be listed there. server.json targets the 2025-12-11 schema. There is no Go registry type, so the binary ships as an MCPB bundle built from a release artifact, per the registry's own guidance for compiled servers. The description is 95 of its 100 permitted characters and leads with 'playable' on purpose: a search for a Pokemon MCP server returns a crowd of PokeAPI data wrappers, and the thing worth saying about this one is that an agent takes a seat and plays. GoReleaser builds the three user-facing commands across linux/darwin/windows and amd64/arm64, matching the Dockerfile's flags. cmd/bench ships as 'pokearena-bench' — a file called 'bench' in a downloads folder tells you nothing. Registry publishing is a separate job so a registry outage cannot cost the binaries. Also ignore /dist/, which goreleaser writes to.
The engine is a pure function and cmd/bench already drives it in-process with no services, but both of those are reachable only from Go. The audience we want — RL and LLM-eval researchers — works in Python, and their tooling reaches for a gym.make-shaped thing. There was no way in that did not start with 'learn Go'. This is the seam: one line of JSON per request on stdin, one per response on stdout. reset / step / legal_actions / observe / close, plus an info handshake that reports protocol version, engine revision and the dataset's provenance, because a result that cannot say what produced it cannot back the reproducibility claim. It reads the embedded dataset, so it runs from any directory with no data/ on disk. Observations are built through the same fog-of-war path the arena uses rather than a second implementation: a side sees its own team in full and only the foe's active, as hp_pct with the absolute HP, stats, EVs, IVs, ability, nature and move ids withheld, and the foe's bench as a count. Tests cover determinism, divergence on a different seed, that no hidden field leaks, and illegal-action rejection.
The engine has been reachable only from Go. The people we most want using it — RL and LLM-eval researchers — work in Python, and their tooling reaches for a gym.make-shaped thing, which is why TextArena and the rest standardised on that shape. This is the way in. A Gymnasium-style single-agent env (a baseline drives the other side) and a PettingZoo-style two-agent env, both subprocessing cmd/pokearena-env. No required runtime dependencies: gymnasium, pettingzoo and numpy are extras, and the package imports and runs without any of them. A missing binary raises an actionable error naming the go install line, never a silent failure. The action space is a fixed Discrete(11) — four move slots, Struggle, six switches — never renumbered between turns, with legality carried as a mask, because an action space that renumbers is not learnable. step() now accepts every shape a caller might plausibly hold: a bare int, a legal_actions() record, that record's inner action object, or its index. It previously forwarded the whole record to the engine, which rejected it for a missing 'kind' the caller never typed — step(legal_actions()[0]) is the first line anyone writes, and it failed. A bad shape now raises a Python TypeError naming the forms that work instead of surfacing a message from the Go layer. 52 tests pass against the binary and skip — loudly, saying nothing was verified — without it.
The @latest note was written before the dataset was embedded and listed two blockers. Only one is left: the tag.
Tagging by hand means a clean local checkout and remembering the gates. Add a 'version' input that does it from the Actions tab instead. The tag job runs build, vet and test before creating anything, so a commit that fails the suite never gets tagged. Everything that can refuse a release happens while the tag still does not exist: an unpushed tag costs nothing to abandon, a pushed one is public history that a release may already point at. Tagging lives in this workflow rather than a dedicated one because a tag pushed with GITHUB_TOKEN cannot trigger another workflow — that is GitHub's recursion guard — so a separate tagger would create the tag and then nothing would happen. Doing both in one run needs no secret. The registry publish was gated on 'event_name == push', which would have skipped it for a tag this workflow cut itself. It is now gated on a flag the release job sets, true for a tag push or a freshly created tag and false when only rebuilding an existing tag's artifacts, which is the case the guard was actually there for.
golangci-lint enforces locale US and gofumpt, which is stricter than the gofmt I had been checking with. Three British spellings and one composite literal that gofumpt splits across lines.
The most likely first experience of anyone who installs pokearena-mcp from the MCP registry is a refused connection: the default gateway is ws://localhost:8080 and they have no arena running. They got 'connection refused' and nothing else — no hint that an arena is a thing they need, let alone how to get one. An agent relaying that message could not help either. Name the URL we tried and both ways forward. Only an unreachable dial gets this; a gateway that answered and refused us has a different problem and keeps its own error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The engine was never the problem. Someone could find the GitHub page and then
not install it, not legally use it, and not run it — so nothing downstream of
"found the repo" worked. This fixes that chain end to end.
The four blockers
module pokearenawas not a resolvable import path.go install github.com/shaumik/PokeArena/cmd/...@latestfailed and the repo did not appearon pkg.go.dev at all. Renamed, which rewrote 402 import lines. The sed was
scoped to import-shaped references only — a naive replace would have corrupted
"pokearena-mcp","pokearena.events"and"pokearena.work", a tool name andtwo RabbitMQ exchange names.
No LICENSE meant all rights reserved. Unusable in published research,
unusable commercially, and rejected on sight by the registries and awesome-lists
worth being listed in. MIT, keeping the trademark position the README already
stated.
cmd/benchcould not run outside a clone. It loaded the dex viadomain.LoadDex("data")and defaulted-teamsto a relative path, so it diedat dex load from any other directory. The zero-setup entry point only ever
opened from inside a checkout. The module-root embed now covers
benchmark-teams.json,_provenance.jsonandmodel-pricing.json, and benchfalls back to it when the dataset directory is absent. An explicit
-datathatdoes not exist is still a hard error.
No tags meant no registry listing. The official MCP Registry only accepts
artifacts from GitHub or GitLab releases, so the project was structurally
ineligible for the place MCP clients browse.
What else is here
same README admits is unverified.
cmd/benchis the first runnable block;docker compose upkeeps its section further down. The heading is nowConnect your agent (Pv-Agent), repairing an anchorARCHITECTURE.mdandmcp-protocol.mdwere already linking to.AGENTS.mdfor the agent doing the discovering: the no-infra path, whatdoes and does not need services, the MCP tool surface, observation and action
shapes.
pip install pokearena— Gymnasium and PettingZoo shaped, subprocessing anew
cmd/pokearena-envthat speaks JSONL over stdio. No required runtimedependencies. Observations come from the same
ai.MakeViewredaction path theMCP server and PvP socket use, so fog of war is not reimplemented for a second
consumer.
server.json. The registry has no Go package type, sothe server ships as an MCPB bundle built from a release artifact. GoReleaser
covers three commands across linux/darwin/windows and amd64/arm64.
release.ymltakes aversioninput; thetag job runs build/vet/test before creating the tag. Tagging lives in that
workflow rather than a dedicated one because a tag pushed with
GITHUB_TOKENcannot trigger another workflow, so a separate tagger would tag and then do
nothing.
CITATION.cff, a Claude Code plugin, anddocs/deeper-search-played-worse.md— benchmark.md §6 written up assomething a stranger will read, keeping the part that makes it honest: the
corrected search model is weaker than the buggy one, because the phantom KO
had been inducing helpful aggression.
Verification
go build ./...,go vet ./...,gofmt -l .clean. Fullgo test ./...—17 packages, all pass.
nothing about the engine was verified rather than reporting a false green.
This was the risk worth checking — an embedded dataset that drifts from
data/would make published numbers unreproducible, which is the project'score claim.
server.jsonvalidates against the 2025-12-11 schema and against the liveregistry via
mcp-publisher validate. All four workflow dispatch pathssimulated.
Known gaps
gymnasium/pettingzoocould not be installed in the build container,so only the degraded no-dependency path was exercised. The subclassing is
guarded and provably import-safe, but nobody has watched
PokeArenaEnvactually inherit from
gymnasium.Env. Worth a CI job with[all]installedbefore the Python package is announced anywhere.
benchmark.md§6 may want a re-sweep. A fresh run of the depth sweep gaveexpectimax 37.5% [31.6, 43.8] where the doc publishes 36.7% [30.8, 42.9].
Intervals overlap so nothing is contradicted, but
_provenance.jsonshows thedataset moved after that table was measured, and §6 is described in that doc
as its only load-bearing measurement.
golangci-lintcould not run here — the container's build targets go1.25while
go.modis on 1.26. Pre-existing, unrelated to this branch; CI runs thepinned v2.12.2.
POKEARENA_GATEWAY_URLstill defaults tows://localhost:8080, so anyonearriving via the MCP Registry must run
docker compose upbefore the serverdoes anything. The benchmark's friction is now zero; the MCP path's is not.
Nothing here is merged-and-live until
v0.1.0is tagged — release binaries, theregistry record,
go installand pkg.go.dev all wait on it.docs/publishing.mdis the runbook.
🤖 Generated with Claude Code
https://claude.ai/code/session_012m9mPxYV2MeV7kGt3rXc63
Generated by Claude Code