feat: consolidate eval cli

[mikasenghaas]

Summary

Migrates the eval CLI from verifiers into the prime-rl ecosystem: uv run eval is prime-rl's entrypoint for evaluating any model on any environment, standalone or as the online evals of a training run.

Why:

• consolidation of features: adaptive concurrency, the local dashboard, resume from the trace stream, live rollouts
• easier code sharing between the CLI entrypoints (rl, sft, eval)
• direct access to the pre-installed environments (prime-envs submodule)
• one workspace for post-training: training and evals

Important:

• the verifiers eval CLI stays for a transition period as the quick in-process path, renamed uv run vf-eval so prime-rl owns eval (feat: remove the eval CLI verifiers#2610 removes it later); env servers and their streamed traces (feat: stream served episodes turn by turn verifiers#2512, merged; the submodule points at verifiers main) are consumed here only
• evals run through env servers, for scalability; the launcher (eval, sft, rl) owns them
• --resume continues an interrupted run from its trace stream: landed episodes rejoin the epoch, only the owed rollouts run; num_examples/group_size may change, the model, sampling and env config may not
• multi-env capable ([[source]] blocks); the single-env command shorthands are preserved
• adaptive concurrency only against local deployments (it reads vLLM /metrics); against an API, concurrency is pinned via min_inflight = max_inflight (the default pins 128)
• prime-rl's core dependencies are plain CPU; install them and the environments with uv sync --all-packages
• Prime runs and traces integrate through the single monitor abstraction (--monitors.prime, --monitors.wandb, the file monitor)

Examples:

uv run eval gsm8k                                            # default model on Prime Inference
uv run eval gsm8k -n 32 -r 4 -m openai/gpt-5.6-luna           # 32 tasks × 4 rollouts, another model
uv run eval terminal-bench-2 --env.taskset.tasks '["fix-git"]' --env.agent.harness.id bash -n 1 -r 2
uv run eval @ configs/debug/eval/single-turn.toml            # gsm8k, null harness
uv run eval @ configs/debug/eval/multi-turn.toml             # 16 terminal-bench-2 tasks, bash harness
uv run eval @ configs/debug/eval/multi-env.toml              # terminal-bench-2 under bash and rlm side by side
uv run eval @ configs/debug/eval/aime2026.toml               # AIME 2026, avg@16
uv run eval @ configs/debug/eval/tb2.toml                    # Terminal-Bench 2, avg@4
uv run eval @ configs/debug/eval/resume.toml --run.name x    # then: --resume
uv run eval gsm8k --monitors.prime                           # stream the evaluation to the platform

Breaking

• uv run evals is uv run eval (EvalsConfig → EvalConfig); config shape: [eval.client] → [client], [eval.concurrency] → [concurrency], [[eval.source]] → [[source]], eval.num_examples/eval.group_size → top level. [online] is gone: online evals are spawned by sft.
• eval has no [ckpt] block; --resume is a switch that reads the run's trace stream.
• SFT online evals: no public entrypoint (uv run sft with an [eval] block spawns it); logs/attempt_N/evals.log → eval.log, evals.json → eval.json, W&B label online-eval; EvalsEvalConfig → SFTOnlineEvalConfig.
• ratio is accepted on training sources only, interval on online eval sources only; a standalone eval's [[source]] takes neither.
• orchestrator.env_server_base_port and the eval entrypoint's env_server_base_port are removed: launcher-managed env servers bind an OS-assigned port and publish it (configs/attempt_N/resolved/envs/<split>/<name>.address); serve.address pins one. uv run env-server without serve.address binds an OS-assigned port (address_file publishes it).
• configs/evals/swe.toml is removed (uv run eval swebench-verified --env.agent.harness.id bash).

Verification

• Unit: uv run pytest tests/unit green (configs, orchestrator, eval, monitors incl. the live reader).
• Debug configs: single-turn, multi-turn, multi-env and aime2026 run to completion against Prime Inference. Resume: single-env and multi-env runs interrupted with SIGTERM and resumed hold exactly examples × group_size unique episodes with the plan and epoch metrics over the full set; resuming with a larger group_size runs only the extra rollouts; resuming with another model is refused.
• Core install (uv sync without extras) runs uv run eval; an SFT run with online evals starts one env server per source.
• Live streaming, on the final code: gsm8k 16 single-turn episodes, terminal-bench-2 fix-git with bash (13-14 turns, the pending preview alternating with committed turns in the live files), kuhn-poker 4 two-seat episodes (8 traces), a 5-step reverse-text RL run (768 train episodes with token ids, logprobs and masks); no assembly mismatch.
• Platform: --monitors.prime on a two-rollout fix-git eval opened the evaluation 9 s after launch, streamed both rollouts and closed it; the dashboard linked to it throughout.
• Dashboard behaviour checked in headless Chromium: live rows and viewer follow, handover in place, metrics pane per env, filter menus, platform button states.

🤖 Generated with Claude Code

---

Note

Medium Risk
Breaking CLI and config renames affect existing scripts; env-server address discovery replaces fixed ports across rl, sft, and eval launch paths.

Overview
Replaces uv run evals with uv run eval as the standalone evaluation entrypoint, sharing the orchestrator eval pipeline (per-source env servers, adaptive or pinned concurrency, file/W&B/Prime monitors). Config is flattened ([[source]], [client], [concurrency] at top level); --resume continues from the trace stream instead of a [ckpt] cursor; SFT online eval is renamed and typed as SFTOnlineEvalConfig (no public [online] block).

Env server wiring changes: fixed env_server_base_port ranges are removed—launcher-managed servers bind OS-assigned loopback ports and publish addresses via address_file / configs/.../envs/<split>/<name>.address. Training ratio and online-eval interval are scoped to the right source types only.

Docs and examples add docs/eval.md, per-example eval.toml files, and configs/debug/eval/* smokes; configs/evals/swe.toml is deleted. CI adds a gsm8k eval integration test on VM runners.

Dashboard gains live rollout APIs and eval-centric metrics (plan-driven progress, platform links, eval run status), renames evals.log → eval.log, and tightens static asset revalidation.

^{Reviewed by Cursor Bugbot for commit 82f2f0e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit db514fd. Configure here.}

[hallerite]

Thanks for addressing the first round. I checked the updated stack: the eval runner now imports without torch, server ownership is fixed, and resume restores landed episodes and the full epoch count. I found two remaining resume correctness issues and a concurrency issue in the live dashboard cache, detailed inline.

Validation: 27 targeted prime-rl tests and 99 verifiers non-live tests passed, plus local reproductions of the issues below.

[hallerite]

LGTM, great work