feat: stream served episodes turn by turn by mikasenghaas · Pull Request #2512 · PrimeIntellect-ai/verifiers

mikasenghaas · 2026-09-02T20:14:16Z

Summary

Served episodes stream turn by turn over the env-serve wire (verifiers/v1/serve/delta.py). Trace.notify fires at every phase change and after every recorded turn; the worker's DeltaStreamer diffs the run's live traces against what it already sent and ships the new part (header once, appended nodes/calls/errors, semantic links, changed scalars), so every byte crosses the wire once. A cursor advances only after its delta is on the wire. EnvClient.run assembles the episode from the deltas (EpisodeAssembly, which copies the nodes it keeps so a later link never mutates a delta the caller holds), validates it once, and takes on_delta so a caller can relay or persist the stream; the client's receive loop survives a malformed frame.
pending preview: the interception server hands each model request's uncommitted tail (tool results, user turns) to Trace.preview, so a watcher sees a tool result before the model has answered; the committed turn replaces it and the finished record never carries it.
The eval CLI is uv run vf-eval and runs in-process only. Serving an env to many consumers is prime-rl's job (its uv run eval runs env servers and consumes this stream), so the CLI keeps the quick local path and never assembles deltas; its TUI footer and console log point at prime-rl's eval. The env-server e2e fixture drives a worker pool through EnvClient directly.

Based on main; the eval CLI stays for a transition period (#2610 removes it later). prime-rl PrimeIntellect-ai/prime-rl#3471 consumes the stream for its live traces.

Breaking

Env-serve wire: replies are 4 frames (client_id, request_id, kind, data) with kind = delta | reply; RunResponse carries the episode head and per-trace counts instead of the traces. A client and server must be on the same side of this change.
Every console script is prefixed: vf-eval, vf-validate, vf-debug, vf-replay, vf-init, vf-gepa. Generic names collide with the scripts of a workspace that installs verifiers as a dependency (prime-rl's eval), and the winner is whichever package was installed last. vf-eval has no [serve] / --serve.* / --no-serve: rollouts always run in-process; use prime-rl's uv run eval for env-server runs. Existing venvs: uv sync --reinstall-package verifiers --reinstall-package prime-rl regenerates the scripts.
Env.run_slot takes on_trace; Trace gains watch/notify/preview/clear_preview.

Verification

tests/v1/test_serve_delta.py: turns, a link onto an earlier node, a retried attempt and the final reward round-trip through DeltaStreamer → EpisodeAssembly → WireEpisode; a refused send is diffed again; the pending preview streams, is replaced by the committed turn and never reaches the record; assembling never mutates a delta; the field lists cover every serialized Trace field.
uv run pytest tests/v1 -m "not e2e" green apart from the config-parse cases of tasksets not installed in the venv; ruff clean.
End to end through prime-rl: gsm8k single turn, terminal-bench-2 fix-git with bash and with the rlm harness delegating to a sub-agent (semantic links subagent_call/subagent_return), kuhn-poker two-seat episodes, 5-step reverse-text RL runs with token ids and logprobs, including router replay plus sampling replay on a MoE; no assembly mismatch.

🤖 Generated with Claude Code

Note

Stream served episodes turn by turn with delta frames

Adds DeltaStreamer in delta.py that watches trace changes during run_slot and sends incremental deltas; the client EpisodeAssembly reconstructs a WireEpisode from those deltas plus a final episode head and TraceSummary counts.
Changes the wire protocol to three-frame messages (request ID, kind, data). Delta frames are relayed by the pool and client without closing the request; reply frames close it. Updated in server.py, pool.py, and client.py.
Adds a trace watcher/preview API in trace.py. Env.run_slot gains an on_trace callback; Rollout.open/close and InterceptionServer notify the trace on phase changes and pending turn previews.
PendingTurn.commit/commit_prompt now clear the trace's pending preview after committing.
Risk: RunResponse in types.py removes the serialized WireEpisode field and replaces it with an episode head and TraceSummary list — any out-of-tree consumer expecting the full episode in the run response will break.

Changes since #2512 opened

Renamed console script entry points in pyproject.toml and updated all documentation and usage strings to use vf- prefixed command names [02a6eff]
Refactored trace preview system from unkeyed to keyed architecture [55500a3]
Updated all trace preview and clear operations to use keyed API with PendingTurn instances as keys [55500a3]
Added abandon mechanism for cleaning up previews when turns fail or are cancelled [55500a3]
Changed verifiers.v1.trace.Trace to store preview entries by id() of the key object rather than by the key object itself [73d602d]
Updated PRIME_RL_HINT constant in the hint.py module [fc7e45b]

^{Macroscope summarized fe771d8.}

Note

High Risk
Breaking env-serve wire protocol and RunResponse shape require matching client/server versions; CLI renames and removal of eval’s serve path change how users run evaluations.

Overview
This PR streams env-server episodes incrementally instead of returning one fat RunResponse. Workers use DeltaStreamer to diff live traces on Trace.notify() and send delta frames; clients EpisodeAssembly rebuilds traces and validate once against a final head + TraceSummary counts. The wire format is now [request_id, kind, data] with kind = delta | reply; the pool relays deltas without closing the request.

Trace gains watch / notify / keyed preview / clear_preview; rollouts and the interception server fire notifications (including pending tool/user tails before commit). Env.run_slot adds on_trace so the server can subscribe traces to the streamer.

vf-eval is renamed and narrowed: console scripts are vf-* to avoid clashing with prime-rl’s eval, docs/config comments follow suit, and vf-eval always runs in-process—[serve] / --no-serve are removed from eval config—with dashboard/log hints to use prime-rl for env-server scale and live dashboards.

^{Reviewed by Cursor Bugbot for commit fc7e45b. Bugbot is set up for automated code reviews on this repo. Configure here.}

Evaluation lives in prime-rl (`uv run evals`): multi-env, adaptive concurrency, cursor checkpoints, dashboard and platform upload. Keeping a second single-env `eval` CLI here only confused which one to use, so it goes: the entrypoint, its rich live view, traces.jsonl resume, platform push, EvalConfig and the root configs/*.toml, plus the evaluation doc and skill. RunConfig and default_run_name move to configs/cli/run.py for validate and gepa. output.py types its run directory through a small protocol. The platform module keeps the sample format and credentials prime-rl's monitors use. The e2e suite runs envs through a test-local runner (tests/v1/runner.py) covering the in-process and env-server paths. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

`uv run eval` replaces `uv run evals` so verifiers users keep their command; the sidecar is `online-eval`. Modules, configs (EvalConfig, OnlineEvalConfig, SFTEvalConfig; the orchestrator's interval block is ScheduledEvalConfig), log files and resolved-config names follow. examples/eval/ gains TOMLs for the CLI usage verifiers used to document (best-of-n, wiki-search, terminal-bench-2 with retries, rlm in docker with sampling) and a README. A GPU integration test runs `uv run eval` against a local vLLM server from configs/ci/integration. The verifiers submodule points at the branch that removes its `eval` console script, so the shared venv has one `eval`; re-pin to main once PrimeIntellect-ai/verifiers#2512 lands. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

prime-rl owns the upload path (its train and eval monitors), so the sample format and credentials move there too. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…dy created (#2569) ## Summary A hosted evaluation creates its evaluation record before the sandbox exists and hands the sandbox that id as `$EVALUATION_ID`. Until now the sandbox's `uv run eval` opened a run of its own through prime-runs, so a hosted v1 eval would produce a second, unlinked evaluation while the platform's record sat at running. This adds the one knob the platform needs to run the same command a researcher runs locally. - **`run.attach <evaluation-id>`** on `EvalConfig`: `open_run` passes it to `prime_runs.init(id=...)`. prime-runs 0.1.2 (PrimeIntellect-ai/prime#913, released) attaches eval runs the way it already attached training runs: no create, sinks keyed to that id, `/finalize` on a clean finish, failure marking left to the launcher (the platform reads the sandbox exit code), and the launcher's `metadata` document left alone. Requires `push`, validated on the config. - **No local fallback when attaching.** Without `run.attach` the behaviour is unchanged: a run that cannot be opened is logged and the eval continues locally. With it, a failed attach or the SDK's `PRIME_RUNS_MODE=disabled` kill switch raises: a local run nobody reads is not a substitute while the launcher's run waits until it times out. - Docs bullet under "Common config values", `prime-runs>=0.1.2` and the relock (the lock diff is the one package). Nothing changes for local runs; `run.attach` is unset unless a launcher sets it. ## Verified against prod With this branch and prime-runs from `main` (== 0.1.2): an evaluation created launcher-style through the evals API (`metadata={"launcher": "hosted-smoke"}`), then ``` eval alphabet-sort -n 1 -r 1 --no-serve --no-rich --run.attach <that id> ``` exited 0 with `--push: completed -> .../evaluations/<that id>`. Afterwards the evaluation had `total_samples` 1, the run's metrics (`avg_reward`, `avg_metrics.alphabet_sort`, `avg_error`), its original metadata byte for byte, and status PROCESSING; no second evaluation was created. The same command without `--run.attach` still creates its own run as before (checked earlier on the SDK side). Smoke evaluations deleted. ## Test plan - New `tests/v1/test_platform_run.py` (mocked `pr.init`): attach needs push; no attach → `id=None` and the created run's id is assigned; attach → `id=<attach>` and the config carries that id; failed attach raises; failed open without attach still falls back to a local run; kill switch refuses an attach. - `tests/v1/test_configs.py` still parses every checked-in config. ruff check + format clean. ## Follow-up (not here) `EvalsBackend.attach` makes no request, so a wrong id only surfaces on the first upload (404 in the run's footer) and the eval still exits 0. For hosted evals the id is always the platform's own; the platform's v1 runner will check `total_samples` on exit rather than trusting the exit code alone. Relates to #2512 (if the `eval` console script moves, the hosted runner's command moves with it). Linear: ENG-5781. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01XrcqANSZ24BALuWS9MciYv  --- > [!NOTE] > **Medium Risk** > Changes platform run lifecycle for hosted evals (hard failures on attach); local-only evals are unchanged when `run.attach` is unset. > > **Overview** > Adds **`run.attach <evaluation-id>`** so hosted sandboxes can upload into an evaluation the launcher already created instead of opening a second run. `open_run` passes that id to **`prime_runs.init(id=...)`** (bump to **0.1.2**); the config gets the same id for traces and the run dir. > > **Attach is strict:** it requires **`push`**, rejects **`PRIME_RUNS_MODE=disabled`**, and **does not fall back** to a local run on attach failure—unlike a normal push open, which still continues locally. Docs cover **`push`** and **`run.attach`** under common eval config. > > New **`tests/v1/test_platform_run.py`** mocks `pr.init` for attach validation, success paths, fallback without attach, and kill-switch behavior. > > Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 2a989a8. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).     > [!NOTE] > ### Add `run.attach` to stream hosted evals into existing platform runs in v1 > - `RunConfig.attach` identifies an existing platform run to attach a hosted sandbox evaluation to. The `attach_needs_push` validator rejects configurations that set `run.attach` with push disabled. > - [platform.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2569/files#diff-8644072f827be448db5b0010e0969c6c86d9bbf07191ca00a993f416d123235f) `open_run` passes the attach id to online SDK initialization. Attached evaluations fail with a runtime error if attachment cannot be performed, instead of falling back to a disabled local run. Unattached runs keep the existing fallback behavior. > - The SDK kill switch rejects attach requests before initialization, preventing a silent conversion to a local run. > - Raises the minimum `prime-runs` dependency from 0.1.1 to 0.1.2. > - Risk: `open_run` now raises an attachment-specific runtime error on SDK failures for attached runs (no local fallback); reviewers should verify callers in [platform.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2569/files#diff-8644072f827be448db5b0010e0969c6c86d9bbf07191ca00a993f416d123235f) handle this error path. > >  > > <a href="https://app.macroscope.com">Macroscope</a> summarized 2a989a8. >  >  Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

Main moved the eval CLI's platform upload onto prime-runs (#2415, #2569, #2577) and taught the runner `run.attach`. All of that lives in files this branch deletes, so the merge keeps the deletions and drops the new `tests/v1/test_platform_run.py` with them. Nothing in verifiers imports `prime_runs` any more, so the dependency goes too; prime-rl depends on it directly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

A served rollout was visible only as its final episode; the env-serve wire now streams each trace as it changes. The rollout fires Trace.notify at every phase change and the interception proxy after every recorded turn; the worker's DeltaStreamer diffs the live traces against what it already sent and ships the new part (header once, appended nodes/calls/errors, links onto earlier nodes, changed scalars). The reply that ends a run carries the episode head and per-trace counts, and the client assembles the episode from the deltas, so the stream costs about what the one-shot reply did. Frames gain a kind (`delta` | `reply`); the pool relays deltas without closing the request. EnvClient.run takes on_update to watch the assembly grow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

A consumer that persists the stream needs the delta itself, not only the assembly it produced. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

One primitive for a served rollout's progress: the delta. The client still assembles the episode it returns from the same deltas. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

`reasoning_details` items (`reasoning.text`, `reasoning.summary`, `reasoning.encrypted`) were hashed as raw provider state, so a harness that replays them through its own SDK, which concatenates the streamed `format` field per chunk, never matched the committed assistant node and forked a branch every turn. Their text is already the reasoning content; only the fields required on replay (an encrypted blob, a signature) count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The eval CLI is gone, so the per-taskset smoke test shells out to nothing; it now runs each taskset in-process through the shared runner (one capped rollout, the taskset's own harness). The e2e config builder drops its stale platform-push kwarg. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

A tool result reaches the framework inside the harness's next model request and its node only lands when that turn commits, after the model has answered, so a watcher saw it a whole model call late. The interception server now hands the request's uncommitted tail to the trace as a preview; the delta stream ships it as pending, the assembly keeps it until the turn's nodes land, and the finished record never carries it.

A send that failed was logged and its delta forgotten, so the client tripped over the count check at the end of an otherwise good run; a flush scheduled at the last notify could also start after the streamer had exited. Each trace's delta now carries the cursor it would leave behind, stored only after the send succeeds, and the exit cancels the pending schedule and lets in-flight flushes finish before the final one. The client's receive loop logs and skips a malformed frame or a raising delta handler instead of dying and hanging every request. Trace.clear_preview replaces poking the private field from graph.py.

prime-rl's `uv run eval` replaces it: the `eval` console script, its runner, resume, rich dashboard, platform upload and example configs go, along with the `prime-runs` dependency. `RunConfig` moves to `configs/cli/run.py` for the run-directory CLIs that stay (validate, gepa), and the e2e suite runs tasksets in-process.

Main fixed the cause in the bash harness's streaming path (#2604), so replayed reasoning details match verbatim and the hash needs no special case; the regression test for the mangled form goes with it.

hallerite

The streaming approach looks useful. I found one mutation issue at the boundary between the client assembly and consumers that buffer deltas, described inline.

The scaffold installs only verifiers, which has no eval executable since the eval CLI moved to prime-rl. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The assembly kept the node dicts of the delta itself, so a later links delta grew semantic_parents inside a delta the caller already held. A consumer that serializes deltas after the fact (prime-rl's buffered live writer) then saw the link twice: once in the node, once as the link update. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The CLI's removal moves to its own PR for a transition period; this branch carries only the streamed traces. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Serving an env to many consumers is prime-rl's job: its eval runs env servers and consumes their delta stream. The CLI keeps the quick local path, so it never assembles deltas. The env-server test fixture drives a worker pool through EnvClient directly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Both packages install an `eval` script and whichever is installed last wins, so in a prime-rl workspace the verifiers entry hands over to prime-rl's entrypoint. Standalone, the TUI footer and the console log recommend prime-rl's eval: streamed traces, a live dashboard, multi-env runs and env servers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

macroscopeapp · 2026-09-16T00:10:08Z

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds turn-by-turn episode streaming with a new delta wire protocol and client-side assembly, while also changing the eval CLI from server-backed execution to in-process execution. The resulting changes span shared serving, trace, rollout, interception, retry, serialization, and dashboard paths, creating a broad runtime and compatibility surface.

^{You can add or adjust custom eligibility rules. Learn more.}

prime-rl's eval owns the `eval` script; two packages installing the same script leave the winner to install order, and the verifiers entry should not look for prime-rl. The CLI keeps working under `vf-eval` for the transition. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

`vf-eval`, `vf-validate`, `vf-debug`, `vf-replay`, `vf-init`, `vf-gepa`: generic names collide with the scripts of a workspace that installs verifiers as a dependency (prime-rl's `eval`), and the winner is whichever package was installed last. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Parallel agents share one trace, so a single preview list let one request's preview overwrite another's and any commit clear them all. Each request now previews under its own key; a commit clears only its own, and a request that fails or is cancelled before committing takes its preview back before the failure is recorded. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 55500a3. Configure here.}

A PendingTurn is a dataclass and not hashable. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

hallerite

Thanks for fixing the delta aliasing issue; the node/link copying addresses the earlier finding, and the 99 non-live verifiers tests passed locally. One remaining compatibility issue: holding the CLI-removal PR does not by itself preserve hosted evals, because this PR renames the executable they invoke.

hallerite

lgtm

## Summary Migrates the eval CLI from verifiers into the prime-rl ecosystem: `uv run eval` is prime-rl's entrypoint for evaluating any model on any environment, standalone or as the online evals of a training run. Why: - consolidation of features: adaptive concurrency, the local dashboard, resume from the trace stream, live rollouts - easier code sharing between the CLI entrypoints (`rl`, `sft`, `eval`) - direct access to the pre-installed environments (`prime-envs` submodule) - one workspace for post-training: training and evals Important: - the verifiers eval CLI stays for a transition period as the quick in-process path, renamed `uv run vf-eval` so prime-rl owns `eval` (PrimeIntellect-ai/verifiers#2610 removes it later); env servers and their streamed traces (PrimeIntellect-ai/verifiers#2512, merged; the submodule points at verifiers `main`) are consumed here only - evals run through env servers, for scalability; the launcher (`eval`, `sft`, `rl`) owns them - `--resume` continues an interrupted run from its trace stream: landed episodes rejoin the epoch, only the owed rollouts run; `num_examples`/`group_size` may change, the model, sampling and env config may not - multi-env capable (`[[source]]` blocks); the single-env command shorthands are preserved - adaptive concurrency only against local deployments (it reads vLLM `/metrics`); against an API, concurrency is pinned via `min_inflight = max_inflight` (the default pins 128) - prime-rl's core dependencies are plain CPU; install them and the environments with `uv sync --all-packages` - Prime runs and traces integrate through the single monitor abstraction (`--monitors.prime`, `--monitors.wandb`, the file monitor) Examples: ```bash uv run eval gsm8k # default model on Prime Inference uv run eval gsm8k -n 32 -r 4 -m openai/gpt-5.6-luna # 32 tasks × 4 rollouts, another model uv run eval terminal-bench-2 --env.taskset.tasks '["fix-git"]' --env.agent.harness.id bash -n 1 -r 2 uv run eval @ configs/debug/eval/single-turn.toml # gsm8k, null harness uv run eval @ configs/debug/eval/multi-turn.toml # 16 terminal-bench-2 tasks, bash harness uv run eval @ configs/debug/eval/multi-env.toml # terminal-bench-2 under bash and rlm side by side uv run eval @ configs/debug/eval/aime2026.toml # AIME 2026, avg@16 uv run eval @ configs/debug/eval/tb2.toml # Terminal-Bench 2, avg@4 uv run eval @ configs/debug/eval/resume.toml --run.name x # then: --resume uv run eval gsm8k --monitors.prime # stream the evaluation to the platform ``` ## Breaking - `uv run evals` is `uv run eval` (`EvalsConfig` → `EvalConfig`); config shape: `[eval.client]` → `[client]`, `[eval.concurrency]` → `[concurrency]`, `[[eval.source]]` → `[[source]]`, `eval.num_examples`/`eval.group_size` → top level. `[online]` is gone: online evals are spawned by `sft`. - `eval` has no `[ckpt]` block; `--resume` is a switch that reads the run's trace stream. - SFT online evals: no public entrypoint (`uv run sft` with an `[eval]` block spawns it); `logs/attempt_N/evals.log` → `eval.log`, `evals.json` → `eval.json`, W&B label `online-eval`; `EvalsEvalConfig` → `SFTOnlineEvalConfig`. - `ratio` is accepted on training sources only, `interval` on online eval sources only; a standalone eval's `[[source]]` takes neither. - `orchestrator.env_server_base_port` and the eval entrypoint's `env_server_base_port` are removed: launcher-managed env servers bind an OS-assigned port and publish it (`configs/attempt_N/resolved/envs/<split>/<name>.address`); `serve.address` pins one. `uv run env-server` without `serve.address` binds an OS-assigned port (`address_file` publishes it). - `configs/evals/swe.toml` is removed (`uv run eval swebench-verified --env.agent.harness.id bash`). ## Verification - Unit: `uv run pytest tests/unit` green (configs, orchestrator, eval, monitors incl. the live reader). - Debug configs: single-turn, multi-turn, multi-env and aime2026 run to completion against Prime Inference. Resume: single-env and multi-env runs interrupted with SIGTERM and resumed hold exactly examples × group_size unique episodes with the plan and epoch metrics over the full set; resuming with a larger `group_size` runs only the extra rollouts; resuming with another model is refused. - Core install (`uv sync` without extras) runs `uv run eval`; an SFT run with online evals starts one env server per source. - Live streaming, on the final code: gsm8k 16 single-turn episodes, terminal-bench-2 fix-git with bash (13-14 turns, the pending preview alternating with committed turns in the live files), kuhn-poker 4 two-seat episodes (8 traces), a 5-step reverse-text RL run (768 train episodes with token ids, logprobs and masks); no assembly mismatch. - Platform: `--monitors.prime` on a two-rollout fix-git eval opened the evaluation 9 s after launch, streamed both rollouts and closed it; the dashboard linked to it throughout. - Dashboard behaviour checked in headless Chromium: live rows and viewer follow, handover in place, metrics pane per env, filter menus, platform button states. 🤖 Generated with [Claude Code](https://claude.com/claude-code)  --- > [!NOTE] > **Medium Risk** > Breaking CLI and config renames affect existing scripts; env-server address discovery replaces fixed ports across `rl`, `sft`, and `eval` launch paths. > > **Overview** > **Replaces `uv run evals` with `uv run eval`** as the standalone evaluation entrypoint, sharing the orchestrator eval pipeline (per-source env servers, adaptive or pinned concurrency, file/W&B/Prime monitors). Config is **flattened** (`[[source]]`, `[client]`, `[concurrency]` at top level); **`--resume`** continues from the trace stream instead of a `[ckpt]` cursor; SFT **online eval** is renamed and typed as `SFTOnlineEvalConfig` (no public `[online]` block). > > **Env server wiring changes:** fixed `env_server_base_port` ranges are removed—launcher-managed servers bind **OS-assigned loopback ports** and publish addresses via **`address_file`** / `configs/.../envs/<split>/<name>.address`. Training **`ratio`** and online-eval **`interval`** are scoped to the right source types only. > > **Docs and examples** add `docs/eval.md`, per-example `eval.toml` files, and `configs/debug/eval/*` smokes; **`configs/evals/swe.toml` is deleted**. CI adds a **gsm8k eval** integration test on VM runners. > > **Dashboard** gains live rollout APIs and eval-centric metrics (plan-driven progress, platform links, eval run status), renames **`evals.log` → `eval.log`**, and tightens static asset revalidation. > > Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 82f2f0e. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).  --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

mikasenghaas and others added 2 commits September 2, 2026 20:13

docs: point evaluation at prime-rl's uv run eval

061d384

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mikasenghaas mentioned this pull request Sep 2, 2026

feat: consolidate eval cli PrimeIntellect-ai/prime-rl#3471

Merged

feat: drop the platform sample format

08a6681

prime-rl owns the upload path (its train and eval monitors), so the sample format and credentials move there too. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

kcoopermiller mentioned this pull request Sep 9, 2026

feat(v1): run.attach streams the eval into a run the platform already created #2569

Merged

mikasenghaas and others added 2 commits September 14, 2026 18:03

mikasenghaas changed the title ~~feat: remove the eval CLI~~ feat: consolidate evals + streamed traces Sep 15, 2026

mikasenghaas and others added 3 commits September 15, 2026 01:24

hand the applied delta to EnvClient.run's on_update

c0ab937

A consumer that persists the stream needs the delta itself, not only the assembly it produced. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

EnvClient.run hands each delta to on_delta

b963fb2

One primitive for a served rollout's progress: the delta. The client still assembles the episode it returns from the same deltas. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mikasenghaas mentioned this pull request Sep 15, 2026

fix: preserve streamed reasoning details #2604

Merged

mikasenghaas and others added 6 commits September 15, 2026 02:52

Merge branch 'main' into feat/remove-eval-cli

62d2a36

Merge remote-tracking branch 'origin/main' into feat/remove-eval-cli

77fb1be

mikasenghaas mentioned this pull request Sep 15, 2026

feat: remove the eval CLI #2610

Open

mikasenghaas added 2 commits September 15, 2026 17:25

drop the reasoning-details hash workaround

e7a7b0f

Main fixed the cause in the bash harness's streaming path (#2604), so replayed reasoning details match verbatim and the hash needs no special case; the regression test for the mangled form goes with it.

Merge branch 'feat/drop-eval-cli' into feat/remove-eval-cli

1e94977

mikasenghaas changed the title ~~feat: consolidate evals + streamed traces~~ feat: stream served episodes turn by turn Sep 15, 2026

mikasenghaas changed the base branch from main to feat/drop-eval-cli September 15, 2026 17:25

mikasenghaas requested review from hallerite and samsja September 15, 2026 18:39

hallerite reviewed Sep 15, 2026

View reviewed changes

Comment thread verifiers/v1/serve/delta.py

mikasenghaas and others added 2 commits September 15, 2026 21:59

scaffold README points eval at prime-rl

067fe68

The scaffold installs only verifiers, which has no eval executable since the eval CLI moved to prime-rl. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Merge branch 'feat/drop-eval-cli' into feat/remove-eval-cli

6a32575

mikasenghaas and others added 3 commits September 15, 2026 22:00

keep the eval CLI

1e5fe56

The CLI's removal moves to its own PR for a transition period; this branch carries only the streamed traces. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mikasenghaas changed the base branch from feat/drop-eval-cli to main September 15, 2026 23:42

mikasenghaas and others added 2 commits September 15, 2026 23:50

ruff: sort imports, dict literal

f451f7b

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mikasenghaas marked this pull request as ready for review September 16, 2026 00:03

mikasenghaas requested a review from hallerite September 16, 2026 00:06

macroscopeapp Bot reviewed Sep 16, 2026

View reviewed changes

Comment thread verifiers/v1/interception/server.py Outdated

Comment thread verifiers/v1/cli/eval/main.py Outdated

mikasenghaas and others added 4 commits September 16, 2026 00:11

AGENTS.md: the eval CLI is vf-eval

f454a51

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

macroscopeapp Bot reviewed Sep 16, 2026

View reviewed changes

Comment thread verifiers/v1/trace.py Outdated

cursor Bot reviewed Sep 16, 2026

View reviewed changes

Comment thread verifiers/v1/trace.py Outdated

mikasenghaas and others added 2 commits September 16, 2026 00:45

preview keys by request identity

73d602d

A PendingTurn is a dataclass and not hashable. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

hint: vf-eval will be deprecated, move to prime-rl's eval

fc7e45b

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mikasenghaas commented Sep 16, 2026

View reviewed changes

Comment thread verifiers/v1/cli/eval/hint.py Outdated

samsja approved these changes Sep 16, 2026

View reviewed changes

hallerite reviewed Sep 16, 2026

View reviewed changes

Comment thread pyproject.toml

hallerite requested a review from samsja September 16, 2026 12:18

mikasenghaas requested a review from hallerite September 16, 2026 17:13

hallerite approved these changes Sep 16, 2026

View reviewed changes

mikasenghaas merged commit 5a69fb4 into main Sep 16, 2026
14 checks passed

mikasenghaas deleted the feat/remove-eval-cli branch September 16, 2026 17:53

Navigation Menu

Uh oh!

feat: stream served episodes turn by turn - #2512

feat: stream served episodes turn by turn#2512
mikasenghaas merged 29 commits into
mainfrom
feat/remove-eval-cli

mikasenghaas commented Sep 2, 2026 •

edited by macroscopeapp Bot

Loading

Uh oh!

hallerite left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

macroscopeapp Bot commented Sep 16, 2026 •

edited

Loading

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Uh oh!

hallerite left a comment

Uh oh!

Uh oh!

hallerite left a comment •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

mikasenghaas commented Sep 2, 2026 • edited by macroscopeapp Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Breaking

Verification

Stream served episodes turn by turn with delta frames

Changes since #2512 opened

Uh oh!

hallerite left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

macroscopeapp Bot commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approvability

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hallerite left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hallerite left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mikasenghaas commented Sep 2, 2026 •

edited by macroscopeapp Bot

Loading

macroscopeapp Bot commented Sep 16, 2026 •

edited

Loading

hallerite left a comment •

edited

Loading