Skip to content

feat(evals): Record and replay eval model calls with Roach - #1972

Open
sentry-junior[bot] wants to merge 2 commits into
mainfrom
feat/roach-evals
Open

feat(evals): Record and replay eval model calls with Roach#1972
sentry-junior[bot] wants to merge 2 commits into
mainfrom
feat/roach-evals

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

The MCP evals (packages/mcp-server-evals) call OpenRouter on every run. This sends those calls through Roach, Sentry's shared record/replay proxy. CI replays recorded calls and records only the new ones.

The change is config only, in the same shape as getsentry/warden#564:

  • roach.json sets tenant getsentry/toolkit, allows https://openrouter.ai, and has one model rule for POST https://openrouter.ai/api/v1/. It has no values patterns. Through a local Roach, I sent the requests the evals build twice: two search_events embedded-agent calls and the first call of McpToolCallTaskRunner. Both runs gave byte-identical bodies and the same keys, and the second run replayed all of them. The tool outputs that feed later steps (get_issue_details, get_trace_details from the MSW fixtures) were also identical when I called them a minute apart. I could not capture the tool-prediction scorer's request, because its tool discovery crashes locally (see below). Its prompt comes from the static tool list. The API key travels in Authorization, which Roach redacts on its own, so there is no secretEnv.
  • vitest.config.ts adds the Roach global setup and setup file. The Roach setup file goes before src/setup-env.ts, so MSW, which throws on unhandled requests, does not catch Roach's own session calls.
  • eval.yml passes ROACH_TOKEN to the eval step and moves the job to Node 24, because Roach needs 24.14+. The workflow runs on pull_request, not pull_request_target, so fork PRs get no secrets and stay replay-only. The fork policy does not change.
  • @sentry/roach 0.2.0 is in the catalog, and in minimumReleaseAgeExclude because it was published today. The agents peer line in the lockfile is pnpm 11.8.0's own output after adding the package.
  • docs/testing/overview.md has a short "Eval Recordings" section.

Blockers before merge

  • Needs fix(client): Keep control calls out of in-process HTTP mocks roach#30 released. On 0.2.0, MSW catches the Roach session call, and every eval fails with Roach POST session failed with HTTP 500: Unhandled request. I reproduced this against the live service. With that fix applied locally, a replay run with no token against the live service fails cleanly with 412 misses, as expected. After the release, bump the catalog entry and drop the minimumReleaseAgeExclude line. fix(vitest): Allow off mode on Node before 24.14 roach#31, which lets ROACH_MODE=off run on Node 22, is a nice-to-have for local runs on toolkit's Node 22 minimum.
  • ROACH_TOKEN access. getsentry/toolkit must be added to the selected repositories of the ROACH_TOKEN org secret, and an org admin has to do that. Without it, CI runs replay-only and records nothing.
  • The eval suite is already broken on main. vitest-evals 0.4.0 does not work with vitest 5. The reporter fails to load (Package subpath './reporters' is not defined), and describeEval throws at collection. The latest vitest-evals still peers vitest <5. continue-on-error hides this, and the Eval run on main reports success with no results file. Roach records nothing until the evals run again; test(evals): Migrate to vitest-evals harness #1132 may be the path. Also, ToolPredictionScorer's tool discovery (sentry mcp under tsx) crashes locally with Package subpath './client' is not defined by "exports" from @sentry/api. That happens with and without Roach.

via @dcramer.

--

View Session [Sentry]

@vercel

vercel Bot commented Nov 5, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
sentry-local Ready Ready Preview Nov 5, 2026 4:49am UTC

Request Review

@dcramer
dcramer marked this pull request as ready for review November 5, 2026 02:51
@github-actions github-actions Bot added the risk: medium PR risk score: medium label Nov 5, 2026
@sentry-junior sentry-junior Bot mentioned this pull request Nov 5, 2026
Merged
dcramer added a commit that referenced this pull request Nov 5, 2026
@sentry-junior @dcramer
fix(evals): Run the eval suite on vitest-evals 0.18 (#1973)
02ca9bf 
The Eval workflow on `main` reports green, but no evals have run.
`eval:ci` fails at startup:

```
Failed to load custom Reporter from vitest-evals/reporter
cause: Package subpath './reporters' is not defined by "exports" (ERR_PACKAGE_PATH_NOT_EXPORTED)
```

The cause is that vitest-evals 0.4.0 imports `vitest/reporters`, which
vitest 5 removed. `continue-on-error` on "Run evals" hides the failure,
so no `eval-results.json` gets written and the status check step throws.
Example run:
https://github.com/getsentry/toolkit/actions/runs/41951775819

This PR moves the eval package to vitest-evals 0.18.0, which supports
vitest 5 (getsentry/vitest-evals#113, peer range `vitest >=5 <6`). The
whole repo stays on vitest 5; there is no vitest 4 pin.

## Changes

- **Bump `vitest-evals` to `^0.18.0`.** I added `vitest-evals@0.18.0`
and its `@vitest-evals/core@0.18.0` and `@vitest-evals/report-ui@0.18.0`
dependencies to `minimumReleaseAgeExclude`, because they were published
today. Drop those entries once they are past the 24h window.
- **Import from `vitest-evals/legacy`.** 0.18 keeps the scorer-first API
(`describeEval` with `data`/`task`/`scorers`, `ToolCallScorer`, `Score`,
`ScoreFn`, `BaseScorerOptions`) under `vitest-evals/legacy`. This is an
import-path change only, in the 33 eval imports and the example in
`docs/testing/overview.md`. The eval code is unchanged.
- **Remove the unused root `vitest-evals` dev dependency.** Nothing at
the root imports it.
- **Set `SENTRY_CLI_NO_TELEMETRY=1` for the scorer's `sentry mcp`
process.**
- `ToolPredictionScorer` starts `pnpm --dir packages/cli run cli mcp`
from source. On `main`, that process exits immediately with ``Package
subpath './client' is not defined by "exports"``, and every scorer then
fails with `MCPClientError: Connection closed`.
- The exit comes from `lazyTelemetry()` in `src/lib/db/index.ts`. It
loads `../telemetry.js` with `createRequire`, so its imports resolve
`@sentry/api/client` under the `require` condition, but `@sentry/api`
exports `./client` only for `import`.
- Evals shouldn't send CLI telemetry anyway. The CLI bug itself is not
fixed here: `pnpm --filter sentry run cli help` also fails from source.
It needs its own CLI fix.

`continue-on-error` and the eval status check are unchanged.

## Verification

- **Local, no LLM key:** with no key in the sandbox, `pnpm eval:ci` on
vitest 5.0.5 loads the reporter, collects and runs all 27 files and 121
evals, and writes `eval-results.json`. On `main` it stops at startup.
Every eval failed, and the only cause was the missing provider key. No
`Connection closed` errors.
- **Checks:** `pnpm run tsc`, `pnpm run lint`, and `pnpm install
--frozen-lockfile` pass. The full `pnpm run test` didn't finish inside
my sandbox's time limit, so I'm relying on CI for it.

## Related

- #1132 (draft, "Migrate to vitest-evals harness") moves these evals to
the harness API. It touches the same eval files, so expect import
conflicts there. This PR keeps the current scorer-first suite running
until then.
- #1972 (Roach recordings for evals) depends on this, because Roach
records nothing while the suite doesn't run. Both PRs change
`pnpm-lock.yaml` and `pnpm-workspace.yaml`, so #1972 needs a rebase
after this merges.

<!-- junior-request-attribution:start -->
via @dcramer.
<!-- junior-request-attribution:end -->

<!-- junior-session-footer:start -->
<!-- junior-conversation-id:agent%3A1de0f02c391e347294cc7ce28353bcfe -->

--

[View
Session](https://junior-prod.sentry.dev/conversations/agent%3A1de0f02c391e347294cc7ce28353bcfe)
[[Sentry]](https://sentry.sentry.io/explore/conversations/agent%3A1de0f02c391e347294cc7ce28353bcfe/?project=4510944073809921)

<!-- junior-session-footer:end -->

Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
Co-authored-by: David Cramer <23610+dcramer@users.noreply.github.com>
sentry-junior Bot and others added 2 commits November 5, 2026 04:44
@sentry-junior @dcramer
feat(evals): Record and replay eval model calls with Roach
f990119 
The MCP evals call OpenRouter on every run. Roach, Sentry's shared
recording proxy, now sits in front of those calls: it replays a call it
has a recording for, and records the rest when the run has ROACH_TOKEN.

The setup is config only: roach.json names the OpenRouter host and one
rule for its API, vitest.config.ts adds the Roach global setup and setup
file, and the Eval workflow passes ROACH_TOKEN and moves to Node 24,
which Roach needs. Fork PRs get no secrets, so they only replay.

The Roach setup file loads before setup-env.ts so that MSW does not
catch Roach's own control calls. That needs getsentry/roach#30.

Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
@sentry-junior @dcramer
test(evals): Use unreleased roach main (revert before merge)
bbc5c92 
Point @sentry/roach at getsentry/roach@efa6885 (packages/roach) through
a pnpm git dependency, so CI can test record and replay before the next
Roach release. pnpm runs the package's prepack for git dependencies, so
allowBuilds lists the exact tarball.

Drop this commit and set the catalog to the released version before
merging.

Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
@sentry-junior sentry-junior Bot mentioned this pull request Nov 5, 2026
Draft

This branch was successfully deployed

1 active and 1 inactive deployments
Actions — bbc5c929 Deployed Nov 5, 2026 by sentry-junior[bot] via eval #1563
Preview — bbc5c929 Deployed Nov 5, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dcramer dcramer dcramer approved these changes

Assignees

@dcramer dcramer

Labels

risk: medium PR risk score: medium

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@dcramer