[sentry-junior]

#1973 got the eval suite running on vitest-evals 0.18, but it did that by moving every import to the temporary vitest-evals/legacy entrypoint. This PR moves every suite onto the root harness API: describeEval(name, { harness, judges, judgeThreshold }, (it) => ...). No vitest-evals/legacy imports are left.

This is a migration only. Cases, expected tool calls, structured-output checks, thresholds, timeouts and test names are all unchanged. Product code and eval prompts are untouched.

How the conversion works

• Each eval case is the harness input. The task still gets only the prompt string. Suite-level judges read expectedTools and expected from the case, which the vitest-evals README allows ("put criteria on input"). I didn't use per-case toSatisfyJudge(...) matchers, because they apply a threshold to each judge separately. The legacy API averaged all scorers and compared that average to the threshold, and keeping one run() per case with automatic judges gives the same pass/fail.
• createTaskHarness wraps the existing task runners (NoOpTaskRunner, McpToolCallTaskRunner, the embedded-agent tasks). It returns a normalized transcript: the prompt, the tool calls, then the output. defineEvalCases registers one test per case.
• ToolPredictionJudge is ToolPredictionScorer converted to a judge. The prompt, model, schema and scoring are the same.
• StructuredOutputJudge stays a local judge with the same matching logic. The built-in judge's fuzzy matching (substring, case-insensitive) would change scores.
• ToolCallJudge wraps the built-in vitest-evals judge so it can read expectedTools from the case. The legacy ToolCallScorer was already a thin wrapper around this same scorer.

task.meta.eval keeps the shape that eval.yml's status check reads (avgScore, scores[].metadata.rationale). The vitest-evals/reporter CI path doesn't change. Nothing changes in vitest.config.ts, eval.yml, package.json or the lockfile.

Relation to #1132

This replaces the harness part of @dcramer's #1132. It reuses that PR's harness-adapter design (createTaskHarness, which turns tool calls into transcript events) and its judge conversions, updated for 0.18. It leaves out a few parts of #1132, because each one changes what the evals measure:

• the mcp-core search agent prompt and config changes (search-events, search-issues, search-issue-events)
• routing the eval model through the embedded-agent provider factory, and the new McpToolCallTaskRunner system prompt
• the all-or-nothing structured-output scoring and the per-judge thresholds

Any of these can be a follow-up PR.

Eval cases and test names are the same, but the harness changes what each run records, so any Roach recordings (#1972) will need to re-record once.

via @dcramer.

--

View Session [Sentry]

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
sentry-local	Ready	Preview	Nov 5, 2026 4:57am UTC

[dcramer]

whats the cause of the toatl lines increase?