[sentry-junior]

getsentry/toolkit#1974 moved 27 eval files off vitest-evals/legacy, and the migration needed ~117 lines of adapters: utils/harness.ts +95 and utils/toolCallJudge.ts +22. This PR closes the two gaps behind those adapters. Both changes are additive: existing 0.18 call sites and legacy/ are untouched.

1. Per-case expectations for automatic suite judges

applyAutomaticJudges only passed input/output/toolCalls/run/session to suite judges. That meant the built-in ToolCallJudge and StructuredOutputJudge never saw a case's expectedTools/expected unless you called toSatisfyJudge(...), and that path checks each judge against its own threshold. Toolkit worked around this by packing the expectations into the harness input and re-wrapping every judge to unpack them.

The fix is an optional second argument to run:

await run(input, { expected, expectedTools, ...customJudgeParams });

• The harness never sees these values.
• They are spread into every automatic judge's context. Core fields (input, output, run, toolCalls, …) always win.
• toSatisfyJudge(...) reuses them for the returned run. Explicit matcher options still override them.
• They are judge params, not run data, so they aren't stored on the run. That keeps non-JSON values such as the RegExp expectations toolkit uses out of the serialized run.

Threshold semantics (checked, unchanged): the suite judgeThreshold is already compared against the average score of all automatic judges, and a null score counts as 0. That is the same rule as legacy threshold, so toolkit needs no opt-in. This PR documents it on DescribeEvalOptions.judgeThreshold and in the docs, and adds a test where one judge scores 0, another scores 1, and the case passes at 0.5.

2. createTaskHarness(name, task)

createHarness deliberately rejects a top-level toolCalls result (there is a test for it), so a plain task returning { result, toolCalls } had to build transcript events by hand. I chose a small built-in wrapper over loosening createHarness:

• It keeps createHarness's strict "ordered events or messages" contract and its existing rejection.
• It maps the legacy (input) => string | { result, toolCalls } task shape 1:1. The wrapper builds user → tool_call (+ tool_result when a call has a result) → assistant events itself. Errors, fallback traces and artifacts come from createHarness.
• It is positional, like createJudge(name, assess). An object form createTaskHarness({ name, task }) wraps to 4 lines under oxfmt in every toolkit eval file, which would make the files longer, not shorter. The positional form also matches toolkit's existing local helper, so call sites don't change.

Proof in toolkit

I used a scratch copy of toolkit at 198335d (not pushed) and swapped in the packed branch build. I rewrote the utils and all 27 eval files to the new API. tsc --noEmit and oxfmt --check pass on mcp-server-evals.

I also ran a mock-model smoke eval: MockLanguageModelV3 driving McpToolCallTaskRunner through the real MCP stdio server and MSW mocks. Results:

• ToolCallJudge scored 1 when expectedTools came from run(...).
• An it.fails case with the wrong expectedTools failed as expected.
• An averaged-threshold case passed at 0.5.

	before (#1974)	after
utils/harness.ts	95	27 (EvalCase + a 10-line defineEvalCases)
utils/toolCallJudge.ts	22	deleted (built-in ToolCallJudge)
utils/index.ts	12	10
utils/structuredOutputJudge.ts / toolPredictionJudge.ts	271 / 247	270 / 248 (read ctx.expected / ctx.expectedTools instead of ctx.input.*)
utils total	747	655 (−92)
get-issue.eval.ts	46	44
list-teams.eval.ts	70	69
search-events-agent.eval.ts	289	283
27 eval files total	2853	2790 (−63)

Most of the savings are in the adapters. Per-file changes are only import moves. The rest of the ~10-line-per-file growth in #1974 comes from code layout (a separate const cases, the describeEval options block), not from the API.

get-issue.eval.ts:

-import { describeEval } from "vitest-evals";
+import { describeEval, createTaskHarness, ToolCallJudge } from "vitest-evals";
 import {
-  createTaskHarness,
   defineEvalCases,
   FIXTURES,
   McpToolCallTaskRunner,
-  ToolCallJudge,
   type EvalCase,
 } from "./utils";

The adapter it no longer needs (utils/harness.ts):

-export function createTaskHarness(name: string, task: EvalTaskRunner): EvalHarness {
-  return createHarness<EvalCase, string>({
-    name,
-    run: async ({ input: { input } }) => {
-      … try/catch + attachHarnessRunToError(createFailedHarnessRun(input, error)) …
-      const events: TranscriptEvent[] = [ user, ...toolCalls.map(→ tool_call), assistant ];
-      return { output: result.result, events };
-    },
-  });
-}
 export function defineEvalCases(it, cases, { timeout = 60000 } = {}) {
-  for (const testCase of cases) {
-    it(testCase.name ?? testCase.input, { timeout }, async ({ run }) => {
-      await run(testCase);
+  for (const { input, name, ...expectations } of cases) {
+    it(name ?? input, { timeout }, async ({ run }) => {
+      await run(input, expectations);

Possible follow-up (not in this PR)

A cases option on describeEval would remove toolkit's last helper (defineEvalCases). it.for(cases)(…) already covers most of it, so I left it out.

Open questions

1. Shape of the run options. This PR uses a flat run(input, { expected, expectedTools, ...custom }). That's terse, but any future non-judge run option, such as a per-run timeout or metadata, would share a namespace with custom judge params. The alternative is a nested run(input, { judge: { … } }).
2. Typing. EvalRunOptions types expected and expectedTools but has an index signature for custom params, so a typo in a custom param name won't be caught. Inferring the allowed keys from the suite's judges tuple is possible but adds generics to describeEval.
3. createTaskHarness scope. It has no usage/traces passthrough, and non-object tool arguments are dropped, the same as the toolkit adapter did. Anything richer should use createHarness. Is that the boundary you want?

pnpm test has one failure in this sandbox, report-ui "returns 500 when asset reads fail", and it fails the same way on main here. The sandbox user can still read a chmod 000 file, so this is not caused by this change.

via @dcramer.

--

View Session [Sentry]

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
vitest-evals-docs	Ready	Preview	Nov 5, 2026 5:48am UTC