feat(vitest-evals): Pass case expectations to suite judges and add createTaskHarness

README.md

@@ -230,8 +230,13 @@ Harness-backed suites stay close to plain Vitest:
   when the harness output type includes `undefined`
 - judges own their prompt, rubric, and parsing; LLM-backed judges use
   `ctx.runJudge(...)` from a configured `judgeHarness`
-- pass scenario-specific judge criteria explicitly to matcher options, or bind
-  suite-wide criteria on the judge instance
+- pass scenario-specific judge criteria as `run(input, { expected,
+  expectedTools })` so automatic suite judges see them, or explicitly to
+  matcher options; bind suite-wide criteria on the judge instance
+- suite-level `judgeThreshold` compares the average score of all automatic
+  judges, like the legacy `threshold`
+- wrap a plain `(input) => string | { result, toolCalls }` task function with
+  `createTaskHarness(...)` instead of building transcript events by hand
 - reporter output separates application, judge, and combined token and
   `costUsd` usage from normalized runs. First-party harnesses attach native
   spans when provider/runtime data is available; `createHarness(...)` attaches a fallback

docs/architecture.md

@@ -67,6 +67,11 @@ event stream is flat:
 - `type: "tool_call"` for a tool request
 - `type: "tool_result"` for a completed or failed tool execution
 
+`createTaskHarness(...)` is a thin wrapper over `createHarness(...)` for plain
+task functions that return `string | { result, toolCalls }`; it builds the
+user, `tool_call`, `tool_result`, and assistant events itself, so
+`createHarness(...)` keeps rejecting top-level `toolCalls`.
+
 This is the reporter and judge contract. `createHarness(...)` may accept strict
 camelCase message-shaped inputs at its lightweight result boundary: assistant
 `toolCalls`, `role: "tool"` results keyed by `toolCallId`, or AI SDK-style
@@ -244,14 +249,17 @@ For each eval test in a harness-backed suite:
 5. The harness returns a `HarnessRun` with `result.output`, `result.session`,
    `usage`, `timings`, `artifacts`, and `errors`.
 6. Core stores that run on `task.meta.harness` for the reporter.
-7. Automatic suite-level judges run against the normalized run/session pair.
+7. Automatic suite-level judges run against the normalized run/session pair,
+   plus any per-case judge params passed as `run(input, options)`. The suite
+   `judgeThreshold` is compared against the average of their scores.
 8. Each judge-harness invocation is stored under its judge score as `judgeRuns`.
 9. The eval test asserts on the same returned result and session.
 10. The reporter renders the recorded metadata and includes judge usage in totals
    without re-executing either harness.
 
 Explicit `expect(result).toSatisfyJudge(...)` calls use the run's typed output
-and reuse registered input, metadata, and harness context
+and reuse registered input, `run(input, options)` judge params, metadata, and
+harness context
 when `result` came from the fixture-backed `run(...)`. Inside an eval test,
 calls on registered output objects or session objects reuse that exact run
 context; other raw values fall back to the current test's most recent

packages/docs/src/content/docs/docs/harnesses/custom.mdx

@@ -68,6 +68,27 @@ export const qaHarness = createHarness<string, string>({
 });
 ```
 
+### Plain Task Functions
+
+When the entrypoint already returns its output and the tool calls it made, use
+`createTaskHarness()` and skip the transcript. It records the user input, one
+`tool_call` event per call (plus a `tool_result` when the call has a `result`),
+and the assistant output.
+
+```ts title="evals/qaHarness.ts"
+import { createTaskHarness } from "vitest-evals";
+import { answerQuestionWithTools } from "../src/questionFlow";
+
+export const qaHarness = createTaskHarness("qa-app", async (input: string) => {
+  const { text, toolCalls } = await answerQuestionWithTools(input);
+
+  return { result: text, toolCalls };
+});
+```
+
+A task can also return the output string alone. Legacy
+`(input) => Promise<string | { result, toolCalls }>` task functions fit as-is.
+
 Return a full `HarnessRun` only when you need complete control over the
 session, usage, timings, artifacts, traces, and errors. Full runs must already
 use canonical `session.events`; `createHarness()` is the preferred custom

packages/docs/src/content/docs/docs/judges.mdx

@@ -74,5 +74,46 @@ describeEval(
 );
 ```
 
+## Per-Case Expectations
+
+Suite-level judges also need case-specific expectations. Pass them as the
+second `run(...)` argument. The harness never sees them; each automatic judge
+receives them in its context, and `toSatisfyJudge(...)` reuses them for the
+returned run unless matcher options override them.
+
+```ts title="evals/capital.eval.ts"
+import {
+  describeEval,
+  StructuredOutputJudge,
+  ToolCallJudge,
+} from "vitest-evals";
+import { qaHarness } from "./qaHarness";
+
+describeEval(
+  "capital questions",
+  {
+    harness: qaHarness,
+    judges: [ToolCallJudge(), StructuredOutputJudge()],
+    judgeThreshold: 0.75,
+  },
+  (it) => {
+    it.for([
+      {
+        input: "What is the capital of France?",
+        expected: { answer: "Paris" },
+        expectedTools: ["lookupCapital"],
+      },
+    ])("$input", async ({ input, ...expectations }, { run }) => {
+      await run(input, expectations);
+    });
+  },
+);
+```
+
+`judgeThreshold` compares the average score of all automatic judges, so one
+judge can fall short while the case still passes. A `null` score counts as 0.
+Use explicit `toSatisfyJudge(...)` assertions when each judge must pass on its
+own threshold.
+
 Keep deterministic judges close to the harness contract. Use custom judges for
 rubrics that need domain-specific scoring or reusable explanations.

packages/docs/src/content/docs/docs/judges/structured-output.mdx

@@ -5,7 +5,7 @@ editUrl: false
 ---
 
 `StructuredOutputJudge()` compares the harness output with expected values from
-the judge config or matcher options. Use it for deterministic checks when the harness
+the judge config, `run(input, { expected })`, or matcher options. Use it for deterministic checks when the harness
 returns a parsed domain object instead of raw provider text.
 
 ## Usage

packages/docs/src/content/docs/docs/judges/tool-call.mdx

@@ -25,8 +25,16 @@ describeEval("capital questions", {
 
 ## Arguments
 
-Use matcher options when a case-specific tool argument is the important
-assertion.
+Pass case-specific expected tools as a `run(...)` option so the suite judge
+checks them:
+
+```ts title="evals/capital.eval.ts"
+await run("What is the capital of France?", {
+  expectedTools: [{ name: "lookupCapital", arguments: { country: "France" } }],
+});
+```
+
+Or use matcher options when the tool argument is a separate assertion.
 
 ```ts title="evals/capital.eval.ts"
 const result = await run("What is the capital of France?");

packages/vitest-evals/README.md

@@ -43,10 +43,13 @@ workflow.
   not return traces themselves. Span attributes include typed
   OpenTelemetry GenAI semantic keys while still allowing provider-specific
   metadata
-- scenario-specific judge criteria should live in `input` or explicit matcher
-  options, depending on whether the app or only the judge needs them
+- scenario-specific judge criteria should live in `input` when the app needs
+  them; per-case judge expectations go in `run(input, { expected,
+  expectedTools })` or explicit matcher options when only the judge needs them
 - suite-level `judges` are optional and run automatically after each `run(...)`
-- suite-level `judgeThreshold` controls fail-on-score for those automatic judges
+- suite-level `judgeThreshold` controls fail-on-score for those automatic
+  judges; it compares the average score of all automatic judges (a `null`
+  score counts as 0) and defaults to 1
 - every judge is a named object with `assess(ctx)`
 - every judge receives `JudgeContext` with typed `input`, typed `output`, the
   normalized run/session, and tool calls; `output` is only optional
@@ -130,6 +133,33 @@ describeEval("refund agent", { harness }, (it) => {
 });
 ```
 
+When suite-level judges need per-case expectations, pass them as the second
+`run(...)` argument. The harness never sees them; every automatic judge
+receives them in its context, and `toSatisfyJudge(...)` reuses them for the
+returned run unless matcher options override them:
+
+```ts
+describeEval(
+  "refund agent",
+  {
+    harness,
+    judges: [ToolCallJudge(), StructuredOutputJudge()],
+    judgeThreshold: 0.75,
+  },
+  (it) => {
+    it.for([
+      {
+        input: "Refund invoice inv_123",
+        expected: { status: "approved" },
+        expectedTools: ["lookupInvoice", "createRefund"],
+      },
+    ])("$input", async ({ input, ...expectations }, { run }) => {
+      await run(input, expectations);
+    });
+  },
+);
+```
+
 ## Terminal Reporting
 
 The terminal reporter has two eval report levels. Normal mode prints compact
@@ -205,6 +235,40 @@ factory and a `Runner` or `runner` factory. The harness calls
 normalization and replay hooks when the app needs a custom entrypoint or
 structured domain output mapping.
 
+## Plain Task Functions
+
+When the system under test is a function that returns its output and,
+optionally, the tool calls it made, wrap it with `createTaskHarness(...)`
+instead of building transcript events by hand. Legacy
+`(input) => Promise<string | { result, toolCalls }>` task functions fit as-is:
+
+```ts
+import { createTaskHarness, describeEval, ToolCallJudge } from "vitest-evals";
+
+describeEval(
+  "refund agent",
+  {
+    harness: createTaskHarness("refund-agent", async (input) => {
+      const { text, toolCalls } = await runRefundAgent(input);
+      return { result: text, toolCalls };
+    }),
+    judges: [ToolCallJudge()],
+  },
+  (it) => {
+    it("looks up the invoice", async ({ run }) => {
+      await run("Refund invoice inv_123", {
+        expectedTools: ["lookupInvoice"],
+      });
+    });
+  },
+);
+```
+
+The harness records the user input, one `tool_call` event per reported call
+(plus a `tool_result` event when the call has a `result`), and the assistant
+output. Use `createHarness(...)` when the app also needs to report usage,
+traces, or a richer transcript.
+
 ## Custom App Harnesses
 
 First-party harness packages are conveniences, not the only supported path. If
@@ -331,8 +395,8 @@ Use `Harness.run(...)` for the application under test. Calling
 `ctx.harness.run(...)` from inside a judge runs the application a second time,
 so reserve that for judges that intentionally need a second execution. Put
 criteria on `input` when they are part of the scenario itself; pass
-case-specific judge criteria through matcher options, or configure suite-wide
-criteria on the judge instance.
+case-specific judge criteria through `run(input, options)` or matcher options,
+or configure suite-wide criteria on the judge instance.
 
 `createHarness(...)` lightweight results must return at least one normalized
 event, either directly as `events` or from strict camelCase `messages`. Stored