Repository navigation
fix(evals): Run the eval suite on vitest-evals 0.18 - #1973
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Member
|
oops, i meant to have you get vitest-evals working on vitest 5, not downgrade us to vitest 4 |
dcramer
pushed a commit
to getsentry/vitest-evals
that referenced
this pull request
Nov 5, 2026
Makes `vitest-evals` work on Vitest 5. getsentry/toolkit uses Vitest 5, and its eval suite can't load the reporter from the `vitest-evals` 0.4.0 release it depends on, because that release imports `vitest/reporters`, which Vitest 5 removed. The reporter on `main` already imports from `vitest/node`, and Vitest 5 still exports `DefaultReporter` and `VerboseReporter` from there, so it runs. What broke were the types and the test fixtures. > [!WARNING] > **Breaking change for Vitest 4 consumers.** The `vitest-evals` peer dependency changes from `vitest >=4 <5` to `vitest >=5 <6`. One `declare module "vitest"` block can't merge with both the Vitest 4 `Assertion<T>` and the Vitest 5 `Assertion<R, T>`, so the typed matchers no longer support Vitest 4. Ship this as a minor bump (**0.18.0**), not a patch. ### Changes - **Matcher types:** Vitest 5 declares `Assertion<R extends void | Promise<void> = void, T = unknown>`. The augmentations in `index.ts` (`toSatisfyJudge`) and `legacy.ts` (`toEval`) used `Assertion<T = any>`, so `tsc` failed with TS2428. Both now use the Vitest 5 signature. - **Peer ranges:** `vitest-evals` requires `vitest >=5 <6`, as the warning above describes. `@vitest-evals/github-reporter` doesn't augment Vitest types, so it accepts `>=4 <6`. - **Workspace:** `vitest` and `@vitest/coverage-v8` move to `^5.1.0`, and the lockfile is regenerated with pnpm 10.33.0. - **Reporter tests:** Vitest 5's base reporter reads `state` from the context and calls `benchmarks()` on each test case, so the fake context and `TestCase` now provide both. The verbose fallback test now expects `file:12`, because Vitest 5 prints the line number without the column. ### Verification - `pnpm typecheck`, `pnpm lint`, `pnpm build`, and `pnpm release:check` pass. - `pnpm test:ci`: 408 tests pass and 1 fails. The failure is in `report-ui`, "returns 500 when asset reads fail". It fails the same way on `main` with Vitest 4 in my sandbox, where the file is still readable after `chmod 000`, so this change didn't cause it. - Consumer check: I packed this branch and installed it into getsentry/toolkit `main`, which runs Vitest 5.0.5. With its imports switched from `vitest-evals` to `vitest-evals/legacy`, `vitest run --reporter=vitest-evals/reporter` loaded the reporter. All 27 files and 121 evals ran and wrote `eval-results.json`, and `tsc` passed for the eval package. The evals themselves failed only because the sandbox has no provider API keys. ### Follow-up After 0.18.0 is released, toolkit can move from `vitest-evals` 0.4.0 to it and keep Vitest 5. That change will replace getsentry/toolkit#1973's current approach, which pins Vitest 4 for the eval package. <!-- junior-request-attribution:start --> via @dcramer. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:agent%3A1de0f02c391e347294cc7ce28353bcfe --> -- [View Session](https://junior-prod.sentry.dev/conversations/agent%3A1de0f02c391e347294cc7ce28353bcfe) [[Sentry]](https://sentry.sentry.io/explore/conversations/agent%3A1de0f02c391e347294cc7ce28353bcfe/?project=4510944073809921) <!-- junior-session-footer:end --> Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>
vitest-evals 0.4.0 imports `vitest/reporters`, which vitest 5 removed, so `eval:ci` failed at startup and no evals ran. vitest-evals 0.18.0 supports vitest 5 (peer `>=5 <6`). - Bump the catalog to ^0.18.0 and keep the eval package on vitest 5. Allow vitest-evals and its @vitest-evals/core and report-ui 0.18.0 deps past minimumReleaseAge, as they were published today. - Import the scorer-first API from `vitest-evals/legacy`, where 0.18 keeps it. - Remove the unused root vitest-evals dev dependency. - Set SENTRY_CLI_NO_TELEMETRY=1 for the scorer's `sentry mcp` process. Evals must not send telemetry, and it skips the lazy telemetry require that fails when the CLI runs from source. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
sentry-junior
Bot
force-pushed
the
fix/evals-vitest4
branch
from
November 5, 2026 04:20
84bda39 to
50cdbb1
Compare
sentry-junior
Bot
changed the title
dcramer
marked this pull request as ready for review
November 5, 2026 04:31
Member
|
can we not also bump vitest to 5? or is it too much effort |
dcramer
approved these changes
Nov 5, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Eval workflow on
mainreports green, but no evals have run.eval:cifails at startup:The cause is that vitest-evals 0.4.0 imports
vitest/reporters, which vitest 5 removed.continue-on-erroron "Run evals" hides the failure, so noeval-results.jsongets written and the status check step throws. Example run: https://github.com/getsentry/toolkit/actions/runs/41951775819This PR moves the eval package to vitest-evals 0.18.0, which supports vitest 5 (getsentry/vitest-evals#113, peer range
vitest >=5 <6). The whole repo stays on vitest 5; there is no vitest 4 pin.Changes
vitest-evalsto^0.18.0. I addedvitest-evals@0.18.0and its@vitest-evals/core@0.18.0and@vitest-evals/report-ui@0.18.0dependencies tominimumReleaseAgeExclude, because they were published today. Drop those entries once they are past the 24h window.vitest-evals/legacy. 0.18 keeps the scorer-first API (describeEvalwithdata/task/scorers,ToolCallScorer,Score,ScoreFn,BaseScorerOptions) undervitest-evals/legacy. This is an import-path change only, in the 33 eval imports and the example indocs/testing/overview.md. The eval code is unchanged.vitest-evalsdev dependency. Nothing at the root imports it.SENTRY_CLI_NO_TELEMETRY=1for the scorer'ssentry mcpprocess.ToolPredictionScorerstartspnpm --dir packages/cli run cli mcpfrom source. Onmain, that process exits immediately withPackage subpath './client' is not defined by "exports", and every scorer then fails withMCPClientError: Connection closed.lazyTelemetry()insrc/lib/db/index.ts. It loads../telemetry.jswithcreateRequire, so its imports resolve@sentry/api/clientunder therequirecondition, but@sentry/apiexports./clientonly forimport.pnpm --filter sentry run cli helpalso fails from source. It needs its own CLI fix.continue-on-errorand the eval status check are unchanged.Verification
pnpm eval:cion vitest 5.0.5 loads the reporter, collects and runs all 27 files and 121 evals, and writeseval-results.json. Onmainit stops at startup. Every eval failed, and the only cause was the missing provider key. NoConnection closederrors.pnpm run tsc,pnpm run lint, andpnpm install --frozen-lockfilepass. The fullpnpm run testdidn't finish inside my sandbox's time limit, so I'm relying on CI for it.Related
pnpm-lock.yamlandpnpm-workspace.yaml, so feat(evals): Record and replay eval model calls with Roach #1972 needs a rebase after this merges.via @dcramer.
--
View Session [Sentry]