Skip to content

feat(packs): vitest 4 → 5 candidate pack - #57

Closed
lucasbraum wants to merge 1 commit into
mainfrom
claude/pensive-galileo-asurhw
Closed

lucasbraum wants to merge 1 commit into
mainfrom
claude/pensive-galileo-asurhw

Conversation

@lucasbraum

Copy link
Copy Markdown
Collaborator

Pack

vitest, >=4 <5 → >=5 <6. Status: candidate (1 false positive among breaking findings; see below). Queue entry set to candidate. No pack-request issue existed for it.

Sources

No official codemod exists for this major.

Rules and behavior notes

Rules:

  • matcher-types (breaking, detect + compiler claims, agent): matchers on the global jest.Matchers, or on a one-parameter Assertion<T> / Matchers<R>. Site = the declare global block or the interface.
  • jest-dom-matchers (breaking, detect, agent): import '@testing-library/jest-dom[/vitest]', whose types no longer reach expect.
  • bench-api (breaking, type diff, agent): bench and the Bench* types removed.
  • removed-entrypoints (breaking, detect, rewrites): vitest/coverage and vitest/reporters → vitest/node, vitest/environments and vitest/snapshot → vitest/runtime.
  • removed-entrypoints-manual (breaking, detect, agent): vitest/runners, vitest/suite, vitest/mocker, vitest/internal/module-runner.
  • sequential-removed (breaking, type diff, agent): test.sequential, describe.sequential, sequential option.

Behavior notes (all "Decisions for you", some also test follow-ups): coverage-patterns (also a finding, found from source: bare directory in coverage.include/exclude), clear-mocks-default, test-name-pattern, inline-projects-extend, hoisted-calls-top-level, mock-behavior, assertion-timing, pretty-format-output, fake-timers-temporal, artifact-locations, config-resolution, browser-mode, environment-requirements, node-api.

Ground-truth repositories

  • votingworks/vxsuite @ 5cb3c7d9c1abc891e7b12de57c0278790dd6160a, upgrade commit bb04c5fb01aa9bcd285170d2277df04ad5d7e09c (vitest 4.1.11 → 5.0.2, with @vitest/coverage-istanbul 5.0.2): 30 expected findings (3 matcher-types, 11 jest-dom-matchers, 3 bench-api, 13 coverage-patterns).
  • vitorvasc/opentelemetry-ecosystem-explorer @ 46d5b37d00bbf27f394e33cee64a0e206c31315f (directory ecosystem-explorer), upgrade PR head 2b0497b35d0e49dbeeb0d6174ad09aed03d46697 (4.1.11 → 5.0.0): 2 expected findings (jest-dom-matchers).

Expected findings come from pnpm packs:truth (their compiler plus the upgrade commit's changed lines), read by hand; the 169 compiler diagnostics under vxsuite's libs/image-utils/src/vitest_setup.ts are evidence of that one declaration and are recorded once at it. Exclusions are listed in the $comment of ground-truth.json.

Considered and dropped: siemens/element (a real migration, but its @simpl/* and @siemens-ux/* packages come from code.siemens.com, so it cannot install from a public registry); google/budoux, ckeditor/ckeditor5-vue, gopherium/gophenberg, justinlocsei/larkspur and abraham/nutmeg (version bumps with no source change, so no site).

uptide pack test output

New pack (pnpm uptide pack test vitest --write; the ~18,900 generic (unverified) lines are omitted):

uptide pack test vitest · candidate
fixtures      5 cases, 17 sites · pass
ground truth
  votingworks/vxsuite@5cb3c7d9c1ab             vitest 4.1.11 → 5.0.2  predicted 15334, expected 30, false positives 15320, false negatives 16
    with @vitest/coverage-istanbul 4.1.11 → 5.0.2
  vitorvasc/opentelemetry-ecosystem-explorer@46d5b37d00bb vitest 4.1.11 → 5.0.0  predicted 3621, expected 2, false positives 3619, false negatives 0

  rule                         precision  recall    tp   fp   fn
  bench-api                          15%     67%     2   11    1
  coverage-patterns                 100%      0%     0    0   13
  generic                             0%    100%     0 18927    0
  jest-dom-matchers                 100%     85%    11    0    2
  matcher-types                      75%    100%     3    1    0
  overall (sites)                     0%     50%    16 18939   16
  breaking (sites)                   94%     50%    16    1   16

false positives among breaking:
  votingworks/vxsuite  libs/image-utils/src/jest_pdf_snapshot.ts:42  matcher-types (breaking)

false negatives (16)
  13 x coverage-patterns (vitest.config.ts files that import a local `defineConfig` wrapper, not vitest)
  jest-dom-matchers: apps/print/frontend/src/setupTests.ts:2, libs/dev-dock/frontend/src/setupTests.ts:1 (files that do not import vitest)
  bench-api: libs/hmpb/src/render_ballot.bench.ts:16

Existing packs, rerun in this sandbox, all unchanged:

zod     verified  overall 100% / 84%, breaking 100% / 71%, 0 false positives  (PASS)
stripe  verified  PASS
ai      verified  PASS
react   verified  excalidraw + tldraw, breaking 100% precision (89 sites, 0 false), recall 69%  (PASS, same numbers as the recorded verification)

react note: the first run could not install tldraw (its packageManager is yarn@4.7.0 and the yarn on PATH here is 1.22.22; the installer runs it directly). That is an environment issue, not a regression. With a corepack shim on PATH (corepack enable --install-directory <dir>), the rerun matched the recorded numbers.

pnpm lint, pnpm typecheck and pnpm test pass.

What is not covered

  • Why it is a candidate: one false positive among breaking findings. jest_pdf_snapshot.ts:42 is a real new TS2551 (toMatchImageSnapshot missing on Assertion), but it is a symptom of the declaration at vitest_setup.ts:10, which the pack reports. check did not trace it to that cause (a global jest.Matchers augmentation), so it stands as a second breaking site. The ground truth records one site at the cause, as docs/packs.md says, and I did not add the symptom to pass.
  • Pack detect runs only on files that import the package, so these expected sites are missed: 13 bare-directory coverage entries (configs that import a shared defineConfig wrapper), and jest-dom imports in two setup files that import nothing from vitest. Recall on breaking sites is 50%.
  • check could not resolve imports in 69 of 71 vxsuite files (workspace packages are not built), so only a few of the 169 compiler diagnostics are visible there.
  • bench-api claims 11 inferred unverified sites on Bench* types in vxsuite that the upgrade did not touch (overall tally only, not breaking), and the describe wrapper at render_ballot.bench.ts:16 is missed.
  • Rules and notes with no evidence in either repo, resting on fixtures and the guide only: removed-entrypoints*, sequential-removed, nested vi.mock, and all behavior notes except coverage-patterns.
  • A second repo that really migrated and installs here is needed for verified to mean more than one monorepo: the other candidates found are version-only bumps.

Tooling friction

  • pnpm uptide check from source fails inside its worker (Cannot find module .../packages/core/src/errors.js); pnpm build and the dist CLI work. pack test is unaffected.
  • pnpm packs:truth requires full 40-character SHAs, and for a file the upgrade moved it prints the new path with the old line number; the paths in the ground truth were read from git diff -M.
  • The README packs table includes candidates and the README test caps it at 150 lines; adding the row went one line over, so this PR joins two lines of the documentation list (no content change).
  • vitest.dev and its guide are not reachable from the sandbox; the guide was read from the vitest repo over git.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YNCujWu9MXTm48xoYVKHJC


Generated by Claude Code

A candidate pack for Vitest 4 → 5, scored against votingworks/vxsuite and
vitorvasc/opentelemetry-ecosystem-explorer at the commit before their own
upgrade. Rules: matcher-types, jest-dom-matchers, bench-api,
removed-entrypoints (rewritten for the four with a drop-in path),
removed-entrypoints-manual, sequential-removed; behavior notes for what the
compiler cannot see, and coverage-patterns found from source. One false
positive among breaking findings keeps it a candidate.

The README packs table is regenerated, and one line of the documentation list
is joined so the README stays within its 150-line test.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YNCujWu9MXTm48xoYVKHJC
Signed-off-by: Claude <noreply@anthropic.com>
@lucasbraum lucasbraum added the pack-candidate Migration pack proposed by the daily pack task label Oct 10, 2026 — with Claude
lucasbraum added a commit that referenced this pull request Oct 10, 2026
…n a repository augmentation (#60)

## What and why

**1. Shared root through a setup or declaration file.** In the vitest
pack test (#57, ground truth votingworks/vxsuite), `check` reported a
breaking finding at `libs/image-utils/src/jest_pdf_snapshot.ts:42`
(TS2551, `toMatchImageSnapshot` missing on `Assertion`) beside the
declaration at `libs/image-utils/src/vitest_setup.ts:10` the pack
already reports. The tracer did not link them: #51 handles parameters
only, and this is a global augmentation. Two things turned out to be
true on inspection:

- The failing member is not a parameter and not declared in the setup
file either: `toMatchImageSnapshot` comes from `jest-image-snapshot`'s
own types, which declare it on the global `jest.Matchers`. The setup
file augments that same interface (`declare global { namespace jest {
interface Matchers<R> { toMatchImage…; toMatchPdfSnapshot… } } }`) and
registers all three matchers with `expect.extend`. Vitest 5 stops
reading the global namespace, so every matcher on it has to move to
`Matchers<R, T>` of "vitest" together, and the repository's augmentation
is where that edit goes.

The cause tracer (`adapters/typescript/cause.ts`, `augmentationCause`)
now handles missing-member diagnostics (TS2339, TS2551): the member is
resolved in the baseline program, where it still exists, and

- when its declaration is in a repository file, the diagnostic is
anchored there: a `declare global` block at the block, a `declare module
"x"` augmentation at the interface inside it, any other repository
declaration at itself;
- when its declaration is a package's (or another fixture's) on an
interface the repository augments, it is anchored at the repository's
augmentation of that interface, this workspace's first, with the package
named in the reason;
- one site is enough, unlike a parameter: the edit is never at the call.
The anchor is also the diagnostic's `root`, so sites in other workspaces
fold with it after the merge (#51's model, same report fields:
`anchorOnly`, `downstream`, `root`, "N call sites").

It is tried after the compiler-option cause and before the parameter
cause. On vxsuite the finding is now one site at `vitest_setup.ts:10`,
with the `jest_pdf_snapshot.ts` call as evidence.

**2. README line limit.** `readme.test.ts` capped the file at 150 lines
and every pack row pushed against it. The test now excludes the
generated block between `<!-- packs:start -->` and `<!-- packs:end -->`
from the count. #57's two joined documentation lines are not on `main`
(that change lives on #57's branch), so there is nothing to restore
here; once this merges, #57 can drop that edit.

No pack's rules, expectations or ground truth were edited.

## How it was verified

Unit tests in `cause.test.ts` on a new fixture pair,
`fixtures/deps/matchers-v1` (an assertion that reads the global
`checks.Matchers`) and `matchers-v2` (its own `Matchers<R, T>`, the
global no longer read), plus `matchers-extra-v1` (a package declaring a
matcher on that global):

- three call sites in two files, for a matcher the setup file declares,
fold into one anchor at the setup file's `declare global` line, each
site carrying the cause and the root;
- a call site for a matcher another package declares on the interface
the setup file augments anchors at that augmentation, naming the
package;
- a member the package itself dropped is left at the site.

`pnpm lint`, `pnpm typecheck`, `pnpm docs:links`, `pnpm docs:packs
--check`, the private-material guard, core (930 tests) and CLI (350
tests) pass.

Pack tests, before (main at 45677eb) and after (this branch), breaking
findings; vitest run in a scratch worktree with #57's pack copied in
uncommitted:

| pack | before: precision / recall, breaking FP | after: precision /
recall, breaking FP |
|---|---|---|
| vitest (#57, candidate) | 94% / 50%, 1 (`jest_pdf_snapshot.ts:42`) |
**100% / 50%, 0** |
| zod | 100% / 71%, 0 | 100% / 71%, 0, identical |
| stripe | 100% / 90%, 0 | 100% / 90%, 0, identical |
| ai | 100% / 36%, 0 | 100% / 36%, 0, identical |
| react | 100% / 69%, 0 | 100% / 69%, 0, identical |

vitest per rule after: `matcher-types` 100% / 100% (3 of 3), was 75% /
100%. `pack test vitest` still exits non-zero on this branch only
because #57's `verification.json` records 94% / 1 false positive and no
longer matches; that file is #57's to rewrite with `--write`. Its 16
false negatives and the 18,900-odd `generic (unverified)` sites (not
breaking) are unchanged.

## Checklist

- [x] Tests for the change (a bug fix starts with a failing test)
- [x] `pnpm lint && pnpm typecheck && pnpm test` pass
- [x] README and docs updated if behavior changed
(`docs/architecture.md`, `docs/concepts.md`)
- [x] Changeset added if a user would notice (`pnpm changeset`): minor
- [x] No new dependency, or it was agreed in an issue
- [x] No private code, names or paths in fixtures, tests or docs
- [x] The rules in CONTRIBUTING.md ("Rules that do not bend") still hold

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018u9b3Pp5H68DdYZGp5Qg1U

---
_Generated by [Claude
Code](https://claude.ai/code/session_018u9b3Pp5H68DdYZGp5Qg1U)_

---------

Signed-off-by: Lucas Braum <lucas.braum@deskfy.io>
Co-authored-by: Lucas Braum <lucas.braum@deskfy.io>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

Copy link
Copy Markdown
Collaborator Author

Superseded by #63


Generated by Claude Code

@lucasbraum lucasbraum closed this Oct 10, 2026
lucasbraum added a commit that referenced this pull request Oct 10, 2026
Supersedes #57.

## Pack

`vitest`, `>=4 <5` → `>=5 <6`. Status: **verified** (2 public repos,
100% precision on breaking findings, recall 50%). The only breaking
false positive in #57 (`libs/image-utils/src/jest_pdf_snapshot.ts:42`)
is gone since #60 folds missing-member diagnostics (TS2339/TS2551) into
one finding at the repository's own declaration. Rules, expectations,
ground truth and companions are unchanged from #57.

Changes over #57:
- The README edit that joined two lines is undone (the line limit no
longer counts the packs table).
- `verification.json`, `packs/queue.json` and the changeset say
verified; the README packs table is regenerated (`pnpm docs:packs`;
`--check` and `pnpm docs:links` pass).
- Tests that used vitest as the example of a package without a pack
(`packages/cli/src/cli.test.ts`,
`packages/core/src/fix/isolate.test.ts`) use `lodash` instead, since
vitest now has a verified pack. Only the example package changed.

## Sources

- Migrating to Vitest 5.0: https://vitest.dev/guide/migration (read as
`docs/guide/migration/index.md` of vitest-dev/vitest at 69e6369)
- Vitest 5.0.0 release notes:
https://github.com/vitest-dev/vitest/releases/tag/v5.0.0
- Benchmarking guide: https://vitest.dev/guide/benchmarking
- Extending matchers: https://vitest.dev/guide/extending-matchers

No official codemod exists for this major.

## Rules and behavior notes

Rules: `matcher-types`, `jest-dom-matchers`, `bench-api`,
`removed-entrypoints` (rewrites four entry points),
`removed-entrypoints-manual`, `sequential-removed`. Behavior notes:
`coverage-patterns` (also a finding), `clear-mocks-default`,
`test-name-pattern`, `inline-projects-extend`,
`hoisted-calls-top-level`, `mock-behavior`, `assertion-timing`,
`pretty-format-output`, `fake-timers-temporal`, `artifact-locations`,
`config-resolution`, `browser-mode`, `environment-requirements`,
`node-api`.

## Ground-truth repositories

- votingworks/vxsuite @ `5cb3c7d9c1abc891e7b12de57c0278790dd6160a`,
upgrade commit bb04c5fb01aa9bcd285170d2277df04ad5d7e09c (vitest 4.1.11 →
5.0.2, with `@vitest/coverage-istanbul`): 30 expected findings.
- vitorvasc/opentelemetry-ecosystem-explorer @ `46d5b37d00bb` (directory
`ecosystem-explorer`), upgrade PR head
2b0497b35d0e49dbeeb0d6174ad09aed03d46697 (4.1.11 → 5.0.0): 2 expected
findings.

Dropped candidates are listed in #57 (siemens/element cannot install
from a public registry; the others are version-only bumps).

## `uptide pack test` output

```
uptide pack test vitest · verified, recorded candidate   (before --write; now recorded verified)
fixtures      5 cases, 17 sites · pass
ground truth
  votingworks/vxsuite@5cb3c7d9c1ab             vitest 4.1.11 → 5.0.2  predicted 15333, expected 30, false positives 15319, false negatives 16
    with @vitest/coverage-istanbul 4.1.11 → 5.0.2
  vitorvasc/opentelemetry-ecosystem-explorer@46d5b37d00bb vitest 4.1.11 → 5.0.0  predicted 3621, expected 2, false positives 3619, false negatives 0

  rule                         precision  recall    tp   fp   fn
  bench-api                          15%     67%     2   11    1
  coverage-patterns                 100%      0%     0    0   13
  generic                             0%    100%     0 18927    0
  jest-dom-matchers                 100%     85%    11    0    2
  matcher-types                     100%    100%     3    0    0
  overall (sites)                     0%     50%    16 18938   16
  breaking (sites)                  100%     50%    16    0   16

PASS vitest: verified
```

(The ~18,900 `generic (unverified)` lines are omitted; the 11
`bench-api` false positives are unverified inferred sites, not
breaking.)

Existing packs, rerun, all unchanged:

```
zod     verified  overall 100% / 84%, breaking 100% / 71%  (PASS)
stripe  verified  overall 100% / 90%, breaking 100% / 90%  (PASS)
ai      verified  overall 100% / 77%, breaking 100% / 36%  (PASS)
react   verified  breaking 100% / 69% (89 sites, 0 false)  (PASS)
```

Environment note: tldraw (react) pins Yarn Berry and the sandbox `yarn`
is classic, so the react run used a corepack shim on PATH (sandbox
only). With that shim on PATH, stripe's nextjs/saas-starter failed to
install (`MODULE_NOT_FOUND`); stripe passes without the shim.
Environment issues, not regressions.

`pnpm lint`, `pnpm typecheck` and `pnpm test` pass.

## What is not covered

- Pack `detect` runs only on files that import the package, so these
expected sites are missed: 13 bare-directory `coverage.include/exclude`
entries in configs that import a shared `defineConfig` wrapper, and
jest-dom imports in two setup files that import nothing from `vitest`.
Breaking recall is 50%.
- `check` could not resolve imports in 69 of 71 vxsuite files (workspace
packages are not built), so only a few of the 169 compiler diagnostics
are visible there.
- `bench-api` misses the describe wrapper at
`render_ballot.bench.ts:16`.
- Rules and notes with no evidence in either repo, resting on fixtures
and the guide only: `removed-entrypoints*`, `sequential-removed`, nested
`vi.mock`, and all behavior notes except `coverage-patterns`.
- The two repos are one monorepo and one small app; a third that really
migrated would make "verified" stronger.

## Tooling friction

- `pnpm uptide check` from source fails inside its worker (`Cannot find
module .../packages/core/src/errors.js`); `pnpm build` and the dist CLI
work. `pack test` is unaffected.
- `pnpm packs:truth` requires full 40-character SHAs and prints the new
path with the old line number for moved files.
- A corepack shim on PATH fixes Yarn Berry repos but breaks the pnpm
install of nextjs/saas-starter in the same run; the packs had to be run
with and without it.
- vitest.dev is not reachable from the sandbox.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01NwSXH7Dy3r4KjJhwKYBuxT

---
_Generated by [Claude
Code](https://claude.ai/code/session_01NwSXH7Dy3r4KjJhwKYBuxT)_

---------

Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pack-candidate Migration pack proposed by the daily pack task

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants