test(packs): public ground truth for assisted mechanical rules (UPT-69) by lucasbraum · Pull Request #156 · uptide-dev/uptide

lucasbraum · 2026-10-15T16:52:55Z

UPT-69. I looked for public upgrade commits in which a person made exactly the change one of the assisted mechanical rules (UPT-48) makes. For each pack, the ground truth was drafted with packs:truth, labeled by hand and committed before scoring. Then the pack was re-scored once with --write. Existing truth is unchanged.

Pack	Rule	New public repos	Rule tp/fp	Result
zod	`error-params`	vzdc-artcc/website (13 sites), CANShift/canshift-core (2)	39 / 0	mechanical again; pack stays verified
stripe	`api-version-unpinned`	starl3xx/wallet-to-social (3), AtlasDevHQ/atlas (2)	5 / 0	mechanical again; pack stays verified
stripe	`fixture-cast`	none installable	n/a	still assisted
next	`stable-cache-exports`	benshabbat/GamesForMyKids (1), HiroshiIgarash/koremina (2)	3 / 0	exercised, but the pack now measures candidate (see below)
lucide-react	`renamed-icons`	none in range	n/a	still assisted
vitest	`removed-entrypoints`	none	n/a	still assisted

Needs your decision: next is now candidate

In both new next entries I listed @next/eslint-plugin-next under with. That was a drafting mistake: the package moved only as a transitive dependency of eslint-config-next, and neither repository declares it in package.json. docs/packs.md says with comes from package.json; exif-photo-blog, by contrast, does declare it. check does not move the plugin, so the gate fails with "check leaves part of the upgrade behind" and --write recorded candidate.

Per the rule, I did not change the truth after scoring. Your options:

Drop the two with entries in a reviewed follow-up and re-score.
Keep next as candidate.

Other things the run showed

next: one middleware-proxy false positive, deprecated severity, at GamesForMyKids middleware.ts:69. That file is a supabase helper the upgrade left alone. It is not a breaking false positive.
zod, vzdc-artcc/website: 350 findings at unverified severity (287 generic, 63 types). These do not count as breaking false positives. They are probably caused by the Prisma client, which is generated and therefore missing when installed scripts-off.
zod, vzdc-artcc/website: 36 implicit-any (7006) diagnostics were left out of the truth. They come from mapping over .errors, which is any at zod 4. The .errors sites themselves are recorded under error-issues.
next drafts: the helper's known resolution artifacts were left out of the truth (next/font/google TS2305, Script/styled-jsx TS2322, implicit-any), following note 4 in next's RESEARCH.md.

Searched, not usable

stripe fixture-cast: fix(deps): update dependency stripe to v17 opencollective/opencollective-api#10416 is a real human change. The PR is a Renovate bump, and Henrique's commit 0c0d846 changes three casts to as unknown as Stripe.X. But the repo pins node 20.x with engine-strict=true, so it cannot be installed under the harness. No other candidate was found across about 1,800 scanned stripe bump PRs.
stripe api-version-unpinned, skipped candidates:
- jetstreamapp/jetstream: engine pin.
- icco/etu-web: private @icco GitHub Packages.
- ohmiler/milerdev: Claude co-authored.
lucide-react renamed-icons: no 0.x → 1.x upgrade renamed a listed icon. The listed renames that do exist (direktiv, marimo, immarkus, coai: MousePointerSquare → SquareMousePointer) happened in 0.x → 0.x bumps, which are outside the pack's range.
vitest removed-entrypoints: I scanned about 790 merged Vitest 5 PRs and none changed a vitest/{coverage,reporters,environments,snapshot} import in TS. Near misses:
- gyng/save-in: the change is in a .mjs JSDoc type.
- nuxt/test-utils and mayoite: the import moved without a version bump.
Search method: gh search prs/commits/code over upgrade titles, old and new API names, and codemod names. Every diff was checked with git diff parent..commit, and versions came from the lockfiles.

Checks

pnpm lint passes.
public:check is clean.
Pack unit tests: scaffold.test.ts fails locally on main as well. pack-test.test.ts passes on its own and is flaky only under the full parallel run.

🤖 Generated with Claude Code

lucasbraum · 2026-10-15T18:00:33Z

Correction (9137ebf, then re-score a7a1377).

9137ebf fixes a transcription error against the pre-declared method; it is not a change made to fit the run. docs/packs.md says with lists the packages the upgrade moved, at the versions its package.json chose. Neither of the two new entries declares @next/eslint-plugin-next in package.json at its upgrade commit:

benshabbat/GamesForMyKids, in gamesformykids/package.json
HiroshiIgarash/koremina, in package.json

You can verify this in each upgrade commit's package.json. The package moved only in the lockfile, as a transitive dependency of eslint-config-next. The commit removes it from those two with lists and changes no expected site.

Re-score (a7a1377). I re-scored next once with --write, and it measures verified: 34 breaking sites, 0 breaking false positives, and stable-cache-exports at 3 true positives / 0 false positives. The README table and changeset are updated to match.

One false positive remains, at deprecated severity, so it is not breaking: middleware-proxy at GamesForMyKids middleware.ts:69.

🤖 Generated with Claude Code

… (UPT-69) vzdc-artcc/website and CANShift/canshift-core each rewrote zod 3 error params (required_error, invalid_type_error) to error in their own zod 4 upgrade commit. Drafted with packs:truth and labeled by hand before scoring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

…T-69) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

…ades (UPT-69) starl3xx/wallet-to-social and AtlasDevHQ/atlas each pinned their unpinned Stripe clients in their own stripe major upgrade commit. Drafted with packs:truth and labeled by hand before scoring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

…c sites (UPT-69) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

…grades (UPT-69) benshabbat/GamesForMyKids and HiroshiIgarash/koremina each moved their aliased unstable_cacheLife/unstable_cacheTag imports to the stable names in their own Next 15 to 16 upgrade commit. Drafted with packs:truth and labeled by hand before scoring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

…res candidate (UPT-69) stable-cache-exports: 3 true positives, 0 false positives in the two new repositories. The run measures candidate because both new entries list @next/eslint-plugin-next under `with`, a transitive dependency neither package.json declares, and check does not move it. Recorded as measured; the truth is left as committed before scoring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

…PT-69) Fixes a transcription error against the pre-declared method, not a change to fit the run. docs/packs.md says `with` lists the packages the upgrade moved at the versions its package.json chose. @next/eslint-plugin-next is declared in neither benshabbat/GamesForMyKids (gamesformykids/package.json) nor HiroshiIgarash/koremina (package.json) at their upgrade commits; it only moved in the lockfile as a transitive dependency of eslint-config-next. Removed from those two `with` lists. No expected site changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

…ed (UPT-69) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

lucasbraum · 2026-10-15T18:02:43Z

Rebased on main after #157 (UPT-63), now at 156147f.

The only conflict was the README packs table. I took main's README and regenerated the table with pnpm docs:packs; no hand edits.
The zod, stripe and next verification.json files are byte-identical to the pre-rebase head, so I didn't re-score.
Focused tests pass: zod, stripe and next packs, contract, pack-test, companions (196 passed, 1 skipped).

🤖 Generated with Claude Code

…d-move fixes Rebased on main after #155, #156 and #157 and re-ran pack test --write on the unchanged ground truth: breaking 3 TP / 0 FP / 17 FN, and the run now measures verified. The record stays candidate (allowed since #154): seven rewrite rules are unexercised by public ground truth and recall is 15%. README table regenerated with docs:packs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FX2nBoNEvDh9v71yDhEzxL Signed-off-by: Claude <noreply@anthropic.com>

@lucasbraum

This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## uptide@0.13.0 ### Minor Changes - [#134](#134) [`b795705`](b795705) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Packs can name alternate leads: packages a repository declares instead of the pack's own (`react-router-dom` for `react-router`). `check` and `fix` asked for either name use the pack; where only the alternate is declared, the lead is analyzed at the version installed through it, pnpm stores included, and `fix` declares it beside the alternate at the target before upgrading both. - [#149](#149) [`3303118`](3303118) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add a candidate express 4 to 5 migration pack. Route strings that path-to-regexp 8 rejects or reads differently are found on calls of an Express app or router (including lists registered in a loop); `req.params` values widened to `string | string[]`, `req.param`, `res.sendfile`, `express.static.mime` and `express.query` are claimed from the compiler; `res.redirect(url, status)`, `app.param(fn)`, `res.redirect('back')` and `res.send(<number>)` are found in the source; runtime defaults (async errors, `req.body`, the query parser, `urlencoded`, dotfiles, Node 18) go to decisions. No rule rewrites. With the pack in use, `check` no longer adds its text-matched express 5 runtime findings. - [#148](#148) [`5143c46`](5143c46) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add a candidate joi 17 to 18 migration pack. `new ValidationError(message, {}, original)`, which stops compiling for code written against joi 17.0–17.6, is rewritten to pass `[]`; the Node.js 20 requirement and the top-level domains joi 18 no longer accepts in `email()`, `domain()` and `uri()` are reported as decisions. Scored against two public repositories, one of them a negative control. - [#131](#131) [`413d3da`](413d3da) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add a verified lucide-react 0.x to 1.x migration pack. Brand icons removed in 1.x are found at their imports and every use, with a guide whose replacement choice goes to the owner; icon names dropped during 0.x are renamed at the import or member; deep `dist/` imports, untyped icon lookups, the `absoluteStrokeWidth` and `IconNode` deprecations, and the `aria-hidden` default for repositories before 0.489.0 are reported. Lucide's lockstep packages move together and the consumer's build is required. Verified against three public repositories, one of them a negative control. - [#132](#132) [`107c107`](107c107) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add a verified React Router 6 → 7 migration pack, which also runs for applications that declare only `react-router-dom`. It moves `react-router-dom` imports to `react-router`. It reports, with a guide for the agent, the rest of the upgrade: the removed `future` prop and router option, `react-router-dom/server`, lowercase `formMethod` comparisons and `json`/`defer` calls (found by syntax where the compiler is blind), the other removed exports, and `generatePath` calls that pass `null` for a required parameter. The six v6 future flags that become defaults are reported as decisions where the repository's router setup has not enabled them. React Router 7 → 8 is a separate upgrade. - [#142](#142) [`9c57934`](9c57934) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check` and `fix` plan what the target version requires: a package the target, or anything moving with it, declares as a peer or dependency at a range the installed copy rejects moves to the lowest release of its major that satisfies it (openai 7 moves zod 3.24.2 to 3.25.0), shown apart from release-group companions ("moved because openai 7.4.0 requires zod ^3.25 || ^4.0"). A move only a new major makes is never made: it is reported as a blocker with the exact range in `check`, the PR body and `pack test`. The npm install scope guard no longer lets a declared root dependency change because a moved package peers on it. - [#128](#128) [`06f11ab`](06f11ab) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add verified Sentry JavaScript SDK 10 to 11 migration packs for `@sentry/nextjs` and `@sentry/browser`, with the SDK's lockstep release group, a required build for Next.js, and inventory-scoped decisions for data collection, span streaming, sampling and platform minimums. Mechanical rules move `withSentryConfig` to `@sentry/nextjs/config` and the removed build options under `webpack`, remove `enableLogs`/`enableMetrics: true`, and rename exports, integration names and `httpIntegration` options; privacy, sampling, profiling and OpenTelemetry choices stay assisted and are reported for the owner. Fix `uptide pack new` dropping registry entries long enough to be wrapped. - [#155](#155) [`922c5ab`](922c5ab) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide pack test` applies a stricter verified gate: at least five expected breaking sites across the public ground-truth repositories, besides two repositories and no breaking false positive. Upgrades the compiler cannot see are gated on their behavior notes instead. Mechanical rules no real site exercises are recorded as unexercised, and `check` and `fix` run their sites as assisted. `@sentry/browser` and `@sentry/react` become candidates (fewer than five real breaking sites). A pack can lower the minimum only with a written reason in its RESEARCH.md. The README packs table shows each pack's breaking-site count. The output shows the gate's class, site counts and every reason a pack is a candidate. - [#147](#147) [`0b88170`](0b88170) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check` and `fix` remove a deprecated `@types` stub instead of upgrading to it. When the release the upgrade would move `@types/<pkg>` to is deprecated on npm, and `<pkg>`'s target declares its own types (`types`, `typings` or a `types` export), the plan removes it: "Removed: @types/ioredis, ioredis 5 ships its own types", in `check` and in the PR body. `check` now diffs such a package against its own target types instead of failing on a stub with no declarations, and `fix` removes the entry in the same install, with the install scope guard allowing exactly that removal. - [#150](#150) [`4c3212b`](4c3212b) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check typescript` judges a TypeScript 5 to 6 upgrade with both compilers: every project of the workspace is compiled with the installed TypeScript and with the target's, and what is new at the target is a finding; the bundled compiler never stands in, and a target without a JavaScript API (TypeScript 7) is reported as not compiled. A candidate typescript 5 to 6 pack reports the tsconfig options 6.0 deprecates where they are set, the rootDir default, module-keyword namespaces and import assertions. A peer that rejects the target (typescript-eslint on TypeScript 6) is now reported as a peer conflict even when it is not grouped with the package. - [#137](#137) [`21894d7`](21894d7) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add `verify.build` to uptide.yml: the package.json script a migration pack's required build runs. Unset, Uptide runs `build:ci`, then `build-ci`, then `build`. When a build fails before the framework's build starts because it needs an environment variable or a database (`prisma db push` without `DATABASE_URL`), the pull request and the report say so ("build needs DATABASE_URL; set verify.build") instead of a bare "build failed". `uptide init` writes the setting as a comment. ### Patch Changes - [#141](#141) [`9debb6d`](9debb6d) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check` no longer reports a pre-existing type error as a breaking site when the upgrade only changes its wording, such as a type the message prints in full. An error at the same place with a different reason is still reported. - [#129](#129) [`e796e4c`](e796e4c) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add a candidate ioredis 5 to 6 migration pack: client constructions that inherit the RESP3, reconnect and keep-alive defaults ioredis 6 changed, compiler-blind sites (`mode === 'subscriber'`, commands in a `connect` listener with the offline queue off, AUTH recognized by command name), dependents that bundle ioredis 5, instrumentation that stops patching ioredis 6, and Node 20. It is scored against five public upgrades and stays a candidate: none of them has a breaking site to verify it with. - [#139](#139) [`9ba005a`](9ba005a) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Allow local check and fix smoke tests of registered candidate packs with --allow-candidate, with an explicit unverified banner. Reject candidate opt-in in the Action and with publication flags, and prevent stored candidate runs from being published. Label pack-claimed assisted sites by rule id even with --no-llm. - [#157](#157) [`f3af6a1`](f3af6a1) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check` no longer reports a blocker for a direct dependency whose plain dependency range rejects the target (`@azure/ai-projects` on `openai ^6`, verdaccio pinning express 4): the package manager gives it its own nested copy, so it is listed as staying, with the new major that would accept the target when there is one. `pack test` counts a `with` move that `check` leaves and explains (a blocker, a new-major stay or a peer conflict) as explained, and one of a package that does not depend on the lead as the repository's choice; neither fails the run. - [#130](#130) [`5ced450`](5ced450) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add a candidate openai 4 to 7 migration pack: named path parameters, `del` → `delete`, promoted beta chat helpers, renamed runner events, removed entry points and request-option overloads are rewritten where unambiguous; tool-call unions, `httpAgent`, `fileFromPath` and the other removals get agent guides; `APIError.headers`, Web `Response` bodies and double-encoded path ids are detected even in code typed `any`; Node 22, zod, fetch and client-subclass decisions carry repository evidence. It stays a candidate (the dependency is still checked and fixed as generic) until two engine gaps its ground truth exposed are closed. - [#136](#136) [`578a042`](578a042) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Fold implicit-any errors (TS7006, TS7031, TS7019) inside an option the target removed into the pack finding on that option, as downstream evidence: the callbacks lost the types the option gave them, and the same edit resolves them. They no longer stand as separate breaking sites. - [#143](#143) [`883e587`](883e587) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide pack test` scores each expected site at the severity of the rule or behavior note it belongs to, so a missed note that only needs a look, or a decision, no longer counts as a missed break. Behavior-note sites get their own precision and recall in the output and in `verification.json` (`behavior`, with the counts behind each rate), false negatives show their severity, and a rate with nothing to measure prints `n/a` instead of 100%, in `pack test` and in the README's packs table. - [#154](#154) [`7aa3d58`](7aa3d58) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide pack test` fails a pack's `verification.json` only when it claims more than the run measures (recorded verified, measured candidate). A pack recorded as candidate on purpose passes, with `recorded candidate by choice; measures verified`, and a record whose numbers differ gets a note instead of a failure. A `generic` ground-truth site now counts as a true positive for the rule that claims it. - [#152](#152) [`ca31aee`](ca31aee) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide pack test` takes several packs: `uptide pack test zod ai`. - [#159](#159) [`1e157fd`](1e157fd) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide pack test --json` times each ground-truth repository: `timing` gives the fetch, install and check wall time and the process's peak memory. - [#144](#144) [`0881185`](0881185) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Ground truth can name a site for an upgrade that only adds lines inside an existing call, such as an option added to a constructor: the site is the call's start line, so a pack finding there scores as a true positive instead of a false positive. `pnpm packs:truth` drafts these anchors from insert-only hunks for review, and docs/packs.md and CONTRIBUTING describe the rule. - [#156](#156) [`9039e3c`](9039e3c) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Public ground truth now exercises three rules that ran as assisted: zod `error-params`, stripe `api-version-unpinned` and next `stable-cache-exports`. Their rewrites run mechanically again. All three packs stay verified. - [#138](#138) [`0db5640`](0db5640) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Respect repository tsconfig file selection, recursive project references, and workspace project options when checking dependency upgrades. Compiler diagnostics outside every TypeScript project no longer produce breaking findings; pack rules and build-configuration findings retain their own scope. Repositories without a tsconfig retain importer-based compilation, with that fallback stated in the report. Anchor missing members supplied by dependency augmentations at their setup import, retaining affected calls as evidence instead of counting each call as a separate breaking edit. - [#151](#151) [`9ddaaff`](9ddaaff) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add a candidate redis (node-redis) 4 to 5 and 5 to 6 migration pack, one rule set per major: an upgrade gets the rules of the majors it crosses, so 4 to 6 gets both. 4 to 5 covers `legacyMode`, `commandOptions` and isolation, scan iterators that yield batches, string SCAN cursors, boolean replies that became numbers, enums that became constants, `keepAlive`, `@redis/graph` and internal `dist` imports; 5 to 6 covers the RESP3 client type, `unstableResp3`, and the client defaults 6 changes (RESP3, command timeout, keep-alive delay, maintenance notifications) with a decision per client. Scored against five public upgrades; it stays a candidate until the stricter verified gate lands, since the core 4 to 5 API changes have no real site yet. - [#135](#135) [`42a3b54`](42a3b54) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Move only packages released with the upgraded one. A pack's declared release group now bounds the lead's npm scope, and without one a same-scope package moves only when a pin, a peer or a release at the target relates it to the group. Packages of the scope that are versioned on their own (`@sentry/vite-plugin` 5.x, `@sentry/cli`, the old `@sentry/utils` 8.x) are reported as "stays at X" in `check`, the pull request and `pack test` instead of making `fix` refuse the whole upgrade. Registers the verified `@sentry/react` 10 to 11 pack. - [#146](#146) [`09b8682`](09b8682) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Report Stripe clients pinned to different API versions at the top of `check` and of the PR body, as a decision for the owner: every `new Stripe(...)` in TypeScript and JavaScript sources, scripts included, with the version it speaks after the upgrade (an unpinned client speaks the SDK default). The PR's decisions list, for each client behind the target, the changelog entries it would cross for the resources it calls. No pin is rewritten. - [#140](#140) [`bc428d1`](bc428d1) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Keep migration PR bodies within their size budget while preserving summaries, representative diffs, decisions, review warnings and verification ahead of changelog and transitive detail. Keep every evidenced Stripe change visible and uncapped; group and cap only other changes to used resources, link the full changelog, and identify omitted text in the retained report. - [#145](#145) [`3c8902d`](3c8902d) Thanks [@lucasbraum](https://github.com/lucasbraum)! - The zod pack migrates `ZodError#errors` to `.issues` by rule: reads (`error.errors[0].message`, `error?.errors`) and destructuring (`const { errors } = error` becomes `const { issues: errors } = error`) where the type checker resolves the receiver to `ZodError`, and reads off an untyped catch binding of a zod parse. Writes are left to the agent. - [#127](#127) [`c4ead4f`](c4ead4f) Thanks [@lucasbraum](https://github.com/lucasbraum)! - zod: `check` now reports `ZodError#formErrors` and `#errors` read off an untyped catch binding of a zod `parse`/`parseAsync` (`catch (error: any) { return error.formErrors.fieldErrors }`), which zod 4 removed and the compiler cannot see. Files that only import a schema from the repository's own module (`@/lib/zod`) are covered too. `fix` rewrites `<err>.formErrors.fieldErrors` and `<err>.formErrors.formErrors` to `z.flattenError(<err>)`, importing `z` where it is missing; every other read goes to the agent. Co-authored-by: uptide-release[bot] <338861287+uptide-release[bot]@users.noreply.github.com>

…185) Closes UPT-68. A file importing a schema from a local module could assign `await schema.safeParseAsync(...)` to a variable, then read `result.error.errors` without `check` reporting it. The usage scan stopped tracing at the local variable and skipped common member names. Follow variable initializers back to the import, with cycle protection. The existing compiler resolver still proves the member belongs to the dependency, so unrelated `.errors`, shadowed inputs, and untyped values are not guessed. Reported Zod sites use the existing `error-issues` rewrite. No ground truth or thresholds changed. Validation: - Regression failed before the fix (zero of four expected reads) and passes afterward. Covers sync/async results, aliases, optional/non-null access, a local barrel/path alias, unrelated properties, shadowing, and cycles. - 54 focused adapter/Zod tests pass. - Real Zod 3.25.76 → 4.6.5 repro: six reads reported breaking/mechanical, all six rewrites applied, unrelated `form.errors` preserved, and `tsc --noEmit` passes after upgrade. - `pnpm lint` and `pnpm typecheck` pass. Lint retains the existing Biome schema-version notice and warning in `packs/next/platform.test.ts`. The issue's secondary `error-params` concern predates #156, which added public coverage and restored mechanical status. Current score comparison follows below. `pnpm test` passes (full core and CLI suites). ### Zod pack scores Both full runs used the same pinned ground truth and installed repositories: clean baseline `aee6b82` in a separate worktree, then this change. `pnpm -s uptide pack test zod --offline --json` passes on both; status remains **verified**, fixtures remain **8 cases / 20 sites passing**, and breaking false positives remain **0**. | Metric | Before | After | | --- | ---: | ---: | | Breaking precision | 100.00% | 100.00% | | Breaking recall | 51.76% | 53.66% | | Overall precision (all severities) | 23.87% | 23.03% | | Overall recall (all severities) | 85.25% | 86.33% | | Breaking TP / FP / FN | 44 / 0 / 41 | 44 / 0 / 38 | | Overall TP / FP / FN | 237 / 756 / 41 | 240 / 802 / 38 | | `error-issues` TP / FP / FN | 51 / 0 / 2 | 52 / 0 / 1 | | `error-params` TP / FP / FN | 39 / 0 / 3 | 39 / 0 / 3 | | `generic` TP / FP / FN | 80 / 532 / 28 | 82 / 534 / 26 | | `ip` TP / FP / FN | 1 / 0 / 0 | 1 / 0 / 0 | | `record-key` TP / FP / FN | 2 / 0 / 9 | 2 / 0 / 9 | | `string-format` TP / FP / FN | 27 / 0 / 2 | 27 / 0 / 2 | | `types` TP / FP / FN | 17 / 244 / 17 | 17 / 288 / 17 | Three existing misses are now found: AwesomeIntune `src/app/api/ideas/route.ts:201` and `src/app/api/submit-tool/route.ts:17`, and vzdc-artcc/website `components/SoloCertification/SoloForm.tsx:27`. **Tradeoff:** PATH gains 46 unmatched findings, all **unverified** (44 `types`, 2 `generic`), lowering overall precision. There are no new breaking false positives. The scorer counts these newly matched sites at their reported severity; this is why overall TP increases while breaking TP stays at 44. The `error-params` row is unchanged at 39 / 0 / 3, including 15 true positives in public repositories (24 in storefront). Its earlier zero-public-hit issue is already addressed. Both clean baseline and changed runs flag the committed verification record as stale: the committed breaking counts are 66 / 0 / 37, whereas this environment measures baseline 44 / 0 / 41. `verification.json` is left unchanged rather than folding that pre-existing discrepancy into this scanner fix. Neither run overclaims the pack's verified status. Not run: the full Papermark checkout, other packs' public-repository scoring, or CI watching. The real-Zod repro covers the reported local-schema/async-result shape; general assignment-flow analysis and scoped-project importer selection are unchanged. Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

…uid() (UPT-86) (#200) UPT-86. Breaking recall 54% → 65% (44 → 55 TP, 38 → 30 FN), 0 breaking FP. Ground truth and thresholds unchanged. ## Part 1: 66 vs 44 **44 / 0 / 38 is the number.** A clean local run (`packs:fetch`, `pack test zod`) and CI's `pack zod` job on main (`4c3dc23`, run 39009240360) agree exactly. The recorded 66 / 0 / 37 does not reproduce, not even at its own commit: `9039e3c` (#156) scores 38 / 0 / 37 here. What I could establish: breaking TP counts only sites `check` reports at `breaking`. For vzdc-artcc/website, 0 of its breaking sites are TP: its findings come back `unverified` with "57 of 95 files cannot resolve their imports; compile signal skipped". The files import `@/generated/prisma/*`, which `prisma generate` writes in a postinstall, and ground-truth installs run scripts off. `tsc` on the cached checkout confirms 234 TS2307 on those paths. #156 predicted this ("probably caused by the Prisma client"). I did not run `prisma generate`, so I could not show that a generated client yields 66; I believe that is the difference, but it is not demonstrated. The `unverified` findings cost recall, not precision. `verification.json` is regenerated from the run (`--write`); `docs:packs` updated README and compatibility. The README's 64% → 65% looks like a one-point gain; against what CI measures, the gain is 54% → 65%. ## Part 2: recall Reach first: of the 38 misses, 33 are in files that import zod directly (the scan reads them); 5 are not (`import("zod").X` only, `LessonForm.tsx` reading `error.errors` with no zod import). The misses are rule gaps, not unread files. | group | misses | cause | |---|---|---| | root exports now only in `zod/v3` (`SafeParseReturnType`, `ZodTypeDef`, `ZodEffects`) | 15 (7 FN here; 8 already seen as `types`) | **fixed**: the diff counts them as unchanged because `exportedFrom` overlaps on `./v3` | | `.refine(fn, { message })` → `error` | 6 | not done: deprecated, still compiles; needs a detect rule on the second argument | | compiler-only TS2322/TS2345 flowing across files (PATH `post-runs`, `resume-run`; band `server`, `sse`, `stdio`) | 10 | not done: no zod API at the line | | `import("zod").ZodTypeAny` and `ZodRawShape`/`_def`/`ZodOptional` name checks | 8 | not done: deprecated names, `_def` reads | | `errorMap` / `ctx.defaultError` | 2 | not done | | rule | before TP/FP/FN | after | |---|---|---| | removed-exports (new) | n/a | 7 / 8 / 0 (all 15 on truth lines; 8 labelled `types` there, so counted as a wrong rule) | | generic | 82 / 534 / 26 | 82 / 527 / 19 | | error-params, error-issues, string-format, record-key | unchanged | unchanged | | **breaking** | 44 / 0 / 38 | 55 / 0 / 30 | `uuid()` → `guid()` (the calmcp case): `string-format` wrote `z.uuid()`, which in zod 4 also checks the RFC version and variant; zod 3 accepted any 8-4-4-4-12 hex id. It now writes `z.guid()`. **Decision for you:** the public upgrades in the truth mostly use `z.uuid()`; I chose behaviour preservation. Fixture and guide updated; line-level scores unchanged. Checks: `biome` and `tsc` clean; pack unit tests pass except `scaffold.test.ts` and `pack-test.test.ts` "each companion", which fail on a clean main here too. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Lucas Braum <lucasbraum@gmail.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>

lucasbraum and others added 9 commits October 15, 2026 11:02

test(packs): re-score zod; error-params exercised by public sites (UP…

d682a00

…T-69) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

test(packs): re-score stripe; api-version-unpinned exercised by publi…

8fbea20

…c sites (UPT-69) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

docs(packs): packs table and changeset for the UPT-69 re-scores

17a9cd0

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

test(packs): re-score next after the with correction; measures verifi…

156147f

…ed (UPT-69) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

lucasbraum force-pushed the lucasbraum/upt-69-public-truth-assisted-rules branch from a7a1377 to 156147f Compare October 15, 2026 18:02

lucasbraum merged commit 9039e3c into main Oct 15, 2026
10 checks passed

lucasbraum deleted the lucasbraum/upt-69-public-truth-assisted-rules branch October 15, 2026 18:27

uptide-release Bot mentioned this pull request Oct 15, 2026

Version Packages #133

Merged

lucasbraum mentioned this pull request Oct 15, 2026

feat(packs): candidate express 4 to 5 pack (UPT-55) #149

Merged

8 tasks done

lucasbraum mentioned this pull request Oct 15, 2026

feat(packs): add candidate openai 4 to 7 migration pack #130

Merged

7 of 8 tasks

lucasbraum mentioned this pull request Oct 17, 2026

fix(check): detect Zod errors through local schema imports (UPT-68) #185

Merged

lucasbraum mentioned this pull request Oct 17, 2026

feat(zod): report root exports that moved to zod/v3, uuid() becomes guid() (UPT-86) #200

Merged

Navigation Menu

Repository navigation

test(packs): public ground truth for assisted mechanical rules (UPT-69) - #156

test(packs): public ground truth for assisted mechanical rules (UPT-69)#156
lucasbraum merged 9 commits into
mainfrom
lucasbraum/upt-69-public-truth-assisted-rules

lucasbraum commented Oct 15, 2026

Uh oh!

lucasbraum commented Oct 15, 2026

Uh oh!

lucasbraum commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

lucasbraum commented Oct 15, 2026

Needs your decision: next is now candidate

Other things the run showed

Searched, not usable

Checks

Uh oh!

lucasbraum commented Oct 15, 2026

Uh oh!

lucasbraum commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant