feat(packs): stricter verified gate: real breaking sites, unexercised rewrites run as assisted (UPT-48)

[lucasbraum]

Phase 2 report

1. Sentry demoted. @sentry/browser (3 real breaking sites) and @sentry/react (2) are recorded candidate. packages/core/src/packs/sentry/RESEARCH.md says they need at least 5 real breaking sites each, and lists repositories to search later: cal.com, midday, rallly, inbox-zero, unkey, twenty.

2. Unexercised rules run as assisted. pack test --write ran once on every pack except joi and redis. Only the two Sentry statuses changed; no precision, recall or count moved. Each record now carries class, breakingSites, behaviorSites and unexercised.

   • activePacks wraps each pack: transform refuses an edit whose rule is unexercised (fix hands the site to the agent with the guide), and check demotes such a site to assisted. The wrapper checks the rule of the applied edit, so the hand-written zod and stripe transforms are covered.
   • Stripe's pin and fixture-cast edits have no rewrite of their own. A new Pack.mechanical list makes the gate count them.
   • Now assisted: lucide-react renamed-icons; next stable-cache-exports; vitest removed-entrypoints; zod error-members, error-params; stripe api-version-unpinned, fixture-cast; and the Sentry rules each record lists.
   • zod error-members: no truth added. I scanned merged PRs that bump zod 3→4 and remove a .formErrors/.errors read (chore(deps): Update dependency zod to v4 PrefectHQ/prefect#22795, [deps]: Update zod to v4 bitwarden/ai-plugins#102, chore(deps): upgrade Zod to v4.4.3 jesseiancary/plinth#41, chore: express v5 + zod v4 Seven74AI/music-library#176 and others). Every catch-binding read is narrowed by instanceof z.ZodError, so it is typed and compiler-visible (error-issues), not read through any. test: Add comprehensive test suite + fix Zod v4 validation crash in all controllers Naeha-S/Bloggonut#2 and fix(smart_read): guard zod-v4 error issues and require non-empty path ooples/token-optimizer-mcp#167 do read .errors in a catch, but on zod 4 already: they are later crash fixes, not 3→4 upgrades. So error-members runs as assisted.

3. zod error-params, 0 of 3 public sites: not fixed. All three are found as breaking true positives but filed under generic. They are errorMap sites: z.literal(true, { errorMap }) in AwesomeIntune, and { errorMap } plus a z.ZodErrorMap function in PATH. The error-params rewrite only handles required_error/invalid_type_error and skips errorMap ("needs manual merging"). zod has no ruleOf, so the plan files a site under error-params only when the rewrite applies or the error text names those two keys. Re-filing them under error-params would count them as evidence for a rewrite that never ran on them. The right fix is a separate assisted error-map rule with its own guide (zod 4: error replaces errorMap, and the map's signature changed). That is a pack change, left for a follow-up.

4. joi and redis stay candidate (not written). Why redis's record differed on purpose: it is still the scaffold record (0 repos, zero counts), never written. feat(packs): candidate redis (node-redis) 4 → 5 and 5 → 6 pack (UPT-58) #151 held it at candidate until this gate landed, because:

   • breaking recall is 61% (11/18);
   • behavior notes are 50/50, with all four client-defaults predictions false positives;
   • the core 4→5 API changes (command-options, legacy-mode, scan-iterators, number-replies, keep-alive) have no real site;
   • two engine gaps cost sites: no compile errors in files that reach the client through a local wrapper, and errors probably subtracted where the installed version already had one.

   Under this gate it measures verified (5 repos, 20 real breaking sites, 0 breaking FP). keep-alive, quit-disconnect, scan-iterators and unstable-resp3 would run as assisted.

5. Override and README. meta.minBreakingSites lowers the minimum for one pack, and pack test fails a pack that sets it without a reason under ## Breaking-site minimum in its RESEARCH.md. No pack uses it. The README packs table has a "Breaking sites" column, and docs/packs.md and CONTRIBUTING describe the gate.

Also: --write now formats a record the way biome does (a short list inline), so a written record passes lint. The fix-machinery tests use zod's required_error rewrite as their mechanical example; they mock the registry wrapper to keep it as written.

Local runs: the @sentry/react truth repo pins pnpm 12.8.2, which corepack 0.36 cannot launch (pnpm 12 ships a native binary its postinstall fetches). I scored it with a local pnpm 12.11.2 through a scratch shim. CI's corepack may hit the same thing if the repository is not in its cache. These tests also fail on main locally (environment): required-build, peer-preflight, scaffold, release, telemetry sender, third-party notices.

---

Phase 1: the gate and its impact (measured before the records were written)

The gate

verifiedGate (contract.ts) replaces statusOf. gateOf (pack-test.ts) feeds it from one run; "real" means public repositories, so a fixture repository is scored and never counts.

	breaking class (default)	behavior-only class
applies when	any real expected site is under a rule or generic	every real expected site is a behavior note's, whatever its severity (ioredis 6)
repositories	≥ 2 public	≥ 2 public
real sites	≥ 5 expected breaking sites (VERIFIED_MIN_BREAKING_SITES)	≥ 5 expected note sites (VERIFIED_MIN_BEHAVIOR_SITES)
false positives	0 among breaking findings	0 among breaking findings, 0 on note sites, 0 on mechanical rules
found	≥ 1 breaking site	≥ 1 note site; per-note recall is in the rules table (#143)

Unexercised rewrites: a rule with a rewrite and no true positive in public ground truth is listed as unexercised. It never changes the status.

• pack test prints a gate line (class, real site counts, status), each reason a pack is a candidate, and the unexercised rules.
• The computed record carries class, breakingSites, behaviorSites and unexercised, so every record is now stale (a note, not a failure) until --write.
• overclaims() is unchanged from fix(packs): fail a verification record only when it claims more than the run measures #154: a record fails only when it records verified and the run measures candidate; recorded candidate by choice passes.

Not in phase 1

• fix running unexercised sites as assisted: it needs the unexercised list in each record, which only phase 2's --write produces. Wrapping transform for the hand-written packs (zod, stripe) is the work there.
• A per-pack override of the minimum with a written reason in RESEARCH.md.
• The README breaking-site column.

Impact: every registered pack, under the new gate

Each pack was run one at a time from source (pnpm uptide pack test <id> --json), each with its own cache directory, deleted after the pack. Site counts are expected sites in public ground truth. "Breaking tp/fp/fn" is across every scored repository, fixtures included.

pack	recorded	measures now	real breaking / note sites	breaking tp/fp/fn	unexercised mechanical rules	why
ai	verified	verified	11 / 0	9/0/2	—
lucide-react	verified	verified	11 / 0	11/0/0	renamed-icons
next	verified	verified	13 / 0	11/0/2	stable-cache-exports
react	verified	verified	130 / 0	89/0/40	—
react-router	verified	verified	18 / 0	14/0/2	—	see below
stripe	verified	verified	16 / 0	18/0/2	—
vitest	verified	verified	32 / 13	16/0/16	removed-entrypoints
zod	verified	verified	85 / 0	59/0/21	error-members, error-params	error-params' TPs are all in the fixture repo; its 3 public sites are missed
@sentry/nextjs	verified	verified	8 / 0	8/0/0	enable-logs, enable-metrics, http-integration-options, integration-names, renamed-exports, streamed-span-wrapper
@sentry/browser	verified	candidate (overclaim, fails)	3 / 0	3/0/0	enable-metrics, integration-names, renamed-exports, streamed-span-wrapper	3 breaking sites, needs 5
@sentry/react	verified	candidate (overclaim, fails)	2 / 0 in truth	—	enable-logs, enable-metrics, integration-names, renamed-exports, streamed-span-wrapper	2 breaking sites in its truth, needs 5. Locally one repo failed to install (pnpm crashed with MODULE_NOT_FOUND, environmental), but the count alone drops it
joi	candidate	verified	108 / 0	108/0/0	—	recorded candidate by choice: passes
redis	candidate	verified	20 / 2	11/0/7	keep-alive, quit-disconnect, scan-iterators, unstable-resp3	recorded candidate by choice: passes
ioredis	candidate	candidate (behavior class)	2 / 13	0/0/2	—	3 false positives on client-defaults note sites

Drops: only @sentry/browser and @sentry/react. Both now fail pack test as overclaims, so the Packs job is red on this PR until phase 2.

react-router: on main, the only mechanical rule is dom-imports (29 real TPs). removed-json, removed-defer, form-method-case, dom-server-entry and future-option have no rewrite; they are already assisted, so the gate turns none of them assisted. Separately, dom-imports has 336 non-breaking false positives at the rule level (precision 8%), which no gate criterion reads.

Mechanical rules that would turn assisted (unexercised): 24 rule entries across 8 packs, 15 distinct ids. lucide-react renamed-icons; next stable-cache-exports; vitest removed-entrypoints; zod error-members, error-params; redis keep-alive, quit-disconnect, scan-iterators, unstable-resp3; and the Sentry rules listed per pack above.

Tests

contract.test.ts has a pass and a fail case for each criterion, plus the unexercised list and the behavior-only class (verified, too few sites, false positives on notes and rules, never chosen with a rule site). pack-test.test.ts covers gateOf: fixture sites never count or exercise a rule, and an unscored repository is no evidence. (scaffold.test.ts fails locally with formatted: false on main as well; unrelated.)

🤖 Generated with Claude Code

[lucasbraum]

CI on 02f3019 failed in two places, both from this PR, and ac063b8 fixes them:

• smoke (20/22): the no-LLM fix expected zod's required_error sites to migrate by rule. They are assisted now, so there was no by-rule commit and uptide/zod-4.6.5~2 did not exist. The fixtures now use z.record(value) (record-key: mechanical, exercised by real ground truth), with the same site counts. The smoke checks that rule's line and the two-argument form. I checked the npm fixture locally with the built CLI: 2 auto-fixable, verification passed, two commits on the branch.
• test (20/22): scaffold.test saw a fresh pack's record as stale, because pack new did not write the gate's fields. It now writes class, site counts and the example rewrite as unexercised, in the formatter's layout.

The gate is unchanged.