feat(packs): stricter verified gate: real breaking sites, unexercised rewrites run as assisted (UPT-48)

.changeset/stricter-verified-gate.md

@@ -0,0 +1,5 @@
+---
+"uptide": minor
+---
+
+`uptide pack test` applies a stricter verified gate: at least five expected breaking sites across the public ground-truth repositories, besides two repositories and no breaking false positive. Upgrades the compiler cannot see are gated on their behavior notes instead. Mechanical rules no real site exercises are recorded as unexercised, and `check` and `fix` run their sites as assisted. `@sentry/browser` and `@sentry/react` become candidates (fewer than five real breaking sites). A pack can lower the minimum only with a written reason in its RESEARCH.md. The README packs table shows each pack's breaking-site count. The output shows the gate's class, site counts and every reason a pack is a candidate.

CONTRIBUTING.md

@@ -94,9 +94,11 @@ Rules for a pack change:
    by `uptide pack test`. Expected findings come from the repository's own upgrade and the
    compiler, never from Uptide's output. No private code, paths or names anywhere: fixtures
    are synthetic, ground truth is public.
-4. A pack is verified only with ground truth from two public repositories and no false
-   positive among its breaking findings; until then it ships as a candidate, and the CLI
-   treats the dependency as generic.
+4. A pack is verified only with ground truth from two public repositories, at least five
+   expected breaking sites across them, and no false positive among its breaking findings
+   (docs/packs.md has the behavior-only variant); until then it ships as a candidate, and
+   the CLI treats the dependency as generic. A rewrite no real site exercises is listed
+   as unexercised and runs as assisted: give each mechanical rule at least one real site.
 5. Server API changes (Stripe API versions) are never rewritten by rule.
 
 ## Write a pack
@@ -123,8 +125,9 @@ open a **Pack request** issue, and say you are writing it.
    (`// @uptide <rule>`, `keep` for one it must leave alone) and `after.ts`, what the rules
    must produce. `pnpm uptide pack test <package> --fixtures-only` runs them in seconds;
    `--update-fixtures` writes each `after.ts` from what the rules do, for you to read.
-5. **Find ground truth.** Two public repositories that made this upgrade in a commit you
-   can point at. Pin the commit before it, and draft the expected findings from their own
+5. **Find ground truth.** Two or more public repositories that made this upgrade in a
+   commit you can point at, with at least five breaking sites between them and a real site
+   for every rule that rewrites. Pin the commit before it, and draft the expected findings from their own
    compiler and their own upgrade commit:
    ```sh
    pnpm packs:truth <package> owner/name <commit-before> <upgrade-commit> --to <version>
@@ -144,6 +147,8 @@ open a **Pack request** issue, and say you are writing it.
    A false negative is a site to cover or a known gap to say in the pull request. A site
    expected under a behavior note scores at the note's `severity`, and behavior-note sites
    get their own precision and recall: a missed "needs a look" is not a missed break.
+   The `gate` line says the class, the real site counts, and why the pack is a candidate if
+   it is one; rules it lists as unexercised run as assisted until a real site covers them.
 7. **Smoke-test locally** without editing `verification.json`:
    ```sh
    pnpm uptide check <package> --allow-candidate --cwd /path/to/local-fixture

README.md

@@ -50,26 +50,26 @@ Everything else: [getting started](docs/getting-started.md).
 A pack is everything Uptide knows about one upgrade: rules, a guide for the agent, behavior
 checks, and ground truth from public repositories it is scored against. Precision and recall
 are for breaking findings on that ground truth, as recorded by `uptide pack test`; a pack is
-verified with two or more public repositories and no false positive. The contract:
-[migration packs](docs/packs.md).
+verified with two or more public repositories, five real breaking sites, and no false positive.
+The contract: [migration packs](docs/packs.md).
 
 <!-- packs:start -->
-| Package | Range | Precision | Recall | Ground-truth repositories | Status |
-| --- | --- | ---: | ---: | --- | --- |
-| `ai` | 6.x → 7.x | 100% | 82% | vercel/chatbot, miurla/morphic | verified |
-| `ioredis` | 5.x → 6.x | n/a | 0% | ferni-ai/voiceai, syzygyhack/open-foundry, simondietschi/dokunc, safe-global/safe-client-gateway, kazumadev619-dev/FishingConditionsApp | candidate |
-| `joi` | 17.x → 18.x | n/a | n/a | edtech4good/edtech-lms-api, senacor/azure-function-middleware | candidate |
-| `lucide-react` | 0.x → 1.x | 100% | 100% | tradeblocks-org/tradeblocks, spizeck/SFPCA, cubicsteak/keep | verified |
-| `next` | 15.x → 16.x | 100% | 85% | sambecker/exif-photo-blog, stoczec/wortkarte | verified |
-| `react` | 18.x → 19.x | 100% | 69% | excalidraw/excalidraw, tldraw/tldraw | verified |
-| `react-router` | 6.x → 7.x | 100% | 88% | Arize-ai/phoenix, getlago/lago-front | verified |
-| `redis` | >=4 <6 → >=5 <7 | n/a | n/a | deanmoses/tacocat-gallery-sam, swoofer/mcp-coordinator, deanmoses/tacocat-gallery-sam, UNN-Devotek/FCM-Fallout-Chat-Mod, tearingItUp786/blog | candidate |
-| `@sentry/browser` | 10.x → 11.x | 100% | 100% | Automancer-Ltd/automancer-site, sbomify/sbomify | verified |
-| `@sentry/nextjs` | 10.x → 11.x | 100% | 100% | cohstats/coh3-stats, xdoubleu/tools.xdoubleu.com, maximunited/yomu | verified |
-| `@sentry/react` | 10.x → 11.x | 100% | 100% | owine/youth-activity-scheduler, successible/cleanslate | verified |
-| `stripe` | 14 and newer | 100% | 90% | unkeyed/unkey, nextjs/saas-starter | verified |
-| `vitest` | 4.x → 5.x | 100% | 50% | votingworks/vxsuite, vitorvasc/opentelemetry-ecosystem-explorer | verified |
-| `zod` | 3.x → 4.x | 100% | 74% | ugurkocde/AwesomeIntune, howardyang2009/PATH, band-ai/band-sdk-typescript | verified |
+| Package | Range | Precision | Recall | Breaking sites | Ground-truth repositories | Status |
+| --- | --- | ---: | ---: | ---: | --- | --- |
+| `ai` | 6.x → 7.x | 100% | 82% | 11 | vercel/chatbot, miurla/morphic | verified |
+| `ioredis` | 5.x → 6.x | n/a | 0% | 2 | ferni-ai/voiceai, syzygyhack/open-foundry, simondietschi/dokunc, safe-global/safe-client-gateway, kazumadev619-dev/FishingConditionsApp | candidate |
+| `joi` | 17.x → 18.x | n/a | n/a | — | edtech4good/edtech-lms-api, senacor/azure-function-middleware | candidate |
+| `lucide-react` | 0.x → 1.x | 100% | 100% | 11 | tradeblocks-org/tradeblocks, spizeck/SFPCA, cubicsteak/keep | verified |
+| `next` | 15.x → 16.x | 100% | 85% | 13 | sambecker/exif-photo-blog, stoczec/wortkarte | verified |
+| `react` | 18.x → 19.x | 100% | 69% | 130 | excalidraw/excalidraw, tldraw/tldraw | verified |
+| `react-router` | 6.x → 7.x | 100% | 88% | 18 | Arize-ai/phoenix, getlago/lago-front | verified |
+| `redis` | >=4 <6 → >=5 <7 | n/a | n/a | — | deanmoses/tacocat-gallery-sam, swoofer/mcp-coordinator, deanmoses/tacocat-gallery-sam, UNN-Devotek/FCM-Fallout-Chat-Mod, tearingItUp786/blog | candidate |
+| `@sentry/browser` | 10.x → 11.x | 100% | 100% | 3 | Automancer-Ltd/automancer-site, sbomify/sbomify | candidate |
+| `@sentry/nextjs` | 10.x → 11.x | 100% | 100% | 8 | cohstats/coh3-stats, xdoubleu/tools.xdoubleu.com, maximunited/yomu | verified |
+| `@sentry/react` | 10.x → 11.x | 100% | 100% | 2 | owine/youth-activity-scheduler, successible/cleanslate | candidate |
+| `stripe` | 14 and newer | 100% | 90% | 16 | unkeyed/unkey, nextjs/saas-starter | verified |
+| `vitest` | 4.x → 5.x | 100% | 50% | 32 | votingworks/vxsuite, vitorvasc/opentelemetry-ecosystem-explorer | verified |
+| `zod` | 3.x → 4.x | 100% | 74% | 85 | ugurkocde/AwesomeIntune, howardyang2009/PATH, band-ai/band-sdk-typescript | verified |
 <!-- packs:end -->
 
 Any other dependency gets the same analysis without a pack, and `fix` migrates it with the

docs/packs.md

@@ -297,9 +297,27 @@ measures verified`; one whose numbers merely differ passes with a note (`--write
 
 ## Verified or candidate
 
-A pack is **verified** only with ground truth from at least two public repositories and no
-false positive among its breaking findings there (precision 100%), with at least one
-breaking finding predicted. Anything else is a **candidate**:
+A pack is **verified** only with ground truth from at least two public repositories, no
+false positive among its breaking findings there (precision 100%), at least one breaking
+finding predicted, and at least five expected breaking sites across the public repositories
+(fixture repositories never count). An upgrade the compiler cannot see, where every
+expected site is a behavior note's (ioredis 6: new runtime defaults), is judged on its notes
+instead, whatever their severity: at least five expected note sites, one found, and no false positive on a note or a
+mechanical rule. `pack test` prints the gate's class, the site counts and every reason a
+pack is a candidate.
+
+A rule with a `rewrite` that no public repository's ground truth exercises (no true
+positive) is **unexercised**: `pack test` lists it and records it in `verification.json`
+(`unexercised`), because nothing real confirms its rewrite yet. `check` and `fix` read the record:
+its sites are assisted (the agent, with the rule's guide), never rewritten. Detection is
+unaffected, and so is the pack's status. A pack whose `transform` edits for a rule that has no
+`rewrite` of its own (Stripe's pin) lists that rule in `mechanical`, so the gate counts it too.
+
+A pack may ask for fewer real breaking sites with `meta.minBreakingSites`, only for a major whose
+guide lists fewer breaking changes, and only with the reason written under
+`## Breaking-site minimum` in its RESEARCH.md; `pack test` fails a pack that sets it without one.
+
+Anything else is a **candidate**:
 
 - it ships in the tree, and `pack test` scores it;
 - by default, `check`, `list` and `fix` treat the dependency as generic: no `verified` label, the
@@ -312,7 +330,8 @@ breaking finding predicted. Anything else is a **candidate**:
 The CLI reads the status from `verification.json`, which `pack test --write` writes and
 which records the digest of the ground-truth file it was measured against, with precision,
 recall and the counts behind them for breaking findings and for behavior-note sites
-(`behavior`, absent from records written before it). The gate reads the breaking ones only. CI runs
+(`behavior`, absent from records written before it), the gate's class, the real breaking and
+behavior-note site counts, and the unexercised rules. CI runs
 `pack test` for every pack on every pull request that touches packs and fails when the
 record claims verified where the evidence measures candidate, so the label cannot outlive
 what supports it.

packages/cli/smoke/fixtures/npm-workspaces/packages/api/src/signup.ts

@@ -1,8 +1,10 @@
 import { z } from 'zod';
 
 export const signup = z.object({
-  email: z.string({ required_error: 'Email is required' }),
-  plan: z.enum(['free', 'pro'], { invalid_type_error: 'Unknown plan' }),
+  email: z.string().min(3),
+  plan: z.enum(['free', 'pro']),
+  metadata: z.record(z.string()),
+  limits: z.record(z.number()),
 });
 
 export type Signup = z.infer<typeof signup>;

packages/cli/smoke/fixtures/npm-workspaces/packages/shared/src/index.ts

@@ -1,11 +1,9 @@
 import { z } from 'zod';
 
 export const user = z.object({
-  name: z.string({
-    required_error: 'Name is required',
-    invalid_type_error: 'Name must be a string',
-  }),
-  age: z.number({ invalid_type_error: 'Age must be a number' }).int(),
+  name: z.string(),
+  labels: z.record(z.string()),
+  scores: z.record(z.number().int()),
 });
 
 export type User = z.infer<typeof user>;

packages/cli/smoke/fixtures/npm/src/schema.ts

@@ -1,11 +1,9 @@
 import { z } from 'zod';
 
 export const user = z.object({
-  name: z.string({
-    required_error: 'Name is required',
-    invalid_type_error: 'Name must be a string',
-  }),
-  age: z.number({ invalid_type_error: 'Age must be a number' }).int(),
+  name: z.string(),
+  labels: z.record(z.string()),
+  scores: z.record(z.number().int()),
 });
 
 export type User = z.infer<typeof user>;

packages/cli/smoke/fixtures/pnpm/packages/api/src/signup.ts

@@ -1,8 +1,10 @@
 import { z } from 'zod';
 
 export const signup = z.object({
-  email: z.string({ required_error: 'Email is required' }),
-  plan: z.enum(['free', 'pro'], { invalid_type_error: 'Unknown plan' }),
+  email: z.string().min(3),
+  plan: z.enum(['free', 'pro']),
+  metadata: z.record(z.string()),
+  limits: z.record(z.number()),
 });
 
 export type Signup = z.infer<typeof signup>;

packages/cli/smoke/fixtures/pnpm/packages/shared/src/index.ts

@@ -1,11 +1,9 @@
 import { z } from 'zod';
 
 export const user = z.object({
-  name: z.string({
-    required_error: 'Name is required',
-    invalid_type_error: 'Name must be a string',
-  }),
-  age: z.number({ invalid_type_error: 'Age must be a number' }).int(),
+  name: z.string(),
+  labels: z.record(z.string()),
+  scores: z.record(z.number().int()),
 });
 
 export type User = z.infer<typeof user>;

packages/cli/smoke/fixtures/yarn-berry/src/schema.ts

@@ -1,11 +1,9 @@
 import { z } from 'zod';
 
 export const user = z.object({
-  name: z.string({
-    required_error: 'Name is required',
-    invalid_type_error: 'Name must be a string',
-  }),
-  age: z.number({ invalid_type_error: 'Age must be a number' }).int(),
+  name: z.string(),
+  labels: z.record(z.string()),
+  scores: z.record(z.number().int()),
 });
 
 export type User = z.infer<typeof user>;

packages/cli/smoke/fixtures/yarn/src/schema.ts

@@ -1,11 +1,9 @@
 import { z } from 'zod';
 
 export const user = z.object({
-  name: z.string({
-    required_error: 'Name is required',
-    invalid_type_error: 'Name must be a string',
-  }),
-  age: z.number({ invalid_type_error: 'Age must be a number' }).int(),
+  name: z.string(),
+  labels: z.record(z.string()),
+  scores: z.record(z.number().int()),
 });
 
 export type User = z.infer<typeof user>;

packages/cli/smoke/in-container.mjs

@@ -149,7 +149,13 @@ for (const fixture of ['npm', 'npm-workspaces', 'pnpm', 'yarn', 'yarn-berry']) {
     check.out,
     /zod\s+3\.23\.8 → 4\.6\.5\s+major · (?:--target|latest on npm)\s+verified\s+✗ \d+ breaking/,
   );
-  has(`${manager} check`, check.out, 'New error API (required_error → error)');
+  // zod's record-key rewrite: mechanical, exercised by real ground truth (unlike required_error,
+  // which UPT-48 runs as assisted until a real site covers it).
+  has(
+    `${manager} check`,
+    check.out,
+    'z.record(valueSchema) requires an explicit string key schema',
+  );
   matches(
     `${manager} check`,
     check.out,
@@ -308,7 +314,7 @@ for (const fixture of ['npm', 'npm-workspaces', 'pnpm', 'yarn', 'yarn-berry']) {
   for (const file of files)
     expect(
       `${fixture}: ${file} migrated`,
-      !/required_error|invalid_type_error/.test(onBranch(repo, zodBranch, file)),
+      !/z\.record\((?!z\.string\(\), )/.test(onBranch(repo, zodBranch, file)),
     );
   if (fixture === 'npm-workspaces') {
     expect(

packages/cli/src/pack.ts

@@ -159,6 +159,17 @@ export function formatPackTest(report: PackTestReport, color = false): string {
     );
   }
   out.push('');
+  if (!report.fixturesOnly && report.repos.length > 0) {
+    const g = report.gate;
+    out.push(
+      `gate          ${g.class === 'behavior' ? 'behavior-only, ' : ''}${g.breakingSites} breaking site${g.breakingSites === 1 ? '' : 's'}, ${g.behaviorSites} behavior-note site${g.behaviorSites === 1 ? '' : 's'} in public ground truth · ${g.status}`,
+    );
+    for (const reason of g.reasons) out.push(`  ${c.yellow('·')} ${reason}`);
+    if (g.unexercised.length)
+      out.push(
+        `  ${c.yellow('·')} no real site exercises ${g.unexercised.join(', ')}: their rewrites run as assisted`,
+      );
+  }
   if (report.overclaimed)
     out.push(
       `${c.red('✗')} verification.json records verified; this run measures candidate: \`uptide pack test ${report.package} --write\` records it`,

packages/cli/src/readme.test.ts

@@ -73,7 +73,7 @@ describe('README', () => {
   it('lists the verified packs from a generated table, and privacy in four lines plus the table', () => {
     const packs = section('<!-- packs:start -->', '<!-- packs:end -->');
     expect(packs).toContain(
-      '| Package | Range | Precision | Recall | Ground-truth repositories | Status |',
+      '| Package | Range | Precision | Recall | Breaking sites | Ground-truth repositories | Status |',
     );
     for (const name of ['`zod`', '`stripe`', '`ai`']) expect(packs).toContain(`| ${name} |`);
     const privacy = section('## Privacy', '## Documentation');

packages/core/src/check/check.ts

@@ -434,7 +434,13 @@ export async function check(opts: CheckOptions): Promise<CheckResult> {
     for (const report of results.flat()) {
       // Packs whose rules carry their own rewrite (zod, and every pack built on the contract).
       const pack = packsOf(opts).find((p) => p.name === report.name);
-      if (!pack || !(pack.rules as readonly { rewrite?: unknown }[]).some((r) => r.rewrite))
+      if (
+        !pack ||
+        !(
+          (pack.rules as readonly { rewrite?: unknown }[]).some((r) => r.rewrite) ||
+          pack.unexercised?.length
+        )
+      )
         continue;
       const sources = new Map<string, string>();
       report.findings = report.findings.map((finding) => {

packages/core/src/fix/fix.test.ts

@@ -25,6 +25,13 @@ import {
 } from './verify.js';
 import { bumpCatalog, bumpVersions } from './versions.js';
 
+// zod's required_error rewrite has no real ground-truth site yet, so the registry runs it as
+// assisted (UPT-48). These tests exercise the fix machinery with that rule as written.
+vi.mock('../packs/fixability.js', async (importOriginal) => ({
+  ...(await importOriginal<typeof import('../packs/fixability.js')>()),
+  withUnexercised: (pack: unknown) => pack,
+}));
+
 const scratch = mkdtempSync(join(tmpdir(), 'uptide-fix-tests-'));
 afterAll(() => rmSync(scratch, { recursive: true, force: true }));
 const fixture = () => zodFixture(scratch);

packages/core/src/fix/isolate.test.ts

@@ -30,6 +30,13 @@ import { command, git, quietEnv } from './process.js';
 import { zodFixture } from './test-fixture.js';
 import type { FixRequest } from './types.js';
 
+// zod's required_error rewrite has no real ground-truth site yet, so the registry runs it as
+// assisted (UPT-48). These tests exercise the fix machinery with that rule as written.
+vi.mock('../packs/fixability.js', async (importOriginal) => ({
+  ...(await importOriginal<typeof import('../packs/fixability.js')>()),
+  withUnexercised: (pack: unknown) => pack,
+}));
+
 const scratch = realpathSync(mkdtempSync(join(tmpdir(), 'uptide-isolate-')));
 // Clones made by these tests live under the scratch directory, not in the machine's temporary root.
 process.env.UPTIDE_RUNS_DIR = join(scratch, 'runs');

packages/core/src/index.ts

@@ -177,6 +177,7 @@ export * from './onboarding/node.js';
 export {
   type BehaviorNote,
   definePack,
+  type GateResult,
   type GroundTruth,
   type GroundTruthFinding,
   type GroundTruthRepo,
@@ -188,8 +189,11 @@ export {
   type RegisteredPack,
   rateLabel,
   replaceAtSite,
+  VERIFIED_MIN_BEHAVIOR_SITES,
+  VERIFIED_MIN_BREAKING_SITES,
   VERIFIED_MIN_REPOS,
   type VerificationTally,
+  verifiedGate,
 } from './packs/contract.js';
 export { activePacks, packStatus, registeredPacks } from './packs/index.js';
 export { leadName, leadNames, packLeading } from './packs/leads.js';

packages/core/src/packs/ai/verification.json

@@ -15,5 +15,9 @@
     "falsePositives": 0,
     "truePositives": 0,
     "falseNegatives": 0
-  }
+  },
+  "class": "breaking",
+  "breakingSites": 11,
+  "behaviorSites": 0,
+  "unexercised": []
 }