fix(packs): allow safe local candidate smoke tests (UPT-51)

.changeset/local-candidate-smoke.md

@@ -0,0 +1,5 @@
+---
+"uptide": patch
+---
+
+Allow local check and fix smoke tests of registered candidate packs with --allow-candidate, with an explicit unverified banner. Reject candidate opt-in in the Action and with publication flags, and prevent stored candidate runs from being published. Label pack-claimed assisted sites by rule id even with --no-llm.

CONTRIBUTING.md

@@ -140,7 +140,16 @@ open a **Pack request** issue, and say you are writing it.
    ```
    Every false positive among breaking findings fails it. Fix the rule, not the ground truth.
    A false negative is a site to cover or a known gap to say in the pull request.
-7. **Open the pull request** with the pack template (`?template=pack.md`), with the
+7. **Smoke-test locally** without editing `verification.json`:
+   ```sh
+   pnpm uptide check <package> --allow-candidate --cwd /path/to/local-fixture
+   pnpm uptide fix <package> --allow-candidate --no-llm --cwd /path/to/local-fixture
+   ```
+   Only registered packs with a recorded `candidate` status qualify. The output, stored
+   report and PR body preview say **candidate pack, not verified**. The Action rejects this
+   flag, as do `--pr` and `--yes`; candidate runs cannot be published later with `uptide pr`.
+   Verified packs behave as usual. Do not mark a pack verified to smoke-test it.
+8. **Open the pull request** with the pack template (`?template=pack.md`), with the
    `pack test` output pasted in, and a changeset.
 
 `pnpm uptide` runs the CLI from source, so nothing needs building while you work.

README.md

@@ -153,8 +153,8 @@ The full statement and the isolation model: [privacy](docs/privacy.md) and [SECU
 
 ## Contributing
 
-Setup, the rules that do not bend, how to write a migration pack and how to sign off:
-[CONTRIBUTING.md](CONTRIBUTING.md). Security problems go through [SECURITY.md](SECURITY.md).
+[CONTRIBUTING.md](CONTRIBUTING.md) covers setup, safety, pack authoring and sign-off. Smoke-test packs with `check --allow-candidate` or `fix --allow-candidate`: reports say “candidate pack, not verified”.
+The flag is local only: the Action, `--pr` and `--yes` reject it. Security problems: [SECURITY.md](SECURITY.md).
 
 ## License
 

docs/commands/check.md

@@ -137,3 +137,12 @@ an npm package, without a repository. Tarballs, surfaces and runtime probes are
 changes found at your call sites; **2** bad arguments, unsupported repository, no network, or
 nothing breaking was found but a dependency failed to analyze (the report still lists the
 others). See [concepts](../concepts.md#exit-codes).
+
+### Candidate pack smoke tests
+
+`--allow-candidate` opts in to registered packs whose verification record says `candidate`.
+The output and report carry “candidate pack, not verified”; verified packs are unchanged,
+and unregistered or blocked packs keep their normal behavior. This flag is for local pack
+development only: the Action rejects it, and `fix` refuses it with `--pr` or `--yes`.
+The generated PR body is a local preview; a stored candidate run cannot be published.
+See [Write a pack](../../CONTRIBUTING.md#write-a-pack).

docs/commands/fix.md

@@ -296,3 +296,12 @@ root are ever removed.
 
 **0** migration verified: no new type errors, tests pass; **1** verification failed or sites
 remain for manual work; **2** uptide could not answer. See [concepts](../concepts.md#exit-codes).
+
+### Candidate pack smoke tests
+
+`--allow-candidate` opts in to registered packs whose verification record says `candidate`.
+The output and report carry “candidate pack, not verified”; verified packs are unchanged,
+and unregistered or blocked packs keep their normal behavior. This flag is for local pack
+development only: the Action rejects it, and `fix` refuses it with `--pr` or `--yes`.
+The generated PR body is a local preview; a stored candidate run cannot be published.
+See [Write a pack](../../CONTRIBUTING.md#write-a-pack).

docs/packs.md

@@ -269,8 +269,12 @@ false positive among its breaking findings there (precision 100%), with at least
 breaking finding predicted. Anything else is a **candidate**:
 
 - it ships in the tree, and `pack test` scores it;
-- `check`, `list` and `fix` treat the dependency as generic: no `verified` label, the
+- by default, `check`, `list` and `fix` treat the dependency as generic: no `verified` label, the
   generic tier's evidence rule, the agent alone in `fix`.
+- local `check --allow-candidate` and `fix --allow-candidate` opt in only to registered packs
+  with an explicit candidate record. Reports and the PR body preview say **candidate pack,
+  not verified**. The Action, `--pr` and `--yes` reject this flag; stored candidate runs
+  cannot be published. Never edit `verification.json` just to smoke-test a pack.
 
 The CLI reads the status from `verification.json`, which `pack test --write` writes and
 which records the digest of the ground-truth file it was measured against. CI runs
@@ -280,7 +284,7 @@ record no longer matches the evidence, so the label cannot outlive what supports
 ## The registry, the queue, CI
 
 - `packages/core/src/packs/registry.ts` is generated: `pack new` renders it again with the
-  new entry. Every pack in it is scored by CI; only verified ones are used by the CLI.
+  new entry. Every pack in it is scored by CI; only verified ones are used by default.
 - `packs/queue.json` ranks the next packages worth a pack (`pnpm packs:queue`) by direct use
   × log2(1 + breaking changes). Direct use is how many sample repositories
   (`fixtures/corpus.json` and the pinned public applications in

packages/cli/src/alternate-lead.test.ts

@@ -13,7 +13,19 @@ vi.mock('@uptide/core', async (original) => {
     releaseGroup: [{ name: 'router-dom', source }],
     alternateLeads: [{ name: 'router-dom', source }],
   });
-  return { ...core, activePacks: () => [...core.activePacks(), router] };
+  const candidate = {
+    ...router,
+    name: 'candidate-router',
+    alternateLeads: [{ name: 'candidate-dom', source }],
+  };
+  return {
+    ...core,
+    activePacks: (allowCandidate = false) => [
+      ...core.activePacks(allowCandidate),
+      router,
+      ...(allowCandidate ? [candidate] : []),
+    ],
+  };
 });
 
 /** Declares `router-dom` only, as most React Router 6 applications declare `react-router-dom`. */
@@ -57,3 +69,33 @@ describe('a pack lead asked for, or declared, through its alternate lead', () =>
     expect(io.stderr()).toContain('router is not a dependency');
   });
 });
+
+it.each(['check', 'fix'])(
+  'candidate alternate leads pass dependency preflight for %s',
+  async (command) => {
+    const e = fakeEngine({
+      installed: async () => new Map([['candidate-dom', '1.0.0']]),
+      declared: async () => new Map([['candidate-dom', '^1.0.0']]),
+    });
+    const cwd = pnpmGitRepo({
+      'package.json': JSON.stringify({
+        name: 'shop',
+        packageManager: 'pnpm@10.17.1',
+        dependencies: { 'candidate-dom': '^1.0.0' },
+      }),
+      'node_modules/candidate-dom/package.json': '{"name":"candidate-dom","version":"1.0.0"}',
+    });
+    const io = memoryIo({ cwd });
+    expect(
+      await run(
+        [command, 'candidate-dom', '--allow-candidate', ...(command === 'fix' ? ['--no-llm'] : [])],
+        io,
+        e,
+      ),
+    ).toBe(0);
+    expect(e.calls[0]).toMatchObject({
+      allowCandidate: true,
+      only: command === 'check' ? ['candidate-router'] : 'candidate-router',
+    });
+  },
+);

packages/cli/src/cli.test.ts

@@ -895,3 +895,37 @@ it('exits 1 when pr refuses a tree that was not verified', async () => {
   expect(await run(['pr', '--branch', 'uptide/zod-4.6.5', '--yes'], io, engine)).toBe(1);
   expect(io.stderr()).toContain('nothing pushed');
 });
+
+it.each([['--pr'], ['--yes'], ['--pr', '--yes']])(
+  'refuses --allow-candidate with publication flags %j before the engine',
+  async (...flags) => {
+    const engine = fakeEngine();
+    const io = memoryIo({ cwd: '/does-not-exist' });
+    expect(await run(['fix', 'zod', '--allow-candidate', ...flags], io, engine)).toBe(2);
+    expect(io.stderr()).toContain('--allow-candidate cannot be used with --pr or --yes');
+    expect(engine.calls).toEqual([]);
+  },
+);
+
+it('forwards the local candidate opt-in for check and fix', async () => {
+  const engine = fakeEngine();
+  await run(['check', 'zod', '--allow-candidate'], memoryIo({ cwd: npmRepo() }), engine);
+  await run(
+    ['fix', 'zod', '--allow-candidate', '--no-llm'],
+    memoryIo({ cwd: pnpmGitRepo() }),
+    engine,
+  );
+  expect(engine.calls).toHaveLength(2);
+  expect(engine.calls).toEqual([
+    expect.objectContaining({ allowCandidate: true }),
+    expect.objectContaining({ allowCandidate: true }),
+  ]);
+});
+
+it.each(['check', 'fix'])('rejects --allow-candidate in GitHub Actions for %s', async (command) => {
+  const engine = fakeEngine();
+  const io = memoryIo({ cwd: '/does-not-exist', env: { GITHUB_ACTIONS: 'true' } });
+  expect(await run([command, 'zod', '--allow-candidate'], io, engine)).toBe(2);
+  expect(io.stderr()).toContain('--allow-candidate is not available in the Uptide Action');
+  expect(engine.calls).toEqual([]);
+});

packages/cli/src/cli.ts

@@ -239,9 +239,10 @@ export async function run(
   const withAlternates = async (
     repo: Repo,
     deps: Awaited<ReturnType<typeof locateDependencies>>,
+    allowCandidate = false,
   ): Promise<Awaited<ReturnType<typeof locateDependencies>>> => {
     const alternates = (name: string) =>
-      (packLeading(name, activePacks())?.alternateLeads ?? []).map((a) => a.name);
+      (packLeading(name, activePacks(allowCandidate))?.alternateLeads ?? []).map((a) => a.name);
     const wanted = deps.filter((d) => d.workspaces.length === 0).flatMap((d) => alternates(d.name));
     if (!wanted.length) return deps;
     const found = await locateDependencies(repo, engine, wanted);
@@ -258,6 +259,7 @@ export async function run(
     repo: Repo,
     only: string[] | undefined,
     progress: Progress,
+    allowCandidate = false,
   ): Promise<unknown> =>
     progress.phase(
       'Dependencies',
@@ -270,7 +272,7 @@ export async function run(
         if (only) {
           // Asked for by name and absent is an error, not an empty answer, unless a pack's
           // alternate lead is declared instead (`react-router-dom` for `react-router`).
-          const present = await withAlternates(repo, deps);
+          const present = await withAlternates(repo, deps, allowCandidate);
           const absent = present.filter((d) => d.workspaces.length === 0).map((d) => d.name);
           if (absent.length > 0) throw notADependency(repo, absent);
           requireInstalled(repo, present);
@@ -453,6 +455,7 @@ ${PRIVACY}`,
   shared(
     program
       .command('check [packages...]')
+      .option('--allow-candidate', 'use registered candidate packs for local smoke tests only')
       .description('Which of your call sites an upgrade breaks, and what fixing them costs')
       .option('--only <packages>', 'deprecated alias for positional package names')
       .option('--group <name>', 'check every member of a group shown by uptide list')
@@ -504,6 +507,7 @@ ${EXIT_CODES(
         flags: Shared & {
           only?: string;
           group?: string;
+          allowCandidate?: boolean;
           target?: string[];
           html?: string | true;
           open?: boolean;
@@ -520,14 +524,16 @@ ${EXIT_CODES(
         act(
           flags,
           async ({ ui, progress, cwd }) => {
+            if (flags.allowCandidate && io.env.GITHUB_ACTIONS === 'true')
+              throw new CliError('--allow-candidate is not available in the Uptide Action');
             if (flags.open && !flags.html) throw new CliError('--open requires --html');
             const started = io.now();
             const quiet = !flags.verbose;
             const requested = [...new Set([...packages, ...(list(flags.only) ?? [])])];
             // An alternate lead's name (`react-router-dom`) asks for its pack's lead.
             const only = requested.includes('all')
               ? requested
-              : leadNames(requested, activePacks());
+              : leadNames(requested, activePacks(flags.allowCandidate));
             if ((!flags.group && only.length === 0) || only.includes('all'))
               throw new CliError(
                 'check requires one or more package names. Start with uptide list.',
@@ -582,7 +588,7 @@ ${EXIT_CODES(
             // Named after what was asked for; an alternate lead's target is its lead's.
             const targets = Object.fromEntries(
               Object.entries(parseTargets(flags.target, [...new Set([...requested, ...only])])).map(
-                ([name, version]) => [leadName(name, activePacks()), version],
+                ([name, version]) => [leadName(name, activePacks(flags.allowCandidate)), version],
               ),
             );
             const htmlReport = async (report: CheckReport): Promise<void> => {
@@ -628,7 +634,7 @@ ${EXIT_CODES(
             };
             // Without a terminal there is no live line: say what started, then how long it took.
             if (quiet && !ui.interactive) io.err(`uptide check · ${repoLine(headerOf(repo, 0))}\n`);
-            await requireDependencies(repo, only, progress);
+            await requireDependencies(repo, only, progress, flags.allowCandidate);
             const report = await progress.phase(
               only
                 ? `Analysis of ${only.join(', ')}`
@@ -644,6 +650,7 @@ ${EXIT_CODES(
                       compile: flags.compile ?? true,
                       runtime: flags.runtime ?? true,
                       allDeps: flags.allDeps,
+                      ...(flags.allowCandidate ? { allowCandidate: true } : {}),
                       ...(flags.workspaces
                         ? { workspaceConcurrency: Number(flags.workspaces) }
                         : {}),
@@ -855,6 +862,10 @@ ${EXIT_CODES('a plan was made', 'not used', ';\n     or a dependency failed to a
         ).choices(['patch', 'sdk']),
       )
       .option('--debug', 'keep the temporary clone and print every phase (--keep-temp --verbose)')
+      .option(
+        '--allow-candidate',
+        'use registered candidate packs locally; never with --pr or --yes',
+      )
       .option('--pr', 'open a draft PR after verification passes; requires --yes')
       .option(
         '--base <branch>',
@@ -907,6 +918,7 @@ ${PRIVACY}`,
           only?: string;
           target?: string;
           allowPeer?: string[];
+          allowCandidate?: boolean;
           includeDeprecated?: boolean;
           pinCurrentApi?: boolean;
           llm?: boolean;
@@ -930,6 +942,10 @@ ${PRIVACY}`,
           flags,
           async ({ ui, progress, cwd }) => {
             const started = io.now();
+            if (flags.allowCandidate && (flags.pr || flags.yes))
+              throw new CliError('--allow-candidate cannot be used with --pr or --yes');
+            if (flags.allowCandidate && io.env.GITHUB_ACTIONS === 'true')
+              throw new CliError('--allow-candidate is not available in the Uptide Action');
             const quiet = !flags.verbose && !flags.debug;
             const loaded = settings(cwd, flags.config);
             if (packageName && flags.only && packageName !== flags.only)
@@ -946,7 +962,7 @@ ${PRIVACY}`,
             namedOutside(loaded, [requested]);
             // An alternate lead's name (`react-router-dom`) fixes its pack's lead, and a target
             // named after the alternate is the lead's.
-            const only = leadName(requested, activePacks());
+            const only = leadName(requested, activePacks(flags.allowCandidate));
             const target =
               only === requested
                 ? flags.target
@@ -967,7 +983,7 @@ ${PRIVACY}`,
               });
             // No pack, so no rule: without the agent there is nothing this command can do,
             // and it says so before it touches anything.
-            const hasPack = activePacks().some((pack) => pack.name === only);
+            const hasPack = activePacks(flags.allowCandidate).some((pack) => pack.name === only);
             if (!hasPack && (!llmOn || !selection.available)) throw noAgentForGeneric(only, !llmOn);
             if (flags.pinCurrentApi && only !== 'stripe')
               throw new CliError('--pin-current-api applies to stripe only', {
@@ -993,6 +1009,7 @@ ${PRIVACY}`,
                 const deps = await withAlternates(
                   found,
                   await locateDependencies(found, engine, [only]),
+                  flags.allowCandidate,
                 );
                 if (deps.every((d) => d.workspaces.length === 0))
                   throw notADependency(found, [requested]);
@@ -1071,6 +1088,7 @@ ${PRIVACY}`,
                     only,
                     target: target,
                     includeDeprecated: flags.includeDeprecated,
+                    ...(flags.allowCandidate ? { allowCandidate: true } : {}),
                     ...(peers.length ? { also: peers } : {}),
                     ...(flags.allowPeer?.length ? { allowPeer: flags.allowPeer } : {}),
                     ...(flags.pinCurrentApi ? { pinCurrentApi: true } : {}),

packages/cli/src/format-check.test.ts

@@ -639,3 +639,13 @@ it('says which importing files the memory cap left out, per workspace, in the su
   expect(details).toContain('  apps/web/src/sdk.ts');
   expect(formatCheck(storefront, shown)).not.toContain('memory cap');
 });
+
+it.each([false, true])('shows the candidate banner in check output (details=%s)', (details) => {
+  const report = structuredClone(storefront);
+  const first = report.packages[0];
+  if (!first) throw new Error('missing fixture package');
+  first.packWarning = 'candidate pack, not verified';
+  expect(formatCheck(report, { ...shown, details })).toContain(
+    `${first.name}: candidate pack, not verified`,
+  );
+});

packages/cli/src/format-check.ts

@@ -483,6 +483,8 @@ export function formatCheck(report: CheckReport, opts: FormatCheckOptions = {}):
     lines.push(terminalHeader('check', opts.header, opts.color ?? true), '');
   }
   const rows = checkRows(report, colors);
+  for (const p of report.packages)
+    if (p.packWarning) lines.push(colors.yellow(`${p.name}: ${p.packWarning}`), '');
   if (opts.details) {
     lines.push(formatCheckDetails(report, { color: opts.color, all: opts.all }).trimEnd(), '');
   } else {
@@ -503,7 +505,10 @@ export function formatCheck(report: CheckReport, opts: FormatCheckOptions = {}):
             text: move(row),
             tone: row.bump.startsWith('major') ? ('yellow' as const) : ('dim' as const),
           },
-          { text: row.p.tier === 'verified' ? 'verified' : '', tone: 'green' as const },
+          {
+            text: row.p.packWarning ? 'candidate' : row.p.tier === 'verified' ? 'verified' : '',
+            tone: 'green' as const,
+          },
           { text: row.verdict.replace(COLOR, '') },
           { text: row.by, tone: 'dim' as const },
         ]),

packages/cli/src/jobs.ts

@@ -22,9 +22,11 @@ export type CheckRequest = Pick<
   | 'allDeps'
   | 'workspaceConcurrency'
   | 'maxTimeMs'
+  | 'allowCandidate'
 > & { checkResults?: CheckReport };
 
 export interface FixRequest {
+  allowCandidate?: boolean;
   provider?: string;
   model?: string;
   cwd: string;