fix(packs): score expected sites at their rule's or note's severity (UPT-52)

.changeset/pack-test-behavior-notes.md

@@ -0,0 +1,5 @@
+---
+"uptide": patch
+---
+
+`uptide pack test` scores each expected site at the severity of the rule or behavior note it belongs to, so a missed note that only needs a look, or a decision, no longer counts as a missed break. Behavior-note sites get their own precision and recall in the output and in `verification.json` (`behavior`, with the counts behind each rate), false negatives show their severity, and a rate with nothing to measure prints `n/a` instead of 100%, in `pack test` and in the README's packs table.

CONTRIBUTING.md

@@ -139,7 +139,9 @@ open a **Pack request** issue, and say you are writing it.
    pnpm uptide pack test <package> --write  # records the status in verification.json
    ```
    Every false positive among breaking findings fails it. Fix the rule, not the ground truth.
-   A false negative is a site to cover or a known gap to say in the pull request.
+   A false negative is a site to cover or a known gap to say in the pull request. A site
+   expected under a behavior note scores at the note's `severity`, and behavior-note sites
+   get their own precision and recall: a missed "needs a look" is not a missed break.
 7. **Smoke-test locally** without editing `verification.json`:
    ```sh
    pnpm uptide check <package> --allow-candidate --cwd /path/to/local-fixture
@@ -154,6 +156,31 @@ open a **Pack request** issue, and say you are writing it.
 
 `pnpm uptide` runs the CLI from source, so nothing needs building while you work.
 
+## Run pack test locally
+
+`pnpm uptide pack test` installs each ground-truth repository with the package manager its
+lockfile names, scripts off. For a full run:
+
+- **Corepack for Yarn only.** Repositories that pin Yarn Berry in `packageManager` (tldraw,
+  `yarn@4.7.0`) fail under Yarn 1. Run `corepack enable yarn`, not `corepack enable`: that
+  also replaces your global pnpm with a shim, and tests that run pnpm outside this checkout
+  then try to download a pnpm of corepack's choosing.
+- **Where Yarn 4 comes from.** Corepack fetches Yarn 2 and later from repo.yarnpkg.com. Where
+  that host is blocked (a proxy, an allowlist), fetch it from npm instead:
+  `export COREPACK_NPM_REGISTRY=https://registry.npmjs.org COREPACK_ENABLE_DOWNLOAD_PROMPT=0`.
+- **Disk.** Each repository stays cached with its dependencies installed, 2 to 8 GB per pack.
+  To score every pack on a bounded disk, give each pack its own caches and delete them after:
+  ```sh
+  for p in ai zod; do   # each pack you score
+    export UPTIDE_GROUND_TRUTH_CACHE=/tmp/uptide-truth/$p UPTIDE_CACHE_DIR=/tmp/uptide-cache/$p
+    pnpm uptide pack test $p
+    rm -rf "$UPTIDE_GROUND_TRUTH_CACHE" "$UPTIDE_CACHE_DIR"
+  done
+  ```
+  Comparing two checkouts (a before and after table), score each pack in both before you
+  delete its caches. In-tree fixtures (`fixtures/repos/storefront`) are installed inside the
+  checkout itself, so run each checkout once without `--offline`.
+
 ## Rules that do not bend
 
 These are what make Uptide safe to run on someone else's code. A pull request that breaks

docs/packs.md

@@ -255,10 +255,21 @@ findings:
 
 - **per rule**: a site counts for a rule when file, line and rule all match;
 - **overall**: at the line level, every severity;
+- **behavior notes**: at the line level, the sites of the pack's behavior notes, whatever
+  their severity;
 - **breaking**: at the line level, breaking findings only.
 
-It prints precision and recall for each, and every false positive and false negative as
-`repo  file:line  rule`, and sites found under another rule. It exits non-zero on any false
+An expected site has the severity of the rule or note it is expected under: a rule's
+`severity`; a note's `severity`, or breaking when the note reports findings without one (as
+`check` reports them); `behavior` for a note `check` never reports as a finding (`decision`,
+`test-follow-up`); breaking for `generic`. A predicted site has the severity `check` reported.
+A missed site counts against breaking recall only when its severity is breaking: a missed
+`deprecated` note ("needs a look") or a decision is not a missed break.
+
+It prints precision and recall for each, `n/a` where there was nothing to measure (nothing
+predicted, or nothing expected), every false positive as `repo  file:line  rule (severity)`,
+every false negative the same way with the severity it was expected at, and sites found under
+another rule. It exits non-zero on any false
 positive among breaking findings, any fixture failure, or a `verification.json` that no
 longer matches what it measured (`--write` records it).
 
@@ -277,7 +288,9 @@ breaking finding predicted. Anything else is a **candidate**:
   cannot be published. Never edit `verification.json` just to smoke-test a pack.
 
 The CLI reads the status from `verification.json`, which `pack test --write` writes and
-which records the digest of the ground-truth file it was measured against. CI runs
+which records the digest of the ground-truth file it was measured against, with precision,
+recall and the counts behind them for breaking findings and for behavior-note sites
+(`behavior`, absent from records written before it). The gate reads the breaking ones only. CI runs
 `pack test` for every pack on every pull request that touches packs and fails when the
 record no longer matches the evidence, so the label cannot outlive what supports it.
 
@@ -318,14 +331,16 @@ pnpm uptide pack test ai --write         # record the measured status in verific
 pnpm uptide pack test ai --fixtures-only --update-fixtures  # write after.ts from the rules
 ```
 
-`pack test` prints precision and recall per rule, overall and for breaking findings, and
-every false positive and false negative as `repository  file:line  rule`. Exit codes: **0**
+`pack test` prints precision and recall per rule, overall, for behavior-note sites and for
+breaking findings, and every false positive and false negative as
+`repository  file:line  rule (severity)`. Exit codes: **0**
 every pack passed, **1** a false positive among breaking findings, a failing fixture, or a
 `verification.json` the run does not support, **2** it could not run (not in a checkout, bad
 arguments).
 
 The "Verified packs" table in the README is generated from each pack's `verification.json`
-and ground truth by `pnpm docs:packs`; CI fails when it is stale.
+and ground truth by `pnpm docs:packs`; CI fails when it is stale. Its precision and recall are
+the breaking ones, `n/a` when nothing breaking was predicted or expected.
 
 ### Config files and transactional edits
 

packages/cli/src/pack.test.ts

@@ -0,0 +1,51 @@
+import { type PackTestReport, registeredPacks, unfinishedPack } from '@uptide/core';
+import { expect, it } from 'vitest';
+import { formatPackTest } from './pack.js';
+
+it('prints behavior notes on their own row, n/a where nothing was predicted, and each miss with its severity', () => {
+  const entry = registeredPacks()[0];
+  if (!entry) throw new Error('no registered pack');
+  const tally = (tp: number, fp: number, fn: number) => ({
+    truePositives: tp,
+    falsePositives: fp,
+    falseNegatives: fn,
+    precision: tp + fp ? tp / (tp + fp) : 1,
+    recall: tp + fn ? tp / (tp + fn) : 1,
+  });
+  const site = (line: number, rule: string, severity: 'breaking' | 'deprecated') => ({
+    file: 'src/client.ts',
+    line,
+    rule,
+    severity,
+  });
+  const report: PackTestReport = {
+    ...unfinishedPack(entry, ''),
+    problems: [],
+    repos: [
+      {
+        label: 'owner/name',
+        from: '5.0.0',
+        to: '6.0.0',
+        predicted: [site(13, 'client-defaults', 'deprecated')],
+        expected: [
+          site(9, 'client-defaults', 'deprecated'),
+          site(20, 'handshake-auth', 'breaking'),
+        ],
+        falsePositives: [site(13, 'client-defaults', 'deprecated')],
+        falseNegatives: [
+          site(9, 'client-defaults', 'deprecated'),
+          site(20, 'handshake-auth', 'breaking'),
+        ],
+        wrongRule: [],
+      },
+    ],
+    overall: tally(0, 1, 2),
+    breaking: tally(0, 0, 1),
+    behavior: tally(0, 1, 2),
+  };
+  const out = formatPackTest(report);
+  expect(out).toMatch(/behavior notes \(sites\)\s+0%\s+0%\s+0\s+1\s+2/);
+  expect(out).toMatch(/breaking \(sites\)\s+n\/a\s+0%\s+0\s+0\s+1/);
+  expect(out).toContain('src/client.ts:9  client-defaults (deprecated)');
+  expect(out).toContain('src/client.ts:20  handshake-auth (breaking)');
+});

packages/cli/src/pack.ts

@@ -1,6 +1,6 @@
 import { existsSync, readFileSync } from 'node:fs';
 import { dirname, join } from 'node:path';
-import type { PackTestReport, RegisteredPack, Tally } from '@uptide/core';
+import { type PackTestReport, type RegisteredPack, rateLabel, type Tally } from '@uptide/core';
 import pc from 'picocolors';
 import { CliError } from './errors.js';
 
@@ -48,10 +48,8 @@ export function packsToTest(
   return [entry];
 }
 
-const pct = (n: number): string => `${Math.round(n * 100)}%`;
-
 function row(name: string, t: Tally): string {
-  return `  ${name.padEnd(28)} ${pct(t.precision).padStart(9)} ${pct(t.recall).padStart(7)} ${String(t.truePositives).padStart(5)} ${String(t.falsePositives).padStart(4)} ${String(t.falseNegatives).padStart(4)}`;
+  return `  ${name.padEnd(28)} ${rateLabel(t, 'precision').padStart(9)} ${rateLabel(t, 'recall').padStart(7)} ${String(t.truePositives).padStart(5)} ${String(t.falsePositives).padStart(4)} ${String(t.falseNegatives).padStart(4)}`;
 }
 
 /** What `uptide pack test` prints for one pack. */
@@ -109,6 +107,8 @@ export function formatPackTest(report: PackTestReport, color = false): string {
     for (const [rule, t] of Object.entries(report.rules).sort(([a], [b]) => a.localeCompare(b)))
       out.push(row(rule, t));
     out.push(row('overall (sites)', report.overall));
+    // Absent from a report written before behavior notes were scored on their own.
+    if (report.behavior) out.push(row('behavior notes (sites)', report.behavior));
     out.push(c.bold(row('breaking (sites)', report.breaking)));
     const listed = (
       title: string,
@@ -133,7 +133,12 @@ export function formatPackTest(report: PackTestReport, color = false): string {
     listed(
       'false negatives',
       report.repos.flatMap((r) =>
-        r.falseNegatives.map((s) => ({ label: r.label, file: s.file, line: s.line, text: s.rule })),
+        r.falseNegatives.map((s) => ({
+          label: r.label,
+          file: s.file,
+          line: s.line,
+          text: s.severity ? `${s.rule} (${s.severity})` : s.rule,
+        })),
       ),
     );
     listed(

packages/core/src/index.ts

@@ -186,8 +186,10 @@ export {
   type PackStatus,
   type PackVerification,
   type RegisteredPack,
+  rateLabel,
   replaceAtSite,
   VERIFIED_MIN_REPOS,
+  type VerificationTally,
 } from './packs/contract.js';
 export { activePacks, packStatus, registeredPacks } from './packs/index.js';
 export { leadName, leadNames, packLeading } from './packs/leads.js';

packages/core/src/packs/ai/verification.json

@@ -5,6 +5,15 @@
   "breaking": {
     "precision": 1,
     "recall": 0.8182,
-    "falsePositives": 0
+    "falsePositives": 0,
+    "truePositives": 9,
+    "falseNegatives": 2
+  },
+  "behavior": {
+    "precision": 1,
+    "recall": 1,
+    "falsePositives": 0,
+    "truePositives": 0,
+    "falseNegatives": 0
   }
 }

packages/core/src/packs/contract.ts

@@ -157,7 +157,39 @@ export interface PackVerification {
   truth: string;
   /** Public repositories in the ground truth (fixtures excluded). */
   repos: number;
-  breaking: { precision: number; recall: number; falsePositives: number };
+  breaking: VerificationTally;
+  /** Sites of behavior notes, whatever their severity. Absent from records written before it. */
+  behavior?: VerificationTally;
+}
+
+/**
+ * Precision and recall as recorded. A ratio with nothing to measure reads 1; the counts, absent
+ * from records written before they were kept, tell that apart from a perfect score.
+ */
+export interface VerificationTally {
+  precision: number;
+  recall: number;
+  falsePositives: number;
+  truePositives?: number;
+  falseNegatives?: number;
+}
+
+/** A run's tally as `verification.json` records it. */
+export function verificationTally(t: Required<VerificationTally>): Required<VerificationTally> {
+  return {
+    precision: Number(t.precision.toFixed(4)),
+    recall: Number(t.recall.toFixed(4)),
+    falsePositives: t.falsePositives,
+    truePositives: t.truePositives,
+    falseNegatives: t.falseNegatives,
+  };
+}
+
+/** A precision or recall for people: `n/a` when nothing was predicted, or nothing expected. */
+export function rateLabel(tally: VerificationTally, rate: 'precision' | 'recall'): string {
+  const missed = rate === 'precision' ? tally.falsePositives : tally.falseNegatives;
+  if (tally.truePositives === 0 && missed === 0) return 'n/a';
+  return `${Math.round(tally[rate] * 100)}%`;
 }
 
 /**

packages/core/src/packs/lucide-react/verification.json

@@ -5,6 +5,15 @@
   "breaking": {
     "precision": 1,
     "recall": 1,
-    "falsePositives": 0
+    "falsePositives": 0,
+    "truePositives": 11,
+    "falseNegatives": 0
+  },
+  "behavior": {
+    "precision": 1,
+    "recall": 1,
+    "falsePositives": 0,
+    "truePositives": 0,
+    "falseNegatives": 0
   }
 }

packages/core/src/packs/next/verification.json

@@ -5,6 +5,15 @@
   "breaking": {
     "precision": 1,
     "recall": 0.8462,
-    "falsePositives": 0
+    "falsePositives": 0,
+    "truePositives": 11,
+    "falseNegatives": 2
+  },
+  "behavior": {
+    "precision": 1,
+    "recall": 1,
+    "falsePositives": 0,
+    "truePositives": 0,
+    "falseNegatives": 0
   }
 }

packages/core/src/packs/react/verification.json

@@ -5,6 +5,15 @@
   "breaking": {
     "precision": 1,
     "recall": 0.6899,
-    "falsePositives": 0
+    "falsePositives": 0,
+    "truePositives": 89,
+    "falseNegatives": 40
+  },
+  "behavior": {
+    "precision": 1,
+    "recall": 1,
+    "falsePositives": 0,
+    "truePositives": 0,
+    "falseNegatives": 0
   }
 }

packages/core/src/packs/sentry-browser/verification.json

@@ -5,6 +5,15 @@
   "breaking": {
     "precision": 1,
     "recall": 1,
-    "falsePositives": 0
+    "falsePositives": 0,
+    "truePositives": 3,
+    "falseNegatives": 0
+  },
+  "behavior": {
+    "precision": 1,
+    "recall": 1,
+    "falsePositives": 0,
+    "truePositives": 0,
+    "falseNegatives": 0
   }
 }

packages/core/src/packs/sentry-nextjs/verification.json

@@ -5,6 +5,15 @@
   "breaking": {
     "precision": 1,
     "recall": 1,
-    "falsePositives": 0
+    "falsePositives": 0,
+    "truePositives": 8,
+    "falseNegatives": 0
+  },
+  "behavior": {
+    "precision": 1,
+    "recall": 1,
+    "falsePositives": 0,
+    "truePositives": 0,
+    "falseNegatives": 0
   }
 }

packages/core/src/packs/sentry-react/verification.json

@@ -5,6 +5,15 @@
   "breaking": {
     "precision": 1,
     "recall": 1,
-    "falsePositives": 0
+    "falsePositives": 0,
+    "truePositives": 2,
+    "falseNegatives": 0
+  },
+  "behavior": {
+    "precision": 1,
+    "recall": 1,
+    "falsePositives": 0,
+    "truePositives": 0,
+    "falseNegatives": 0
   }
 }

packages/core/src/packs/stripe/verification.json

@@ -5,6 +5,15 @@
   "breaking": {
     "precision": 1,
     "recall": 0.9,
-    "falsePositives": 0
+    "falsePositives": 0,
+    "truePositives": 18,
+    "falseNegatives": 2
+  },
+  "behavior": {
+    "precision": 1,
+    "recall": 1,
+    "falsePositives": 0,
+    "truePositives": 0,
+    "falseNegatives": 0
   }
 }