From 2a3b16e0186587ac321d5c1be086417d36fbca36 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 20:04:59 +0000 Subject: [PATCH 1/3] fix(packs): score expected sites at their rule's or note's severity `pack test` counted every missed site as a breaking false negative unless a *rule* with that id was deprecated: `BehaviorNote.severity` was never read, so missed behavior-note sites that only need a look (ioredis `client-defaults`) were reported as missed breaks. An expected site now takes the severity of the rule or note it is expected under: a note's own `severity`, breaking when it reports findings without one (as `check` reports them), and `behavior` for a note `check` never reports. Behavior-note sites get their own precision and recall, in the output and in verification.json (`behavior`); both tallies record the counts behind their rates, so a zero denominator prints `n/a` in `pack test` and the README table instead of 100%. False negatives print their severity. Older records still read as before. Fixes UPT-52. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011t9c3ZLsM9B3KiYGPFDyyK Signed-off-by: Claude --- .changeset/pack-test-behavior-notes.md | 5 + CONTRIBUTING.md | 4 +- docs/packs.md | 27 ++- packages/cli/src/pack.test.ts | 51 ++++++ packages/cli/src/pack.ts | 15 +- packages/core/src/index.ts | 2 + packages/core/src/packs/contract.ts | 34 +++- .../src/packs/tooling/behavior-score.test.ts | 164 ++++++++++++++++++ packages/core/src/packs/tooling/pack-test.ts | 142 ++++++++++----- packages/core/src/packs/tooling/scaffold.ts | 17 +- scripts/docs-packs.ts | 5 +- 11 files changed, 402 insertions(+), 64 deletions(-) create mode 100644 .changeset/pack-test-behavior-notes.md create mode 100644 packages/cli/src/pack.test.ts create mode 100644 packages/core/src/packs/tooling/behavior-score.test.ts diff --git a/.changeset/pack-test-behavior-notes.md b/.changeset/pack-test-behavior-notes.md new file mode 100644 index 0000000..1bb97c0 --- /dev/null +++ b/.changeset/pack-test-behavior-notes.md @@ -0,0 +1,5 @@ +--- +"uptide": patch +--- + +`uptide pack test` scores each expected site at the severity of the rule or behavior note it belongs to, so a missed note that only needs a look, or a decision, no longer counts as a missed break. Behavior-note sites get their own precision and recall in the output and in `verification.json` (`behavior`, with the counts behind each rate), false negatives show their severity, and a rate with nothing to measure prints `n/a` instead of 100%, in `pack test` and in the README's packs table. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6cbf3ce..ac2ec04 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -139,7 +139,9 @@ open a **Pack request** issue, and say you are writing it. pnpm uptide pack test --write # records the status in verification.json ``` Every false positive among breaking findings fails it. Fix the rule, not the ground truth. - A false negative is a site to cover or a known gap to say in the pull request. + A false negative is a site to cover or a known gap to say in the pull request. A site + expected under a behavior note scores at the note's `severity`, and behavior-note sites + get their own precision and recall: a missed "needs a look" is not a missed break. 7. **Smoke-test locally** without editing `verification.json`: ```sh pnpm uptide check --allow-candidate --cwd /path/to/local-fixture diff --git a/docs/packs.md b/docs/packs.md index 1ad07b7..9bd8b01 100644 --- a/docs/packs.md +++ b/docs/packs.md @@ -255,10 +255,21 @@ findings: - **per rule**: a site counts for a rule when file, line and rule all match; - **overall**: at the line level, every severity; +- **behavior notes**: at the line level, the sites of the pack's behavior notes, whatever + their severity; - **breaking**: at the line level, breaking findings only. -It prints precision and recall for each, and every false positive and false negative as -`repo file:line rule`, and sites found under another rule. It exits non-zero on any false +An expected site has the severity of the rule or note it is expected under: a rule's +`severity`; a note's `severity`, or breaking when the note reports findings without one (as +`check` reports them); `behavior` for a note `check` never reports as a finding (`decision`, +`test-follow-up`); breaking for `generic`. A predicted site has the severity `check` reported. +A missed site counts against breaking recall only when its severity is breaking: a missed +`deprecated` note ("needs a look") or a decision is not a missed break. + +It prints precision and recall for each, `n/a` where there was nothing to measure (nothing +predicted, or nothing expected), every false positive as `repo file:line rule (severity)`, +every false negative the same way with the severity it was expected at, and sites found under +another rule. It exits non-zero on any false positive among breaking findings, any fixture failure, or a `verification.json` that no longer matches what it measured (`--write` records it). @@ -277,7 +288,9 @@ breaking finding predicted. Anything else is a **candidate**: cannot be published. Never edit `verification.json` just to smoke-test a pack. The CLI reads the status from `verification.json`, which `pack test --write` writes and -which records the digest of the ground-truth file it was measured against. CI runs +which records the digest of the ground-truth file it was measured against, with precision, +recall and the counts behind them for breaking findings and for behavior-note sites +(`behavior`, absent from records written before it). The gate reads the breaking ones only. CI runs `pack test` for every pack on every pull request that touches packs and fails when the record no longer matches the evidence, so the label cannot outlive what supports it. @@ -318,14 +331,16 @@ pnpm uptide pack test ai --write # record the measured status in verific pnpm uptide pack test ai --fixtures-only --update-fixtures # write after.ts from the rules ``` -`pack test` prints precision and recall per rule, overall and for breaking findings, and -every false positive and false negative as `repository file:line rule`. Exit codes: **0** +`pack test` prints precision and recall per rule, overall, for behavior-note sites and for +breaking findings, and every false positive and false negative as +`repository file:line rule (severity)`. Exit codes: **0** every pack passed, **1** a false positive among breaking findings, a failing fixture, or a `verification.json` the run does not support, **2** it could not run (not in a checkout, bad arguments). The "Verified packs" table in the README is generated from each pack's `verification.json` -and ground truth by `pnpm docs:packs`; CI fails when it is stale. +and ground truth by `pnpm docs:packs`; CI fails when it is stale. Its precision and recall are +the breaking ones, `n/a` when nothing breaking was predicted or expected. ### Config files and transactional edits diff --git a/packages/cli/src/pack.test.ts b/packages/cli/src/pack.test.ts new file mode 100644 index 0000000..658e3bb --- /dev/null +++ b/packages/cli/src/pack.test.ts @@ -0,0 +1,51 @@ +import { type PackTestReport, registeredPacks, unfinishedPack } from '@uptide/core'; +import { expect, it } from 'vitest'; +import { formatPackTest } from './pack.js'; + +it('prints behavior notes on their own row, n/a where nothing was predicted, and each miss with its severity', () => { + const entry = registeredPacks()[0]; + if (!entry) throw new Error('no registered pack'); + const tally = (tp: number, fp: number, fn: number) => ({ + truePositives: tp, + falsePositives: fp, + falseNegatives: fn, + precision: tp + fp ? tp / (tp + fp) : 1, + recall: tp + fn ? tp / (tp + fn) : 1, + }); + const site = (line: number, rule: string, severity: 'breaking' | 'deprecated') => ({ + file: 'src/client.ts', + line, + rule, + severity, + }); + const report: PackTestReport = { + ...unfinishedPack(entry, ''), + problems: [], + repos: [ + { + label: 'owner/name', + from: '5.0.0', + to: '6.0.0', + predicted: [site(13, 'client-defaults', 'deprecated')], + expected: [ + site(9, 'client-defaults', 'deprecated'), + site(20, 'handshake-auth', 'breaking'), + ], + falsePositives: [site(13, 'client-defaults', 'deprecated')], + falseNegatives: [ + site(9, 'client-defaults', 'deprecated'), + site(20, 'handshake-auth', 'breaking'), + ], + wrongRule: [], + }, + ], + overall: tally(0, 1, 2), + breaking: tally(0, 0, 1), + behavior: tally(0, 1, 2), + }; + const out = formatPackTest(report); + expect(out).toMatch(/behavior notes \(sites\)\s+0%\s+0%\s+0\s+1\s+2/); + expect(out).toMatch(/breaking \(sites\)\s+n\/a\s+0%\s+0\s+0\s+1/); + expect(out).toContain('src/client.ts:9 client-defaults (deprecated)'); + expect(out).toContain('src/client.ts:20 handshake-auth (breaking)'); +}); diff --git a/packages/cli/src/pack.ts b/packages/cli/src/pack.ts index 6977a0b..f201205 100644 --- a/packages/cli/src/pack.ts +++ b/packages/cli/src/pack.ts @@ -1,6 +1,6 @@ import { existsSync, readFileSync } from 'node:fs'; import { dirname, join } from 'node:path'; -import type { PackTestReport, RegisteredPack, Tally } from '@uptide/core'; +import { type PackTestReport, type RegisteredPack, rateLabel, type Tally } from '@uptide/core'; import pc from 'picocolors'; import { CliError } from './errors.js'; @@ -48,10 +48,8 @@ export function packsToTest( return [entry]; } -const pct = (n: number): string => `${Math.round(n * 100)}%`; - function row(name: string, t: Tally): string { - return ` ${name.padEnd(28)} ${pct(t.precision).padStart(9)} ${pct(t.recall).padStart(7)} ${String(t.truePositives).padStart(5)} ${String(t.falsePositives).padStart(4)} ${String(t.falseNegatives).padStart(4)}`; + return ` ${name.padEnd(28)} ${rateLabel(t, 'precision').padStart(9)} ${rateLabel(t, 'recall').padStart(7)} ${String(t.truePositives).padStart(5)} ${String(t.falsePositives).padStart(4)} ${String(t.falseNegatives).padStart(4)}`; } /** What `uptide pack test` prints for one pack. */ @@ -109,6 +107,8 @@ export function formatPackTest(report: PackTestReport, color = false): string { for (const [rule, t] of Object.entries(report.rules).sort(([a], [b]) => a.localeCompare(b))) out.push(row(rule, t)); out.push(row('overall (sites)', report.overall)); + // Absent from a report written before behavior notes were scored on their own. + if (report.behavior) out.push(row('behavior notes (sites)', report.behavior)); out.push(c.bold(row('breaking (sites)', report.breaking))); const listed = ( title: string, @@ -133,7 +133,12 @@ export function formatPackTest(report: PackTestReport, color = false): string { listed( 'false negatives', report.repos.flatMap((r) => - r.falseNegatives.map((s) => ({ label: r.label, file: s.file, line: s.line, text: s.rule })), + r.falseNegatives.map((s) => ({ + label: r.label, + file: s.file, + line: s.line, + text: s.severity ? `${s.rule} (${s.severity})` : s.rule, + })), ), ); listed( diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 0060ee0..fdcb298 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -186,8 +186,10 @@ export { type PackStatus, type PackVerification, type RegisteredPack, + rateLabel, replaceAtSite, VERIFIED_MIN_REPOS, + type VerificationTally, } from './packs/contract.js'; export { activePacks, packStatus, registeredPacks } from './packs/index.js'; export { leadName, leadNames, packLeading } from './packs/leads.js'; diff --git a/packages/core/src/packs/contract.ts b/packages/core/src/packs/contract.ts index 774b0e9..968d2cf 100644 --- a/packages/core/src/packs/contract.ts +++ b/packages/core/src/packs/contract.ts @@ -157,7 +157,39 @@ export interface PackVerification { truth: string; /** Public repositories in the ground truth (fixtures excluded). */ repos: number; - breaking: { precision: number; recall: number; falsePositives: number }; + breaking: VerificationTally; + /** Sites of behavior notes, whatever their severity. Absent from records written before it. */ + behavior?: VerificationTally; +} + +/** + * Precision and recall as recorded. A ratio with nothing to measure reads 1; the counts, absent + * from records written before they were kept, tell that apart from a perfect score. + */ +export interface VerificationTally { + precision: number; + recall: number; + falsePositives: number; + truePositives?: number; + falseNegatives?: number; +} + +/** A run's tally as `verification.json` records it. */ +export function verificationTally(t: Required): Required { + return { + precision: Number(t.precision.toFixed(4)), + recall: Number(t.recall.toFixed(4)), + falsePositives: t.falsePositives, + truePositives: t.truePositives, + falseNegatives: t.falseNegatives, + }; +} + +/** A precision or recall for people: `n/a` when nothing was predicted, or nothing expected. */ +export function rateLabel(tally: VerificationTally, rate: 'precision' | 'recall'): string { + const missed = rate === 'precision' ? tally.falsePositives : tally.falseNegatives; + if (tally.truePositives === 0 && missed === 0) return 'n/a'; + return `${Math.round(tally[rate] * 100)}%`; } /** diff --git a/packages/core/src/packs/tooling/behavior-score.test.ts b/packages/core/src/packs/tooling/behavior-score.test.ts new file mode 100644 index 0000000..ae7f147 --- /dev/null +++ b/packages/core/src/packs/tooling/behavior-score.test.ts @@ -0,0 +1,164 @@ +import { describe, expect, it } from 'vitest'; +import { definePack, rateLabel } from '../contract.js'; +import { type ScoredSite, scoreSites, scoreTruth, siteSeverity } from './pack-test.js'; + +// Synthetic, modeled on the ioredis 5 → 6 ground truth (PR 129): every expected site belongs to a +// behavior note. `client-defaults` only needs a look (deprecated); `handshake-auth` is declared +// breaking; `node-minimum` is a decision `check` never reports as a site. +const pack = definePack({ + meta: { + package: 'redis-like', + from: '>=5 <6', + to: '>=6 <7', + sources: [{ title: 'guide', url: 'https://example.com/guide' }], + maintainer: 'uptide-dev', + }, + rules: [ + { + id: 'duplicate-signature', + summary: 'duplicate() became generic', + severity: 'breaking', + kinds: ['signature'], + symbols: /^TS2416$/, + guide: 'make the override generic too', + }, + ], + behavior: [ + { + id: 'client-defaults', + summary: 'a client keeps RESP3 and the new retry defaults', + reported: ['finding', 'decision'], + severity: 'deprecated', + detect: () => [], + }, + { + id: 'handshake-auth', + summary: 'AUTH moves into HELLO', + reported: ['finding'], + severity: 'breaking', + detect: () => [], + }, + { id: 'node-minimum', summary: 'Node 20 or newer', reported: ['decision'] }, + ], + instructions: '', +}); + +const at = (spec: string, rule: string, severity?: ScoredSite['severity']): ScoredSite => { + const [file, line] = spec.split(':') as [string, string]; + return { file, line: Number(line), rule, ...(severity ? { severity } : {}) }; +}; +const repo = (predicted: ScoredSite[], expected: ScoredSite[]) => ({ + predicted, + expected, + ...scoreSites(predicted, expected), +}); +const needsALook = (spec: string) => at(spec, 'client-defaults', 'deprecated'); + +/** The four scored repositories of the ioredis truth, with what the pack predicted there. */ +const repos = [ + // voiceai: five rebuilt constructions; two predicted at the construction, not the run's first + // line; two constructions changed by pure insertion, which are not sites. + repo( + ['src/cache.ts:101', 'src/cache.ts:108', 'src/jobs.ts:92', 'src/jobs.ts:98'] + .concat(['src/pubsub.ts:140', 'src/limiter.ts:57']) + .map(needsALook), + [ + 'src/cache.ts:98', + 'src/cache.ts:108', + 'src/jobs.ts:89', + 'src/jobs.ts:98', + 'src/vectors.ts:134', + ].map((s) => at(s, 'client-defaults')), + ), + // open-foundry: the truth anchors the rewritten acquisition, the pack the call below it. + repo( + [needsALook('packages/api/src/server.ts:138')], + [at('packages/api/src/server.ts:136', 'client-defaults')], + ), + // dokunc: one construction found; the redaction test it misses is a breaking note's. + repo( + [needsALook('apps/collab/src/server.ts:108'), needsALook('apps/web/src/lib/redis.ts:54')], + [ + at('apps/collab/src/server.ts:108', 'client-defaults'), + at('apps/web/src/lib/log.test.ts:20', 'handshake-auth'), + at('apps/web/src/lib/log.test.ts:28', 'handshake-auth'), + ], + ), + // A clean upgrade: the client it reports was left as it was, by choice. + repo([needsALook('src/lib/cache/client.ts:13')], []), +]; + +describe('behavior-note sites', () => { + it('count as breaking misses only when their note is declared breaking', () => { + const score = scoreTruth(pack, repos); + // Before: 6 breaking false negatives, every client-defaults miss among them. + expect(score.breaking).toMatchObject({ + truePositives: 0, + falsePositives: 0, + falseNegatives: 2, + }); + expect(score.overall).toMatchObject({ truePositives: 3, falsePositives: 7, falseNegatives: 6 }); + expect(score.rules['client-defaults']).toMatchObject({ + truePositives: 3, + falsePositives: 7, + falseNegatives: 4, + }); + }); + + it('are scored on their own: precision and recall of every note site', () => { + const { behavior } = scoreTruth(pack, repos); + expect(behavior).toEqual({ + truePositives: 3, + falsePositives: 7, + falseNegatives: 6, + precision: 0.3, + recall: 1 / 3, + }); + }); + + it('take the severity of the rule or note they are expected under', () => { + expect(siteSeverity(pack, 'duplicate-signature')).toBe('breaking'); + expect(siteSeverity(pack, 'client-defaults')).toBe('deprecated'); + expect(siteSeverity(pack, 'handshake-auth')).toBe('breaking'); + // Never reported as a finding: neither breaking nor deprecated. + expect(siteSeverity(pack, 'node-minimum')).toBe('behavior'); + // `check` reports a detected note without a severity as breaking; so does the scorer. + const unmarked = { ...pack, behavior: [{ ...pack.behavior[1], id: 'x', severity: undefined }] }; + expect(siteSeverity(unmarked as typeof pack, 'x')).toBe('breaking'); + expect(siteSeverity(pack, 'generic')).toBe('breaking'); + }); + + it('leave a decision-only note out of the breaking misses', () => { + const score = scoreTruth(pack, [repo([], [at('package.json:4', 'node-minimum')])]); + expect(score.breaking.falseNegatives).toBe(0); + expect(score.behavior.falseNegatives).toBe(1); + }); +}); + +describe('rateLabel', () => { + it('says n/a for a zero denominator, where the ratio reads 100%', () => { + const none = { + precision: 1, + recall: 0, + truePositives: 0, + falsePositives: 0, + falseNegatives: 2, + }; + expect(rateLabel(none, 'precision')).toBe('n/a'); + expect(rateLabel(none, 'recall')).toBe('0%'); + expect(rateLabel({ ...none, falseNegatives: 0, recall: 1 }, 'recall')).toBe('n/a'); + expect( + rateLabel( + { precision: 0.9, recall: 0.5, truePositives: 9, falsePositives: 1, falseNegatives: 9 }, + 'precision', + ), + ).toBe('90%'); + }); + + it('reads a record written before the counts were kept as it did', () => { + expect(rateLabel({ precision: 1, recall: 0.7375, falsePositives: 0 }, 'precision')).toBe( + '100%', + ); + expect(rateLabel({ precision: 1, recall: 0.7375, falsePositives: 0 }, 'recall')).toBe('74%'); + }); +}); diff --git a/packages/core/src/packs/tooling/pack-test.ts b/packages/core/src/packs/tooling/pack-test.ts index e0fe341..d5b682c 100644 --- a/packages/core/src/packs/tooling/pack-test.ts +++ b/packages/core/src/packs/tooling/pack-test.ts @@ -16,16 +16,21 @@ import { recordedStatus, statusOf, truthDigest, + verificationTally, } from '../contract.js'; import { type FixtureResult, fixturesPass, runFixtures } from './fixtures.js'; import { type EnsureOptions, ensureFixture, ensureRepo, label, truthProblems } from './truth.js'; -/** A site as scored: repository-relative file, line, the rule it belongs to, its severity. */ +/** + * A site as scored: repository-relative file, line, the rule it belongs to, its severity. A + * predicted site has the severity `check` reported; an expected one, its rule's or note's + * (`siteSeverity`), where `behavior` is a note `check` never reports as a finding. + */ export interface ScoredSite { file: string; line: number; rule: string; - severity?: 'breaking' | 'deprecated' | 'unverified'; + severity?: 'breaking' | 'deprecated' | 'unverified' | 'behavior'; } export interface Tally { @@ -83,6 +88,8 @@ export interface PackTestReport { overall: Tally; /** Ground truth, at the line level, breaking findings only: the verified gate. */ breaking: Tally; + /** Ground truth, at the line level, the sites of behavior notes, whatever their severity. */ + behavior: Tally; passed: boolean; } @@ -171,13 +178,87 @@ export function scoreSites( }; } +/** + * The severity of an expected site: its rule's, or its note's as `check` reports the note's + * findings (breaking unless the note says otherwise). A note `check` never reports as a finding + * is `behavior`; a site no rule or note covers (`generic`) is breaking. + */ +export function siteSeverity( + pack: Pick, + rule: string, +): 'breaking' | 'deprecated' | 'behavior' { + const own = pack.rules.find((r) => r.id === rule); + if (own) return own.severity; + const note = pack.behavior.find((b) => b.id === rule); + if (!note) return 'breaking'; + if (note.severity) return note.severity; + return note.detect && note.reported.includes('finding') ? 'breaking' : 'behavior'; +} + +/** + * Every scored repository together: per rule, at the line level, for breaking findings, and + * for the sites of behavior notes. + */ +export function scoreTruth( + pack: Pick, + scored: Pick[], +): Pick { + const all = (k: K) => + scored.flatMap((r) => r[k]); + const fp = all('falsePositives'); + const fn = all('falseNegatives'); + const predicted = all('predicted'); + const overall = tally(predicted.length - fp.length, fp.length, fn.length); + const breakingPredicted = predicted.filter((p) => p.severity === 'breaking'); + const breakingFp = fp.filter((p) => p.severity === 'breaking'); + // A miss is a breaking one when the rule or note it was expected under is. + const breaking = tally( + breakingPredicted.length - breakingFp.length, + breakingFp.length, + fn.filter((f) => siteSeverity(pack, f.rule) === 'breaking').length, + ); + const notes = new Set(pack.behavior.map((b) => b.id)); + const ofNote = (s: ScoredSite) => notes.has(s.rule); + const noteFp = fp.filter(ofNote).length; + const behavior = tally( + predicted.filter(ofNote).length - noteFp, + noteFp, + fn.filter(ofNote).length, + ); + const rules: Record = {}; + const ids = new Set([ + ...pack.rules.map((r) => r.id), + ...pack.behavior.map((b) => b.id), + ...predicted.map((p) => p.rule), + ...all('expected').map((e) => e.rule), + ]); + for (const id of ids) { + let tp = 0; + let rfp = 0; + let rfn = 0; + for (const r of scored) { + const want = new Set(r.expected.filter((e) => e.rule === id).map(key)); + const got = new Set(r.predicted.filter((p) => p.rule === id).map(key)); + for (const k of got) (want.has(k) ? () => tp++ : () => rfp++)(); + for (const k of want) if (!got.has(k)) rfn++; + } + if (tp + rfp + rfn > 0) rules[id] = tally(tp, rfp, rfn); + } + return { rules, overall, breaking, behavior }; +} + async function scoreRepo( pack: Pack, entry: GroundTruthRepo, root: string, options: EnsureOptions, ): Promise { - const expected = entry.findings.map((f) => ({ file: f.file, line: f.line, rule: f.rule })); + const expected = entry.findings.map((f) => ({ + file: f.file, + line: f.line, + rule: f.rule, + severity: siteSeverity(pack, f.rule), + })); const base: RepoScore = { label: label(entry), ...(entry.repo ? { repo: entry.repo } : {}), @@ -295,58 +376,25 @@ export async function testPack( if (!options.fixturesOnly && problems.length === 0) for (const repo of truth.repos) repos.push(await scoreRepo(pack, repo, options.root, options)); - const scored = repos.filter((r) => !r.error); - const all = (k: K) => - scored.flatMap((r) => r[k]); - const fp = all('falsePositives'); - const fn = all('falseNegatives'); - const predicted = all('predicted'); - const overall = tally(predicted.length - fp.length, fp.length, fn.length); - const breakingPredicted = predicted.filter((p) => p.severity === 'breaking'); - const breakingFp = fp.filter((p) => p.severity === 'breaking'); - // A miss is a breaking one unless the rule it was expected under is deprecated. - const deprecatedRules = new Set( - pack.rules.filter((r) => r.severity === 'deprecated').map((r) => r.id), + const { rules, overall, breaking, behavior } = scoreTruth( + pack, + repos.filter((r) => !r.error), ); - const breaking = tally( - breakingPredicted.length - breakingFp.length, - breakingFp.length, - fn.filter((f) => !deprecatedRules.has(f.rule)).length, - ); - const rules: Record = {}; - const ids = new Set([ - ...pack.rules.map((r) => r.id), - ...pack.behavior.map((b) => b.id), - ...predicted.map((p) => p.rule), - ...all('expected').map((e) => e.rule), - ]); - for (const id of ids) { - let tp = 0; - let rfp = 0; - let rfn = 0; - for (const r of scored) { - const want = new Set(r.expected.filter((e) => e.rule === id).map(key)); - const got = new Set(r.predicted.filter((p) => p.rule === id).map(key)); - for (const k of got) (want.has(k) ? () => tp++ : () => rfp++)(); - for (const k of want) if (!got.has(k)) rfn++; - } - if (tp + rfp + rfn > 0) rules[id] = tally(tp, rfp, rfn); - } // A pack whose upgrade leaves part of the group behind produces an inconsistent install. const status: PackStatus = repos.some((r) => r.error || r.leftBehind?.length) || options.fixturesOnly ? 'candidate' - : statusOf(truth, { falsePositives: breakingFp.length, predicted: breakingPredicted.length }); + : statusOf(truth, { + falsePositives: breaking.falsePositives, + predicted: breaking.truePositives + breaking.falsePositives, + }); const verification: PackVerification = { status, truth: read ? truthDigest(read.text) : '', repos: publicRepos(truth).length, - breaking: { - precision: Number(breaking.precision.toFixed(4)), - recall: Number(breaking.recall.toFixed(4)), - falsePositives: breakingFp.length, - }, + breaking: verificationTally(breaking), + behavior: verificationTally(behavior), }; const recorded = recordedStatus(entry.verification); // Fixtures alone say nothing about ground truth: only a full run can find the record stale. @@ -360,7 +408,7 @@ export async function testPack( const passed = scoredAll && fixturesPass(fixtures) && - breakingFp.length === 0 && + breaking.falsePositives === 0 && !repos.some((r) => r.leftBehind?.length) && !stale; return { @@ -377,6 +425,7 @@ export async function testPack( rules, overall, breaking, + behavior, passed, }; } @@ -415,6 +464,7 @@ export function unfinishedPack(entry: RegisteredPack, reason: string): PackTestR rules: {}, overall: none, breaking: none, + behavior: none, passed: false, }; } diff --git a/packages/core/src/packs/tooling/scaffold.ts b/packages/core/src/packs/tooling/scaffold.ts index a7545ef..aad2ffa 100644 --- a/packages/core/src/packs/tooling/scaffold.ts +++ b/packages/core/src/packs/tooling/scaffold.ts @@ -3,7 +3,12 @@ import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; import { join, relative } from 'node:path'; import { UptideError } from '../../errors.js'; import { minVersion } from '../../fetch/range.js'; -import { type GroundTruth, type PackVerification, truthDigest } from '../contract.js'; +import { + type GroundTruth, + type PackVerification, + truthDigest, + verificationTally, +} from '../contract.js'; import { packsDir } from './pack-test.js'; import { packConstant, packDir, parseRegistry, renderRegistry } from './registry-file.js'; @@ -148,11 +153,19 @@ export function scaffoldPack(options: ScaffoldOptions): ScaffoldResult { }).status === 0; // What `pack test` records for a pack with no ground-truth repository yet, tied to the // ground-truth file as formatted: the first run finds it current. + const nothing = { + precision: 1, + recall: 1, + truePositives: 0, + falsePositives: 0, + falseNegatives: 0, + }; const verification: PackVerification = { status: 'candidate', truth: truthDigest(readFileSync(join(home, 'ground-truth.json'), 'utf8')), repos: 0, - breaking: { precision: 1, recall: 1, falsePositives: 0 }, + breaking: verificationTally(nothing), + behavior: verificationTally(nothing), }; const verificationFile = join(home, 'verification.json'); writeFileSync(verificationFile, `${JSON.stringify(verification, null, 2)}\n`); diff --git a/scripts/docs-packs.ts b/scripts/docs-packs.ts index 4c53133..aab8eb6 100644 --- a/scripts/docs-packs.ts +++ b/scripts/docs-packs.ts @@ -9,6 +9,7 @@ import { readFileSync, writeFileSync } from 'node:fs'; import { join } from 'node:path'; import { fileURLToPath } from 'node:url'; +import { rateLabel } from '../packages/core/src/packs/contract.ts'; import { registeredPacks } from '../packages/core/src/packs/index.ts'; const root = fileURLToPath(new URL('..', import.meta.url)); @@ -29,8 +30,6 @@ function rangeLabel(range: string): string { return range; } -const percent = (n: number): string => `${Math.round(n * 100)}%`; - export function packsTable(): string { const rows = registeredPacks().map(({ dir, pack, verification }) => { const truth = JSON.parse( @@ -41,7 +40,7 @@ export function packsTable(): string { pack.meta.from === pack.meta.to ? rangeLabel(pack.meta.from) : `${rangeLabel(pack.meta.from)} → ${rangeLabel(pack.meta.to)}`; - return `| \`${pack.meta.package}\` | ${range} | ${percent(verification.breaking.precision)} | ${percent(verification.breaking.recall)} | ${repos.join(', ')} | ${verification.status} |`; + return `| \`${pack.meta.package}\` | ${range} | ${rateLabel(verification.breaking, 'precision')} | ${rateLabel(verification.breaking, 'recall')} | ${repos.join(', ')} | ${verification.status} |`; }); return [ '| Package | Range | Precision | Recall | Ground-truth repositories | Status |', From 34bc36187a3e3932242bb18b2f4ce012f5ae64a0 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 22:28:52 +0000 Subject: [PATCH 2/3] chore(packs): record behavior-note tallies and the counts behind each rate `uptide pack test --write` for every registered pack, against the same ground truth: each verification.json gains `behavior` and the true-positive and false-negative counts under `breaking`. Status, truth digest, repositories, and breaking precision, recall and false positives are unchanged for all ten packs. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011t9c3ZLsM9B3KiYGPFDyyK Signed-off-by: Claude --- packages/core/src/packs/ai/verification.json | 11 ++++++++++- .../core/src/packs/lucide-react/verification.json | 11 ++++++++++- packages/core/src/packs/next/verification.json | 11 ++++++++++- packages/core/src/packs/react/verification.json | 11 ++++++++++- .../core/src/packs/sentry-browser/verification.json | 11 ++++++++++- .../core/src/packs/sentry-nextjs/verification.json | 11 ++++++++++- .../core/src/packs/sentry-react/verification.json | 11 ++++++++++- packages/core/src/packs/stripe/verification.json | 11 ++++++++++- packages/core/src/packs/vitest/verification.json | 11 ++++++++++- packages/core/src/packs/zod/verification.json | 11 ++++++++++- 10 files changed, 100 insertions(+), 10 deletions(-) diff --git a/packages/core/src/packs/ai/verification.json b/packages/core/src/packs/ai/verification.json index 21cdaa4..32312cd 100644 --- a/packages/core/src/packs/ai/verification.json +++ b/packages/core/src/packs/ai/verification.json @@ -5,6 +5,15 @@ "breaking": { "precision": 1, "recall": 0.8182, - "falsePositives": 0 + "falsePositives": 0, + "truePositives": 9, + "falseNegatives": 2 + }, + "behavior": { + "precision": 1, + "recall": 1, + "falsePositives": 0, + "truePositives": 0, + "falseNegatives": 0 } } diff --git a/packages/core/src/packs/lucide-react/verification.json b/packages/core/src/packs/lucide-react/verification.json index c809202..12a9308 100644 --- a/packages/core/src/packs/lucide-react/verification.json +++ b/packages/core/src/packs/lucide-react/verification.json @@ -5,6 +5,15 @@ "breaking": { "precision": 1, "recall": 1, - "falsePositives": 0 + "falsePositives": 0, + "truePositives": 11, + "falseNegatives": 0 + }, + "behavior": { + "precision": 1, + "recall": 1, + "falsePositives": 0, + "truePositives": 0, + "falseNegatives": 0 } } diff --git a/packages/core/src/packs/next/verification.json b/packages/core/src/packs/next/verification.json index 876641d..abffd1c 100644 --- a/packages/core/src/packs/next/verification.json +++ b/packages/core/src/packs/next/verification.json @@ -5,6 +5,15 @@ "breaking": { "precision": 1, "recall": 0.8462, - "falsePositives": 0 + "falsePositives": 0, + "truePositives": 11, + "falseNegatives": 2 + }, + "behavior": { + "precision": 1, + "recall": 1, + "falsePositives": 0, + "truePositives": 0, + "falseNegatives": 0 } } diff --git a/packages/core/src/packs/react/verification.json b/packages/core/src/packs/react/verification.json index 49069a8..0842e0c 100644 --- a/packages/core/src/packs/react/verification.json +++ b/packages/core/src/packs/react/verification.json @@ -5,6 +5,15 @@ "breaking": { "precision": 1, "recall": 0.6899, - "falsePositives": 0 + "falsePositives": 0, + "truePositives": 89, + "falseNegatives": 40 + }, + "behavior": { + "precision": 1, + "recall": 1, + "falsePositives": 0, + "truePositives": 0, + "falseNegatives": 0 } } diff --git a/packages/core/src/packs/sentry-browser/verification.json b/packages/core/src/packs/sentry-browser/verification.json index 80a0d09..c7b547d 100644 --- a/packages/core/src/packs/sentry-browser/verification.json +++ b/packages/core/src/packs/sentry-browser/verification.json @@ -5,6 +5,15 @@ "breaking": { "precision": 1, "recall": 1, - "falsePositives": 0 + "falsePositives": 0, + "truePositives": 3, + "falseNegatives": 0 + }, + "behavior": { + "precision": 1, + "recall": 1, + "falsePositives": 0, + "truePositives": 0, + "falseNegatives": 0 } } diff --git a/packages/core/src/packs/sentry-nextjs/verification.json b/packages/core/src/packs/sentry-nextjs/verification.json index 12c3fef..5d3e3c6 100644 --- a/packages/core/src/packs/sentry-nextjs/verification.json +++ b/packages/core/src/packs/sentry-nextjs/verification.json @@ -5,6 +5,15 @@ "breaking": { "precision": 1, "recall": 1, - "falsePositives": 0 + "falsePositives": 0, + "truePositives": 8, + "falseNegatives": 0 + }, + "behavior": { + "precision": 1, + "recall": 1, + "falsePositives": 0, + "truePositives": 0, + "falseNegatives": 0 } } diff --git a/packages/core/src/packs/sentry-react/verification.json b/packages/core/src/packs/sentry-react/verification.json index 18d96ef..5994af5 100644 --- a/packages/core/src/packs/sentry-react/verification.json +++ b/packages/core/src/packs/sentry-react/verification.json @@ -5,6 +5,15 @@ "breaking": { "precision": 1, "recall": 1, - "falsePositives": 0 + "falsePositives": 0, + "truePositives": 2, + "falseNegatives": 0 + }, + "behavior": { + "precision": 1, + "recall": 1, + "falsePositives": 0, + "truePositives": 0, + "falseNegatives": 0 } } diff --git a/packages/core/src/packs/stripe/verification.json b/packages/core/src/packs/stripe/verification.json index 9036546..6774abc 100644 --- a/packages/core/src/packs/stripe/verification.json +++ b/packages/core/src/packs/stripe/verification.json @@ -5,6 +5,15 @@ "breaking": { "precision": 1, "recall": 0.9, - "falsePositives": 0 + "falsePositives": 0, + "truePositives": 18, + "falseNegatives": 2 + }, + "behavior": { + "precision": 1, + "recall": 1, + "falsePositives": 0, + "truePositives": 0, + "falseNegatives": 0 } } diff --git a/packages/core/src/packs/vitest/verification.json b/packages/core/src/packs/vitest/verification.json index 515ceeb..71f0b3c 100644 --- a/packages/core/src/packs/vitest/verification.json +++ b/packages/core/src/packs/vitest/verification.json @@ -5,6 +5,15 @@ "breaking": { "precision": 1, "recall": 0.5, - "falsePositives": 0 + "falsePositives": 0, + "truePositives": 16, + "falseNegatives": 16 + }, + "behavior": { + "precision": 1, + "recall": 0, + "falsePositives": 0, + "truePositives": 0, + "falseNegatives": 13 } } diff --git a/packages/core/src/packs/zod/verification.json b/packages/core/src/packs/zod/verification.json index 0570c61..6eb5ce7 100644 --- a/packages/core/src/packs/zod/verification.json +++ b/packages/core/src/packs/zod/verification.json @@ -5,6 +5,15 @@ "breaking": { "precision": 1, "recall": 0.7375, - "falsePositives": 0 + "falsePositives": 0, + "truePositives": 59, + "falseNegatives": 21 + }, + "behavior": { + "precision": 1, + "recall": 1, + "falsePositives": 0, + "truePositives": 0, + "falseNegatives": 0 } } From 9308233f569baacfbf8eda87561733e52c764d3e Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 22:28:52 +0000 Subject: [PATCH 3/3] docs(contributing): run pack test locally What a full `pnpm uptide pack test` run needs on a contributor's machine: corepack enabled for Yarn only (a blanket `corepack enable` shims the global pnpm too), COREPACK_NPM_REGISTRY where repo.yarnpkg.com is blocked, and per-pack cache directories evicted after each pack to bound disk. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011t9c3ZLsM9B3KiYGPFDyyK Signed-off-by: Claude --- CONTRIBUTING.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ac2ec04..8ea3097 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -156,6 +156,31 @@ open a **Pack request** issue, and say you are writing it. `pnpm uptide` runs the CLI from source, so nothing needs building while you work. +## Run pack test locally + +`pnpm uptide pack test` installs each ground-truth repository with the package manager its +lockfile names, scripts off. For a full run: + +- **Corepack for Yarn only.** Repositories that pin Yarn Berry in `packageManager` (tldraw, + `yarn@4.7.0`) fail under Yarn 1. Run `corepack enable yarn`, not `corepack enable`: that + also replaces your global pnpm with a shim, and tests that run pnpm outside this checkout + then try to download a pnpm of corepack's choosing. +- **Where Yarn 4 comes from.** Corepack fetches Yarn 2 and later from repo.yarnpkg.com. Where + that host is blocked (a proxy, an allowlist), fetch it from npm instead: + `export COREPACK_NPM_REGISTRY=https://registry.npmjs.org COREPACK_ENABLE_DOWNLOAD_PROMPT=0`. +- **Disk.** Each repository stays cached with its dependencies installed, 2 to 8 GB per pack. + To score every pack on a bounded disk, give each pack its own caches and delete them after: + ```sh + for p in ai zod; do # each pack you score + export UPTIDE_GROUND_TRUTH_CACHE=/tmp/uptide-truth/$p UPTIDE_CACHE_DIR=/tmp/uptide-cache/$p + pnpm uptide pack test $p + rm -rf "$UPTIDE_GROUND_TRUTH_CACHE" "$UPTIDE_CACHE_DIR" + done + ``` + Comparing two checkouts (a before and after table), score each pack in both before you + delete its caches. In-tree fixtures (`fixtures/repos/storefront`) are installed inside the + checkout itself, so run each checkout once without `--offline`. + ## Rules that do not bend These are what make Uptide safe to run on someone else's code. A pull request that breaks