diff --git a/.changeset/promote-typescript-express.md b/.changeset/promote-typescript-express.md new file mode 100644 index 00000000..532c3d14 --- /dev/null +++ b/.changeset/promote-typescript-express.md @@ -0,0 +1,5 @@ +--- +"uptide": minor +--- + +The typescript 5 to 6 and express 4 to 5 packs are now verified, so `check`, `list` and `fix` use them by default. Both measure verified under the stricter gate on unchanged ground truth: typescript across 7 public repositories (1,399 real breaking sites, no breaking false positive; its `downlevel-iteration`, `import-assertions` and `module-keyword` rewrites stay assisted until a real site exercises them), express across 4 (86 real breaking sites, no breaking false positive). diff --git a/.changeset/score-folded-evidence.md b/.changeset/score-folded-evidence.md new file mode 100644 index 00000000..92411bab --- /dev/null +++ b/.changeset/score-folded-evidence.md @@ -0,0 +1,5 @@ +--- +"uptide": patch +--- + +`uptide pack test` counts a ground-truth site that a folded finding lists as its evidence as covered by that finding, so recall reflects what `check` reports. The typescript pack's breaking recall goes from 17% to 99% on unchanged ground truth: its TypeScript 6 `types` default errors fold into one tsconfig finding each. diff --git a/README.md b/README.md index 1a990fed..42346382 100644 --- a/README.md +++ b/README.md @@ -57,7 +57,7 @@ The contract: [migration packs](docs/packs.md). | Package | Range | Precision | Recall | Breaking sites | Ground-truth repositories | Status | | --- | --- | ---: | ---: | ---: | --- | --- | | `ai` | 6.x → 7.x | 100% | 82% | 11 | vercel/chatbot, miurla/morphic | verified | -| `express` | 4.x → 5.x | 100% | 85% | 86 | netlify/cli, bcgov/tenant-management-system, amiantos/lurker, openhab/openhab-cloud | candidate | +| `express` | 4.x → 5.x | 100% | 85% | 86 | netlify/cli, bcgov/tenant-management-system, amiantos/lurker, openhab/openhab-cloud | verified | | `ioredis` | 5.x → 6.x | n/a | 0% | 2 | ferni-ai/voiceai, syzygyhack/open-foundry, simondietschi/dokunc, safe-global/safe-client-gateway, kazumadev619-dev/FishingConditionsApp | candidate | | `joi` | 17.x → 18.x | n/a | n/a | — | edtech4good/edtech-lms-api, senacor/azure-function-middleware | candidate | | `lucide-react` | 0.x → 1.x | 100% | 100% | 11 | tradeblocks-org/tradeblocks, spizeck/SFPCA, cubicsteak/keep | verified | @@ -70,7 +70,7 @@ The contract: [migration packs](docs/packs.md). | `@sentry/nextjs` | 10.x → 11.x | 100% | 100% | 8 | cohstats/coh3-stats, xdoubleu/tools.xdoubleu.com, maximunited/yomu | verified | | `@sentry/react` | 10.x → 11.x | 100% | 100% | 2 | owine/youth-activity-scheduler, successible/cleanslate | candidate | | `stripe` | 14 and newer | 100% | 92% | 21 | unkeyed/unkey, nextjs/saas-starter, starl3xx/wallet-to-social, AtlasDevHQ/atlas | verified | -| `typescript` | 5.x → 6.x | 100% | 17% | 1399 | m-lyon/recipe, robrechtme/recipes, JosephMaynard/dependency-radar, CroffZ/github-release-notifier, SignalK/signalk-derived-data, shogo82148/typescript-action, Borewit/music-metadata | candidate | +| `typescript` | 5.x → 6.x | 100% | 99% | 1399 | m-lyon/recipe, robrechtme/recipes, JosephMaynard/dependency-radar, CroffZ/github-release-notifier, SignalK/signalk-derived-data, shogo82148/typescript-action, Borewit/music-metadata | verified | | `vitest` | 4.x → 5.x | 100% | 50% | 32 | votingworks/vxsuite, vitorvasc/opentelemetry-ecosystem-explorer | verified | | `zod` | 3.x → 4.x | 100% | 64% | 219 | ugurkocde/AwesomeIntune, howardyang2009/PATH, band-ai/band-sdk-typescript, vzdc-artcc/website, CANShift/canshift-core | verified | diff --git a/docs/packs.md b/docs/packs.md index 50d1f888..7299d565 100644 --- a/docs/packs.md +++ b/docs/packs.md @@ -140,7 +140,9 @@ site is reported is part of the contract (`reported`): findings are claimed and folds compiler errors that one edit elsewhere resolves into a finding at that edit, as their evidence. The typescript pack folds the "Cannot find name" errors of the 6.0 `types: []` default into one at the tsconfig that leaves - `types` unset. + `types` unset. `pack test` counts a ground-truth site listed as such evidence as + covered, at the site's own rule and severity; evidence the truth does not name is not + scored. - `decision`: the pull request lists it under "Decisions for you", with the files and lines. - `test-follow-up`: an assertion is updated only after the migrated code made that test fail, and the test then has to pass (zod's reworded default messages). diff --git a/packages/cli/src/__snapshots__/list-design.test.ts.snap b/packages/cli/src/__snapshots__/list-design.test.ts.snap index 983f9e83..47bb8813 100644 --- a/packages/cli/src/__snapshots__/list-design.test.ts.snap +++ b/packages/cli/src/__snapshots__/list-design.test.ts.snap @@ -29,7 +29,7 @@ PACKAGES cookie-plugin 1.0.0 → 2.0.0 major 1 file 1 ref TOOLING 19 packages, used by scripts and config · --all - typescript 5.0.0 → 6.0.0 major 0 files + typescript 5.0.0 → 6.0.0 major 0 files verified compiler major: check build and tsconfig webpack 5.0.0 → 6.0.0 major 0 files bundler major: check build config @@ -53,27 +53,27 @@ exports[`Nest terminal design, TTY=true 1`] = ` GROUPS upgrade together @nestjs/* 10 packages → 12.x npx uptide check --group nestjs - @nestjs/cli 10.0.0 → 12.0.0 2 majors behind 0 files + @nestjs/cli 10.0.0 → 12.0.0 2 majors behind 0 files   known configuration or build tool - @nestjs/common 10.4.0 → 12.0.0 2 majors behind 1 file 2 calls - @nestjs/core 10.4.0 → 12.0.0 2 majors behind 1 file 1 call - @nestjs/platform-express 10.3.0 → 12.0.0 2 majors behind 1 file 1 ref - @nestjs/platform-fastify 10.4.0 → 12.0.0 2 majors behind 1 file 1 call - @nestjs/schematics 10.0.0 → 12.0.0 2 majors behind 0 files + @nestjs/common 10.4.0 → 12.0.0 2 majors behind 1 file 2 calls  + @nestjs/core 10.4.0 → 12.0.0 2 majors behind 1 file 1 call  + @nestjs/platform-express 10.3.0 → 12.0.0 2 majors behind 1 file 1 ref  + @nestjs/platform-fastify 10.4.0 → 12.0.0 2 majors behind 1 file 1 call  + @nestjs/schematics 10.0.0 → 12.0.0 2 majors behind 0 files   known configuration or build tool  referenced by configuration - @nestjs/swagger 7.4.2 → 12.0.0 5 majors behind 1 file 1 call + @nestjs/swagger 7.4.2 → 12.0.0 5 majors behind 1 file 1 call  @fastify/static 6.12.0 → 10.1.5 4 majors behind peer of @nestjs/platform-fastify nodemailer 6.9.0 → 8.0.0 2 majors behind peer of @nestjs/core reflect-metadata 0.1.0 → 1.0.0 major peer of @nestjs/common, @nestjs/core PACKAGES - cookie-plugin 1.0.0 → 2.0.0 major 1 file 1 ref + cookie-plugin 1.0.0 → 2.0.0 major 1 file 1 ref  TOOLING 19 packages, used by scripts and config · --all - typescript 5.0.0 → 6.0.0 major 0 files + typescript 5.0.0 → 6.0.0 major 0 files verified  compiler major: check build and tsconfig - webpack 5.0.0 → 6.0.0 major 0 files + webpack 5.0.0 → 6.0.0 major 0 files   bundler major: check build config POSSIBLY UNUSED 2 packages, no usage found by Uptide's scan; verify before removing · --all diff --git a/packages/core/src/packs/express/RESEARCH.md b/packages/core/src/packs/express/RESEARCH.md index eed60404..13c08361 100644 --- a/packages/core/src/packs/express/RESEARCH.md +++ b/packages/core/src/packs/express/RESEARCH.md @@ -129,10 +129,11 @@ Searched read-only (blobless clones, `git log`/`show`, `gh search`): ## Measured pack test -`pnpm uptide pack test express --write` with per-pack caches (`UPTIDE_GROUND_TRUTH_CACHE`, -`UPTIDE_CACHE_DIR`), rebased on UPT-48 (#155) and UPT-63 (#157), 2026-10-15: **passes, measures -verified; recorded candidate by choice** (below). 4 public repositories, 86 expected breaking sites, -no unexercised rewrite. +`pnpm uptide pack test express --write` on main (`530f90c`, after UPT-48 #155 and UPT-63 #157) +with its own `UPTIDE_CACHE_DIR`, 2026-10-15: **passes, measures verified, recorded verified.** 4 +public repositories, 86 expected breaking sites, no unexercised rewrite. The ground truth is +unchanged (same digest as the candidate record), and the numbers match the run that first measured +verified. | Rule / scoring group | Precision | Recall | TP | FP | FN | | --- | ---: | ---: | ---: | ---: | ---: | @@ -153,9 +154,11 @@ ground truth was not changed: - openhab: body-parser and serve-static moved to 2.x, the repository's choice (neither depends on express). -**Why candidate.** Kept a candidate on purpose until a person reviews it: the http-proxy-middleware -peer conflict means `fix` would leave an install the netlify upgrade never had, and every rule -outside `route-syntax` and `params-string-array` is measured by fixtures only. +**Known gaps of the verified record.** The http-proxy-middleware peer conflict means `fix` would +leave an install the netlify upgrade never had (`check` names the conflict, it does not move the +package), and every rule outside `route-syntax` and `params-string-array` is measured by fixtures +only. The pack was held as a candidate on these grounds until UPT-48's gate was on main; the gate +does not cover either, so they are stated here. **Recall misses, all kept.** Lurker's handler annotations and bcgov's destructure are where the upgrade fixed a params type, not where the compiler reports it. netlify keeps three route paths as @@ -177,9 +180,14 @@ them rewrites, so under UPT-48 none is an unexercised rewrite. ## UPT-48 gate -Scored under the stricter gate (#155): 86 expected breaking sites across 4 repositories, per-rule -TP/FP/FN above, `unexercised: []` (no rule rewrites). It measures verified; the record says -candidate by choice. +Verified under the stricter gate (#155): + +- Repositories: 4 public (gate: ≥2). +- Real breaking sites: 86 expected (gate: ≥5). +- Breaking false positives: 0 (gate: 0). Breaking precision 100%, recall 85% (73 of 86). +- Behavior notes: 2 expected `req-body-undefined` sites, none predicted (recall 0%, no false + positive). +- Unexercised: none (`unexercised: []`). No rule rewrites, so no rewrite runs unconfirmed. ## What the engine cannot express diff --git a/packages/core/src/packs/express/verification.json b/packages/core/src/packs/express/verification.json index 07b4c2f0..2a4a5a81 100644 --- a/packages/core/src/packs/express/verification.json +++ b/packages/core/src/packs/express/verification.json @@ -1,5 +1,5 @@ { - "status": "candidate", + "status": "verified", "truth": "sha256-d07b7e14c0c525dae477ce9085f51713a46a89ecebdb260f8f9c4d48eef88e99", "repos": 4, "breaking": { diff --git a/packages/core/src/packs/tooling/pack-test.test.ts b/packages/core/src/packs/tooling/pack-test.test.ts index 14840b73..b250af51 100644 --- a/packages/core/src/packs/tooling/pack-test.test.ts +++ b/packages/core/src/packs/tooling/pack-test.test.ts @@ -142,6 +142,15 @@ it('scores an anchored finding at its anchor; the call sites listed as evidence expect(predictedSites(report, aiPack).map((s) => `${s.file}:${s.line} ${s.rule}`)).toEqual([ 'packages/lib/hook.ts:6 tool-context', ]); + // Evidence the ground truth names is covered by the anchor, at the truth's rule and severity; + // evidence it does not name (x.tsx:11) stays unscored. + const expected = [ + { file: 'apps/b/src/y.tsx', line: 59, rule: 'generic', severity: 'breaking' as const }, + ]; + expect(predictedSites(report, aiPack, expected)).toEqual([ + { file: 'apps/b/src/y.tsx', line: 59, rule: 'generic', severity: 'breaking' }, + { file: 'packages/lib/hook.ts', line: 6, rule: 'tool-context', severity: 'breaking' }, + ]); }); it('counts a required move as moved, and a move check reports only a new major makes as explained', () => { diff --git a/packages/core/src/packs/tooling/pack-test.ts b/packages/core/src/packs/tooling/pack-test.ts index df7d7d2c..2b4992d3 100644 --- a/packages/core/src/packs/tooling/pack-test.ts +++ b/packages/core/src/packs/tooling/pack-test.ts @@ -141,8 +141,14 @@ const key = (s: { file: string; line: number }): string => `${s.file}:${s.line}` /** * The sites `check` shows for the package, from its plan: the findings worth acting on, * grouped under the rule `fix` would apply. Repository-relative, as ground truth is written. + * An `expected` site an anchored finding lists as evidence is shown too (see below). */ -export function predictedSites(report: CheckReport, pack: Pack): ScoredSite[] { +export function predictedSites( + report: CheckReport, + pack: Pack, + expected: ScoredSite[] = [], +): ScoredSite[] { + const wanted = new Map(expected.map((e) => [key(e), e])); const own = new Set([...pack.rules.map((r) => r.id), ...pack.behavior.map((b) => b.id)]); const sites = new Map(); for (const p of report.packages) { @@ -184,6 +190,14 @@ export function predictedSites(report: CheckReport, pack: Pack): ScoredSite[] { // One site, one rule: the more severe group wins, as the plan lists it first. if (!sites.has(key(site))) sites.set(key(site), site); } + // Evidence the ground truth names is covered: `check` reports the line under its anchor + // (TypeScript 6's `types` errors under one tsconfig finding, #136's option cascades), so + // it scores as the site the truth expects, at that site's rule and severity. Evidence the + // truth does not name stays unscored. + for (const k of evidence) { + const site = wanted.get(k); + if (site && !sites.has(k)) sites.set(k, { ...site }); + } } return [...sites.values()].sort((a, b) => a.file.localeCompare(b.file) || a.line - b.line); } @@ -350,7 +364,7 @@ async function scoreRepo( if (installed.length > 0 && !installed.includes(entry.from)) base.installedMismatch = `the lockfile has ${installed.join(', ')}, the ground truth says ${entry.from}`; const own = report.packages.filter((p) => p.name === pack.name); - const predicted = predictedSites(report, pack); + const predicted = predictedSites(report, pack, expected); return { ...base, ...companionOutcome(own, entry.with, (name) => dependsOn(project, dir, name, pack.name)), diff --git a/packages/core/src/packs/typescript/RESEARCH.md b/packages/core/src/packs/typescript/RESEARCH.md index af1cabda..b1769097 100644 --- a/packages/core/src/packs/typescript/RESEARCH.md +++ b/packages/core/src/packs/typescript/RESEARCH.md @@ -217,26 +217,33 @@ The recall figure measures the original truth's convention (every error line a s the engine's (one site per edit). It is not a count of breaks `check` misses: every folded error is listed under its tsconfig finding. -## The verified gate, now and under UPT-48 - -Under today's gate the run measures **verified**: seven public repositories, no breaking false -positive, 1,379 breaking sites predicted. The task asked to hold the claim until the stricter gate -(UPT-48) lands, so `verification.json` keeps the candidate record and is not written by this pull -request. After rebasing on UPT-48, `pack test --write` re-scores and records whatever that gate -says. - -Against UPT-48 as specified: - -- ≥2 repositories: 7. -- ≥5 expected breaking sites: 1,399. +**Scoring the folded evidence.** Since then `pack test` counts a ground-truth site that an +anchored finding lists as evidence as covered by that finding, at the site's own rule and +severity (`predictedSites`, as for #136's option cascades). The ground truth did not change. +Re-scored 2026-10-15: breaking recall **99% (1,379 of 1,399)**, 0 breaking false positives, +the same as before the fold. The 20 breaking misses are the second-round and same-change lines +listed under "Measured `pack test`". + +## Verified under UPT-48 + +`pnpm uptide pack test typescript --write` on main (`530f90c`, UPT-48's gate), with its own +`UPTIDE_CACHE_DIR`, 2026-10-15: **PASS, measures verified, recorded verified.** The ground truth +is unchanged (same digest as the candidate record). + +- Repositories: 7 public (gate: ≥2). +- Real breaking sites: 1,399 expected (gate: ≥5). +- Breaking false positives: 0 (gate: 0). Breaking precision 100%, recall 17% (240 of 1,399; the + folded `types` errors above), 99% (1,379) once the scorer counts folded evidence (above). +- Behavior notes: precision 20%, recall 50% (2 TP, 8 FP, 2 FN), all `changed-defaults` anchors at + `tsconfig*.json:1`. Notes do not count against a breaking-class gate. - Per-rule real-world true positives: `base-url` 4, `module-resolution-node10` 2, `target-es5` 1, `root-dir` 1 (all assisted). -- The three mechanical rewrites (`downlevel-iteration`, `module-keyword`, `import-assertions`) - met **no** real-world site. Under UPT-48 they are unexercised and run as assisted. The fixtures - are their only evidence. +- **Unexercised** (recorded in `verification.json`): `downlevel-iteration`, `import-assertions`, + `module-keyword`. No real site exercises their rewrites, so `check` and `fix` run them assisted; + the fixtures are their only evidence. - `module-resolution-classic`, `module-format-removed`, `out-file`, `interop-false` and - `always-strict-false` are detection only and also unexercised. Each matches only the exact value - 6.0 reports. + `always-strict-false` are detection only with no real site either. Each matches only the exact + value 6.0 reports. ## Known limits diff --git a/packages/core/src/packs/typescript/verification.json b/packages/core/src/packs/typescript/verification.json index 9d225d18..6382c7a4 100644 --- a/packages/core/src/packs/typescript/verification.json +++ b/packages/core/src/packs/typescript/verification.json @@ -1,13 +1,13 @@ { - "status": "candidate", + "status": "verified", "truth": "sha256-b463fcdce83820c229ab46ab51a2afb62c3da5f4f5e6a2b53fee63f302755e68", "repos": 7, "breaking": { "precision": 1, - "recall": 0.1716, + "recall": 0.9857, "falsePositives": 0, - "truePositives": 240, - "falseNegatives": 1159 + "truePositives": 1379, + "falseNegatives": 20 }, "behavior": { "precision": 0.2,