Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/promote-typescript-express.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"uptide": minor
---

The typescript 5 to 6 and express 4 to 5 packs are now verified, so `check`, `list` and `fix` use them by default. Both measure verified under the stricter gate on unchanged ground truth: typescript across 7 public repositories (1,399 real breaking sites, no breaking false positive; its `downlevel-iteration`, `import-assertions` and `module-keyword` rewrites stay assisted until a real site exercises them), express across 4 (86 real breaking sites, no breaking false positive).
5 changes: 5 additions & 0 deletions .changeset/score-folded-evidence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"uptide": patch
---

`uptide pack test` counts a ground-truth site that a folded finding lists as its evidence as covered by that finding, so recall reflects what `check` reports. The typescript pack's breaking recall goes from 17% to 99% on unchanged ground truth: its TypeScript 6 `types` default errors fold into one tsconfig finding each.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ The contract: [migration packs](docs/packs.md).
| Package | Range | Precision | Recall | Breaking sites | Ground-truth repositories | Status |
| --- | --- | ---: | ---: | ---: | --- | --- |
| `ai` | 6.x → 7.x | 100% | 82% | 11 | vercel/chatbot, miurla/morphic | verified |
| `express` | 4.x → 5.x | 100% | 85% | 86 | netlify/cli, bcgov/tenant-management-system, amiantos/lurker, openhab/openhab-cloud | candidate |
| `express` | 4.x → 5.x | 100% | 85% | 86 | netlify/cli, bcgov/tenant-management-system, amiantos/lurker, openhab/openhab-cloud | verified |
| `ioredis` | 5.x → 6.x | n/a | 0% | 2 | ferni-ai/voiceai, syzygyhack/open-foundry, simondietschi/dokunc, safe-global/safe-client-gateway, kazumadev619-dev/FishingConditionsApp | candidate |
| `joi` | 17.x → 18.x | n/a | n/a | — | edtech4good/edtech-lms-api, senacor/azure-function-middleware | candidate |
| `lucide-react` | 0.x → 1.x | 100% | 100% | 11 | tradeblocks-org/tradeblocks, spizeck/SFPCA, cubicsteak/keep | verified |
Expand All @@ -70,7 +70,7 @@ The contract: [migration packs](docs/packs.md).
| `@sentry/nextjs` | 10.x → 11.x | 100% | 100% | 8 | cohstats/coh3-stats, xdoubleu/tools.xdoubleu.com, maximunited/yomu | verified |
| `@sentry/react` | 10.x → 11.x | 100% | 100% | 2 | owine/youth-activity-scheduler, successible/cleanslate | candidate |
| `stripe` | 14 and newer | 100% | 92% | 21 | unkeyed/unkey, nextjs/saas-starter, starl3xx/wallet-to-social, AtlasDevHQ/atlas | verified |
| `typescript` | 5.x → 6.x | 100% | 17% | 1399 | m-lyon/recipe, robrechtme/recipes, JosephMaynard/dependency-radar, CroffZ/github-release-notifier, SignalK/signalk-derived-data, shogo82148/typescript-action, Borewit/music-metadata | candidate |
| `typescript` | 5.x → 6.x | 100% | 99% | 1399 | m-lyon/recipe, robrechtme/recipes, JosephMaynard/dependency-radar, CroffZ/github-release-notifier, SignalK/signalk-derived-data, shogo82148/typescript-action, Borewit/music-metadata | verified |
| `vitest` | 4.x → 5.x | 100% | 50% | 32 | votingworks/vxsuite, vitorvasc/opentelemetry-ecosystem-explorer | verified |
| `zod` | 3.x → 4.x | 100% | 64% | 219 | ugurkocde/AwesomeIntune, howardyang2009/PATH, band-ai/band-sdk-typescript, vzdc-artcc/website, CANShift/canshift-core | verified |
<!-- packs:end -->
Expand Down
4 changes: 3 additions & 1 deletion docs/packs.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,9 @@ site is reported is part of the contract (`reported`):
findings are claimed and folds compiler errors that one edit elsewhere resolves into a
finding at that edit, as their evidence. The typescript pack folds the "Cannot find
name" errors of the 6.0 `types: []` default into one at the tsconfig that leaves
`types` unset.
`types` unset. `pack test` counts a ground-truth site listed as such evidence as
covered, at the site's own rule and severity; evidence the truth does not name is not
scored.
- `decision`: the pull request lists it under "Decisions for you", with the files and lines.
- `test-follow-up`: an assertion is updated only after the migrated code made that test
fail, and the test then has to pass (zod's reworded default messages).
Expand Down
22 changes: 11 additions & 11 deletions packages/cli/src/__snapshots__/list-design.test.ts.snap
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ PACKAGES
cookie-plugin 1.0.0 → 2.0.0 major 1 file 1 ref

TOOLING 19 packages, used by scripts and config · --all
typescript 5.0.0 → 6.0.0 major 0 files
typescript 5.0.0 → 6.0.0 major 0 files verified
compiler major: check build and tsconfig
webpack 5.0.0 → 6.0.0 major 0 files
bundler major: check build config
Expand All @@ -53,27 +53,27 @@ exports[`Nest terminal design, TTY=true 1`] = `

GROUPS upgrade together
@nestjs/* 10 packages → 12.x npx uptide check --group nestjs
@nestjs/cli 10.0.0 → 12.0.0 2 majors behind 0 files
@nestjs/cli 10.0.0 → 12.0.0 2 majors behind 0 files 
 known configuration or build tool
@nestjs/common 10.4.0 → 12.0.0 2 majors behind 1 file 2 calls
@nestjs/core 10.4.0 → 12.0.0 2 majors behind 1 file 1 call
@nestjs/platform-express 10.3.0 → 12.0.0 2 majors behind 1 file 1 ref
@nestjs/platform-fastify 10.4.0 → 12.0.0 2 majors behind 1 file 1 call
@nestjs/schematics 10.0.0 → 12.0.0 2 majors behind 0 files
@nestjs/common 10.4.0 → 12.0.0 2 majors behind 1 file 2 calls 
@nestjs/core 10.4.0 → 12.0.0 2 majors behind 1 file 1 call 
@nestjs/platform-express 10.3.0 → 12.0.0 2 majors behind 1 file 1 ref 
@nestjs/platform-fastify 10.4.0 → 12.0.0 2 majors behind 1 file 1 call 
@nestjs/schematics 10.0.0 → 12.0.0 2 majors behind 0 files 
 known configuration or build tool
 referenced by configuration
@nestjs/swagger 7.4.2 → 12.0.0 5 majors behind 1 file 1 call
@nestjs/swagger 7.4.2 → 12.0.0 5 majors behind 1 file 1 call 
@fastify/static 6.12.0 → 10.1.5 4 majors behind peer of @nestjs/platform-fastify
nodemailer 6.9.0 → 8.0.0 2 majors behind peer of @nestjs/core
reflect-metadata 0.1.0 → 1.0.0 major peer of @nestjs/common, @nestjs/core

PACKAGES
cookie-plugin 1.0.0 → 2.0.0 major 1 file 1 ref
cookie-plugin 1.0.0 → 2.0.0 major 1 file 1 ref 

TOOLING 19 packages, used by scripts and config · --all
typescript 5.0.0 → 6.0.0 major 0 files
typescript 5.0.0 → 6.0.0 major 0 files verified
 compiler major: check build and tsconfig
webpack 5.0.0 → 6.0.0 major 0 files
webpack 5.0.0 → 6.0.0 major 0 files 
 bundler major: check build config

POSSIBLY UNUSED 2 packages, no usage found by Uptide's scan; verify before removing · --all
Expand Down
28 changes: 18 additions & 10 deletions packages/core/src/packs/express/RESEARCH.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,10 +129,11 @@ Searched read-only (blobless clones, `git log`/`show`, `gh search`):

## Measured pack test

`pnpm uptide pack test express --write` with per-pack caches (`UPTIDE_GROUND_TRUTH_CACHE`,
`UPTIDE_CACHE_DIR`), rebased on UPT-48 (#155) and UPT-63 (#157), 2026-10-15: **passes, measures
verified; recorded candidate by choice** (below). 4 public repositories, 86 expected breaking sites,
no unexercised rewrite.
`pnpm uptide pack test express --write` on main (`530f90c`, after UPT-48 #155 and UPT-63 #157)
with its own `UPTIDE_CACHE_DIR`, 2026-10-15: **passes, measures verified, recorded verified.** 4
public repositories, 86 expected breaking sites, no unexercised rewrite. The ground truth is
unchanged (same digest as the candidate record), and the numbers match the run that first measured
verified.

| Rule / scoring group | Precision | Recall | TP | FP | FN |
| --- | ---: | ---: | ---: | ---: | ---: |
Expand All @@ -153,9 +154,11 @@ ground truth was not changed:
- openhab: body-parser and serve-static moved to 2.x, the repository's choice (neither depends on
express).

**Why candidate.** Kept a candidate on purpose until a person reviews it: the http-proxy-middleware
peer conflict means `fix` would leave an install the netlify upgrade never had, and every rule
outside `route-syntax` and `params-string-array` is measured by fixtures only.
**Known gaps of the verified record.** The http-proxy-middleware peer conflict means `fix` would
leave an install the netlify upgrade never had (`check` names the conflict, it does not move the
package), and every rule outside `route-syntax` and `params-string-array` is measured by fixtures
only. The pack was held as a candidate on these grounds until UPT-48's gate was on main; the gate
does not cover either, so they are stated here.

**Recall misses, all kept.** Lurker's handler annotations and bcgov's destructure are where the
upgrade fixed a params type, not where the compiler reports it. netlify keeps three route paths as
Expand All @@ -177,9 +180,14 @@ them rewrites, so under UPT-48 none is an unexercised rewrite.

## UPT-48 gate

Scored under the stricter gate (#155): 86 expected breaking sites across 4 repositories, per-rule
TP/FP/FN above, `unexercised: []` (no rule rewrites). It measures verified; the record says
candidate by choice.
Verified under the stricter gate (#155):

- Repositories: 4 public (gate: ≥2).
- Real breaking sites: 86 expected (gate: ≥5).
- Breaking false positives: 0 (gate: 0). Breaking precision 100%, recall 85% (73 of 86).
- Behavior notes: 2 expected `req-body-undefined` sites, none predicted (recall 0%, no false
positive).
- Unexercised: none (`unexercised: []`). No rule rewrites, so no rewrite runs unconfirmed.

## What the engine cannot express

Expand Down
2 changes: 1 addition & 1 deletion packages/core/src/packs/express/verification.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"status": "candidate",
"status": "verified",
"truth": "sha256-d07b7e14c0c525dae477ce9085f51713a46a89ecebdb260f8f9c4d48eef88e99",
"repos": 4,
"breaking": {
Expand Down
9 changes: 9 additions & 0 deletions packages/core/src/packs/tooling/pack-test.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,15 @@ it('scores an anchored finding at its anchor; the call sites listed as evidence
expect(predictedSites(report, aiPack).map((s) => `${s.file}:${s.line} ${s.rule}`)).toEqual([
'packages/lib/hook.ts:6 tool-context',
]);
// Evidence the ground truth names is covered by the anchor, at the truth's rule and severity;
// evidence it does not name (x.tsx:11) stays unscored.
const expected = [
{ file: 'apps/b/src/y.tsx', line: 59, rule: 'generic', severity: 'breaking' as const },
];
expect(predictedSites(report, aiPack, expected)).toEqual([
{ file: 'apps/b/src/y.tsx', line: 59, rule: 'generic', severity: 'breaking' },
{ file: 'packages/lib/hook.ts', line: 6, rule: 'tool-context', severity: 'breaking' },
]);
});

it('counts a required move as moved, and a move check reports only a new major makes as explained', () => {
Expand Down
18 changes: 16 additions & 2 deletions packages/core/src/packs/tooling/pack-test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -141,8 +141,14 @@ const key = (s: { file: string; line: number }): string => `${s.file}:${s.line}`
/**
* The sites `check` shows for the package, from its plan: the findings worth acting on,
* grouped under the rule `fix` would apply. Repository-relative, as ground truth is written.
* An `expected` site an anchored finding lists as evidence is shown too (see below).
*/
export function predictedSites(report: CheckReport, pack: Pack): ScoredSite[] {
export function predictedSites(
report: CheckReport,
pack: Pack,
expected: ScoredSite[] = [],
): ScoredSite[] {
const wanted = new Map(expected.map((e) => [key(e), e]));
const own = new Set([...pack.rules.map((r) => r.id), ...pack.behavior.map((b) => b.id)]);
const sites = new Map<string, ScoredSite>();
for (const p of report.packages) {
Expand Down Expand Up @@ -184,6 +190,14 @@ export function predictedSites(report: CheckReport, pack: Pack): ScoredSite[] {
// One site, one rule: the more severe group wins, as the plan lists it first.
if (!sites.has(key(site))) sites.set(key(site), site);
}
// Evidence the ground truth names is covered: `check` reports the line under its anchor
// (TypeScript 6's `types` errors under one tsconfig finding, #136's option cascades), so
// it scores as the site the truth expects, at that site's rule and severity. Evidence the
// truth does not name stays unscored.
for (const k of evidence) {
const site = wanted.get(k);
if (site && !sites.has(k)) sites.set(k, { ...site });
}
}
return [...sites.values()].sort((a, b) => a.file.localeCompare(b.file) || a.line - b.line);
}
Expand Down Expand Up @@ -350,7 +364,7 @@ async function scoreRepo(
if (installed.length > 0 && !installed.includes(entry.from))
base.installedMismatch = `the lockfile has ${installed.join(', ')}, the ground truth says ${entry.from}`;
const own = report.packages.filter((p) => p.name === pack.name);
const predicted = predictedSites(report, pack);
const predicted = predictedSites(report, pack, expected);
return {
...base,
...companionOutcome(own, entry.with, (name) => dependsOn(project, dir, name, pack.name)),
Expand Down
41 changes: 24 additions & 17 deletions packages/core/src/packs/typescript/RESEARCH.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,26 +217,33 @@ The recall figure measures the original truth's convention (every error line a s
the engine's (one site per edit). It is not a count of breaks `check` misses: every folded
error is listed under its tsconfig finding.

## The verified gate, now and under UPT-48

Under today's gate the run measures **verified**: seven public repositories, no breaking false
positive, 1,379 breaking sites predicted. The task asked to hold the claim until the stricter gate
(UPT-48) lands, so `verification.json` keeps the candidate record and is not written by this pull
request. After rebasing on UPT-48, `pack test --write` re-scores and records whatever that gate
says.

Against UPT-48 as specified:

- ≥2 repositories: 7.
- ≥5 expected breaking sites: 1,399.
**Scoring the folded evidence.** Since then `pack test` counts a ground-truth site that an
anchored finding lists as evidence as covered by that finding, at the site's own rule and
severity (`predictedSites`, as for #136's option cascades). The ground truth did not change.
Re-scored 2026-10-15: breaking recall **99% (1,379 of 1,399)**, 0 breaking false positives,
the same as before the fold. The 20 breaking misses are the second-round and same-change lines
listed under "Measured `pack test`".

## Verified under UPT-48

`pnpm uptide pack test typescript --write` on main (`530f90c`, UPT-48's gate), with its own
`UPTIDE_CACHE_DIR`, 2026-10-15: **PASS, measures verified, recorded verified.** The ground truth
is unchanged (same digest as the candidate record).

- Repositories: 7 public (gate: ≥2).
- Real breaking sites: 1,399 expected (gate: ≥5).
- Breaking false positives: 0 (gate: 0). Breaking precision 100%, recall 17% (240 of 1,399; the
folded `types` errors above), 99% (1,379) once the scorer counts folded evidence (above).
- Behavior notes: precision 20%, recall 50% (2 TP, 8 FP, 2 FN), all `changed-defaults` anchors at
`tsconfig*.json:1`. Notes do not count against a breaking-class gate.
- Per-rule real-world true positives: `base-url` 4, `module-resolution-node10` 2, `target-es5` 1,
`root-dir` 1 (all assisted).
- The three mechanical rewrites (`downlevel-iteration`, `module-keyword`, `import-assertions`)
met **no** real-world site. Under UPT-48 they are unexercised and run as assisted. The fixtures
are their only evidence.
- **Unexercised** (recorded in `verification.json`): `downlevel-iteration`, `import-assertions`,
`module-keyword`. No real site exercises their rewrites, so `check` and `fix` run them assisted;
the fixtures are their only evidence.
- `module-resolution-classic`, `module-format-removed`, `out-file`, `interop-false` and
`always-strict-false` are detection only and also unexercised. Each matches only the exact value
6.0 reports.
`always-strict-false` are detection only with no real site either. Each matches only the exact
value 6.0 reports.

## Known limits

Expand Down
8 changes: 4 additions & 4 deletions packages/core/src/packs/typescript/verification.json
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
{
"status": "candidate",
"status": "verified",
"truth": "sha256-b463fcdce83820c229ab46ab51a2afb62c3da5f4f5e6a2b53fee63f302755e68",
"repos": 7,
"breaking": {
"precision": 1,
"recall": 0.1716,
"recall": 0.9857,
"falsePositives": 0,
"truePositives": 240,
"falseNegatives": 1159
"truePositives": 1379,
"falseNegatives": 20
},
"behavior": {
"precision": 0.2,
Expand Down
Loading