Repository navigation
feat(pack): ground-truth preparation runs the known code generators (UPT-95) - #218
Merged
Merged
Conversation
lucasbraum
force-pushed
the
lucasbraum/upt-95-ground-truth-generators
branch
from
October 18, 2026 00:59
34419e2 to
daacbbe
Compare
…(UPT-95) Declared before any re-score: after the install and before check, the ground-truth preparation runs the repository's Prisma and GraphQL codegen generators (UPT-90's detection), for every pack and repository. A failed generator is reported and the repository is scored as before. check still runs no scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
…UPT-95) ensureRepo runs UPT-90's generateClients on the installed project after the install, for every repository packs:fetch and pack test prepare, and records each outcome in the cache marker. A cache installed before this change is generated on next use without a re-fetch or re-install, also offline. pack test reports a generator that failed, timed out or was refused for that repository; it is scored without the generated code. check still runs no scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
…aration (UPT-95) Declared before the re-score that follows: Prisma 5 and 6 fetch engine binaries from outside the registry when their install script did not run, so that generator fails and is reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
…UPT-95) Prisma 5 and 6 fetch engine binaries from binaries.prisma.sh when their install script did not run. Ground truth passes an unreachable PRISMA_ENGINES_MIRROR through generateClients' new env parameter, so that generator fails and is reported. Also report the line of a generator's output that says what failed, not its trailing summary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
…erator fails the pack (UPT-95) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
… generated code fails (UPT-95) On the Packs runner, prisma generate for vzdc-artcc (and every other Prisma repository) failed: the CLI downloads its schema engine (and, for Prisma 5/6, the query library) on first use, and the ground-truth mirror block refuses it. Locally ~/.cache/prisma already held the engines. ensureRepo now fetches those engines in uptide's own code before any generator runs: the commit the locked @prisma/engines-version names, for this machine's binary target, from binaries.prisma.sh, verified against the published sha256, kept under the cache's prisma-engines/, and handed to Prisma through PRISMA_SCHEMA_ENGINE_BINARY and PRISMA_QUERY_ENGINE_LIBRARY. The generators stay behind the blocked mirror. A cached failed generation runs again. pack test no longer scores a repository whose generator failed: it is incomplete, unscored, and fails the pack, unless its ground truth entry declares withoutGenerated with the reason. Four entries declare it, all failures in the repository itself: dokunc and FishingConditionsApp (ioredis, prisma.config.ts needs a database URL), lago-front (react-router, local schema without a Query root) and m-lyon/recipe (typescript, codegen.ts without a schema). Their verification records get the new truth digest only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
lucasbraum
force-pushed
the
lucasbraum/upt-95-ground-truth-generators
branch
from
October 18, 2026 04:30
daacbbe to
f64e72f
Compare
lucasbraum
added a commit
that referenced
this pull request
Oct 18, 2026
…UPT-99) pnpm uptide pack test zod --write, then pnpm docs:packs. With #218's generator step and this branch's folding: breaking found at any severity 216/249, as breaking 117/249, 0 breaking false positives, verified, the numbers CI measured for this branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
…ey come from (UPT-99) (#222) Closes UPT-99. Stacked on #218 (base: `lucasbraum/upt-95-ground-truth-generators`). **Rule first.** The folding rule went into docs/concepts.md ("One root cause, one finding") and docs/packs.md (Scoring) in 2cb4eef, before any scoring run. The engine change is 7f66699. ## What changes A new compile error whose value the compiler traces to another new error's expression is that site's consequence. It folds into the breaking finding there as a **downstream error**. It no longer counts as a breaking site of its own. - **Tracing is compiler-only (`flowOrigin`, adapters/typescript/cause.ts).** It starts at the expression at the error and follows the value back through: - variable and destructured declarations; - a callback whose parameter is typed only by an `any` receiver (`errors.map(e => …)`); - members of returned object literals; - casts to a type whose import is itself a new error (`as SafeParseReturnType<…>`); - calls of repository functions. A call is followed only when every return whose type changed between the installed program and the target leads to such an expression. Types are compared by an order-free shape (members, unions and intersections sorted), because the two programs print the same Prisma payload types in a different member order. There is no line-distance fallback: no trace means no origin. - **Folding (`foldConsequences`, check/root-cause.ts)** runs per workspace after the pack has run. The root is the innermost breaking finding (not a cause cluster) whose span holds the origin. Chains go to the first root, and loops don't fold. A diagnostic with no breaking finding at its origin stays as it was. The verdict is recounted. - **Output:** - default view: `49 sites, +54 downstream errors` on the rule's line; - `--details`: each folded error listed under its root; - `--json`: the root finding's `downstream`, plus `plan[].downstream`. Downstream errors are not sites, so `fix` selects only the root. Cause clusters no longer leak their `downstream` onto their expanded sites. - **`fix` verification is unchanged:** it still compiles the whole workspace and compares every diagnostic with the baseline. Folding changes what is reported, never what has to compile. - **Fixtures** (`fixtures/deps/issues-v1`/`-v2`, `cause.test.ts`): - a `.errors` read whose `map(e => e.message)` (TS7006) and lost narrowing (TS18048) fold into the root; - a removed import in the same file (TS2305) stays; - a call whose return also changed somewhere untraceable is not followed. `root-cause.test.ts` covers the fold: chains, orphans, non-breaking roots, loops and the plan count. No ground-truth, gate or threshold change, and no `verification.json` rewrite. ## Before / after "Before" is #218's CI run [39076300528](https://github.com/uptide-dev/uptide/actions/runs/39076300528) at 34419e2 (pack-test artifacts). "After" is a local re-score of this branch: zod first, then one pack at a time, each with its own cache, deleted after. The columns are #212's: breaking ground-truth sites found by a finding at any severity, and found as breaking. | Pack | Breaking sites | Any severity, before | Any severity, after | As breaking, before | As breaking, after | Breaking FPs before → after | Status before → after | | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- | | @sentry/browser | 22 | 82% (18) | 82% (18) | 82% (18) | 82% (18) | 0 → 0 | verified → verified | | @sentry/nextjs | 8 | 100% (8) | 100% (8) | 100% (8) | 100% (8) | 0 → 0 | verified → verified | | @sentry/react | 29 | 72% (21) | 72% (21) | 72% (21) | 72% (21) | 3 → 3 | candidate → candidate | | ai | 11 | 82% (9) | 82% (9) | 82% (9) | 82% (9) | 0 → 0 | verified → verified | | express | 86 | 86% (74) | 86% (74) | 86% (74) | 86% (74) | 0 → 0 | verified → verified | | ioredis | 2 | 0% (0) | 0% (0) | 0% (0) | 0% (0) | 0 → 0 | candidate → candidate | | joi | 108 | 100% (108) | 100% (108) | 100% (108) | 100% (108) | 0 → 0 | verified → verified | | lucide-react | 11 | 100% (11) | 100% (11) | 100% (11) | 100% (11) | 0 → 0 | verified → verified | | next | 34 | 88% (30) | 88% (30) | 88% (30) | 88% (30) | 0 → 0 | verified → verified | | openai | 20 | 15% (3) | 15% (3) | 15% (3) | 15% (3) | 0 → 0 | verified → verified | | react | 130 | 95% (123) | 95% (123) | 95% (123) | 95% (123) | 0 → 0 | verified → verified | | react-router | 18 | 89% (16) | 89% (16) | 78% (14) | 78% (14) | 0 → 0 | verified → verified | | redis | 20 | 85% (17) | 85% (17) | 75% (15) | 75% (15) | 0 → 0 | verified → verified | | stripe | 25 | 92% (23) | 92% (23) | 92% (23) | 92% (23) | 0 → 0 | verified → verified | | typescript | 1399 | 99% (1379) | 99% (1379) | 99% (1379) | 99% (1379) | 0 → 0 | verified → verified | | vitest | 32 | 97% (31) | 97% (31) | 97% (31) | 97% (31) | 0 → 0 | verified → verified | | **zod** | 249 | 93% (232) | 87% (216) | 22% (55) | 47% (117) | 0 → 0 | verified → verified | No pack loses a true positive. Site by site, every true-positive site of every pack in the "before" run is still found, with one exception: zod's 17, which #218 itself loses (see below). ### zod, with vzdc-artcc compiling The CI "before" is not the state this issue is about. **On #218's CI, vzdc-artcc's `prisma generate` failed.** The artifact reports `generator failed: prisma generate … ECONNREFUSED 127.0.0.1:9`: Prisma 7 fetches `schema-engine` on the Linux runner, and #218's engine-download block refuses it. So vzdc-artcc did not compile there, and zod stayed at its pre-generator numbers. Locally the engines are already in `~/.cache/prisma`, so the project generates and compiles. #218's own local re-score at the same commit (34419e2) is the meaningful zod "before": | zod | Any severity | As breaking | Breaking TP / FP / FN | Status | | --- | ---: | ---: | --- | --- | | #218 CI (generator failed) | 93% (232) | 22% (55) | 55 / 0 / 17 | verified | | #218 local (generator ran) | 87% (216) | 47% (117) | 117 / **58** / 33 | candidate (gate fails) | | this PR, local | 87% (216) | 47% (117) | 117 / **0** / 33 | **verified** (gate passes) | All 58 vzdc-artcc false positives fold: - 54 under the `error-issues` sites (`return {errors: result.error.errors}` in the server actions), from the forms' `errors.map(e => e.message)` (TS7006) and the lost narrowing (`savedAirport` possibly undefined, TS18048). - 4 under the `SafeParseReturnType` import in `actions/event.ts`/`actions/eventPosition.ts` (`removed-exports`). Those actions cast the parse result to it, so their `.errors` is `any` without an error of its own. The 17 vzdc misses are the same before and after. They are #218's: the 15 `{ message }` generic sites that compile clean under zod 4, plus the two `.errors` reads behind that cast. **The CI generator failure is #218's.** Once this PR is merged into #218's branch, CI will still score vzdc-artcc without the generated client. The zod gate will pass there either way. The folding is only exercised where `prisma generate` succeeds. ## Local notes - @sentry/react ran with the same pnpm-12 shim and LFS pass-through env as UPT-64/UPT-95. It exits 1 on its existing 3 false positives, as before. - Tests: - new and related suites pass: cause, root-cause, unattributed, option-cascades, format-check, format-check-details. - Failing on this branch and on its base alike (fixture installs, scaffold, zod contract against the generated vzdc project, the check verdict JSON): 20 tests. - Under a full parallel run, some `feedback.test.ts` and `fix.test.ts` cases time out on both. Run alone, all 89 pass on this branch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Lucas Braum <lucasbraum@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Merged
lucasbraum
pushed a commit
that referenced
this pull request
Oct 18, 2026
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## uptide@0.15.1 ### Patch Changes - [#220](#220) [`cf86d65`](cf86d65) Thanks [@lucasbraum](https://github.com/lucasbraum)! - With an assisting agent, a root pin Uptide cannot update safely (a nested override such as `overrides > pkg > peerDependencies > react`) no longer stops `fix` with exit 2: the run continues as an agent-led draft that leaves the pin as written and names it, with its exact path and value, at the top of the PR body. The agent-led peer reason now names each blocker's first release that accepts the target, or says none does. An agent-led draft must pass the repository's clean install (`npm ci` or the manager's frozen install) before delivery; if it does not, the PR says so at the top. Without an agent, with `--no-llm`, or on a safety refusal, `fix` still exits 2. - [#232](#232) [`e4a2885`](e4a2885) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide fix` spends less wall time on test runs and dependency copies: - The agent's own `run_tests` check runs only its runner's related tests when the `test` script builds first and then runs a script that is just the runner (`pnpm run build && pnpm run test:unit`). The gate after the agent's edits and the final verification run exactly what they ran before. - The upgrade's install moves the installed `node_modules` out of its temporary worktree instead of copying it. - Each phase of the fix report lists the repository's commands it ran (installs, generators, builds, type-checks, tests, lint, format) with their count and summed wall time. An agent-led run's clean install has its own phase. - [#223](#223) [`2dc2162`](2dc2162) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `fix` no longer stops with "baseline tests modified the working tree" when the test run only writes new untracked files, such as a junit `test-report.xml` that is not gitignored. Those files are recorded as test artifacts (`testArtifacts` in the report), kept out of every commit, the agent's diff and the pending-changes check, and removed before each later test run, `uptide verify` included. A change to a tracked file during baseline tests still refuses the run. - [#215](#215) [`abddcb3`](abddcb3) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check` no longer lists possible impact (type changes nothing confirmed as breaking) on its first screen: each package says it in one line, `+N possible impact in M files, not counted as breaking`, and `--details` and `--json` still have every site. A package whose types were partly verified no longer repeats its coverage line in the heading, and a monorepo's unbuilt workspace chains are one count on that line. - [#213](#213) [`2de3a18`](2de3a18) Thanks [@lucasbraum](https://github.com/lucasbraum)! - A companion pack that moves with the lead's fix (react beside next) now gets its own scan context and inventory, and the PR body carries a section with its review notes. - [#212](#212) [`733cfbe`](733cfbe) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `pack test --write` records how many breaking sites a finding landed on at any severity and as breaking, out of all breaking sites it scored, and the pack tables show both ("Found, any severity", "Found as breaking") in place of the single recall column. Packs recorded before it show `—`. - [#224](#224) [`09f68df`](09f68df) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `fix` on npm no longer stops when the lock-only install hits ERESOLVE from the old lockfile. It retries without the lockfile entries of the moved packages, their companions and the packages whose peers they satisfy, and only then with the whole lockfile re-resolved; the report says which step worked, and the lockfile check reports the churn as usual. It never uses `--legacy-peer-deps` or `--force`. When every step fails, `fix` ends blocked, naming the conflicting packages from npm's ERESOLVE report. `fix --json` now prints JSON on every exit, a failure included; when the run already has its report (a PR body over GitHub's size limit), that report comes with the error. - [#221](#221) [`ff8de8e`](ff8de8e) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `fix` no longer rejects an agent patch for test files the parent commit already failed, whichever test scope ran them: when the test script builds first and the build failed on the error being fixed, the start had no test results to compare with, so a correct patch was reverted every round. The patch's tests also run with the pack's follow-ups in place (a test asserting the old Stripe API version), which `fix` still applies and reports after the agent. When a PR body cannot fit, the stored report.json the error points to is now written first. `check` and `fix` now also analyze files that reach a package's wrapper through a barrel (`import { getStripeClient } from '../stripe'` with `export * from './client'`). Edited files are formatted with oxfmt and dprint as well as biome and prettier, from the workspace that holds them, with its own configuration, else the root's. The stripe pack adds `request-options`: request options passed in the params slot (`subscriptions.cancel(id, { idempotencyKey })`), which SDK 22 rejects, move to the third argument. It runs assisted until a ground-truth repository exercises it. - [#226](#226) [`448c3cb`](448c3cb) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide fix` judges consumed workspace builds on the final head, not on the tree right after the bump. A build that failed after the bump and passes once the edits land no longer starts a feedback round, adds an "Agent-led draft" reason or shows "Build failed" in the PR body. The builds run again only when the tree changed since their last result, and not at all when the verification build already built those workspaces. The fix report's `buildRuns` gives the number of builds the run executed and their wall time. - [#218](#218) [`13e343b`](13e343b) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check` folds a compile error into the breaking site it comes from when the compiler traces its value there (an `errors.map(e => e.message)` under a `result.error.errors` that zod 4 no longer types, and the narrowing that depended on it). The rule's line shows `+N downstream errors`, `--details` lists them under their site, and `--json` carries them as that finding's `downstream`. They are no longer separate breaking sites. `fix` still verifies against every error. - [#214](#214) [`195ae10`](195ae10) Thanks [@lucasbraum](https://github.com/lucasbraum)! - A rewrite rule below 80% precision over at least 5 scored ground-truth sites now runs as assisted in `fix` (react-router `dom-imports`); smaller samples (next `middleware-proxy`) stay mechanical and are flagged in the compatibility table. Pack status is unchanged. - [#218](#218) [`13e343b`](13e343b) Thanks [@lucasbraum](https://github.com/lucasbraum)! - pack test runs a ground-truth repository's code generators (prisma generate, local graphql-codegen) after its install, as fix does. It fetches the Prisma engines the locked version needs itself, so generators stay off the network. A repository whose generator still fails is incomplete and fails the pack, unless its ground truth declares `withoutGenerated` - [#227](#227) [`ae175a9`](ae175a9) Thanks [@lucasbraum](https://github.com/lucasbraum)! - A migration whose evidence is larger than GitHub's PR description limit no longer stops `fix` with "Required migration evidence exceeds the PR body budget" and loses the run. Long required lists get a compact form when they do not fit. Lockfile churn shows counts per kind of change and the first packages. Manual sites are grouped by rule, with counts and the first locations. Remaining type errors show the first entries. Every full list stays in report.json and, in the Uptide Action, in the job summary, which the description links to. If even the compact form does not fit, the description is a short summary that says so at the top: the verdict, the checks, the counts and where the full report is. The branch and report.json are always kept. An agent's decisions under "Decisions for you" name the files the agent changed, five at most, and count the rest, instead of every file in the task's scope. A verification-feedback task scoped to the whole repository made each of its decisions a list of hundreds of files. - [#219](#219) [`12dc612`](12dc612) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `fix zod` rewrites `z.string().uuid()` to `z.guid()` by rule without `--include-deprecated`: zod 4's `uuid()` also checks the RFC version and variant, so ids zod 3 accepted stop validating, and `check` now counts the site as breaking. `fix` no longer reports false pre-existing type errors on a TypeScript 7 repository whose `lib` names a library newer than its `target` (`lib: ["ES2023"]` under `ES2022`). Co-authored-by: uptide-release[bot] <338861287+uptide-release[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes UPT-95.
Method first. The docs/packs.md change was committed before any scoring run (694b141). The engine-download rule (dbdbac7) was also committed before the final re-score. After the install and before
check, ground-truth preparation (ensureRepo, so bothpacks:fetchandpack test) runs UPT-90'sgenerateClientson the installed project. That meansprisma generatefor aschema.prisma, and GraphQL codegen for acodegen.ts/ymlwith a local schema. It applies to every pack and every repository.checkstill runs no scripts. Fixtures are synthetic and installed in the tree, so they get no generator step.pack testreports a generator that failed, timed out or was refused asgenerator failed: <command> (<reason>)(generatorFailedin--json). That repository is scored without the generated code, as before.PRISMA_ENGINES_MIRRORto an address nothing listens on, passed through a new optionalenvparameter ofgenerateClients. Prisma 5/6 download their engine from binaries.prisma.sh when the install script did not run. Checked on a cleanprisma@5.22.0install with scripts off:generatefails withECONNREFUSED 127.0.0.1:9. Prisma 7's client generator needs no engine.verification.jsonis rewritten.Before / after
Before is CI run 39069556978, at this PR's base 213e1a5. After is a local re-score at this branch, one pack at a time, each with its own cache (as CI), deleted after. The columns are #212's, computed the same way from the
pack test --jsonsites: breaking ground-truth sites of every scored repository, found by a finding at any severity, and found as breaking.Generators that ran (11 repositories, 9 packs).
~/.cache/prisma: HiroshiIgarash/koremina (next), csnyder256/ux-struggle-detector (openai) and UNN-Devotek/FCM-Fallout-Chat-Mod (redis). On a fresh runner the download block makes these fail and be reported. Their scores are the same either way.prisma.config.tsneedsDATABASE_URL/DATABASE_URL_DIRECT.Query root type must be provided.schema.Only zod moves.
zod: the vzdc-artcc sites, and a new gate failure
All 63
error-issues/error-paramssites in vzdc-artcc/website wereunverifiedbefore. Now:error-issues, all 13error-params).actions/event.ts:165andactions/eventPosition.ts:263. The parse result there is castas SafeParseReturnType<any, any>, soresult.error.errorsisanyand nothing fires. This is a real finder gap, not an environment one.Other vzdc-artcc truth sites:
genericbreaking sites ({ message: … }params) were hit only by the file-levelunverifiedfinding. They now compile clean under zod 4 (messageis deprecated, not removed), so they are misses. That is the any-severity drop, 232 → 216.genericsites move fromunverifiedtodeprecated.error-issuessite is newly found as breaking.58 new breaking false positives, all
generic, all in vzdc-artcc. These are compile diagnostics in the form components, such ascomponents/Airports/AirportForm.tsx:17,21,24. They come fromerrors.map(e => e.message)over aZodError.errorsresult that isanyat zod 4, and from the narrowing that falls out of it (savedAirportpossibly undefined). The truth entry says it leaves these out on purpose ("the 36 implicit-any diagnostics (7006) in components that map over a ZodError.errors result typed any at zod 4"). Before this change they were hidden because the files did not compile at all.So zod now measures candidate (gate: "58 false positives among breaking findings"). Its recorded
verifiedmakespack test zodexit 1, and the zod pack job will fail on this PR. I didn't change the ground truth, the gate or the record, per the issue's rules. Three ways forward, your call:pack test zod --write).Local notes
truth.test.tscovers a cached install that gets generated once, offline, and reports a failing generator.generate.test.tspasses. Two tooling tests fail locally on origin/main too (the scaffold formatter check, and "passes every registered pack: each companion it names carries its source").🤖 Generated with Claude Code