Repository navigation
fix(fix,zod): TS 7 libs in the fix type-check, uuid() as breaking (UPT-100) - #219
Merged
lucasbraum merged 1 commit intoOct 18, 2026
Merged
Conversation
…T-100) The fix phase's compiler host for repositories without a compiler API (TypeScript 7) served only the libs the target's default pulls in, so `lib: ["ES2023"]` under `target: "ES2022"` failed with TS6053 and every `Object.hasOwn` became a false pre-existing error. It now loads any lib by name through compile.ts's bundledLib, as the analysis does since UPT-73. zod 4 keeps z.string().uuid() as a deprecated alias that also checks the RFC version and variant, so the type diff called the site deprecated and fix never selected it. The zod pack's foldFindings marks it breaking and string-format rewrites it to z.guid() without --include-deprecated. Closes UPT-100. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
lucasbraum
deleted the
lucasbraum/upt-100-bench-calmcp-baseline-and-zod-rules
branch
October 18, 2026 00:26
Merged
lucasbraum
added a commit
that referenced
this pull request
Oct 18, 2026
… (UPT-104) (#226) Closes UPT-104. ## What and why `fix` builds the workspaces whose output a consumer reads (`prepareWorkspace`) right after the bump, before any edit, so the agent's type-check sees fresh declarations. Those results were then reported as the run's build and turned into `Build cannot run (…)` fallback reasons, and nothing ever built them again. When the edits fixed the build, the run still: - forced a feedback round (`round === 1 && fallback.length`) with nothing left to fix, - put an **Agent-led draft** banner at the top of the PR body, - showed `Build after the bump … — failed` / `❌ build` in the PR body. That's what happened on calmcp (#219) and on dms-saas (#221's final run, uptide-bench fork PR #8). ## Change (`packages/core/src/fix/run.ts`) - The consumed builds are judged on the head the edits left. They run again before the verdict and after each feedback round, and the reported `verification.build` holds those results. - Build fallback reasons are recomputed from the latest builds after each build, so a reason the head no longer shows is dropped. The reasons that don't depend on the head (planner refusals, agent-led pins, unknown baseline) stay. - The consumed builds run again only when the tree changed since their last result: HEAD's tree plus the diff and untracked names over it, marked `ponytail:` because an untracked file edited in place goes unseen. A run whose after-bump build passed and that made no later edits does not build again. - When the verification build (generic or `requiresBuild` runs) already built the same workspaces on the head, its results are used instead of a separate build. - New report field `buildRuns: { count, ms }`: every build the run executed (preparation and verification) and their wall time. ## Tests `fix.test.ts`, `consumed workspace builds on the final head (UPT-104)`: 1. The after-bump build fails and the fixed head builds: no feedback round, no `agentFallback`, the agent is never called, the build line says `passed`, no banner, and 3 builds (baseline, after the bump, edited head). 2. The after-bump build passes and nothing changes afterwards: no further build (2 builds). 3. A `requiresBuild` pack: the consumed workspace is built once on the head, by the verification build. Tests 2 and 3 each fail with their guard removed. The existing test that counted 2 builds now expects 3, with the third on the edited head. `pnpm test`, `pnpm typecheck` and `pnpm lint` pass; the only lint warnings are the two that predate this change. ## Bench (#169 harness, Uptide arm only) Same harness, model (`claude-sonnet-5-5`, medium) and $1 cap as #169. **Before** is origin/main `2dc2162` plus only the `buildRuns` counter (a local commit, not pushed); **after** is this branch. One run each, sequential. | Run | Rounds | Cost | Wall | Builds (count / time) | Fallback banner | PR body build line | Verdict | |---|---|---|---|---|---|---|---| | calmcp before | 1 | $0.117 | 76 s | 1 / 0.5 s | yes: `Build cannot run (.)`, zodDoc.ts TS2322 | `.` `tsc -p tsconfig.json` — **failed** (`❌ build`) | PASS | | calmcp after | **0** | **$0.082** | **58 s** | 2 / 0.7 s | **none** | `.` `tsc -p tsconfig.json` — **passed** (`✅ build`) | PASS | | dms-saas before | 1 | $0.391 | 300 s | 2 / 8.6 s | yes: `Build cannot run (packages/dms-saas)`, the stripe errors the agent then fixed | `packages/dms-saas` — **failed** (`❌ build`) | FAIL | | dms-saas after | **0** | **$0.294** | **256 s** | 3 / 15.5 s | **none** | `packages/dms-saas` — **passed** (`✅ build`) | PASS | Phases: - calmcp before: analyze 5 s, baseline 3 s, install 6 s, rules 0 s, agent 33 s ($0.089), verify 2 s, **feedback 16 s ($0.028)** - calmcp after: analyze 6 s, baseline 3 s, install 6 s, rules 0 s, agent 35 s ($0.082), verify 2 s - dms-saas before: analyze 23 s, baseline 26 s, install 21 s, rules 0 s, agent 120 s ($0.271), verify 26 s, **feedback 69 s ($0.120)** - dms-saas after: analyze 25 s, baseline 30 s, install 24 s, rules 0 s, agent 128 s ($0.294), verify 35 s The extra build is the head rebuild: +0.2 s on calmcp and +6.9 s on dms-saas. It replaces a feedback round of 16 s / $0.028 and 69 s / $0.120. How to read the dms-saas rows: - **The verdicts differ because of the agent's edits, not this change.** Before, the agent bumped the API version and a basil-payloads test failed as new. After, 3 failing test files were marked pre-existing and the run passed. Both runs left 0 new type errors. - **The first after run on dms-saas failed and is not in the table.** It exited 2 after 423 s with `Required migration evidence exceeds the PR body budget`. A run that throws removes its clone, so no report survived, and its spend went unrecorded (no `--json` output). I reran it once with `--keep-temp` (no other difference): that's the row above, with a 31 KB PR body. The overflow is the open "PR body overflow" follow-up from UPT-101, and I couldn't attribute it to this change: removing a fallback reason only shortens the body. Nothing here fixes it. No fork PRs were opened for these runs, and nothing was pushed outside this branch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Lucas Braum <lucasbraum@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
lucasbraum
pushed a commit
that referenced
this pull request
Oct 18, 2026
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## uptide@0.15.1 ### Patch Changes - [#220](#220) [`cf86d65`](cf86d65) Thanks [@lucasbraum](https://github.com/lucasbraum)! - With an assisting agent, a root pin Uptide cannot update safely (a nested override such as `overrides > pkg > peerDependencies > react`) no longer stops `fix` with exit 2: the run continues as an agent-led draft that leaves the pin as written and names it, with its exact path and value, at the top of the PR body. The agent-led peer reason now names each blocker's first release that accepts the target, or says none does. An agent-led draft must pass the repository's clean install (`npm ci` or the manager's frozen install) before delivery; if it does not, the PR says so at the top. Without an agent, with `--no-llm`, or on a safety refusal, `fix` still exits 2. - [#232](#232) [`e4a2885`](e4a2885) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide fix` spends less wall time on test runs and dependency copies: - The agent's own `run_tests` check runs only its runner's related tests when the `test` script builds first and then runs a script that is just the runner (`pnpm run build && pnpm run test:unit`). The gate after the agent's edits and the final verification run exactly what they ran before. - The upgrade's install moves the installed `node_modules` out of its temporary worktree instead of copying it. - Each phase of the fix report lists the repository's commands it ran (installs, generators, builds, type-checks, tests, lint, format) with their count and summed wall time. An agent-led run's clean install has its own phase. - [#223](#223) [`2dc2162`](2dc2162) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `fix` no longer stops with "baseline tests modified the working tree" when the test run only writes new untracked files, such as a junit `test-report.xml` that is not gitignored. Those files are recorded as test artifacts (`testArtifacts` in the report), kept out of every commit, the agent's diff and the pending-changes check, and removed before each later test run, `uptide verify` included. A change to a tracked file during baseline tests still refuses the run. - [#215](#215) [`abddcb3`](abddcb3) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check` no longer lists possible impact (type changes nothing confirmed as breaking) on its first screen: each package says it in one line, `+N possible impact in M files, not counted as breaking`, and `--details` and `--json` still have every site. A package whose types were partly verified no longer repeats its coverage line in the heading, and a monorepo's unbuilt workspace chains are one count on that line. - [#213](#213) [`2de3a18`](2de3a18) Thanks [@lucasbraum](https://github.com/lucasbraum)! - A companion pack that moves with the lead's fix (react beside next) now gets its own scan context and inventory, and the PR body carries a section with its review notes. - [#212](#212) [`733cfbe`](733cfbe) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `pack test --write` records how many breaking sites a finding landed on at any severity and as breaking, out of all breaking sites it scored, and the pack tables show both ("Found, any severity", "Found as breaking") in place of the single recall column. Packs recorded before it show `—`. - [#224](#224) [`09f68df`](09f68df) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `fix` on npm no longer stops when the lock-only install hits ERESOLVE from the old lockfile. It retries without the lockfile entries of the moved packages, their companions and the packages whose peers they satisfy, and only then with the whole lockfile re-resolved; the report says which step worked, and the lockfile check reports the churn as usual. It never uses `--legacy-peer-deps` or `--force`. When every step fails, `fix` ends blocked, naming the conflicting packages from npm's ERESOLVE report. `fix --json` now prints JSON on every exit, a failure included; when the run already has its report (a PR body over GitHub's size limit), that report comes with the error. - [#221](#221) [`ff8de8e`](ff8de8e) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `fix` no longer rejects an agent patch for test files the parent commit already failed, whichever test scope ran them: when the test script builds first and the build failed on the error being fixed, the start had no test results to compare with, so a correct patch was reverted every round. The patch's tests also run with the pack's follow-ups in place (a test asserting the old Stripe API version), which `fix` still applies and reports after the agent. When a PR body cannot fit, the stored report.json the error points to is now written first. `check` and `fix` now also analyze files that reach a package's wrapper through a barrel (`import { getStripeClient } from '../stripe'` with `export * from './client'`). Edited files are formatted with oxfmt and dprint as well as biome and prettier, from the workspace that holds them, with its own configuration, else the root's. The stripe pack adds `request-options`: request options passed in the params slot (`subscriptions.cancel(id, { idempotencyKey })`), which SDK 22 rejects, move to the third argument. It runs assisted until a ground-truth repository exercises it. - [#226](#226) [`448c3cb`](448c3cb) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide fix` judges consumed workspace builds on the final head, not on the tree right after the bump. A build that failed after the bump and passes once the edits land no longer starts a feedback round, adds an "Agent-led draft" reason or shows "Build failed" in the PR body. The builds run again only when the tree changed since their last result, and not at all when the verification build already built those workspaces. The fix report's `buildRuns` gives the number of builds the run executed and their wall time. - [#218](#218) [`13e343b`](13e343b) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check` folds a compile error into the breaking site it comes from when the compiler traces its value there (an `errors.map(e => e.message)` under a `result.error.errors` that zod 4 no longer types, and the narrowing that depended on it). The rule's line shows `+N downstream errors`, `--details` lists them under their site, and `--json` carries them as that finding's `downstream`. They are no longer separate breaking sites. `fix` still verifies against every error. - [#214](#214) [`195ae10`](195ae10) Thanks [@lucasbraum](https://github.com/lucasbraum)! - A rewrite rule below 80% precision over at least 5 scored ground-truth sites now runs as assisted in `fix` (react-router `dom-imports`); smaller samples (next `middleware-proxy`) stay mechanical and are flagged in the compatibility table. Pack status is unchanged. - [#218](#218) [`13e343b`](13e343b) Thanks [@lucasbraum](https://github.com/lucasbraum)! - pack test runs a ground-truth repository's code generators (prisma generate, local graphql-codegen) after its install, as fix does. It fetches the Prisma engines the locked version needs itself, so generators stay off the network. A repository whose generator still fails is incomplete and fails the pack, unless its ground truth declares `withoutGenerated` - [#227](#227) [`ae175a9`](ae175a9) Thanks [@lucasbraum](https://github.com/lucasbraum)! - A migration whose evidence is larger than GitHub's PR description limit no longer stops `fix` with "Required migration evidence exceeds the PR body budget" and loses the run. Long required lists get a compact form when they do not fit. Lockfile churn shows counts per kind of change and the first packages. Manual sites are grouped by rule, with counts and the first locations. Remaining type errors show the first entries. Every full list stays in report.json and, in the Uptide Action, in the job summary, which the description links to. If even the compact form does not fit, the description is a short summary that says so at the top: the verdict, the checks, the counts and where the full report is. The branch and report.json are always kept. An agent's decisions under "Decisions for you" name the files the agent changed, five at most, and count the rest, instead of every file in the task's scope. A verification-feedback task scoped to the whole repository made each of its decisions a list of hundreds of files. - [#219](#219) [`12dc612`](12dc612) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `fix zod` rewrites `z.string().uuid()` to `z.guid()` by rule without `--include-deprecated`: zod 4's `uuid()` also checks the RFC version and variant, so ids zod 3 accepted stop validating, and `check` now counts the site as breaking. `fix` no longer reports false pre-existing type errors on a TypeScript 7 repository whose `lib` names a library newer than its `target` (`lib: ["ES2023"]` under `ES2022`). Co-authored-by: uptide-release[bot] <338861287+uptide-release[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes UPT-100.
On calmcp (zod 3.25.76 → 4.6.5, from the #169 bench), Uptide reported 5 pre-existing type errors on a repository whose own
tscis clean at the parent commit. It also sent every site to the agent.1. False baseline errors (TypeScript 7 libs in the fix type-check)
calmcp runs TypeScript 7.0.2, which ships no compiler API and no
lib.*.d.ts, sofixtype-checks it with the bundled compiler. Its tsconfig hastarget: "ES2022"andlib: ["ES2023"]. The fix phase's host (verify.tsbundledHost) served only the libs a probe program with that target had loaded, andlib.es2023.d.tswas not one of them. That produced TS6053 (/__uptide_lib__/lib.es2023.d.tsnot found) and 4 × TS2550 (Object.hasOwn). UPT-73 fixed the same gap in the analysis (compile.ts), but not in the fix phase's own checker.Fix:
bundledHostloads any lib by name throughcompile.ts'sbundledLib, the same source the analysis uses. Fixture:verify.test.ts, a TS 7-style install withlib: ["ES2023"]undertarget: "ES2022"usingObject.hasOwn. It failed before the fix (TS6053, TS2304Object) and passes now.2. No zod rule fired (
uuid()→guid())Reach and import shape were fine:
checkfoundsrc/tools/create.ts:20 z.string().uuid()asZodString#uuid, fixabilitymechanical. The type diff calls it deprecated, though, andfixselects only breaking sites unless--include-deprecatedis set, sostring-formatnever saw it. The agent then changed it during the verification-feedback round, after 4 tests failed.But
uuid()is a behavior change, not only a deprecation: zod 4'suuid()also checks the RFC version and variant, so ids zod 3 accepted stop validating. That is what broke calmcp's tests.Fix: the zod pack's
foldFindingsmarksZodString#uuidsites breaking (evidencepack, with that reason).string-formatrewrites them toz.guid()without the flag. The other formats (email, url, datetime, base64) stay deprecated and flag-gated. The rule id is unchanged, so ground truth and per-rule scores still line up. Fixture:zod.test.ts, the calmcp shape (const uuid = z.string().uuid()) is promoted and rewritten flagless, andemail()is not..errors→.issuesdid not fire because calmcp has no.errorsreads (nothing to match), not because of a bug.Bench: calmcp, Uptide arm only, before / after
Same harness (
bench/copied from the #169 branch, uncommitted, unchanged), same model (claude-sonnet-5-5, medium), same caps ($1 Uptide budget), same parent commit, andbench/check.shfor the final checks. Before = this branch's base (origin/main 195ae10). After = this branch. One run each.check.sh)create.ts:20bystring-format)uuid()→guid()made byPhases (ms, cost):
The 5 agent-led sites in
src/tools/zodDoc.tsare type-level code over zod internals (ZodTypeAnyparameters,_def,.description). They match the zod pack'stypesrule, which is agent work by design, so they stay agent-led.Not fixed here (follow-up)
The remaining feedback round in the after run is spurious.
verification.buildkeeps the build result from right after the bump (tsc -p tsconfig.jsonfailing onzodDoc.ts), not the build at the final head. At the final head,tsc -p tsconfig.jsonexits 0. The feedback agent got that stale error, found nothing to fix and made no edits. The PR body also says "Build … failed". The before run (and the #169 rerun, "❌ build") has the same defect.Checks
pnpm --filter @uptide/core exec vitest run: 186 files, 2253 passedpnpm typecheck,pnpm exec biome check .: clean (existing warnings only)uuid()sites in the ground truth now count as breaking instead of deprecated, which may move zod's breaking counts inverification.json. Not run locally.No fork PRs were made for these two runs. Nothing was pushed outside this repository.
🤖 Generated with Claude Code