bench: UpgradeBench v0 pilot harness (UPT-24) by lucasbraum · Pull Request #169 · uptide-dev/uptide

lucasbraum · 2026-10-16T05:28:40Z

UPT-24 pilot: the UpgradeBench harness, run once on 3 held-out repositories × 2 arms. It is an internal test of the harness, not a result to publish. Only the scripts are committed; the results below are not.

Harness (`bench/`)

selection.json: 3 public repositories, each with a human's merged upgrade in a verified pack's range (zod, stripe, react). None is in any pack's ground-truth.json (excluded.json, 63 repos). It was committed in bf88ead, before either arm ran.
run.sh <claude|uptide|human> <id>: both arms start from the same fresh clone of the parent, installed with lifecycle scripts off before the clock starts, with pushes to the original repository disabled.
- Both use Uptide's default executor model, claude-sonnet-5-5 at medium effort.
- Claude Code gets the brief's prompt verbatim, with claude -p --bare and Bash only inside Claude Code's sandbox (claude-sandbox.json).
- Uptide runs fix <pkg> --target <B> from this branch's build, with default settings: SDK executor, $1 budget.
- human checks out the human's commit as the reference.
check.sh: the repository's own install, type-check, lint and tests on a clean worktree of the run's head.
judge.sh: a fork in uptide-bench whose default branch is reset to the parent, the run pushed as bench/<run>, and a PR on the fork. Pushes go to the fork's explicit URL only.
score.mjs: one scorer for every run, blind to arm. It never reads key.tsv, and finds the reference by commit sha. --unblind joins the arms in afterwards. score.test.mjs covers the hunk logic.

Results (1 run per pair)

repo	arm	exit	coverage, src	coverage, all	local install / types / lint / tests	fork CI	cost	wall
calmcp, zod 3.25.76→4.6.5	human		16/16	27/27	ok / ok / ok / ok	`build` fails at `npm audit` (new advisory)
	Uptide	1	6/16	6/27	ok / ok / ok / 4 fail	same as human	$0.07	51 s
	Claude Code	0	6/16	6/27	ok / ok / ok / ok	same as human	$0.10	43 s
dms-saas, stripe 18.5.0→22.6.2	human		32/32	51/51	ok / ok / ok / ok	pass
	Uptide	1	9/32	10/51	ok / 1 error / format / ok	fail (lint)	$0.24	161 s
	Claude Code	0	15/32	16/51	ok / ok / ok / ok¹	pass	$0.27	115 s
enki, react 18.3.1→19.2.0	human		29/29	29/29	ok / ok / ok / ok	fails only at SonarQube (secret)
	Uptide	2	0/29	0/29	no changes	no PR	$0.00	9 s
	Claude Code	0	10/29	10/29	`npm ci` fails (ERESOLVE)	fails at `npm ci`	$0.13	94 s

Coverage counts the human's hunks whose old lines the run also changed; "src" leaves out tests and scripts.
Unrelated changes: 0 for every run.
Cost is priced with Uptide's table for both arms.
¹ One MongoDB-backed test timed out at 5 s on the first full pass. It passed alone twice and in a full re-run.

What each run did:

calmcp: both arms fixed the type-level zod 4 changes in zodDoc.ts. Only Claude Code made the uuid()→guid() behavior change. Uptide's PR says ❌ tests and "Behavior not checked".
dms-saas: Uptide left one subscriptions.cancel site with idempotencyKey in the params (TS2353) and an unformatted test. Its PR said "1 new, Verification: FAIL".
enki: Uptide stopped before cloning. Claude Code bumped react-leaflet but kept @entur/react-component-toggle@0.0.3 behind an override that npm ci rejects.

Escaped regression candidates (missed human hunks that every check passes, read by hand):

calmcp, both arms: the .strict() the human added to 5 MCP tool schemas. zod 4's JSON Schema no longer emits additionalProperties: false, so tool inputs now advertise that unknown keys are allowed.
dms-saas, both arms: billing_mode: classic for new subscriptions, the narrowed credit-note enums and the checkout managed_payments change. These compile and pass tests.

States what it did not verify: both arms, every run. The regex in score.mjs matches everything, so this is my reading:

Claude Code named the tests the sandbox blocked and the runtime checks it skipped.
Uptide's verification table marks ⚠️/❌ per check and lists decisions for the owner, such as moving webhook endpoints to the new API version.

Spend: about $0.81 for the six runs, plus $0.28 for sandbox probes and three runs discarded after the harness fixes below. That is about $1.10 of the $30 cap.

Uptide issues the pilot found

The compiler host misses lib.es2023.d.ts (/__uptide_lib__/…, TS6053) on a TypeScript 7 project. That reports 5 baseline errors that do not exist (calmcp is tsc-clean).
No workspace-dependency build before the baseline. dms-saas's @antelopejs/interface-dms-saas resolves through dist, so 31 of its 48 "baseline" errors are missing modules and types end "not verified (type resolution failed)".
A false companion conflict: "@entur/tokens 3.20.0 has no release that agrees with @entur/react-component-toggle 1.0.0", although toggle 1.0.0 depends only on react and react-dom. fix react exits 2 with no next command.
The zod pack has no rule for uuid() strictness, and the stripe agent missed one cancel site that a compile error pointed at.

What should change before the full run (10 repos × 3 runs)

Unfair conditions between arms:

The arms run under different isolation. Claude Code's sandbox could not start MongoMemoryServer (11 test files unrunnable) and, before allowLocalBinding, supertest. Uptide runs unsandboxed. Run both arms in the same container or VM, with no Claude-specific sandbox; inside it, --dangerously-skip-permissions is acceptable. Auto mode refuses it on the host.
Uptide refuses an uninstalled checkout, so the harness now installs the parent for both arms (untimed). Keep that and state it.
Claude Code 2.1.283 does not know claude-sonnet-5-5. It reports cost at older Sonnet rates (about 1.8×), and enforces --max-budget-usd at those rates. The harness reprices from modelUsage.
Decide in advance what happens when Uptide stops with a printed next command (peer plans), and whether a human commit with a Co-Authored-By: Claude trailer (calmcp) counts as the human answer.

Fork CI problems:

calmcp's CI fails on the human reference too: npm audit sees an advisory published after the commit.
enki's fork-PR workflow skips a PR made inside the fork, and its internal one needs a Sonar secret.
dependency-review fails on forks.
Selection rule: open the human reference PR first, and keep a repository only if its CI passes.
score.mjs compares CI at check level; it should compare the failing step (enki's two failures are different steps).

Flaky steps:

One MongoDB-backed test timed out at 5 s under load. Re-run failing tests once and record the flake.
gh search commits hit its secondary rate limit after about 6 queries while selecting.

Metrics:

Hunk overlap penalises a different valid fix and counts the human's own new tests.
Better: apply the human's test changes to each run's head and run them. That is an automated "escaped regressions vs the human answer".
Replace the "states unverified" regex with a rubric.

Cost per run: $0.07–$0.27, so the full run is about $12–$20 at this model.

Rerun on current main (2026-10-17), same 3 repos, 1 round

The first run above stays as it was. Same harness (bench/ from this branch, unchanged), model (claude-sonnet-5-5, medium), budget caps, starting commits and judging as the first run. Uptide is built from origin/main at 195ae10 (fix --target, SDK executor, $1 budget). Claude Code is 2.1.283 with the same sandbox settings. No harness, scoring or ground-truth changes were needed. Forks are in uptide-bench only: calmcp Uptide, calmcp Claude, dms-saas Uptide, dms-saas Claude, enki Claude.

repo	arm	tests	types / lint	sites left unfixed	files changed (src + lock/manifest)	coverage, all	fork CI	cost	wall
calmcp, zod 3.25.76→4.6.5	Uptide	pass	pass / pass	0	2 + 2	7/27	same as human	$0.16	97 s
	Claude Code	pass	pass / pass	0	2 + 2	7/27	same as human	$0.11	47 s
dms-saas, stripe 18.5.0→22.6.2	Uptide	pass²	1 error / format (2 files)	1 (`idempotencyKey` in a `subscriptions.cancel`, `workspaces-admin.ts:78`)	8 + 2	10/51	fail	$0.69	391 s
	Claude Code	pass²	pass / pass	0	8 + 2	13/51	pass	$0.31	149 s
enki, react 18.3.1→19.2.0	Uptide	no changes	no changes	n/a, stopped	0	0/29	no PR	$0.00	9 s
	Claude Code	no (`npm ci` fails, ERESOLVE)	not run	0 by its own run, but the install fails	7 + 2	7/29	fail (`build_and_preview`, as for the human)	$0.11	67 s

² test/private-admission.test.ts times out at 5 s (MongoMemoryServer) on the first full pass, on the human reference and both arms alike. It passes 16/16 alone on all three. Same flake as the first run; treated as a pass.

Uptide, per run

calmcp: 5 sites, 0 mechanical, 5 agent-led (types), 1 verification-feedback round. agentFallback is set: the zod pack had no mechanical rule to apply (rules phase 1 ms) and the build did not compile after the mechanical step, so it went agent-led. Verification: ✅ tests, ❌ build, "Behavior ⚠️ not checked". It also still reports 5 baseline type errors on a tsc-clean repo (Object.hasOwn, issue 1 below is not fixed).
dms-saas: 11 sites, 2 mechanical, 9 agent-led, 3 verification-feedback rounds. agentFallback is set: the build failed on 8 type errors after the mechanical step. It ended with 1 type error and a format failure in webhook-credit-notes.ts and its test, and its PR says ❌ type-check ❌ build ❌ tests.
enki: 0 sites, 0 rounds, no fallback. It stopped before cloning: package.json overrides > @reach/dialog > peerDependencies > react may pin react, but its selector or value cannot be updated safely. Nothing was changed.

Per repo

calmcp: Claude Code, narrowly. Identical result on every check and on coverage; Claude Code is cheaper ($0.11 vs $0.16) and twice as fast. Neither added the .strict() the human did.
dms-saas: Claude Code. It compiles, lints and passes fork CI; Uptide leaves a cancel site and unformatted files, and costs 2.2× as much and 2.6× the time.
enki: neither. Claude Code has a diff but its lockfile cannot be installed (@entur/react-component-toggle@0.0.3 keeps a react@^18.3.1 peer); Uptide made no change.

What changed in Uptide versus the first run

calmcp: better. Exit 1 with 4 failing tests became exit 0 with all checks green, and it now makes the uuid()→guid() change it missed before (coverage 6/27 → 7/27). Cost $0.07 → $0.16, wall 51 s → 97 s.
dms-saas: same defect, higher price. It still leaves one subscriptions.cancel site with idempotencyKey (a different file) and an unformatted test, and fork CI still fails on lint. Coverage 10/51 either way. Cost $0.24 → $0.69, wall 161 s → 391 s (3 feedback rounds).
enki: still no PR, different reason. The false @entur/tokens companion conflict is gone; it now refuses on the @reach/dialog peer override. Exit 2 both times.

Claude Code, for comparison with its first run: calmcp 6/27 → 7/27; dms-saas 16/51 → 13/51, same pass; enki 10/29 → 7/29 and still an npm ci ERESOLVE. Cost and wall time moved by tens of percent either way, which one round cannot separate from noise.

Spend: $0.85 Uptide + $0.53 Claude Code = $1.38 for the six runs, no discarded runs. The harness copied from this branch ran unchanged on a worktree of main (bench/ is not on main).

🤖 Generated with Claude Code

…uth (UPT-24) Three public repositories with a human's merged upgrade in a verified pack's range (zod, stripe, react), recorded before either arm runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

… pilot (UPT-24) run.sh starts both arms from the same fresh clone of the parent, with lifecycle scripts off and pushes disabled; the Claude Code arm runs Bash only inside Claude Code's sandbox. check.sh runs the repository's own checks, judge.sh opens a pull request on an uptide-bench fork, and score.mjs scores every run the same way without reading its arm. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

…e (UPT-24) Probes showed headless Claude Code denying git commit and compound commands that need approval; every Bash call already runs inside the OS sandbox, so all Bash is allowed. Claude Code reports an unknown model's cost at older rates, so its tokens are priced with Uptide's table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

… sandbox (UPT-24) The first run showed Uptide refusing an uninstalled checkout (a developer's is installed) and the sandbox failing supertest's local listen, so Claude Code could not run tests Uptide runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

…s without commits (UPT-24) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

…T-100) (#219) Closes UPT-100. On calmcp (zod 3.25.76 → 4.6.5, from the #169 bench), Uptide reported 5 pre-existing type errors on a repository whose own `tsc` is clean at the parent commit. It also sent every site to the agent. ## 1. False baseline errors (TypeScript 7 libs in the fix type-check) calmcp runs TypeScript 7.0.2, which ships no compiler API and no `lib.*.d.ts`, so `fix` type-checks it with the bundled compiler. Its tsconfig has `target: "ES2022"` and `lib: ["ES2023"]`. The fix phase's host (`verify.ts` `bundledHost`) served only the libs a probe program with that target had loaded, and `lib.es2023.d.ts` was not one of them. That produced TS6053 (`/__uptide_lib__/lib.es2023.d.ts` not found) and 4 × TS2550 (`Object.hasOwn`). UPT-73 fixed the same gap in the analysis (`compile.ts`), but not in the fix phase's own checker. Fix: `bundledHost` loads any lib by name through `compile.ts`'s `bundledLib`, the same source the analysis uses. Fixture: `verify.test.ts`, a TS 7-style install with `lib: ["ES2023"]` under `target: "ES2022"` using `Object.hasOwn`. It failed before the fix (TS6053, TS2304 `Object`) and passes now. ## 2. No zod rule fired (`uuid()` → `guid()`) Reach and import shape were fine: `check` found `src/tools/create.ts:20 z.string().uuid()` as `ZodString#uuid`, fixability `mechanical`. The type diff calls it **deprecated**, though, and `fix` selects only breaking sites unless `--include-deprecated` is set, so `string-format` never saw it. The agent then changed it during the verification-feedback round, after 4 tests failed. But `uuid()` is a behavior change, not only a deprecation: zod 4's `uuid()` also checks the RFC version and variant, so ids zod 3 accepted stop validating. That is what broke calmcp's tests. Fix: the zod pack's `foldFindings` marks `ZodString#uuid` sites breaking (evidence `pack`, with that reason). `string-format` rewrites them to `z.guid()` without the flag. The other formats (email, url, datetime, base64) stay deprecated and flag-gated. The rule id is unchanged, so ground truth and per-rule scores still line up. Fixture: `zod.test.ts`, the calmcp shape (`const uuid = z.string().uuid()`) is promoted and rewritten flagless, and `email()` is not. `.errors` → `.issues` did not fire because calmcp has no `.errors` reads (nothing to match), not because of a bug. ## Bench: calmcp, Uptide arm only, before / after Same harness (`bench/` copied from the #169 branch, uncommitted, unchanged), same model (`claude-sonnet-5-5`, medium), same caps ($1 Uptide budget), same parent commit, and `bench/check.sh` for the final checks. Before = this branch's base (origin/main 195ae10). After = this branch. One run each. | | before | after | |---|---|---| | exit | 0 | 0 | | install / typecheck / lint / test (`check.sh`) | pass / pass / pass / pass | pass / pass / pass / pass | | baseline type errors reported | **5** (4 × TS2550, 1 × TS6053) | **0** | | sites: mechanical / agent-led | **0 / 5** | **1 / 5** (`create.ts:20` by `string-format`) | | `uuid()` → `guid()` made by | agent, in the feedback round after 4 failing tests | rule, before the agent | | verification-feedback rounds | 1 ($0.137, 69 s) | 1 ($0.027, 16 s, no edits) | | cost | $0.197 | $0.106 | | wall time (run.sh) | 148 s | 89 s | | files changed (src + lock/manifest) | 2 + 2 | 2 + 2 | Phases (ms, cost): | phase | before | after | |---|---|---| | analyze | 8 847 | 11 853 | | baseline | 3 792 | 5 152 | | install | 10 603 | 12 056 | | rules | 1 | 649 | | agent | 40 574, $0.060 | 34 041, $0.079 | | verify | 6 802 | 4 206 | | feedback | 69 262, $0.137 | 16 000, $0.027 | The 5 agent-led sites in `src/tools/zodDoc.ts` are type-level code over zod internals (`ZodTypeAny` parameters, `_def`, `.description`). They match the zod pack's `types` rule, which is agent work by design, so they stay agent-led. ## Not fixed here (follow-up) The remaining feedback round in the after run is spurious. `verification.build` keeps the build result from right after the bump (`tsc -p tsconfig.json` failing on `zodDoc.ts`), not the build at the final head. At the final head, `tsc -p tsconfig.json` exits 0. The feedback agent got that stale error, found nothing to fix and made no edits. The PR body also says "Build … failed". The before run (and the #169 rerun, "❌ build") has the same defect. ## Checks - `pnpm --filter @uptide/core exec vitest run`: 186 files, 2253 passed - `pnpm typecheck`, `pnpm exec biome check .`: clean (existing warnings only) - The Packs workflow re-scores zod on this PR: `uuid()` sites in the ground truth now count as breaking instead of deprecated, which may move zod's breaking counts in `verification.json`. Not run locally. No fork PRs were made for these two runs. Nothing was pushed outside this repository. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Lucas Braum <lucasbraum@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…ate (UPT-102) (#220) UPT-102. From the UpgradeBench rerun (#169): on entur/enki (react 18.3.1 → 19.2.0), `fix` exited 2 before cloning because a nested override "can't be updated safely", with no agent fallback. **1. Unsafe root pin → agent-led draft.** `planRootPins` now collects every pin it cannot update safely (nested or selector keys, non-version values) into one `UnsafeRootPinError` and names them all. With an assisting agent, the run continues on #188's `agentFallback` path. The pins are left as written, and each one is named with its exact path and value in the reason at the top of the PR body. `pinConsentFallback` became `assistedRootPins`, shared by `isolatedFix` and `fix`. Exit 2 is unchanged without an agent, with `--no-llm`, with `updatePins: false`, and on `SafetyRefusal`. **2. Peer blockers named.** The preflight already picked the lowest accepting release (`compatiblePeerVersion`). The agent-led reason only listed names. It now says `X 0.0.3 → 1.0.0, the first release that accepts react 19.2.0`, or `no X release accepts …; 0.0.3 kept with a peer override`. **3. Clean-install gate.** Before delivery, an agent-led draft runs the repo's frozen install (`npm ci --ignore-scripts …`, the manager's equivalent elsewhere) in the fix clone, on the final branch. A failure goes right under the reason (`This branch does not install.`), fails `verification.passed`, and appears as an `Install` row in the terminal summary. Verified runs skip it: the upgrade step already runs the same install, and nothing after it may touch manifests or lockfiles. Tests (fixtures): an unsafe pin refuses without an agent or with `--no-llm`; with an agent it becomes a draft with the pin unchanged and named in the first lines. A failing clean install heads the PR and fails verification. The peer reason names the first accepting release when a later one rejects the target, or says none does. A root-pins unit test covers naming all unsafe pins and leaving them while the safe ones update. Two existing tests updated: the reason wording, and one extra install call. **enki rerun** (same harness, Uptide arm only, claude-sonnet-5-5, $1 cap): the pin refusal is gone. It now stops at a different, pre-existing guard: enki's vitest config writes `test-report.xml` (vitest-sonar-reporter, not gitignored), so `baseline tests modified the working tree`. No draft was delivered and $0 was spent. Checked locally on the same tree, the reasons this PR adds read: toggle 0.0.3 → 1.0.0 and react-leaflet 4.2.1 → 5.0.0 as the first releases accepting react 19.2.0, plus seven `@reach/*`/`@azure/msal-react` overrides left unchanged. The human's merged upgrade made the same choices. The clean-install gate did not run on enki because the run stopped earlier. The baseline-artifact stop is a follow-up. Not done: the agent does not edit the override. `package.json` and lockfiles stay protected from the agent, and changing them would need a reinstall loop. The draft leaves the pin and asks the reviewer to check it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Lucas Braum <lucasbraum@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…-101) (#221) Closes UPT-101. ## What and why On the dms-saas run in the UpgradeBench pilot (#169), Uptide left a `subscriptions.cancel` type error after 3 feedback rounds. It also left an unformatted file, so fork CI failed. This PR fixes both, plus two more defects the reruns exposed. ### 1. Why the cancel site survived: the patch was rejected It wasn't excluded as pre-existing, and the round budget didn't run out. The agent fixed `workspaces-admin.ts` correctly in every round, and the gate rejected the patch every time. - **The gate compared against the round start only.** dms-saas's `test` script runs `pnpm run build` first. At the round's start the build failed on the error being fixed, so no test file failed there. After the patch the build passed and vitest showed the 4 test files that fail at the parent too, which looked new. `judge` (`sdk-agent.ts`) now also marks failures against the parent baseline, the way final verification does. It compares test files whatever scope ran them: the gate runs related tests, the parent ran the full suite. A run that names no test file never matches across scopes. - **The site was never planned.** `workspaces-admin.ts` imports `getStripeClient` from the `../stripe` barrel (`export * from "./client"`), two hops from `stripe`. Root selection took one hop (`scopedImporters`). It now also takes the importers of a barrel among the direct importers, under the same memory cap, which still reports what it drops. `check` now reports the site. - **Exposed by the first rerun: the site batch's gate failed the pin assertion.** Once every site compiled, the batch failed `test/stripe-basil-payloads.test.ts`, which asserts the old API version. The pack's follow-up updates that assertion, but only after the agent. The earlier runs only got past this because the build kept failing. The gate now runs the batch's tests with the pack's follow-ups in place, then puts them back. `fix` still applies and reports them as before. - **Mechanical rule: `request-options`** (stripe pack, with fixtures). SDK 18–21 have both `(id, params?, options?)` and `(id, options?)`. SDK 22 drops the second, so `cancel(id, { idempotencyKey })` type-checks the options as `SubscriptionCancelParams` (TS2353). When the literal holds only RequestOptions keys and is the last argument of a method call, the rule passes `undefined` as params, as the human upgrade did. A literal that mixes params and options stays assisted. No ground-truth repo exercises the rule (dms-saas is held out of ground truth by the bench), so under UPT-48's gate it runs **assisted** with its guide. `verification.json` lists it as unexercised, and the docs tables were regenerated. ### 2. Formatting dms-saas formats with **oxfmt**, configured at the root and again in `packages/dms-saas`. It uses none of prettier, biome or dprint. `style.ts` now detects oxfmt and dprint as well. It runs each tool from the package that holds the file (nearest `package.json`), with that package's config and binary, else the root's, as UPT-74's builds do. It runs in the fix clone on the edited files only. A formatter that fails isn't thrown. The lint after it checks the same files, and the PR body reports a FAILED lint. In the after run the `style` commit touched one file, one the migration had already edited. ### Also `fix` refused the oversized PR body with "review the full report.json", but the report was written after the body. It's now written first. ## Bench: dms-saas, Uptide arm, before / after Same harness (`bench/` from the pilot branch, not committed), `claude-sonnet-5-5`, effort medium, $1 limit. Checks are the repository's own commands via `bench/check.sh`. | | Before: 195ae10, run 2530720c | After: c745ed6, run 0503a533 | | --- | --- | --- | | `uptide fix` exit | 1 | 0 | | Type-check | ✗ 1 error, TS2353 `workspaces-admin.ts:78` | ✓ | | Lint / format | ✗ oxfmt, 1 file | ✓ | | Unit tests (local) | 1 file fails: `private-admission`, a 5 s Mongo timeout that fails on every arm, human included | same | | Fork CI | ✗ [uptide-bench/AntelopeJS--dms-saas#7](uptide-bench/AntelopeJS--dms-saas#7) (lint) | ✓ [uptide-bench/AntelopeJS--dms-saas#8](uptide-bench/AntelopeJS--dms-saas#8) | | Cost | $0.60 | $0.40 | | Wall time | 519 s | 310 s | | Feedback rounds | 3, all rejected | 1, no change | For comparison, the Claude Code arm in the #169 rerun: $0.31 / 149 s, clean. ### Where the time went (UPT-14 phases) | Phase | Before | After | | --- | ---: | ---: | | analyze | 34 s | 27 s | | baseline | 29 s | 30 s | | install | 27 s | 22 s | | rules | 0.1 s | 0.1 s | | agent | 71 s · $0.24 | 125 s · $0.29 | | verify | 10 s | 27 s | | feedback | 330 s · $0.36 | 66 s · $0.11 | Before, 64% of the run was the three rejected feedback rounds. After, the agent handles all 10 sites in one accepted batch. The remaining 66 s round is waste (see follow-ups). ### Runs There were four runs. The before run. A first after run (`6cdcd1e5`, at 19dae27) that exited 2: the site batch was rejected (pin assertion, third bullet above), every site went manual, and the required PR-body evidence went over 60 kB. A diagnostic `fix --keep-temp` in a scratch clone of the same commit, outside the harness, to get the stored report. Then the final after run. Total spend ≈ $2.3: $0.60 + ~$0.65 for 6cdcd1e5 (not recorded, since the run ended before writing its JSON) + $0.65 + $0.40. ## How it was verified Each new test fails without its fix and passes with it: - `sdk-agent.test.ts`: the parent fails the same file with another command while the round-start build fails, so the batch is accepted; a new file or an unnamed failure is still rejected. The batch's tests also run with the pack's follow-ups in place, and the follow-up file is put back. - `memory.test.ts`: the importers of a barrel that re-exports a root are taken; one importing only a non-dependent name isn't. - `style.test.ts`: oxfmt is configured in a workspace and at the root, runs from each package with its files, and an untouched file stays untouched. - `stripe.test.ts` plus `fixtures/request-options`: 2 sites rewritten, a mixed params-and-options literal kept, and an `iterations` TS2353 isn't claimed. `pnpm lint`, `pnpm typecheck`, and core and CLI tests pass (core: 2257 passed). `pnpm docs:packs` regenerated the tables, and `public:check` is clean. ## Not done / follow-ups - **Stale build fallback.** The post-install workspace rebuild of `packages/dms-saas` fails because its code doesn't compile at the target yet. That failure stays in `agentFallback` after the migration fixes it. It triggers the wasted 66 s / $0.11 round and puts a false "Agent-led draft" banner on uptide-bench/AntelopeJS--dms-saas#8. The rebuild should be re-run on the migrated tree before it counts as a fallback. - **A run where every site ends manual still can't publish.** The diagnostic report rendered more than 60 kB of rank-0 evidence. The report is kept now, but the body still refuses. - **`request-options` stays assisted** until a ground-truth repository with that shape is added. ## Checklist - [x] Tests for the change (a bug fix starts with a failing test) - [x] `pnpm lint && pnpm typecheck && pnpm test` pass - [x] README and docs updated if behavior changed - [x] Changeset added if a user would notice (`pnpm changeset`) - [x] No new dependency, or it was agreed in an issue - [x] No private code, names or paths in fixtures, tests or docs - [x] The rules in CONTRIBUTING.md ("Rules that do not bend") still hold 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Lucas Braum <lucasbraum@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…T-105) (#223) Closes UPT-105. Refs UPT-102 (not closed: see the enki rerun below). ## What and why On enki (UpgradeBench, #169), the vitest junit reporter writes `test-report.xml`, which the repository does not gitignore. The baseline-tests guard treated any untracked file as a change, so `fix` stopped with "baseline tests modified the working tree" before doing anything. Now: - Untracked files are snapshotted before the baseline tests (`untrackedFiles`). After the run, a change to a **tracked** file still refuses, as before. **New untracked** files are test artifacts (`testArtifacts` in `verify.ts`). - Artifacts go in the private clone's `.git/info/exclude`, anchored at the repo top (with the project's subdirectory prefix, and pattern characters escaped). No status, diff, `git add` or `ls-files --exclude-standard` sees them. That keeps them out of commits, the agent's diff (including tests the agent runs itself), the follow-up and style steps, and the `verificationPending` check, from one place. - They are removed right away, and again before each later test run: `clearing()` wraps `services.tests` once, after baseline, so settle, the rules' transactions, the agent gate and the feedback rounds all go through it. - The report records them (`testArtifacts`). `uptide verify` runs in a fresh clone, so it re-applies the exclude and clearing from the stored run; otherwise its final test run would leave the file and fail. - The same applies to the Stripe pin path (`pin.ts`). `fix` and `verify` always run in a private clone (`isolatedFix`/`isolatedVerify`), so writing `info/exclude` never touches the user's checkout. The comment there names the ceiling if either ever runs in place. ## Tests - `fix.test.ts`: the fixture's test script writes an untracked `test-report.xml` and **fails if the previous run's file is still there**. Baseline and final verification pass, `testArtifacts` is `['test-report.xml']`, the file is in no commit and not in `changedFiles`, nothing is pending. `uptide verify` from a fresh clone (no exclude entry, no file on disk) also passes. - `fix.test.ts`: a baseline test run that writes an untracked report **and** changes a tracked file still refuses. - `verify.test.ts`: a project in a subdirectory, with names containing `[ ] # ! space`, is hidden and nothing else is. - Reverting the run change brings back the original refusal. Reverting only the reverify change fails the verify assertion. - `pnpm lint`, `pnpm typecheck`, `pnpm test`: green locally (core 2268 passed). ## enki rerun (#169 harness, Uptide arm only, claude-sonnet-5-5, $1 cap) Run `7f90b41c`, from this branch's build (8d1d8bb). | | | |---|---| | Baseline tests | **passed the guard** (this fix) | | Stopped at | lockfile install of react 19.2.0 + `@entur/react-component-toggle` 1.0.0: `npm install --package-lock-only` → `ERESOLVE`, exit 2 | | Clean-install check (#220) | not reached: no draft | | Checks | none on a draft: head is the parent commit | | Cost | $0.00 (no agent call) | | Wall time | 54 s | | Fork CI | none: no commits, so no fork pull request | The cause, reproduced outside Uptide: starting from the parent's `package-lock.json`, `npm install --package-lock-only` fails with the same `ERESOLVE` **even with the human upgrade's exact package.json** (ed37dd2). Without the old lockfile, the same manifest resolves (exit 0). So Uptide's companion set matches the human's. What blocks is npm's resolution starting from the old lockfile, not the plan. Two defects show up here: the install step has no fallback for this, and `fix --json` printed nothing on this exit 2. The run ended as a raw npm error instead of a "blocked by" result. UPT-102's done-criterion (a draft that installs, or a clear "blocked by") is not met yet, so this PR does not close it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Lucas Braum <lucasbraum@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…otherwise (UPT-107) (#224) Closes UPT-107. Refs UPT-102 (not closed: enki delivered no draft and did not end blocked; see below). ## What and why On enki (UpgradeBench, #169), `npm install --package-lock-only` failed with ERESOLVE from the parent's lockfile, although the same manifest resolves fresh. `fix` stopped with exit 2, and `fix --json` printed nothing. **Staged retry** (`managers/upgrade.ts`). For npm only, and only on ERESOLVE, the lock-only install is retried from smaller lockfiles, smallest re-resolve first: 1. **moved entries**: the old lockfile without the entries of the moved packages (lead and companions), every locked package that declares one of them as a peer, and everything installed below those. Links are kept. 2. **whole lockfile**: every installed entry dropped, so npm resolves everything fresh. `--legacy-peer-deps` and `--force` are never used (the tests assert it on every install call). - The step that worked is recorded as `lockfile.eresolveRetry` (`{ step, dropped, conflicts }`). It shows in the text summary and the PR body ("Lockfile re-resolved"). - The parent-commit comparison (`parentLock`) starts from the same reduced lockfile. So the lock guard reports churn as usual: what the fresh resolve moves at the parent too shows as `parentChurn`, not as the upgrade's. - If every step fails, `fix` ends with `INSTALL_BLOCKED` (`InstallBlockedError`). Its message lists the package lines from npm's ERESOLVE report (While resolving / Found / Could not resolve dependency / Conflicting peer dependency / `peer … from …`). It crosses the worker boundary intact. **`fix --json` on every exit.** Exit 0 and 1 already printed the report. On a failure (exit 2, and exit 1 from a `CliError`), `fix --json` now prints `{ error: { code, message } }`. A blocked install adds `blocked: { attempts, conflicts }`. This is scoped to `fix`: `pr` already writes its own error JSON and then rethrows. **Second commit, found by the rerun.** The paid enki run got past ERESOLVE and ran the agent. It then stopped at "Required migration evidence exceeds the PR body budget". That refusal is thrown after `report.json` is written in the clone, and the clone is then removed. So the JSON had the error but no report: no cost, no lockfile. The refusal is now a `ReportedError` (`PR_BODY_TOO_LARGE`) carrying the report, and `fix --json` prints the report with `error` on top. Whether the run should stop there at all is unchanged here (follow-up). ## Tests - `managers/eresolve.test.ts` uses a stand-in npm: a small registry, keeps locked versions, and fails with npm's real ERESOLVE text (synthetic names, `fixtures/lockfiles/eresolve/npm-11.txt`) while a conflict holds. - **moved entries**: the retry starts without `toggle, ui, ui-dom` and keeps `unrelated` locked; toggle re-resolves to 1.1.0. - **whole lockfile**: used only when step 1 still conflicts; the fresh resolve's churn lands in `parentChurn`, and `outOfScope` is empty. - **all fail**: `INSTALL_BLOCKED` after exactly 3 attempts, with the conflict lines. - `cli.test.ts`: `fix --json` exit 2 prints JSON for a blocked install (with `blocked`), for any other failure, and for a `ReportedError` (with the report). The text output carries the same message. - `format-fix.test.ts`: the summary names the retry step. - `pnpm lint`, `pnpm typecheck`, `pnpm test`: green locally (core 2271, cli 515). `pnpm public:check` with the denylist: clean. ## enki rerun (#169 harness, Uptide arm only, claude-sonnet-5-5, $1 cap) Run `a8ff50ca`, from this branch's first commit (fd3037f). | | | |---|---| | Exit | 2: `Required migration evidence exceeds the PR body budget` (after install, agent and verification) | | `--json` | printed `{ error }` (before the second commit it carried no report) | | Install step | **moved entries** (see the $0 replay below) | | Lockfile churn | 0 out of scope; 85 parent churn; 5 moved subtrees | | Clean install (#220) | `npm ci --ignore-scripts` passed in the replay; the paid run's result was lost with its report | | Checks | install/typecheck/lint/test 0, but on the parent commit: no commits were delivered | | Cost | unknown, at most the $1 cap. The agent estimated $0.54 for 9 sites; the harness read $0 because no report came back | | Wall time | 193 s | | Fork CI | none: no commits, so no fork pull request | **$0 replay.** The same CLI and flags were run on a copy of the run's clone, with `--keep-temp` and an invalid API key so every agent call fails. It takes the same agent-led plan: react, react-dom 19.2.0, @types/react, @types/react-dom, `@entur/react-component-toggle` 1.0.0, react-leaflet 5.0.0, unsafe `@reach/*` overrides left alone. The lock-only install hit ERESOLVE (`Found: @entur/react-component-toggle@0.0.3` / `Conflicting peer dependency: react@19.2.0`). Step 1 resolved it after dropping the entries of 82 react-peer packages; step 2 was not needed. It ended at exit 1 with a 23 KB PR body. In the paid run, the agent's edits pushed the required (never-trimmed) evidence past GitHub's 65,536-character limit. UPT-102 stays open: enki now installs, but its draft is refused at the PR body. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Lucas Braum <lucasbraum@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

… (UPT-104) (#226) Closes UPT-104. ## What and why `fix` builds the workspaces whose output a consumer reads (`prepareWorkspace`) right after the bump, before any edit, so the agent's type-check sees fresh declarations. Those results were then reported as the run's build and turned into `Build cannot run (…)` fallback reasons, and nothing ever built them again. When the edits fixed the build, the run still: - forced a feedback round (`round === 1 && fallback.length`) with nothing left to fix, - put an **Agent-led draft** banner at the top of the PR body, - showed `Build after the bump … — failed` / `❌ build` in the PR body. That's what happened on calmcp (#219) and on dms-saas (#221's final run, uptide-bench fork PR #8). ## Change (`packages/core/src/fix/run.ts`) - The consumed builds are judged on the head the edits left. They run again before the verdict and after each feedback round, and the reported `verification.build` holds those results. - Build fallback reasons are recomputed from the latest builds after each build, so a reason the head no longer shows is dropped. The reasons that don't depend on the head (planner refusals, agent-led pins, unknown baseline) stay. - The consumed builds run again only when the tree changed since their last result: HEAD's tree plus the diff and untracked names over it, marked `ponytail:` because an untracked file edited in place goes unseen. A run whose after-bump build passed and that made no later edits does not build again. - When the verification build (generic or `requiresBuild` runs) already built the same workspaces on the head, its results are used instead of a separate build. - New report field `buildRuns: { count, ms }`: every build the run executed (preparation and verification) and their wall time. ## Tests `fix.test.ts`, `consumed workspace builds on the final head (UPT-104)`: 1. The after-bump build fails and the fixed head builds: no feedback round, no `agentFallback`, the agent is never called, the build line says `passed`, no banner, and 3 builds (baseline, after the bump, edited head). 2. The after-bump build passes and nothing changes afterwards: no further build (2 builds). 3. A `requiresBuild` pack: the consumed workspace is built once on the head, by the verification build. Tests 2 and 3 each fail with their guard removed. The existing test that counted 2 builds now expects 3, with the third on the edited head. `pnpm test`, `pnpm typecheck` and `pnpm lint` pass; the only lint warnings are the two that predate this change. ## Bench (#169 harness, Uptide arm only) Same harness, model (`claude-sonnet-5-5`, medium) and $1 cap as #169. **Before** is origin/main `2dc2162` plus only the `buildRuns` counter (a local commit, not pushed); **after** is this branch. One run each, sequential. | Run | Rounds | Cost | Wall | Builds (count / time) | Fallback banner | PR body build line | Verdict | |---|---|---|---|---|---|---|---| | calmcp before | 1 | $0.117 | 76 s | 1 / 0.5 s | yes: `Build cannot run (.)`, zodDoc.ts TS2322 | `.` `tsc -p tsconfig.json` — **failed** (`❌ build`) | PASS | | calmcp after | **0** | **$0.082** | **58 s** | 2 / 0.7 s | **none** | `.` `tsc -p tsconfig.json` — **passed** (`✅ build`) | PASS | | dms-saas before | 1 | $0.391 | 300 s | 2 / 8.6 s | yes: `Build cannot run (packages/dms-saas)`, the stripe errors the agent then fixed | `packages/dms-saas` — **failed** (`❌ build`) | FAIL | | dms-saas after | **0** | **$0.294** | **256 s** | 3 / 15.5 s | **none** | `packages/dms-saas` — **passed** (`✅ build`) | PASS | Phases: - calmcp before: analyze 5 s, baseline 3 s, install 6 s, rules 0 s, agent 33 s ($0.089), verify 2 s, **feedback 16 s ($0.028)** - calmcp after: analyze 6 s, baseline 3 s, install 6 s, rules 0 s, agent 35 s ($0.082), verify 2 s - dms-saas before: analyze 23 s, baseline 26 s, install 21 s, rules 0 s, agent 120 s ($0.271), verify 26 s, **feedback 69 s ($0.120)** - dms-saas after: analyze 25 s, baseline 30 s, install 24 s, rules 0 s, agent 128 s ($0.294), verify 35 s The extra build is the head rebuild: +0.2 s on calmcp and +6.9 s on dms-saas. It replaces a feedback round of 16 s / $0.028 and 69 s / $0.120. How to read the dms-saas rows: - **The verdicts differ because of the agent's edits, not this change.** Before, the agent bumped the API version and a basil-payloads test failed as new. After, 3 failing test files were marked pre-existing and the run passed. Both runs left 0 new type errors. - **The first after run on dms-saas failed and is not in the table.** It exited 2 after 423 s with `Required migration evidence exceeds the PR body budget`. A run that throws removes its clone, so no report survived, and its spend went unrecorded (no `--json` output). I reran it once with `--keep-temp` (no other difference): that's the row above, with a 31 KB PR body. The overflow is the open "PR body overflow" follow-up from UPT-101, and I couldn't attribute it to this change: removing a fallback reason only shortens the body. Nothing here fixes it. No fork PRs were opened for these runs, and nothing was pushed outside this branch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Lucas Braum <lucasbraum@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

) Closes UPT-106. Closes UPT-102: enki now delivers an agent-led draft, uptide-bench/entur--enki#5. ## Problem `prBody()` threw "Required migration evidence exceeds the PR body budget" when the rank-0 sections alone were over the budget. That threw away the whole finished run. It happened on enki (a8ff50ca, after #224) and on dms-saas (#221). ## Change - **No section is exempt.** `fitPieces` returns `undefined` instead of throwing. Required lists that can grow without bound get a compact form, which is used only when the whole list does not fit. These lists still go before anything droppable. - Lockfile churn (outside the upgrade, and parent churn): counts per kind of change (added, removed, version changed, re-resolved at the same version), then the first 10 packages. - Manual sites under "Worth a look": grouped by rule, with a count and the first 5 locations each. - Remaining type errors: the first 10, then a count. - **Full lists** stay in report.json. The Action's job summary now lists every breaking site (it was capped at 50). Compact forms link to it through a new `runUrl` field, set from `GITHUB_SERVER_URL`/`GITHUB_REPOSITORY`/`GITHUB_RUN_ID` when the run is in Actions. - **Minimal body.** If the compact form still does not fit, the body is a fixed-size summary: the verdict, the checks (types, tests, builds, lint, clean install), the counts (sites by outcome, lockfile, files, decisions), the branch and commit, and where the full report is. It opens with a "Shortened description" notice. - **The run is kept.** `fix()` and the isolated run's `finish()` write report.json before the body, and nothing throws for size any more, so a finished run always keeps its commits and report.json. `uptide pr` sends the shortened body instead of refusing. - **Decision lines** (second commit, found by the enki rerun). An agent decision named every file in its task's scope. The verification-feedback task was scoped to 317 files, so its 2 decisions came to 46 KB. A decision now names the files the task changed, five at most, and counts the rest. ## Fixtures Written first; both failed on main with the throw. - 300 manual sites (3 rules) and 100 + 100 churned lockfile entries. At the 60 k budget every section is present: lockfile sections, risk table, What changed, Worth a look, Decisions, Verification details, Run details. Manual sites are compact and the lockfile lists are whole. At 8 k the lockfile lists are compact too: "100 entries: 25 added, 25 removed, 25 re-resolved at the same version, 25 version changed", with the first 10 packages. - The same report with an oversized decision renders the minimal body. - A 317-file task with one changed file: the decision names only that file. Without diffs it names 5 files and "312 more files". ## enki rerun (#169 harness) Uptide arm only, claude-sonnet-5-5, default $1 cap, one run: 60949226. | | | |---|---| | Exit | 0, draft delivered (branch `uptide/react-19.2.0`, verified commit `3f05f25`) | | PR body | 804 chars: minimal fallback (on 06bab33). With 16b2abe the same stored report renders whole at 34,718 chars (77,681 before), no shortening | | Checks (harness) | install 0 · typecheck 0 · lint 0 · test 0 | | #220 clean install | `npm ci --ignore-scripts --no-audit --no-fund` passed | | Sites | 3 by rule, 8 by the agent, 1 manual; verification PASS, 0 new type errors | | Cost / wall time | $0.17 (estimate $0.54) · 160 s | | Fork CI | uptide-bench/entur--enki#5: npm ci, `npm run check` and tests passed. The job fails at SonarQube Scan (no `SONAR_TOKEN` in the fork), so build is skipped. Fork PRs #1–#4 fail the same way | The run used the build of 06bab33. The fork PR's body is therefore the minimal form; the 34.7 KB figure is a $0 re-render of the stored report.json with 16b2abe. The report records `uptideDirty` because the harness files under `bench/` were copied in untracked, as in earlier reruns. ## Left as is - Lockfile churn is not in the job summary. The run-report schema (`uptide.run/v1`) has no lockfile field, so the full churn list is only in report.json. - `ReportedError` and `PR_BODY_TOO_LARGE` stay. Nothing throws `PR_BODY_TOO_LARGE` any more, but `fix --json` still uses `ReportedError` for reporting any failure after report.json is written. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Lucas Braum <lucasbraum@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…nds per phase (UPT-103) (#232) Closes UPT-103. ## What and why After UPT-100/101/104/106, Uptide's cost is at or below Claude Code's on the pilot repositories (#169). Wall time is the gap. This PR measures where the time goes and makes two cuts. Neither one changes the gate after the agent's edits or the final verification. **1. Per-phase commands in the fix report** (`d0a77c3`). Each `phases[]` entry now has `runs`: the repository's commands the phase ran (`install`, `generate`, `build`, `types`, `tests`, `lint`, `format`), each with a count and its summed wall time. An agent-led run's clean install (#220) gets its own `clean-install` phase, so the phases add up to the run. The tables below come from this field. **What the baseline showed.** These are the costs I could act on, across the three repos: - **node_modules copies around the install.** `isolatedUpgrade` installs in a temporary worktree. It then `cpSync`s the new `node_modules` into the clone, file by file and not even copy-on-write, and deletes the worktree. In a $0 replay with a CPU profile (invalid API key, no model spend), that copy was 7.5 s on enki and 5.1 s on dms-saas. npm itself took only about 10.5 s of enki's 31 s install. - **Test runs in the agent phase on dms-saas: 2 runs, 42.6 s.** It is the largest line item there apart from the model. The agent's `run_tests` asks for the task's related tests. But dms-saas's `test` script is `pnpm run build && … && pnpm run test:unit`, so related scoping found no bare runner and ran the whole script, build included. The model's own time is the largest item everywhere (about 31 s, 71 s and 50 s), but I found no deterministic cut for it (see "What did not work"). **2. The agent's own test check runs its runner's related tests** (`26574a9`). This is a new `agentCheck` test option, used only by the agent's own `run_tests`. When a `test` script ends by running a script that is just the runner, the check runs that runner's related tests. Everything else is unchanged: - The gate after the agent's edits still runs its own `related` plan (the whole script on dms-saas). - The final verification runs the full suite. - A check result never stands in for a gate result: separate cache entries. - Where the narrowing changes no command (calmcp, enki: the script already is the runner), the check stays the gate's own run, and the gate reuses its result on the same tree, as before. **3. The installed tree is moved, not copied** (`049023c`). `isolatedUpgrade` renames the new `node_modules` out of the worktree, which is removed anyway. The tree is the same, links into the worktree are relocated as before, and it falls back to a copy across filesystems (`EXDEV`). ## Results The #169 harness (`bench/run.sh`), Uptide arm only, `claude-sonnet-5-5` at medium effort, default $1 limit, once per repository. Before is `origin/main` plus the counting commit `d0a77c3` (no behavior change). After is `049023c`; the last commit only adds a test. `bench/` is untracked in both checkouts, which is why both reports say `uptideDirty`. Claude Code's numbers are from the #169 rerun (same harness and model). | repo | Claude Code (#169) | Uptide before | Uptide after | |---|---|---|---| | calmcp, zod 3.25.76→4.6.5 | $0.11, 47 s | $0.094, 60 s | $0.085, **53 s** | | dms-saas, stripe 18.5.0→22.6.2 | $0.31, 149 s | $0.265, 224 s | $0.279, **210 s** | | enki, react 18.3.1→19.2.0 | $0.11, 67 s (its diff fails `npm ci`) | $0.176, 147 s | $0.161, **136 s** | **Same checks, same verdicts, same diff shape.** On every repository, before and after match on all of these: - `verification.passed` (true) - the site outcomes: calmcp 1 mechanical + 5 agent, dms-saas 2 + 10, enki 3 + 8 + 1 manual - the changed file list: 4, 11 and 12 files, identical - test, lint and build statuses, and no new type errors - feedback rounds: 0, 0 and 1 - enki's clean install, which passed One run each: the model's own time moves by ±5 s between runs, which is the size of calmcp's whole change. <details><summary>calmcp phases</summary> | line item | before | after | |---|---:|---:| | **wall (harness)** | **60 s** | **53 s** | | cost | $0.094 | $0.085 | | outside `fix()`: preflight, clone with node_modules copy, delivery, clone removal | 3 s | 3 s | | analyze (check; workspace prep builds) | 5.7 s | 5.5 s | | baseline phase | 3.3 s | 3.2 s | | ↳ baseline type-check | 1× 0.8 s | 1× 0.8 s | | ↳ baseline tests | 1× 1.8 s | 1× 1.8 s | | install phase (bump install, consumed rebuild, target types) | 6.1 s | 4 s | | ↳ upgrade install | 1× 4.8 s | 1× 2.8 s | | agent phase | 38.2 s ($0.094) | 33.5 s ($0.085) | | ↳ checks inside it (types, lint, format) | 0.9 s | 1 s | | ↳ tests inside it (agent check + gate) | 2× 1.8 s | 2× 1.9 s | | ↳ model and tool turns (the rest) | 35.5 s | 30.6 s | | feedback rounds | 0: – | 0: – | | final verify (build, format, full tests, types, lint) | 2.6 s | 2.7 s | | clean install (agent-led, #220) | – | – | | installs, all phases | 1× 4.8 s | 1× 2.8 s | | tests, all phases | 4× 5.1 s | 4× 5.3 s | | type-checks, all phases | 8× 2.7 s | 8× 2.7 s | | builds (`buildRuns`) | 2× 0.7 s | 2× 0.6 s | | generators | 2× 0 s | 2× 0 s | | publish | not run (no `--pr` in the harness) | not run | | agent turns | 19 | 17 | </details> <details><summary>dms-saas phases</summary> | line item | before | after | |---|---:|---:| | **wall (harness)** | **224 s** | **210 s** | | cost | $0.265 | $0.279 | | outside `fix()`: preflight, clone with node_modules copy, delivery, clone removal | 9 s | 10 s | | analyze (check; workspace prep builds) | 23 s | 22.7 s | | baseline phase | 25.8 s | 25.5 s | | ↳ baseline type-check | 1× 3.6 s | 1× 3.7 s | | ↳ baseline tests | 1× 21.6 s | 1× 21.5 s | | install phase (bump install, consumed rebuild, target types) | 22 s | 14.7 s | | ↳ upgrade install | 1× 12.4 s | 1× 5.3 s | | agent phase | 112.4 s ($0.265) | 107.3 s ($0.279) | | ↳ checks inside it (types, lint, format) | 1.5 s | 1.3 s | | ↳ tests inside it (agent check + gate) | 2× 42.6 s | 2× 35.3 s | | ↳ model and tool turns (the rest) | 68.4 s | 70.7 s | | feedback rounds | 0: – | 0: – | | final verify (build, format, full tests, types, lint) | 31.1 s | 29.1 s | | clean install (agent-led, #220) | – | – | | installs, all phases | 1× 12.4 s | 1× 5.3 s | | tests, all phases | 4× 85.6 s | 4× 76.6 s | | type-checks, all phases | 8× 11.4 s | 8× 11.2 s | | builds (`buildRuns`) | 3× 13.5 s | 3× 13.1 s | | generators | 2× 0 s | 2× 0 s | | publish | not run (no `--pr` in the harness) | not run | | agent turns | 34 | 34 | </details> <details><summary>enki phases</summary> | line item | before | after | |---|---:|---:| | **wall (harness)** | **147 s** | **136 s** | | cost | $0.176 | $0.161 | | outside `fix()`: preflight, clone with node_modules copy, delivery, clone removal | 11 s | 11 s | | analyze (check; workspace prep builds) | 12.8 s | 12.7 s | | baseline phase | 7 s | 7 s | | ↳ baseline type-check | 1× 2.8 s | 1× 2.6 s | | ↳ baseline tests | 1× 3.9 s | 1× 4.1 s | | install phase (bump install, consumed rebuild, target types) | 32.9 s | 23 s | | ↳ upgrade install | 1× 30.9 s | 1× 21.1 s | | agent phase | 26.6 s ($0.085) | 27.4 s ($0.085) | | ↳ checks inside it (types, lint, format) | 1.1 s | 1 s | | ↳ tests inside it (agent check + gate) | 1× 1.1 s | 1× 1.2 s | | ↳ model and tool turns (the rest) | 24.4 s | 25.2 s | | feedback rounds | 1: 44 s ($0.092) | 1: 40.8 s ($0.076) | | final verify (build, format, full tests, types, lint) | 5.7 s | 5.8 s | | clean install (agent-led, #220) | 6 s | 7.7 s | | installs, all phases | 2× 36.9 s | 2× 28.7 s | | tests, all phases | 5× 15.5 s | 5× 16.3 s | | type-checks, all phases | 11× 15.3 s | 11× 15 s | | builds (`buildRuns`) | 0× 0 s | 0× 0 s | | generators | 2× 0 s | 2× 0 s | | publish | not run (no `--pr` in the harness) | not run | | agent turns | 17 + 19 | 17 + 13 | </details> Where the cuts show up: - **upgrade install:** calmcp 4.8 → 2.8 s, dms-saas 12.4 → 5.3 s, enki 30.9 → 21.1 s. - **agent-phase tests on dms-saas:** 42.6 → 35.3 s. The narrowed check took about 14 s instead of 21 s. Tests on calmcp and enki: same counts and times as before. ## What is left versus Claude Code - **calmcp, 53 s vs 47 s.** About 31 s is the model: 17 turns, one tool call each. The other ~19 s is work Claude Code does not do in this form: analysis 5.5 s, install 4 s, baseline 3.2 s, final verify 2.7 s, clone and delivery 3 s. - **dms-saas, 210 s vs 149 s.** Tests are the gap: four runs of the `test` script family take 76.6 s (baseline 21.5, agent check ≈14, gate ≈21, final ≈20). - Every full run builds before running vitest. vitest alone is about 17 s. - Related tests barely help here: every route imports the Stripe client, so "related" is most of the suite. - The gate and the final run are the verification this PR must not trade away. - The rest: analysis 22.7 s (the stripe 18→22 type diff and call-site compile) and the model, about 71 s for 34 turns. - **enki, 136 s vs 67 s.** Claude Code's diff does not install (`npm ci` ERESOLVE), so it skipped what costs Uptide here: the upgrade install (21 s), the clean install (7.7 s) and the agent-led feedback round (41 s). The round is forced by the peer-blocker and root-pin fallback. ## What did not work (reverted, not in this PR) An intermediate build also (a) asked the agent to batch reads and edits as parallel tool calls in one message, and (b) narrowed the agent's check in feedback rounds. It ran on all three repos ("after (reverted)" runs 8e0f068f, d48a70c0, 72b2bc57): - (a) changed nothing: dms-saas made the same 34 turns with the same tool counts. - (b) made enki slower, 147 → 163 s. `vitest related` over the feedback task's files cost more than the full suite, and splitting the check from the gate lost the gate's cache hit. Both are reverted. The final change keeps the check on the gate's run wherever narrowing changes nothing. ## How it was verified Fixture tests: - `generic.test.ts`, "narrows only the agent's own check, …" (3 cases) — **the final verify still runs the full gate**. It covers a chained script, a plain runner, and an agent-led run. It records every test call and its options and checks: - the order is baseline full, agent check, gate (as before), final full suite; - the final call is `fullSuite` without `related`; - the verify phase built, type-checked and ran tests once each; - the agent-led run ran its clean install (#220) and it passed. - `fix.test.ts`, "scopes only the agent's own check to the runner a test script chain ends with": `agentCheck` gives `vitest related …`, while `related` alone and `fullSuite` still give the whole `pnpm run build && pnpm run test:unit`. - `sdk-agent.test.ts`: the agent's `run_tests` uses `checkTests`, repeats reuse its own result, and the gate on the same tree still runs `tests`. - `isolated-install.test.ts`: the clone's `node_modules` holds the same inode the install wrote (a move, not a copy). Relative links are kept and absolute links into the worktree are relocated. - `generic.test.ts` phases test: the baseline phase's `runs` counts its type-check and test run. `pnpm lint && pnpm typecheck && pnpm test` pass (core 2,292 tests, cli 516). The bench runs needed no fork PRs: nothing was pushed anywhere but this branch. ## Checklist - [x] Tests for the change - [x] `pnpm lint && pnpm typecheck && pnpm test` pass - [x] README and docs updated if behavior changed (no CLI flag or documented output changed; the report field is described in `types.ts`) - [x] Changeset added (`agent-phase-wall-time.md`) - [x] No new dependency - [x] No private code, names or paths in fixtures, tests or docs - [x] The rules in CONTRIBUTING.md ("Rules that do not bend") still hold 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Lucas Braum <lucasbraum@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

lucasbraum · 2026-10-18T17:31:25Z

Round 3: published `uptide@0.15.1` vs Claude Code (2026-10-18), same 3 repos, 1 run each

Setup:

Uptide arm: npx -y uptide@0.15.1 fix <pkg> --target <B>. That is the published package, not a local build; version.uptide: "0.15.1" is in each run's meta.
Claude Code arm: 2.1.283 (version.claude in meta), claude-sonnet-5-5 at medium effort. Same prompt, sandbox settings and tools as both earlier rounds, with --max-budget-usd 1.
Both arms: fresh clone of the same parent commit, parent installed (untimed) before the clock starts.
Harness: bench/ from this branch rebased on origin/main 41a2e75, plus one local commit that is not pushed. That commit swaps the Uptide command for npx, records versions and executor in meta, and adds a build check (each repo's npm run build / pnpm run build).
Human references: re-checked locally. All 5 checks pass on all 3, and dms-saas passes 681/681 tests with no flake this time.

Read this first: the published CLI ran a different executor. npx uptide@0.15.1 does not install the optional @anthropic-ai/claude-agent-sdk peer, so all three Uptide runs used the patch executor (executor: "patch" in meta). Both earlier rounds ran from a workspace build with the SDK executor, so this round is not like-for-like with Uptide's earlier numbers. It does measure what a user gets from the README command.

calmcp, zod 3.25.76→4.6.5

round	arm	frozen install	types	lint	tests	build	fork CI	cost	wall	files (code + lock/manifest)	coverage vs human
3 (0.15.1)	Uptide	ok	5 errors	ok	1 fail	fail	fail: env failures as human, plus `Tool surface` and `docker` fail building	$0.03	52 s	1 + 2	1/27
	Claude Code	ok	ok	ok	ok	ok	same as human (env only)	$0.11	49 s	2 + 2	7/27
2 (main `195ae10`)	Uptide	ok	ok	ok	ok	–	same as human	$0.16	97 s	2 + 2	7/27
	Claude Code	ok	ok	ok	ok	–	same as human	$0.11	47 s	2 + 2	7/27
1 (pilot)	Uptide	ok	ok	ok	4 fail	–	same as human	$0.07	51 s	–	6/27
	Claude Code	ok	ok	ok	ok	–	same as human	$0.10	43 s	–	6/27

Fork CI, environmental: calmcp's build jobs stop at npm audit on the human reference too, because an advisory was published after the commit. dependency-review fails on forks.
Fork CI, real: the Uptide PR also fails Tool surface/diff and docker, because tsc fails on them.
vs human: the human changed 5 code files. Uptide 0.15.1 made only the uuid()→guid() change in create.ts. It left zodDoc.ts untouched, which is where the 5 type errors are, and its PR says so (❌ type-check, build, tests). Claude Code changed zodDoc.ts and create.ts, the same result as round 2. Neither added the human's .strict() on the tool schemas.
PRs: Uptide, Claude Code.

dms-saas, stripe 18.5.0→22.6.2

round	arm	frozen install	types	lint	tests	build	fork CI	cost	wall	files (code + lock/manifest)	coverage vs human
3 (0.15.1)	Uptide	ok	5 errors	ok	pass³	fail	fail at type-check	$0.18	242 s	7 + 2	6/51
	Claude Code	ok	ok	ok	1 real fail³	ok	fail at `test:unit`, same test	$0.25	121 s	8 + 2	13/51
2 (main `195ae10`)	Uptide	ok	1 error	format	pass²	–	fail	$0.69	391 s	8 + 2	10/51
	Claude Code	ok	ok	ok	pass²	–	pass	$0.31	149 s	8 + 2	13/51
1 (pilot)	Uptide	ok	1 error	format	ok	–	fail (lint)	$0.24	161 s	–	10/51
	Claude Code	ok	ok	ok	ok¹	–	pass	$0.27	115 s	–	16/51

³ The usual MongoMemoryServer flake: private-admission.test.ts timed out at 5 s on the full pass for both arms, and passes alone on both. Claude Code's second failure is real.

subscription-effect-admission.test.ts › "does not compensate an ambiguous replacement schedule" fails with Cannot read properties of undefined (reading 'retrieve'), locally and in fork CI.
Claude Code replaced the removed phase iterations with a duration read through stripe.prices.retrieve. That is the human's approach too, but the human also added prices to that test's Stripe mock.
Claude Code could not see the failure. Its summary says 11 Mongo-backed test files cannot start inside its sandbox, which is the known isolation difference between arms.
Uptide 0.15.1: for the first time it fixed every subscriptions.cancel idempotency site, and lint is clean. It left phase iterations in plan-schedule.ts and the now open-ended credit-note type/status enums in webhook-credit-notes.ts, which are the 5 type errors. Its PR body says ❌ type-check/build/tests. The patch agent wrote that it cannot fix the credit-note payload without a cast or the payload builder's declarations.
vs human: the human changed 22 files. Neither arm made the billing_mode, managed_payments or provisioning changes.
PRs: Uptide, Claude Code.

enki, react 18.3.1→19.2.0

round	arm	frozen install	types	lint	tests	build	fork CI	cost	wall	files (code + lock/manifest)	coverage vs human
3 (0.15.1)	Uptide	ok	4 errors	ok	ok (58/58)	fail	fails only at SonarQube Scan (missing secret, as human)	$0.06	158 s	6 + 2	9/29
	Claude Code	fail (ERESOLVE)	not run	not run	not run	not run	fail at `npm ci`	$0.11	67 s	7 + 2	10/29
2 (main `195ae10`)	Uptide	no changes (exit 2)					no PR	$0.00	9 s	0	0/29
	Claude Code	fail (ERESOLVE)	not run			–	fail (`build_and_preview`)	$0.11	67 s	7 + 2	7/29
1 (pilot)	Uptide	no changes (exit 2)					no PR	$0.00	9 s	–	0/29
	Claude Code	fail (ERESOLVE)	not run			–	fail at `npm ci`	$0.13	94 s	–	10/29

Uptide's first PR on enki. It now follows the peer blockers instead of stopping: @entur/react-component-toggle 0.0.3→1.0.0 and react-leaflet 4→5, the same manifest bumps the human made. It left the @reach/* and @azure/msal-react overrides pinned to 18.3.1 and listed them for review.
Uptide's 4 type errors:
- 3 files import @react-types/datepicker, which its new lockfile no longer has at the top level of node_modules. enki uses that transitive dependency without declaring it; the human's lockfile keeps it.
- One implicit any in Providers/index.tsx.
Fork CI does not catch these errors. The fork workflow runs npm ci, prettier and tests, then stops at SonarQube Scan because the fork has no Sonar secret. npm run build (tsc && vite build) never runs, so the type errors only show up locally.
Claude Code: the same npm ci ERESOLVE as both earlier rounds. It kept @entur/react-component-toggle@0.0.3, whose react@^18.3.1 peer the lockfile cannot satisfy.
vs human: the human changed 11 code files. Both arms touched the same 6 react-leaflet/useRef files, and Claude Code also touched Providers/index.tsx. Neither changed Markers.tsx, QuayPopupButtonPanel.tsx, QuaySequenceIndexTooltip.tsx or usePopupOpeningOnFocus.ts.
PRs: Uptide, Claude Code.

Summary

calmcp: Claude Code wins. Uptide 0.15.1 regressed from all-green in round 2.
dms-saas: neither is mergeable. Uptide has 5 type errors; Claude Code compiles but breaks one test.
enki: neither is mergeable. Uptide's version installs and passes tests but fails the type-check and build; Claude Code's version does not install.
Coverage vs human: Uptide went down on calmcp (7→1) and dms-saas (10→6), and up on enki (0→9). Claude Code is within ±3 of its earlier rounds on all three.
Wall time: Uptide is faster than round 2 on calmcp and dms-saas (patch agent, fewer tokens). It produced weaker diffs on both.
I can't separate how much of Uptide's change comes from 0.15.1 itself and how much from the patch executor. A run of the same version with the SDK peer installed would settle it. I did not run one, because that is a different configuration from the one asked for, not a rerun.

Uptide 0.15.1 issues this round found

Silent executor switch. Without the optional SDK peer, npx uptide fix quietly uses the patch executor. Its own notice still says "The Anthropic SDK executor can also send files throughout the temporary clone", and neither the log nor the PR body mentions the fallback.
Wrong build line on enki. The PR's verification row says "build not required (pack-verified migration)", while npm run build fails on the same 4 type errors.
Lockfile drops an undeclared transitive dependency. The regenerated enki lockfile loses the top-level @react-types/datepicker that the repo imports.

Notes

Claude Code cap: the brief's $1. The earlier rounds actually ran at the harness default of $6 (CLAUDE_CAP_USD unset). No Claude Code run in any round came close to either cap: the largest this round is $0.25 at Uptide's prices, about $0.46 at Claude Code's own.
No reruns or discarded runs. No run failed for an infrastructure reason. The only re-execution is the established flake check: re-running the two failing dms-saas test files alone, on the human reference and both arms.
Spend: $0.28 Uptide + $0.47 Claude Code = $0.75.

🤖 Generated with Claude Code

…LVE retry (UPT-109) (#234) Closes UPT-109 once the enki check below is done. Refs UPT-107 (#224). ## What and why In the 0.15.1 pilot (#169), enki's Uptide PR installed and passed tests, but the type-check failed: 3 files import `@react-types/datepicker`, which enki uses without declaring it, and the new lockfile no longer had it. **Cause.** The UPT-107 "moved entries" retry drops every package that declares `react` as a peer, and that includes `@react-aria/datepicker`. It re-resolved from 3.15.2 to 3.17.1, and 3.17.1 no longer depends on `@react-types/datepicker`. Nothing required that package any more, so npm pruned it. **Fix** (`managers/upgrade.ts`, npm only): - After a retry step succeeds, compare the top level of the old and new lockfile (root and workspaces). Then scan tracked `.[cm]?[jt]sx?` files for bare `from` / `import` / `require` specifiers. A package that left the top level and is imported is a dropped import. - If there is one, the same step runs again. It keeps the dropped package and every locked package that requires it, transitively, at their locked versions. Moved packages are never kept. - If that retry hits ERESOLVE, the first retry's lockfile stands. - `lockfile.undeclared` (`{ name, version, kept }`) lists each one. The CLI summary and the PR body show it under "Imported but not declared": the package is imported but not declared in package.json, and it was kept at a version or the new lockfile no longer has it. Both suggest adding it to package.json. - The same report also covers a normal install with no retry, where the only possible outcome is reporting it. - `package.json` is never changed. On enki this should keep `@react-aria/datepicker` (its react peer range already accepts 19) and its requirers at their old versions, and so keep `@react-types/datepicker` 3.13.2. ## Tests - `eresolve.test.ts`, stand-in npm (now resolves dependencies transitively and prunes unrequired entries, like npm): - **kept**: `picker` peers on the moved `ui`, so the first retry drops it. `picker` 1.1.0 no longer brings `picker-types`, which `src/date.ts` imports. The second retry starts with `picker` and `picker-types` locked, and the lockfile keeps both at 1.0.0. The report has `undeclared: [{ picker-types, 1.0.0, kept: true }]`, and `package.json` is unchanged. - **cannot keep**: `picker` 1.0.0 conflicts too, so the second retry hits ERESOLVE. The first retry's lockfile stands (`picker` 1.1.0, no `picker-types`), with `kept: false`. - Both failed on main before the fix. - `format-fix.test.ts`, `report.test.ts`: the summary and PR body lines for both kept and not kept. - Locally: `pnpm lint` (only the existing warning/info from main), `pnpm typecheck`, core 2296 passed, cli 517 passed (after `pnpm build`, which the notices and telemetry tests need). ## Not done - **enki not re-run.** No bench runs for this PR, so the type-check result on enki is still unconfirmed. Running the #169 harness's Uptide arm on enki would confirm it. - pnpm and yarn are not covered. The retry is npm only, and their strict layouts already break undeclared imports. - The import scan is a regex over JS/TS files. It does not read `.vue` / `.svelte` files or `/// <reference types>`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Lucas Braum <lucasbraum@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…ed" in its place (UPT-110) (#233) Closes UPT-110. ## What and why In the 0.15.1 pilot (#169), enki's PR said "build not required (pack-verified migration)", but `npm run build` fails on the final head. A pack without `requiresBuild` never ran the repository's build, and the report filled that gap with "not required". That reads like a pass. ## Change - **`fix` runs the repository's build once on the final head** when the pack doesn't gate on it (`run.ts`). This happens after the feedback rounds, in a new `build` phase so the phases still add up to the run's wall time. Its results join `verification.build`. - **`verify` rebuilds on its own head** (`reverify.ts`). Before, it reused the stored build result for packs that don't require one, so the result could be stale. - **The build line** (`verifiedBy`, `report.ts`) now shows one of these: - `✅ build` - `❌ build: <first error line>` (escaped for the table, max 160 characters), plus the existing `verify.build` hint - `⚠️ build not run`, or `⚠️ build not run: no build script` When a pack exists but doesn't require the build, "(not required by the pack)" or "(not required by this candidate pack)" goes after the result. Generic runs, pin mode and packs that require a build get no note. ## Decisions to flag 1. **A build the pack doesn't require is reported but doesn't block.** `verification.passed`, `buildFallback` and `publicationBlockers` work as before. So a PR can be ready for review while the line says `❌ build … (not required by the pack)`. If a build that fails on the head (and passed at baseline) should block, that's a follow-up. It would also need a baseline build to tell "already failing" apart from "newly broken", and that costs a second build per run. 2. **This costs one more build per pack-verified run.** In the consumed-workspace case (UPT-104), `packages/internal` is now built twice on the final head: once by the consumed rebuild and once by the head build. I didn't dedupe this. ## Tests - New fixture in `required-build.test.ts`: a pack with `requiresBuild` false and a failing repo build renders `❌ build: src/db.ts(3,8): error TS2307: … (not required by the pack)`. It also checks the body has no "build not required" and the build is not a blocker. Without the fix it fails: the build is never called. - It replaces "leaves packs without requiresBuild unchanged", which asserted the old behavior. - Updated build-call counts and rows in three UPT-104 tests in `fix.test.ts`, and the wording in report and report-budget snapshots. - Lint clean on the changed files. On main, `biome check` already reports 1 warning and 1 info. Typecheck passes. After `pnpm build`: core 2293 passed, cli 516 passed. No bench runs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Lucas Braum <lucasbraum@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…ntly (UPT-108) (#237) UPT-108. `npx uptide@0.15.1 fix` never installed `@anthropic-ai/claude-agent-sdk` (an optional peer), so the Anthropic default quietly ran the patch executor while the notice and docs described the SDK ([pilot round 3](#169 (comment))). ## What changes - **The SDK is fetched on the first `fix` that needs it, not shipped.** The package stays at 13.6 MB and keeps the SDK as an optional peer. Lookup order: 1. An SDK resolvable next to uptide. The Action's install path is unchanged. 2. `<cache>/agent-sdk/<version>`, using `defaultCacheDir()`, so `UPTIDE_CACHE_DIR` and `XDG_CACHE_HOME` apply. 3. Otherwise the first `fix` that would use it runs `npm install --prefix <temp> --no-save --include=optional @anthropic-ai/claude-agent-sdk@0.3.296`. That goes through `command()`, so scripts are off and secrets are stripped from the env. The temp dir is renamed into place: the cache holds a complete install or nothing. One line reports it, with the size. Later runs reuse it. `SDK_VERSION` is pinned by a package test to the CLI's devDependency and peer range. - **Never a silent switch.** If the SDK can't run (the install failed, it failed to load, or there's no Claude Code binary for the platform), the default uses the patch executor. It says so on the first line of stderr and at the top of the PR body, with the reason and how to get the SDK executor. `--agent sdk` without a usable SDK stops before any clone instead of failing mid-run. - **`report.json`** gains `llm.executor` (`sdk` | `patch`, set whenever assisted fixes are on). `llm.sdkMissing` is now the reason string, where it used to be `true`. - **Notice:** the privacy note names the SDK only when the SDK will run, and the `LLM:` line names the executor. - **Docs:** `fix.md`, `privacy.md` and `architecture.md`, plus a new dated decision in ADR 0016 (the 2026-10-12 optional-peer decision is marked superseded). The changeset is `minor`. What the first lines look like: ``` # first fix, nothing installed (node:22 container) Agent SDK 0.3.296 is not installed: installing it once into /root/.cache/uptide/agent-sdk/0.3.296 … 295 MB, reused by later runs uptide fix · zod · smoke-npm (npm) LLM: anthropic / claude-sonnet-5-5 · executor sdk · budget $1.00 # install fails (@Anthropic-AI scope pointed at an unreachable registry) Agent SDK 0.3.296 is not installed: installing it once into /tmp/offline-cache/agent-sdk/0.3.296 … failed warning: Assisted fixes use the patch executor, not the Agent SDK: @anthropic-ai/claude-agent-sdk is not installed, and installing @anthropic-ai/claude-agent-sdk@0.3.296 into /tmp/offline-cache/agent-sdk/0.3.296 failed (npm error code ECONNREFUSED). For the SDK executor, let Uptide reach the npm registry once to install it into its cache, or install @anthropic-ai/claude-agent-sdk@0.3.296 next to Uptide; --agent patch chooses the patch executor and drops this note. uptide fix · zod · smoke-npm (npm) LLM: anthropic / claude-sonnet-5-5 · executor patch · budget $1.00 ``` The PR body of that run starts with the same note, after the title, as `> ⚠️ …`. ## Sizes node:22 on linux arm64, the packed CLI. | | before (main) | this PR | |---|---|---| | tarball | 2,549,083 B | 2,548,325 B | | `npm install` of the tarball / plain `npx uptide check` | 13.6 MB, 3 packages | 13.6 MB, 3 packages, no SDK anywhere on disk | | first `fix` that needs the SDK | none (silent patch fallback) | downloads into the cache once: 295 MB reported by Uptide (`du`: 310 MB) | | later `fix` runs | | 0 B | I tried an `optionalDependencies` version first (77bfe7c, then reverted). Every install grew to 323.6 MB and 110 packages, `check` included: binary 250 MB, SDK 5.5 MB, its peers 36 MB. That's why it's fetched on demand. ## Smoke (clean node:22 container, packed tarball, fake key, the npm smoke fixture) The zod sites there are all mechanical, so no model call is made. | case | result | |---|---| | plain `npx -y file:<tgz> check` | exit 1 (breaking found, as expected); npx install 13.6 MB; **no `claude-agent-sdk*` directory on disk**, no `agent-sdk` cache | | plain `npx` `fix`, first run | one install line (295 MB), `executor sdk`; `report.json` has `executor: "sdk"`; exit 0 in 13 s | | plain `npx` `fix`, second run | **no install line**, `executor sdk`; exit 0 in 8 s | | SDK installed next to uptide, empty cache | no install line, `executor sdk`, nothing written under `agent-sdk` | | fetch fails (`.npmrc` sends `@anthropic-ai` to `127.0.0.1:9`) | warning on the first line after the failed install line, `executor patch`; `report.json` has `executor: "patch"` and `sdkMissing` with the reason; PR body line 3 is the note; no partial cache left; exit 0 | | cache dir can't be created (`UPTIDE_CACHE_DIR=/etc/passwd/cache`) | loud fallback with `ENOTDIR` in the reason, no crash (df41ef8). The run then exits 2 because Uptide's registry cache can't be written either. That is pre-existing and unrelated. | An earlier round, on the optional-dependency build, also confirmed that npm, pnpm 10, yarn 1 and yarn 4 `dlx` resolve and run the SDK from where they install it. `pnpm turbo run typecheck test` for `uptide` and `@uptide/core` passes. `biome check .` has only the pre-existing warning and info. ## Pilot rerun: Uptide arm only, packed tarball via plain `npx -y file:<tgz>` Same harness as round 3, `claude-sonnet-5-5`, $1 (`--max-cost` default), human references from round 3. All three runs report `executor: sdk`. | repo | run | executor | frozen install | types | lint | tests | build | fork CI | cost | wall | coverage vs human | |---|---|---|---|---|---|---|---|---|---|---|---| | calmcp, zod 3→4 | **this PR** | sdk | ok | ok | ok | ok | ok | same as human: only `npm audit` (×2) and `dependency-review` fail, both environmental | $0.09 | 53 s | 6/27 | | | 0.15.1 (#169) | patch | ok | 5 errors | ok | 1 fail | fail | fail: env, plus `Tool surface` and `docker` | $0.03 | 52 s | 1/27 | | dms-saas, stripe 18→22 | **this PR** | sdk | ok | 10 errors | ok | 1 fail³ | fail | fail at typecheck | $1.02 | 612 s | 0/51 | | | 0.15.1 (#169) | patch | ok | 5 errors | ok | pass³ | fail | fail at type-check | $0.18 | 242 s | 6/51 | | enki, react 18→19 | **this PR** | sdk | ok | ok | ok | ok (58/58) | ok | fails only at SonarQube Scan (no secret on forks, as human) | $0.18 | 164 s | 10/29 | | | 0.15.1 (#169) | patch | ok | 4 errors | ok | ok | fail | fails only at SonarQube Scan | $0.06 | 158 s | 9/29 | ³ The MongoMemoryServer flake: `private-admission.test.ts` timed out at 5 s. - **calmcp and enki:** every local check now passes. Round 3 failed the type-check and build on both. - On enki, the session also replaced the undeclared `@react-types/datepicker` import with `@entur/datepicker`'s own `TimeValue`. That was round 3's lockfile finding. The scorer counts those 2 hunks as "unrelated" only because the human left that file alone. - **dms-saas got worse, and not because of the executor's edits.** Four SDK sessions ran (the site session plus 3 feedback rounds). The gate rejected the first three with `tests fail: packages/dms-saas`; the local check shows that same flaky test failing. The fourth stopped at the budget. Every edit was discarded, so the branch is only the version bump, the formatter and the lockfile. That gives 0/51 coverage and the full set of upstream type errors. The gate letting one known flaky test throw away correct-looking sessions is its own follow-up, not part of this PR. - **Cost:** dms-saas ended $0.02 over the $1 limit. The last SDK session stopped at `error_max_budget_usd` after crossing it. - **Not rerun after the redesign:** the pilot ran on the optional-dependency build (77bfe7c, tarball sha256 `a2408063…`). The executor path didn't change after that: same SDK 0.3.296, same binary, same session code. Only where the SDK is imported from changed, and the smoke above covers that. - Fork PRs, `uptide-bench` only: [calmcp#9](uptide-bench/consetto--calmcp#9), [dms-saas#11](uptide-bench/AntelopeJS--dms-saas#11), [enki#8](uptide-bench/entur--enki#8). The harness change (an `UPTIDE_SPEC` override, plus executor read from `llm.executor`) was applied to a copy of `bench/`, not committed. ## For review - **Terms:** ADR 0016 has an open question to Anthropic, "Is an npm dependency 'preinstalling' Claude Code in a product?". Installing the published package on first use is the same question in another form. The ADR records it as still open. - **Windows:** the fetch spawns `npm.cmd` without a shell, which is untested. If it fails there, the run falls back loudly. - **Pre-existing, not touched:** `UPTIDE_CACHE_DIR` under `/proc` makes `check` and `fix` spin forever before any output. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Lucas Braum <lucasbraum@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

lucasbraum · 2026-10-18T22:20:11Z

Round 4: published `uptide@0.16.0` (SDK required) vs Claude Code (2026-10-18), same 3 repos, 1 run each

Setup:

Uptide arm: npx -y uptide@0.16.0 fix <pkg> --target <B> --yes --json, with UPTIDE_CACHE_DIR set to an empty directory per run. Package and target are the same as every earlier round. Each run's meta has version.uptide: "0.16.0" and executor: "sdk", and sdkDownload holds the CLI's line: Agent SDK 0.3.296 is not installed: installing it once into <run>/uptide-cache/agent-sdk/0.3.296 … 280 MB, reused by later runs. All three runs downloaded the SDK, and its download time is inside the wall time.
Claude Code arm: 2.1.283, claude-sonnet-5-5 at medium effort, --max-budget-usd 1. Same prompt, sandbox settings and parent commits as round 3.
Both arms: a fresh clone at the same parent commit, with the parent installed (untimed) before the clock starts.
Human references: round 3's runs, reused unchanged.
Harness: bench/ from this branch rebased on origin/main 63b7c9b, plus round 3's local commit and one more. That commit adds the 0.16.0 command and records executor from llm.executor and the SDK line in meta. It also fixes judge.sh --collect: gh pr checks --watch returned at once while a new PR had no check runs yet. It now waits until the PR has check runs, none of them is pending, and the count hasn't changed for a minute, and only then reads CI. Both commits are local, not pushed.
Reruns: none. Every run ran once and no run hit an infrastructure failure.

calmcp, zod 3.25.76→4.6.5

run	arm	executor	frozen install	types	lint	tests	build	flaky	fork CI	cost	wall	files (code + lock/manifest)	coverage vs human
round 4 (0.16.0)	Uptide	sdk	ok	ok	ok	ok	ok	none	same as human: env only¹	$0.08	62 s	2 + 2	6/27
	Claude Code		ok	ok	ok	ok	ok	none	same as human: env only¹	$0.10	45 s	2 + 2	6/27
#237 (tarball, SDK)	Uptide	sdk	ok	ok	ok	ok	ok		env only¹	$0.09	53 s	2 + 2	6/27
round 3 (0.15.1)	Uptide	patch	ok	5 errors	ok	1 fail	fail		fail: env, plus `Tool surface` and `docker`	$0.03	52 s	1 + 2	1/27
	Claude Code		ok	ok	ok	ok	ok		env only¹	$0.11	49 s	2 + 2	7/27

¹ CI/build (22) and CI/build (24) fail at "Audit runtime dependencies" (npm audit sees an advisory published after the commit), and dependency-review fails on forks. All of these fail on the human reference too. Tool surface/diff, btp-descriptors and docker pass on both arms.

vs human: both arms changed the same 2 of the human's 5 code files (zodDoc.ts, create.ts), including uuid()→guid(). Both missed the same 3 files: index.ts, schema.test.ts and tool-surface.mjs. Neither added the human's .strict() on the tool schemas, the escaped-regression candidate every round has had.
Uptide: 1 SDK session, 0 feedback rounds, verification passed. It is the same result as feat(fix): fetch the Agent SDK on the first fix, never fall back silently (UPT-108) #237.
PRs: Uptide, Claude Code.

dms-saas, stripe 18.5.0→22.6.2

run	arm	executor	frozen install	types	lint	tests	build	flaky	fork CI	cost	wall	files (code + lock/manifest)	coverage vs human
round 4 (0.16.0)	Uptide	sdk	ok	ok	ok	ok	ok	none	pass (`ci/ubuntu`)	$0.27	222 s	9 + 2	10/51
	Claude Code		no commit²						no PR	$0.18	95 s	0 committed (7 + 2 uncommitted)	0/51 (8/51 uncommitted)
#238 (tarball, SDK)	Uptide	sdk	ok	ok	ok	ok	ok	none	pass	$0.27	234 s	9 + 2	10/51
#237 (tarball, SDK)	Uptide	sdk	ok	10 errors	ok	1 fail	fail	(not detected)	fail at typecheck	$1.02	612 s	1 + 2	0/51
round 3 (0.15.1)	Uptide	patch	ok	5 errors	ok	pass³	fail		fail at type-check	$0.18	242 s	7 + 2	6/51
	Claude Code		ok	ok	ok	1 real fail³	ok		fail at `test:unit`	$0.25	121 s	8 + 2	13/51

² Claude Code finished without committing, although the prompt says "Commit". The run ended normally after 17 turns, with no permission denials. Its summary says, "I have not committed anything. Given the failures above and the API-version decision, I'd rather you look first." By the harness's rule the result is its head, which is the parent: no PR, and the local checks passed only because they ran on the parent. I did not rerun, because this is the arm's own behavior, not an infrastructure failure.

The uncommitted tree, outside the score: I checked it as a stash commit, with no PR. It fails the type-check and the build on 4 errors in webhook-credit-notes.ts (the open-ended credit-note type/status enums). Lint passes. Tests: 672/673, and the one failure is the known flake: private-admission.test.ts timed out at 5 s and passes 16/16 alone. Coverage is 8/51.
Its summary reports about 570 baseline type errors. That's because it ran the type-check without building interface-dms-saas first. It also reports 11 Mongo-backed test files that can't start inside its sandbox.

³ The MongoMemoryServer flake in private-admission.test.ts, as in earlier rounds.

Uptide: 1 SDK session, accepted, 0 feedback rounds, with no file recorded as flaky. The flake did not fire in this run, locally or inside Uptide, so UPT-111's rerun path was again not exercised on the real flake. The result matches fix(fix): rerun a session's failing test files once before rejecting its edits (UPT-111) #238: same file count, same coverage, about the same cost.
vs human: the human changed 21 code files. Uptide changed 7 of them plus 2 test files the human left alone (stripe-basil-payloads.test.ts, stripe-credit-note-webhook.test.ts). Neither arm made the billing_mode, managed_payments, checkout or provisioning changes, the escaped-regression candidates from round 1.
PRs: Uptide. Claude Code has none.

enki, react 18.3.1→19.2.0

run	arm	executor	frozen install	types	lint	tests	build	flaky	fork CI	cost	wall	files (code + lock/manifest)	coverage vs human
round 4 (0.16.0)	Uptide	sdk	ok	ok	ok	ok	ok	none	fails only at SonarQube Scan (no secret on forks, as human)	$0.14	180 s	7 + 2	10/29
	Claude Code		fail (ERESOLVE)	not run	not run	not run	not run		fail at `Run npm ci`	$0.10	64 s	7 + 2	7/29
#237 (tarball, SDK)	Uptide	sdk	ok	ok	ok	ok	ok		SonarQube only	$0.18	164 s	10 + 2	10/29
round 3 (0.15.1)	Uptide	patch	ok	4 errors	ok	ok	fail		SonarQube only	$0.06	158 s	6 + 2	9/29
	Claude Code		fail (ERESOLVE)	not run	not run	not run	not run		fail at `npm ci`	$0.11	67 s	7 + 2	10/29

Fork CI: Pull request build (external) is skipped for a PR opened inside the fork on both arms, as before. Pull request build and deploy/build_and_preview fails at different steps. For Uptide, npm ci, prettier, tests and build pass, and it stops at SonarQube Scan, which is environmental. For Claude Code it stops at npm ci, which is a real failure.
Claude Code: the same ERESOLVE in all 4 rounds. It keeps @entur/react-component-toggle@0.0.3, whose react@^18.3.1 peer can't be satisfied, next to react@19.2.0 and react-leaflet@5.0.0.
Uptide: 2 SDK sessions, 1 feedback round, verification passed, with no file recorded as flaky.
vs human: both arms changed the same 7 of the human's 11 code files. Both missed Markers.tsx, QuayPopupButtonPanel.tsx, QuaySequenceIndexTooltip.tsx and usePopupOpeningOnFocus.ts. feat(fix): fetch the Agent SDK on the first fix, never fall back silently (UPT-108) #237 also touched 3 files the human didn't, including the @react-types/datepicker replacement. This run touched none outside the human's files.
PRs: Uptide, Claude Code.

Summary

repo	Uptide 0.16.0	Claude Code
calmcp	all green, fork CI env-only	all green, fork CI env-only (tie)
dms-saas	all green, fork CI passes	no commit; the uncommitted tree fails types and build
enki	all green locally; fork CI env-only (Sonar)	lockfile doesn't install (4th round in a row)

Uptide 0.16.0 is mergeable on all 3 repos for the first time: every local check passes and fork CI fails only where the human reference fails too.
Uptide vs its own earlier runs:
- 0.16.0 reproduces the feat(fix): fetch the Agent SDK on the first fix, never fall back silently (UPT-108) #237 results on calmcp and enki, and the fix(fix): rerun a session's failing test files once before rejecting its edits (UPT-111) #238 result on dms-saas, within about $0.04 and 30 s, so the SDK path works as published.
- Against round 3's patch executor, it fixes every regression.
Claude Code:
- On calmcp it matches Uptide.
- On dms-saas it didn't deliver: no commit, and its tree has 4 type errors.
- On enki it has failed npm ci in all 4 rounds.
- Its coverage moved by ±3 hunks across rounds, about the noise of one run.
Escaped-regression candidates are unchanged: calmcp's .strict(), and dms-saas's billing_mode, managed_payments and provisioning changes. Both arms miss them, every round.
Cost: Uptide $0.50, Claude Code $0.38, both priced with Uptide's table. Uptide's wall time includes three 280 MB SDK downloads.
One run per pair: ties and small gaps are not significant.

Notes

Fork CI waits: every PR's checks were read only after they registered and finished, with no timeout.
Flaky files: none in any committed run, in Uptide's llm.tasks[].flaky or locally. The only flake is in the uncommitted Claude tree, re-run alone as in earlier rounds.
Spend: $0.50 Uptide + $0.38 Claude Code = $0.88 (Claude Code's own price for its runs: $0.66).

🤖 Generated with Claude Code

lucasbraum · 2026-10-18T22:55:47Z

Round 5: Claude Code on `claude-fable-5-1` (2026-10-18), same 3 repos, 1 run each

Setup:

Claude Code arm only: Claude Code 2.1.283, claude-fable-5-1 at medium effort, --max-budget-usd 5. The sandbox settings and parent commits are the same as round 4. Each run's meta records model: "claude-fable-5-1".
Prompt: round 4's task text, plus "When done, commit all your changes on the current branch. Do not ask for review."
Uptide side: round 4's runs as is (0.16.0 on claude-sonnet-5-5). Not rerun.
Both: a fresh clone at the same parent commit, with the parent installed (untimed) before the clock starts. Each fork PR is in uptide-bench, and its CI was read with round 4's wait: until check runs exist, none is pending, and the count holds for a minute.
Human references: round 3's, reused unchanged.
Harness: round 4's bench/ (fc02550) plus one local commit. That commit adds BENCH_MODEL and CLAUDE_PROMPT_EXTRA, a Fable row in the cost table (Uptide's pricing.ts: $10 input, $50 output, $0.25 cache read, $12.5 cache write, per million), and model and claudeCodeCostUsd in meta. Not pushed.
Reruns: none. All 3 runs ran once and none hit an infrastructure failure.
Not like for like with round 4's Claude Code column: the prompt gained the commit sentence and the model changed, both at once. The dms-saas difference below is likely due to the prompt at least as much as to the model.

calmcp, zod 3.25.76→4.6.5

run	frozen install	types	lint	tests	build	flaky	fork CI	cost	wall	files (code + lock/manifest)	coverage vs human
Claude Code / Fable	ok	ok	ok	ok (248)	ok	none	same as human: env only¹	$1.50	227 s	3 + 2	7/27
Uptide 0.16.0 / Sonnet (r4)	ok	ok	ok	ok	ok	none	env only¹	$0.08	62 s	2 + 2	6/27
Claude Code / Sonnet (r4)	ok	ok	ok	ok	ok	none	env only¹	$0.10	45 s	2 + 2	6/27

¹ CI/build (22) and CI/build (24) fail at "Audit runtime dependencies", on high-severity advisories against @modelcontextprotocol/sdk and node-forge published after the commit. dependency-review fails with "not supported on this repository" (no dependency graph on the fork). The human reference fails the same checks. Tool surface/diff, btp-descriptors and docker pass.

vs human: it changed the same 2 code files as both round-4 arms (zodDoc.ts, create.ts), including uuid()→guid() with the same reasoning as the human. It also changed one more, calmCreate.ts: issue.path.map(String), a type-level fix the human didn't need.
- In zodDoc.ts, it recognizes string formats by schema.type === 'string', where the human used instanceof z.ZodStringFormat. It reads descriptions through .description, where the human used z.globalRegistry. Same behavior for the schemas here.
- It missed index.ts, schema.test.ts and tool-surface.mjs.
Semantic change no check catches: it did not add the human's .strict() on the tool input schemas. Its commit message notes that the MCP SDK now converts schemas with zod 4's toJSONSchema, but it didn't notice that zod 4 dropped additionalProperties: false. So a parameter the model invents is now dropped without an error. This is the same escaped-regression candidate every arm has missed in every round.
PR: Claude Code / Fable.

dms-saas, stripe 18.5.0→22.6.2

run	frozen install	types	lint	tests	build	flaky	fork CI	cost	wall	files (code + lock/manifest)	coverage vs human
Claude Code / Fable	ok	ok	ok	672/673, flake only²	ok	`private-admission.test.ts`	pass (`ci/ubuntu`)	$3.37	486 s	9 + 2	14/51
Uptide 0.16.0 / Sonnet (r4)	ok	ok	ok	ok	ok	none	pass	$0.27	222 s	9 + 2	10/51
Claude Code / Sonnet (r4)	no commit						no PR	$0.18	95 s	0 (7 + 2 uncommitted)	0/51

² The known MongoMemoryServer flake: private-admission.test.ts timed out at 5 s in the full run, then passed 16/16 alone 5 times out of 5. The run doesn't touch that file. Fork CI ran the full suite green.

It committed this time: 1 commit and a clean tree. Round 4's Sonnet run, without the extra sentence, ended with "I'd rather you look first" and no commit.
vs human: it changed 7 of the human's 21 code files, the same 7 as Uptide, plus deferred.ts and stripe-basil-payloads.test.ts, which the human renamed.
- It pinned STRIPE_API_VERSION to 2026-08-26.dahlia as the human did, and updated the test assertion where the human renamed and extended the file.
- Its credit-note open-enum handling matches the human's almost line for line: a "mixed" fallback for unknown types and a shared resolveCreditNoteStatus.
- It made the subscriptions.cancel(id, undefined, opts) change at all 4 sites.
Semantic changes no check catches:
- iterations: 1 → duration: the human reads the price's recurring.interval and interval_count from Stripe and throws on a non-recurring price. Fable passes targetPlan.interval from the database with interval_count: 1 hard-coded, and adds an interval field to ScheduleDowngradeInput. The two match for prices this code creates, but a Dashboard price with interval_count > 1 would get a phase of 1 interval instead of 1 billing cycle. Its summary flags this as "the only runtime-behaviour change" and asks for a manual test-mode check.
- Clover's billing_mode default: the human pins new subscriptions to billing_mode: classic because clover's flexible default changes proration and Customer Portal cancellations. Fable didn't, though its summary says "I read the v19–v22 changelogs; nothing else this codebase touches was removed". It also missed the managed_payments, checkout and provisioning (payment_behavior: "default_incomplete") changes. These are the same escaped-regression candidates as every round, and Uptide misses them too.
- Its sandbox couldn't start mongod (exit 48), so 11 Mongo-backed test files (167 tests) never ran inside the session. It said so in its summary.
PR: Claude Code / Fable.

enki, react 18.3.1→19.2.0

run	frozen install	types	lint	tests	build	flaky	fork CI	cost	wall	files (code + lock/manifest)	coverage vs human
Claude Code / Fable	ok	ok	ok	ok (58)	ok	none	same as human: fails only at SonarQube Scan³	$3.28	558 s	8 + 2	14/29
Uptide 0.16.0 / Sonnet (r4)	ok	ok	ok	ok	ok	none	SonarQube only³	$0.14	180 s	7 + 2	10/29
Claude Code / Sonnet (r4)	fail (ERESOLVE)	not run	not run	not run	not run		fail at `npm ci`	$0.10	64 s	7 + 2	7/29

³ In Pull request build and deploy/build_and_preview, npm ci, npm run check and the tests pass, and the job stops at SonarQube Scan, which has no secret on forks. Pull request build (external) is skipped for a PR opened inside the fork.

First Claude Code run to pass npm ci on enki in 5 rounds. It found that the existing overrides block ("@reach/dialog": {"peerDependencies": {...}}) was a no-op, because npm reads peerDependencies there as a package name. It rewrote the block to "react": "$react" and added an override for @entur/react-component-toggle. It also added one for react-side-effect, dropped the @azure/msal-react entry, and removed @types/react-leaflet. Its summary reports npm ls with 0 invalid peers.
vs human: it changed 8 of the human's 11 code files, one more than Uptide (usePopupOpeningOnFocus.ts). It missed Markers.tsx, QuayPopupButtonPanel.tsx and QuaySequenceIndexTooltip.tsx.
Semantic change no check catches: it kept @entur/react-component-toggle@0.0.3, whose peer is react@^18.3.1, and forced it onto React 19 through the override. The human and Uptide upgraded it to 1.0.0, and the human also upgraded @entur/rollup-plugin-react-component-toggle to 1.0.0. Fable compared only against the latest, 1.1.2, which needs react@^19.2.7, and decided against 1.x. The toggle is imported in index.tsx, App, LandingPage and NavBar, and its sandbox-feature loading has no test. The human also added rxjs 7.8.2 and moved @types/react to 19.2.2, where Fable chose 19.2.18. Fable's source changes are type-level (useReducer generics, useRef initial values, RefObject), as are both other arms'.
PR: Claude Code / Fable.

Summary

repo	Uptide 0.16.0 / Sonnet (r4)	Claude Code / Fable (r5)	Claude Code / Sonnet (r4)
calmcp	all green, fork CI env-only, $0.08, 62 s, 6/27	all green, fork CI env-only, $1.50, 227 s, 7/27	all green, fork CI env-only, $0.10, 45 s, 6/27
dms-saas	all green, fork CI passes, $0.27, 222 s, 10/51	green except the known flake, fork CI passes, $3.37, 486 s, 14/51	no commit, $0.18, 95 s
enki	all green, fork CI Sonar-only, $0.14, 180 s, 10/29	all green, fork CI Sonar-only, $3.28, 558 s, 14/29	`npm ci` fails (ERESOLVE), $0.10, 64 s, 7/29
total	$0.50, 7.7 min	$8.14, 21.2 min	$0.38, 3.4 min

Fable ties Uptide on mergeability: all 3 repos pass the local checks and fork CI fails only where the human reference fails. It's the first Claude Code configuration to do that. On dms-saas the commit sentence probably mattered as much as the model.
Fable covers more of the human's changes: 7/27, 14/51 and 14/29, against Uptide's 6, 10 and 10. With one run per pair, a few hunks is within noise, except enki's install fix. Sonnet failed there 4 rounds in a row and Fable got it right on the first try.
Cost and time: Fable cost 16× Uptide ($8.14 vs $0.50) and took 2.7× the wall time (21 vs 8 min), with 34, 52 and 65 turns. dms-saas, at $3.37, came closest to the $5 cap. Fable's list price is 5× Sonnet's, and it also used more tokens: 1.7M cache reads on dms-saas alone.
Semantic risks no check catches:
- The same escaped-regression candidates as every round: calmcp's .strict(), and dms-saas's billing_mode, managed_payments and provisioning changes. Fable misses them too, as do Uptide and Claude Code on Sonnet.
- New this round: Fable added two behavioral choices the other arms didn't make: forcing react-component-toggle@0.0.3 onto React 19 through an override, and taking the schedule phase's duration from the database plan instead of the Stripe price. It flagged both in its summary, with its reasons.
Loss for Uptide on coverage, win on cost: at matched mergeability, Uptide is roughly 16× cheaper and 2.7× faster, and Fable produces somewhat fuller diffs.

Notes

Spend: $8.14, all Fable. Claude Code's own price for each run matched Uptide's table to the cent ($1.50, $3.37, $3.28), so no repricing was needed. The cap held, with the highest run at $3.37.
Flaky files: only dms-saas's private-admission.test.ts, rerun alone as in earlier rounds.
Fork CI waits: every PR's checks were read only after they registered and finished, with no timeout.
Fable worked around the sandbox: on enki, npm ci in the repo hit a sandbox deny on an .idea/ folder inside a tarball, so Fable proved the lockfile with npm ci in a temp directory. On dms-saas, mongod couldn't bind. Neither affected the harness's own checks, which run outside the sandbox.

🤖 Generated with Claude Code

selection.json gains calmcp's five `silent` sites, the inputSchema lines where the human's upgrade added .strict() so tools/list keeps advertising additionalProperties: false (ADR 0017). score.mjs counts a site as caught when the run's diff touches its line and the file at the run's head has the keep edit, or when the PR body names the record; a site a PR body's "Behavior change (<record>)" line names that the selection lacks is a silent false positive. Both arms are scored by the same two columns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

) The SDK pre-run estimate charged a full uncached token allowance for every site even though one session shares rule knowledge. On alanagoyal this predicted $1.08 for 18 sites and spent $0.1336, forcing an unnecessary budget increase. Estimate SDK sessions from distinct migration rules, a small per-site context allowance, and a session minimum, with an explicit 1.5× safety margin applied before rounding to cents. Keep declaration-only anchors in the site allowance without charging another independent rule. The patch executor's estimate and both executors' hard budget checks remain unchanged. | Recorded run | Before | Actual | After | After / actual | | --- | ---: | ---: | ---: | ---: | | alanagoyal, React 18 → 19 | $1.08 | $0.1336105 | $0.23 | 1.72× | | calmcp, pilot round 4 | $0.30 | $0.0842459 | $0.14 | 1.66× | | dms-saas, pilot round 4 | $0.60 | $0.2740438 | $0.42 | 1.53× | | enki, pilot round 4 | $0.54 | $0.1431526 | $0.29 | 2.03× | The original reports and bench metadata agree. [#169](#169 round 5 reuses round 4's Uptide runs, so it is not counted again. Mechanical sites were already correctly excluded. The old token allowance implicitly covered retries; it had no explicit feedback multiplier. These runs used zero or one feedback round, not three. Recorded fixtures and `docs/cost-estimate-calibration.md` document the evidence, formula, and limits. This is a four-run in-sample calibration, not a guarantee on unseen migrations. The safety margin reduces the risk of turning a free preflight stop into a paid mid-run budget stop. Recorded-pair tests require estimates between 1.3× and 2.5× actual spend. No paid runs were made. Install caching is outside this PR's scope. UPT-14/UPT-111 behavior is preserved: preflight rejects estimates above the limit, equality is allowed, Action/App runs work within the original limit, feedback spends only the remainder, and flake reruns and per-call reservations are unchanged. Tests cover recorded pairs, mechanical exclusion, anchors, model pricing, and SDK preflight/runtime budget boundaries. Validation passed: `pnpm lint` (existing Biome schema notice and fixture warning), `pnpm typecheck`, and the estimator and generic fix tests. Tests also verify that the margin is applied before cent rounding. Refs UPT-94 (cost-estimate part only). --------- Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

…253) Stripe's clover request API changes omitted billing mode to flexible without a compiler error. This adds cited behavior records, API-specific preserving edits, and decisions in the existing report sections. Coverage and behavior: - Cover all six APIs in the pinned changelog: subscriptions, invoice previews, subscription schedules, quotes, subscription-mode Checkout, and Payment Links. Exact paths are root `billing_mode` for subscriptions/schedules, `subscription_data.billing_mode` for Checkout/quotes, and `subscription_details.billing_mode` or `schedule_details.billing_mode` for invoice previews. Existing-subscription schedules/previews are excluded. Unknown quote recurrence and Payment Links' undocumented preserving parameter become cited decisions. - Match the effective request API version. Retained pre-clover pins produce no finding. Selected pin migrations add creation sites to the agent plan; post-edit verification checks the actual pin and classic value. Raw compiler findings are classified with the existing rule classifier. Payload `api_version` values never move clients. SDK 12/13 upgrades are supported; the record's SDK range is a candidate filter, not the behavior boundary. - Remove full Stripe pages and `.gitattributes`. Records keep quoted lines, URLs, read dates, and full-page SHA256 hashes. `build:stripe-changelog` fetches, verifies hashes and quotes, fails on drift requiring author review, and writes snapshots only to a gitignored cache. The zod snapshot contract is retained. Independent dms-saas audit of parent `6390f434622db5d6e546854b435fb1017c3cc665`, checked against human PR #70: - Existing expected sites remain `packages/dms-saas/src/workspaces/provisioning.ts:355` and `packages/dms-saas/src/stripe/workspace-checkout.ts:211` (the parameter builder shared by the initial/retry wrapper). - Additional calls: `src/stripe/plan-schedule.ts:52` uses `from_subscription`; `src/stripe/upcoming-invoice.ts:107` previews an existing `subscription`. Neither creates a subscription with the new default. No quotes or Payment Links calls exist. No ground-truth entries were added or changed based on Uptide output. Full rationale and source references are in `packages/core/src/packs/stripe/RESEARCH.md`. Validation: - Rebased onto **#251 / UPT-94** (`773dbbe`). A regression test passes real quoted Stripe findings through the pack transform and `sdkCostEstimate`: they remain manual sites under `stripe/billing-mode-default-flexible`, add their own rule cost, and repeated occurrences count as separate sites while sharing that rule cost. No production cost-estimator changes were needed. - Post-rebase validation: **790 core tests passed / 14 skipped**, including all fix tests, behavior records, Stripe triggers and cost estimates; TypeScript and formatting checks pass. The first broader invocation omitted the package config and was superseded by this successful configured run. - After the rebase, the dms-saas parent scan still matches exactly the same two sites and preserving edits. Per the conditional rerun request, no additional benchmark run or spend was incurred; the successful fork run below remains the benchmark evidence. - **13/13 verified-pack gates passed; no FP regressions against origin/main.** All breaking and silent FP counts remain zero. Existing behavior-note FPs (Redis 2, TypeScript 8) and auxiliary mechanical tallies (Next 1, React Router 336) match origin/main. No expectations or gate were relaxed. The broader offline command exits 1 only because candidate packs ioredis, joi, openai and @sentry/react lack cached repositories; every verified pack passes. - Stripe standalone pack test passed and recorded verified: breaking **23 TP / 0 FP / 2 FN**; the frozen silent-oracle discrepancy is explained below. - Source fetch/hash/quote build, TypeScript check, build, formatting and diff checks pass. **22 trigger/plan tests**, **150 fix/SDK/record/pack-contract tests**, and **5 benchmark scorer tests** pass. Two initial five-second timeouts under concurrent pack load passed on rerun with a 30-second test timeout. - Uptide-only #169 rerun `4776187e`, commit `f82fca0`: [dms-saas fork PR #16](uptide-bench/AntelopeJS--dms-saas#16), **MERGEABLE**, **2/2 silent sites caught and edited**, **0 silent FP**, **$0.2669476**, **255 seconds**, exit **0**. Both original sites resolve to explicit classic afterward. Independent harness install, typecheck, lint and tests all exit **0**. The isolated clone's pre-existing missing frontend dependencies are resolved by the independent full install. - An earlier validation rerun exposed the raw-finding rule-classification bug and caught 0/2; it was fixed and regression-tested before the successful run. That attempt cost **$0.2888555 / 331 seconds**. Total for the two review reruns: **$0.5558031 / 586 seconds** (setup excluded). Neither attempt changed ground truth. The frozen unkey pack expectations still include two creation sites whose clients retain `2023-10-16`. Correct pin handling suppresses those findings in `check`, so the unchanged silent oracle scores **0 TP / 0 FP / 2 FN**. This discrepancy is disclosed and recorded in verification.json; the ground truth was not adjusted after scoring. During `fix`, selecting the pin migration activates the creation-site findings, as the dms-saas rerun verifies. Changeset: **minor**, for the new runtime-behavior detection capability. Signed-off commits, rebased on origin/main; shared run/report edits remain localized. No upstream third-party writes or CI watching. UPT-113 --------- Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

lucasbraum · 2026-10-19T11:26:31Z

UpgradeBench v1 pre-run: 5 unseen repos, Uptide 0.17.1 (Sonnet, $1) vs Claude Code (Fable, $5), once each

Selection, arms and the rule's amendment were committed and pushed before any arm ran. They are in bench/v1/repos.md and bench/v1/arms.md, and the full report is in bench/v1/pre5/results.md, all in #256.

The amendment, 2026-10-19T10:04Z: a second source, merged Renovate/Dependabot major bumps that a human fixed before merge. Human commit search alone found too few qualifying upgrades. "tsc clean at parent" and "license OK" are now stated outright.
Mergeable: every local check and fork CI result passes, or fails only where the human commit's fails.

repo	pack	Uptide 0.17.1 (Sonnet)	Claude Code (Fable)
dekkerglen/CubeCobra	express 4→5	no PR: refused because npm 11 ≠ `packageManager` npm@10.9.0 · $0.00 · 57 s	no PR: $5 cap after 53 turns, 28 files uncommitted · $5.09 · 861 s
ZM-BAD/DAG-chat	vitest 4→5	no PR: "vitest is not a dependency" (app in `frontend/`) · $0.00 · 0 s	mergeable: green, CI 15/15 · $0.73 · 129 s
brendanlong/lion-reader	@sentry/nextjs 10→11	not mergeable: test gate rejected a fixture fix, so the type-check fails and CI fails Type Check, Unit, E2E · $0.48 · 522 s	no PR: $5 cap after 79 turns, 8 files uncommitted · $5.06 · 711 s
sCypher-me/cypher-browser	lucide-react 0→1	mergeable: checks as human, CI pass, exit 1 only because the baseline build fails too · $0.19 · 109 s	mergeable: checks as human, CI pass · $1.33 · 201 s
GlennLab/LlamaDesk	react-router 6→7	mergeable: green, CI 2/2 · $0.05 · 48 s (rerun¹)	mergeable: CI 2/2, one flaky test locally · $0.34 · 53 s
mergeable		2/5	3/5
cost, total (median)		$0.72 ($0.05)	$12.55 ($1.33)
wall, total (median)		12.3 min (57 s)	32.6 min (201 s)

¹ The one infrastructure rerun: Uptide named its branch uptide/react-router-7.18.4, and the harness looked for uptide/react-router-dom-*. The harness is fixed, and the first run ($0.04) is kept outside the score.

Uptide's losses are engine bugs, written up as Linear-ready issues in results.md, not fixed here:
1. A packageManager npm pin refuses a run that would install fine.
2. Nested non-workspace projects aren't found.
3. The test gate blocks type-only fixture updates.
4. A build that fails identically at baseline fails verification.
Claude Code's losses are both budget: it commits only at the end, so $5 ran out with work uncommitted. That work mirrored the human's on both repos.
Semantic changes no check catches:
- On lion-reader, Uptide's dataCollection uses a cookie deny list, which collects session cookies; the human set cookies: false.
- On cypher-browser, each run picked a different replacement for lucide's removed brand icons. Claude Code re-vendored the old logos.
Uptide's silent-change findings: relevant on all 3 repos where it ran: 7 on lion-reader, the human acting on 3 of them; the aria-hidden change on cypher-browser; and startTransition on LlamaDesk.
Fork CI that tells nothing: on CubeCobra, every job skips on forks (if: github.repository == …), the human's too.
One run per pair: these are signals, not significance.

🤖 Generated with Claude Code

lucasbraum and others added 6 commits October 15, 2026 21:59

bench: format the sandbox settings (UPT-24)

f0cf19a

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

bench: read Uptide's cost and PR body from its JSON, skip PRs for run…

48aa022

…s without commits (UPT-24) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>

lucasbraum added the no-changeset Change needs no release note label Oct 16, 2026

This was referenced Oct 17, 2026

fix(fix,zod): TS 7 libs in the fix type-check, uuid() as breaking (UPT-100) #219

Merged

fix: unsafe root pins become agent-led drafts, with a clean-install gate (UPT-102) #220

Merged

lucasbraum mentioned this pull request Oct 18, 2026

fix: dms-saas cancel site survives feedback, oxfmt per workspace (UPT-101) #221

Merged

7 tasks done

lucasbraum mentioned this pull request Oct 18, 2026

fix: untracked files written by baseline tests are test artifacts (UPT-105) #223

Merged

lucasbraum mentioned this pull request Oct 18, 2026

fix: retry npm ERESOLVE with the smallest re-resolve, blocked report otherwise (UPT-107) #224

Merged

This was referenced Oct 18, 2026

fix(fix): judge consumed builds on the final head, not after the bump (UPT-104) #226

Merged

fix: an oversized PR body is shortened, never a lost run (UPT-106) #227

Merged

lucasbraum mentioned this pull request Oct 18, 2026

perf(fix): cut install copies and the agent's test check, count commands per phase (UPT-103) #232

Merged

7 tasks done

This was referenced Oct 18, 2026

fix: the PR's build line states the build's result, never "not required" in its place (UPT-110) #233

Merged

fix: keep an undeclared transitive the repo imports through the ERESOLVE retry (UPT-109) #234

Merged

lucasbraum mentioned this pull request Oct 18, 2026

feat(fix): fetch the Agent SDK on the first fix, never fall back silently (UPT-108) #237

Merged

lucasbraum mentioned this pull request Oct 18, 2026

fix(fix): rerun a session's failing test files once before rejecting its edits (UPT-111) #238

Merged

This was referenced Oct 19, 2026

feat(packs): behavior records with cited sources; zod's MCP input schema record (UPT-114) #244

Merged

fix(fix): calibrate SDK cost estimates from recorded runs (UPT-94) #251

Merged

feat(stripe): catch silent billing-mode changes end to end (UPT-113) #253

Merged

Navigation Menu

Repository navigation

bench: UpgradeBench v0 pilot harness (UPT-24) - #169

bench: UpgradeBench v0 pilot harness (UPT-24)#169
lucasbraum wants to merge 7 commits into
mainfrom
bench/upgradebench-v0-pilot

lucasbraum commented Oct 16, 2026 •

edited

Loading

Uh oh!

lucasbraum commented Oct 18, 2026

Uh oh!

lucasbraum commented Oct 18, 2026

Uh oh!

lucasbraum commented Oct 18, 2026

Uh oh!

lucasbraum commented Oct 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

lucasbraum commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Harness (bench/)

Results (1 run per pair)

Uptide issues the pilot found

What should change before the full run (10 repos × 3 runs)

Rerun on current main (2026-10-17), same 3 repos, 1 round

Uh oh!

lucasbraum commented Oct 18, 2026

Round 3: published uptide@0.15.1 vs Claude Code (2026-10-18), same 3 repos, 1 run each

calmcp, zod 3.25.76→4.6.5

dms-saas, stripe 18.5.0→22.6.2

enki, react 18.3.1→19.2.0

Summary

Uptide 0.15.1 issues this round found

Notes

Uh oh!

lucasbraum commented Oct 18, 2026

Round 4: published uptide@0.16.0 (SDK required) vs Claude Code (2026-10-18), same 3 repos, 1 run each

calmcp, zod 3.25.76→4.6.5

dms-saas, stripe 18.5.0→22.6.2

enki, react 18.3.1→19.2.0

Summary

Notes

Uh oh!

lucasbraum commented Oct 18, 2026

Round 5: Claude Code on claude-fable-5-1 (2026-10-18), same 3 repos, 1 run each

calmcp, zod 3.25.76→4.6.5

dms-saas, stripe 18.5.0→22.6.2

enki, react 18.3.1→19.2.0

Summary

Notes

Uh oh!

lucasbraum commented Oct 19, 2026

UpgradeBench v1 pre-run: 5 unseen repos, Uptide 0.17.1 (Sonnet, $1) vs Claude Code (Fable, $5), once each

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

lucasbraum commented Oct 16, 2026 •

edited

Loading

Harness (`bench/`)

Round 3: published `uptide@0.15.1` vs Claude Code (2026-10-18), same 3 repos, 1 run each

Round 4: published `uptide@0.16.0` (SDK required) vs Claude Code (2026-10-18), same 3 repos, 1 run each

Round 5: Claude Code on `claude-fable-5-1` (2026-10-18), same 3 repos, 1 run each