Skip to content

fix: unsafe root pins become agent-led drafts, with a clean-install gate (UPT-102) - #220

Merged
lucasbraum merged 1 commit into
mainfrom
lucasbraum/upt-102-pin-refusal-agent-draft
Oct 18, 2026
Merged

fix: unsafe root pins become agent-led drafts, with a clean-install gate (UPT-102)#220
lucasbraum merged 1 commit into
mainfrom
lucasbraum/upt-102-pin-refusal-agent-draft

Conversation

@lucasbraum

@lucasbraum lucasbraum commented Oct 17, 2026

Copy link
Copy Markdown
Collaborator

UPT-102. From the UpgradeBench rerun (#169): on entur/enki (react 18.3.1 → 19.2.0), fix exited 2 before cloning because a nested override "can't be updated safely", with no agent fallback.

1. Unsafe root pin → agent-led draft. planRootPins now collects every pin it cannot update safely (nested or selector keys, non-version values) into one UnsafeRootPinError and names them all. With an assisting agent, the run continues on #188's agentFallback path. The pins are left as written, and each one is named with its exact path and value in the reason at the top of the PR body. pinConsentFallback became assistedRootPins, shared by isolatedFix and fix. Exit 2 is unchanged without an agent, with --no-llm, with updatePins: false, and on SafetyRefusal.

2. Peer blockers named. The preflight already picked the lowest accepting release (compatiblePeerVersion). The agent-led reason only listed names. It now says X 0.0.3 → 1.0.0, the first release that accepts react 19.2.0, or no X release accepts …; 0.0.3 kept with a peer override.

3. Clean-install gate. Before delivery, an agent-led draft runs the repo's frozen install (npm ci --ignore-scripts …, the manager's equivalent elsewhere) in the fix clone, on the final branch. A failure goes right under the reason (This branch does not install.), fails verification.passed, and appears as an Install row in the terminal summary. Verified runs skip it: the upgrade step already runs the same install, and nothing after it may touch manifests or lockfiles.

Tests (fixtures): an unsafe pin refuses without an agent or with --no-llm; with an agent it becomes a draft with the pin unchanged and named in the first lines. A failing clean install heads the PR and fails verification. The peer reason names the first accepting release when a later one rejects the target, or says none does. A root-pins unit test covers naming all unsafe pins and leaving them while the safe ones update. Two existing tests updated: the reason wording, and one extra install call.

enki rerun (same harness, Uptide arm only, claude-sonnet-5-5, $1 cap): the pin refusal is gone. It now stops at a different, pre-existing guard: enki's vitest config writes test-report.xml (vitest-sonar-reporter, not gitignored), so baseline tests modified the working tree. No draft was delivered and $0 was spent. Checked locally on the same tree, the reasons this PR adds read: toggle 0.0.3 → 1.0.0 and react-leaflet 4.2.1 → 5.0.0 as the first releases accepting react 19.2.0, plus seven @reach/*/@azure/msal-react overrides left unchanged. The human's merged upgrade made the same choices. The clean-install gate did not run on enki because the run stopped earlier. The baseline-artifact stop is a follow-up.

Not done: the agent does not edit the override. package.json and lockfiles stay protected from the agent, and changing them would need a reinstall loop. The draft leaves the pin and asks the reviewer to check it.

🤖 Generated with Claude Code

@lucasbraum @claude
fix: unsafe root pins become agent-led drafts, with a clean-install g…
c1b6131 
…ate (UPT-102)

With an assisting agent, a root pin Uptide cannot update safely (enki's
`overrides > @reach/dialog > peerDependencies > react`) no longer exits 2.
It continues on UPT-78's agentFallback path: the pin is left as written
and named, with exact path and value, at the top of the PR body. Every
unsafe pin is collected and named at once (UnsafeRootPinError). Exit 2
stays without an agent, with --no-llm, and on SafetyRefusal.

The agent-led peer reason names each blocker's first release that
accepts the target, or says none does and an override keeps it.

An agent-led draft runs the repository's frozen install (npm ci) in the
fix clone before delivery. A failure heads the PR body and fails
verification.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
@lucasbraum
lucasbraum merged commit cf86d65 into main Oct 18, 2026
28 checks passed
@lucasbraum
lucasbraum deleted the lucasbraum/upt-102-pin-refusal-agent-draft branch October 18, 2026 00:26
@uptide-release uptide-release Bot mentioned this pull request Oct 18, 2026
Merged
@lucasbraum lucasbraum mentioned this pull request Oct 18, 2026
Merged
lucasbraum added a commit that referenced this pull request Oct 18, 2026
@lucasbraum @claude
fix: untracked files written by baseline tests are test artifacts (UP…
2dc2162 
…T-105) (#223)

Closes UPT-105. Refs UPT-102 (not closed: see the enki rerun below).

## What and why

On enki (UpgradeBench, #169), the vitest junit reporter
writes `test-report.xml`, which the repository does not gitignore. The
baseline-tests guard treated any untracked file as a change, so `fix`
stopped with "baseline tests modified the working tree" before doing
anything.

Now:

- Untracked files are snapshotted before the baseline tests
(`untrackedFiles`). After the run, a change to a **tracked** file still
refuses, as before. **New untracked** files are test artifacts
(`testArtifacts` in `verify.ts`).
- Artifacts go in the private clone's `.git/info/exclude`, anchored at
the repo top (with the project's subdirectory prefix, and pattern
characters escaped). No status, diff, `git add` or `ls-files
--exclude-standard` sees them. That keeps them out of commits, the
agent's diff (including tests the agent runs itself), the follow-up and
style steps, and the `verificationPending` check, from one place.
- They are removed right away, and again before each later test run:
`clearing()` wraps `services.tests` once, after baseline, so settle, the
rules' transactions, the agent gate and the feedback rounds all go
through it.
- The report records them (`testArtifacts`). `uptide verify` runs in a
fresh clone, so it re-applies the exclude and clearing from the stored
run; otherwise its final test run would leave the file and fail.
- The same applies to the Stripe pin path (`pin.ts`).

`fix` and `verify` always run in a private clone
(`isolatedFix`/`isolatedVerify`), so writing `info/exclude` never
touches the user's checkout. The comment there names the ceiling if
either ever runs in place.

## Tests

- `fix.test.ts`: the fixture's test script writes an untracked
`test-report.xml` and **fails if the previous run's file is still
there**. Baseline and final verification pass, `testArtifacts` is
`['test-report.xml']`, the file is in no commit and not in
`changedFiles`, nothing is pending. `uptide verify` from a fresh clone
(no exclude entry, no file on disk) also passes.
- `fix.test.ts`: a baseline test run that writes an untracked report
**and** changes a tracked file still refuses.
- `verify.test.ts`: a project in a subdirectory, with names containing
`[ ] # ! space`, is hidden and nothing else is.
- Reverting the run change brings back the original refusal. Reverting
only the reverify change fails the verify assertion.
- `pnpm lint`, `pnpm typecheck`, `pnpm test`: green locally (core 2268
passed).

## enki rerun (#169 harness, Uptide arm only, claude-sonnet-5-5, $1 cap)

Run `7f90b41c`, from this branch's build (8d1d8bb).

| | |
|---|---|
| Baseline tests | **passed the guard** (this fix) |
| Stopped at | lockfile install of react 19.2.0 +
`@entur/react-component-toggle` 1.0.0: `npm install --package-lock-only`
→ `ERESOLVE`, exit 2 |
| Clean-install check (#220) | not reached: no draft |
| Checks | none on a draft: head is the parent commit |
| Cost | $0.00 (no agent call) |
| Wall time | 54 s |
| Fork CI | none: no commits, so no fork pull request |

The cause, reproduced outside Uptide: starting from the parent's
`package-lock.json`, `npm install --package-lock-only` fails with the
same `ERESOLVE` **even with the human upgrade's exact package.json**
(ed37dd2). Without the old lockfile, the same manifest resolves (exit
0). So Uptide's companion set matches the human's. What blocks is npm's
resolution starting from the old lockfile, not the plan. Two defects
show up here: the install step has no fallback for this, and `fix
--json` printed nothing on this exit 2. The run ended as a raw npm error
instead of a "blocked by" result.

UPT-102's done-criterion (a draft that installs, or a clear "blocked
by") is not met yet, so this PR does not close it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@lucasbraum lucasbraum mentioned this pull request Oct 18, 2026
Merged
lucasbraum added a commit that referenced this pull request Oct 18, 2026
@lucasbraum @claude
fix: retry npm ERESOLVE with the smallest re-resolve, blocked report …
09f68df 
…otherwise (UPT-107) (#224)

Closes UPT-107. Refs UPT-102 (not closed: enki delivered no draft and
did not end blocked; see below).

## What and why

On enki (UpgradeBench, #169), `npm install
--package-lock-only` failed with ERESOLVE from the parent's lockfile,
although the same manifest resolves fresh. `fix` stopped with exit 2,
and `fix --json` printed nothing.

**Staged retry** (`managers/upgrade.ts`). For npm only, and only on
ERESOLVE, the lock-only install is retried from smaller lockfiles,
smallest re-resolve first:

1. **moved entries**: the old lockfile without the entries of the moved
packages (lead and companions), every locked package that declares one
of them as a peer, and everything installed below those. Links are kept.
2. **whole lockfile**: every installed entry dropped, so npm resolves
everything fresh.

`--legacy-peer-deps` and `--force` are never used (the tests assert it
on every install call).

- The step that worked is recorded as `lockfile.eresolveRetry` (`{ step,
dropped, conflicts }`). It shows in the text summary and the PR body
("Lockfile re-resolved").
- The parent-commit comparison (`parentLock`) starts from the same
reduced lockfile. So the lock guard reports churn as usual: what the
fresh resolve moves at the parent too shows as `parentChurn`, not as the
upgrade's.
- If every step fails, `fix` ends with `INSTALL_BLOCKED`
(`InstallBlockedError`). Its message lists the package lines from npm's
ERESOLVE report (While resolving / Found / Could not resolve dependency
/ Conflicting peer dependency / `peer … from …`). It crosses the worker
boundary intact.

**`fix --json` on every exit.** Exit 0 and 1 already printed the report.
On a failure (exit 2, and exit 1 from a `CliError`), `fix --json` now
prints `{ error: { code, message } }`. A blocked install adds `blocked:
{ attempts, conflicts }`. This is scoped to `fix`: `pr` already writes
its own error JSON and then rethrows.

**Second commit, found by the rerun.** The paid enki run got past
ERESOLVE and ran the agent. It then stopped at "Required migration
evidence exceeds the PR body budget". That refusal is thrown after
`report.json` is written in the clone, and the clone is then removed. So
the JSON had the error but no report: no cost, no lockfile. The refusal
is now a `ReportedError` (`PR_BODY_TOO_LARGE`) carrying the report, and
`fix --json` prints the report with `error` on top. Whether the run
should stop there at all is unchanged here (follow-up).

## Tests

- `managers/eresolve.test.ts` uses a stand-in npm: a small registry,
keeps locked versions, and fails with npm's real ERESOLVE text
(synthetic names, `fixtures/lockfiles/eresolve/npm-11.txt`) while a
conflict holds.
- **moved entries**: the retry starts without `toggle, ui, ui-dom` and
keeps `unrelated` locked; toggle re-resolves to 1.1.0.
- **whole lockfile**: used only when step 1 still conflicts; the fresh
resolve's churn lands in `parentChurn`, and `outOfScope` is empty.
- **all fail**: `INSTALL_BLOCKED` after exactly 3 attempts, with the
conflict lines.
- `cli.test.ts`: `fix --json` exit 2 prints JSON for a blocked install
(with `blocked`), for any other failure, and for a `ReportedError` (with
the report). The text output carries the same message.
- `format-fix.test.ts`: the summary names the retry step.
- `pnpm lint`, `pnpm typecheck`, `pnpm test`: green locally (core 2271,
cli 515). `pnpm public:check` with the denylist: clean.

## enki rerun (#169 harness, Uptide arm only, claude-sonnet-5-5, $1 cap)

Run `a8ff50ca`, from this branch's first commit (fd3037f).

| | |
|---|---|
| Exit | 2: `Required migration evidence exceeds the PR body budget`
(after install, agent and verification) |
| `--json` | printed `{ error }` (before the second commit it carried no
report) |
| Install step | **moved entries** (see the $0 replay below) |
| Lockfile churn | 0 out of scope; 85 parent churn; 5 moved subtrees |
| Clean install (#220) | `npm ci --ignore-scripts` passed in the replay;
the paid run's result was lost with its report |
| Checks | install/typecheck/lint/test 0, but on the parent commit: no
commits were delivered |
| Cost | unknown, at most the $1 cap. The agent estimated $0.54 for 9
sites; the harness read $0 because no report came back |
| Wall time | 193 s |
| Fork CI | none: no commits, so no fork pull request |

**$0 replay.** The same CLI and flags were run on a copy of the run's
clone, with `--keep-temp` and an invalid API key so every agent call
fails. It takes the same agent-led plan: react, react-dom 19.2.0,
@types/react, @types/react-dom, `@entur/react-component-toggle` 1.0.0,
react-leaflet 5.0.0, unsafe `@reach/*` overrides left alone. The
lock-only install hit ERESOLVE (`Found:
@entur/react-component-toggle@0.0.3` / `Conflicting peer dependency:
react@19.2.0`). Step 1 resolved it after dropping the entries of 82
react-peer packages; step 2 was not needed. It ended at exit 1 with a 23
KB PR body. In the paid run, the agent's edits pushed the required
(never-trimmed) evidence past GitHub's 65,536-character limit.

UPT-102 stays open: enki now installs, but its draft is refused at the
PR body.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@lucasbraum lucasbraum mentioned this pull request Oct 18, 2026
Merged
lucasbraum added a commit that referenced this pull request Oct 18, 2026
@lucasbraum @claude
fix: an oversized PR body is shortened, never a lost run (UPT-106) (#227
ae175a9 
)

Closes UPT-106. Closes UPT-102: enki now delivers an agent-led draft,
uptide-bench/entur--enki#5.

## Problem

`prBody()` threw "Required migration evidence exceeds the PR body
budget" when the rank-0 sections alone were over the budget. That threw
away the whole finished run. It happened on enki (a8ff50ca, after #224)
and on dms-saas (#221).

## Change

- **No section is exempt.** `fitPieces` returns `undefined` instead of
throwing. Required lists that can grow without bound get a compact form,
which is used only when the whole list does not fit. These lists still
go before anything droppable.
- Lockfile churn (outside the upgrade, and parent churn): counts per
kind of change (added, removed, version changed, re-resolved at the same
version), then the first 10 packages.
- Manual sites under "Worth a look": grouped by rule, with a count and
the first 5 locations each.
  - Remaining type errors: the first 10, then a count.
- **Full lists** stay in report.json. The Action's job summary now lists
every breaking site (it was capped at 50). Compact forms link to it
through a new `runUrl` field, set from
`GITHUB_SERVER_URL`/`GITHUB_REPOSITORY`/`GITHUB_RUN_ID` when the run is
in Actions.
- **Minimal body.** If the compact form still does not fit, the body is
a fixed-size summary: the verdict, the checks (types, tests, builds,
lint, clean install), the counts (sites by outcome, lockfile, files,
decisions), the branch and commit, and where the full report is. It
opens with a "Shortened description" notice.
- **The run is kept.** `fix()` and the isolated run's `finish()` write
report.json before the body, and nothing throws for size any more, so a
finished run always keeps its commits and report.json. `uptide pr` sends
the shortened body instead of refusing.
- **Decision lines** (second commit, found by the enki rerun). An agent
decision named every file in its task's scope. The verification-feedback
task was scoped to 317 files, so its 2 decisions came to 46 KB. A
decision now names the files the task changed, five at most, and counts
the rest.

## Fixtures

Written first; both failed on main with the throw.
- 300 manual sites (3 rules) and 100 + 100 churned lockfile entries. At
the 60 k budget every section is present: lockfile sections, risk table,
What changed, Worth a look, Decisions, Verification details, Run
details. Manual sites are compact and the lockfile lists are whole. At 8
k the lockfile lists are compact too: "100 entries: 25 added, 25
removed, 25 re-resolved at the same version, 25 version changed", with
the first 10 packages.
- The same report with an oversized decision renders the minimal body.
- A 317-file task with one changed file: the decision names only that
file. Without diffs it names 5 files and "312 more files".

## enki rerun (#169 harness)

Uptide arm only, claude-sonnet-5-5, default $1 cap, one run: 60949226.

| | |
|---|---|
| Exit | 0, draft delivered (branch `uptide/react-19.2.0`, verified
commit `3f05f25`) |
| PR body | 804 chars: minimal fallback (on 06bab33). With 16b2abe the
same stored report renders whole at 34,718 chars (77,681 before), no
shortening |
| Checks (harness) | install 0 · typecheck 0 · lint 0 · test 0 |
| #220 clean install | `npm ci --ignore-scripts --no-audit --no-fund`
passed |
| Sites | 3 by rule, 8 by the agent, 1 manual; verification PASS, 0 new
type errors |
| Cost / wall time | $0.17 (estimate $0.54) · 160 s |
| Fork CI | uptide-bench/entur--enki#5: npm ci, `npm run check` and
tests passed. The job fails at SonarQube Scan (no `SONAR_TOKEN` in the
fork), so build is skipped. Fork PRs #1–#4 fail the same way |

The run used the build of 06bab33. The fork PR's body is therefore the
minimal form; the 34.7 KB figure is a $0 re-render of the stored
report.json with 16b2abe. The report records `uptideDirty` because the
harness files under `bench/` were copied in untracked, as in earlier
reruns.

## Left as is

- Lockfile churn is not in the job summary. The run-report schema
(`uptide.run/v1`) has no lockfile field, so the full churn list is only
in report.json.
- `ReportedError` and `PR_BODY_TOO_LARGE` stay. Nothing throws
`PR_BODY_TOO_LARGE` any more, but `fix --json` still uses
`ReportedError` for reporting any failure after report.json is written.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@lucasbraum lucasbraum mentioned this pull request Oct 18, 2026
Merged
7 tasks done
lucasbraum added a commit that referenced this pull request Oct 18, 2026
@lucasbraum @claude
perf(fix): cut install copies and the agent's test check, count comma…
e4a2885 
…nds per phase (UPT-103) (#232)

Closes UPT-103.

## What and why

After UPT-100/101/104/106, Uptide's cost is at or below Claude Code's on
the pilot repositories (#169). Wall time is the gap.
This PR measures where the time goes and makes two cuts. Neither one
changes the gate after the agent's edits or the final verification.

**1. Per-phase commands in the fix report** (`d0a77c3`). Each `phases[]`
entry now has `runs`: the repository's commands the phase ran
(`install`, `generate`, `build`, `types`, `tests`, `lint`, `format`),
each with a count and its summed wall time. An agent-led run's clean
install (#220) gets its own `clean-install` phase, so the phases add up
to the run. The tables below come from this field.

**What the baseline showed.** These are the costs I could act on, across
the three repos:

- **node_modules copies around the install.** `isolatedUpgrade` installs
in a temporary worktree. It then `cpSync`s the new `node_modules` into
the clone, file by file and not even copy-on-write, and deletes the
worktree. In a $0 replay with a CPU profile (invalid API key, no model
spend), that copy was 7.5 s on enki and 5.1 s on dms-saas. npm itself
took only about 10.5 s of enki's 31 s install.
- **Test runs in the agent phase on dms-saas: 2 runs, 42.6 s.** It is
the largest line item there apart from the model. The agent's
`run_tests` asks for the task's related tests. But dms-saas's `test`
script is `pnpm run build && … && pnpm run test:unit`, so related
scoping found no bare runner and ran the whole script, build included.

The model's own time is the largest item everywhere (about 31 s, 71 s
and 50 s), but I found no deterministic cut for it (see "What did not
work").

**2. The agent's own test check runs its runner's related tests**
(`26574a9`). This is a new `agentCheck` test option, used only by the
agent's own `run_tests`. When a `test` script ends by running a script
that is just the runner, the check runs that runner's related tests.
Everything else is unchanged:
- The gate after the agent's edits still runs its own `related` plan
(the whole script on dms-saas).
- The final verification runs the full suite.
- A check result never stands in for a gate result: separate cache
entries.
- Where the narrowing changes no command (calmcp, enki: the script
already is the runner), the check stays the gate's own run, and the gate
reuses its result on the same tree, as before.

**3. The installed tree is moved, not copied** (`049023c`).
`isolatedUpgrade` renames the new `node_modules` out of the worktree,
which is removed anyway. The tree is the same, links into the worktree
are relocated as before, and it falls back to a copy across filesystems
(`EXDEV`).

## Results

The #169 harness (`bench/run.sh`), Uptide arm only, `claude-sonnet-5-5`
at medium effort, default $1 limit, once per repository. Before is
`origin/main` plus the counting commit `d0a77c3` (no behavior change).
After is `049023c`; the last commit only adds a test. `bench/` is
untracked in both checkouts, which is why both reports say
`uptideDirty`. Claude Code's numbers are from the #169 rerun (same
harness and model).

| repo | Claude Code (#169) | Uptide before | Uptide after |
|---|---|---|---|
| calmcp, zod 3.25.76→4.6.5 | $0.11, 47 s | $0.094, 60 s | $0.085, **53
s** |
| dms-saas, stripe 18.5.0→22.6.2 | $0.31, 149 s | $0.265, 224 s |
$0.279, **210 s** |
| enki, react 18.3.1→19.2.0 | $0.11, 67 s (its diff fails `npm ci`) |
$0.176, 147 s | $0.161, **136 s** |

**Same checks, same verdicts, same diff shape.** On every repository,
before and after match on all of these:
- `verification.passed` (true)
- the site outcomes: calmcp 1 mechanical + 5 agent, dms-saas 2 + 10,
enki 3 + 8 + 1 manual
- the changed file list: 4, 11 and 12 files, identical
- test, lint and build statuses, and no new type errors
- feedback rounds: 0, 0 and 1
- enki's clean install, which passed

One run each: the model's own time moves by ±5 s between runs, which is
the size of calmcp's whole change.

<details><summary>calmcp phases</summary>

| line item | before | after |
|---|---:|---:|
| **wall (harness)** | **60 s** | **53 s** |
| cost | $0.094 | $0.085 |
| outside `fix()`: preflight, clone with node_modules copy, delivery,
clone removal | 3 s | 3 s |
| analyze (check; workspace prep builds) | 5.7 s | 5.5 s |
| baseline phase | 3.3 s | 3.2 s |
| ↳ baseline type-check | 1× 0.8 s | 1× 0.8 s |
| ↳ baseline tests | 1× 1.8 s | 1× 1.8 s |
| install phase (bump install, consumed rebuild, target types) | 6.1 s |
4 s |
| ↳ upgrade install | 1× 4.8 s | 1× 2.8 s |
| agent phase | 38.2 s ($0.094) | 33.5 s ($0.085) |
| ↳ checks inside it (types, lint, format) | 0.9 s | 1 s |
| ↳ tests inside it (agent check + gate) | 2× 1.8 s | 2× 1.9 s |
| ↳ model and tool turns (the rest) | 35.5 s | 30.6 s |
| feedback rounds | 0: – | 0: – |
| final verify (build, format, full tests, types, lint) | 2.6 s | 2.7 s
|
| clean install (agent-led, #220) | – | – |
| installs, all phases | 1× 4.8 s | 1× 2.8 s |
| tests, all phases | 4× 5.1 s | 4× 5.3 s |
| type-checks, all phases | 8× 2.7 s | 8× 2.7 s |
| builds (`buildRuns`) | 2× 0.7 s | 2× 0.6 s |
| generators | 2× 0 s | 2× 0 s |
| publish | not run (no `--pr` in the harness) | not run |
| agent turns | 19 | 17 |

</details>

<details><summary>dms-saas phases</summary>

| line item | before | after |
|---|---:|---:|
| **wall (harness)** | **224 s** | **210 s** |
| cost | $0.265 | $0.279 |
| outside `fix()`: preflight, clone with node_modules copy, delivery,
clone removal | 9 s | 10 s |
| analyze (check; workspace prep builds) | 23 s | 22.7 s |
| baseline phase | 25.8 s | 25.5 s |
| ↳ baseline type-check | 1× 3.6 s | 1× 3.7 s |
| ↳ baseline tests | 1× 21.6 s | 1× 21.5 s |
| install phase (bump install, consumed rebuild, target types) | 22 s |
14.7 s |
| ↳ upgrade install | 1× 12.4 s | 1× 5.3 s |
| agent phase | 112.4 s ($0.265) | 107.3 s ($0.279) |
| ↳ checks inside it (types, lint, format) | 1.5 s | 1.3 s |
| ↳ tests inside it (agent check + gate) | 2× 42.6 s | 2× 35.3 s |
| ↳ model and tool turns (the rest) | 68.4 s | 70.7 s |
| feedback rounds | 0: – | 0: – |
| final verify (build, format, full tests, types, lint) | 31.1 s | 29.1
s |
| clean install (agent-led, #220) | – | – |
| installs, all phases | 1× 12.4 s | 1× 5.3 s |
| tests, all phases | 4× 85.6 s | 4× 76.6 s |
| type-checks, all phases | 8× 11.4 s | 8× 11.2 s |
| builds (`buildRuns`) | 3× 13.5 s | 3× 13.1 s |
| generators | 2× 0 s | 2× 0 s |
| publish | not run (no `--pr` in the harness) | not run |
| agent turns | 34 | 34 |

</details>

<details><summary>enki phases</summary>

| line item | before | after |
|---|---:|---:|
| **wall (harness)** | **147 s** | **136 s** |
| cost | $0.176 | $0.161 |
| outside `fix()`: preflight, clone with node_modules copy, delivery,
clone removal | 11 s | 11 s |
| analyze (check; workspace prep builds) | 12.8 s | 12.7 s |
| baseline phase | 7 s | 7 s |
| ↳ baseline type-check | 1× 2.8 s | 1× 2.6 s |
| ↳ baseline tests | 1× 3.9 s | 1× 4.1 s |
| install phase (bump install, consumed rebuild, target types) | 32.9 s
| 23 s |
| ↳ upgrade install | 1× 30.9 s | 1× 21.1 s |
| agent phase | 26.6 s ($0.085) | 27.4 s ($0.085) |
| ↳ checks inside it (types, lint, format) | 1.1 s | 1 s |
| ↳ tests inside it (agent check + gate) | 1× 1.1 s | 1× 1.2 s |
| ↳ model and tool turns (the rest) | 24.4 s | 25.2 s |
| feedback rounds | 1: 44 s ($0.092) | 1: 40.8 s ($0.076) |
| final verify (build, format, full tests, types, lint) | 5.7 s | 5.8 s
|
| clean install (agent-led, #220) | 6 s | 7.7 s |
| installs, all phases | 2× 36.9 s | 2× 28.7 s |
| tests, all phases | 5× 15.5 s | 5× 16.3 s |
| type-checks, all phases | 11× 15.3 s | 11× 15 s |
| builds (`buildRuns`) | 0× 0 s | 0× 0 s |
| generators | 2× 0 s | 2× 0 s |
| publish | not run (no `--pr` in the harness) | not run |
| agent turns | 17 + 19 | 17 + 13 |

</details>

Where the cuts show up:
- **upgrade install:** calmcp 4.8 → 2.8 s, dms-saas 12.4 → 5.3 s, enki
30.9 → 21.1 s.
- **agent-phase tests on dms-saas:** 42.6 → 35.3 s. The narrowed check
took about 14 s instead of 21 s.

Tests on calmcp and enki: same counts and times as before.

## What is left versus Claude Code

- **calmcp, 53 s vs 47 s.** About 31 s is the model: 17 turns, one tool
call each. The other ~19 s is work Claude Code does not do in this form:
analysis 5.5 s, install 4 s, baseline 3.2 s, final verify 2.7 s, clone
and delivery 3 s.
- **dms-saas, 210 s vs 149 s.** Tests are the gap: four runs of the
`test` script family take 76.6 s (baseline 21.5, agent check ≈14, gate
≈21, final ≈20).
- Every full run builds before running vitest. vitest alone is about 17
s.
- Related tests barely help here: every route imports the Stripe client,
so "related" is most of the suite.
- The gate and the final run are the verification this PR must not trade
away.
- The rest: analysis 22.7 s (the stripe 18→22 type diff and call-site
compile) and the model, about 71 s for 34 turns.
- **enki, 136 s vs 67 s.** Claude Code's diff does not install (`npm ci`
ERESOLVE), so it skipped what costs Uptide here: the upgrade install (21
s), the clean install (7.7 s) and the agent-led feedback round (41 s).
The round is forced by the peer-blocker and root-pin fallback.

## What did not work (reverted, not in this PR)

An intermediate build also (a) asked the agent to batch reads and edits
as parallel tool calls in one message, and (b) narrowed the agent's
check in feedback rounds. It ran on all three repos ("after (reverted)"
runs 8e0f068f, d48a70c0, 72b2bc57):
- (a) changed nothing: dms-saas made the same 34 turns with the same
tool counts.
- (b) made enki slower, 147 → 163 s. `vitest related` over the feedback
task's files cost more than the full suite, and splitting the check from
the gate lost the gate's cache hit.

Both are reverted. The final change keeps the check on the gate's run
wherever narrowing changes nothing.

## How it was verified

Fixture tests:
- `generic.test.ts`, "narrows only the agent's own check, …" (3 cases) —
**the final verify still runs the full gate**. It covers a chained
script, a plain runner, and an agent-led run. It records every test call
and its options and checks:
- the order is baseline full, agent check, gate (as before), final full
suite;
  - the final call is `fullSuite` without `related`;
  - the verify phase built, type-checked and ran tests once each;
  - the agent-led run ran its clean install (#220) and it passed.
- `fix.test.ts`, "scopes only the agent's own check to the runner a test
script chain ends with": `agentCheck` gives `vitest related …`, while
`related` alone and `fullSuite` still give the whole `pnpm run build &&
pnpm run test:unit`.
- `sdk-agent.test.ts`: the agent's `run_tests` uses `checkTests`,
repeats reuse its own result, and the gate on the same tree still runs
`tests`.
- `isolated-install.test.ts`: the clone's `node_modules` holds the same
inode the install wrote (a move, not a copy). Relative links are kept
and absolute links into the worktree are relocated.
- `generic.test.ts` phases test: the baseline phase's `runs` counts its
type-check and test run.

`pnpm lint && pnpm typecheck && pnpm test` pass (core 2,292 tests, cli
516). The bench runs needed no fork PRs: nothing was pushed anywhere but
this branch.

## Checklist

- [x] Tests for the change
- [x] `pnpm lint && pnpm typecheck && pnpm test` pass
- [x] README and docs updated if behavior changed (no CLI flag or
documented output changed; the report field is described in `types.ts`)
- [x] Changeset added (`agent-phase-wall-time.md`)
- [x] No new dependency
- [x] No private code, names or paths in fixtures, tests or docs
- [x] The rules in CONTRIBUTING.md ("Rules that do not bend") still hold

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
lucasbraum pushed a commit that referenced this pull request Oct 18, 2026
@uptide-release
Version Packages (#217)
41a2e75 
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.


# Releases
## uptide@0.15.1

### Patch Changes

- [#220](#220)
[`cf86d65`](cf86d65)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - With an assisting
agent, a root pin Uptide cannot update safely (a nested override such as
`overrides > pkg > peerDependencies > react`) no longer stops `fix` with
exit 2: the run continues as an agent-led draft that leaves the pin as
written and names it, with its exact path and value, at the top of the
PR body. The agent-led peer reason now names each blocker's first
release that accepts the target, or says none does. An agent-led draft
must pass the repository's clean install (`npm ci` or the manager's
frozen install) before delivery; if it does not, the PR says so at the
top. Without an agent, with `--no-llm`, or on a safety refusal, `fix`
still exits 2.

- [#232](#232)
[`e4a2885`](e4a2885)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide fix`
spends less wall time on test runs and dependency copies:
  
- The agent's own `run_tests` check runs only its runner's related tests
when the `test` script builds first and then runs a script that is just
the runner (`pnpm run build && pnpm run test:unit`). The gate after the
agent's edits and the final verification run exactly what they ran
before.
- The upgrade's install moves the installed `node_modules` out of its
temporary worktree instead of copying it.
- Each phase of the fix report lists the repository's commands it ran
(installs, generators, builds, type-checks, tests, lint, format) with
their count and summed wall time. An agent-led run's clean install has
its own phase.

- [#223](#223)
[`2dc2162`](2dc2162)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `fix` no longer
stops with "baseline tests modified the working tree" when the test run
only writes new untracked files, such as a junit `test-report.xml` that
is not gitignored. Those files are recorded as test artifacts
(`testArtifacts` in the report), kept out of every commit, the agent's
diff and the pending-changes check, and removed before each later test
run, `uptide verify` included. A change to a tracked file during
baseline tests still refuses the run.

- [#215](#215)
[`abddcb3`](abddcb3)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check` no longer
lists possible impact (type changes nothing confirmed as breaking) on
its first screen: each package says it in one line, `+N possible impact
in M files, not counted as breaking`, and `--details` and `--json` still
have every site. A package whose types were partly verified no longer
repeats its coverage line in the heading, and a monorepo's unbuilt
workspace chains are one count on that line.

- [#213](#213)
[`2de3a18`](2de3a18)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - A companion pack
that moves with the lead's fix (react beside next) now gets its own scan
context and inventory, and the PR body carries a section with its review
notes.

- [#212](#212)
[`733cfbe`](733cfbe)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `pack test
--write` records how many breaking sites a finding landed on at any
severity and as breaking, out of all breaking sites it scored, and the
pack tables show both ("Found, any severity", "Found as breaking") in
place of the single recall column. Packs recorded before it show `—`.

- [#224](#224)
[`09f68df`](09f68df)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `fix` on npm no
longer stops when the lock-only install hits ERESOLVE from the old
lockfile. It retries without the lockfile entries of the moved packages,
their companions and the packages whose peers they satisfy, and only
then with the whole lockfile re-resolved; the report says which step
worked, and the lockfile check reports the churn as usual. It never uses
`--legacy-peer-deps` or `--force`. When every step fails, `fix` ends
blocked, naming the conflicting packages from npm's ERESOLVE report.
`fix --json` now prints JSON on every exit, a failure included; when the
run already has its report (a PR body over GitHub's size limit), that
report comes with the error.

- [#221](#221)
[`ff8de8e`](ff8de8e)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `fix` no longer
rejects an agent patch for test files the parent commit already failed,
whichever test scope ran them: when the test script builds first and the
build failed on the error being fixed, the start had no test results to
compare with, so a correct patch was reverted every round. The patch's
tests also run with the pack's follow-ups in place (a test asserting the
old Stripe API version), which `fix` still applies and reports after the
agent. When a PR body cannot fit, the stored report.json the error
points to is now written first. `check` and `fix` now also analyze files
that reach a package's wrapper through a barrel (`import {
getStripeClient } from '../stripe'` with `export * from './client'`).
Edited files are formatted with oxfmt and dprint as well as biome and
prettier, from the workspace that holds them, with its own
configuration, else the root's. The stripe pack adds `request-options`:
request options passed in the params slot (`subscriptions.cancel(id, {
idempotencyKey })`), which SDK 22 rejects, move to the third argument.
It runs assisted until a ground-truth repository exercises it.

- [#226](#226)
[`448c3cb`](448c3cb)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide fix`
judges consumed workspace builds on the final head, not on the tree
right after the bump. A build that failed after the bump and passes once
the edits land no longer starts a feedback round, adds an "Agent-led
draft" reason or shows "Build failed" in the PR body. The builds run
again only when the tree changed since their last result, and not at all
when the verification build already built those workspaces. The fix
report's `buildRuns` gives the number of builds the run executed and
their wall time.

- [#218](#218)
[`13e343b`](13e343b)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check` folds a
compile error into the breaking site it comes from when the compiler
traces its value there (an `errors.map(e => e.message)` under a
`result.error.errors` that zod 4 no longer types, and the narrowing that
depended on it). The rule's line shows `+N downstream errors`,
`--details` lists them under their site, and `--json` carries them as
that finding's `downstream`. They are no longer separate breaking sites.
`fix` still verifies against every error.

- [#214](#214)
[`195ae10`](195ae10)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - A rewrite rule
below 80% precision over at least 5 scored ground-truth sites now runs
as assisted in `fix` (react-router `dom-imports`); smaller samples (next
`middleware-proxy`) stay mechanical and are flagged in the compatibility
table. Pack status is unchanged.

- [#218](#218)
[`13e343b`](13e343b)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - pack test runs a
ground-truth repository's code generators (prisma generate, local
graphql-codegen) after its install, as fix does. It fetches the Prisma
engines the locked version needs itself, so generators stay off the
network. A repository whose generator still fails is incomplete and
fails the pack, unless its ground truth declares `withoutGenerated`

- [#227](#227)
[`ae175a9`](ae175a9)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - A migration whose
evidence is larger than GitHub's PR description limit no longer stops
`fix` with "Required migration evidence exceeds the PR body budget" and
loses the run. Long required lists get a compact form when they do not
fit. Lockfile churn shows counts per kind of change and the first
packages. Manual sites are grouped by rule, with counts and the first
locations. Remaining type errors show the first entries. Every full list
stays in report.json and, in the Uptide Action, in the job summary,
which the description links to. If even the compact form does not fit,
the description is a short summary that says so at the top: the verdict,
the checks, the counts and where the full report is. The branch and
report.json are always kept.
  
An agent's decisions under "Decisions for you" name the files the agent
changed, five at most, and count the rest, instead of every file in the
task's scope. A verification-feedback task scoped to the whole
repository made each of its decisions a list of hundreds of files.

- [#219](#219)
[`12dc612`](12dc612)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `fix zod`
rewrites `z.string().uuid()` to `z.guid()` by rule without
`--include-deprecated`: zod 4's `uuid()` also checks the RFC version and
variant, so ids zod 3 accepted stop validating, and `check` now counts
the site as breaking. `fix` no longer reports false pre-existing type
errors on a TypeScript 7 repository whose `lib` names a library newer
than its `target` (`lib: ["ES2023"]` under `ES2022`).

Co-authored-by: uptide-release[bot] <338861287+uptide-release[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@lucasbraum