Skip to content

fix: untracked files written by baseline tests are test artifacts (UPT-105) - #223

Merged
lucasbraum merged 1 commit into
mainfrom
lucasbraum/upt-105-baseline-test-artifacts
Oct 18, 2026
Merged

lucasbraum merged 1 commit into
mainfrom
lucasbraum/upt-105-baseline-test-artifacts

Conversation

@lucasbraum

Copy link
Copy Markdown
Collaborator

Closes UPT-105. Refs UPT-102 (not closed: see the enki rerun below).

What and why

On enki (UpgradeBench, #169), the vitest junit reporter writes test-report.xml, which the repository does not gitignore. The baseline-tests guard treated any untracked file as a change, so fix stopped with "baseline tests modified the working tree" before doing anything.

Now:

  • Untracked files are snapshotted before the baseline tests (untrackedFiles). After the run, a change to a tracked file still refuses, as before. New untracked files are test artifacts (testArtifacts in verify.ts).
  • Artifacts go in the private clone's .git/info/exclude, anchored at the repo top (with the project's subdirectory prefix, and pattern characters escaped). No status, diff, git add or ls-files --exclude-standard sees them. That keeps them out of commits, the agent's diff (including tests the agent runs itself), the follow-up and style steps, and the verificationPending check, from one place.
  • They are removed right away, and again before each later test run: clearing() wraps services.tests once, after baseline, so settle, the rules' transactions, the agent gate and the feedback rounds all go through it.
  • The report records them (testArtifacts). uptide verify runs in a fresh clone, so it re-applies the exclude and clearing from the stored run; otherwise its final test run would leave the file and fail.
  • The same applies to the Stripe pin path (pin.ts).

fix and verify always run in a private clone (isolatedFix/isolatedVerify), so writing info/exclude never touches the user's checkout. The comment there names the ceiling if either ever runs in place.

Tests

  • fix.test.ts: the fixture's test script writes an untracked test-report.xml and fails if the previous run's file is still there. Baseline and final verification pass, testArtifacts is ['test-report.xml'], the file is in no commit and not in changedFiles, nothing is pending. uptide verify from a fresh clone (no exclude entry, no file on disk) also passes.
  • fix.test.ts: a baseline test run that writes an untracked report and changes a tracked file still refuses.
  • verify.test.ts: a project in a subdirectory, with names containing [ ] # ! space, is hidden and nothing else is.
  • Reverting the run change brings back the original refusal. Reverting only the reverify change fails the verify assertion.
  • pnpm lint, pnpm typecheck, pnpm test: green locally (core 2268 passed).

enki rerun (#169 harness, Uptide arm only, claude-sonnet-5-5, $1 cap)

Run 7f90b41c, from this branch's build (8d1d8bb).

Baseline tests passed the guard (this fix)
Stopped at lockfile install of react 19.2.0 + @entur/react-component-toggle 1.0.0: npm install --package-lock-only → ERESOLVE, exit 2
Clean-install check (#220) not reached: no draft
Checks none on a draft: head is the parent commit
Cost $0.00 (no agent call)
Wall time 54 s
Fork CI none: no commits, so no fork pull request

The cause, reproduced outside Uptide: starting from the parent's package-lock.json, npm install --package-lock-only fails with the same ERESOLVE even with the human upgrade's exact package.json (ed37dd2). Without the old lockfile, the same manifest resolves (exit 0). So Uptide's companion set matches the human's. What blocks is npm's resolution starting from the old lockfile, not the plan. Two defects show up here: the install step has no fallback for this, and fix --json printed nothing on this exit 2. The run ended as a raw npm error instead of a "blocked by" result.

UPT-102's done-criterion (a draft that installs, or a clear "blocked by") is not met yet, so this PR does not close it.

🤖 Generated with Claude Code

…T-105)

A test run that writes a report nobody gitignores (enki's vitest junit
reporter writes test-report.xml) stopped fix with "baseline tests
modified the working tree". Untracked files are now snapshotted before
the baseline tests; new ones are recorded as testArtifacts, hidden from
git through the private clone's info/exclude (so no commit, agent diff,
follow-up, style step or verificationPending check sees them) and
removed before each later test run, uptide verify included. A change to
a tracked file during baseline tests still refuses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
@lucasbraum
lucasbraum merged commit 2dc2162 into main Oct 18, 2026
12 checks passed
@lucasbraum
lucasbraum deleted the lucasbraum/upt-105-baseline-test-artifacts branch October 18, 2026 01:39
@uptide-release uptide-release Bot mentioned this pull request Oct 18, 2026
lucasbraum pushed a commit that referenced this pull request Oct 18, 2026
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.


# Releases
## uptide@0.15.1

### Patch Changes

- [#220](#220)
[`cf86d65`](cf86d65)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - With an assisting
agent, a root pin Uptide cannot update safely (a nested override such as
`overrides > pkg > peerDependencies > react`) no longer stops `fix` with
exit 2: the run continues as an agent-led draft that leaves the pin as
written and names it, with its exact path and value, at the top of the
PR body. The agent-led peer reason now names each blocker's first
release that accepts the target, or says none does. An agent-led draft
must pass the repository's clean install (`npm ci` or the manager's
frozen install) before delivery; if it does not, the PR says so at the
top. Without an agent, with `--no-llm`, or on a safety refusal, `fix`
still exits 2.

- [#232](#232)
[`e4a2885`](e4a2885)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide fix`
spends less wall time on test runs and dependency copies:
  
- The agent's own `run_tests` check runs only its runner's related tests
when the `test` script builds first and then runs a script that is just
the runner (`pnpm run build && pnpm run test:unit`). The gate after the
agent's edits and the final verification run exactly what they ran
before.
- The upgrade's install moves the installed `node_modules` out of its
temporary worktree instead of copying it.
- Each phase of the fix report lists the repository's commands it ran
(installs, generators, builds, type-checks, tests, lint, format) with
their count and summed wall time. An agent-led run's clean install has
its own phase.

- [#223](#223)
[`2dc2162`](2dc2162)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `fix` no longer
stops with "baseline tests modified the working tree" when the test run
only writes new untracked files, such as a junit `test-report.xml` that
is not gitignored. Those files are recorded as test artifacts
(`testArtifacts` in the report), kept out of every commit, the agent's
diff and the pending-changes check, and removed before each later test
run, `uptide verify` included. A change to a tracked file during
baseline tests still refuses the run.

- [#215](#215)
[`abddcb3`](abddcb3)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check` no longer
lists possible impact (type changes nothing confirmed as breaking) on
its first screen: each package says it in one line, `+N possible impact
in M files, not counted as breaking`, and `--details` and `--json` still
have every site. A package whose types were partly verified no longer
repeats its coverage line in the heading, and a monorepo's unbuilt
workspace chains are one count on that line.

- [#213](#213)
[`2de3a18`](2de3a18)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - A companion pack
that moves with the lead's fix (react beside next) now gets its own scan
context and inventory, and the PR body carries a section with its review
notes.

- [#212](#212)
[`733cfbe`](733cfbe)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `pack test
--write` records how many breaking sites a finding landed on at any
severity and as breaking, out of all breaking sites it scored, and the
pack tables show both ("Found, any severity", "Found as breaking") in
place of the single recall column. Packs recorded before it show `—`.

- [#224](#224)
[`09f68df`](09f68df)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `fix` on npm no
longer stops when the lock-only install hits ERESOLVE from the old
lockfile. It retries without the lockfile entries of the moved packages,
their companions and the packages whose peers they satisfy, and only
then with the whole lockfile re-resolved; the report says which step
worked, and the lockfile check reports the churn as usual. It never uses
`--legacy-peer-deps` or `--force`. When every step fails, `fix` ends
blocked, naming the conflicting packages from npm's ERESOLVE report.
`fix --json` now prints JSON on every exit, a failure included; when the
run already has its report (a PR body over GitHub's size limit), that
report comes with the error.

- [#221](#221)
[`ff8de8e`](ff8de8e)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `fix` no longer
rejects an agent patch for test files the parent commit already failed,
whichever test scope ran them: when the test script builds first and the
build failed on the error being fixed, the start had no test results to
compare with, so a correct patch was reverted every round. The patch's
tests also run with the pack's follow-ups in place (a test asserting the
old Stripe API version), which `fix` still applies and reports after the
agent. When a PR body cannot fit, the stored report.json the error
points to is now written first. `check` and `fix` now also analyze files
that reach a package's wrapper through a barrel (`import {
getStripeClient } from '../stripe'` with `export * from './client'`).
Edited files are formatted with oxfmt and dprint as well as biome and
prettier, from the workspace that holds them, with its own
configuration, else the root's. The stripe pack adds `request-options`:
request options passed in the params slot (`subscriptions.cancel(id, {
idempotencyKey })`), which SDK 22 rejects, move to the third argument.
It runs assisted until a ground-truth repository exercises it.

- [#226](#226)
[`448c3cb`](448c3cb)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide fix`
judges consumed workspace builds on the final head, not on the tree
right after the bump. A build that failed after the bump and passes once
the edits land no longer starts a feedback round, adds an "Agent-led
draft" reason or shows "Build failed" in the PR body. The builds run
again only when the tree changed since their last result, and not at all
when the verification build already built those workspaces. The fix
report's `buildRuns` gives the number of builds the run executed and
their wall time.

- [#218](#218)
[`13e343b`](13e343b)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check` folds a
compile error into the breaking site it comes from when the compiler
traces its value there (an `errors.map(e => e.message)` under a
`result.error.errors` that zod 4 no longer types, and the narrowing that
depended on it). The rule's line shows `+N downstream errors`,
`--details` lists them under their site, and `--json` carries them as
that finding's `downstream`. They are no longer separate breaking sites.
`fix` still verifies against every error.

- [#214](#214)
[`195ae10`](195ae10)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - A rewrite rule
below 80% precision over at least 5 scored ground-truth sites now runs
as assisted in `fix` (react-router `dom-imports`); smaller samples (next
`middleware-proxy`) stay mechanical and are flagged in the compatibility
table. Pack status is unchanged.

- [#218](#218)
[`13e343b`](13e343b)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - pack test runs a
ground-truth repository's code generators (prisma generate, local
graphql-codegen) after its install, as fix does. It fetches the Prisma
engines the locked version needs itself, so generators stay off the
network. A repository whose generator still fails is incomplete and
fails the pack, unless its ground truth declares `withoutGenerated`

- [#227](#227)
[`ae175a9`](ae175a9)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - A migration whose
evidence is larger than GitHub's PR description limit no longer stops
`fix` with "Required migration evidence exceeds the PR body budget" and
loses the run. Long required lists get a compact form when they do not
fit. Lockfile churn shows counts per kind of change and the first
packages. Manual sites are grouped by rule, with counts and the first
locations. Remaining type errors show the first entries. Every full list
stays in report.json and, in the Uptide Action, in the job summary,
which the description links to. If even the compact form does not fit,
the description is a short summary that says so at the top: the verdict,
the checks, the counts and where the full report is. The branch and
report.json are always kept.
  
An agent's decisions under "Decisions for you" name the files the agent
changed, five at most, and count the rest, instead of every file in the
task's scope. A verification-feedback task scoped to the whole
repository made each of its decisions a list of hundreds of files.

- [#219](#219)
[`12dc612`](12dc612)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `fix zod`
rewrites `z.string().uuid()` to `z.guid()` by rule without
`--include-deprecated`: zod 4's `uuid()` also checks the RFC version and
variant, so ids zod 3 accepted stop validating, and `check` now counts
the site as breaking. `fix` no longer reports false pre-existing type
errors on a TypeScript 7 repository whose `lib` names a library newer
than its `target` (`lib: ["ES2023"]` under `ES2022`).

Co-authored-by: uptide-release[bot] <338861287+uptide-release[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant