Skip to content

fix(packs): score expected sites at their rule's or note's severity (UPT-52) - #143

Merged
lucasbraum merged 3 commits into
mainfrom
claude/amazing-albattani-8m8r4h
Oct 15, 2026
Merged

fix(packs): score expected sites at their rule's or note's severity (UPT-52)#143
lucasbraum merged 3 commits into
mainfrom
claude/amazing-albattani-8m8r4h

Conversation

@lucasbraum

@lucasbraum lucasbraum commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator

Fixes UPT-52.

What and why

pack test counted every missed expected site as a breaking false negative unless a rule with that id was deprecated. BehaviorNote.severity was never read, so a missed behavior-note site that only needs a look counted as a missed break. Found by the ioredis pack (#129): "Breaking: 0 predicted, 0 FP, 6 FN", where four of the six misses are client-defaults, a note declared severity: 'deprecated'.

This PR:

  • Scores each expected site at the severity of the rule or note it belongs to (siteSeverity): a rule's severity; a note's severity, or breaking when the note reports findings without one (what check reports for it); behavior for a note check never reports as a finding (decision, test-follow-up); breaking for generic. Breaking recall counts only misses whose severity is breaking.
  • Reports behavior-note precision/recall on its own: a behavior notes (sites) row in pack test, behavior in --json, and behavior in verification.json.
  • Records the counts behind each rate (truePositives, falseNegatives) in verification.json, so a zero denominator prints n/a instead of 100% in pack test and in the README packs table (rateLabel). Backward compatible: both new fields are optional, a record without them prints as before, and the runtime gate (recordedStatus) reads nothing new.
  • False negatives print their severity, as false positives already did.
  • Scoring moves into a pure scoreTruth(pack, repos), with no change in behavior, so it can be tested without fetching repositories.
  • CONTRIBUTING gets "Run pack test locally": corepack for Yarn only, COREPACK_NPM_REGISTRY for Yarn 4, per-pack cache dirs and eviction.

check and fix are untouched: no file under check/, fix/ or the adapters changes.

One decision to review: notes declared breaking stay breaking

A note's own severity decides, not the fact that it is a note. Two reasons:

  • vitest (verified) has 13 truth sites under coverage-patterns, a note declared severity: 'breaking' with detect. check shows those findings as breaking, so their false positives must keep counting against the verified gate. Treating every note site as non-breaking would hide them and change vitest's verified numbers.
  • So ioredis on feat(packs): add a candidate ioredis 5 to 6 pack #129 goes from 6 breaking FN to 2, not 0: the two left are handshake-auth sites (log.test.ts:20, :28), and that note declares severity: 'breaking'. If the note should only need a look, that is a one-word change in the pack on feat(packs): add a candidate ioredis 5 to 6 pack #129, not a scorer change.

How it was verified

Failing test first. packages/core/src/packs/tooling/behavior-score.test.ts uses synthetic sites modeled on the ioredis ground truth (voiceai, open-foundry, dokunc, plus the FishingConditionsApp control). On main's scorer it reproduces the bug exactly:

FAIL  behavior-note sites > count as breaking misses only when their note is declared breaking
-   "falseNegatives": 2,
+   "falseNegatives": 6,

All six of its tests failed before the change and pass after. packages/cli/src/pack.test.ts covers the new row, n/a, and the false-negative severity.

End to end on #129 (local only, nothing pushed). #129's head, scored before and after cherry-picking this commit, on the same five public repositories:

ioredis Breaking P / R (TP/FP/FN) Behavior notes P / R (TP/FP/FN)
#129 as is 100%* / 0% (0/0/6) not reported
#129 + this PR n/a / 0% (0/0/2) 30% / 33% (3/7/6)

* printed as 100% before this PR, with 0 predicted.

pack test ioredis with this PR 
  rule                         precision  recall    tp   fp   fn
  client-defaults                    30%     43%     3    7    4
  handshake-auth                     n/a      0%     0    0    2
  overall (sites)                    30%     33%     3    7    6
  behavior notes (sites)             30%     33%     3    7    6
  breaking (sites)                   n/a      0%     0    0    2

false positives (7)
  ferni-ai/voiceai@c59ed0982f96  src/memory/storage/redis-cache/redis-cache.ts:101  client-defaults (deprecated)
  ferni-ai/voiceai@c59ed0982f96  src/services/data/redis-pubsub.ts:140  client-defaults (deprecated)
  ferni-ai/voiceai@c59ed0982f96  src/services/platform/rate-limiter.ts:57  client-defaults (deprecated)
  ferni-ai/voiceai@c59ed0982f96  src/services/workflows/jobs/redis-job-queue.ts:92  client-defaults (deprecated)
  syzygyhack/open-foundry@73910018b5e8  packages/api/src/server.ts:138  client-defaults (deprecated)
  simondietschi/dokunc@83681ed0ccc5  apps/web/src/lib/redis.ts:54  client-defaults (deprecated)
  kazumadev619-dev/FishingConditionsApp@8ba3f0d24bc7  src/lib/cache/client.ts:13  client-defaults (deprecated)

false negatives (6)
  ferni-ai/voiceai@c59ed0982f96  src/memory/storage/redis-cache/redis-cache.ts:98  client-defaults (deprecated)
  ferni-ai/voiceai@c59ed0982f96  src/services/workflows/jobs/redis-job-queue.ts:89  client-defaults (deprecated)
  ferni-ai/voiceai@c59ed0982f96  src/memory/vectors/embedding-cache.ts:134  client-defaults (deprecated)
  syzygyhack/open-foundry@73910018b5e8  packages/api/src/server.ts:136  client-defaults (deprecated)
  simondietschi/dokunc@83681ed0ccc5  apps/web/src/lib/log.test.ts:20  handshake-auth (breaking)
  simondietschi/dokunc@83681ed0ccc5  apps/web/src/lib/log.test.ts:28  handshake-auth (breaking)

PASS ioredis: candidate

Re-score of every registered pack, against main at 9ba005a (includes #139). Each pack was fetched once (scripts off), scored on main and on this branch, then its caches were evicted:

Pack Status Breaking P / R (TP/FP/FN), main @ 9ba005a Breaking, this PR Behavior notes, this PR pack test
ai verified → verified 100% / 82% (9/0/2) 100% / 82% (9/0/2) same n/a / n/a (0/0/0) pass → pass
lucide-react verified → verified 100% / 100% (11/0/0) 100% / 100% (11/0/0) same n/a / n/a (0/0/0) pass → pass
next verified → verified 100% / 85% (11/0/2) 100% / 85% (11/0/2) same n/a / n/a (0/0/0) pass → pass
react verified → verified 100% / 69% (89/0/40) 100% / 69% (89/0/40) same n/a / n/a (0/0/0) pass → pass
sentry-browser verified → verified 100% / 100% (3/0/0) 100% / 100% (3/0/0) same n/a / n/a (0/0/0) pass → pass
sentry-nextjs verified → verified 100% / 100% (8/0/0) 100% / 100% (8/0/0) same n/a / n/a (0/0/0) pass → pass
sentry-react verified → verified 100% / 100% (2/0/0) 100% / 100% (2/0/0) same n/a / n/a (0/0/0) pass → pass
stripe verified → verified 100% / 90% (18/0/2) 100% / 90% (18/0/2) same n/a / n/a (0/0/0) pass → pass
vitest verified → verified 100% / 50% (16/0/16) 100% / 50% (16/0/16) same n/a / 0% (0/0/13) pass → pass
zod verified → verified 100% / 74% (59/0/21) 100% / 74% (59/0/21) same n/a / n/a (0/0/0) pass → pass

Breaking precision, recall and false positives are identical for every pack, and so is each status. The only difference in the records is additive: behavior plus the truePositives/falseNegatives counts (commit 2). vitest is the one pack with behavior-note sites in its truth: 13 coverage-patterns sites, a note declared breaking. They stay breaking misses, as before, and are now also visible as behavior-note recall 0%.

How the runs were made: Yarn 4 through corepack enable yarn (tldraw) and COREPACK_NPM_REGISTRY, as the new CONTRIBUTING section says. A first pass with a blanket corepack enable could not install the repositories that pin pnpm (wortkarte, saas-starter, both sentry-react repositories) on main and on this branch alike. I re-scored those three packs on all checkouts with the documented setup; the table is that complete run.

pnpm lint, pnpm typecheck and pnpm test pass (core 1650 passed / 32 skipped, CLI 453 passed). pnpm docs:packs --check and pnpm docs:links pass.

Checklist

  • Tests for the change (a bug fix starts with a failing test)
  • pnpm lint && pnpm typecheck && pnpm test pass
  • README and docs updated if behavior changed (docs/packs.md "Scoring", CONTRIBUTING; the README table is generated and unchanged in content)
  • Changeset added if a user would notice (patch)
  • No new dependency, or it was agreed in an issue
  • No private code, names or paths in fixtures, tests or docs (test sites are synthetic)
  • The rules in CONTRIBUTING.md ("Rules that do not bend") still hold

🤖 Generated with Claude Code

https://claude.ai/code/session_011t9c3ZLsM9B3KiYGPFDyyK

Generated by Claude Code

claude added 3 commits October 14, 2026 21:00
@claude
fix(packs): score expected sites at their rule's or note's severity
2a3b16e 
`pack test` counted every missed site as a breaking false negative unless a
*rule* with that id was deprecated: `BehaviorNote.severity` was never read, so
missed behavior-note sites that only need a look (ioredis `client-defaults`)
were reported as missed breaks.

An expected site now takes the severity of the rule or note it is expected
under: a note's own `severity`, breaking when it reports findings without one
(as `check` reports them), and `behavior` for a note `check` never reports.
Behavior-note sites get their own precision and recall, in the output and in
verification.json (`behavior`); both tallies record the counts behind their
rates, so a zero denominator prints `n/a` in `pack test` and the README table
instead of 100%. False negatives print their severity. Older records still
read as before.

Fixes UPT-52.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011t9c3ZLsM9B3KiYGPFDyyK
Signed-off-by: Claude <noreply@anthropic.com>
@claude
chore(packs): record behavior-note tallies and the counts behind each…
34bc361 
… rate

`uptide pack test --write` for every registered pack, against the same
ground truth: each verification.json gains `behavior` and the true-positive
and false-negative counts under `breaking`. Status, truth digest,
repositories, and breaking precision, recall and false positives are
unchanged for all ten packs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011t9c3ZLsM9B3KiYGPFDyyK
Signed-off-by: Claude <noreply@anthropic.com>
@claude
docs(contributing): run pack test locally
9308233 
What a full `pnpm uptide pack test` run needs on a contributor's machine:
corepack enabled for Yarn only (a blanket `corepack enable` shims the
global pnpm too), COREPACK_NPM_REGISTRY where repo.yarnpkg.com is blocked,
and per-pack cache directories evicted after each pack to bound disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011t9c3ZLsM9B3KiYGPFDyyK
Signed-off-by: Claude <noreply@anthropic.com>
@lucasbraum lucasbraum mentioned this pull request Oct 14, 2026
Merged
7 tasks done
@lucasbraum
lucasbraum merged commit 883e587 into main Oct 15, 2026
10 checks passed
@lucasbraum
lucasbraum deleted the claude/amazing-albattani-8m8r4h branch October 15, 2026 00:26
lucasbraum added a commit that referenced this pull request Oct 15, 2026
@lucasbraum @claude
feat(packs): anchor a pure insertion into an unchanged call as a grou…
0881185 
…nd-truth site (UPT-53) (#144)

Fixes
[UPT-53](https://linear.app/uptide/issue/UPT-53/score-added-only-lines-as-ground-truth-sites).
**Stacked on [#143](#143
(UPT-52): review only the last commit; this PR retargets to `main` once
that one merges.

## What and why

docs/packs.md counted no pure insertion as a site. So an upgrade that
only **adds** lines inside an existing call could not be written as
ground truth. The typical example is an option added to a constructor:
open-foundry pinned `protocol: 2` inside `new RedisClient(redisUrl, {`
at `packages/api/src/server.ts:138`. A pack that reports the same
constructor can never score a true positive there; its finding shows as
a false positive. Found by the ioredis pack
([#129](#129)).

This PR:

- **Defines the site.** A pure insertion is a site when it adds to a
call or object the upgrade otherwise left unchanged. The site is the
start line of the innermost call (or `new`) whose parentheses hold the
inserted lines, through object and array literals, or of the object
literal when no call takes it. It is not a site when it falls between
statements, inside a function body (even one passed to a call), or
inside a call the upgrade also changed (those changed lines are already
its site).
- **`pnpm packs:truth` drafts these anchors**, for review like every
other drafted site. The JSON draft gets `anchors: [{ file, line, text,
within, inserted }]`; `changed` and `inserted` are as before. The diff
reading moves from `scripts/packs-truth.ts` into
`packages/core/src/packs/tooling/truth-diff.ts` (`diffLines`,
`enclosingLiteral`, `insertionAnchors`), where it is unit-tested; the
script's output for `changed` and `inserted` is unchanged.
- **Docs**: docs/packs.md "Ground truth" (the anchor rule, what
`packs:truth` prints) and CONTRIBUTING "Write a pack" step 5.

Scoring needs no change: sites are file and line, so an anchored site
scores like any other. No pack's ground truth changes here. Adding the
anchors to ioredis's truth is #129's call.

## How it was verified

**Failing test first.**
`packages/core/src/packs/tooling/truth-diff.test.ts` is modeled on
open-foundry `server.ts:135–145`, with synthetic code and the same diff
shape: lines 135–137 rewritten (the truth's site at `:136`) plus a pure
insertion after `:138`, inside the constructor call. Before this change
there was no anchor (the module did not exist), and the prediction at
`:138` scores as a false positive. After: the insertion is drafted at
`:138`, and with that site in the truth the prediction is a true
positive. Negative cases cover insertions between statements, into a
callback body, after the closing `)`, and into a call whose lines also
changed. Writing the tests also caught a real bug before it shipped: an
argument added after a trailing comma was missed.

**On the real repositories** (read only, scripts off), `pnpm packs:truth
ioredis …` on this branch drafts:

| Repository | Anchor | Inserted | ioredis pack today |
| --- | --- | --- | --- |
| syzygyhack/open-foundry | `packages/api/src/server.ts:138`
`redisClient = new RedisClient(redisUrl, {` | `protocol: 2,` | false
positive at `:138` |
| ferni-ai/voiceai | `src/services/data/redis-pubsub.ts:140`
`this.publisher = new Redis(redisUrl, {` | `protocol: 2,` | false
positive at `:140` |
| ferni-ai/voiceai | `src/services/platform/rate-limiter.ts:57`
`redisClient = new Redis(redisUrl, {` | `protocol: 2,` | false positive
at `:57` |
| simondietschi/dokunc | `apps/web/src/lib/redis.ts:54` `const client =
new Redis(url, {` | `retryStrategy: reconnectDelay,` | false positive at
`:54` |

These are exactly the four "real constructions the upgrade changed, but
by pure insertion" that #129's RESEARCH.md lists as unscorable. voiceai
also drafts one object anchor
(`src/utils/__tests__/error-scrub.test.ts:44`, `return {` gaining
`status: 'ok'`) from the @sentry/node bump in the same commit. The
reviewer drops it, as the drafting flow intends.

**Re-score of every registered pack.** Base: this branch's parent
(#143), itself on main at `9ba005a`. Same fetch, scripts off, caches
evicted per pack:

| Pack | Status | Breaking P / R (TP/FP/FN), #143 | Breaking, this PR |
Behavior notes, this PR | pack test |
| --- | --- | --- | --- | --- | --- |
| `ai` | verified → verified | 100% / 82% (9/0/2) | 100% / 82% (9/0/2)
same | n/a / n/a (0/0/0) | pass → pass |
| `lucide-react` | verified → verified | 100% / 100% (11/0/0) | 100% /
100% (11/0/0) same | n/a / n/a (0/0/0) | pass → pass |
| `next` | verified → verified | 100% / 85% (11/0/2) | 100% / 85%
(11/0/2) same | n/a / n/a (0/0/0) | pass → pass |
| `react` | verified → verified | 100% / 69% (89/0/40) | 100% / 69%
(89/0/40) same | n/a / n/a (0/0/0) | pass → pass |
| `sentry-browser` | verified → verified | 100% / 100% (3/0/0) | 100% /
100% (3/0/0) same | n/a / n/a (0/0/0) | pass → pass |
| `sentry-nextjs` | verified → verified | 100% / 100% (8/0/0) | 100% /
100% (8/0/0) same | n/a / n/a (0/0/0) | pass → pass |
| `sentry-react` | verified → verified | 100% / 100% (2/0/0) | 100% /
100% (2/0/0) same | n/a / n/a (0/0/0) | pass → pass |
| `stripe` | verified → verified | 100% / 90% (18/0/2) | 100% / 90%
(18/0/2) same | n/a / n/a (0/0/0) | pass → pass |
| `vitest` | verified → verified | 100% / 50% (16/0/16) | 100% / 50%
(16/0/16) same | n/a / 0% (0/0/13) | pass → pass |
| `zod` | verified → verified | 100% / 74% (59/0/21) | 100% / 74%
(59/0/21) same | n/a / n/a (0/0/0) | pass → pass |

Identical for every pack: this PR changes drafting and docs, not
scoring. `pack test --write` changed no record.

`pnpm lint`, `pnpm typecheck` and `pnpm test` pass (core 1656 passed /
32 skipped, CLI 453 passed). `pnpm docs:links` passes.

## Checklist

- [x] Tests for the change (a bug fix starts with a failing test)
- [x] `pnpm lint && pnpm typecheck && pnpm test` pass
- [x] README and docs updated if behavior changed (docs/packs.md,
CONTRIBUTING)
- [x] Changeset added if a user would notice (patch)
- [x] No new dependency, or it was agreed in an issue (uses core's
existing `ts-morph`)
- [x] No private code, names or paths in fixtures, tests or docs (test
code is synthetic; the table cites public repositories already in #129)
- [x] The rules in CONTRIBUTING.md ("Rules that do not bend") still hold

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011t9c3ZLsM9B3KiYGPFDyyK

---
_Generated by [Claude
Code](https://claude.ai/code/session_011t9c3ZLsM9B3KiYGPFDyyK)_

---------

Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
@uptide-release uptide-release Bot mentioned this pull request Oct 15, 2026
Merged
lucasbraum pushed a commit that referenced this pull request Oct 15, 2026
@claude
fix(packs): rescore ioredis on main with insertion anchors and note s…
e16104b 
…everities

Rebased on main (#143, #144). The ground truth gains the four ioredis
insertion anchors packs:truth now drafts (protocol: 2 and retryStrategy
added inside unchanged constructor calls); no earlier site moved. Behavior
notes 70% precision, 54% recall; still a candidate with no breaking false
positive and no breaking prediction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017BYqQRCpXk6jz9vcCnx21H
Signed-off-by: Claude <noreply@anthropic.com>
@lucasbraum lucasbraum mentioned this pull request Oct 15, 2026
Merged
7 of 8 tasks
lucasbraum pushed a commit that referenced this pull request Oct 15, 2026
@claude
fix(packs): rescore ioredis on main with insertion anchors and note s…
30579da 
…everities

Rebased on main (#143, #144). The ground truth gains the four ioredis
insertion anchors packs:truth now drafts (protocol: 2 and retryStrategy
added inside unchanged constructor calls); no earlier site moved. Behavior
notes 70% precision, 54% recall; still a candidate with no breaking false
positive and no breaking prediction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017BYqQRCpXk6jz9vcCnx21H
Signed-off-by: Claude <noreply@anthropic.com>
lucasbraum added a commit that referenced this pull request Oct 15, 2026
@lucasbraum @claude
feat(packs): add a candidate ioredis 5 to 6 pack (#129)
e796e4c 
## Pack

`ioredis` `>=5 <6` → `>=6 <7` (queue entry `ioredis`, now `blocked`).
**Candidate, not verified.** No breaking false positive on five public
merged upgrades. But none of them has a compiler diagnostic or a
breaking site in application code; the only breaking expected sites are
two lines of test data, which the pack doesn't predict. So the gate (≥1
predicted breaking finding) can't be met honestly. Evidence:
`packages/core/src/packs/ioredis/RESEARCH.md`.

**Rebased on main `3c8902d` (2026-10-15).** The only conflicts were
`registry.ts` (both entries kept, alphabetical) and the generated README
table (regenerated). The rebase also picks up #145's re-recorded
react-router `verification.json`; the Packs workflow had been red on
main since #143 without it.

After the rebase, #143 and #144 changed scoring and the ground-truth
rule:
- I re-ran `pnpm packs:truth` on all five repositories: still no
compiler diagnostic.
- I added the four ioredis insertion anchors it now drafts, unchanged.
- I dropped one anchor from the @sentry/node bump.
- No earlier site was moved or removed.
- The revision is recorded in the truth's `$comment` and in RESEARCH.md.

## Sources

All read 2026-10-14 and pinned:
- the official [Upgrading from v5 to
v6](https://github.com/redis/ioredis/wiki/Upgrading-from-v5-to-v6) (wiki
`e859962`)
- the CHANGELOG and README at `v6.0.0`
- #2127 (RESP3), #2160 (defaults), #2206 (String.prototype, only in
6.1.0-beta.1)
- the BullMQ 6.0.0 changelog
- the published `ioredis` 5.11.1/6.0.0 and `@ioredis/commands`
1.10.0/2.0.0 tarballs, diffed
- the dependents' registry manifests

v6 removes **no** command (373 → 382 methods) and no public export.
Reply shapes and types are unchanged for default (`"legacy"`) clients,
and so are event names and `exec()` types. Breaking changes: Node ≥20,
RESP3 by default, plus new `retryStrategy`/`keepAlive` defaults.

## Rules and behavior notes

**Assisted rules** (no rewrite):
- `bundled-ioredis`: TS2322/2345/2739-41 naming `Redis<"legacy">` not
assignable, i.e. a dependent that installs its own ioredis 5. The guide
gives options (a)–(d), so the agent's choice lands in *Decisions for
you*. Never a cast.
- `duplicate-signature`: TS2416/2430 on `'duplicate'` overrides.

**Finding notes** (syntactic, binding-aware):
- `client-defaults` (deprecated, also a decision): constructions that
leave `protocol`/`retryStrategy`/`keepAlive` at the changed defaults.
Tests are skipped.
- `subscriber-mode` (breaking): `mode === 'subscriber'`.
- `offline-queue-connect` (breaking): a `connect` listener sending a
command when `enableOfflineQueue: false`.
- `handshake-auth` (breaking): AUTH recognized by command name. v6 sends
credentials inside `HELLO 3 AUTH`.

**Decision notes** (with `applies()` evidence):
- `node-20`
- `bundled-copies`: bullmq <6, bull, @hocuspocus/extension-redis,
@chat-adapter/state-ioredis, cache-manager-ioredis-yet
- `instrumentation`: OTel `instrumentation-ioredis` <0.70 /
`auto-instrumentations-node` <0.80
- `stale-types`: `@types/ioredis`
- `string-prototype`: target exactly 6.0.0
- `also-review`

**Not declared:**
- No release group: ioredis ships alone.
- No `requiresBuild`: no case of a build failing where tsc passes.
- Explicit peers rejecting 6 (typeorm, ioredis-mock,
@liaoliaots/nestjs-redis) are left to the engine's peer-conflict report.

## Ground-truth repositories

| Repository | Parent | Upgrade | Findings |
| --- | --- | --- | --- |
| ferni-ai/voiceai | `c59ed09` | `c6838d5` (#703) | 7 `client-defaults`
(protocol: 2 pins, 2 of them insertion anchors) |
| syzygyhack/open-foundry | `7391001` | `a466e0d` | 2 `client-defaults`
(1 anchor) |
| simondietschi/dokunc | `83681ed` | `c475390` (#15) | 2
`client-defaults` (v5 retryStrategy, 1 anchor), 2 `handshake-auth` |
| safe-global/safe-client-gateway | `14bd05b` | `80135d2` (#3409) | 0
(control, Yarn Berry) |
| kazumadev619-dev/FishingConditionsApp | `8ba3f0d` | `cdecc03` (#235) |
0 (control, npm) |

Screened out: elie222/inbox-zero#3821. Its TS2322 (two copies via bullmq
5) is a `packs:truth` linking artifact; a real scripts-off install
compiles it. Its upgrade also moved bullmq 5 → 6.

## `uptide pack test` output

```
uptide pack test ioredis · candidate
fixtures      6 cases, 19 sites · pass
ground truth
  ferni-ai/voiceai@c59ed0982f96                ioredis 5.8.2 → 6.0.0  predicted 6, expected 7, false positives 2, false negatives 3
  syzygyhack/open-foundry@73910018b5e8         ioredis 5.10.1 → 6.0.0  predicted 1, expected 2, false positives 0, false negatives 1
  simondietschi/dokunc@83681ed0ccc5            ioredis 5.11.1 → 6.0.0  predicted 2, expected 4, false positives 0, false negatives 2
    ✗ blocked: @hocuspocus/extension-redis 4.7.0 depends on ioredis ~5.6.1; no release accepts ioredis 6.0.0
  safe-global/safe-client-gateway@14bd05b98bf4 ioredis 5.11.1 → 6.0.0  predicted 0, expected 0, false positives 0, false negatives 0
  kazumadev619-dev/FishingConditionsApp@8ba3f0d24bc7 ioredis 5.11.1 → 6.0.0  predicted 1, expected 0, false positives 1, false negatives 0
    with @types/ioredis 4.28.10 → 5.0.0

  rule                         precision  recall    tp   fp   fn
  client-defaults                    70%     64%     7    3    4
  handshake-auth                     n/a      0%     0    0    2
  overall (sites)                    70%     54%     7    3    6
  behavior notes (sites)             70%     54%     7    3    6
  breaking (sites)                   n/a      0%     0    0    2

PASS ioredis: candidate
```

The dokunc "blocked" line is the engine's own report (#142) of the
bundled hocuspocus copy that `bundled-copies` describes; the repository
upgraded anyway, keeping both copies.

**Remaining false positives:**
- voiceai `:101`/`:92` against truth `:98`/`:89`: the anchor is at the
first line of the rewritten run.
- FishingConditionsApp `client.ts:13`: a client deliberately left on
RESP3.

**Remaining false negatives:**
- voiceai `embedding-cache.ts:134`: the module name is hidden from
static scans.
- voiceai `:98`/`:89` and open-foundry `:136`: anchor offsets.
- dokunc `log.test.ts:20/28`: `handshake-auth`, test data.

**Engine gaps still open** (RESEARCH.md):
1. A runtime-only major can't reach `verified`.
2. Detection reaches only files with static usages.
3. There is no "remove a package" outcome (`@types/ioredis`).
4. `with` can't model a required dependent major.
5. `packs:truth` links per workspace, and `--directory` fails with a
root lockfile.

Gaps 2 and 3 of the original list were resolved by #143 and #144.

## Checklist

- [x] Every rule and note comes from a source listed in `meta.sources`
- [x] Every rule that rewrites or detects has a fixture case, with a
site it must leave alone (`keep`). The two compiler-claim rules are
unit-tested instead, because a `message:` marker can't contain `"`.
- [x] Ground truth: public repositories at the commit before their
upgrade; expected findings read from their own upgrade commit and the
compiler, never copied from Uptide's output
- [ ] At least two public repositories for `verified`; otherwise the
pack ships as a candidate (5 repos, still a candidate: no predicted
breaking site)
- [x] `pnpm uptide pack test ioredis` passes and its output is pasted
above; `verification.json` written with `--write`
- [x] `pnpm lint && pnpm typecheck && pnpm test` pass: core 1727 passed
/ 32 network-skipped, CLI 454 passed; lint at baseline; `docs:packs
--check` current. All 10 CI checks pass on `30579da`, Pack test
included.
- [x] Changeset added
- [x] No private code, names or paths anywhere: fixtures are synthetic,
ground truth is public

No third-party repository was pushed to, commented on or modified. Every
install ran with scripts off, and no bare `npx` was used.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_017BYqQRCpXk6jz9vcCnx21H

---------

Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
@lucasbraum lucasbraum mentioned this pull request Oct 15, 2026
Merged
lucasbraum added a commit that referenced this pull request Oct 15, 2026
@lucasbraum @claude
feat(packs): stricter verified gate: real breaking sites, unexercised…
922c5ab 
… rewrites run as assisted (UPT-48) (#155)

## Phase 2 report

1. **Sentry demoted.** `@sentry/browser` (3 real breaking sites) and
`@sentry/react` (2) are recorded `candidate`.
`packages/core/src/packs/sentry/RESEARCH.md` says they need at least 5
real breaking sites each, and lists repositories to search later:
cal.com, midday, rallly, inbox-zero, unkey, twenty.
2. **Unexercised rules run as assisted.** `pack test --write` ran once
on every pack except joi and redis. Only the two Sentry statuses
changed; no precision, recall or count moved. Each record now carries
`class`, `breakingSites`, `behaviorSites` and `unexercised`.
- `activePacks` wraps each pack: `transform` refuses an edit whose rule
is unexercised (fix hands the site to the agent with the guide), and
check demotes such a site to assisted. The wrapper checks the rule of
the applied edit, so the hand-written zod and stripe transforms are
covered.
- Stripe's pin and fixture-cast edits have no `rewrite` of their own. A
new `Pack.mechanical` list makes the gate count them.
- Now assisted: lucide-react `renamed-icons`; next
`stable-cache-exports`; vitest `removed-entrypoints`; zod
`error-members`, `error-params`; stripe `api-version-unpinned`,
`fixture-cast`; and the Sentry rules each record lists.
- **zod `error-members`: no truth added.** I scanned merged PRs that
bump zod 3→4 and remove a `.formErrors`/`.errors` read
(PrefectHQ/prefect#22795, bitwarden/ai-plugins#102,
jesseiancary/plinth#41, Seven74AI/music-library#176 and others). Every
catch-binding read is narrowed by `instanceof z.ZodError`, so it is
typed and compiler-visible (`error-issues`), not read through `any`.
Naeha-S/Bloggonut#2 and ooples/token-optimizer-mcp#167 do read `.errors`
in a catch, but on zod 4 already: they are later crash fixes, not 3→4
upgrades. So `error-members` runs as assisted.
3. **zod `error-params`, 0 of 3 public sites: not fixed.** All three are
found as breaking true positives but filed under `generic`. They are
`errorMap` sites: `z.literal(true, { errorMap })` in AwesomeIntune, and
`{ errorMap }` plus a `z.ZodErrorMap` function in PATH. The
`error-params` rewrite only handles
`required_error`/`invalid_type_error` and skips `errorMap` ("needs
manual merging"). zod has no `ruleOf`, so the plan files a site under
`error-params` only when the rewrite applies or the error text names
those two keys. Re-filing them under `error-params` would count them as
evidence for a rewrite that never ran on them. The right fix is a
separate assisted `error-map` rule with its own guide (zod 4: `error`
replaces `errorMap`, and the map's signature changed). That is a pack
change, left for a follow-up.
4. **joi and redis stay candidate** (not written). Why redis's record
differed on purpose: it is still the scaffold record (0 repos, zero
counts), never written. #151 held it at candidate until this gate
landed, because:
   - breaking recall is 61% (11/18);
- behavior notes are 50/50, with all four `client-defaults` predictions
false positives;
- the core 4→5 API changes (`command-options`, `legacy-mode`,
`scan-iterators`, `number-replies`, `keep-alive`) have no real site;
- two engine gaps cost sites: no compile errors in files that reach the
client through a local wrapper, and errors probably subtracted where the
installed version already had one.

Under this gate it measures verified (5 repos, 20 real breaking sites, 0
breaking FP). `keep-alive`, `quit-disconnect`, `scan-iterators` and
`unstable-resp3` would run as assisted.
5. **Override and README.** `meta.minBreakingSites` lowers the minimum
for one pack, and `pack test` fails a pack that sets it without a reason
under `## Breaking-site minimum` in its RESEARCH.md. No pack uses it.
The README packs table has a "Breaking sites" column, and docs/packs.md
and CONTRIBUTING describe the gate.

Also: `--write` now formats a record the way biome does (a short list
inline), so a written record passes lint. The fix-machinery tests use
zod's `required_error` rewrite as their mechanical example; they mock
the registry wrapper to keep it as written.

Local runs: the `@sentry/react` truth repo pins pnpm 12.8.2, which
corepack 0.36 cannot launch (pnpm 12 ships a native binary its
postinstall fetches). I scored it with a local pnpm 12.11.2 through a
scratch shim. CI's corepack may hit the same thing if the repository is
not in its cache. These tests also fail on main locally (environment):
required-build, peer-preflight, scaffold, release, telemetry sender,
third-party notices.

---

## Phase 1: the gate and its impact (measured before the records were
written)

## The gate

`verifiedGate` (contract.ts) replaces `statusOf`. `gateOf`
(pack-test.ts) feeds it from one run; "real" means public repositories,
so a fixture repository is scored and never counts.

| | breaking class (default) | behavior-only class |
|---|---|---|
| applies when | any real expected site is under a rule or `generic` |
every real expected site is a behavior note's, whatever its severity
(ioredis 6) |
| repositories | ≥ 2 public | ≥ 2 public |
| real sites | ≥ 5 expected breaking sites
(`VERIFIED_MIN_BREAKING_SITES`) | ≥ 5 expected note sites
(`VERIFIED_MIN_BEHAVIOR_SITES`) |
| false positives | 0 among breaking findings | 0 among breaking
findings, 0 on note sites, 0 on mechanical rules |
| found | ≥ 1 breaking site | ≥ 1 note site; per-note recall is in the
rules table (#143) |

**Unexercised rewrites**: a rule with a `rewrite` and no true positive
in public ground truth is listed as unexercised. It never changes the
status.

- `pack test` prints a `gate` line (class, real site counts, status),
each reason a pack is a candidate, and the unexercised rules.
- The computed record carries `class`, `breakingSites`, `behaviorSites`
and `unexercised`, so every record is now stale (a note, not a failure)
until `--write`.
- `overclaims()` is unchanged from #154: a record fails only when it
records verified and the run measures candidate; recorded candidate by
choice passes.

**Not in phase 1**
- `fix` running unexercised sites as assisted: it needs the
`unexercised` list in each record, which only phase 2's `--write`
produces. Wrapping `transform` for the hand-written packs (zod, stripe)
is the work there.
- A per-pack override of the minimum with a written reason in
RESEARCH.md.
- The README breaking-site column.

## Impact: every registered pack, under the new gate

Each pack was run one at a time from source (`pnpm uptide pack test <id>
--json`), each with its own cache directory, deleted after the pack.
Site counts are expected sites in public ground truth. "Breaking
tp/fp/fn" is across every scored repository, fixtures included.

| pack | recorded | measures now | real breaking / note sites | breaking
tp/fp/fn | unexercised mechanical rules | why |
|---|---|---|---|---|---|---|
| ai | verified | verified | 11 / 0 | 9/0/2 | — | |
| lucide-react | verified | verified | 11 / 0 | 11/0/0 | renamed-icons |
|
| next | verified | verified | 13 / 0 | 11/0/2 | stable-cache-exports |
|
| react | verified | verified | 130 / 0 | 89/0/40 | — | |
| react-router | verified | verified | 18 / 0 | 14/0/2 | — | see below |
| stripe | verified | verified | 16 / 0 | 18/0/2 | — | |
| vitest | verified | verified | 32 / 13 | 16/0/16 | removed-entrypoints
| |
| zod | verified | verified | 85 / 0 | 59/0/21 | error-members,
error-params | error-params' TPs are all in the fixture repo; its 3
public sites are missed |
| @sentry/nextjs | verified | verified | 8 / 0 | 8/0/0 | enable-logs,
enable-metrics, http-integration-options, integration-names,
renamed-exports, streamed-span-wrapper | |
| **@sentry/browser** | verified | **candidate** (overclaim, fails) | 3
/ 0 | 3/0/0 | enable-metrics, integration-names, renamed-exports,
streamed-span-wrapper | 3 breaking sites, needs 5 |
| **@sentry/react** | verified | **candidate** (overclaim, fails) | 2 /
0 in truth | — | enable-logs, enable-metrics, integration-names,
renamed-exports, streamed-span-wrapper | 2 breaking sites in its truth,
needs 5. Locally one repo failed to install (pnpm crashed with
MODULE_NOT_FOUND, environmental), but the count alone drops it |
| joi | candidate | verified | 108 / 0 | 108/0/0 | — | recorded
candidate by choice: passes |
| redis | candidate | verified | 20 / 2 | 11/0/7 | keep-alive,
quit-disconnect, scan-iterators, unstable-resp3 | recorded candidate by
choice: passes |
| ioredis | candidate | candidate (behavior class) | 2 / 13 | 0/0/2 | —
| 3 false positives on `client-defaults` note sites |

**Drops**: only @sentry/browser and @sentry/react. Both now fail `pack
test` as overclaims, so the Packs job is red on this PR until phase 2.

**react-router**: on main, the only mechanical rule is `dom-imports` (29
real TPs). `removed-json`, `removed-defer`, `form-method-case`,
`dom-server-entry` and `future-option` have no `rewrite`; they are
already assisted, so the gate turns none of them assisted. Separately,
`dom-imports` has 336 non-breaking false positives at the rule level
(precision 8%), which no gate criterion reads.

**Mechanical rules that would turn assisted** (unexercised): 24 rule
entries across 8 packs, 15 distinct ids. lucide-react `renamed-icons`;
next `stable-cache-exports`; vitest `removed-entrypoints`; zod
`error-members`, `error-params`; redis `keep-alive`, `quit-disconnect`,
`scan-iterators`, `unstable-resp3`; and the Sentry rules listed per pack
above.

## Tests

`contract.test.ts` has a pass and a fail case for each criterion, plus
the unexercised list and the behavior-only class (verified, too few
sites, false positives on notes and rules, never chosen with a rule
site). `pack-test.test.ts` covers `gateOf`: fixture sites never count or
exercise a rule, and an unscored repository is no evidence.
(`scaffold.test.ts` fails locally with `formatted: false` on main as
well; unrelated.)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
lucasbraum pushed a commit that referenced this pull request Oct 15, 2026
@uptide-release
Version Packages (#133)
530f90c 
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.


# Releases
## uptide@0.13.0

### Minor Changes

- [#134](#134)
[`b795705`](b795705)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Packs can name
alternate leads: packages a repository declares instead of the pack's
own (`react-router-dom` for `react-router`). `check` and `fix` asked for
either name use the pack; where only the alternate is declared, the lead
is analyzed at the version installed through it, pnpm stores included,
and `fix` declares it beside the alternate at the target before
upgrading both.

- [#149](#149)
[`3303118`](3303118)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add a candidate
express 4 to 5 migration pack. Route strings that path-to-regexp 8
rejects or reads differently are found on calls of an Express app or
router (including lists registered in a loop); `req.params` values
widened to `string | string[]`, `req.param`, `res.sendfile`,
`express.static.mime` and `express.query` are claimed from the compiler;
`res.redirect(url, status)`, `app.param(fn)`, `res.redirect('back')` and
`res.send(<number>)` are found in the source; runtime defaults (async
errors, `req.body`, the query parser, `urlencoded`, dotfiles, Node 18)
go to decisions. No rule rewrites. With the pack in use, `check` no
longer adds its text-matched express 5 runtime findings.

- [#148](#148)
[`5143c46`](5143c46)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add a candidate
joi 17 to 18 migration pack. `new ValidationError(message, {},
original)`, which stops compiling for code written against joi
17.0–17.6, is rewritten to pass `[]`; the Node.js 20 requirement and the
top-level domains joi 18 no longer accepts in `email()`, `domain()` and
`uri()` are reported as decisions. Scored against two public
repositories, one of them a negative control.

- [#131](#131)
[`413d3da`](413d3da)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add a verified
lucide-react 0.x to 1.x migration pack. Brand icons removed in 1.x are
found at their imports and every use, with a guide whose replacement
choice goes to the owner; icon names dropped during 0.x are renamed at
the import or member; deep `dist/` imports, untyped icon lookups, the
`absoluteStrokeWidth` and `IconNode` deprecations, and the `aria-hidden`
default for repositories before 0.489.0 are reported. Lucide's lockstep
packages move together and the consumer's build is required. Verified
against three public repositories, one of them a negative control.

- [#132](#132)
[`107c107`](107c107)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add a verified
React Router 6 → 7 migration pack, which also runs for applications that
declare only `react-router-dom`. It moves `react-router-dom` imports to
`react-router`. It reports, with a guide for the agent, the rest of the
upgrade: the removed `future` prop and router option,
`react-router-dom/server`, lowercase `formMethod` comparisons and
`json`/`defer` calls (found by syntax where the compiler is blind), the
other removed exports, and `generatePath` calls that pass `null` for a
required parameter. The six v6 future flags that become defaults are
reported as decisions where the repository's router setup has not
enabled them. React Router 7 → 8 is a separate upgrade.

- [#142](#142)
[`9c57934`](9c57934)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check` and `fix`
plan what the target version requires: a package the target, or anything
moving with it, declares as a peer or dependency at a range the
installed copy rejects moves to the lowest release of its major that
satisfies it (openai 7 moves zod 3.24.2 to 3.25.0), shown apart from
release-group companions ("moved because openai 7.4.0 requires zod ^3.25
|| ^4.0"). A move only a new major makes is never made: it is reported
as a blocker with the exact range in `check`, the PR body and `pack
test`. The npm install scope guard no longer lets a declared root
dependency change because a moved package peers on it.

- [#128](#128)
[`06f11ab`](06f11ab)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add verified
Sentry JavaScript SDK 10 to 11 migration packs for `@sentry/nextjs` and
`@sentry/browser`, with the SDK's lockstep release group, a required
build for Next.js, and inventory-scoped decisions for data collection,
span streaming, sampling and platform minimums. Mechanical rules move
`withSentryConfig` to `@sentry/nextjs/config` and the removed build
options under `webpack`, remove `enableLogs`/`enableMetrics: true`, and
rename exports, integration names and `httpIntegration` options;
privacy, sampling, profiling and OpenTelemetry choices stay assisted and
are reported for the owner. Fix `uptide pack new` dropping registry
entries long enough to be wrapped.

- [#155](#155)
[`922c5ab`](922c5ab)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide pack
test` applies a stricter verified gate: at least five expected breaking
sites across the public ground-truth repositories, besides two
repositories and no breaking false positive. Upgrades the compiler
cannot see are gated on their behavior notes instead. Mechanical rules
no real site exercises are recorded as unexercised, and `check` and
`fix` run their sites as assisted. `@sentry/browser` and `@sentry/react`
become candidates (fewer than five real breaking sites). A pack can
lower the minimum only with a written reason in its RESEARCH.md. The
README packs table shows each pack's breaking-site count. The output
shows the gate's class, site counts and every reason a pack is a
candidate.

- [#147](#147)
[`0b88170`](0b88170)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check` and `fix`
remove a deprecated `@types` stub instead of upgrading to it. When the
release the upgrade would move `@types/<pkg>` to is deprecated on npm,
and `<pkg>`'s target declares its own types (`types`, `typings` or a
`types` export), the plan removes it: "Removed: @types/ioredis, ioredis
5 ships its own types", in `check` and in the PR body. `check` now diffs
such a package against its own target types instead of failing on a stub
with no declarations, and `fix` removes the entry in the same install,
with the install scope guard allowing exactly that removal.

- [#150](#150)
[`4c3212b`](4c3212b)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check
typescript` judges a TypeScript 5 to 6 upgrade with both compilers:
every project of the workspace is compiled with the installed TypeScript
and with the target's, and what is new at the target is a finding; the
bundled compiler never stands in, and a target without a JavaScript API
(TypeScript 7) is reported as not compiled. A candidate typescript 5 to
6 pack reports the tsconfig options 6.0 deprecates where they are set,
the rootDir default, module-keyword namespaces and import assertions. A
peer that rejects the target (typescript-eslint on TypeScript 6) is now
reported as a peer conflict even when it is not grouped with the
package.

- [#137](#137)
[`21894d7`](21894d7)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add
`verify.build` to uptide.yml: the package.json script a migration pack's
required build runs. Unset, Uptide runs `build:ci`, then `build-ci`,
then `build`. When a build fails before the framework's build starts
because it needs an environment variable or a database (`prisma db push`
without `DATABASE_URL`), the pull request and the report say so ("build
needs DATABASE_URL; set verify.build") instead of a bare "build failed".
`uptide init` writes the setting as a comment.

### Patch Changes

- [#141](#141)
[`9debb6d`](9debb6d)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check` no longer
reports a pre-existing type error as a breaking site when the upgrade
only changes its wording, such as a type the message prints in full. An
error at the same place with a different reason is still reported.

- [#129](#129)
[`e796e4c`](e796e4c)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add a candidate
ioredis 5 to 6 migration pack: client constructions that inherit the
RESP3, reconnect and keep-alive defaults ioredis 6 changed,
compiler-blind sites (`mode === 'subscriber'`, commands in a `connect`
listener with the offline queue off, AUTH recognized by command name),
dependents that bundle ioredis 5, instrumentation that stops patching
ioredis 6, and Node 20. It is scored against five public upgrades and
stays a candidate: none of them has a breaking site to verify it with.

- [#139](#139)
[`9ba005a`](9ba005a)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Allow local check
and fix smoke tests of registered candidate packs with
--allow-candidate, with an explicit unverified banner. Reject candidate
opt-in in the Action and with publication flags, and prevent stored
candidate runs from being published. Label pack-claimed assisted sites
by rule id even with --no-llm.

- [#157](#157)
[`f3af6a1`](f3af6a1)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `check` no longer
reports a blocker for a direct dependency whose plain dependency range
rejects the target (`@azure/ai-projects` on `openai ^6`, verdaccio
pinning express 4): the package manager gives it its own nested copy, so
it is listed as staying, with the new major that would accept the target
when there is one. `pack test` counts a `with` move that `check` leaves
and explains (a blocker, a new-major stay or a peer conflict) as
explained, and one of a package that does not depend on the lead as the
repository's choice; neither fails the run.

- [#130](#130)
[`5ced450`](5ced450)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add a candidate
openai 4 to 7 migration pack: named path parameters, `del` → `delete`,
promoted beta chat helpers, renamed runner events, removed entry points
and request-option overloads are rewritten where unambiguous; tool-call
unions, `httpAgent`, `fileFromPath` and the other removals get agent
guides; `APIError.headers`, Web `Response` bodies and double-encoded
path ids are detected even in code typed `any`; Node 22, zod, fetch and
client-subclass decisions carry repository evidence. It stays a
candidate (the dependency is still checked and fixed as generic) until
two engine gaps its ground truth exposed are closed.

- [#136](#136)
[`578a042`](578a042)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Fold implicit-any
errors (TS7006, TS7031, TS7019) inside an option the target removed into
the pack finding on that option, as downstream evidence: the callbacks
lost the types the option gave them, and the same edit resolves them.
They no longer stand as separate breaking sites.

- [#143](#143)
[`883e587`](883e587)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide pack
test` scores each expected site at the severity of the rule or behavior
note it belongs to, so a missed note that only needs a look, or a
decision, no longer counts as a missed break. Behavior-note sites get
their own precision and recall in the output and in `verification.json`
(`behavior`, with the counts behind each rate), false negatives show
their severity, and a rate with nothing to measure prints `n/a` instead
of 100%, in `pack test` and in the README's packs table.

- [#154](#154)
[`7aa3d58`](7aa3d58)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide pack
test` fails a pack's `verification.json` only when it claims more than
the run measures (recorded verified, measured candidate). A pack
recorded as candidate on purpose passes, with `recorded candidate by
choice; measures verified`, and a record whose numbers differ gets a
note instead of a failure. A `generic` ground-truth site now counts as a
true positive for the rule that claims it.

- [#152](#152)
[`ca31aee`](ca31aee)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide pack
test` takes several packs: `uptide pack test zod ai`.

- [#159](#159)
[`1e157fd`](1e157fd)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide pack test
--json` times each ground-truth repository: `timing` gives the fetch,
install and check wall time and the process's peak memory.

- [#144](#144)
[`0881185`](0881185)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Ground truth can
name a site for an upgrade that only adds lines inside an existing call,
such as an option added to a constructor: the site is the call's start
line, so a pack finding there scores as a true positive instead of a
false positive. `pnpm packs:truth` drafts these anchors from insert-only
hunks for review, and docs/packs.md and CONTRIBUTING describe the rule.

- [#156](#156)
[`9039e3c`](9039e3c)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Public ground
truth now exercises three rules that ran as assisted: zod
`error-params`, stripe `api-version-unpinned` and next
`stable-cache-exports`. Their rewrites run mechanically again. All three
packs stay verified.

- [#138](#138)
[`0db5640`](0db5640)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Respect
repository tsconfig file selection, recursive project references, and
workspace project options when checking dependency upgrades. Compiler
diagnostics outside every TypeScript project no longer produce breaking
findings; pack rules and build-configuration findings retain their own
scope. Repositories without a tsconfig retain importer-based
compilation, with that fallback stated in the report.
  
Anchor missing members supplied by dependency augmentations at their
setup import, retaining affected calls as evidence instead of counting
each call as a separate breaking edit.

- [#151](#151)
[`9ddaaff`](9ddaaff)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Add a candidate
redis (node-redis) 4 to 5 and 5 to 6 migration pack, one rule set per
major: an upgrade gets the rules of the majors it crosses, so 4 to 6
gets both. 4 to 5 covers `legacyMode`, `commandOptions` and isolation,
scan iterators that yield batches, string SCAN cursors, boolean replies
that became numbers, enums that became constants, `keepAlive`,
`@redis/graph` and internal `dist` imports; 5 to 6 covers the RESP3
client type, `unstableResp3`, and the client defaults 6 changes (RESP3,
command timeout, keep-alive delay, maintenance notifications) with a
decision per client. Scored against five public upgrades; it stays a
candidate until the stricter verified gate lands, since the core 4 to 5
API changes have no real site yet.

- [#135](#135)
[`42a3b54`](42a3b54)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Move only
packages released with the upgraded one. A pack's declared release group
now bounds the lead's npm scope, and without one a same-scope package
moves only when a pin, a peer or a release at the target relates it to
the group. Packages of the scope that are versioned on their own
(`@sentry/vite-plugin` 5.x, `@sentry/cli`, the old `@sentry/utils` 8.x)
are reported as "stays at X" in `check`, the pull request and `pack
test` instead of making `fix` refuse the whole upgrade. Registers the
verified `@sentry/react` 10 to 11 pack.

- [#146](#146)
[`09b8682`](09b8682)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Report Stripe
clients pinned to different API versions at the top of `check` and of
the PR body, as a decision for the owner: every `new Stripe(...)` in
TypeScript and JavaScript sources, scripts included, with the version it
speaks after the upgrade (an unpinned client speaks the SDK default).
The PR's decisions list, for each client behind the target, the
changelog entries it would cross for the resources it calls. No pin is
rewritten.

- [#140](#140)
[`bc428d1`](bc428d1)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Keep migration PR
bodies within their size budget while preserving summaries,
representative diffs, decisions, review warnings and verification ahead
of changelog and transitive detail. Keep every evidenced Stripe change
visible and uncapped; group and cap only other changes to used
resources, link the full changelog, and identify omitted text in the
retained report.

- [#145](#145)
[`3c8902d`](3c8902d)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - The zod pack
migrates `ZodError#errors` to `.issues` by rule: reads
(`error.errors[0].message`, `error?.errors`) and destructuring (`const {
errors } = error` becomes `const { issues: errors } = error`) where the
type checker resolves the receiver to `ZodError`, and reads off an
untyped catch binding of a zod parse. Writes are left to the agent.

- [#127](#127)
[`c4ead4f`](c4ead4f)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - zod: `check` now
reports `ZodError#formErrors` and `#errors` read off an untyped catch
binding of a zod `parse`/`parseAsync` (`catch (error: any) { return
error.formErrors.fieldErrors }`), which zod 4 removed and the compiler
cannot see. Files that only import a schema from the repository's own
module (`@/lib/zod`) are covered too. `fix` rewrites
`<err>.formErrors.fieldErrors` and `<err>.formErrors.formErrors` to
`z.flattenError(<err>)`, importing `z` where it is missing; every other
read goes to the agent.

Co-authored-by: uptide-release[bot] <338861287+uptide-release[bot]@users.noreply.github.com>
@lucasbraum lucasbraum mentioned this pull request Oct 17, 2026
Merged
lucasbraum added a commit that referenced this pull request Oct 17, 2026
@lucasbraum @claude
fix(react): report RefObject<T> declarations that need | null (UPT-…
4550da5 
…87) (#201)

Fixes UPT-87.

## Reconciliation first
The committed `verification.json` said 69% (89 TP / 40 FN). A fresh
local `pack test react --offline` measures **107 TP / 22 FN (83%)**, and
CI's `pack react` job on 4c3dc23 measures the same numbers (artifact
`pack-test-react` of run 39009240360). So it is a stale record, not the
environment: #198 left the react tally as it was (its message says so),
and the scorer changed in #143 and #155 after the record was written.

## Reach, then rules (22 breaking misses)
| cause | misses |
|---|---|
| reached; the compiler reports at the use site, truth marks the
`RefObject<T>` declaration (`ref-object-nullable`) | 16 |
| `global-jsx-namespace` with no finding in the file | 4 |
| `generic` in dev-docs | 2 |

Most of the 16 declarations use the global `React.RefObject` with no
react import, so `detect` never read them.

## Change
- `ref-object-nullable` gets a syntax-tree `detect`: `RefObject<T>` /
`React.RefObject<T>` where `T` is an element type or a type parameter
bounded by one, has no `null`, and is not a cast.
- The pack also reads every project file containing `RefObject<` (same
pattern as the ai pack, UPT-33).
- Near-miss fixtures: `| null`, `RefObject<number>`, a cast. A cast is
excluded because it would be a breaking false positive at
`TldrawUiContextualToolbar.tsx:56` (truth lists :55, not :56); :55 stays
missed.
- `verification.json`, README and `docs/compatibility.md` regenerated.

## Before / after (same pinned ground truth, not touched)
| | before (measured) | after |
|---|---|---|
| breaking recall | 83% (107/129) | **94.6% (122/129)** |
| breaking FP | 0 | 0 |
| unverified FP | 8 | 8 |
| `ref-object-nullable` | 38 TP / 16 FN | 53 TP / 1 FN |
| other rules | unchanged | unchanged |

Pack stays verified; thresholds untouched.

## Remaining 7 misses
- `dev-docs/src/pages/index.tsx:1`,
`dev-docs/src/components/Homepage/index.tsx:1` (excalidraw): its
tsconfig says it is not used in compilation.
- `examples/excalidraw/with-nextjs/src/app/page.tsx:19`: a separate
example package, outside the analyzed program.
- `apps/dotcom/client/tsconfig.json:15` (tldraw): a config file; truth
marks the cause, no diagnostic lands there.
- `ButtonIconCycle.tsx:23`, `apps/examples/src/ExampleWrapper.tsx:111`:
global-JSX sites with no finding in the file; **not investigated beyond
that**.
- `TldrawUiContextualToolbar.tsx:55`: the cast, see above.

Suspect truth entries to review (not changed): :55 vs :56 above are the
same cast, one listed and one not.

Core tests pass (`pnpm test` in `packages/core`).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lucasbraum @claude