Repository navigation
fix: price Sonnet 5.5 cache reads at $0.10 and make SDK cost auditable per model (UPT-128, UPT-127) - #268
Merged
lucasbraum merged 3 commits intoOct 20, 2026
Conversation
…ok (UPT-128) Anthropic's pricing page (read 2026-10-20) prices Sonnet 5.5 and Opus 5.5 cache hits at 0.05x base input: $0.10/MTok on Sonnet 5.5, not $0.20. Every other Anthropic cell (input, output, cache read, 5m and 1h cache write) for the five listed models matches the page. pricing.test.ts pins each Anthropic cell to one expected table, and the OpenAI and Gemini rows as an inline snapshot, so a changed cell fails CI. sdkCostEstimate prices uncached token equivalents only, so its numbers and the recorded cost-runs.json expectations do not move. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
…, keep the SDK figure apart (UPT-127) The SDK executor reported costUsd straight from the Agent SDK's total_cost_usd, whose price table is inside its binary, and merged uncached input, cache reads and cache writes into one inputTokens (sdk-agent.ts:1058-1062). Nothing could be recomputed by hand. - sdkSpend reads the result's modelUsage: per model, uncached input, output, cache reads and cache writes. modelUsage has no TTL split; the main loop's usage.cache_creation does, so its 1-hour writes go to the session model. - costUsd is tokens x pricing.ts rates per class; the SDK total is sdkReportedCostUsd, never mixed in. llm.models lists tokens, rates and cost per model in report.json and --json; inputTokens is uncached input only. - --max-cost (feedback rounds, typical session, the Action's budget) counts the higher of computed and SDK-reported cost. - A gap above 5% goes to llm.warnings and the terminal, not the PR body. - A model missing from pricing.ts gets no guessed rate: the SDK figure, marked unverified, with a warning. - The patch executor reports the same per-model row (one model per call). Real run, zod 3 -> 4.6.7 on consetto/calmcp: computed $0.1622007, SDK $0.1622007. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
|
Review complete. No blocking issues — approved ✅; 3 nitpicks below. 🧹 Nitpicks (3) — 🟢 3 low
Reworks LLM cost accounting in the fix pipeline around the SDK's own reported spend, adds cache-aware pricing and a per-model cost breakdown, and corrects the Sonnet cache-read rate.
Three low-severity issues remain: Reviewed commit: b95d7fa |
…n; drop the unverified provider snapshot CI's "No private material" step (scripts/private-material.mjs) failed both test jobs: `.changeset/auditable-sdk-cost.md` matches PRIVATE_DOC in scripts/public-tree.mjs (`audit[^/]*\.md`). It is now sdk-cost-per-model.md. Locally the step exits early without UPTIDE_PRIVATE_DENYLIST, so it never ran. pricing.test.ts no longer pins the OpenAI and Gemini rows as an inline snapshot: they were not re-checked (out of scope), and a pin would read as verified. The Anthropic rows stay pinned against a literal table. docs/model-pricing.md states the 5-minute assumption for cache writes by any model other than the session's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
lucasbraum
deleted the
lucasbraum/upt-127-uptide-fix-json-cost-cannot-be-audited
branch
October 20, 2026 17:31
Merged
lucasbraum
pushed a commit
that referenced
this pull request
Oct 20, 2026
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## uptide@0.18.2 ### Patch Changes - [#267](#267) [`907dffb`](907dffb) Thanks [@lucasbraum](https://github.com/lucasbraum)! - The agent can no longer change dependencies outside the upgrade. An agent edit that changes or removes an existing dependency in any `package.json`, or touches `overrides`, `resolutions` or the `pnpm` field, is refused and reverted. Uptide sends the refusal back to the agent in the next feedback round, naming the package, and records it in `report.json` (`outOfScopeDependencyEdits`). That includes the upgraded package and its planned companions, whose versions the engine wrote: only a rewrite to the same value is allowed. The agent may still add a new dependency: it is kept, recorded (`agentAddedDependencies`) and listed under "Added by the agent" in the PR body. Lockfiles, `.npmrc`, `.yarnrc(.yml)`, `.pnpmfile.cjs`, `pnpm-workspace.yaml` and new `package.json` files are read-only for the agent. - [#267](#267) [`907dffb`](907dffb) Thanks [@lucasbraum](https://github.com/lucasbraum)! - A companion that would move a new major is planned only when the upgraded package asks for it: its own manifest at the target pins the companion, the companion's manifest declares the upgraded package, or the upgraded package's pack names it. When a third package asks for it (`@guardian/libs` 33 pinning `@guardian/ophan-tracker-js` 3.1.0 during a typescript upgrade), Uptide no longer moves it. The PR lists it under "Blocked" with the declaring package, the manifest field and the range. Release pairs still move with their lead: `react-dom` with `react`, `@ai-sdk/*` with `ai`, `@sentry/profiling-node` with `@sentry/node`. - [#267](#267) [`907dffb`](907dffb) Thanks [@lucasbraum](https://github.com/lucasbraum)! - A package that does not move no longer drags another one through its pins: `fix typescript` on guardian/support-dotcom-components moved `@guardian/ophan-tracker-js` 2.8.3 → 3.1.0 because `@guardian/libs` 33.0.0, which stayed, pins it. A pin the same manifest also declares as a peer the install satisfies moves nothing either, and a pin reason now names the manifest field and the installed version. - [#266](#266) [`a5e3f57`](a5e3f57) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide fix` no longer upgrades a peer blocker across a major that no migration pack covers. Before, `redis` 5 → 6 next to `typeorm` 0.3 (which peers on redis ^5) moved typeorm to 1.1.0 and ended in an agent-led draft. Now the blocker keeps its version, with a peer override scoped to it in package.json: npm `overrides.<blocker>.<peer>`, pnpm `pnpm.peerDependencyRules.allowedVersions`, Yarn `resolutions["<blocker>/<peer>"]`. The lockfile scope check still covers the install. The PR body names the override, which release was not taken and why. In `report.json`, each `peerConflicts` entry now has `strategy` (`override`, `blocked` or `none`), `declined` and `override`. If verification fails with the override, the blocker is reported as a blocker and is still not upgraded. A blocker you name on the command line, or one whose major a pack covers, is upgraded as before. - [#264](#264) [`2089d0a`](2089d0a) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `uptide fix` no longer stops when the package manager resolves a different version than the target. Yarn classic ignored the pin for a direct dependency and resolved a range such as `^7.0.105` to its newest match (7.0.146). Now the lockfile resolves exactly the target with npm, pnpm, Yarn classic and Yarn Berry, and the manifest keeps its range. If the package manager still resolves another version, the run continues only when that version is in the same major, inside the manifest range and covered by the pack. The PR body, the terminal summary and `report.json` (`resolvedVersion`, `lockfile.resolved`) then name both versions. Otherwise the run stops with an error that names the target, the resolved version and the reason. - [#268](#268) [`4cee03a`](4cee03a) Thanks [@lucasbraum](https://github.com/lucasbraum)! - `fix` computes the agent's cost itself, per model and token class, from Uptide's price table: `report.json` and `--json` list each model's uncached input, cache read, 5-minute and 1-hour cache write and output tokens, the rate applied to each and the cost, so the total can be redone by hand. The Agent SDK's own figure is kept apart as `sdkReportedCostUsd`; `--max-cost` counts the higher of the two, a gap above 5% is printed as a warning, and a model missing from the table uses the SDK's figure, marked unverified. `llm.inputTokens` now counts uncached input only. - [#268](#268) [`4cee03a`](4cee03a) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Sonnet 5.5 cache reads are priced at the published $0.10 per million tokens (0.05x input), not $0.20. Every Anthropic rate in the price table was checked against Anthropic's pricing page on 2026-10-20, and a test now pins each one. Co-authored-by: uptide-release[bot] <338861287+uptide-release[bot]@users.noreply.github.com>
lucasbraum
added a commit
that referenced
this pull request
Oct 20, 2026
…and, PR-template rule, dev set (UPT-24) ecommerce's RESP3 is missed at the location; the committed mention rule is a footnote. Unrelated changes are judged by hand per run: Uptide's ophan-tracker 2->3 makes support-dotcom-components not mergeable without edits; Claude Code's strut test edits follow the required aieo bump. yea's PR-description check is out of the bar for both arms. Arm A cost cites #268. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
lucasbraum
added a commit
that referenced
this pull request
Oct 22, 2026
…o the upgrade's copy (UPT-134, UPT-135) (#271) Hotfix for 0.18.3. Three signed-off commits, in order: UPT-134, UPT-135, and the small group-title fix. ## UPT-134: the agent silently does nothing **Root cause** (`packages/core/src/fix/sdk-agent.ts`, `sdkRunner`, lines 1102–1117 at c64fb88). When the Anthropic API refuses the request (invalid key, empty credit balance, a model the key cannot use), Agent SDK 0.3.296 emits one `result` with `subtype: "success"`, `is_error: true`, `num_turns: 1`, `total_cost_usd: 0` and the API's message in `result`, then throws `Claude Code returned an error result: …`. The runner's `catch` kept the result and dropped the throw (`if (!result) throw err`), and the return only looked at `subtype !== 'success'`. So the task came back as a clean session with no summary: gate `no-change`, every site "SDK agent made no change", and up to 3 feedback rounds that hit the same error. That is the reported signature: `agent` a few seconds, `feedback` a few seconds, `$0.00`. Measured directly against SDK 0.3.296 with a bad key and with a model the key cannot use: both return `{"subtype":"success","is_error":true,"num_turns":1,"total_cost_usd":0}` in about 1.2 s, then throw. **Not a 0.18.2 regression.** Both lines date from 405f429 (2026-10-12). 0.18.1 from npm reproduces the same thing (below). PRs #267 and #268 are not involved. **UpgradeBench riseup-oss/mcp (jest 29 → 30) is a different case.** Uptide found 0 sites, and types, tests and build pass after the bump, so no agent task was planned (`tasks: []`). The ground truth agrees: the bump alone equals the human commit. That $0 was legitimate; the run just never said why. It does now. **Fix** - `sdkRunner` keeps the API's message (`sessionError`). A `stop_reason: refusal` is recorded as `stop: 'refusal'`. - That task's gate is `error`, and whatever the session wrote before the error is reset. Feedback starts no further session after an error. - Safety net, whatever the cause: every task that keeps nothing records why in `llm.tasks[].noChange`. The reasons are the API's error, no turn, no output tokens, a refusal, a stop, every edit rejected (with the gate's reason), or the agent's own `couldNotDo`. - When nothing the agent did stayed, `llm.agentIdle` carries the reason. The terminal and the PR body print "The agent did not change anything: <reason>". The PR body no longer says "The code was migrated" when every site is still manual. ## UPT-135: pnpm lock guard false positive I could not get pnpm 10.34 to produce the reported resolutions on a fresh fixture, so the shape is reproduced at the guard with synthetic lockfiles (`fixtures/lockfiles/onto-upgrade`, pnpm, npm and Yarn Berry). The "same version, different resolution" entries were not peer suffixes. pnpm moved unchanged packages' dependencies (`qs`, `iconv-lite`) onto the newer copies express 5 brings. The copies they left were outside the upgrade's subtree, so each one read as "→ none", and every package depending on them read as "X → X (same version, different resolution)". On c64fb88 the fixture gives exactly that list. A run like this falls back to an agent-led draft. **Fix** (`packages/core/src/fix/managers/lock-guard.ts`, shared by all three managers) - An edge of an unmoved package may move onto the upgrade's copy when it stays in the same major (the same minor below 1.0) and is strictly newer. Importers' own declarations still move only through the plan. - A copy that left is in scope when everything that reached it is: the subtree, an accepted dependent, or another such copy. - Each move is listed as housekeeping: `housekeeping.onto` in report.json, a line under **Lockfile housekeeping** in the PR body, and a count in the terminal. - **npm** had the same false positive (a nested copy dropped for the hoisted one) and is fixed by the same code. - **Yarn Berry** had it too, in a different form: the descriptor `qs@npm:^6.10.0` changes 6.12.0 → 6.16.1 in place. Yarn keys descriptors apart from the upgrade's own `qs@npm:^6.14.0`, so the upgrade's copy is matched by content there, and an in-place descriptor move is accepted under the same rule. npm placements are still matched by key only: a nested npm copy that changes in place is another copy on disk, and stays refused (existing `npm-lock.test.ts` case). - Still refused: a move to a release the upgrade did not bring (the control), a downgrade, a move across a major, a 0.x minor, and anything the repository declares itself. ## Small: group titles The PR body headed each group of sites with its key. That was the rule id (`params-string-array`) or, for sites no pack rule covers, the changed declaration's path. `e.query` is how @types/express names `express.query`. The run now records each site rule's summary (`ruleSummaries`), and groups are titled by it. A declaration path gets the compiler's title at its sites, or "`e.query` was removed in express 5.3.0". Stored reports without summaries keep the id. ## Evidence Case A: riseup-oss/mcp at `0d0e55a` (bench secondary base), read-only clone, remotes removed. Case B: a local express 4.21.2 → 5.3.0 fixture on pnpm, with 6 `req.params`-as-string sites and 1 `express.query()` site (all manual: the express pack has no rewrites), plus `@bull-board/express` 5.23.0, which keeps express 4 as its own dependency. Every run used `--max-cost 0.25`, with the key loaded per command. | Run | Uptide | Key | Result | `llm` | |---|---|---|---|---| | A before | 0.18.2 npm | real | 0 sites, nothing said | tasks `[]`, $0, 0/0 tokens | | A before | 0.18.1 npm | real | same | tasks `[]`, $0 | | A after | this branch | real | PR body: "The agent did not change anything: nothing was left for it: no site needed it, and the type-check, tests and build had no new failure to repair." | tasks `[]`, $0 | | B before | 0.18.2 npm | rejected by the API | 7 manual, "SDK agent made no change", 4 sessions (site task + 3 feedback rounds) | each: turns 1, 0/0 tokens, $0, gate `no-change`, no stop, no error | | B before | 0.18.1 npm | rejected by the API | identical | identical | | B after | this branch | rejected by the API | 7 manual; terminal and PR body: "The agent did not change anything: the session failed: Invalid API key · Fix external API key"; 1 session, no feedback round | turns 1, 0/0 tokens, $0, gate `error`, `error` and `noChange` set | | B | 0.18.2 npm | real | 7/7 fixed by the agent | turns 14, 2,354 output tokens, $0.0605, gate `accepted` | | B after | this branch | real | 7/7 fixed by the agent, no `agentIdle` | turns 14, 2,508 output tokens, $0.0636, 22.0 s, gate `accepted` | With a working key, case B runs fine on 0.18.2. Its before-state ($0, 0 fixed) needs the API to refuse the request, which is the failure this fixes. Case B on pnpm 10.34 wrote no out-of-scope entries on either version; the reported out-of-scope list is reproduced at the guard (above). Spend: $0.12 in total ($0.0605 + $0.0636); every other run and probe was $0. ## Checks (run locally the way CI runs them, API keys unset) ```text pnpm install --frozen-lockfile --ignore-scripts ok pnpm lint Checked 952 files. No fixes applied. 2 warnings, 1 info (all in files this PR does not touch) node scripts/private-material.mjs 1513 tracked files, 1 private identifiers: clean (EVENT=pull_request, dummy UPTIDE_PRIVATE_DENYLIST) pnpm build Tasks: 2 successful, 2 total pnpm typecheck Tasks: 3 successful, 3 total pnpm test @uptide/core Test Files 203 passed | 4 skipped (207) Tests 2487 passed | 37 skipped (2524) pnpm test uptide (cli) Test Files 48 passed (48) Tests 539 passed (539) pnpm docs:packs --check / pnpm docs:links ok; 84 Markdown files, every relative link resolves ``` Under `turbo run test` on my machine, 6 core tests (npmrc, list) and 5 CLI tests (failures, smoke) fail on this branch and on a clean c64fb88 alike. Turbo gives the tests `TMPDIR=/private/tmp`, and a stray `/private/tmp/package.json` left by an earlier bench run makes `/tmp` look like a project root. The totals above are each package's suite run directly. CI's runner has neither. Mutation checks: 12/12 mutations killed for UPT-134, 14/14 for UPT-135 (after two tests were added for the two that first survived), and 5/5 for the titles. ## Open questions - The tester's actual API error is unknown. With this build their run will print it. Worth asking them to re-run, or to send `llm.tasks` from the 0.18.2 report.json if they still have it. - UPT-135 accepts a dependency moving forward onto the upgrade's copy within a major. That is a real resolution change for an unchanged package (another package's `qs` moves forward), only listed, not refused. Is that the line you want, or should it stay refused outside agent-led runs? - `uptide fix` on a checkout without `node_modules` fails with "express has no upgrade to 5.3.0" (`fix/run.ts:496`): the check finds no installed version. That is separate from this PR and is not fixed here. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Lucas Braum <lucasbraum@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits, in order: UPT-128 (the price table), then UPT-127 (auditable SDK cost).
UPT-128: price table
Source: https://platform.claude.com/docs/en/about-claude/pricing, read 2026-10-20 (cited at the top of the Anthropic table in
pricing.ts).There is one change: Sonnet 5.5 cache reads.
docs/model-pricing.mdis updated to match.pricing.test.tspins every Anthropic cell to a single literal expected table (toEqual). Setting Sonnet 5.5 back to 0.20 fails it. OpenAI and Gemini are not re-checked and not pinned (out of scope).sdkCostEstimate(UPT-94) does not move. It prices uncached token equivalents only (cachedInputTokens: 0), so the cache-read rate never enters it.cost-estimate.test.tspasses unmodified before and after (recorded runs: $0.23 for alanagoyal; $0.14 and $0.27 for the unit cases; all four within 1.3–2.5× of actual).SDK_ESTIMATE_MARGINandcost-runs.jsonare unchanged. TheactualUsdvalues there are the SDK's own totals with no token breakdown, so they cannot be repriced.UPT-127: auditable SDK cost
Where tokens were merged:
packages/core/src/fix/sdk-agent.ts:1058-1062(on main) setinputTokens = input_tokens + cache_read_input_tokens + cache_creation_input_tokensandcostUsd = result.total_cost_usd. The patch path did the same atpackages/core/src/llm/fixer.ts:217(billedInput), on top ofadapters.ts:89, which already adds cache reads to input.sdkSpend(result, model)readsresult.modelUsage, which the SDK documents as the field for token and cost accounting (it covers subagents and compaction, whichusagedoes not). For each model it takes uncached input, output, cache reads and cache writes.modelUsagehas none.result.usage.cache_creationhas one, but for the main loop only. Its 1-hour writes are assigned to the session model; any other model's writes are priced as 5-minute writes.costUsd= tokens ×pricing.tsrate, per class and model, rounded up to the nanodollar.sdkReportedCostUsdkeeps the SDK'stotal_cost_usdseparate, and each row also carries its own SDK figure.report.jsonand--jsongainllm.models[]:inputTokens,cacheReadTokens,cacheWriteTokens,cacheWriteHourTokens,outputTokens,rates,costUsd,sdkReportedCostUsd. The same fields appear on eachllm.tasks[]record.costUsdandinputTokenskeep their names.inputTokensnow means uncached input only, documented indocs/model-pricing.mdand the types.--max-costcountsmax(computed, SDK-reported)(chargedUsd) for the feedback-round budget, the typical-session check and the Action's per-run budget.feedback.test.ts"counts the higher…" runs computed $0.05 / SDK $0.10 and the reverse. Both stop exactly where $0.10 sessions do. MakingchargedUsdignore the SDK figure fails the test.llm.warningswith both numbers, printed to stderr. It is never in the PR body:notesfeed the PR,warningsdo not.ratesand is markedunverified: true,costUsdis the SDK's figure,llm.costUnverifiedis set, and the run warns. If there is nomodelUsageat all, the SDK total is used, also marked unverified. Tested insdk-cost.test.ts.assisted.tsandrun.tsadd rows up by model (addSpend).schema/uptide.run.v1.jsonis untouched: the run report copies named fields only, and itscostUsdis now the computed figure.Real run: hand calculation
uptide fix --only zod --max-cost 0.25 --jsonon a local clone of publicconsetto/calmcpatbdaef39^(zod ^3.24.1 → 4.6.7). Origin was removed from the clone, and nothing was pushed or commented. Fix ran in its temporary clone and verification passed.claude-sonnet-5-5, one SDK session, 31 turns, no feedback rounds. Preflight estimate $0.15 for 10 sites.Computed $0.1622007 = SDK-reported $0.1622007, a 0% gap. At the old $0.20 cache-read rate the computed figure would have been $0.1878409, 15.8% above the SDK. So the SDK does use Sonnet 5.5's $0.10 rate, and the UPT-128 fix is right. These numbers are pinned in
sdk-cost.test.ts, with the arithmetic in the comment.Verification
pnpm test: core 2460 passed / 37 skipped (205 files), cli 538 passed (48 files)pnpm typecheck: 3/3 tasks successfulpnpm exec biome check .: no errors; the 2 warnings and 1 info are the same as on mainFollow-up commit (66f17f1)
CI's "No private material" step failed both test jobs.
.changeset/auditable-sdk-cost.mdmatchedPRIVATE_DOC(audit[^/]*\.md) inscripts/public-tree.mjs:41, so it is renamedsdk-cost-per-model.md. Locally that step exits early withoutUPTIDE_PRIVATE_DENYLIST, so it never ran. Run with a dummy denylist, it now reportsclean. The same commit drops the unverified OpenAI/Gemini snapshot and states the 5-minute assumption for non-session-model cache writes indocs/model-pricing.md.Open questions (answered)
maxBudgetUsdinside a session with its own table. If that table ever undercharged, one session could pass the cap by Uptide's figures.max()only protects what comes after the session. Should the session's budget be scaled by the observed computed/SDK ratio?🤖 Generated with Claude Code