Skip to content

fix(fix): calibrate SDK cost estimates from recorded runs (UPT-94) - #251

Merged
lucasbraum merged 2 commits into
mainfrom
fix/upt-94-cost-estimate
Oct 19, 2026
Merged

lucasbraum merged 2 commits into
mainfrom
fix/upt-94-cost-estimate

Conversation

@lucasbraum

@lucasbraum lucasbraum commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

The SDK pre-run estimate charged a full uncached token allowance for every site even though one session shares rule knowledge. On alanagoyal this predicted $1.08 for 18 sites and spent $0.1336, forcing an unnecessary budget increase.

Estimate SDK sessions from distinct migration rules, a small per-site context allowance, and a session minimum, with an explicit 1.5× safety margin applied before rounding to cents. Keep declaration-only anchors in the site allowance without charging another independent rule. The patch executor's estimate and both executors' hard budget checks remain unchanged.

Recorded run Before Actual After After / actual
alanagoyal, React 18 → 19 $1.08 $0.1336105 $0.23 1.72×
calmcp, pilot round 4 $0.30 $0.0842459 $0.14 1.66×
dms-saas, pilot round 4 $0.60 $0.2740438 $0.42 1.53×
enki, pilot round 4 $0.54 $0.1431526 $0.29 2.03×

The original reports and bench metadata agree. #169's round 5 reuses round 4's Uptide runs, so it is not counted again. Mechanical sites were already correctly excluded. The old token allowance implicitly covered retries; it had no explicit feedback multiplier. These runs used zero or one feedback round, not three.

Recorded fixtures and docs/cost-estimate-calibration.md document the evidence, formula, and limits. This is a four-run in-sample calibration, not a guarantee on unseen migrations. The safety margin reduces the risk of turning a free preflight stop into a paid mid-run budget stop. Recorded-pair tests require estimates between 1.3× and 2.5× actual spend. No paid runs were made. Install caching is outside this PR's scope.

UPT-14/UPT-111 behavior is preserved: preflight rejects estimates above the limit, equality is allowed, Action/App runs work within the original limit, feedback spends only the remainder, and flake reruns and per-call reservations are unchanged. Tests cover recorded pairs, mechanical exclusion, anchors, model pricing, and SDK preflight/runtime budget boundaries.

Validation passed: pnpm lint (existing Biome schema notice and fixture warning), pnpm typecheck, and the estimator and generic fix tests. Tests also verify that the margin is applied before cent rounding.

Refs UPT-94 (cost-estimate part only).

Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
@lucasbraum

Copy link
Copy Markdown
Collaborator Author

@tenki-reviewer

@tenki-reviewer

tenki-reviewer Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Review complete. No blocking issues — approved ✅; 1 nitpick below.

🧹 Nitpicks (1) — 🟢 1 low
  • 🟢 Doc's calmcp rule count contradicts the fixture (cost-estimate-calibration.md:16) — The doc records 1 distinct rule and a $0.14 estimate for calmcp (docs/cost-estimate-calibration.md:16, :67), but the fixture's calmcp TS2740 site has no rule field, so sdkCostEstimate falls back to the change path and counts 2 rules (packages/core/src/llm/__fixtures__/cost-runs.json:270), yielding $0.15 and a ratio of ~1.78 instead of the documented 1.66.

This PR introduces a pre-run LLM cost estimator for sdk agent runs in packages/core/src/llm/fixer.ts, combining a session floor with per-rule and per-site allowances multiplied by a 1.5× margin, and wires it into fix/run.ts so the estimate is checked against configured spending limits before any agent spend. Calibration is grounded in a new recorded-runs fixture, with docs explaining the derivation and tests asserting estimates stay within a 1.3×–2.5× band of actuals; configured limits are unchanged. One small documentation inconsistency remains around the calmcp rule count (see inline comment).

Files Change
packages/core/src/llm/fixer.ts Adds sdkCostEstimate with session floor, per-rule/per-site allowances, and 1.5× margin.
packages/core/src/fix/run.ts Selects the SDK estimate for sdk agent runs and gates preflight against spending limits.
packages/core/src/llm/__fixtures__/cost-runs.json Adds recorded calibration runs used as the estimator's ground truth.
packages/core/src/llm/cost-estimate.test.ts, packages/core/src/fix/generic.test.ts New tests covering estimate calibration band and gating behavior.
docs/cost-estimate-calibration.md, .changeset/quiet-rules-estimate.md Documents the calibration methodology and adds a changeset.

Reviewed commit: cd84e4f

@lucasbraum
lucasbraum merged commit 773dbbe into main Oct 19, 2026
13 checks passed
@lucasbraum
lucasbraum deleted the fix/upt-94-cost-estimate branch October 19, 2026 05:45
@uptide-release uptide-release Bot mentioned this pull request Oct 19, 2026
lucasbraum added a commit that referenced this pull request Oct 19, 2026
…253)

Stripe's clover request API changes omitted billing mode to flexible
without a compiler error. This adds cited behavior records, API-specific
preserving edits, and decisions in the existing report sections.

Coverage and behavior:

- Cover all six APIs in the pinned changelog: subscriptions, invoice
previews, subscription schedules, quotes, subscription-mode Checkout,
and Payment Links. Exact paths are root `billing_mode` for
subscriptions/schedules, `subscription_data.billing_mode` for
Checkout/quotes, and `subscription_details.billing_mode` or
`schedule_details.billing_mode` for invoice previews.
Existing-subscription schedules/previews are excluded. Unknown quote
recurrence and Payment Links' undocumented preserving parameter become
cited decisions.
- Match the effective request API version. Retained pre-clover pins
produce no finding. Selected pin migrations add creation sites to the
agent plan; post-edit verification checks the actual pin and classic
value. Raw compiler findings are classified with the existing rule
classifier. Payload `api_version` values never move clients. SDK 12/13
upgrades are supported; the record's SDK range is a candidate filter,
not the behavior boundary.
- Remove full Stripe pages and `.gitattributes`. Records keep quoted
lines, URLs, read dates, and full-page SHA256 hashes.
`build:stripe-changelog` fetches, verifies hashes and quotes, fails on
drift requiring author review, and writes snapshots only to a gitignored
cache. The zod snapshot contract is retained.

Independent dms-saas audit of parent
`6390f434622db5d6e546854b435fb1017c3cc665`, checked against human PR
#70:

- Existing expected sites remain
`packages/dms-saas/src/workspaces/provisioning.ts:355` and
`packages/dms-saas/src/stripe/workspace-checkout.ts:211` (the parameter
builder shared by the initial/retry wrapper).
- Additional calls: `src/stripe/plan-schedule.ts:52` uses
`from_subscription`; `src/stripe/upcoming-invoice.ts:107` previews an
existing `subscription`. Neither creates a subscription with the new
default. No quotes or Payment Links calls exist. No ground-truth entries
were added or changed based on Uptide output. Full rationale and source
references are in `packages/core/src/packs/stripe/RESEARCH.md`.

Validation:

- Rebased onto **#251 / UPT-94** (`773dbbe`). A regression test passes
real quoted Stripe findings through the pack transform and
`sdkCostEstimate`: they remain manual sites under
`stripe/billing-mode-default-flexible`, add their own rule cost, and
repeated occurrences count as separate sites while sharing that rule
cost. No production cost-estimator changes were needed.
- Post-rebase validation: **790 core tests passed / 14 skipped**,
including all fix tests, behavior records, Stripe triggers and cost
estimates; TypeScript and formatting checks pass. The first broader
invocation omitted the package config and was superseded by this
successful configured run.
- After the rebase, the dms-saas parent scan still matches exactly the
same two sites and preserving edits. Per the conditional rerun request,
no additional benchmark run or spend was incurred; the successful fork
run below remains the benchmark evidence.

- **13/13 verified-pack gates passed; no FP regressions against
origin/main.** All breaking and silent FP counts remain zero. Existing
behavior-note FPs (Redis 2, TypeScript 8) and auxiliary mechanical
tallies (Next 1, React Router 336) match origin/main. No expectations or
gate were relaxed. The broader offline command exits 1 only because
candidate packs ioredis, joi, openai and @sentry/react lack cached
repositories; every verified pack passes.
- Stripe standalone pack test passed and recorded verified: breaking
**23 TP / 0 FP / 2 FN**; the frozen silent-oracle discrepancy is
explained below.
- Source fetch/hash/quote build, TypeScript check, build, formatting and
diff checks pass. **22 trigger/plan tests**, **150
fix/SDK/record/pack-contract tests**, and **5 benchmark scorer tests**
pass. Two initial five-second timeouts under concurrent pack load passed
on rerun with a 30-second test timeout.
- Uptide-only #169 rerun `4776187e`, commit `f82fca0`: [dms-saas fork PR
#16](uptide-bench/AntelopeJS--dms-saas#16),
**MERGEABLE**, **2/2 silent sites caught and edited**, **0 silent FP**,
**$0.2669476**, **255 seconds**, exit **0**. Both original sites resolve
to explicit classic afterward. Independent harness install, typecheck,
lint and tests all exit **0**. The isolated clone's pre-existing missing
frontend dependencies are resolved by the independent full install.
- An earlier validation rerun exposed the raw-finding
rule-classification bug and caught 0/2; it was fixed and
regression-tested before the successful run. That attempt cost
**$0.2888555 / 331 seconds**. Total for the two review reruns:
**$0.5558031 / 586 seconds** (setup excluded). Neither attempt changed
ground truth.

The frozen unkey pack expectations still include two creation sites
whose clients retain `2023-10-16`. Correct pin handling suppresses those
findings in `check`, so the unchanged silent oracle scores **0 TP / 0 FP
/ 2 FN**. This discrepancy is disclosed and recorded in
verification.json; the ground truth was not adjusted after scoring.
During `fix`, selecting the pin migration activates the creation-site
findings, as the dms-saas rerun verifies.

Changeset: **minor**, for the new runtime-behavior detection capability.
Signed-off commits, rebased on origin/main; shared run/report edits
remain localized. No upstream third-party writes or CI watching.

UPT-113

---------

Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
lucasbraum pushed a commit that referenced this pull request Oct 19, 2026
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.


# Releases
## uptide@0.18.0

### Minor Changes

- [#261](#261)
[`93f029e`](93f029e)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Sentry 10 → 11:
report every `Sentry.init` whose data collection widens because
`dataCollection` is unset, citing the migration guide, and give the
agent the guide's exact v10 baseline. After a fix, a site counts as kept
only when every category is at least that restrictive, so a partial edit
such as a cookie deny list stays open in the PR instead of being
reported as kept. The agent now keeps privacy settings at their most
restrictive value, or leaves them as a decision, when no cited edit
exists.

- [#253](#253)
[`e48373d`](e48373d)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Find Stripe's
clover billing-mode default change at subscription creation sites,
across the six affected APIs. Respect request API-version pins, preserve
classic billing at documented parameter paths, and report unresolved
parameters or semantics as decisions. Verify quoted sources against
full-page hashes at pack-build time without vendoring Stripe pages.
Score the dms-saas silent billing change in UpgradeBench.

### Patch Changes

- [#260](#260)
[`8ea1a87`](8ea1a87)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - A build that
fails the same way before and after the upgrade (same command, status
and output; only durations and clock times may differ) no longer fails
verification. Builds are now compared to the baseline the same way tests
are. The PR says `⚠️ build: failing at baseline, not checked`, and the
verdict reads `no new errors`, not `verification passed`. A build that
passed at baseline and fails after the upgrade still fails, and so does
one that fails differently. Without a pack, a run like this still does
not publish, because its build cannot verify the bump.

- [#259](#259)
[`0db73c8`](0db73c8)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Let the agent add
a property the upgraded types made required to a test fixture: only to
the object the compiler's TS2739, TS2740 or TS2741 error points at, and
only with a literal value. Every other test input change is still
rejected. Each such property is listed under "Tests changed: review" in
the PR body and in report.json (file, line, property and the value the
agent chose). Name a test that asserts a dependency default the upgrade
changed (Sentry's `dataCollection` and `attachStacktrace`) under
"Decisions for you", instead of leaving it to fail unexplained.

- [#251](#251)
[`773dbbe`](773dbbe)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Calibrate the SDK
pre-run cost estimate from recorded migrations, accounting for shared
rule knowledge instead of charging a full session allowance per site.
Keep the configured spending limit and per-call budget enforcement
unchanged.

- [#258](#258)
[`3a4898f`](3a4898f)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Discover
independent nested projects when fixing a dependency absent from the
root project. Run installation, migration and verification in the
selected project's directory, preserving its lockfile and package
manager. When several projects declare the dependency, name their
manifests and require an explicit directory.

- [#257](#257)
[`7b9ed7f`](7b9ed7f)
Thanks [@lucasbraum](https://github.com/lucasbraum)! - Run pinned npm
through Corepack when local npm differs. When the pin is unavailable,
allow local npm with a compatible v3 lockfile, record the fallback in
the report, and reject frozen installs that rewrite the lockfile.

Co-authored-by: uptide-release[bot] <338861287+uptide-release[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant