Repository navigation
fix(fix): calibrate SDK cost estimates from recorded runs (UPT-94) - #251
Merged
Merged
Conversation
Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
Collaborator
Author
|
Review complete. No blocking issues — approved ✅; 1 nitpick below. 🧹 Nitpicks (1) — 🟢 1 low
This PR introduces a pre-run LLM cost estimator for
Reviewed commit: cd84e4f |
Merged
lucasbraum
added a commit
that referenced
this pull request
Oct 19, 2026
…253) Stripe's clover request API changes omitted billing mode to flexible without a compiler error. This adds cited behavior records, API-specific preserving edits, and decisions in the existing report sections. Coverage and behavior: - Cover all six APIs in the pinned changelog: subscriptions, invoice previews, subscription schedules, quotes, subscription-mode Checkout, and Payment Links. Exact paths are root `billing_mode` for subscriptions/schedules, `subscription_data.billing_mode` for Checkout/quotes, and `subscription_details.billing_mode` or `schedule_details.billing_mode` for invoice previews. Existing-subscription schedules/previews are excluded. Unknown quote recurrence and Payment Links' undocumented preserving parameter become cited decisions. - Match the effective request API version. Retained pre-clover pins produce no finding. Selected pin migrations add creation sites to the agent plan; post-edit verification checks the actual pin and classic value. Raw compiler findings are classified with the existing rule classifier. Payload `api_version` values never move clients. SDK 12/13 upgrades are supported; the record's SDK range is a candidate filter, not the behavior boundary. - Remove full Stripe pages and `.gitattributes`. Records keep quoted lines, URLs, read dates, and full-page SHA256 hashes. `build:stripe-changelog` fetches, verifies hashes and quotes, fails on drift requiring author review, and writes snapshots only to a gitignored cache. The zod snapshot contract is retained. Independent dms-saas audit of parent `6390f434622db5d6e546854b435fb1017c3cc665`, checked against human PR #70: - Existing expected sites remain `packages/dms-saas/src/workspaces/provisioning.ts:355` and `packages/dms-saas/src/stripe/workspace-checkout.ts:211` (the parameter builder shared by the initial/retry wrapper). - Additional calls: `src/stripe/plan-schedule.ts:52` uses `from_subscription`; `src/stripe/upcoming-invoice.ts:107` previews an existing `subscription`. Neither creates a subscription with the new default. No quotes or Payment Links calls exist. No ground-truth entries were added or changed based on Uptide output. Full rationale and source references are in `packages/core/src/packs/stripe/RESEARCH.md`. Validation: - Rebased onto **#251 / UPT-94** (`773dbbe`). A regression test passes real quoted Stripe findings through the pack transform and `sdkCostEstimate`: they remain manual sites under `stripe/billing-mode-default-flexible`, add their own rule cost, and repeated occurrences count as separate sites while sharing that rule cost. No production cost-estimator changes were needed. - Post-rebase validation: **790 core tests passed / 14 skipped**, including all fix tests, behavior records, Stripe triggers and cost estimates; TypeScript and formatting checks pass. The first broader invocation omitted the package config and was superseded by this successful configured run. - After the rebase, the dms-saas parent scan still matches exactly the same two sites and preserving edits. Per the conditional rerun request, no additional benchmark run or spend was incurred; the successful fork run below remains the benchmark evidence. - **13/13 verified-pack gates passed; no FP regressions against origin/main.** All breaking and silent FP counts remain zero. Existing behavior-note FPs (Redis 2, TypeScript 8) and auxiliary mechanical tallies (Next 1, React Router 336) match origin/main. No expectations or gate were relaxed. The broader offline command exits 1 only because candidate packs ioredis, joi, openai and @sentry/react lack cached repositories; every verified pack passes. - Stripe standalone pack test passed and recorded verified: breaking **23 TP / 0 FP / 2 FN**; the frozen silent-oracle discrepancy is explained below. - Source fetch/hash/quote build, TypeScript check, build, formatting and diff checks pass. **22 trigger/plan tests**, **150 fix/SDK/record/pack-contract tests**, and **5 benchmark scorer tests** pass. Two initial five-second timeouts under concurrent pack load passed on rerun with a 30-second test timeout. - Uptide-only #169 rerun `4776187e`, commit `f82fca0`: [dms-saas fork PR #16](uptide-bench/AntelopeJS--dms-saas#16), **MERGEABLE**, **2/2 silent sites caught and edited**, **0 silent FP**, **$0.2669476**, **255 seconds**, exit **0**. Both original sites resolve to explicit classic afterward. Independent harness install, typecheck, lint and tests all exit **0**. The isolated clone's pre-existing missing frontend dependencies are resolved by the independent full install. - An earlier validation rerun exposed the raw-finding rule-classification bug and caught 0/2; it was fixed and regression-tested before the successful run. That attempt cost **$0.2888555 / 331 seconds**. Total for the two review reruns: **$0.5558031 / 586 seconds** (setup excluded). Neither attempt changed ground truth. The frozen unkey pack expectations still include two creation sites whose clients retain `2023-10-16`. Correct pin handling suppresses those findings in `check`, so the unchanged silent oracle scores **0 TP / 0 FP / 2 FN**. This discrepancy is disclosed and recorded in verification.json; the ground truth was not adjusted after scoring. During `fix`, selecting the pin migration activates the creation-site findings, as the dms-saas rerun verifies. Changeset: **minor**, for the new runtime-behavior detection capability. Signed-off commits, rebased on origin/main; shared run/report edits remain localized. No upstream third-party writes or CI watching. UPT-113 --------- Signed-off-by: Lucas Braum <lucasbraum@gmail.com>
lucasbraum
pushed a commit
that referenced
this pull request
Oct 19, 2026
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## uptide@0.18.0 ### Minor Changes - [#261](#261) [`93f029e`](93f029e) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Sentry 10 → 11: report every `Sentry.init` whose data collection widens because `dataCollection` is unset, citing the migration guide, and give the agent the guide's exact v10 baseline. After a fix, a site counts as kept only when every category is at least that restrictive, so a partial edit such as a cookie deny list stays open in the PR instead of being reported as kept. The agent now keeps privacy settings at their most restrictive value, or leaves them as a decision, when no cited edit exists. - [#253](#253) [`e48373d`](e48373d) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Find Stripe's clover billing-mode default change at subscription creation sites, across the six affected APIs. Respect request API-version pins, preserve classic billing at documented parameter paths, and report unresolved parameters or semantics as decisions. Verify quoted sources against full-page hashes at pack-build time without vendoring Stripe pages. Score the dms-saas silent billing change in UpgradeBench. ### Patch Changes - [#260](#260) [`8ea1a87`](8ea1a87) Thanks [@lucasbraum](https://github.com/lucasbraum)! - A build that fails the same way before and after the upgrade (same command, status and output; only durations and clock times may differ) no longer fails verification. Builds are now compared to the baseline the same way tests are. The PR says `⚠️ build: failing at baseline, not checked`, and the verdict reads `no new errors`, not `verification passed`. A build that passed at baseline and fails after the upgrade still fails, and so does one that fails differently. Without a pack, a run like this still does not publish, because its build cannot verify the bump. - [#259](#259) [`0db73c8`](0db73c8) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Let the agent add a property the upgraded types made required to a test fixture: only to the object the compiler's TS2739, TS2740 or TS2741 error points at, and only with a literal value. Every other test input change is still rejected. Each such property is listed under "Tests changed: review" in the PR body and in report.json (file, line, property and the value the agent chose). Name a test that asserts a dependency default the upgrade changed (Sentry's `dataCollection` and `attachStacktrace`) under "Decisions for you", instead of leaving it to fail unexplained. - [#251](#251) [`773dbbe`](773dbbe) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Calibrate the SDK pre-run cost estimate from recorded migrations, accounting for shared rule knowledge instead of charging a full session allowance per site. Keep the configured spending limit and per-call budget enforcement unchanged. - [#258](#258) [`3a4898f`](3a4898f) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Discover independent nested projects when fixing a dependency absent from the root project. Run installation, migration and verification in the selected project's directory, preserving its lockfile and package manager. When several projects declare the dependency, name their manifests and require an explicit directory. - [#257](#257) [`7b9ed7f`](7b9ed7f) Thanks [@lucasbraum](https://github.com/lucasbraum)! - Run pinned npm through Corepack when local npm differs. When the pin is unavailable, allow local npm with a compatible v3 lockfile, record the fallback in the report, and reject frozen installs that rewrite the lockfile. Co-authored-by: uptide-release[bot] <338861287+uptide-release[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The SDK pre-run estimate charged a full uncached token allowance for every site even though one session shares rule knowledge. On alanagoyal this predicted $1.08 for 18 sites and spent $0.1336, forcing an unnecessary budget increase.
Estimate SDK sessions from distinct migration rules, a small per-site context allowance, and a session minimum, with an explicit 1.5× safety margin applied before rounding to cents. Keep declaration-only anchors in the site allowance without charging another independent rule. The patch executor's estimate and both executors' hard budget checks remain unchanged.
The original reports and bench metadata agree. #169's round 5 reuses round 4's Uptide runs, so it is not counted again. Mechanical sites were already correctly excluded. The old token allowance implicitly covered retries; it had no explicit feedback multiplier. These runs used zero or one feedback round, not three.
Recorded fixtures and
docs/cost-estimate-calibration.mddocument the evidence, formula, and limits. This is a four-run in-sample calibration, not a guarantee on unseen migrations. The safety margin reduces the risk of turning a free preflight stop into a paid mid-run budget stop. Recorded-pair tests require estimates between 1.3× and 2.5× actual spend. No paid runs were made. Install caching is outside this PR's scope.UPT-14/UPT-111 behavior is preserved: preflight rejects estimates above the limit, equality is allowed, Action/App runs work within the original limit, feedback spends only the remainder, and flake reruns and per-call reservations are unchanged. Tests cover recorded pairs, mechanical exclusion, anchors, model pricing, and SDK preflight/runtime budget boundaries.
Validation passed:
pnpm lint(existing Biome schema notice and fixture warning),pnpm typecheck, and the estimator and generic fix tests. Tests also verify that the margin is applied before cent rounding.Refs UPT-94 (cost-estimate part only).