fix(stella-model): retry an afford-hinted OpenRouter 402 at a reduced max_tokens; refuse underfunded arenabench cloud runs - #3309
Open
macanderson wants to merge 2 commits into
Open
fix(stella-model): retry an afford-hinted OpenRouter 402 at a reduced max_tokens; refuse underfunded arenabench cloud runs#3309macanderson wants to merge 2 commits into
macanderson wants to merge 2 commits into
Conversation
…duced max_tokens
OpenRouter answers a request whose max_tokens ceiling the credit balance
cannot cover with HTTP 402 naming the ceiling it can still fund ("You
requested up to 128000 tokens, but can only afford 47365"). Treating
that as a terminal billing error aborted whole bench trials the balance
could still have paid for: runs h2h891 (16 trials lost), fivetools5
(30/30), gate89high1 (35/89).
The shared chat-completions adapter now resends such a rejection ONCE
with max_tokens reduced to 90% of the hint, when the hint clears an
8192-token floor — the same one-shot resend shape as the
mandatory-reasoning recovery, gated on actually addressing OpenRouter
(#1285's endpoint-not-identity lesson). A hint-less 402, a below-floor
hint, and a second 402 all abort exactly as before.
Declared as the parity matrix's fifth axis (BillingPosture):
AffordHintRetry for openrouter with a wiremock witness that fails
without the resend, TerminalAbort with a note everywhere else. The
stream-fallback axis also gains the enforcement the other axes already
had (witness existence, uniqueness, axis coverage, cli completeness),
which it shipped without; the cli completeness tests move to
config/tests/parity.rs so the parent stays under the file-size ratchet.
Three cloud runs were lost whole to an empty OpenRouter balance — h2h891 (16 trials), fivetools5 (30/30), gate89high1 (35/89): the submit path fanned out one Batch job per trial while every model call inside them was going to answer HTTP 402, and each container burned compute to score a 0.0 indistinguishable from an agent loss. cloud run now asks GET https://openrouter.ai/api/v1/credits (shape verified against the live endpoint: data.total_credits - data.total_usage) before anything is uploaded or submitted, and refuses when the remaining balance is under trials x --est-per-trial (default $1.00), naming the balance, the projection, and --skip-balance-check as the override. The check refuses only what it positively knows: a missing OPENROUTER_API_KEY, a network failure, or a shape surprise fails OPEN with a printed warning — a broken preflight must never block a funded run. Decisions live in the new stdlib-only arenabench/balance.py; the one HTTP call is injected through CloudExecutor as the credits seam, the same shape as ls_remote, so the tests neither read the environment nor reach the network. cloud.py itself stays under the 1500-line ratchet.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
This was referenced Aug 14, 2026
Contributor
Reviewer's GuideAdds a fifth provider parity axis for billing behavior and implements an OpenRouter-specific afford-hinted 402 retry in the Stella model adapter, while introducing an ArenaBench cloud-run preflight that checks OpenRouter credits and refuses clearly when the projected trial spend exceeds the remaining balance, with supporting tests and docs updates. Sequence diagram for OpenRouter afford-hinted 402 retry in ZaiProvider.complete_innersequenceDiagram
participant Client
participant ZaiProvider
participant OpenRouter
Client->>ZaiProvider: complete_inner(req, observer)
ZaiProvider->>OpenRouter: complete_attempt(req, observer, retry=false)
OpenRouter-->>ZaiProvider: ProviderError::Terminal(detail)
alt rejects_disabled_reasoning(detail) && id == openrouter && req.reasoning == Some(false)
ZaiProvider->>OpenRouter: complete_attempt(req, observer, retry=true)
OpenRouter-->>ZaiProvider: CompletionResult
ZaiProvider-->>Client: CompletionResult
else serves_openrouter() && afford_capped_max_tokens(detail)
ZaiProvider->>ZaiProvider: afford_capped_max_tokens(detail)
ZaiProvider->>ZaiProvider: set reduced.max_output_tokens
ZaiProvider->>OpenRouter: complete_attempt(reduced, observer, retry=false)
OpenRouter-->>ZaiProvider: CompletionResult
ZaiProvider-->>Client: CompletionResult
else
ZaiProvider-->>Client: ProviderError::Terminal(detail)
end
Sequence diagram for ArenaBench cloud run OpenRouter balance preflightsequenceDiagram
actor Operator
participant CloudExecutor
participant balance_preflight as balance.preflight
participant fetch_credits as fetch_openrouter_credits
participant OpenRouter
Operator->>CloudExecutor: _cmd_cloud_run(args)
CloudExecutor->>CloudExecutor: plans = plan_trials(spec)
alt args.skip_balance_check is False
CloudExecutor->>balance_preflight: preflight(trials=len(plans), est_per_trial, api_key, fetch=executor.openrouter_credits)
alt api_key is None
balance_preflight-->>CloudExecutor: None (check skipped)
else fetch_openrouter_credits(api_key) raises BalanceUnknownError
balance_preflight->>CloudExecutor: out("balance : check skipped — ...")
balance_preflight-->>CloudExecutor: None
else remaining_credits < projected_spend
balance_preflight->>fetch_credits: fetch_openrouter_credits(api_key)
fetch_credits->>OpenRouter: GET CREDITS_URL
OpenRouter-->>fetch_credits: JSON credits
fetch_credits-->>balance_preflight: payload
balance_preflight-->>CloudExecutor: refusal_message
CloudExecutor->>Operator: print("error: " + refusal_message)
CloudExecutor-->>Operator: return 2
else remaining_credits >= projected_spend
balance_preflight-->>CloudExecutor: None (balance covers projection)
end
end
opt submission proceeds
CloudExecutor->>CloudExecutor: resolve_sut, upload, submit
end
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Three paid bench runs were killed by the same mechanism: OpenRouter answers a request whose
max_tokensceiling the credit balance cannot cover with HTTP 402 whose body literally says "You requested up to 128000 tokens, but can only afford 47365" — and Stella classified every 402 as a terminal provider error and aborted the trial, while ArenaBench happily submitted full grids against the empty balance.Evidence — the three lost runs:
h2h891(16 trials lost),fivetools5(30/30 lost),gate89high1(35/89 lost). This PR is the record for the fix; no pre-existing issue tracks it, so it closes nothing.Two fixes, one per commit:
A — engine-side graceful degrade (
crates/stella-model). The shared chat-completions adapter (zai.rs::complete_inner) now resends an OpenRouter 402 carrying the afford hint once, withmax_tokensreduced to 90% of the hinted ceiling, when the hint clears an 8192-token floor (below that a turn buys a truncation, not an answer — thefinish_reason: lengthshape from the 2026-07-31 bundle). A hint-less 402, a below-floor hint, and a second 402 abort exactly as before. Gated on actually addressing OpenRouter (serves_openrouter(), the #1285 endpoint-not-identity lesson); the hint is parsed defensively from the gateway's own prose (afford_hint_tokens— no hint / no digits / overflow all mean "no retry").Declared as the parity matrix's fifth axis,
BillingPosture(provider_parity.rs):AffordHintRetryforopenrouterwith a named witness,TerminalAbortwith a reviewable note for the other nine ids — behavior for every other provider is unchanged.Exemplars followed (per CLAUDE.md): the one-shot resend copies the crate's own
rejects_disabled_reasoningmandatory-reasoning recovery shape incomplete_inner, and the new axis copiesOverflowPosture's recognize-a-wire-signal-and-recover shape, enforcement tests included.B — ArenaBench preflight (
arenabench/).arenabench cloud runnow queriesGET https://openrouter.ai/api/v1/credits(endpoint shape verified against the live API on 2026-08-14:{"data":{"total_credits":1650,"total_usage":1606.027034237}}; remaining =total_credits - total_usage) before anything is uploaded, resolved, or submitted, and refuses when remaining < trials ×--est-per-trial(new flag, default $1.00), naming the balance, the projection, and the--skip-balance-checkoverride in the refusal. It fails OPEN — missingOPENROUTER_API_KEY, network error, HTTP error, or shape surprise each print a "check skipped" warning and proceed, because a broken preflight must not block a funded run. Decisions live in the new stdlib-onlyarenabench/arenabench/balance.py; the one HTTP call rides an injectablecreditsseam onCloudExecutor, same shape asls_remote(cloud.pystays under the 1500-line ratchet: 1487).The witness
main, pass here)Both flips were demonstrated by running the tests against the unwired code first:
A —
complete_retries_an_afford_hinted_402_with_reduced_max_tokens(zai/tests/error_classify.rs, wiremock): first response 402 with the afford hint; the success mock matches only a body carrying"max_tokens":42628(90% of 47365). Before wiring:FAILED — panicked: the afford-hinted 402 must be retried at the reduced ceiling: Terminal("OpenRouter rejected the request (HTTP 402: payment required): … can only afford 47365 …"). After wiring:ok. The other side is pinned twice: the pre-existing hint-less-402 test now.expect(1)s its mock (no retry without a hint), anda_402_afford_hint_below_the_floor_stays_terminalproves the floor.B —
TestBalancePreflight(tests/test_cloud.py): before wiring the verb,test_an_underfunded_balance_refuses_before_any_upload_or_submitfailed with the transcript showing the run submitting normally (submitted : 1 job(s)); after wiring it refuses with rc 2, zero S3 calls, zero submissions. Sufficient balance submits; acreditsseam raisingOSError("connection refused")proceeds withbalance : check skipped — … connection refused …;--skip-balance-checkprovably never queries the seam. Pure halves (parsing, refusal boundary, fail-open ladder) pinned in the newtests/test_balance.py.The gate
cargo fmt --check(viamake guards-fast, all guards green — file-size, god-files, left-behind, invariants ratchets included)cargo clippy -p stella-model -p stella-cli --all-targets -- -D warningscargo test -p stella-model(404 passed) andcargo test -p stella-cli(full suite, 0 failed) — workspace-wide run left to CI; no other crate is touchedRUSTDOCFLAGS="-D warnings" cargo doc -p stella-model --no-deps(exit 0)uv run --with pytest --no-project pytestthe way CI does — 1021 passed, 3 skipped, 5 failed — all five pre-existing intests/test_sut.py, byte-identical with this PR's changes stashed (host/usr/bin/python3lackstomllib; tracked in arenabench: five test_sut.py tests fail on any stock macOS host — the suite probes the real /usr/bin/python3, which lacks tomllib #3282/arenabench: test_sut.py depends on the host's /usr/bin/python3 — five tests red on stock macOS, green in CI #3301, green in CI)cloud.pylines, noted on arenabench and bench/ configure ruff but no workflow runs it #3055BillingPosturebullet),crates/stella-model/README.md(axis list — which had already droppedOverflowPosture— plus two stale test names),--helptext for the two new flagsNothing left behind
cloud.pyfor when ruff gets a workflow) and arenabench: test_sut.py depends on the host's /usr/bin/python3 — five tests red on stock macOS, green in CI #3301 (apparent duplicate of arenabench: five test_sut.py tests fail on any stock macOS host — the suite probes the real /usr/bin/python3, which lacks tomllib #3282)all_axes_cover_the_same_provider_ids, nostella-clicompleteness test. Since this PR extends exactly those test functions for the new axis, it completes the fourth axis's enforcement in the same motion. Thestella-clicompleteness tests (all five axes) move toconfig/tests/parity.rs— a straight move, no test deleted — because adding two toconfig/tests.rspushed it over the 1500-line ratchet.Ground-rule check
stella-core; no new deps (Rust: none; Python: stdliburllibonly, keeping ArenaBench's stdlib-only contract)Anything reviewers should know?
"can only afford N") in the classified error detail, the same wayrejects_disabled_reasoningmatches upstream wording — there is no machine field for the hint on the wire. The wiremock witness exercises the full classify → parse → resend path, so a rewording of either side fails a test rather than silently disarming the recovery.resolve_sut, so an underfunded run also never triggers a CodeBuild build.ProviderErrorvariant — that would put billing policy instella-protocoland re-issue the same ceiling on retry, which re-rejects identically. The adapter owns the recovery because only it can rewritemax_tokens.Summary by Sourcery
Handle OpenRouter-specific credit exhaustion more gracefully in both the Stella engine and ArenaBench cloud runs, and extend the provider parity matrix and its tests with a new billing axis and full enforcement for all axes.
New Features:
arenabench cloud runthat estimates run cost against current credits, refusing clearly when underfunded and offering a skip flag.Bug Fixes:
Enhancements:
Documentation:
Tests: