Problem
PR #TBD (fix-402-resilience) taught the shared chat-completions adapter to retry an OpenRouter HTTP 402 that names the affordable output ceiling ("You requested up to 128000 tokens, but can only afford 47365") once, with max_tokens reduced to 90% of the hint. The recovery is deliberately per-request: nothing is remembered between turns.
Consequence: while the balance stays low, every subsequent turn re-sends the original ceiling, eats one 402 round-trip, and only then retries reduced. On a bench trial with dozens of turns that is dozens of wasted request/reject cycles (latency, not spend — the 402 bills nothing), and each one risks the gateway's rate limiting.
Where
Sketch
A session-scoped cell (interior-mutable, like StreamRecovery) remembering the last afford-derived ceiling; build_body clamps max_tokens to it when set. It must decay or re-raise: a top-up mid-session should not leave the session capped forever (e.g. clear the latch after N successful turns, or re-try the caller's full ceiling once every M turns).
Constraints discovered
- Parity: the behavior is declared as
BillingPosture::AffordHintRetry in crates/stella-model/src/provider_parity.rs; a latch changes the mechanism wording and needs its witness extended (two turns: second turn must go out already-reduced without an intervening 402).
- The floor (8192) and the 90% margin live beside
afford_capped_max_tokens; keep one copy.
- Byte-stability: the latch must not perturb requests when it never armed (prompt-cache contract, AGENTS.md invariant 7).
Done
A wiremock witness showing: turn 1 → 402(hint) → reduced retry succeeds; turn 2 on the same provider instance goes out with the reduced ceiling on its first attempt. Fails on current main.
Problem
PR #TBD (fix-402-resilience) taught the shared chat-completions adapter to retry an OpenRouter HTTP 402 that names the affordable output ceiling ("You requested up to 128000 tokens, but can only afford 47365") once, with
max_tokensreduced to 90% of the hint. The recovery is deliberately per-request: nothing is remembered between turns.Consequence: while the balance stays low, every subsequent turn re-sends the original ceiling, eats one 402 round-trip, and only then retries reduced. On a bench trial with dozens of turns that is dozens of wasted request/reject cycles (latency, not spend — the 402 bills nothing), and each one risks the gateway's rate limiting.
Where
crates/stella-model/src/zai.rs—complete_inner(the one-shot resend),afford_capped_max_tokens,AFFORD_RETRY_FLOOR_TOKENS.crates/stella-model/src/stream_recovery.rs(StreamRecovery, IMPROVEMENT: streaming→non-streaming fallback on hung or empty streams, with a separate first-byte deadline #2686): armed on a fault, consulted per attempt, bounded so it cannot wedge a session permanently.Sketch
A session-scoped cell (interior-mutable, like
StreamRecovery) remembering the last afford-derived ceiling;build_bodyclampsmax_tokensto it when set. It must decay or re-raise: a top-up mid-session should not leave the session capped forever (e.g. clear the latch after N successful turns, or re-try the caller's full ceiling once every M turns).Constraints discovered
BillingPosture::AffordHintRetryincrates/stella-model/src/provider_parity.rs; a latch changes the mechanism wording and needs its witness extended (two turns: second turn must go out already-reduced without an intervening 402).afford_capped_max_tokens; keep one copy.Done
A wiremock witness showing: turn 1 → 402(hint) → reduced retry succeeds; turn 2 on the same provider instance goes out with the reduced ceiling on its first attempt. Fails on current main.