U2: Routing aggressive overhaul#15
Conversation
…onn pool, circuit recovery) - Hedge top-3 cloud candidates (was top-2), expand window to estTokens<=20K - hedgeRace generalized to N candidates with AbortController per attempt - MAX_RETRIES 10 -> 25; TOTAL_TIMEOUT_MS scaled up (30/60/90/120s) - Per-attempt timeout scaled: 12/20/35/60s (was 8/12/20/30) - Wire upstreamAgent as undici dispatcher on provider fetch - Circuit breaker recovery window 120s -> 30s - Context-aware filter: for >20K tok requests, require 1.5x context headroom Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adjusts the chat completions routing strategy to be more aggressive and resilient under large-context and flaky-upstream conditions by expanding retry/timeout budgets, widening hedging, and using an undici RetryAgent dispatcher for upstream fetches.
Changes:
- Increase retry budget (
MAX_RETRIES) and total retry timeouts, and scale per-attempt timeouts by request size. - Expand hedging from top-2 to top-3 candidates (when eligible) and widen the hedge window to
estTokens <= 20k. - Add a large-context candidate filter (1.5× headroom) and wire
upstreamAgentas the fetch dispatcher; shorten circuit-breaker open window.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (successRate < 0.3) { | ||
| // Open circuit for 2 minutes | ||
| const cbKey = `cb:open:${provider}`; | ||
| await redis.set(cbKey, "open", "EX", 120); | ||
| console.log(`[CIRCUIT-OPEN] ${provider} — success rate ${(successRate * 100).toFixed(0)}% < 30% → circuit open 2min`); | ||
| await redis.set(cbKey, "open", "EX", 30); | ||
| console.log(`[CIRCUIT-OPEN] ${provider} — success rate ${(successRate * 100).toFixed(0)}% < 30% → circuit open 30s`); |
There was a problem hiding this comment.
Circuit breaker open duration was reduced to 30s, but isCircuitOpen() promotes open → half-open whenever TTL <= 30s. With EX 30, the TTL is always <= 30, so the circuit will be promoted to half-open immediately and effectively never block requests. Adjust the half-open promotion threshold (e.g., last few seconds) or keep the open TTL > 30s so the open state meaningfully gates traffic. Also update the stale comment that still says "Open circuit for 2 minutes".
Summary
upstreamAgent(undici RetryAgent) as fetch dispatcher for provider callsTest plan
rtk npx next build-> Errors: 0 | Warnings: 0Note: existing test failures in scanner.test.ts are pre-existing (db mock issue) and unrelated to this change.
Generated with Claude Code