U3: Cooldown reduce 4x#8
Open
jaturapornchai wants to merge 1 commit into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
Reduces the exponential cooldown used for model fail-streak handling to recover the pool faster and mitigate 503 cascades.
Changes:
- Updated
computeCooldownMsbase step from 30s → 10s (4x faster growth). - Reduced cooldown cap from 8 minutes → 2 minutes.
- Updated the function’s explanatory comment to reflect the new intent/schedule (but currently contains an inconsistency).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| * คำนวณ cooldown duration ตาม streak | ||
| * 1 → 30s, 2 → 1min, 3 → 2min, 4 → 4min, 5+ → 8min (cap ไม่เกิน 8 นาที) | ||
| * คำนวณ cooldown duration ตาม streak (aggressive recovery — 4x faster) | ||
| * 1 → 10s, 2 → 20s, 3 → 40s, 4 → 1min, 5+ → 2min cap |
There was a problem hiding this comment.
Docstring cooldown schedule is inconsistent with the implementation and PR description: it says streak 4 → 1min, but the formula returns 80_000ms (80s) for streakCount=4 (and the inline comment on line 23 already reflects 80s). Please update the docstring schedule to match the actual values (e.g., 4 → 80s, 5+ → 2min cap).
Suggested change
| * 1 → 10s, 2 → 20s, 3 → 40s, 4 → 1min, 5+ → 2min cap | |
| * 1 → 10s, 2 → 20s, 3 → 40s, 4 → 80s, 5+ → 2min cap |
chatre7
pushed a commit
to chatre7/bcproxyai
that referenced
this pull request
Apr 17, 2026
- Scanner: fetch models from Ollama /api/tags, estimate context from model size - Gateway: route ollama/model-name directly to local Ollama - Silently skip if Ollama not running (no error spam) - Docker: uses host.docker.internal:11434 to reach host Ollama - Provider color: lime green 💻 - Fix Response body disturbed error in buildProxiedResponse 8 providers: OpenRouter, Kilo, Google, Groq, Cerebras, SambaNova, Mistral, Ollama Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jaturapornchai
added a commit
that referenced
this pull request
May 3, 2026
…ssion, latency, RPM, breaker, fallback) Tackled the audit list end-to-end. Two of the requested ten were already implemented in route.ts (empty-body retry on both proxied + relaxed-retry paths), one was deferred (undici dispatcher re-enable — Node 20 already uses undici keep-alive and the previous nvidia-hang bug has no public fix record), one was scoped out (Thai exam dataset is a separate task). Speed - #1 Client-side RPM throttle (rate-budget.ts) — Redis sliding-window per (provider, model). Skip locally before burning the upstream 429 budget, which is what triggers minutes-long cooldowns. Per-provider RPM defaults in free-model-catalog map documented free-tier limits (Groq 30, SEA-LION 10, Typhoon 200, etc.). Fail-open on Redis outage. - #2 Latency-aware sort — added SambaNova to FAST_STREAM_PROVIDERS, plus a PROVIDER_LATENCY_HINT_MS table so providers with no production data yet rank by their documented first-token latency instead of being pinned at 9999999 (which previously stranded new providers at the back forever). Reliability - #4 Fallback chain — provider-diversity interleave in reorderForLatency: the first 6 candidates round-robin across providers, so a single-provider outage no longer stalls the whole chain. Tail keeps strict latency order. - #5 Circuit breaker (3-state) — acquireCircuitProbe/releaseCircuitProbe in learning.ts adds explicit half-open: when fail_streak>=2 and cooldown has expired, only HALF_OPEN_PROBE_LIMIT concurrent probes are admitted. API exported; wiring into chat/completions left for the fallback-chain refactor follow-up. - #6 Deprecation watcher — deprecatedAfter field on FreeModelCatalogEntry. getActiveFreeModelCatalog filters EOL models from /v1/models, scanner, search, and isHardcodedFreeModel. getModelsDeprecatingSoon emits warn logs from the worker scan cycle for anything within 7 days of EOL. Cerebras llama-3.3-70b (2026-02-16) and qwen-3-235b-a22b-instruct-2507 (2026-05-27) tagged. Quality - #8 Tool-call argument JSON validation — validateToolCallArguments in tool-call-repair.ts. isResponseBad now rejects responses where any tool_call argument string fails JSON.parse or isn't a JSON object; caller falls back to next provider instead of returning malformed tool_calls to clients. - #10 Regression warning — checkRegressionWarning() in learning.ts compares last-1h success rate vs prior-23h baseline per modelId; logs REGRESSION warning when drop >=20pp (with min sample sizes both sides). 10-min cooldown per modelId prevents log spam. Hooked into recordOutcomeLearning fire-and-forget on non-quota fails. Tests 95/95, build 0/0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
computeCooldownMsfor faster pool recoveryTest plan
rtk npx next build— Errors: 0 | Warnings: 0