fix: surface usage and model capabilities on dashboard#4
Open
PKetsuk wants to merge 1 commit into
Open
Conversation
jaturapornchai
added a commit
that referenced
this pull request
May 3, 2026
…ssion, latency, RPM, breaker, fallback) Tackled the audit list end-to-end. Two of the requested ten were already implemented in route.ts (empty-body retry on both proxied + relaxed-retry paths), one was deferred (undici dispatcher re-enable — Node 20 already uses undici keep-alive and the previous nvidia-hang bug has no public fix record), one was scoped out (Thai exam dataset is a separate task). Speed - #1 Client-side RPM throttle (rate-budget.ts) — Redis sliding-window per (provider, model). Skip locally before burning the upstream 429 budget, which is what triggers minutes-long cooldowns. Per-provider RPM defaults in free-model-catalog map documented free-tier limits (Groq 30, SEA-LION 10, Typhoon 200, etc.). Fail-open on Redis outage. - #2 Latency-aware sort — added SambaNova to FAST_STREAM_PROVIDERS, plus a PROVIDER_LATENCY_HINT_MS table so providers with no production data yet rank by their documented first-token latency instead of being pinned at 9999999 (which previously stranded new providers at the back forever). Reliability - #4 Fallback chain — provider-diversity interleave in reorderForLatency: the first 6 candidates round-robin across providers, so a single-provider outage no longer stalls the whole chain. Tail keeps strict latency order. - #5 Circuit breaker (3-state) — acquireCircuitProbe/releaseCircuitProbe in learning.ts adds explicit half-open: when fail_streak>=2 and cooldown has expired, only HALF_OPEN_PROBE_LIMIT concurrent probes are admitted. API exported; wiring into chat/completions left for the fallback-chain refactor follow-up. - #6 Deprecation watcher — deprecatedAfter field on FreeModelCatalogEntry. getActiveFreeModelCatalog filters EOL models from /v1/models, scanner, search, and isHardcodedFreeModel. getModelsDeprecatingSoon emits warn logs from the worker scan cycle for anything within 7 days of EOL. Cerebras llama-3.3-70b (2026-02-16) and qwen-3-235b-a22b-instruct-2507 (2026-05-27) tagged. Quality - #8 Tool-call argument JSON validation — validateToolCallArguments in tool-call-repair.ts. isResponseBad now rejects responses where any tool_call argument string fails JSON.parse or isn't a JSON object; caller falls back to next provider instead of returning malformed tool_calls to clients. - #10 Regression warning — checkRegressionWarning() in learning.ts compares last-1h success rate vs prior-23h baseline per modelId; logs REGRESSION warning when drop >=20pp (with min sample sizes both sides). 10-min cooldown per modelId prevents log spam. Hooked into recordOutcomeLearning fire-and-forget on non-quota fails. Tests 95/95, build 0/0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jaturapornchai
added a commit
that referenced
this pull request
May 3, 2026
#1 HALF_OPEN_PROBE_TTL_SEC bumped 12 → 65 so the slot outlives the longest in-flight request (chat/completions ladders to 60s for ~50KB bodies). Otherwise the slot expired mid-probe, letting another pod fire a parallel probe while the first was still on the wire — defeating the purpose. #5 Static import of @/lib/redis in learning.ts (was dynamic). Lazy import was a holdover from the unit-test concern, but Vitest already mocks the module — no test impact, and we drop a microtask + module-cache lookup from every recordOutcome on the hot path. #4 GATEWAY_REGRESSION_WEBHOOK env wiring. When set, checkRegressionWarning POSTs a Slack/Discord-compatible JSON payload (`text` + `content` for cross-vendor compatibility) on top of the existing console.warn. Hard 5-second timeout so a stuck webhook can't drag down request latency. No-op when unset — safe to deploy without configuration. Tests 127/127, build 0/0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
/api/chatdid not write any token usage, so using the dashboard chat never populated cost dataValidation
npm run build✅npm test[DeepSeek]prefix expectation mismatch)/api/statusnow starts the worker,/api/chatwritestoken_usage,/api/modelsincludessupportsTools/supportsVision