feat: Pattern B+ tool call retry with JSON repair#2
Open
mangsriso wants to merge 3 commits into
Open
Conversation
…nitization - Add jsonrepair for malformed tool_call arguments - Add retryWithToolsModel with 10s AbortController timeout - Enhance buildProxiedResponse with jsonrepair fallback - Strip reasoning/refusal/store fields from requests - Google message sanitizer for tool role messages Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Port Pattern B+ from BCProxyAI (SQLite) to SML Gateway (async PostgreSQL): - Add jsonrepair dependency for malformed tool_call argument repair - Add src/lib/tool-repair.ts: repairToolCallArguments() + hasStructurallyBrokenToolCalls() - Wire repair into non-streaming response handler (sequential retry path) - Wire repair into buildProxiedResponse (final response path) - Add retryWithToolsModel() — async, respects exam system model selection - Set X-SMLGateway-ToolRepair header when retry is used Pattern B+ flow: 1. When tool_calls present, attempt jsonrepair on malformed arguments 2. Check for structural breaks (missing name, null args, oversized args) 3. If broken, retry with next tools-capable model (15s timeout) 4. Return repaired/retried response or best-effort original Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jaturapornchai
added a commit
that referenced
this pull request
May 3, 2026
…ssion, latency, RPM, breaker, fallback) Tackled the audit list end-to-end. Two of the requested ten were already implemented in route.ts (empty-body retry on both proxied + relaxed-retry paths), one was deferred (undici dispatcher re-enable — Node 20 already uses undici keep-alive and the previous nvidia-hang bug has no public fix record), one was scoped out (Thai exam dataset is a separate task). Speed - #1 Client-side RPM throttle (rate-budget.ts) — Redis sliding-window per (provider, model). Skip locally before burning the upstream 429 budget, which is what triggers minutes-long cooldowns. Per-provider RPM defaults in free-model-catalog map documented free-tier limits (Groq 30, SEA-LION 10, Typhoon 200, etc.). Fail-open on Redis outage. - #2 Latency-aware sort — added SambaNova to FAST_STREAM_PROVIDERS, plus a PROVIDER_LATENCY_HINT_MS table so providers with no production data yet rank by their documented first-token latency instead of being pinned at 9999999 (which previously stranded new providers at the back forever). Reliability - #4 Fallback chain — provider-diversity interleave in reorderForLatency: the first 6 candidates round-robin across providers, so a single-provider outage no longer stalls the whole chain. Tail keeps strict latency order. - #5 Circuit breaker (3-state) — acquireCircuitProbe/releaseCircuitProbe in learning.ts adds explicit half-open: when fail_streak>=2 and cooldown has expired, only HALF_OPEN_PROBE_LIMIT concurrent probes are admitted. API exported; wiring into chat/completions left for the fallback-chain refactor follow-up. - #6 Deprecation watcher — deprecatedAfter field on FreeModelCatalogEntry. getActiveFreeModelCatalog filters EOL models from /v1/models, scanner, search, and isHardcodedFreeModel. getModelsDeprecatingSoon emits warn logs from the worker scan cycle for anything within 7 days of EOL. Cerebras llama-3.3-70b (2026-02-16) and qwen-3-235b-a22b-instruct-2507 (2026-05-27) tagged. Quality - #8 Tool-call argument JSON validation — validateToolCallArguments in tool-call-repair.ts. isResponseBad now rejects responses where any tool_call argument string fails JSON.parse or isn't a JSON object; caller falls back to next provider instead of returning malformed tool_calls to clients. - #10 Regression warning — checkRegressionWarning() in learning.ts compares last-1h success rate vs prior-23h baseline per modelId; logs REGRESSION warning when drop >=20pp (with min sample sizes both sides). 10-min cooldown per modelId prevents log spam. Hooked into recordOutcomeLearning fire-and-forget on non-quota fails. Tests 95/95, build 0/0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
supports_tools=1model เมื่อ tool_calls structurally broken (10s timeout + AbortSignal)storefield ที่ Google rejectreasoning/refusalfields + Google tool_call history ordering fixDesign: Pattern B+ (Novel — ไม่มี gateway ไหนทำแบบนี้)
Changes
src/app/v1/chat/completions/route.ts— +210 lines (validation, repair, retry, sanitization)package.json— addjsonrepairdependency (zero deps, 24KB)Test plan
🤖 Generated with Claude Code