Skip to content

feat: Pattern B+ tool call retry with JSON repair#2

Open
mangsriso wants to merge 3 commits into
jaturapornchai:mainfrom
mangsriso:main
Open

feat: Pattern B+ tool call retry with JSON repair#2
mangsriso wants to merge 3 commits into
jaturapornchai:mainfrom
mangsriso:main

Conversation

@mangsriso

Copy link
Copy Markdown

Summary

  • เพิ่ม jsonrepair สำหรับซ่อม malformed tool_call arguments จาก free LLMs
  • เพิ่ม retryWithToolsModel() — retry ไป supports_tools=1 model เมื่อ tool_calls structurally broken (10s timeout + AbortSignal)
  • เพิ่ม hasStructurallyBrokenToolCalls() — ตรวจ missing function.name, null args, oversized args
  • เพิ่ม field whitelist ใน forwardToProvider — strip store field ที่ Google reject
  • เพิ่ม message sanitizer — strip reasoning/refusal fields + Google tool_call history ordering fix
  • เพิ่ม error logging สำหรับ non-200 upstream responses

Design: Pattern B+ (Novel — ไม่มี gateway ไหนทำแบบนี้)

Response → no tool_calls → pass through (zero overhead)
         → tool_calls valid → pass through
         → tool_calls malformed → jsonrepair attempt
             → repair OK → return repaired
             → repair fail → retry with tools-capable model (max 1)

Changes

  • src/app/v1/chat/completions/route.ts — +210 lines (validation, repair, retry, sanitization)
  • package.json — add jsonrepair dependency (zero deps, 24KB)

Test plan

  • Normal request without tools — zero overhead, passes through unchanged
  • Request with tools — proper function call returned with valid JSON args
  • Request with tool_call history (multi-turn) — sanitized for Google, works across providers
  • Streaming request — bypasses validation entirely
  • Invalid model — returns proper error
  • Large payload with tools — works correctly
  • store + tools combined — both fixes work together
  • 41/42 comprehensive checks passed

🤖 Generated with Claude Code

aitma and others added 3 commits April 4, 2026 19:21
…nitization

- Add jsonrepair for malformed tool_call arguments
- Add retryWithToolsModel with 10s AbortController timeout
- Enhance buildProxiedResponse with jsonrepair fallback
- Strip reasoning/refusal/store fields from requests
- Google message sanitizer for tool role messages

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Port Pattern B+ from BCProxyAI (SQLite) to SML Gateway (async PostgreSQL):

- Add jsonrepair dependency for malformed tool_call argument repair
- Add src/lib/tool-repair.ts: repairToolCallArguments() + hasStructurallyBrokenToolCalls()
- Wire repair into non-streaming response handler (sequential retry path)
- Wire repair into buildProxiedResponse (final response path)
- Add retryWithToolsModel() — async, respects exam system model selection
- Set X-SMLGateway-ToolRepair header when retry is used

Pattern B+ flow:
1. When tool_calls present, attempt jsonrepair on malformed arguments
2. Check for structural breaks (missing name, null args, oversized args)
3. If broken, retry with next tools-capable model (15s timeout)
4. Return repaired/retried response or best-effort original

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jaturapornchai added a commit that referenced this pull request May 3, 2026
…ssion, latency, RPM, breaker, fallback)

Tackled the audit list end-to-end. Two of the requested ten were already
implemented in route.ts (empty-body retry on both proxied + relaxed-retry
paths), one was deferred (undici dispatcher re-enable — Node 20 already
uses undici keep-alive and the previous nvidia-hang bug has no public fix
record), one was scoped out (Thai exam dataset is a separate task).

Speed
- #1 Client-side RPM throttle (rate-budget.ts) — Redis sliding-window per
  (provider, model). Skip locally before burning the upstream 429 budget,
  which is what triggers minutes-long cooldowns. Per-provider RPM defaults
  in free-model-catalog map documented free-tier limits (Groq 30, SEA-LION
  10, Typhoon 200, etc.). Fail-open on Redis outage.
- #2 Latency-aware sort — added SambaNova to FAST_STREAM_PROVIDERS, plus a
  PROVIDER_LATENCY_HINT_MS table so providers with no production data yet
  rank by their documented first-token latency instead of being pinned at
  9999999 (which previously stranded new providers at the back forever).

Reliability
- #4 Fallback chain — provider-diversity interleave in reorderForLatency:
  the first 6 candidates round-robin across providers, so a single-provider
  outage no longer stalls the whole chain. Tail keeps strict latency order.
- #5 Circuit breaker (3-state) — acquireCircuitProbe/releaseCircuitProbe
  in learning.ts adds explicit half-open: when fail_streak>=2 and cooldown
  has expired, only HALF_OPEN_PROBE_LIMIT concurrent probes are admitted.
  API exported; wiring into chat/completions left for the fallback-chain
  refactor follow-up.
- #6 Deprecation watcher — deprecatedAfter field on FreeModelCatalogEntry.
  getActiveFreeModelCatalog filters EOL models from /v1/models, scanner,
  search, and isHardcodedFreeModel. getModelsDeprecatingSoon emits warn
  logs from the worker scan cycle for anything within 7 days of EOL.
  Cerebras llama-3.3-70b (2026-02-16) and qwen-3-235b-a22b-instruct-2507
  (2026-05-27) tagged.

Quality
- #8 Tool-call argument JSON validation — validateToolCallArguments in
  tool-call-repair.ts. isResponseBad now rejects responses where any
  tool_call argument string fails JSON.parse or isn't a JSON object;
  caller falls back to next provider instead of returning malformed
  tool_calls to clients.
- #10 Regression warning — checkRegressionWarning() in learning.ts
  compares last-1h success rate vs prior-23h baseline per modelId; logs
  REGRESSION warning when drop >=20pp (with min sample sizes both sides).
  10-min cooldown per modelId prevents log spam. Hooked into
  recordOutcomeLearning fire-and-forget on non-quota fails.

Tests 95/95, build 0/0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant