Skip to content

fix(error-classifier): treat flaky free-model gateway 403/Forbidden as transient#3

Open
anicca-earn wants to merge 1 commit into
BlockRunAI:mainfrom
anicca-earn:fix/flaky-gateway-403-transient-classifier
Open

fix(error-classifier): treat flaky free-model gateway 403/Forbidden as transient#3
anicca-earn wants to merge 1 commit into
BlockRunAI:mainfrom
anicca-earn:fix/flaky-gateway-403-transient-classifier

Conversation

@anicca-earn

Copy link
Copy Markdown

What

Franklin's SOL-chain free-model gateway (sol.blockrun.ai) proxies through an upstream (e.g. NVIDIA NIM) whose credential is flaky — roughly 1 in 3 calls comes back 403 Forbidden Authorization failed even though the gateway's own auth is fine, and the very same request typically succeeds on retry.

classifyAgentError() in src/agent/error-classifier.ts had no branch matching 403. Neither the 401/unauthorized auth branch nor the 500-504 server branch matched it, so the error fell through to the catch-all unknown category with isTransient: false. Since the agent loop's recovery logic (src/agent/loop.ts) only retries when isTransient is true, every session that hit this upstream hiccup died outright instead of retrying past a transient blip.

Fix

Add a branch that classifies "authorization failed" wording (and the general 403 + "forbidden" shape) as category 'server' with isTransient: true. This is deliberately not folded into the existing 'auth' category — 'auth' tells the user to reconfigure their own API key, which isn't the right remedy here (there's nothing wrong with the user's credentials; the gateway's upstream hiccuped). Reusing 'server' means the fix gets the existing transient-retry budget for free, plus loop.ts's server-error streak guard, which automatically switches to the next model in the routing fallback chain if the same model keeps failing — exactly the graceful-degradation behavior a flaky free-model gateway needs, with no new plumbing required.

Testing

Added error classifier treats flaky gateway 403 as transient, not as a user auth failure to test/local.mjs, covering:

  • HTTP 403: Forbidden Authorization failed (the llm.ts-style prefixed form)
  • API error: 403 Forbidden Authorization failed (the exact wording observed in production)
  • HTTP 403: Forbidden (bare form, no echoed "Authorization failed")
  • a real 401 Unauthorized: invalid API key still classifies as non-transient auth — confirming this fix doesn't loosen the existing auth-failure path

Ran the full local suite (npm test: 348/348 pass) and npm run test:strategies (3/3 pass) — no regressions. Live/paid suites (test:e2e, test:free-models) were not run since they require network access and real spend, unrelated to this classifier-only change.

Root cause repro (before fix)

classifyAgentError('API error: 403 Forbidden Authorization failed')
// → { category: 'unknown', isTransient: false }   (before)
// → { category: 'server',  isTransient: true  }   (after)

Free-model routing occasionally proxies through an upstream (e.g. NVIDIA
NIM) whose credential briefly rejects the call with 403 Forbidden /
"Authorization failed" even though the gateway's own auth is fine — the
same request typically succeeds on retry (observed ~1 in 3 calls on the
Solana-chain free-model gateway).

Before this fix, classifyAgentError() had no branch matching "403" —
neither the '401'/unauthorized 'auth' branch nor the 500-504 'server'
branch matched it, so the error fell through to the catch-all 'unknown'
category with isTransient:false. The agent loop's recovery logic only
retries when isTransient is true, so every session hitting this hiccup
died outright instead of retrying past it.

Classify it as 'server' (not 'auth') so it inherits the existing
transient-retry budget and the server-error streak guard's automatic
model-fallback (loop.ts), instead of the non-retryable 'auth' path, which
tells the user to reconfigure their own API key — there's nothing for
them to fix here.

Adds a regression test covering: the HTTP-prefixed form, the exact
production wording (no HTTP prefix), a bare '403 Forbidden' with no
echoed 'Authorization failed', and confirms a real 401/invalid-key error
is untouched (stays non-transient 'auth').
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant