fix(error-classifier): treat flaky free-model gateway 403/Forbidden as transient#3
Open
anicca-earn wants to merge 1 commit into
Open
Conversation
Free-model routing occasionally proxies through an upstream (e.g. NVIDIA NIM) whose credential briefly rejects the call with 403 Forbidden / "Authorization failed" even though the gateway's own auth is fine — the same request typically succeeds on retry (observed ~1 in 3 calls on the Solana-chain free-model gateway). Before this fix, classifyAgentError() had no branch matching "403" — neither the '401'/unauthorized 'auth' branch nor the 500-504 'server' branch matched it, so the error fell through to the catch-all 'unknown' category with isTransient:false. The agent loop's recovery logic only retries when isTransient is true, so every session hitting this hiccup died outright instead of retrying past it. Classify it as 'server' (not 'auth') so it inherits the existing transient-retry budget and the server-error streak guard's automatic model-fallback (loop.ts), instead of the non-retryable 'auth' path, which tells the user to reconfigure their own API key — there's nothing for them to fix here. Adds a regression test covering: the HTTP-prefixed form, the exact production wording (no HTTP prefix), a bare '403 Forbidden' with no echoed 'Authorization failed', and confirms a real 401/invalid-key error is untouched (stays non-transient 'auth').
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Franklin's SOL-chain free-model gateway (
sol.blockrun.ai) proxies through an upstream (e.g. NVIDIA NIM) whose credential is flaky — roughly 1 in 3 calls comes back403 Forbidden Authorization failedeven though the gateway's own auth is fine, and the very same request typically succeeds on retry.classifyAgentError()insrc/agent/error-classifier.tshad no branch matching403. Neither the401/unauthorizedauthbranch nor the500-504serverbranch matched it, so the error fell through to the catch-allunknowncategory withisTransient: false. Since the agent loop's recovery logic (src/agent/loop.ts) only retries whenisTransientistrue, every session that hit this upstream hiccup died outright instead of retrying past a transient blip.Fix
Add a branch that classifies
"authorization failed"wording (and the general403+"forbidden"shape) as category'server'withisTransient: true. This is deliberately not folded into the existing'auth'category —'auth'tells the user to reconfigure their own API key, which isn't the right remedy here (there's nothing wrong with the user's credentials; the gateway's upstream hiccuped). Reusing'server'means the fix gets the existing transient-retry budget for free, plusloop.ts's server-error streak guard, which automatically switches to the next model in the routing fallback chain if the same model keeps failing — exactly the graceful-degradation behavior a flaky free-model gateway needs, with no new plumbing required.Testing
Added
error classifier treats flaky gateway 403 as transient, not as a user auth failuretotest/local.mjs, covering:HTTP 403: Forbidden Authorization failed(thellm.ts-style prefixed form)API error: 403 Forbidden Authorization failed(the exact wording observed in production)HTTP 403: Forbidden(bare form, no echoed "Authorization failed")401 Unauthorized: invalid API keystill classifies as non-transientauth— confirming this fix doesn't loosen the existing auth-failure pathRan the full local suite (
npm test: 348/348 pass) andnpm run test:strategies(3/3 pass) — no regressions. Live/paid suites (test:e2e,test:free-models) were not run since they require network access and real spend, unrelated to this classifier-only change.Root cause repro (before fix)