feat(ai-proxy-multi): let configured HTTP statuses trigger a fallback - #13852
Merged
nic-6443 merged 1 commit intoAug 20, 2026
Merged
Conversation
Only 429 and 5xx could fall back to another instance. A provider that answers 401 for an expired API key or 402 for a drained quota therefore returned that response straight to the client even when other instances with working keys were configured: active health checks can eventually take the instance out of rotation, but they cannot rescue the request that is in flight. Add fallback_http_statuses, an explicit list of upstream statuses that join http_429 / http_5xx in triggering the fallback. It has to be opt-in per status: most 4xx responses are caused by the request itself, and retrying those on another instance would only burn quota on a request that fails everywhere. The decision lives in two places that must agree. base.lua diverts only 429/5xx to the error path, where the body is read and handed to the retry callback; every other status is streamed to the client and never reaches ai-proxy-multi's retry_on_error at all. Both now consult the same helper, so a configured status is diverted and then retried, and max_retries / retry_on_failure_within_ms bound it like any other fallback.
dosubot
Bot
added
the
size:L
AlinsRan
approved these changes
Aug 20, 2026
shreemaan-abhishek
approved these changes
Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Request-level fallback in
ai-proxy-multionly covers 429 and 5xx. A provider that answers 401 for an expired API key, or 402 for a drained quota, returns that response straight to the client even when other instances with working keys are configured. Active health checks can eventually take the bad instance out of rotation, but they cannot rescue the request that is already in flight — which is exactly the multi-key setup people use this plugin for.This adds
fallback_http_statuses, an explicit list of upstream statuses that join thehttp_429/http_5xxentries offallback_strategyin triggering a fallback:{ "fallback_http_statuses": [401, 402], "instances": [ ... ] }It is opt-in per status rather than a blanket "retry 4xx": most 4xx responses are caused by the request itself, and retrying those on another instance only burns quota on a request that will fail everywhere.
The decision lives in two places that have to agree.
ai-proxy/base.luadiverts only 429/5xx to the error path, where the response body is read and handed to the retry callback; every other status is parsed and streamed to the client and never reachesretry_on_errorat all. So relaxing only the retry condition would have had no effect — both now consult the same helper.max_retriesandretry_on_failure_within_msbound these retries like any other fallback, and with no matching status the behaviour is unchanged.Tests cover the fallback on 401/402, the
max_retriesinteraction, the unchanged pass-through when the status is not configured, and the schema range check.