[AI Gateway] Add multi-provider routing for LLM proxies#2486
Merged
Arshardh merged 3 commits intoJul 10, 2026
Merged
Conversation
Introduce additionalProviders on LLM proxies so one proxy can fan out to multiple LLM upstreams, with per-provider conditional upstream auth and inline request/response translators gated on the selected provider. - gateway-controller: additionalProviders + inline transformer in the management API schema and validator; transformProxy exposes each additional provider as a named upstream and injects auth/translator policies with selected_provider execution conditions - platform-api: additionalProviders across openapi, model, dto and service, and carried into the deployment artifact sent to the gateway - kubernetes-operator: additionalProviders on the LlmProxy CRD (types, deepcopy, CRD manifests) and controller payload - examples: multi-provider proxy and per-provider sample configs Translator policy implementations, their build config and the integration feature test follow in a separate PR.
Aakashwije
requested review from
AnuGayan,
Arshardh,
CrowleyRajapakse,
HeshanSudarshana,
HiranyaKavishani,
Krishanx92,
PasanT9,
RakhithaRR,
Tharsanan1,
VirajSalaka,
ashera96,
chamilaadhi,
dushaniw,
hisanhunais,
malinthaprasan,
pubudu538,
renuka-fernando,
senthuran16,
tgtshanika,
tharikaGitHub and
tharindu1st
as code owners
July 6, 2026 18:12
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis PR adds ChangesMulti-provider LLM proxy support
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant Gateway
participant HeaderRouter as openai-header-router
participant AuthPolicy as Upstream Auth Policy
participant Translator as Translator Policy
participant Upstream as Selected Provider
Client->>Gateway: POST /chat/completions (x-provider: anthropic)
Gateway->>HeaderRouter: resolve x-provider header
HeaderRouter->>Gateway: set selected_provider metadata
Gateway->>AuthPolicy: check ExecutionCondition(selected_provider)
AuthPolicy->>Translator: inject provider API key, run translator if defined
Translator->>Upstream: forward transformed request
Upstream-->>Client: response
sequenceDiagram
participant Controller as llmProxyAdapter.Deploy
participant K8sAPI
participant Secret
participant Payload as Deployment Payload
Controller->>Controller: convert LlmProxy spec to JSON map
Controller->>K8sAPI: resolve provider.auth (valueFrom)
K8sAPI->>Secret: fetch secret key
Secret-->>Controller: resolved credential
Controller->>Payload: flatten provider auth value
loop each additionalProviders[i]
Controller->>K8sAPI: resolve additionalProviders[i].auth (valueFrom)
K8sAPI->>Secret: fetch secret key
Secret-->>Controller: resolved credential
Controller->>Payload: flattenAdditionalProviderAuthCredentialValue
end
Controller->>K8sAPI: POST deployment payload
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Aakashwije
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (6)
kubernetes/gateway-operator/internal/controller/llmproxy_controller.go (1)
167-194: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDuplicated resolve/nil-check/flatten pattern between provider and additionalProviders.
The
provider.Authblock (Lines 167-178) and theAdditionalProvidersloop (Lines 179-194) repeat the same resolve → nil-check → flatten sequence with only the field path/flatten target differing. Consider extracting a small helper, e.g.resolveAndFlattenUpstreamAuth(ctx, k8sClient, ns, auth, fieldPath, flattenFn), to reduce duplication and centralize the nil-guard error message format.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kubernetes/gateway-operator/internal/controller/llmproxy_controller.go` around lines 167 - 194, The provider.Auth handling and the AdditionalProviders loop in llmproxy_controller.go repeat the same resolve, nil-check, and flatten flow. Extract that shared logic into a small helper (for example, resolveAndFlattenUpstreamAuth) that takes the auth object, field path, and a flatten callback so both the main provider and additional providers use the same code path. Preserve the existing error handling and nil-guard messages by having the helper call resolveLLMProviderUpstreamAuth and then either flattenUpstreamAuthCredentialValue or flattenAdditionalProviderAuthCredentialValue based on the supplied target.kubernetes/gateway-operator/internal/controller/management_upstream_auth_payload.go (1)
33-64: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winNear-duplicate logic between the two flatten helpers.
flattenUpstreamAuthCredentialValueandflattenAdditionalProviderAuthCredentialValueshare the same "assert object → findauth→ setvalue" shape, differing only in how the parent is located (map key vs. indexed array element). Consider factoring out a sharedsetAuthValue(parent map[string]interface{}, plaintext string) errorhelper to avoid maintaining two copies of this logic.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kubernetes/gateway-operator/internal/controller/management_upstream_auth_payload.go` around lines 33 - 64, The two flatten helpers duplicate the same “get parent auth object and set value” logic, so refactor to reduce maintenance risk. Keep the parent lookup differences in flattenUpstreamAuthCredentialValue and flattenAdditionalProviderAuthCredentialValue, but extract the shared auth update into a helper such as setAuthValue that takes the parent map and plaintext. Update both helpers to call the shared helper after their respective type/range checks, preserving the existing error messages for spec.%s and spec.additionalProviders[%d] paths.gateway/gateway-controller/pkg/config/llm_validator.go (1)
649-694: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDuplicate auth-validation logic vs.
validateUpstreamWithAuth.
validateLLMUpstreamAuthreimplements the same three checks already present invalidateUpstreamWithAuth(Lines 454-485): type required, type must beapi-key, header/value required whenapi-key. The two operate on structurally-identical but distinct generated types (LLMUpstreamAuthvs. the inline upstream-auth object), so a straightforward fix is to extract a primitive-based helper (e.g.,authType string, header, value *string) that both call sites can delegate to.♻️ Suggested extraction
-func (v *LLMValidator) validateLLMUpstreamAuth(fieldPrefix string, auth *api.LLMUpstreamAuth) []ValidationError { - var errors []ValidationError - if auth.Type == "" { - ... - } - ... - return errors -} +func (v *LLMValidator) validateAuthFields(fieldPrefix string, authType string, header, value *string) []ValidationError { + var errors []ValidationError + if authType == "" { + errors = append(errors, ValidationError{Field: fieldPrefix + ".type", Message: "Auth type is required"}) + } else if authType != "api-key" { + errors = append(errors, ValidationError{Field: fieldPrefix + ".type", Message: "Auth type must be 'api-key'"}) + } + if authType == "api-key" { + if header == nil || *header == "" { + errors = append(errors, ValidationError{Field: fieldPrefix + ".header", Message: "Auth header is required when api-key auth type is set"}) + } + if value == nil || *value == "" { + errors = append(errors, ValidationError{Field: fieldPrefix + ".value", Message: "Auth value is required when api-key auth type is set"}) + } + } + return errors +} + +func (v *LLMValidator) validateLLMUpstreamAuth(fieldPrefix string, auth *api.LLMUpstreamAuth) []ValidationError { + return v.validateAuthFields(fieldPrefix, string(auth.Type), auth.Header, auth.Value) +}Then
validateUpstreamWithAuth(Lines 454-485) can delegate tovalidateAuthFieldstoo.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@gateway/gateway-controller/pkg/config/llm_validator.go` around lines 649 - 694, `validateLLMUpstreamAuth` is duplicating the same auth checks already implemented in `validateUpstreamWithAuth`. Extract the shared validation into a small helper that works on primitive inputs (for example, auth type plus header/value pointers or strings) and have both `validateLLMUpstreamAuth` and `validateUpstreamWithAuth` delegate to it. Keep the existing field-specific error paths/messages in each caller while removing the repeated type/header/value logic.gateway/examples/openai-multi-provider-proxy.yaml (1)
76-104: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winNew examples use the deprecated
policies:field.This is a brand-new example file, but it (and the sibling
mistral-openai-proxy.yaml,mistral-provider.yaml,openai-provider.yaml) uses the deprecated top-levelpolicies:list rather thanoperationPolicies:. SinceLLMPolicy/LLMPolicyPathandOperationPolicy/OperationPolicyPathhave the same shape (name/version/paths with path/methods/params), this is essentially a key rename with no behavior change — worth doing in fresh docs to avoid teaching the deprecated pattern.♻️ Suggested rename
- policies: + operationPolicies: - name: api-key-auth version: v1🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@gateway/examples/openai-multi-provider-proxy.yaml` around lines 76 - 104, The example configuration still uses the deprecated top-level policies list, so update this and the related example YAMLs to use operationPolicies instead. This is a straight field rename with the same structure, so keep each policy entry and its nested paths/methods/params unchanged while moving them under operationPolicies. Make sure the openai-header-router and api-key-auth entries remain functionally identical after the rename.gateway/gateway-controller/pkg/utils/llm_transformer.go (1)
162-165: 📐 Maintainability & Code Quality | 🔵 TrivialDuplicate upstream-name resolution logic.
The
ap.Id/ap.Asfallback logic is repeated verbatim in two loops. Consider extracting a small helper, e.g.additionalProviderUpstreamName(ap api.LLMProxyAdditionalProvider) string.Also applies to: 244-247
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@gateway/gateway-controller/pkg/utils/llm_transformer.go` around lines 162 - 165, The ap.Id/ap.As fallback logic is duplicated in both loops inside llm_transformer.go, so extract it into a small helper named something like additionalProviderUpstreamName that takes api.LLMProxyAdditionalProvider and returns the resolved upstream name. Update both call sites in the transformer code to use that helper instead of repeating the nil/empty check and assignment logic.gateway/gateway-controller/pkg/utils/llm_transformer_multiprovider_test.go (1)
119-121: 📐 Maintainability & Code Quality | 🔵 TrivialConsider asserting
Url/BasePathon the generatedUpstreamDefinition.Only
Nameis checked here. Given theUrl/BasePathsplit question raised inllm_transformer.go, asserting the full upstream shape would catch regressions in how the provider context path is represented.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@gateway/gateway-controller/pkg/utils/llm_transformer_multiprovider_test.go` around lines 119 - 121, The test for the generated UpstreamDefinition only verifies Name, so it can miss regressions in how provider routing is represented. Update the assertion in the multiprovider transformer test around result.Spec.UpstreamDefinitions to also validate the generated Url and/or BasePath values, using the same upstream object created by llm_transformer.go so the provider context path split is covered.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@gateway/examples/openai-multi-provider-proxy.yaml`:
- Around line 76-104: The example configuration still uses the deprecated
top-level policies list, so update this and the related example YAMLs to use
operationPolicies instead. This is a straight field rename with the same
structure, so keep each policy entry and its nested paths/methods/params
unchanged while moving them under operationPolicies. Make sure the
openai-header-router and api-key-auth entries remain functionally identical
after the rename.
In `@gateway/gateway-controller/pkg/config/llm_validator.go`:
- Around line 649-694: `validateLLMUpstreamAuth` is duplicating the same auth
checks already implemented in `validateUpstreamWithAuth`. Extract the shared
validation into a small helper that works on primitive inputs (for example, auth
type plus header/value pointers or strings) and have both
`validateLLMUpstreamAuth` and `validateUpstreamWithAuth` delegate to it. Keep
the existing field-specific error paths/messages in each caller while removing
the repeated type/header/value logic.
In `@gateway/gateway-controller/pkg/utils/llm_transformer_multiprovider_test.go`:
- Around line 119-121: The test for the generated UpstreamDefinition only
verifies Name, so it can miss regressions in how provider routing is
represented. Update the assertion in the multiprovider transformer test around
result.Spec.UpstreamDefinitions to also validate the generated Url and/or
BasePath values, using the same upstream object created by llm_transformer.go so
the provider context path split is covered.
In `@gateway/gateway-controller/pkg/utils/llm_transformer.go`:
- Around line 162-165: The ap.Id/ap.As fallback logic is duplicated in both
loops inside llm_transformer.go, so extract it into a small helper named
something like additionalProviderUpstreamName that takes
api.LLMProxyAdditionalProvider and returns the resolved upstream name. Update
both call sites in the transformer code to use that helper instead of repeating
the nil/empty check and assignment logic.
In `@kubernetes/gateway-operator/internal/controller/llmproxy_controller.go`:
- Around line 167-194: The provider.Auth handling and the AdditionalProviders
loop in llmproxy_controller.go repeat the same resolve, nil-check, and flatten
flow. Extract that shared logic into a small helper (for example,
resolveAndFlattenUpstreamAuth) that takes the auth object, field path, and a
flatten callback so both the main provider and additional providers use the same
code path. Preserve the existing error handling and nil-guard messages by having
the helper call resolveLLMProviderUpstreamAuth and then either
flattenUpstreamAuthCredentialValue or
flattenAdditionalProviderAuthCredentialValue based on the supplied target.
In
`@kubernetes/gateway-operator/internal/controller/management_upstream_auth_payload.go`:
- Around line 33-64: The two flatten helpers duplicate the same “get parent auth
object and set value” logic, so refactor to reduce maintenance risk. Keep the
parent lookup differences in flattenUpstreamAuthCredentialValue and
flattenAdditionalProviderAuthCredentialValue, but extract the shared auth update
into a helper such as setAuthValue that takes the parent map and plaintext.
Update both helpers to call the shared helper after their respective type/range
checks, preserving the existing error messages for spec.%s and
spec.additionalProviders[%d] paths.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: db63d46e-d73e-4d99-b795-7d4b119c46af
📒 Files selected for processing (29)
gateway/examples/anthropic-openai-proxy.yamlgateway/examples/anthropic-provider.yamlgateway/examples/azure-openai-provider.yamlgateway/examples/azure-openai-proxy.yamlgateway/examples/gemini-provider.yamlgateway/examples/mistral-openai-proxy.yamlgateway/examples/mistral-provider.yamlgateway/examples/openai-multi-provider-proxy.yamlgateway/examples/openai-provider.yamlgateway/gateway-controller/api/management-openapi.yamlgateway/gateway-controller/pkg/api/management/generated.gogateway/gateway-controller/pkg/config/llm_validator.gogateway/gateway-controller/pkg/utils/llm_transformer.gogateway/gateway-controller/pkg/utils/llm_transformer_multiprovider_test.gokubernetes/gateway-operator/api/v1alpha1/llmproxy_types.gokubernetes/gateway-operator/api/v1alpha1/zz_generated.deepcopy.gokubernetes/gateway-operator/config/crd/bases/gateway.api-platform.wso2.com_llmproxies.yamlkubernetes/gateway-operator/internal/controller/llmproxy_controller.gokubernetes/gateway-operator/internal/controller/llmproxy_controller_test.gokubernetes/gateway-operator/internal/controller/management_upstream_auth_payload.gokubernetes/gateway-operator/internal/controller/management_valuefrom_enqueue.gokubernetes/gateway-operator/internal/controller/management_valuefrom_fingerprint.gokubernetes/helm/operator-helm-chart/crds/gateway.api-platform.wso2.com_llmproxies.yamlplatform-api/api/generated.goplatform-api/internal/dto/llm_deployment.goplatform-api/internal/model/llm.goplatform-api/internal/service/llm.goplatform-api/internal/service/llm_deployment.goplatform-api/resources/openapi.yaml
Arshardh
requested review from
Thushani-Jayasekera,
lasanthaS and
thivindu
as code owners
July 8, 2026 08:59
Comment on lines
+5854
to
+5862
| as: | ||
| type: string | ||
| description: > | ||
| Logical upstream name used by policies to select this provider. Must | ||
| be unique within the proxy. Defaults to `id` when omitted. | ||
| pattern: '^[a-zA-Z0-9\-_]+$' | ||
| minLength: 1 | ||
| maxLength: 100 | ||
| example: anthropic-upstream |
Contributor
There was a problem hiding this comment.
@Arshardh can you remember why we need this as?
Contributor
There was a problem hiding this comment.
We discussed to have an optional alias just in case the provider names will not be readable enough to configure in a routing policy
| Message: "spec.provider.id must consist of lowercase alphanumeric characters, hyphens, or dots, and must start and end with an alphanumeric character", | ||
| }) | ||
| } | ||
| if spec.Provider.Auth != nil { |
Contributor
There was a problem hiding this comment.
Do we need to make provider auth required?
edit upstream to LLM Provider Co-authored-by: Renuka Piyumal Fernando <renukapiyumal@gmail.com>
- examples: add Apache license headers to the nine new LLM provider and proxy samples, matching the short 2026 form used by recent examples - examples: remove the loopback ApiKey documents and their hardcoded key material from the five provider samples; point at api-key.yaml instead and use REPLACE_WITH_... placeholders for the proxies' provider.auth - examples: document the optional `as` alias on openai-multi-provider-proxy - platform-api: add LLMProxyTransformer and the transformer property to LLMProxyAdditionalProvider, mirroring the gateway-controller schema, and carry it through the model, dto, service mappers and deployment artifact - platform-api: align the `as` description with the gateway-controller spec
This was referenced Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Before this change, using more than one LLM vendor from a single app meant standing up a separate proxy and endpoint per vendor, and formatting requests differently for each. There was no way to route one OpenAI-shaped endpoint to multiple upstreams, and translator policies had to be declared separately from the provider they targeted - duplicating the provider
idby hand and keeping it in sync.Relates to #2490
Goals
additionalProviders[].transformer), so provider auth and its translator live together and the provideridis injected automatically.Approach
A proxy declares a primary
providerplus a list ofadditionalProviders, each with its own upstream auth and an inlinetransformer. Theopenai-header-routerpolicy reads thex-providerheader and writes the choice intometadata["selected_provider"]. The controller injects each provider's translator and upstream-auth as conditional policies gated onselected_provider == <provider>, so only the selected provider's translator and key are applied. Each additional provider is exposed as a named upstream for loopback routing.New config reads top-to-bottom as "provider + how to talk to it":
Implemented across three modules:
additionalProviders+ inlinetransformerin the management schema and validator;transformProxyexposes each additional provider as a named upstream and injects auth/translator policies gated on aselected_providerexecution condition.additionalProvidersacross openapi, model, dto and service, carried into the deployment artifact sent to the gateway.additionalProviderson theLlmProxyCRD (types, deepcopy, regenerated CRD manifests) and controller payload.User stories
Documentation
N/A for this PR. The example YAMLs under
gateway/examples/(openai-multi-provider-proxy.yamland per-provider configs) serve as usage documentation. Product-doc updates, if any, will accompany the follow-up translator-policies PR.Automation tests
Unit tests
gateway-controller/pkg/utilscover conditional upstream-auth injection and conditional transformer injection — correctselected_providerexecution condition, injectedidparam, and per-provider isolation.Integration tests
x-providerheader and return their own vendor model.401on wrong/missing key; deploy-time validation returns400for a transformer missingtype/version.llm-proxies.featurescenarios land with the follow-up translator-policies PR (they depend on those policy binaries).Security checks
go vetwas run clean.REPLACE_WITH_*placeholders and non-secret loopback identifiers. Real keys kept in a gitignored local path.Samples
gateway/examples/openai-multi-provider-proxy.yaml- one proxy fanning out to multiple providers with inline transformers.openai-provider.yaml,anthropic-provider.yaml,azure-openai-provider.yaml,mistral-provider.yaml,gemini-provider.yaml.Related PRs
Follow-up (separate PR): translator policy implementations (
openai-to-anthropic/azure-openai/mistral/gemini,openai-header-router), theirbuild.yaml/build-manifest.yamlwiring, and thellm-proxies.featureintegration test.Test environment
gateway-controller/go.mod)1.2.0-M2-SNAPSHOT)