[AI Gateway] Add multi-provider routing for LLM proxies#2586
Merged
Conversation
Introduce additionalProviders on LLM proxies so one proxy can fan out to multiple LLM upstreams, with per-provider conditional upstream auth and inline request/response translators gated on the selected provider. - gateway-controller: additionalProviders + inline transformer in the management API schema and validator; transformProxy exposes each additional provider as a named upstream and injects auth/translator policies with selected_provider execution conditions - platform-api: additionalProviders across openapi, model, dto and service, and carried into the deployment artifact sent to the gateway - kubernetes-operator: additionalProviders on the LlmProxy CRD (types, deepcopy, CRD manifests) and controller payload - examples: multi-provider proxy and per-provider sample configs Translator policy implementations, their build config and the integration feature test follow in a separate PR.
edit upstream to LLM Provider Co-authored-by: Renuka Piyumal Fernando <renukapiyumal@gmail.com>
- examples: add Apache license headers to the nine new LLM provider and proxy samples, matching the short 2026 form used by recent examples - examples: remove the loopback ApiKey documents and their hardcoded key material from the five provider samples; point at api-key.yaml instead and use REPLACE_WITH_... placeholders for the proxies' provider.auth - examples: document the optional `as` alias on openai-multi-provider-proxy - platform-api: add LLMProxyTransformer and the transformer property to LLMProxyAdditionalProvider, mirroring the gateway-controller schema, and carry it through the model, dto, service mappers and deployment artifact - platform-api: align the `as` description with the gateway-controller spec
…ng-clean [AI Gateway] Add multi-provider routing for LLM proxies
Arshardh
requested review from
AnuGayan,
CrowleyRajapakse,
HeshanSudarshana,
HiranyaKavishani,
Krishanx92,
PasanT9,
RakhithaRR,
Tharsanan1,
Thushani-Jayasekera,
VirajSalaka,
ashera96,
chamilaadhi,
dushaniw,
hisanhunais,
lasanthaS,
malinthaprasan,
pubudu538,
renuka-fernando,
senthuran16,
tgtshanika,
tharikaGitHub,
tharindu1st and
thivindu
as code owners
July 10, 2026 10:14
Contributor
|
Warning Review limit reached
Next review available in: 36 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (11)
📝 WalkthroughWalkthroughAdds multi-provider LLM proxy support with additional-provider schemas, platform mappings, conditional gateway transformers and authentication, operator Secret resolution, validation, tests, and provider/proxy examples. ChangesMulti-provider LLM proxy support
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant openai-header-router
participant LLMProviderTransformer
participant Provider
Client->>openai-header-router: Send POST /chat/completions
openai-header-router->>LLMProviderTransformer: Select provider from x-provider
LLMProviderTransformer->>Provider: Apply matching transformer and auth
Provider-->>Client: Return translated provider response
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
renuka-fernando
previously approved these changes
Jul 10, 2026
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (5)
gateway/examples/openai-multi-provider-proxy.yaml (1)
92-119: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse
operationPoliciesinstead of the deprecatedpoliciesfield.Same as
anthropic-openai-proxy.yaml— thepoliciesfield onLLMProxyConfigDatais deprecated pergenerated.go. New examples should useoperationPolicies.♻️ Suggested rename
- policies: + operationPolicies: - name: api-key-auth version: v1 paths: - path: /chat/completions methods: [POST] params: key: X-API-Key in: header - name: openai-header-router version: v1 paths: - path: /chat/completions methods: [POST] params: headerName: x-provider defaultProvider: openai-provider mappings: - headerValue: openai provider: openai-provider - headerValue: anthropic provider: anthropic-provider - headerValue: azure-openai provider: azure-openai-provider - headerValue: mistral provider: mistral-provider - headerValue: gemini provider: gemini-provider🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@gateway/examples/openai-multi-provider-proxy.yaml` around lines 92 - 119, Replace the deprecated policies field with operationPolicies in the LLMProxyConfigData configuration, preserving the existing api-key-auth and openai-header-router entries unchanged.gateway/examples/anthropic-openai-proxy.yaml (1)
45-52: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse
operationPoliciesinstead of the deprecatedpoliciesfield.The
policiesfield onLLMProxyConfigDatais marked deprecated in the generated contract (generated.go): "DEPRECATED - use operationPolicies. Still honoured (treated identically to operationPolicies)." New example files should model the non-deprecated field so users copy the correct pattern.♻️ Suggested rename
- policies: + operationPolicies: - name: openai-to-anthropic version: v1 paths: - path: /chat/completions methods: [POST] params: model: claude-sonnet-4-5-20250929🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@gateway/examples/anthropic-openai-proxy.yaml` around lines 45 - 52, Replace the deprecated `policies` field with `operationPolicies` in the `openai-to-anthropic` configuration, preserving the existing policy name, version, paths, methods, and model parameters.platform-api/resources/openapi.yaml (2)
7873-7877: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win
LLMProxyAdditionalProvider.idmissing length bounds.Sibling
LLMProxyProvider.idenforcesminLength: 1/maxLength: 40, but this field has none, letting arbitrarily long/short values pass OpenAPI validation before failing later at a DB lookup.🔧 Suggested fix
id: type: string description: Unique id of a deployed llm provider + minLength: 1 + maxLength: 40 example: anthropic-provider🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@platform-api/resources/openapi.yaml` around lines 7873 - 7877, Update the LLMProxyAdditionalProvider.id property to match LLMProxyProvider.id by adding minLength: 1 and maxLength: 40 alongside its string type, ensuring OpenAPI validation rejects empty or overly long provider IDs.
7890-7913: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win
LLMProxyTransformer.versionlacks the major-only pattern enforced elsewhere.The description explicitly documents "Major-only translator policy version (for example v1)", mirroring
Policy.version/LLMPolicy.version/OperationPolicy.version, all of which enforcepattern: '^v\d+$'. This field has no such pattern, so a full semver likev1.0.0would pass schema validation despite the stated contract. See related comment ongateway/gateway-controller/pkg/config/llm_validator.go'svalidateLLMProxyTransformer, which also skips this check.🔧 Suggested fix
version: type: string description: > Major-only translator policy version (for example v1). The Gateway Controller resolves it to the installed full version. minLength: 1 + pattern: '^v\d+$' example: v1🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@platform-api/resources/openapi.yaml` around lines 7890 - 7913, Add the same major-only version constraint used by Policy.version, LLMPolicy.version, and OperationPolicy.version to LLMProxyTransformer.version: set its pattern to '^v\d+$' while retaining the existing minLength and example.gateway/gateway-controller/pkg/config/llm_validator.go (1)
613-622: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winHoist the
asformat regex out of the loop.
regexp.MustCompileis called fresh on everyadditionalProvidersentry instead of being precompiled once, unlike the existingv.metadataNameRegexpattern used just above.🔧 Suggested fix
+// add alongside metadataNameRegex on the LLMValidator struct: +// upstreamNameRegex = regexp.MustCompile(`^[a-zA-Z0-9\-_]+$`) ... - if !regexp.MustCompile(`^[a-zA-Z0-9\-_]+$`).MatchString(upstreamName) { + if !v.upstreamNameRegex.MatchString(upstreamName) {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@gateway/gateway-controller/pkg/config/llm_validator.go` around lines 613 - 622, Precompile the `as` name validation regex once outside the additional-providers loop, alongside the existing `v.metadataNameRegex`, and reuse that compiled pattern in the `provider.As` validation within the loop instead of calling regexp.MustCompile for each entry.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@gateway/gateway-controller/api/management-openapi.yaml`:
- Around line 5891-5914: The LLMProxyTransformer version schema currently
permits values beyond the required major-only policy format. Update the version
property under LLMProxyTransformer to validate the ^v\d+$ pattern, preserving
its existing string type and documentation so invalid values such as v1.0 or
latest are rejected at the API boundary.
In `@gateway/gateway-controller/pkg/config/llm_validator.go`:
- Around line 649-664: Update validateLLMProxyTransformer to validate Version
against the project’s major-only format, requiring a value matching ^v\d+$
rather than merely being non-empty. Reuse the existing policy version validation
convention or equivalent validator, while preserving the required-field error
handling and adding an appropriate invalid-format ValidationError for
transformer.version.
In `@kubernetes/gateway-operator/api/v1alpha1/llmproxy_types.go`:
- Around line 35-50: Add a Transformer field to LLMProxyAdditionalProvider using
the operator API’s transformer type and matching JSON/schema markers from the
management contract and gateway model. Update generated deepcopy code and
regenerate both CRD artifacts so spec.additionalProviders[*].transformer is
accepted and preserved.
In `@platform-api/internal/service/llm.go`:
- Around line 1354-1362: In the Create and Update validation flows, add eager
validation for every req.AdditionalProviders entry: verify each Id references an
existing provider and reject duplicate provider IDs, upstream names, or aliases
using the same validation/error patterns used for req.Provider.Id and policy
versions. Update the corresponding request-to-model mapping near the
AdditionalProviders field only after validation succeeds, covering both Create
and Update handlers.
---
Nitpick comments:
In `@gateway/examples/anthropic-openai-proxy.yaml`:
- Around line 45-52: Replace the deprecated `policies` field with
`operationPolicies` in the `openai-to-anthropic` configuration, preserving the
existing policy name, version, paths, methods, and model parameters.
In `@gateway/examples/openai-multi-provider-proxy.yaml`:
- Around line 92-119: Replace the deprecated policies field with
operationPolicies in the LLMProxyConfigData configuration, preserving the
existing api-key-auth and openai-header-router entries unchanged.
In `@gateway/gateway-controller/pkg/config/llm_validator.go`:
- Around line 613-622: Precompile the `as` name validation regex once outside
the additional-providers loop, alongside the existing `v.metadataNameRegex`, and
reuse that compiled pattern in the `provider.As` validation within the loop
instead of calling regexp.MustCompile for each entry.
In `@platform-api/resources/openapi.yaml`:
- Around line 7873-7877: Update the LLMProxyAdditionalProvider.id property to
match LLMProxyProvider.id by adding minLength: 1 and maxLength: 40 alongside its
string type, ensuring OpenAPI validation rejects empty or overly long provider
IDs.
- Around line 7890-7913: Add the same major-only version constraint used by
Policy.version, LLMPolicy.version, and OperationPolicy.version to
LLMProxyTransformer.version: set its pattern to '^v\d+$' while retaining the
existing minLength and example.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: ee03703b-5e45-45ca-bf83-b8f702ab12dc
📒 Files selected for processing (29)
gateway/examples/anthropic-openai-proxy.yamlgateway/examples/anthropic-provider.yamlgateway/examples/azure-openai-provider.yamlgateway/examples/azure-openai-proxy.yamlgateway/examples/gemini-provider.yamlgateway/examples/mistral-openai-proxy.yamlgateway/examples/mistral-provider.yamlgateway/examples/openai-multi-provider-proxy.yamlgateway/examples/openai-provider.yamlgateway/gateway-controller/api/management-openapi.yamlgateway/gateway-controller/pkg/api/management/generated.gogateway/gateway-controller/pkg/config/llm_validator.gogateway/gateway-controller/pkg/utils/llm_transformer.gogateway/gateway-controller/pkg/utils/llm_transformer_multiprovider_test.gokubernetes/gateway-operator/api/v1alpha1/llmproxy_types.gokubernetes/gateway-operator/api/v1alpha1/zz_generated.deepcopy.gokubernetes/gateway-operator/config/crd/bases/gateway.api-platform.wso2.com_llmproxies.yamlkubernetes/gateway-operator/internal/controller/llmproxy_controller.gokubernetes/gateway-operator/internal/controller/llmproxy_controller_test.gokubernetes/gateway-operator/internal/controller/management_upstream_auth_payload.gokubernetes/gateway-operator/internal/controller/management_valuefrom_enqueue.gokubernetes/gateway-operator/internal/controller/management_valuefrom_fingerprint.gokubernetes/helm/operator-helm-chart/crds/gateway.api-platform.wso2.com_llmproxies.yamlplatform-api/api/generated.goplatform-api/internal/dto/llm_deployment.goplatform-api/internal/model/llm.goplatform-api/internal/service/llm.goplatform-api/internal/service/llm_deployment.goplatform-api/resources/openapi.yaml
- Constrain LLMProxyTransformer.version to major-only in the management OpenAPI schema and the gateway-controller LLM validator. - Add operator-side LLMProxyTransformer type + additionalProviders[].transformer field; regenerate deepcopy and both CRD artifacts (operator + Helm). - Eagerly validate additionalProviders (existence + unique upstream names) at LLM proxy Create/Update, covered by Create/Update tests.
[FIX] Address further review comments on multi-provider routing
Fix nil pointer dereference in LLMProviderTransformer test
Krishanx92
approved these changes
Jul 12, 2026
This was referenced Jul 13, 2026
[AI GATEWAY] Rename openai-header-router policy to llm-header-router in multi provider routing
#2633
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merging changes done from: #2486