From dce741f8c4ed6dd19eec49b2103bcce5ba0f56a3 Mon Sep 17 00:00:00 2001 From: Aakash Wijesekara Date: Mon, 13 Jul 2026 14:59:15 +0530 Subject: [PATCH 1/3] Add multi-provider routing documentation --- .../next/llm-proxy/multi-provider-routing.md | 531 ++++++++++++++++++ en/docs/ai-gateway/next/overview.md | 51 +- en/mkdocs.yml | 1 + 3 files changed, 556 insertions(+), 27 deletions(-) create mode 100644 en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md diff --git a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md new file mode 100644 index 000000000..5afb79a53 --- /dev/null +++ b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md @@ -0,0 +1,531 @@ +--- +title: "Multi-Provider Routing for LLM Proxies" +description: "Route OpenAI-compatible LLM proxy requests to multiple providers using header-based selection and provider-specific transformers." +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/multi-provider-routing/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/multi-provider-routing.md +tags: + - ai-gateway + - llm + - routing +author: WSO2 API Platform Documentation Team +last_updated: 2026-07-13 +content_type: "guide" +--- + +# Multi-Provider Routing for LLM Proxies + +## Overview + +Multi-provider routing lets one LLM proxy expose a single OpenAI-compatible endpoint while routing each request to a selected LLM provider. Applications keep one endpoint and one request and response format, even when the upstream provider changes. + +For example, an application can send all requests to `/openai-multi/chat/completions` and select OpenAI or Anthropic with the `x-provider` request header. + +This is useful when you want to: + +- Switch providers without changing application code or endpoint URLs +- Compare provider responses using the same OpenAI-compatible request +- Keep vendor credentials in the gateway instead of distributing them to applications +- Apply proxy-level authentication, rate limits, and guardrails consistently across providers +- Introduce provider fallback or selection logic through a routing policy + +## How It Works + +A multi-provider LLM proxy has: + +- One primary provider in `spec.provider` +- One or more selectable providers in `spec.additionalProviders` +- An `openai-header-router` policy that selects a provider from a request header +- An inline transformer for each additional provider that does not use the OpenAI wire format + +The request flow is: + +```text +OpenAI-compatible client request + | + | x-provider: anthropic + v + Multi-provider LLM proxy + | + | openai-header-router selects anthropic-provider + | openai-to-anthropic transforms the request + | provider loopback authentication is added + v + Anthropic LLM provider + | + | vendor authentication is added + v + Anthropic API + | + | response is transformed to OpenAI format + v +OpenAI-compatible client response +``` + +The router writes the selected provider name to request metadata. The gateway conditionally applies only the authentication and transformer associated with that provider. The primary provider is used when the selection header is missing, empty, or does not match a configured mapping. + +## Before You Begin + +Make sure that: + +- The AI Gateway is running and the management API is available at `http://localhost:9090/api/management/v1`. +- You are using an AI Gateway version that supports multi-provider routing and includes the required router and transformer policies. +- You have credentials for each external LLM provider. +- `curl` and `jq` are installed if you want to follow the command-line examples. + +This guide configures OpenAI as the primary provider and Anthropic as an additional provider. The same configuration model can be extended to Azure OpenAI, Mistral, Gemini, AWS Bedrock, and other providers supported by your AI Gateway version. + +## Understand the Authentication Layers + +Multi-provider routing can involve three different kinds of credentials: + +| Credential | Used by | Purpose | +|------------|---------|---------| +| Vendor credential | LLM provider to external vendor | Authenticates the gateway to OpenAI, Anthropic, or another external service | +| Provider loopback key | LLM proxy to LLM provider | Authenticates the proxy when it routes internally to a protected provider | +| Proxy consumer key | Application to LLM proxy | Authenticates the application invoking the public proxy endpoint | + +Do not use a vendor API key as a loopback or consumer key. Do not commit any of these credentials to source control. + +## Step 1: Deploy the LLM Providers + +Each provider must exist before a proxy can reference it. + +### Deploy the OpenAI provider + +Replace `` with an OpenAI API key. + +```bash +curl -X POST http://localhost:9090/api/management/v1/llm-providers \ + -u admin:admin \ + -H "Content-Type: application/yaml" \ + --data-binary @- <<'EOF' +apiVersion: gateway.api-platform.wso2.com/v1 +kind: LlmProvider +metadata: + name: openai-provider +spec: + displayName: OpenAI Provider + version: v1.0 + template: openai + context: /providers/openai + upstream: + url: https://api.openai.com/v1 + auth: + type: api-key + header: Authorization + value: Bearer + accessControl: + mode: deny_all + exceptions: + - path: /chat/completions + methods: [POST] + operationPolicies: + - name: api-key-auth + version: v1 + paths: + - path: /chat/completions + methods: [POST] + params: + key: X-API-Key + in: header +EOF +``` + +### Deploy the Anthropic provider + +Replace `` with an Anthropic API key. + +```bash +curl -X POST http://localhost:9090/api/management/v1/llm-providers \ + -u admin:admin \ + -H "Content-Type: application/yaml" \ + --data-binary @- <<'EOF' +apiVersion: gateway.api-platform.wso2.com/v1 +kind: LlmProvider +metadata: + name: anthropic-provider +spec: + displayName: Anthropic Provider + version: v1.0 + template: anthropic + context: /providers/anthropic + upstream: + url: https://api.anthropic.com + auth: + type: api-key + header: x-api-key + value: + accessControl: + mode: deny_all + exceptions: + - path: /v1/messages + methods: [POST] + operationPolicies: + - name: api-key-auth + version: v1 + paths: + - path: /v1/messages + methods: [POST] + params: + key: X-API-Key + in: header +EOF +``` + +The vendor credentials under `spec.upstream.auth` are added only when the provider calls its external service. + +## Step 2: Create Provider Loopback Keys + +Because both providers in this example use the `api-key-auth` policy, create an API key for each provider. The proxy uses these keys when routing to the providers through the gateway's internal loopback route. + +```bash +OPENAI_LOOPBACK_KEY=$(curl -s -X POST \ + http://localhost:9090/api/management/v1/llm-providers/openai-provider/api-keys \ + -u admin:admin \ + -H "Content-Type: application/json" \ + -d '{"name":"openai-proxy-loopback"}' \ + | jq -r '.apiKey.apiKey') + +ANTHROPIC_LOOPBACK_KEY=$(curl -s -X POST \ + http://localhost:9090/api/management/v1/llm-providers/anthropic-provider/api-keys \ + -u admin:admin \ + -H "Content-Type: application/json" \ + -d '{"name":"anthropic-proxy-loopback"}' \ + | jq -r '.apiKey.apiKey') +``` + +Verify that both commands returned a value: + +```bash +test -n "$OPENAI_LOOPBACK_KEY" && test "$OPENAI_LOOPBACK_KEY" != "null" +test -n "$ANTHROPIC_LOOPBACK_KEY" && test "$ANTHROPIC_LOOPBACK_KEY" != "null" +``` + +API key values are returned only when they are created or regenerated. Store them securely. + +## Step 3: Deploy the Multi-Provider LLM Proxy + +The following proxy exposes one `/chat/completions` operation. OpenAI is the primary and default provider. Anthropic is an additional selectable provider with an inline request and response transformer. + +```bash +curl -X POST http://localhost:9090/api/management/v1/llm-proxies \ + -u admin:admin \ + -H "Content-Type: application/yaml" \ + --data-binary @- < + transformer: + type: openai-to-azure-openai + version: v1 + params: + model: gpt-4o + apiVersion: "2024-02-15-preview" +``` + +### Mistral + +```yaml +- id: mistral-provider + auth: + type: api-key + header: X-API-Key + value: + transformer: + type: openai-to-mistral + version: v1 + params: + model: mistral-large-latest +``` + +### Gemini + +```yaml +- id: gemini-provider + auth: + type: api-key + header: X-API-Key + value: + transformer: + type: openai-to-gemini + version: v1 + params: + model: gemini-2.5-flash + apiVersion: v1beta +``` + +### AWS Bedrock + +```yaml +- id: aws-bedrock-provider + auth: + type: api-key + header: X-API-Key + value: + transformer: + type: openai-to-aws-bedrock + version: v1 + params: + model: anthropic.claude-3-5-sonnet-20240620-v1:0 +``` + +For example, the matching router entries are: + +```yaml +mappings: + - headerValue: azure-openai + provider: azure-openai-provider + - headerValue: mistral + provider: mistral-provider + - headerValue: gemini + provider: gemini-provider + - headerValue: aws-bedrock + provider: aws-bedrock-provider +``` + +## Use Provider Aliases + +Use `as` when the logical upstream name used by routing policies should differ from the deployed provider ID: + +```yaml +additionalProviders: + - id: anthropic-provider + as: anthropic-upstream + auth: + type: api-key + header: X-API-Key + value: + transformer: + type: openai-to-anthropic + version: v1 + params: + model: claude-sonnet-4-5-20250929 +``` + +When an alias is present, router mappings must select the alias, not the provider ID: + +```yaml +mappings: + - headerValue: anthropic + provider: anthropic-upstream +``` + +The alias must: + +- Contain only letters, numbers, hyphens, or underscores +- Be between 1 and 100 characters +- Be unique within the proxy +- Not match the primary provider ID or another additional provider's effective name + +## Configuration Reference + +### `additionalProviders` + +| Field | Required | Description | +|-------|----------|-------------| +| `id` | Yes | ID of an already deployed `LlmProvider` | +| `as` | No | Logical upstream name used by routing policies; defaults to `id` | +| `auth` | No | API key authentication used by the proxy when calling the provider's internal route | +| `transformer` | No | Request and response transformer applied only when this provider is selected | + +### `transformer` + +| Field | Required | Description | +|-------|----------|-------------| +| `type` | Yes | Installed transformer policy name, such as `openai-to-anthropic` | +| `version` | Yes | Major policy version, such as `v1` | +| `params` | No | Transformer-specific parameters, such as `model` or `apiVersion` | + +### `openai-header-router` parameters + +| Parameter | Required | Default | Description | +|-----------|----------|---------|-------------| +| `headerName` | No | `x-provider` | Request header used for selection | +| `defaultProvider` | Yes | None | Effective provider name selected when no mapping matches | +| `mappings` | Yes | None | Header value to effective provider name mappings; the first match wins | + +## Validation and Troubleshooting + +### The additional provider is not found + +Deploy every provider before deploying the proxy. Each `additionalProviders[].id` must match the `metadata.name` of an existing `LlmProvider`. + +### The proxy reports a duplicate upstream name + +Every effective provider name must be unique. The effective name is `as` when it is configured; otherwise, it is `id`. It must not collide with the primary provider ID. + +### The transformer is rejected during deployment + +Make sure that: + +- `transformer.type` names a transformer supported by your AI Gateway version. +- `transformer.version` uses a major-only version such as `v1`. +- All parameters required by that transformer are present. + +The gateway resolves the major version to an installed full policy version and rejects invalid transformer configuration during deployment. + +### The request always reaches the default provider + +Check that: + +- The routing policy is attached to the same path and method being invoked. +- The request uses the header configured by `headerName`. +- The header value matches a `mappings[].headerValue`. +- The mapping's `provider` matches the additional provider's `as` value when an alias is configured; otherwise, it matches `id`. + +An unknown header value intentionally falls back to `defaultProvider`. + +### The provider returns `401 Unauthorized` + +Confirm which authentication layer rejected the request: + +- A rejection at the proxy usually means the proxy consumer key is missing or invalid. +- A rejection on the provider's loopback route usually means `provider.auth` or `additionalProviders[].auth` contains an invalid provider API key. +- A rejection from the external vendor usually means `LlmProvider.spec.upstream.auth` contains an invalid vendor credential or uses the wrong header format. + +### The configured transformer is not supported + +The AI Gateway distribution includes the router and transformer policies supported by that version. Use a supported `transformer.type` and major version, or upgrade the AI Gateway to a version that includes the required transformer. + +## Security Recommendations + +- Store vendor credentials and loopback keys in a secret manager or Kubernetes `Secret` instead of committing plain-text values. +- Protect the proxy with an authentication policy so applications cannot invoke it anonymously. +- Expose only required provider operations through `accessControl`. +- Apply rate limiting and guardrails at the provider or proxy level according to your governance requirements. +- Use explicit router mappings. Do not accept a client-provided value as an unrestricted upstream name. + +## Complete Example + +For a larger configuration containing OpenAI, Anthropic, Azure OpenAI, Mistral, and Gemini, see [`gateway/examples/openai-multi-provider-proxy.yaml`](https://github.com/wso2/api-platform/blob/main/gateway/examples/openai-multi-provider-proxy.yaml). diff --git a/en/docs/ai-gateway/next/overview.md b/en/docs/ai-gateway/next/overview.md index 642e38f38..78bdb2bdb 100644 --- a/en/docs/ai-gateway/next/overview.md +++ b/en/docs/ai-gateway/next/overview.md @@ -8,7 +8,7 @@ tags: - llm - mcp author: WSO2 API Platform Documentation Team -last_updated: 2026-07-08 +last_updated: 2026-07-13 content_type: "concept" --- @@ -23,31 +23,6 @@ A gateway for managing and securing AI traffic, including Large Language Model ( ## Key Concepts -### LLM Provider - -An LLM Provider represents a managed connection to an upstream AI service, such as OpenAI, Azure OpenAI, or any other LLM API. Platform administrators configure LLM Providers to define enterprise-wide connectivity, governance, and runtime controls, including: - -- The upstream LLM service endpoint -- Authentication credentials (API keys, OAuth tokens, etc.) -- Access control rules for exposed models and endpoints -- Budget and cost control policies, such as token-based rate limiting -- Enterprise-wide guardrails and runtime policies - -Once an LLM Provider is configured and deployed to the AI Gateway, it exposes a managed endpoint that applications can use to securely access the upstream LLM service. - -### App LLM Proxy - -An App LLM Proxy provides an application-specific entry point to an LLM Provider. While the LLM Provider enforces enterprise-wide governance, App LLM Proxies allow application teams to configure application-specific behavior, such as guardrails, prompt decorators, prompt templates, model parameters, and other runtime policies. - -Every App LLM Proxy is associated with an LLM Provider and inherits its administrator-defined access controls, budget limits, and enterprise-wide policies. Each proxy exposes its own URL path (for example, /assistant) and can apply additional application-specific policies without overriding enterprise-wide policies enforced by the platform administrator. - -This enables: - -- Multiple AI applications to securely share a single LLM Provider -- Application-specific guardrails, prompt management, and runtime policies -- Enterprise-wide governance with application-level customization -- Clear separation of responsibilities between platform administrators and application developers - ### LLM Provider Template An LLM Provider Template defines the characteristics and behaviors specific to an AI service provider, such as OpenAI, Azure OpenAI, or other LLM platforms. It describes how the gateway should interpret and extract usage and operational metadata, including prompt, completion, total, and remaining token information, as well as request and response model metadata. @@ -61,6 +36,28 @@ Following templates are shipped out-of-the-box - Azure AI Foundry - Gemini +### LLM Provider + +An LLM Provider represents a connection to an AI backend service such as OpenAI, Azure OpenAI, or other LLM APIs. Platform administrators configure LLM Providers to define: + +- The LLM Provider Template +- The upstream LLM service URL +- Authentication credentials (API keys, tokens) +- Access control rules for which endpoints are exposed +- Budget control policies, such as token-based rate limiting +- Organization-wide policies such as guardrails + +Once configured, the LLM Provider allows traffic to flow through the gateway to the AI backend. + +### LLM Proxy + +An LLM Proxy allows developers to create custom API endpoints that consume an LLM Provider, while inheriting administrator-enforced access control, budgeting and organization-wide policies defined at the provider level. Each proxy gets its own URL context (e.g., `/assistant`) and can have its own policies applied. This enables: + +- Multiple AI applications to share a single LLM Provider +- A single OpenAI-compatible endpoint to route requests to multiple LLM providers. See [Multi-Provider Routing for LLM Proxies](llm-proxy/multi-provider-routing.md). +- Per-application policies such as prompt management and guardrails +- Separation between platform administration and application development + ### MCP Proxy An MCP Proxy routes Model Context Protocol traffic to MCP servers. MCP is a protocol that enables AI assistants to interact with external tools and data sources. With MCP Proxies, you can: @@ -105,7 +102,7 @@ An MCP Proxy routes Model Context Protocol traffic to MCP servers. MCP is a prot AI Guardrails allow you to enforce safety, content, and compliance policies on AI traffic flowing through the AI Gateway. They can be applied at the LLM Provider level (organization-wide), at the LLM Proxy level (per-application), or on MCP Proxies. -The complete and up-to-date guardrail catalogue — with configuration references and examples — is maintained in the gateway-controllers repository: [https://github.com/wso2/gateway-controllers/blob/main/docs/README.md](https://github.com/wso2/gateway-controllers/blob/main/docs/README.md) +The complete and up-to-date guardrail catalogue — with configuration references and examples — is maintained in the gateway-controllers repository: [gateway-controllers documentation](https://github.com/wso2/gateway-controllers/blob/main/docs/README.md) You can extend the AI Gateway with custom guardrail policies by building a custom gateway image using the `ap` CLI. See [Customizing the Gateway by Adding and Removing Policies](../../tools/cli/customizing-gateway-policies.md). diff --git a/en/mkdocs.yml b/en/mkdocs.yml index 9cbf70a9f..be19122ce 100644 --- a/en/mkdocs.yml +++ b/en/mkdocs.yml @@ -543,6 +543,7 @@ nav: - LLM Proxy: - Quick Start Guide: ai-gateway/next/llm-proxy/quick-start-guide.md - LLM Provider Templates: ai-gateway/next/llm-proxy/llm-templates.md + - Multi-Provider Routing: ai-gateway/next/llm-proxy/multi-provider-routing.md - Guardrails: - AWS Bedrock Guardrail: ai-gateway/next/llm-proxy/guardrails/aws-bedrock-guardrail.md - Azure Content Safety: ai-gateway/next/llm-proxy/guardrails/azure-content-safety.md From e12717e20a121529308515afedf56180664594f2 Mon Sep 17 00:00:00 2001 From: Aakash Wijesekara Date: Mon, 13 Jul 2026 17:16:23 +0530 Subject: [PATCH 2/3] Rename header router in documentation --- .../next/llm-proxy/multi-provider-routing.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md index 5afb79a53..78184f786 100644 --- a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md +++ b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md @@ -34,7 +34,7 @@ A multi-provider LLM proxy has: - One primary provider in `spec.provider` - One or more selectable providers in `spec.additionalProviders` -- An `openai-header-router` policy that selects a provider from a request header +- An LLM Header Router policy (`openai-header-router`) that selects a provider from a request header - An inline transformer for each additional provider that does not use the OpenAI wire format The request flow is: @@ -46,7 +46,7 @@ OpenAI-compatible client request v Multi-provider LLM proxy | - | openai-header-router selects anthropic-provider + | LLM Header Router selects anthropic-provider | openai-to-anthropic transforms the request | provider loopback authentication is added v @@ -336,7 +336,7 @@ Header names and mapped header values are matched case-insensitively. Leading an ## Add More Providers -Add each selectable provider under `additionalProviders`, then add a corresponding mapping under `openai-header-router`. +Add each selectable provider under `additionalProviders`, then add a corresponding mapping under the LLM Header Router policy (`openai-header-router`). ### Azure OpenAI @@ -467,7 +467,9 @@ The alias must: | `version` | Yes | Major policy version, such as `v1` | | `params` | No | Transformer-specific parameters, such as `model` or `apiVersion` | -### `openai-header-router` parameters +### LLM Header Router parameters + +Use `openai-header-router` as the policy name in the configuration. | Parameter | Required | Default | Description | |-----------|----------|---------|-------------| From d0df7086dd862856026c38f76ff7dc0619278392 Mon Sep 17 00:00:00 2001 From: Aakash Wijesekara Date: Thu, 16 Jul 2026 11:04:23 +0530 Subject: [PATCH 3/3] Change id as providerId --- en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md index 78184f786..d78787552 100644 --- a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md +++ b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md @@ -266,7 +266,7 @@ spec: EOF ``` -The controller automatically passes the additional provider's effective upstream name to its transformer. Do not add an `id` under `transformer.params`; it is injected from `additionalProviders[].id` or `additionalProviders[].as`. +The controller automatically passes the additional provider's effective upstream name to its transformer. Do not add a `providerId` under `transformer.params`; it is injected from `additionalProviders[].id` or `additionalProviders[].as`. ## Step 4: Create a Proxy Consumer Key