Skip to content

[Improvement]: Support multi-provider routing for LLM proxies #2490

Description

@Aakashwije

Please select the area the issue is related to

Area/AIGateway (AI Gateway runtime/control plane)

Please select the aspect the issue is related to

Aspect/Configuration (Config files, settings, env vars, defaults)

Suggested Improvement

Current limitation:
Using more than one LLM vendor from a single app currently requires a separate
proxy and endpoint per vendor, with requests formatted differently for each.
There's no way to route one OpenAI-shaped endpoint to multiple upstreams. If
you hand-roll this today, the translator policy has to be declared separately
from the provider it targets, duplicating the provider id by hand and keeping
it in sync manually.

Suggested improvement:
Add additionalProviders to the LLM proxy spec so a single proxy can fan out
to multiple LLM providers behind one endpoint, with the target selected
per-request by a header (openai-header-routermetadata.selected_provider).
Each additional provider carries its own upstream auth and an inline
transformer, declared together under additionalProviders[].transformer,
so the provider id is injected automatically instead of being duplicated by
hand. Client requests and responses stay OpenAI-shaped regardless of which
vendor is selected.

Scope:

  • gateway-controller: schema, validation, and policy injection for
    additionalProviders + inline transformer
  • platform-api: management API and deployment artifact support
  • kubernetes-operator: LlmProxy CRD support

Translator policy implementations (openai-to-anthropic/azure-openai/mistral/gemini)
and the openai-header-router policy are tracked separately as a follow-up.

Related Issues

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions