Please select the area the issue is related to
Area/AIGateway (AI Gateway runtime/control plane)
Please select the aspect the issue is related to
Aspect/Configuration (Config files, settings, env vars, defaults)
Suggested Improvement
Current limitation:
Using more than one LLM vendor from a single app currently requires a separate
proxy and endpoint per vendor, with requests formatted differently for each.
There's no way to route one OpenAI-shaped endpoint to multiple upstreams. If
you hand-roll this today, the translator policy has to be declared separately
from the provider it targets, duplicating the provider id by hand and keeping
it in sync manually.
Suggested improvement:
Add additionalProviders to the LLM proxy spec so a single proxy can fan out
to multiple LLM providers behind one endpoint, with the target selected
per-request by a header (openai-header-router → metadata.selected_provider).
Each additional provider carries its own upstream auth and an inline
transformer, declared together under additionalProviders[].transformer,
so the provider id is injected automatically instead of being duplicated by
hand. Client requests and responses stay OpenAI-shaped regardless of which
vendor is selected.
Scope:
- gateway-controller: schema, validation, and policy injection for
additionalProviders + inline transformer
- platform-api: management API and deployment artifact support
- kubernetes-operator:
LlmProxy CRD support
Translator policy implementations (openai-to-anthropic/azure-openai/mistral/gemini)
and the openai-header-router policy are tracked separately as a follow-up.
Related Issues
No response
Please select the area the issue is related to
Area/AIGateway (AI Gateway runtime/control plane)
Please select the aspect the issue is related to
Aspect/Configuration (Config files, settings, env vars, defaults)
Suggested Improvement
Current limitation:
Using more than one LLM vendor from a single app currently requires a separate
proxy and endpoint per vendor, with requests formatted differently for each.
There's no way to route one OpenAI-shaped endpoint to multiple upstreams. If
you hand-roll this today, the translator policy has to be declared separately
from the provider it targets, duplicating the provider id by hand and keeping
it in sync manually.
Suggested improvement:
Add
additionalProvidersto the LLM proxy spec so a single proxy can fan outto multiple LLM providers behind one endpoint, with the target selected
per-request by a header (
openai-header-router→metadata.selected_provider).Each additional provider carries its own upstream auth and an inline
transformer, declared together underadditionalProviders[].transformer,so the provider id is injected automatically instead of being duplicated by
hand. Client requests and responses stay OpenAI-shaped regardless of which
vendor is selected.
Scope:
additionalProviders+ inlinetransformerLlmProxyCRD supportTranslator policy implementations (openai-to-anthropic/azure-openai/mistral/gemini)
and the openai-header-router policy are tracked separately as a follow-up.
Related Issues
No response