Summary
The capability matrix (providers/capabilities/matrix.go) models OpenAI chat sampling parameters — temperature, top_p, tools, response_format, and so on. It has no concept of input modality, so it cannot express "this provider cannot accept images".
Why this matters
Multimodal translation itself is correct: every provider that needs explicit handling (Anthropic, Bedrock Anthropic/Llama/Nova, Cohere, Gemini) maps image_url parts to its native shape, and the OpenAI-compatible providers forward ContentParts through core.Message marshalling. That was fixed in v1.1.11 (#286).
What is still missing is the declaration. A text-only provider has no way to say so, so an image sent to one is handled by whatever that provider's API happens to do with an unexpected content array, rather than by the gateway's own on_unsupported_param policy. Vision support is advertised uniformly across providers that do not uniformly deliver it.
Proposed implementation
- Add a modality dimension to the capability matrix (e.g. accepted input content types per provider), alongside the existing parameter dimension.
- Have
core.EnforceUnsupportedParams — or a sibling check on the same matrix — apply compatibility.on_unsupported_param (warn | drop | reject) to an image part sent to a provider that cannot accept one, so the behaviour is the configured one rather than provider-defined.
- Serialize the modality dimension through
GET /v1/capabilities so callers can discover which providers accept images before routing to them.
- Add tests asserting a text-only provider rejects (or warns/drops, per policy) an
image_url part instead of forwarding it.
Follow-up to #286 (translation, fixed in v1.1.11) and #207 (capability matrix, shipped in v1.2.0).
Summary
The capability matrix (
providers/capabilities/matrix.go) models OpenAI chat sampling parameters —temperature,top_p,tools,response_format, and so on. It has no concept of input modality, so it cannot express "this provider cannot accept images".Why this matters
Multimodal translation itself is correct: every provider that needs explicit handling (Anthropic, Bedrock Anthropic/Llama/Nova, Cohere, Gemini) maps
image_urlparts to its native shape, and the OpenAI-compatible providers forwardContentPartsthroughcore.Messagemarshalling. That was fixed in v1.1.11 (#286).What is still missing is the declaration. A text-only provider has no way to say so, so an image sent to one is handled by whatever that provider's API happens to do with an unexpected content array, rather than by the gateway's own
on_unsupported_parampolicy. Vision support is advertised uniformly across providers that do not uniformly deliver it.Proposed implementation
core.EnforceUnsupportedParams— or a sibling check on the same matrix — applycompatibility.on_unsupported_param(warn | drop | reject) to an image part sent to a provider that cannot accept one, so the behaviour is the configured one rather than provider-defined.GET /v1/capabilitiesso callers can discover which providers accept images before routing to them.image_urlpart instead of forwarding it.Follow-up to #286 (translation, fixed in v1.1.11) and #207 (capability matrix, shipped in v1.2.0).