[vLLM backend][Feature Request] Support vLLM-Omni for native audio/video input

[SlackerProMax]

Problem

The vLLM backend in the official Triton image
(nvcr.io/nvidia/tritonserver:*-vllm-python-py3) only supports the image
modality. In backends/vllm/utils/request.py, the only multimodal branch is
multi_modal_data={"image": ...} — there is no video or audio support.
So even though the vLLM engine supports them, Triton users can't pass audio/video.

Request

Please add vLLM-Omni support to the
official Triton vLLM backend, so Triton can natively serve omni-modal models
(e.g. Qwen3-Omni) with video and audio in multi_modal_data, not just images.
An official omni example model_repository would also help.

Current workaround

Custom Python pre-processing that extracts video frames with ffmpeg and sends
them as multiple images — loses native temporal/audio handling and inflates tokens.

Context

• vLLM-Omni supports text/image/audio/video (v0.24.0, July 2026): https://github.com/vllm-project/vllm-omni
• Env: tritonserver:26.06-vllm-python-py3, Qwen3-VL-8B, single-node GPU

[protonicage]

I would like to see this feature as well to support ASR models.

[protonicage]

Its actually fairly easy to implement by yourself by modfying request.py and the model.py backend. just at the audio input masks and it works out of the box

[SlackerProMax]

Its actually fairly easy to implement by yourself by modfying request.py and the model.py backend. just at the audio input masks and it works out of the box

I understand it works with a local patch, but that is exactly what I’d like to avoid relying on.
For production, modifying request.py and the backend model.py directly means we own that patch forever, including every future upstream upgrade and compatibility break. A supported upstream change or extension point would be much safer.