request_content_type=multipart/form-data should be supported for ASR models like standlalone whisper

[Maze999]

Hi,

When benchmarking non omni models like standalone whisper i see that manual curl command works with request_content_type=multipart/form-data

curl -s http://x.x.x.x/v1/audio/transcriptions -H "Content-Type: multipart/form-data" -F "file=@gold_000_line_000.wav" -F model="openai/whisper-large-v3"
{"text":" שלום צליל אברהם.","usage":{"type":"duration","seconds":2}}

but when trying to simulate it through aiperf by leveraging https://docs.nvidia.com/aiperf/tutorials/model-endpoint-guides/template-endpoint it is hitting a validation error where request_content_type=multipart/form-data is being validated for image_edit, video_generation

aiperf profile
--model openai/whisper-large-v3
--endpoint /v1/audio/transcriptions
--endpoint-type template
--request-content-type multipart/form-data
--extra-inputs 'payload_template:{"file": {{ audio|tojson }}, "model": {{ model|tojson }}}'
--extra-inputs 'response_field:text'
--input-file /persistdir/whisper/saspeech_gold_standard/wavs/inputs.jsonl
--custom-dataset-type single_turn
--url 'http://x.x.x.x'
--request-count 20
--concurrency 4
╭─ Error Running AIPerf System ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ 1 validation error for AIPerfConfig │
│ benchmark.endpoint │
│ Value error, request_content_type=multipart/form-data is only supported for endpoint types that accept form-data (e.g. image_edit, video_generation); endpoint │
│ type template does not. [type=value_error, input_value={'urls': ['http://x.x.x... '{{ model|tojson }}}'}}, input_type=dict] │
│ For further information visit https://errors.pydantic.dev/2.13/v/value_error │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Available routes:

(APIServer pid=1) INFO 06-17 05:42:58 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=1) INFO 06-17 05:42:58 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=1) INFO 06-17 05:42:58 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=1) INFO 06-17 05:42:58 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=1) INFO 06-17 05:42:58 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=1) INFO 06-17 05:42:58 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=1) INFO 06-17 05:42:58 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=1) INFO 06-17 05:42:58 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=1) INFO 06-17 05:42:58 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=1) INFO 06-17 05:42:58 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=1) INFO 06-17 05:42:58 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=1) INFO 06-17 05:42:58 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=1) INFO 06-17 05:42:58 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=1) INFO 06-17 05:42:58 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=1) INFO 06-17 05:42:58 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=1) INFO 06-17 05:42:58 [launcher.py:46] Route: /v1/audio/translations, Methods: POST

Please let me know if any other feasible way exists with aiperf to simulate the above benchmarking, if not please consider supporting --request-content-type multipart/form-data for standalone ASR models like whisper in the upcoming releases

Kindly let me know if any further details are needed

[lkomali]

Hi @Maze999 thanks for filing this. The error message is actually misleading here — fixing the validator won't make this work.

The issue is deeper: when template endpoint uses multipart, the transport only sends actual binary bytes if the JSON field looks like {"b64_data": "...", "filename": "audio.wav", "content_type": "audio/wav"}. {{ audio|tojson }} doesn't produce that shape, so the audio would get sent as a plain string and Whisper would reject it.

No real workaround today — the template path just can't do binary file uploads cleanly.

What actually needs to happen is a dedicated audio_transcription endpoint type that would handle the multipart assembly properly and also let you use the built-in ASR datasets (librispeech, voxpopuli, etc.) instead of a custom input file.

Tracking this as a feature — will follow up here when there's something to test.

[Maze999]

@lkomali
Thank you for looking into this, providing your inputs and considering it for a new feature

Tracking this as a feature — will follow up here when there's something to test.
Sure, i will help validate