Yes — several of the LLM APIs your firewall intercepts accept audio directly, and your own code already encounters it. That's actually why the question matters for llm-fw:
Providers that accept audio input:
- OpenAI — gpt-4o-audio-family models take input_audio content parts (base64 wav/mp3) in regular /chat/completions requests. Your parsers.ts:328 already extracts exactly these
blocks — that parser branch exists because this traffic is real. There's also the Realtime API (streaming voice over WebSocket) and the Whisper transcription endpoints.
- Google Gemini — natively understands audio: inlineData blocks with audio/* mime types (or via the Files API). It can transcribe, summarize, and answer questions about a clip —
which means it will also happily follow instructions spoken inside the clip.
- Mistral (Voxtral models) and Groq (hosted Whisper for transcription) also take audio.
The notable exception is Anthropic — I checked the current API reference rather than answering from memory: the Claude API does not accept audio input. Content blocks are text,
images, and documents (PDFs); the model capability tree has image_input but no audio equivalent. Voice features in Claude apps work by transcribing speech to text on the client
before it ever reaches the API.
So the attack surface from the previous answer is concrete: a user pastes a voice memo into a tool backed by GPT-4o-audio or Gemini, the audio part sails through your proxy as
an opaque base64 blob (audited under nonText, default audit mode), and the model on the other end hears "ignore your previous instructions…" that no stage of your pipeline ever
saw as text. For Anthropic-bound traffic, there's nothing to defend yet — audio can't get in.
Yes — several of the LLM APIs your firewall intercepts accept audio directly, and your own code already encounters it. That's actually why the question matters for llm-fw:
Providers that accept audio input:
blocks — that parser branch exists because this traffic is real. There's also the Realtime API (streaming voice over WebSocket) and the Whisper transcription endpoints.
which means it will also happily follow instructions spoken inside the clip.
The notable exception is Anthropic — I checked the current API reference rather than answering from memory: the Claude API does not accept audio input. Content blocks are text,
images, and documents (PDFs); the model capability tree has image_input but no audio equivalent. Voice features in Claude apps work by transcribing speech to text on the client
before it ever reaches the API.
So the attack surface from the previous answer is concrete: a user pastes a voice memo into a tool backed by GPT-4o-audio or Gemini, the audio part sails through your proxy as
an opaque base64 blob (audited under nonText, default audit mode), and the model on the other end hears "ignore your previous instructions…" that no stage of your pipeline ever
saw as text. For Anthropic-bound traffic, there's nothing to defend yet — audio can't get in.