Add voice message support for Telegram/WhatsApp

[osisdie]

Overview

Both Telegram and WhatsApp support voice messages, but the current brokers only handle text and image messages. Adding voice support would significantly improve the mobile experience.

What to do

• Download voice message audio file from platform API
• Convert OGG/OPUS to a format suitable for transcription
• Transcribe audio to text (options: Whisper API, Claude vision, or platform-native STT)
• Pass transcribed text to Claude Code as the user message
• Reply with the transcription + Claude's response

Relevant files

• external_plugins/whatsapp-channel/broker-relay.ts — WhatsApp broker
• external_plugins/line-channel/broker-relay.ts — LINE broker (similar pattern for reference)

Notes

• WhatsApp sends voice as audio/ogg with codec opus
• Telegram sends voice as .oga files
• Consider adding a VOICE_TRANSCRIPTION_PROVIDER env var for flexibility

[thepandeyakash]

Hi, I’d like to work on this issue.

Plan:

• Add voice message detection in broker-relay

• Download and transcribe audio (using configurable provider via env)

• Inject transcription into Claude pipeline

• Add fallback handling when transcription is not configured

Question:
Do you have a preferred transcription provider (e.g., Whisper) or should I keep it provider-agnostic via env config?

[osisdie]

Hi @thepandeyakash, thanks for jumping in — your plan looks solid and well thought out.

Re: transcription provider — keeping it provider-agnostic via VOICE_TRANSCRIPTION_PROVIDER is the right call. For the initial implementation, I'd recommend starting with the Hugging Face Inference API using openai/whisper-large-v3. The free tier makes it easy to get started without any cost barrier. Down the road we can add more providers — Groq Whisper is another great option (very fast and affordable), and OpenAI Whisper as well.

A few thoughts on scope for the first PR:

• Starting with Telegram voice messages (.oga files) would be ideal — the Telegram broker is more mature and easier to validate end-to-end
• WhatsApp voice support can follow as a second PR
• When no transcription provider is configured, a friendly fallback message like "Voice messages require a transcription provider. Please send text instead." would keep things graceful

Key files for reference:

• external_plugins/telegram-channel/ — this is where voice detection would go
• external_plugins/line-channel/broker-relay.ts — good reference for message type handling patterns

We also have a few contributors in the community who are interested in collaborating on this project. If you'd like, I'm happy to connect you so you can pair up or split the work across Telegram and WhatsApp. No pressure — just wanted you to know you're not working alone here.

Branch off main and open a PR whenever you're ready. Happy to help if you run into anything along the way.

[thepandeyakash]

Hi @thepandeyakash, thanks for jumping in — your plan looks solid and well thought out.

Re: transcription provider — keeping it provider-agnostic via VOICE_TRANSCRIPTION_PROVIDER is the right call. For the initial implementation, I'd recommend starting with the Hugging Face Inference API using openai/whisper-large-v3. The free tier makes it easy to get started without any cost barrier. Down the road we can add more providers — Groq Whisper is another great option (very fast and affordable), and OpenAI Whisper as well.

A few thoughts on scope for the first PR:

• Starting with Telegram voice messages (.oga files) would be ideal — the Telegram broker is more mature and easier to validate end-to-end
• WhatsApp voice support can follow as a second PR
• When no transcription provider is configured, a friendly fallback message like "Voice messages require a transcription provider. Please send text instead." would keep things graceful

Key files for reference:

• external_plugins/telegram-channel/ — this is where voice detection would go
• external_plugins/line-channel/broker-relay.ts — good reference for message type handling patterns

We also have a few contributors in the community who are interested in collaborating on this project. If you'd like, I'm happy to connect you so you can pair up or split the work across Telegram and WhatsApp. No pressure — just wanted you to know you're not working alone here.

Branch off main and open a PR whenever you're ready. Happy to help if you run into anything along the way.

Thanks! I checked the repo but I don’t see a telegram-channel under external_plugins yet.

Should I:

1. Add voice support to an existing channel (e.g., WhatsApp ) for now, or
2. Scaffold a new telegram-channel module and implement voice handling there?

[osisdie]

Hi @thepandeyakash, Good catch on the directory structure! To clarify — the Telegram channel is already implemented, but it lives under .claude/channels/telegram/ as a built-in Claude Code plugin, not under external_plugins/. The external_plugins/ directory is for channels that require a separate broker relay service (WhatsApp, LINE, Slack, Teams).

I originally suggested Telegram first, but after reviewing the architecture more closely, I'd actually recommend starting with WhatsApp instead. Here's why:

• The WhatsApp broker-relay (external_plugins/whatsapp-channel/broker-relay.ts) already detects audio message types and downloads .ogg files — you'd mainly need to add the transcription step on top of that existing flow
• Telegram voice support involves the plugin architecture internally, which is a different integration path and harder to contribute to externally

Sorry for the pivot — WhatsApp will be a much smoother first PR. Let me know if you have any questions about the broker structure!