gpt-4o-transcribe uses first selected language

[perrette]

I have a four language input setting with the first input happening to be German. That is by chance, because I didn't consider the order. The mini version of the model, as well as Whisper, handles that just fine, with most common input being English or French, but the full-fledged gpt-4o-transcribe version misses it and expects strictly German, resulting in messed up transcriptions.

I don't know if it is a bug in the app or in the language model on the back-end side.

[grapefruit89]

Hey @perrette, thanks for reporting this!

I really like this app and want to help the developer avoid burnout, so I try to jump in and help out with troubleshooting wherever I can!

Note

(Full transparency: I used the Google Antigravity desktop version equipped with MCP servers to clone and deeply analyze this repo locally, which helped me quickly trace the architecture for this proposal.)

I looked into the source code, and this is absolutely not a bug on your end, nor is it technically a bug in the backend. It's a fascinating quirk of how the new gpt-4o-transcribe (multimodal) model interprets instructions compared to the classic Whisper model!

The Technical Explanation

When you use Whisper, Dictate passes the active keyboard language (German) to the API via the language="de" parameter. Whisper treats this as a "gentle hint" but relies heavily on its own acoustic auto-detection. If it clearly hears French, it ignores the German hint and outputs French.

However, gpt-4o-transcribe doesn't use the standard Audio API. It uses the Chat API with multimodal audio inputs. To give the model a hint, Dictate dynamically generates a text prompt in DictateController.kt (buildChatAudioInstruction):

// ...
DictateLanguages.englishNameFor(langCode)?.takeIf { it.isNotBlank() }
    ?.let { parts.add("The audio is spoken in $it.") }
// ...

Because GPT-4o is a highly compliant reasoning model, when it receives the absolute text instruction "The audio is spoken in German", it assumes you are stating a strict fact! Instead of auto-detecting, it attempts to follow the instruction by either transcribing French phonetically into German-sounding gibberish, or by directly translating your French speech into German text.

The Fix (@DevEmperor)

The prompt instruction for the single-call multimodal path (chatAudio) needs to be softened so it doesn't override GPT-4o's excellent zero-shot language detection.

In DictateController.kt -> buildChatAudioInstruction(), we should change:
parts.add("The audio is spoken in $it.")
to:
parts.add("The audio is likely spoken in $it, but always transcribe in the language actually spoken.")

This slight semantic tweak will give GPT-4o the context it needs without forcing it to strictly obey the keyboard's layout language.

As a temporary workaround until this is patched: You can go into the Dictate settings and explicitly set the input language to Auto-Detect instead of relying on the keyboard's active layout. This will skip adding the language instruction entirely!

P.S. @DevEmperor I want to emphasize that my intention here is not to act like a know-it-all or dictate how you should develop your app. I'm simply a huge fan of this project, it's one of the most important apps I use daily, and I just want to help bring visibility to these issues so you can focus on the core development! "Why should I still type when there is Dictate?" – this app is probably the reason why phones will shrink again in the next 10 years!