[Feature Request] Real-Time Streaming Transcription (Live Partial Results) for Local Models

[grapefruit89]

Is your feature request related to a problem? Please describe.

Currently, DictateKeyboard operates in a "batch" mode: the user presses record, speaks their sentence or paragraph, stops recording, and then has to wait for the transcription to process and output as a single, large block of text.

This creates a significant disconnect for the user. While speaking, you have no visual feedback on whether the engine is actually understanding you correctly. If a word was misunderstood at the beginning of a long dictation, you only find out at the very end.

Describe the solution you'd like

It would be a massive UX improvement (the "holy grail" of dictation) to implement Live Streaming Transcription. The text should appear in the input field while the user is speaking, updating in real-time.

Since Dictate uses Sherpa-ONNX for local inference, this is technically highly feasible. The codebase already contains a fully functional streaming infrastructure (RealtimeSession, pcmSink, setDictationPreview), which is currently only being utilized for Cloud APIs. The goal is to connect this existing streaming pipeline with the local Sherpa-ONNX backend.

Note

🤖 AI-Assisted Research
The following architectural concept and codebase analysis were conducted with the help of Google Antigravity. I lack the deep, low-level Android programming expertise to figure this out on my own, but I really want to help support the development of this amazing app! I hope this research and the proposed code snippets save you some time and effort. ❤️

🛠️ Proposed Architectural Changes

1. Requirement: Streaming Models
The Sherpa-ONNX OfflineRecognizer (e.g., Whisper) cannot stream natively. To achieve live partial results, a Streaming Model (e.g., Zipformer or Streaming Paraformer) is strictly required.

2. Model Catalog & Config (LocalModelCatalog.kt)

• [MODIFY] Add at least one streaming model (e.g., sherpa-onnx-streaming-zipformer-en-...) to the catalog.
• These models typically consist of an Encoder (encoder.onnx), Decoder (decoder.onnx), and Joiner (joiner.onnx), requiring the OnlineRecognizer from Sherpa-ONNX.

3. Implementing the LocalRealtimeSession Wrapper (LocalRealtimeSession.kt)

• [NEW] Create a new class LocalRealtimeSession : RealtimeSession to wrap the Sherpa-ONNX OnlineRecognizer:
  • sendAudio(pcm16: ByteArray, len: Int): Feed the samples using stream.acceptWaveform(). In a while (recognizer.isReady(stream)) loop, call recognizer.decode(stream). Fetch the intermediate text using recognizer.getResult(stream).text. If the text has changed, fire callbacks.onPartial(text).
  • finish(): Call stream.inputFinished(). Decode the remaining buffer. Fire callbacks.onFinalSegment(text).

4. Adapting the Controller (DictateController.kt)

• [MODIFY] isRealtimeActive(): If the provider is local, check if the installed model is a Streaming Model. If yes, return true.
• [MODIFY] openRealtimeSession(appContext: Context): If a local streaming model is selected, instantiate LocalRealtimeSession instead of calling RealtimeClient.open().
• The best part: The controller's showLive(full: String) callback already handles InputConnection.setComposingText() out-of-the-box, as it was fully programmed for the Cloud APIs!

❓ Open Questions / Decisions

Warning

Model Selection Strategy
Should DictateKeyboard force users to download a dedicated Zipformer model to use Real-Time Dictation? Or should there be a fallback that "fake-streams" via VAD chunking on a standard Whisper model? True streaming models (Zipformer) are vastly more efficient for this use case, but it would introduce a second category of models in the catalog.

Describe alternatives you've considered

I understand that this might be extremely difficult or impossible for the Cloud-based OpenAI Whisper API via standard REST endpoints (since it expects full audio files). However, having this exclusively for Local Models (Sherpa-ONNX) would be a huge selling point for privacy-focused offline dictation.

Additional context

Implementing this would make DictateKeyboard feel just as fast and natural as the proprietary Gboard voice typing, completely eliminating the "waiting for processing" anxiety.

Thanks for considering this and for the amazing work on the app!

[MyButtermilk]

Realtime is already implemented, please check the settings.