Improve dictation hot-path performance

[MyButtermilk]

Summary

This PR reduces avoidable CPU work and allocation pressure in the dictation hot paths, with a focus on realtime streaming, audio encoding, lightweight preference parsing, local transcription reuse, PCM resampling, and Wear recording shutdown behavior.

The changes are intentionally split into small commits so each optimization has a narrow behavioral surface:

• Avoid per-audio-frame kotlinx.serialization JSON object construction for realtime base64 audio payloads.
• Buffer file-to-Base64 encoding and append Base64 chunks with a reusable char buffer instead of per-byte StringBuilder.append calls.
• Parse daily stats and pending milestone markers without split() list allocations.
• Build custom-word glossaries and prompt echo marker checks without repeated split/map/list work.
• Include numThreads in the local recognizer cache key so changing thread count actually rebuilds the native recognizer.
• Normalize PCM16 resampler input to complete sample frames and remove unnecessary interpolation clamps.
• Stop Wear AudioRecord before joining the recorder thread, propagate read/write failures, and make waveform peak tracking atomic.
• Skip realtime audio send work after a session has already closed, including avoiding Base64/ByteString creation when no socket is available.

Why this adds value

Dictation performance is dominated by work that happens repeatedly while the microphone is active or while audio is being uploaded. Even small allocations in those paths matter on Android because they happen many times per second, compete with audio/UI work, and increase garbage collection pressure. This is especially relevant for realtime transcription and Wear devices, where CPU headroom, battery, and thermal budget are tighter than on a desktop.

The biggest improvements are in repeated audio handling:

• Realtime OpenAI, ElevenLabs, and Gemini sessions previously built JSON objects for every audio frame before serializing them. The payload shape is constant except for the Base64 audio body, so this PR keeps provider setup/config messages on structured JSON but emits audio append frames with fixed prefixes/suffixes. That removes object-map/string serialization work from the per-frame path.
• Batch OpenAI-compatible file uploads now copy through Base64OutputStream with a 64 KiB buffer and append encoded chunks through a reusable CharArray. This avoids a large amount of tiny append work while preserving the same multipart payload format.
• Realtime sessions now check done and capture ws before encoding/sending. Closed sessions no longer pay Base64 or ByteString allocation cost for frames that will be dropped anyway.

The smaller parser changes reduce background churn in frequently touched UI/accounting code:

• Daily stats parsing now scans the compact day:words;... store directly and preserves malformed-entry tolerance.
• Pending milestones no longer allocate a list just to read time:<value> or count:<value>.
• Custom-word prompt helpers now scan separators directly and reuse fixed prompt echo markers.

There are also correctness fixes that improve practical performance by preventing stale or slow behavior:

• The local Sherpa recognizer cache key now includes numThreads. Without this, a user-visible thread-count change could keep reusing a recognizer built with the old parallelism, making performance tuning appear ineffective until the model/language changed.
• The PCM16 resampler now drops incomplete trailing bytes before any path uses the length. That prevents odd-length buffers from being interpreted as partial samples. The interpolation clamp was removed because interpolation between valid PCM16 sample values cannot exceed the PCM16 range; doing the clamp inside every sample iteration was redundant.
• The Wear recorder now calls AudioRecord.stop() before joining the recorder thread, which avoids waiting on a blocking read() during stop/cancel. It also surfaces read/write failures instead of silently producing a misleading WAV file and uses an atomic peak value for live waveform polling.

Review process

I used an Oracle/GPT-5.5 Pro review pass to identify hot-path opportunities and risk boundaries, then implemented the items that were low-risk and directly measurable by code inspection/compile-time validation. The review specifically highlighted realtime JSON builders, Base64 string output, split-heavy helpers, PCM16 resampling, local recognizer cache invalidation, and Wear recorder shutdown/error handling. The resulting changes keep provider configuration serialization unchanged and only specialize the stable high-frequency payloads.

Validation

Validated locally on the current main baseline plus this branch:

./gradlew.bat :app:testDebugUnitTest --tests "dev.patrickgold.florisboard.dictate.*" :app:compileDebugKotlin :lib:dictate-core:compileDebugKotlin :wear:compileDebugKotlin

Result: BUILD SUCCESSFUL.

Also ran:

git diff --check origin/main...HEAD

Result: clean.

Focused tests covered:

• DictateAutoFormattingTest
• DictateLanguageMatchTest
• AudioWavTest
• DictateStatsTest
• Pcm16ResamplerTest

New/updated test coverage verifies malformed daily-stat tolerance, fixed-window activity parsing, and odd trailing-byte handling in PCM16 resampling.

Risk and compatibility notes

• Manual JSON construction is limited to constant-shape audio frame messages where the only dynamic value is Base64.NO_WRAP output. Provider setup/config JSON still uses structured serialization.
• The Base64 file upload change preserves the existing string-backed multipart behavior; it only changes buffering and append granularity.
• PCM16 odd trailing bytes are now ignored consistently. Since PCM16 samples require two bytes, this is safer than passing an incomplete sample into resampling logic.
• Wear recorder read failures are now reported instead of silently producing an output file. That can surface real device/audio failures earlier, which is preferable to sending corrupted or incomplete recordings downstream.

[MyButtermilk]

Additional loop update:

• Hardened the phone-side RecordingController with the same shutdown/error-handling pattern used for the Wear recorder.
• AudioRecord.stop() now runs before joining the capture thread, reducing stop/cancel latency when read() is blocking.
• File write and negative AudioRecord.read() failures now stop capture and make stop() return null instead of silently producing a WAV header for bytes that were never written.
• Waveform peak tracking now uses AtomicInteger.getAndSet() plus a once-per-buffer CAS update, avoiding lost peaks between the capture thread and UI polling.
• cancel() now releases the recorder directly and deletes the output without writing a final WAV header first.

Validation rerun after this loop:

./gradlew.bat :app:testDebugUnitTest --tests "dev.patrickgold.florisboard.dictate.*" :app:compileDebugKotlin :lib:dictate-core:compileDebugKotlin :wear:compileDebugKotlin
git diff --check origin/main...HEAD

Both passed locally.

[MyButtermilk]

Additional high-performance loop with Oracle/GPT-5.5 Pro:

• Ran an Oracle review with gpt-5.5-pro / ChatGPT Pro Extended over the Dictate audio, provider, controller, prompt, stats, Wear recorder, and test files.
• Oracle recommended OpenAiCompatibleClient HTTP client reuse as the best next high-impact, low-risk target.
• Implemented a process-wide OkHttpClient cache keyed only by transport-affecting settings:
  • timeout
  • proxy config
  • user-certificate trust mode
• Request-specific values remain outside the cache:
  • API key
  • base URL
  • extra headers
  • transcription/chat API selection

Why this matters:

• Batch transcription and rewording can now reuse OkHttp connection pools instead of building a fresh client/pool for each OpenAiCompatibleClient wrapper.
• This improves repeated cloud dictation, auto-formatting, auto-apply prompt chains, and provider calls that happen sequentially after one recording.
• Keeping auth and provider headers at request level preserves correctness for different accounts and providers while still allowing transport reuse.

Validation rerun after this loop:

./gradlew.bat :app:testDebugUnitTest --tests "dev.patrickgold.florisboard.dictate.*" :app:compileDebugKotlin :lib:dictate-core:compileDebugKotlin :wear:compileDebugKotlin
git diff --check origin/main...HEAD

Both passed locally.

Oracle output was saved locally as C:\tmp\dictate-oracle-high-perf-loop3.md.

[MyButtermilk]

Additional Oracle/GPT-5.5 Pro high-performance loop:

• Ran another Oracle review with gpt-5.5-pro / ChatGPT Pro Extended over the current realtime capture, resampler, controller, and realtime provider code.
• Oracle recommended a bounded realtime audio-frame commit as the next best performance target after the already-merged HTTP client reuse.
• Implemented commit ba8d3fa Bound realtime capture chunks.

What changed:

• Realtime recording now reads smaller capture chunks only when pcmSink is active.
  • AudioRecord still keeps its larger internal buffer.
  • App-side realtime reads are capped to about 40 ms of 16 kHz mono PCM16 audio.
• The realtime pcmSink is called before cache-file WAV writes and waveform peak scanning.
  • This moves audio toward the WebSocket earlier in each capture-loop iteration.
  • WAV fallback behavior remains intact.
• Added Pcm16Resampler.outputLengthBytes(...) and Pcm16Resampler.resampleInto(...).
  • Existing allocating resample(...) API remains available.
  • OpenAI realtime 16 kHz -> 24 kHz conversion now reuses one buffer per realtime session instead of allocating a new output ByteArray per captured chunk.
• Documented the RealtimeSession.sendAudio(...) buffer-consumption contract because callers may now reuse the PCM buffer.
• Added unit tests proving resampleInto(...) matches the existing allocating path and leaves buffer tails untouched.

Expected value:

• Lower capture-to-wire latency for realtime providers by bounding the blocking read size.
• Less jitter because the realtime send path no longer waits behind disk write and peak scanning.
• Lower GC pressure in OpenAI realtime mode by avoiding one resampler output allocation per chunk.

Validation rerun after this loop:

./gradlew.bat :app:testDebugUnitTest --tests "dev.patrickgold.florisboard.dictate.*" :app:compileDebugKotlin :lib:dictate-core:compileDebugKotlin :wear:compileDebugKotlin
git diff --check origin/main...HEAD

Both passed locally. The Gradle run exited with code 0 and all Dictate tests passed.

Oracle output was saved locally as C:\tmp\dictate-oracle-high-perf-loop4.md.

[MyButtermilk]

Additional Oracle/GPT-5.5 Pro high-performance loop:

• Ran another Oracle review with gpt-5.5-pro / ChatGPT Pro Extended over the current audio gate, decode, controller, sink, and prompt code.
• Oracle selected the next single target: stream SpeechGate VAD over recorder-native WAV instead of decoding the whole file into a FloatArray first.
• Implemented commit 971d677 Stream SpeechGate over native WAV.

What changed:

• Added AudioDecode.streamPcm16Mono16kWav(...).
  • It only handles exact recorder-native WAV shape: PCM16, mono, 16 kHz.
  • It returns false for non-native WAVs so the existing full decoder remains the fallback.
  • It streams reusable float windows and supports early stop.
• SpeechGate.hasSpeech(...) now tries the streaming WAV path first.
  • For normal recorded audio, VAD can consume the file window-by-window.
  • If speech is detected early, the gate can stop reading without decoding the rest of the recording.
  • If the file is not exact native WAV or any gate error occurs, existing fail-open/fallback behavior is preserved.
• Added AudioWavTest coverage for:
  • streamed windows matching decodeToMono16k(...)
  • early stop after the first window
  • rejection of non-native sample rates for fallback

Expected value:

• Avoids whole-recording FloatArray allocation for the common recorded-WAV speech gate path.
• Reduces GC pressure and post-stop latency before upload/transcription.
• Enables true early exit when VAD detects speech near the beginning of long recordings.

Validation rerun after this loop:

./gradlew.bat :app:testDebugUnitTest --tests "dev.patrickgold.florisboard.dictate.*" :app:compileDebugKotlin :lib:dictate-core:compileDebugKotlin :wear:compileDebugKotlin
git diff --check origin/main...HEAD

Both passed locally. The focused AudioWavTest run also passed before the full validation.

Oracle output was saved locally as C:\tmp\dictate-oracle-high-perf-loop5.md.

[MyButtermilk]

High-performance loop 6: coalesce realtime IME preview updates

Oracle / GPT-5.5 Pro reviewed the current PR state and picked realtime preview UI coalescing/throttling for the IME path as the next highest-impact target. The recommendation is saved locally at C:\tmp\dictate-oracle-high-perf-loop6.md.

Why this adds performance value

Before this commit, every realtime provider partial could become an immediate editor preview write:

• each callback posted to the main scope,
• _interimText was updated,
• setDictationPreview(newText, prevText) ran for the focused IME field,
• the IME sink recomputed a common prefix and then committed or replaced the changed tail through the editor / InputConnection.

That is expensive in exactly the hot path where providers can emit many small partial revisions per second. Even when the text only changes by a few characters, the keyboard still does main-thread editor work, common-prefix comparisons, InputConnection commits/replacements, and preview-state rewrites at provider callback frequency.

This commit bounds that cost for the IME path by coalescing preview writes to about one update every 66 ms (~15 fps). The latest transcript remains visible quickly enough for the user, but editor writes are no longer tied to token/partial callback frequency. On a fast partial stream, that turns potentially dozens or hundreds of editor writes per second into a small, predictable number of writes per second.

What changed

Commit: 77d8b15 perf(dictate): coalesce realtime IME preview updates

• Added REALTIME_PREVIEW_FRAME_MS = 66L as the IME preview frame budget.
• Added realtimePreviewLatest to track the newest transcript received from realtime callbacks, even when it has not been flushed into the field yet.
• Kept realtimeShown as the text actually written into the field. This separation matters because final commit / fallback cleanup must replace or clear only what is truly visible.
• Replaced direct showLive() preview writes with publishLive() / flushPreview():
  • partials update _interimText immediately for the Smartbar caption,
  • IME preview writes are coalesced,
  • final segments force an immediate flush,
  • overlay behavior is left unchanged.
• Updated stop/finalize to prefer realtimePreviewLatest as the transcript source, so stopping while a preview update is pending does not lose the newest partial.
• Added teardown cleanup for cancel, hide/stash, fallback, error, and cancellation paths so delayed preview jobs cannot re-add text after the session is gone.

Expected user-visible impact

Realtime dictation should feel smoother in text fields under high partial-update rates:

• lower main-thread churn during realtime streaming,
• fewer InputConnection/editor mutations,
• less repeated diff work in ImeDictationSink,
• fewer opportunities for jank or keyboard stalls while the provider is revising partials,
• final text still commits immediately and uses the newest received transcript.

Risk controls

The main correctness risk was stale state: after coalescing, the latest transcript and the visible preview are no longer always identical. The patch handles that explicitly:

• realtimePreviewLatest is used for the transcript when finalizing,
• realtimeShown remains the visible-preview source for clearDictationPreview() and commitDictationFinal(),
• final segments bypass the throttle,
• cancel/failure/stash/cancellation paths cancel any pending flush job before clearing state,
• overlay realtime preview behavior is not throttled by this controller change.

Validation

Ran successfully:

.\gradlew.bat :app:testDebugUnitTest --tests "dev.patrickgold.florisboard.dictate.*" :app:compileDebugKotlin :lib:dictate-core:compileDebugKotlin :wear:compileDebugKotlin
git diff --check origin/main...HEAD

Result: BUILD SUCCESSFUL; diff check clean.

[MyButtermilk]

Review follow-up: realtime final callback ordering

I reviewed the PR after the latest realtime preview coalescing change and found one subtle lifecycle risk in the controller:

• RealtimeCallbacks.onFinalSegment(...) was still doing its final transcript append inside scope.launch { ... } on the main dispatcher.
• RealtimeCallbacks.onClosed() completes the finalize wait immediately from the provider / OkHttp callback thread.
• That means the stop/finalize coroutine could theoretically resume from closed.await(), cancel pending preview work, and compute the transcript before a just-delivered final-segment main-dispatch job had appended its text.

Commit 37ad7ad Fix realtime final callback ordering fixes that by recording realtime text state synchronously in the provider callback before dispatching the coalesced UI/editor update:

• added realtimeTextLock around realtimeFinal / realtimePreviewLatest,
• made realtimePreviewLatest volatile because provider callbacks run off-main,
• moved partial/final transcript accumulation into recordPartial(...) / recordFinalSegment(...),
• left only the UI/editor publication in the main-dispatched publishLive(...),
• reads the final transcript under the same lock in stopRealtimeAndFinalize(...).

This keeps the performance benefit from coalescing editor preview writes, but removes the ordering race between final transcript receipt and session-close finalization.

Validation rerun after the fix:

.\gradlew.bat :app:testDebugUnitTest --tests "dev.patrickgold.florisboard.dictate.*" :app:compileDebugKotlin :lib:dictate-core:compileDebugKotlin :wear:compileDebugKotlin
git diff --check origin/main...HEAD

Result: BUILD SUCCESSFUL; diff check clean. No CI checks are currently reported for this branch.

[MyButtermilk]

Conflict resolution and review follow-up

Merged the latest main (c3bf0fe) into this branch and resolved the conflicts in commit 4ded3fe.

Conflict resolution

• RecordingController now combines PR Improve dictation hot-path performance #172's bounded realtime capture chunks, atomic peak tracking, stop-before-join behavior, and explicit read/write failure handling with the long-form segment rotation added on main.
• Recorder writes and byte accounting remain serialized with rotate() / stop(), but write failures are no longer swallowed or counted as successfully persisted PCM.
• Segment filenames now include a monotonic component so a newly created controller cannot overwrite a segment that is still being transcribed by an earlier recording.
• SpeechGate retains PR Improve dictation hot-path performance #172's streaming native-WAV fast path while preserving the shared Silero model/window access required by the long-form live VAD splitter.

Additional fixes found during review

• Phone and Wear recorders now clean up their native recorder, WAV handle, and partial cache file if AudioRecord.startRecording() fails.
• Wear shutdown now tolerates an interrupted recorder-thread join and still releases the native recorder.
• Error paths reset recorder state consistently instead of leaving an unusable output file behind.

I also re-reviewed the remaining performance changes against the current main implementation, including realtime buffer reuse, manual constant-shape audio JSON frames, shared OkHttp transport caching, PCM16 resampling, allocation-free stats/prompt parsing, local recognizer cache invalidation, and streaming SpeechGate decoding. I found no additional merge blockers.

Validation

Passed locally after resolving the conflicts:

.\gradlew.bat :app:testDebugUnitTest --tests "dev.patrickgold.florisboard.dictate.*" :app:compileDebugKotlin :lib:dictate-core:compileDebugKotlin :wear:compileDebugKotlin
.\gradlew.bat :app:testDebugUnitTest
git diff --check origin/main...HEAD

Results:

• 46 focused Dictate tests passed.
• The complete app unit-test suite passed.
• App, dictate-core, and Wear Kotlin compilation passed.
• Final diff check is clean.
• GitHub now reports the PR as mergeable with a clean merge state.

[MyButtermilk]

@DevEmperor : Please review and merge.