[R&D spike] Alternative STT engines (Parakeet/Nemotron/Apple SpeechAnalyzer) feasibility — BACKLOG, post-MVP

[realproject7]

Status: BACKLOG — post-MVP re-evaluation. Do NOT start without operator prioritization.

Goal

Investigate whether LiveCap should offer STT engines beyond whisper.cpp — specifically to raise "them"-channel accuracy on real audio (the standing core-quality gap; see #111 and the handoff's item 3). This is a research spike + risk assessment, NOT an implementation. Triggered by benchmarking FluidVoice, which ships Parakeet TDT v3 (NVIDIA, CoreML/ANE), Nemotron 3.5 streaming, and Apple's SpeechAnalyzer (macOS 26) alongside whisper.

Why this is parked (the risk profile — recorded so the decision is not re-litigated)

1. Confidence-gate coupling. The phantom-caption suppression (STT: whisper emits phantom captions on silence/ambient noise #92/STT: wrong source-language detection (English transcribed as Chinese) #93) is bound to whisper's real per-token probabilities (full_get_token_prob, crates/livecap-core/src/whisper/engine.rs) and the per-model floor table (STT: per-model confidence-floor table + env override (floors currently hardcoded, seeded on tiny) #109), freshly calibrated in STT live verification & per-model floor calibration on real audio (closes the #92/#93/#94 verification gap) #111. A different engine exposes different (or no) token-probability surface, so the entire gate + calibration would have to be redesigned per engine.
2. Language-stack mismatch. Parakeet/Nemotron/Cohere CoreML models in FluidVoice run via the Swift FluidAudio package. LiveCap's STT core is Rust (whisper-rs). Reaching those models from Rust means an ONNX/CoreML path (e.g. sherpa-onnx) or an FFI bridge — a substantial new subsystem, not a swap.
3. License. FluidVoice is GPLv3 — no code may be copied into MIT LiveCap; any adoption is a clean-room reimplementation.
4. Pipeline entanglement. Bleed suppression ([follow-up] Speaker audio bleeds into the mic channel — duplicate captions, doubled cost, polluted archive #56/[follow-up] #56 bleed suppression insufficient on real speaker audio: 5.3:1 → 0.9:1, need ~0 #64), channel separation, and VAD gating are woven into the current whisper pipeline; a new engine risks a core-wide refactor and regressions in shipped behavior.

The counterweight (why it's worth keeping on the roadmap at all)

Parakeet TDT v3 is 460 MB, ANE-accelerated, 25-language auto-detect, and FluidVoice rates it top-tier on both speed AND accuracy. If #111 concludes that whisper large-v3-turbo is NOT good enough on real "them"-channel audio, an alternative engine becomes the most credible path to the accuracy the operator cares about.

Scope (research only — deliverable is a written assessment, not code)

1. Enumerate candidate engines reachable from a Rust core WITHOUT a Swift dependency: sherpa-onnx (Parakeet/Nemotron ONNX exports?), whisper alternatives (distil-whisper, faster-whisper equivalents), and Apple SpeechAnalyzer via objc2 (macOS 26+, on-device, per-locale asset download — assess bindings maturity).
2. For each: does it expose per-token/segment confidence we can gate on, or would we need a new suppression strategy? Model size, ANE/Metal acceleration, license, language coverage, streaming support.
3. Integration cost estimate against the current pipeline (audio format, VAD handoff, bleed suppression compatibility).
4. Recommendation: keep whisper-only, add ONE alternative behind the existing engine-selection seam, or defer indefinitely — with the STT live verification & per-model floor calibration on real audio (closes the #92/#93/#94 verification gap) #111 accuracy data as the deciding input.

Explicit dependency

Re-evaluate ONLY after #111 (STT calibration) reports whether the whisper model tiers meet the real-audio accuracy bar. If #111 passes, this likely stays deferred.

Routing

Unscheduled backlog research spike. When activated: agent/claude (needs device benchmarking) or a dedicated R&D session.

Part of #107 (backlog row, same tier as #118). Benchmark source: FluidVoice (github.com/altic-dev/FluidVoice, GPLv3).