Skip to content

STT auto-degrade + queue scheduling + ship CoreML (post-MVP, after #111) — behavioral half of #141 #164

Description

@realproject7

Goal (post-MVP — the BEHAVIORAL half of the STT safety belt)

Split from #141. #141 (MVP) ships the observation half — RTF measurement + a "falling behind" signal, dead-CoreML removal, backend logging, log routing. This ticket ships the higher-risk half that actually RESPONDS to a too-heavy model and reshapes the transcribe queue. It is gated on the #111 device calibration because the thresholds and the "does it actually keep up" claim can only be validated with real medium/large-v3-turbo audio on the target Mac.

Do NOT start before #111 has produced real-model RTF data.

Scope

  1. Automatic degradation. When the STT observability MVP: RTF 'falling behind' signal + drop dead CoreML + backend logging + log routing (audit R3/R8/M5) #141 RTF signal sustains past the "falling behind" threshold, take ACTION: skip partials first, and if finals still fall behind, auto-degrade the session to small with a content-free status. Requires the STT live verification & per-model floor calibration on real audio (closes the #92/#93/#94 verification gap) #111 real-model RTF numbers to set the skip/degrade thresholds (a wrong threshold here changes the model mid-session — real behavioral risk, unlike the MVP's harmless signal).
  2. Bound the transcribe queue + channel/kind-aware scheduling. Today pipeline.rs:151 is mpsc::unbounded; only partials shed (at 2500ms, :487); finals never drop. Keep only the latest partial per channel; bound the finals backlog; don't let a burst of system partials delay a mic final. Highest-risk item: this rewrites the transcribe queue, which carries shipped invariants (channel separation, partial/final ordering, the [follow-up] Speaker audio bleeds into the mic channel — duplicate captions, doubled cost, polluted archive #56/[follow-up] Suppressed mic finalization can leave stale streaming block in feed #62 bleed-suppression DropPartial ordering). Adversarial verification + real-audio integration tests mandatory (see the [bug] VAD 30s force-cut duplicates captions + grows silero buffer unbounded on long monologues (M3) #138/STT hot-path: reuse WhisperState + pin language for partials (kill 2x encode) + partial audio_ctx/greedy (audit R1/M1/M2) #140 lessons — subtle bugs live here).
  3. Ship CoreML properly (option 4b). Fetch/generate ggml-{model}-encoder.mlmodelc next to the ggml model so the ANE encoder speedup (1.5–3× on Apple Silicon) actually engages. Model asset SHA-verified like [#110] Settings: whisper model selection with download-on-switch #127. Re-check the audio_ctx×CoreML interaction the STT hot-path: reuse WhisperState + pin language for partials (kill 2x encode) + partial audio_ctx/greedy (audit R1/M1/M2) #140 verification flagged (a present .mlmodelc + reduced audio_ctx = potential heap overflow — but STT hot-path: reuse WhisperState + pin language for partials (kill 2x encode) + partial audio_ctx/greedy (audit R1/M1/M2) #140 dropped audio_ctx, so confirm the interaction is safe before re-enabling either).

Acceptance criteria

  • On the target Mac with a too-heavy model, the watchdog degrades (partial-skip → small) and captions recover to real-time; on an adequate model it never degrades. Thresholds backed by STT live verification & per-model floor calibration on real audio (closes the #92/#93/#94 verification gap) #111 RTF data.
  • Queue is bounded; no unbounded latency accumulation under load; channel separation + bleed-suppression ordering invariants preserved (adversarial verification + integration tests prove it).
  • If CoreML shipped: the loaded backend logs CoreML/ANE, the mlmodelc is SHA-verified, and finals stay accurate (audio_ctx interaction confirmed safe).

Security invariants

Never log/emit caption content; new model asset (if 4b) SHA-verified like #127; no other new runtime dependencies.

Routing

agent/claude — device-bound Rust + RTF benchmarking on this Mac. Depends on #111 (real-model RTF data). Adversarial verification is mandatory for the queue rewrite (item 2).

Part of #134. MVP half already merged as #141.

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/claudeBuilt by local Claude agent (needs this device)enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions