Skip to content

STT live verification & per-model floor calibration on real audio (closes the #92/#93/#94 verification gap) #111

Description

@realproject7

Goal

The three STT behavior fixes shipped in v1.1 (#92 phantom-caption gate, #93 auto-detect CJK suppression, #94 source-language forcing) were verified only via say TTS and tiny-model integration tests — never against real varied audio on the production model. Additionally the confidence floors must be calibrated per model with real evidence, and small vs medium vs large-v3-turbo accuracy compared so the settings copy and default rest on measurement.

Prereqs

Verification matrix (each row = observable pass/fail, evidence via ui-heartbeat.json + saved session files)

  1. Silence/ambient: 10 min with no speech, per model → zero phantom captions (STT: whisper emits phantom captions on silence/ambient noise #92).
  2. Forced Spoken=English on English audio (operator's sample video, YouTube 6ARteKgyEtw) → zero CJK output (Session start: choose spoken (source) language + target language #94).
  3. Auto mode on a JA clip and a ZH clip → correct language detected, no cross-language phantoms (STT: wrong source-language detection (English transcribed as Chinese) #93).
  4. Floor calibration on small (then medium/large-v3-turbo): log mean per-token confidence for real speech vs any phantoms; choose floors with margin; check short-phrase real speech ("Yes." / "좋아요") survives.
  5. Accuracy comparison small vs medium vs large-v3-turbo on the same 5-min sample-video segment: transcript diff, error notes, latency + RSS per model.

Deliverables

Operator-assist (called out up front)

  • One system-audio (SecurityAgent) Allow click after the first re-sign — cannot be automated.
  • Optional 2-min real-voice session for the mic channel (TTS is eaten by bleed suppression); scheduled when convenient.

Routing

agent/claude (PO direct on this Mac — device-bound; verification work per manual §9).

Part of #107. Depends on #108, #109.

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/claudeBuilt by local Claude agent (needs this device)enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions