feat: speaker diarization support

[CJHwong]

Problem

In multi-speaker sessions (meetings, interviews, panels), the transcript is a flat stream of text with no indication of who said what. This makes it harder for the LLM digest to produce accurate meeting summaries, attribute action items, and distinguish perspectives.

Research Summary

Investigated five approaches for adding speaker awareness:

1. Gemini Live API (real-time streaming)

• WebSocket-based, async-native, sends PCM audio chunks
• No speaker diarization support in the Live API. input_transcription returns flat text.
• Session limit: 15 min without compression
• $3/M tokens (~$0.35/hr)
• Verdict: ruled out for diarization

2. Gemini batch API (post-session)

• Send full WAV after session ends, get diarized transcript via structured JSON output
• Prompt-driven: "Transcribe verbatim, identify speakers"
• ~30 sec for a 1-hour recording. ~$0.12/hr on flash.
• Accuracy is LLM-grade (good, not deterministic)
• Verdict: viable as cloud option

3. pyannote-audio (local, PyTorch)

• Gold standard for diarization accuracy
• ~3-4 GB dependency footprint (PyTorch ecosystem)
• CPU: 0.5-1.5x real-time (30-min meeting = 30-90 min processing)
• Apple Silicon MPS: ~3-5x faster than CPU
• HuggingFace gated models (requires auth token)
• Verdict: too heavy and too slow for near-realtime goal

4. sherpa-onnx (local, ONNX)

• ~55 MB total (library + models). Apache 2.0. No HF auth.
• Uses pyannote segmentation model exported to ONNX
• Tested on a 5-min clip: 99s processing time (0.33x RTF)
• Extrapolated: 30-min meeting = ~10 min processing
• Offline only, no streaming
• Verdict: viable as offline fallback, but too slow for near-realtime

5. whisper.cpp tinydiarize (--tdrz)

• Finetuned small.en model that emits [SPEAKER_TURN] tokens at speaker changes
• Near-zero overhead (<10% extra inference cost) since it's the same model architecture
• Precision: 97.7%, Recall: 70.8% (misses ~30% of turns)
• English only, only small.en size available
• pywhispercpp does NOT expose the tdrz_enable flag (trivial 2-line fix to add)
• Verdict: lightweight but limited (English-only, no speaker identification)

Proposed Direction

Insight: speaker change hints may be enough

Full diarization (identifying WHO each speaker is) may be overkill. If the transcript contains [SPEAKER_TURN] markers at speaker boundaries, the LLM digest can infer speaker roles from conversational context and produce structured output like "The PM asked about deadlines, the engineer raised QA concerns."

Implementation plan (phased)

Phase 1: LLM-inferred speaker attribution (no model changes)

• Update digest prompt templates to instruct the LLM: "This transcript is from a multi-speaker session. Identify speaker changes from conversational cues and attribute statements."
• Test with existing transcripts to see how well the LLM handles it without any markers
• Zero dependency cost

Phase 2: Gemini batch diarization (cloud, post-session)

• After session ends, send saved WAV to Gemini API for speaker-labeled transcript
• Merge with existing whisper transcript (align by timestamps)
• Config: diarization: {enabled: true, provider: gemini}
• Dependency: google-genai (~5 MB)

Phase 3 (optional): sherpa-onnx offline fallback

• For users who need offline diarization and can tolerate the processing time
• Config: diarization: {provider: sherpa-onnx}
• Dependency: sherpa-onnx (~55 MB)

Out of scope (for now)

• Real-time streaming diarization (no viable solution found)
• whisper.cpp tinydiarize (English-only limitation conflicts with zh-TW support)
• pyannote-audio (dependency footprint too large)
• MLX-based diarization (nothing production-ready in Python)