Latency: prompt diet + prompt caching + turn-taking tuning + pre-synthesized opener

[teetangh]

Context

Two of the biggest voice-latency levers are turn-taking (endpointing) and LLM prefill. Current settings in apps/voice/streaming.py:

• SileroVADAnalyzer(VADParams(stop_secs=0.8)) (line 482)
• LocalSmartTurnAnalyzerV3 with SmartTurnParams(stop_secs=2.5, max_duration_secs=8) (lines 475-481)
• Deepgram endpointing=300, utterance_end_ms=1500 (lines 504-515)

The streaming system prompt (data/prompts/resolution_voice_streaming/v1.yaml) is long (contains full negotiation policy, confirmation cadence, tool instructions); industry guidance for voice is <600 tokens on the hot path, with prompt caching for the rest. The fixed opener is synthesized fresh on every call.

Implementation plan

1. Prompt diet: split the system prompt into a cacheable stable prefix + minimal per-call context (balance, borrower name, HandoffContext summary). Target < 600 tokens uncached hot path. Keep policy constraints intact — the _validate_offer code gate already backstops policy, so prose duplication can be trimmed.
2. Prompt caching: enable provider prompt caching (OpenAI automatic ≥1024-token prefix; Anthropic cache_control breakpoints if the LLM swap lands) — order messages stable-prefix-first.
3. Turn-taking tuning: sweep stop_secs (VAD 0.4–0.8) and Smart Turn stop_secs (1.2–2.5) against the simulator + a few real calls; measure endpoint-detection latency (user-stops-speaking → LLM run start) via the P0 metrics observer / Pipecat processing metrics. Tighten Deepgram utterance_end_ms if Smart Turn is authoritative.
4. Pre-synthesized opener: cache the fixed opening disclosure audio (it's verbatim per compliance) and play it immediately on on_client_connected instead of waiting for LLM + TTS.
5. Document chosen values + measured budget in the roadmap doc.

Acceptance criteria

• Measured endpoint-detection p50 documented before/after; no increase in false-positive interruptions on the 10 scripted scenarios.
• Opener audio starts < 500 ms after WebSocket start frame.
• System prompt hot path < 600 tokens; cached-prefix hit rate observable in logs.

Dependencies

• P0 fixes PR (metrics observer for measurement).