Latency: config-driven voice LLM — Groq Llama-3.3-70B / Claude Haiku 4.5 with tool-call re-validation

[teetangh]

Context

The streaming path LLM is hardcoded to OpenAI gpt-4o-mini via OpenAILLMService (apps/voice/streaming.py:528-531), chosen for sub-500 ms TTFT over OpenRouter's 2–5 s. Faster options exist: Groq-hosted Llama-3.3-70B (~sub-80 ms TTFT) and Claude Haiku 4.5 (~100–150 ms). settings.yaml already defines an agent_voice role (Groq llama-3.3-70b) that the streaming module ignores, and GROQ_API_KEY is declared in agents/settings.py Secrets.

The risk in swapping is tool-calling fidelity: the negotiation depends on propose_offer and end_call tool calls (streaming.py:110-190) and the deterministic validator _validate_offer (streaming.py:193-362). A new model must reliably call tools, speak the validator's canonical numbers, and follow the confirmation cadence.

Implementation plan

1. Config-driven LLM selection: read the voice LLM provider/model from settings.yaml (voice.llm or the existing llm.roles.agent_voice) — factory returning GroqLLMService / AnthropicLLMService / OpenAILLMService (all available in pipecat; add missing extras to pyproject.toml). Keep gpt-4o-mini as fallback.
2. Verify tool schemas work on each provider (Groq uses OpenAI-compatible tool format; Anthropic needs the pipecat Anthropic service which translates).
3. Re-validation harness: run all 10 scripted scenarios in examples/01-10 through the streaming simulator per provider; check expected outcomes and the pass/fail checks in each example file. Extend tests/test_propose_offer.py if validator interaction changes.
4. Enable prompt caching where supported (Anthropic cache_control; see also issue on prompt diet) to cut TTFT further.
5. Record LLM TTFB per provider from the P0 metrics observer and document the comparison in docs/architecture/05-stt-tts.md or the roadmap doc.

Acceptance criteria

• LLM TTFT p50 < 150 ms on the chosen provider (measured via voice_call_metrics.jsonl).
• All 10 scripted scenarios reach their expected outcomes on the new provider (no regressions in tool-call behavior, canonical-number speech, or confirmation cadence).
• Fallback to gpt-4o-mini works via config change only, no code edit.

Dependencies

• P0 fixes PR (metrics observer).
• Works best after Conversions: code-enforced confirmation cadence before end_call(deal_agreed) #10 (code-enforced confirmation cadence), which de-risks prompt-following differences between models.