Voice-reply latency: adaptive thinking left on, first-chunk cap bug, sentence-only streaming granularity

[pfschaller2027-collab]

What I saw

On a warm session (past the greeting/warmup, so prompt caching should already be hot), a plain conversational reply with no tool use took roughly 11-12 seconds from the user finishing speaking to the first audible word.

Setup: macOS 15.7.9, Intel Core i5-8279U (x86_64), OMP_NUM_THREADS=1 / KMP_DUPLICATE_LIB_OK=TRUE as required, Kokoro (bm_lewis, speed 1.1), Sonnet with effort: "low".

What I expected

The streaming + TTS-pipelining architecture already in main.py/mouth.py is genuinely concurrent — synthesis of chunk N+1 starts the instant chunk N's synthesis finishes, not when N starts playing, verified in the trace. Given that, I expected first audio well under 11s; the slowness had to be upstream of the pipeline itself, not in it.

How to reproduce

1. Launch normally, past the greeting/warmup ping.
2. Ask a plain conversational question that needs no tool use.
3. Watch the [trace] lines already emitted by brain.py/mouth.py/main.py: user-finish → first text delta → first sentence → first TTS synth → first playback.

What I found — three separate, independently measurable causes

1. effort: "low" alone still leaves adaptive thinking on. The model decides for itself whether to spend invisible reasoning tokens before the first visible word; "low" just gives it a smaller budget, not zero. Measured back-to-back, identical prompt, identical warm cache/options otherwise:

• adaptive-but-effort:"low": 2.18s to first text delta
• thinking={"type":"disabled"} added: 1.07s

Roughly half, for a reply that never needed reasoning at all. A voice line has nowhere to show that reasoning anyway.

2. The fast-start chunking only ever saw whole sentences, never clauses. brain.py's _SENTENCE_END regex only splits on .!?. When the model's opening line is one long comma/dash-joined sentence, main.py's first-chunk logic sits idle until the entire sentence has streamed in — even though the first clause was ready in the buffer well before the terminal period arrived. Example (synthetic, not a real transcript): a ~290-char opening sentence held its first clause back for an extra ~1.7s versus when that clause actually existed.

Splitting on clause punctuation too (, ; : — -) hands main.py that first clause the moment it exists. This doesn't fragment anything downstream — the existing batching (emit_batch) already re-combines short pieces back up to a natural chunk size; it just removes a wait main.py had no way to shorten from its own side.

3. Kokoro at OMP_NUM_THREADS=1 synthesizes within a few percent of real time on this CPU, not the faster multiple the current comments assume:

chars	audio duration	synth time	ratio
30	2.83s	2.97s	1.05x
54	4.20s	4.38s	1.04x
99	6.95s	7.16s	1.03x
159	10.45s	11.00s	1.05x

There's no multi-x headroom for the first chunk to spend — it has zero pipeline cover, so every character shipped in it is paid for almost one-for-one in dead air. On top of that I found a real, live-caught bug in the existing first-chunk cap logic (_split_into_pieces's word-boundary fallback): a single clause with no internal comma/dash, sized between the cap and cap*1.5, sailed straight past the cap uncapped, because the fallback loop was gated on a fixed target*1.5 threshold instead of the caller's own hard_cap. Caught live: a 75-char clause shipped whole against a 65-char cap, costing an extra ~1.5s of synthesis before any sound.

Evidence — before/after, same warm session, same question

Before:

user_finish            t=30924.665
brain first text delta t=30931.442   (+6.78s)
first sentence         t=30931.872   (+7.21s)
first TTS synth start  t=30931.873
first TTS synth done   t=30936.418   (+4.55s synth)
first playback         t=30936.419   (+11.75s total)

After (same shape of question, live run through the real app, not a synthetic script):

user_finish             t=1333.878
brain first text delta  t=1334.977   (+1.10s)
first clause            t=1335.532   (+1.65s)
first TTS synth start   t=1335.532
first TTS synth done    t=1340.807   (+5.28s synth)
first playback          t=1340.807   (+6.93s total)

A separate run where the model's first natural clause happened to be short got to first playback in 4.8s; a short yes/no-style reply got there in 3.3s. TTS stayed ahead of playback for the remainder of both multi-chunk replies after the initial chunk.

What I changed

• brain.py: pass thinking={"type": "disabled"} alongside the existing effort; widen _SENTENCE_END to also split on clause punctuation (,;:—-), not just .!?.
• main.py: tighten the first-chunk character cap, and give _split_into_pieces's word-boundary fallback a hard_cap parameter gated correctly on itself instead of the looser target*1.5 (this is also the fix for the live-caught bug above).
• config.py: one added line nudging replies to use comma-paced clauses — this is what actually gives the clause-level streaming something to work with.

None of it touches OMP_NUM_THREADS=1, the thinking-sound-disabled default, playback speed, or the pipeline architecture — all deliberate, already-working choices.

Per CONTRIBUTING.md — not asking for a merge, just leaving the diff in case it's useful as a reference (or entirely skippable) for a rewrite in your own style: pfschaller2027-collab@ad7bc6c