fix(tts-server): bound admission, not generation, by the 5s deadline

[volschin]

Root cause

The live 503 that silently dropped a response turn came from the TTS service itself, not from load shedding.

dgx/tts/api.py::_run_stable_synthesis set deadline = monotonic() + DEFAULT_SYNTHESIS_ADMISSION_TIMEOUT_SECONDS (5.0 s) at request entry and then enforced it over the entire request:

• the polling loop raised SynthesisAdmissionTimeout once the deadline passed even though the worker thread already held the model lock and was generating
• a successfully completed result was discarded if it landed after the deadline
• api.py:99 mapped that to 503 "synthesis admission timed out"

The name said admission; the semantics were "the whole request must finish in 5 s". create_app was called without the argument (server.py), so the 5 s was effectively hardcoded.

Evidence

Lock contention, worker restart, and Traefik shedding are all ruled out:

• qwen3-tts access log: 18:28:54 200, 18:29:08 503, 18:29:11 200 — serialized ~14 s apart with no 499 between them, so the model lock was free and admission was instant. The 5 s elapsed inside generation.
• uvicorn logged the 503 itself; proxy-traefik-1 logged zero 503s that hour.
• No ERROR, traceback, or restart; the container stayed Up (healthy).
• Measured idle latency inside the container: "Ja." 0.3–0.5 s, the sentence that failed 1.5–1.7 s, a long sentence 2.9–3.5 s — ~70 % of the budget with the GPU idle.
• At the moment of failure the shared GPU was busy: vLLM companion-llm reported Running: 1 with active generation at 18:29:05 and qwen3-asr transcribed at 18:29:01.7. A ~3.3× slowdown on a 1.5 s sentence crosses 5 s.

Fix

• CloneRuntime.synthesize takes an admitted event and sets it once the model lock is held and the post-acquire deadline check has passed — never earlier.
• _run_stable_synthesis enforces the deadline only while admitted is unset. Queue time and lock wait stay bounded (the caller may already be gone); in-flight generation runs to completion, bounded by client disconnect (already detected) and the client's own 30 s HTTP timeout.
• Budget is now configurable via SYNTHESIS_ADMISSION_TIMEOUT_SECONDS. A malformed or non-positive value falls back to the default rather than refusing to start — a TTS outage takes every call down with it.

Deliberately preserved: a request that never reaches admission is still rejected at the deadline, so a worker stuck in a saturated executor queue cannot start the model late for a caller who has gone away. The existing tests for that behavior (test_stable_timeout_includes_saturated_executor_queue_time, test_stable_absolute_deadline_prevents_late_executor_model_start) pass unchanged.

Testing

TDD — each test watched failing first:

• test_admitted_synthesis_outlives_the_admission_deadline — admitted before the deadline, completes after → result delivered, cancel event not set
• test_slow_generation_returns_audio_not_503 — end-to-end through the route: generation 3× the budget still returns 200 audio/wav
• test_synthesize_signals_admission_before_generation — the event is set by the time the model is called
• test_admission_timeout_leaves_admission_unsignalled — a request that times out waiting for the lock never signals admission
• test_admission_timeout_from_env_* — override read; empty/malformed/zero/negative fall back
• test_stable_absolute_deadline_rejects_late_success renamed to ..._rejects_never_admitted_late_success, since that is what it actually pins now

venv/bin/pytest -q → 343 passed. ruff format --check / ruff check clean.

Deployment

Needs a GPU image rebuild and a stack update on the DGX; the agent-side retry already masks the symptom in the meantime.