Skip to content

fix(tts): retry transient /v1/audio/speech failures - #28

Merged
volschin merged 1 commit into
mainfrom
fix/tts-transient-retry
Aug 1, 2026
Merged

volschin merged 1 commit into
mainfrom
fix/tts-transient-retry

Conversation

@volschin

@volschin volschin commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Summary

A single 503 Service Unavailable from /v1/audio/speech dropped a whole response turn on a live call: TtsClient.synthesize raised through raise_for_status, VoicePipeline.synthesize_pcm16 logged the failure and returned empty, and process_turn_stream aborted with RuntimeError: TTS returned no audio. The caller heard nothing and got no spoken error. The next request seconds later returned 200 — the outage was brief enough that a bounded retry saves the turn.

Changes

  • agent/tts.pysynthesize retries transient failures:
    • Retryable: RETRY_STATUS = 502/503/504, plus httpx.TransportError.
    • Not retryable: 4xx, and the 500 the server returns for a rejected request body — those repeat deterministically, so retrying only adds dead air before the same failure.
    • RETRY_BACKOFF_S = (0.3, 0.9) → 3 attempts, ≤1.2 s added in the worst case, so a retried turn still lands inside the caller's patience.
    • Each retry logs at WARNING, so recovered 503s stay visible for the still-open server-side root cause.
    • The backoff sleeps are asyncio.sleep cancellation points, so barge-in still cuts a retrying turn.
  • TODO.md — marks the retry item done with the resulting policy; the "root-cause the server-side 503" item stays open.

synthesize_stream is deliberately untouched — /v1/audio/speech/stream is diagnostic-only and not on the production response path.

Testing

TDD — every test watched failing before the implementation existed:

  • 503 then 200 → recovers, 2 requests
  • transport error then 200 → recovers, 2 requests
  • persistent 503 → raises HTTPStatusError after exactly 3 requests
  • 500 → raises after exactly 1 request (no retry)
  • cancelled mid-backoff → propagates CancelledError

Existing test_synthesize_raises_on_http_error retargeted 503 → 500, since 503 is now a retryable status.

venv/bin/pytest -q → 319 passed. ruff format --check / ruff check clean.

A single 503 dropped a whole response turn: synthesize raised, the pipeline
logged the failure and aborted with 'TTS returned no audio', and the caller
heard nothing. The next request seconds later succeeded.

synthesize now retries 502/503/504 and httpx.TransportError on a short
backoff (0.3s, 0.9s; 3 attempts total), logging each retry at WARNING so
recovered outages remain visible for the server-side root cause. 4xx and the
request-body 500 stay non-retryable. The backoff sleeps are cancellation
points, so barge-in still cuts a retrying turn.
@volschin
volschin merged commit 09925c4 into main Aug 1, 2026
5 checks passed
@volschin
volschin deleted the fix/tts-transient-retry branch August 1, 2026 03:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant