fix(tts): retry transient /v1/audio/speech failures - #28
Merged
Merged
Conversation
A single 503 dropped a whole response turn: synthesize raised, the pipeline logged the failure and aborted with 'TTS returned no audio', and the caller heard nothing. The next request seconds later succeeded. synthesize now retries 502/503/504 and httpx.TransportError on a short backoff (0.3s, 0.9s; 3 attempts total), logging each retry at WARNING so recovered outages remain visible for the server-side root cause. 4xx and the request-body 500 stay non-retryable. The backoff sleeps are cancellation points, so barge-in still cuts a retrying turn.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A single
503 Service Unavailablefrom/v1/audio/speechdropped a whole response turn on a live call:TtsClient.synthesizeraised throughraise_for_status,VoicePipeline.synthesize_pcm16logged the failure and returned empty, andprocess_turn_streamaborted withRuntimeError: TTS returned no audio. The caller heard nothing and got no spoken error. The next request seconds later returned 200 — the outage was brief enough that a bounded retry saves the turn.Changes
agent/tts.py—synthesizeretries transient failures:RETRY_STATUS= 502/503/504, plushttpx.TransportError.RETRY_BACKOFF_S = (0.3, 0.9)→ 3 attempts, ≤1.2 s added in the worst case, so a retried turn still lands inside the caller's patience.asyncio.sleepcancellation points, so barge-in still cuts a retrying turn.TODO.md— marks the retry item done with the resulting policy; the "root-cause the server-side 503" item stays open.synthesize_streamis deliberately untouched —/v1/audio/speech/streamis diagnostic-only and not on the production response path.Testing
TDD — every test watched failing before the implementation existed:
HTTPStatusErrorafter exactly 3 requestsCancelledErrorExisting
test_synthesize_raises_on_http_errorretargeted 503 → 500, since 503 is now a retryable status.venv/bin/pytest -q→ 319 passed.ruff format --check/ruff checkclean.