Repository navigation
fix: recover from broken S3 audio streams instead of wedging - #32
Merged
Merged
Conversation
A single mid-stream failure while proxying audio left a dead connection in the shared boto3 client's pool. Every later get_object then returned valid headers with an empty body, so uvicorn aborted the response and Cloudflare served 520 for all playback until the process was restarted. Track bytes streamed against Content-Length and drop the cached client when a stream errors or ends short, so the damage is limited to the one request that hit the dead connection. Connection-level failures on get_object reset it too, and now surface as 502 rather than an unhandled error. Client disconnects (seeking) raise GeneratorExit and are deliberately left alone -- they are routine and leave the connection usable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
GET /api/shows/episodes/{id}/audioreturned Cloudflare 520 for every episode. S3 itself was healthy — the presigned URL played fine, and a fresh process in the same pod streamed the full object.The running process was wedged. Bisecting by
Rangeagainst the origin socket showed it delivered at most one 64 KB chunk (AUDIO_CHUNK_SIZE):The first error in the container log was a one-off
ResponseStreamingError: Connection broken: BrokenPipeError; every error after it wasRuntimeError: Response content shorter than Content-Length. A socket to the S3 endpoint was left inCLOSE_WAIT. The broken connection stayed in the@lru_cached client's pool, so subsequentget_objectcalls returned correct headers with an empty body — uvicorn aborted those responses, cloudflared loggedunexpected EOF, and Cloudflare turned that into 520. It never recovered on its own.Fix
Content-Length; on a short or failed stream, drop the cached client so the next request builds a fresh pool. One request fails instead of all of them, forever.BotoCoreErrorfromget_objecttoo, and return 502 instead of an unhandled error.GeneratorExit, notException, so ordinary seeking does not churn the client — this matches the logs, where months of cancels were harmless and only the read error wedged it.Tests
Four cases added: full stream keeps the client, short stream resets it, mid-stream error resets it and still closes the body, connection error resets it and returns 502.
ruff check,ruff format --check,basedpyright, andpytest(95 passed) all clean.🤖 Generated with Claude Code