Fix WebSocket disconnect during large manual log file uploads (#1062) - #343
Merged
rquidute merged 3 commits intoJul 22, 2026
Merged
Conversation
Uploading a manual test log with hundreds of thousands of lines (e.g. TC-CADMIN-1.17) called logger.info() once per line. Each call triggered a broadcast over the same main WebSocket used by the CLI and a synchronous DB commit, stalling the event loop long enough for the WebSocket's ping/pong keepalive to time out mid-upload. The connection was dropped with ConnectionClosedError, and the next prompt-response send failed with 'Unexpected error uploading file' even though the upload itself had already succeeded (issue #1062). - Batch uploaded log lines into chunks of 500 before calling logger.info(), instead of one call per line, so a large file produces a handful of log/broadcast/DB-commit operations instead of hundreds of thousands. - Collapse the invalid-UTF-8 warning to a single message per upload instead of one per bad line. - Add app.uvicorn_worker.ExtendedTimeoutUvicornWorker, which raises ws_ping_timeout to 60s (matching the value already used by the dev-only start-reload.sh script), and wire it up as the default worker class in gunicorn/start.sh so production tolerates the same event-loop stalls as local dev. - Add unit tests covering chunked logging, invalid UTF-8 handling, and unsupported content-type rejection.
|
Tick the box to add this pull request to the merge queue (same as
|
Contributor
There was a problem hiding this comment.
Code Review
This pull request addresses websocket connection drops during large manual log uploads (GitHub issue #1062) by batching log lines into chunks of 500 before logging them, and by extending the uvicorn worker's websocket ping timeout to 60 seconds. Feedback suggests using .rstrip("\r\n") instead of .rstrip("\n") to properly handle Windows-style line endings and avoid formatting issues.
.rstrip("\n") left a trailing \r on every line of a Windows-style
(CRLF) uploaded log, since the file is read in binary mode and split
only on \n. That stray \r would end up embedded in the joined chunk,
showing up as ^M characters or unexpected double newlines wherever
the chunk is logged or displayed.
Use .rstrip("\r\n") so both Unix and Windows line endings are
handled correctly. Add a regression test with a CRLF-terminated
upload.
antonio-amjr
approved these changes
Jul 22, 2026
mypy's structural check against the UploadFile protocol requires file: BinaryIO exactly; a bare BytesIO() assignment left the attribute inferred as BytesIO, which mypy treats as incompatible with the protocol's BinaryIO annotation despite BytesIO satisfying it at runtime. Annotate the attribute explicitly so handle_uploaded_file(FakeUploadFile(...)) type-checks.
This was referenced Jul 22, 2026
Merged
rquidute
added a commit
that referenced
this pull request
Jul 22, 2026
…#343) (#344) * Fix WebSocket disconnect during large manual log file uploads Uploading a manual test log with hundreds of thousands of lines (e.g. TC-CADMIN-1.17) called logger.info() once per line. Each call triggered a broadcast over the same main WebSocket used by the CLI and a synchronous DB commit, stalling the event loop long enough for the WebSocket's ping/pong keepalive to time out mid-upload. The connection was dropped with ConnectionClosedError, and the next prompt-response send failed with 'Unexpected error uploading file' even though the upload itself had already succeeded (issue #1062). - Batch uploaded log lines into chunks of 500 before calling logger.info(), instead of one call per line, so a large file produces a handful of log/broadcast/DB-commit operations instead of hundreds of thousands. - Collapse the invalid-UTF-8 warning to a single message per upload instead of one per bad line. - Add app.uvicorn_worker.ExtendedTimeoutUvicornWorker, which raises ws_ping_timeout to 60s (matching the value already used by the dev-only start-reload.sh script), and wire it up as the default worker class in gunicorn/start.sh so production tolerates the same event-loop stalls as local dev. - Add unit tests covering chunked logging, invalid UTF-8 handling, and unsupported content-type rejection. * Address review: strip CRLF instead of LF only when batching log lines .rstrip("\n") left a trailing \r on every line of a Windows-style (CRLF) uploaded log, since the file is read in binary mode and split only on \n. That stray \r would end up embedded in the joined chunk, showing up as ^M characters or unexpected double newlines wherever the chunk is logged or displayed. Use .rstrip("\r\n") so both Unix and Windows line endings are handled correctly. Add a regression test with a CRLF-terminated upload. * Fix CI mypy error: annotate FakeUploadFile.file as BinaryIO mypy's structural check against the UploadFile protocol requires file: BinaryIO exactly; a bare BytesIO() assignment left the attribute inferred as BytesIO, which mypy treats as incompatible with the protocol's BinaryIO annotation despite BytesIO satisfying it at runtime. Annotate the attribute explicitly so handle_uploaded_file(FakeUploadFile(...)) type-checks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1062: THCLI reports a WebSocket error after successfully uploading a manual test log (e.g. TC-CADMIN-1.17), even though the upload itself completed.
Root cause
ManualLogUploadStep.handle_uploaded_file()calledlogger.info()once per line of the uploaded file. Uploaded logs can have hundreds of thousands of lines (the reported log had ~191K). Eachlogger.info()call:TestLogHandler, batched, and broadcast over the same main WebSocket the CLI is using (TestUIObserver→socket_connection_manager.broadcast)session.commit()inTestDBObserverThis flood of broadcast + DB-commit work stalls the event loop long enough (~100s in the reported log) that the WebSocket's ping/pong keepalive isn't serviced in time, and the connection gets dropped without a proper close handshake —
ConnectionClosedError: no close frame received or sent. The CLI's very next action (sending the prompt response after upload) hits the broken socket and surfaces asUnexpected error uploading file: ..., even though the file was already uploaded successfully.Fix
logger.info(), instead of one call per line — a large file now produces a handful of log/broadcast/DB-commit operations instead of hundreds of thousands.app.uvicorn_worker.ExtendedTimeoutUvicornWorker, which raisesws_ping_timeoutto 60s (matching the value already used by the dev-onlystart-reload.shscript), and wire it up as the default worker class ingunicorn/start.shso production tolerates the same event-loop stalls as local dev.Testing
matterSDK checkout module not present in this sandbox).