fix: add graceful shutdown support and prevent producer deadlock in add_messages pipeline#283
Open
KRRT7 wants to merge 6 commits into
Open
fix: add graceful shutdown support and prevent producer deadlock in add_messages pipeline#283KRRT7 wants to merge 6 commits into
KRRT7 wants to merge 6 commits into
Conversation
…Result docstring The success property was removed in a previous commit but the docstring still referenced it, causing confusion.
Propagate shutdown_event to dispatcher for coordinated pipeline shutdown. When shutdown is requested, skip remaining chunks instead of processing them after the producer has stopped.
…tion - Extract _is_shutdown(event) to replace three inline copies of the shutdown_event is not None and shutdown_event.is_set() pattern - Fix operator precedence ambiguity in _process_one skip condition - Skip produced_messages increment when shutdown fires mid-chunk-loop to avoid inflating the counter for partially-sent messages - Change MessageAssembly.first_error_msg to str | None = None so the no-error state is explicit; read site falls back to 'Unknown error'
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
shutdown_eventthrough to_dispatcher_taskso it stopsaccepting new work items when shutdown is signaled. Adds a drain loop in the
elsebranch so the producer'sput()calls can unblock and deliver theNonesentinel without hanging._producer_taskwhenshutdown_eventis set, preventing a deadlock when the queue is full atshutdown time.
MessageAssemblyas it arrivesinstead of scanning all chunks at skip-time, eliminating an O(N) scan in
_reassembler_task._is_shutdown(event)helper to replace three inlinecopies of
shutdown_event is not None and shutdown_event.is_set(). Fixesoperator precedence ambiguity in
_process_one. Skipsproduced_messagesincrement when shutdown fires mid-chunk-loop to avoid inflating the counter
for partially-sent messages. Changes
MessageAssembly.first_error_msgtostr | None = Noneso the no-error state is explicit.successproperty from theChunkProcessingResultdocstring.Test plan
drains cleanly without hanging.
Nonesentinel is lost (i.e.result_queuestill receives its sentinel).skip_failed_messages=Trueand confirm the correct error messageis logged without the O(N) scan.
add_messagestests to confirm no regressions.Verification notes
All four items verified by exercising the internal pipeline functions directly
through the library boundary (
uv run python).Shutdown / deadlock (items 1 & 2): ran
_producer_task+_dispatcher_taskconcurrently with
maxsize=2and 10 chunks. Producer confirmed blocking beforeshutdown was signaled; both tasks finished within 3 s.
result_queuereceived 5processed results +
Nonesentinel in the correct terminal position.first_error_msg (item 3): fed a two-chunk message where chunk-0 raises
"first error"and chunk-1 raises"second error". Skip log line containedonly
"first error"— the old O(N) scan is gone.Regression suite (item 4):
test_add_messages_pipeline.py+test_add_messages_streaming.py— 45 passed in 0.14 s (after refactor commit).