Fix flaky backend CI worker timeout - #2204
Merged
Merged
Conversation
JSv4
marked this pull request as ready for review
July 26, 2026 03:24
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
User.DoesNotExistbranch with mocks in aSimpleTestCaseRoot cause
The Backend CI failure observed in #2170 was unrelated to that PR's dependency update. Across repeated failures, an xdist worker died exactly 600 seconds after the second 41% progress marker. A CI-equivalent coverage trace matched that timer start to
MessageVectorSearchTest::test_message_search_nonexistent_userwithin 0.4 ms and on the same worker.The test exercised a full database/permissions/embedding fixture merely to cover the missing-user branch. When the existing thread-based timeout fired, pytest-timeout called
os._exit(1), causing xdist to reportnode down: Not properly terminatedand wait until the outer 100-minute timeout.Impact
The branch test is now deterministic and dependency-free. If another backend test genuinely hangs, CI will report its node ID and stack as a normal pytest failure instead of silently losing the worker.
Validation
1 passed82 passedin 20.12snode downblack --check opencontractserver/tests/test_conversation_search.pyflake8 opencontractserver/tests/test_conversation_search.pygit diff --checkA clean full backend run reached 26% without failures before it was stopped to publish the fix.