Skip to content

Fix flaky backend CI worker timeout - #2204

Merged
JSv4 merged 1 commit into
mainfrom
agent/fix-flaky-backend-ci-2170
Jul 26, 2026
Merged

Fix flaky backend CI worker timeout#2204
JSv4 merged 1 commit into
mainfrom
agent/fix-flaky-backend-ci-2170

Conversation

@JSv4

@JSv4 JSv4 commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • isolate the missing-user message vector-search test from database, permission, and embedder fixtures
  • force the intended User.DoesNotExist branch with mocks in a SimpleTestCase
  • use pytest-timeout's signal mode so xdist reports timed-out tests instead of losing a worker and hanging the controller

Root cause

The Backend CI failure observed in #2170 was unrelated to that PR's dependency update. Across repeated failures, an xdist worker died exactly 600 seconds after the second 41% progress marker. A CI-equivalent coverage trace matched that timer start to MessageVectorSearchTest::test_message_search_nonexistent_user within 0.4 ms and on the same worker.

The test exercised a full database/permissions/embedding fixture merely to cover the missing-user branch. When the existing thread-based timeout fired, pytest-timeout called os._exit(1), causing xdist to report node down: Not properly terminated and wait until the outer 100-minute timeout.

Impact

The branch test is now deterministic and dependency-free. If another backend test genuinely hangs, CI will report its node ID and stack as a normal pytest failure instead of silently losing the worker.

Validation

  • isolated missing-user test: 1 passed
  • complete conversation-search module under xdist and signal timeout: 82 passed in 20.12s
  • controlled 50 ms timeout: reported the exact test and exited xdist cleanly without node down
  • black --check opencontractserver/tests/test_conversation_search.py
  • flake8 opencontractserver/tests/test_conversation_search.py
  • git diff --check

A clean full backend run reached 26% without failures before it was stopped to publish the fix.

@JSv4
JSv4 marked this pull request as ready for review July 26, 2026 03:24
@codecov

codecov Bot commented Jul 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@JSv4
JSv4 merged commit e9f1717 into main Jul 26, 2026
12 of 13 checks passed
@JSv4
JSv4 deleted the agent/fix-flaky-backend-ci-2170 branch July 26, 2026 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant