Repository navigation
Stop a hung CI test run and keep evidence of where it hangs - #852
Merged
Merged
Conversation
A hung xcodebuild test pass printed nothing for 39 minutes until the job timeout, and the cancelled job kept no result bundle. A watchdog now runs beside the CI tasks. After 10 minutes without test output it samples xcodebuild and the test processes into build/ci-logs/hang and sends SIGINT to xcodebuild. The interrupted xcodebuild still writes the result bundle, which marks the running test as canceled, so the step fails early and names the hung test. The xcresult bundle is also uploaded when the job is cancelled.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In run 37164745593 (#851), the
buildjob built the app, then the firstmake test-apppass printed nothing for 39 minutes until the 45-minute job timeout cancelled it. The same code had passed one hour earlier. The run kept no evidence of the hung test:NSUnbufferedIO=YESdoes not help (compared locally).failure(); a timed-out job is cancelled.Change
scripts/ci-test-watchdog.pyruns beside the CI tasks. When xcodebuild runs and nobuild/test-results/*.xcresult.logchanges for 10 minutes, it:sampleoutput of xcodebuild,xctestand the test host (processes started from the build products) tobuild/ci-logs/hang/, which the logs artifact already uploads.The 10-minute limit is far above normal gaps: the build prints continuously, and gaps in test output are about 2 minutes on a slow runner.
Verification
scripts/test_ci_test_watchdog.py: stall detection, log times, and which processes are interrupted and sampled (not the watchdog itself, not an unrelated xcodebuild).while true { sleep(1) }, removed again), with a 30-second limit: the watchdog sampled xcodebuild and the test host, whose stack containsWatchdogHangProbeTests.testHangs(). xcodebuild printed** TEST INTERRUPTED **about 30 seconds after SIGINT, and the result bundle reportedWatchdogHangProbeTests/testHangs()with "Testing was canceled".make checkand actionlint pass.