Skip to content

Stop a hung CI test run and keep evidence of where it hangs - #852

Merged
onevcat merged 1 commit into
mainfrom
ci/test-hang-watchdog
Oct 4, 2026
Merged

onevcat merged 1 commit into
mainfrom
ci/test-hang-watchdog

Conversation

@onevcat

@onevcat onevcat commented Oct 4, 2026

Copy link
Copy Markdown
Owner

Problem

In run 37164745593 (#851), the build job built the app, then the first make test-app pass printed nothing for 39 minutes until the 45-minute job timeout cancelled it. The same code had passed one hour earlier. The run kept no evidence of the hung test:

  • xcodebuild sends test results in bursts, so a hang shows no test lines. NSUnbufferedIO=YES does not help (compared locally).
  • The xcresult upload ran only on failure(); a timed-out job is cancelled.
  • Nothing sampled the processes while they hung.

Change

  • scripts/ci-test-watchdog.py runs beside the CI tasks. When xcodebuild runs and no build/test-results/*.xcresult.log changes for 10 minutes, it:
    1. Prints an error annotation.
    2. Writes the process table and sample output of xcodebuild, xctest and the test host (processes started from the build products) to build/ci-logs/hang/, which the logs artifact already uploads.
    3. Sends SIGINT to xcodebuild (SIGKILL after 3 minutes if it does not stop).
  • An interrupted xcodebuild still writes the result bundle and marks the running test as "Testing was canceled", so the step fails about 10 minutes after the last output and names the hung test.
  • The xcresult bundle is also uploaded when the job is cancelled. The step summary says when the watchdog stopped the run.
  • docs-ai 016.009 records the design and the verification.

The 10-minute limit is far above normal gaps: the build prints continuously, and gaps in test output are about 2 minutes on a slow runner.

Verification

  • scripts/test_ci_test_watchdog.py: stall detection, log times, and which processes are interrupted and sampled (not the watchdog itself, not an unrelated xcodebuild).
  • Local end-to-end run on an iOS 27 simulator with a temporary test that never returns (while true { sleep(1) }, removed again), with a 30-second limit: the watchdog sampled xcodebuild and the test host, whose stack contains WatchdogHangProbeTests.testHangs(). xcodebuild printed ** TEST INTERRUPTED ** about 30 seconds after SIGINT, and the result bundle reported WatchdogHangProbeTests/testHangs() with "Testing was canceled".
  • make check and actionlint pass.

A hung xcodebuild test pass printed nothing for 39 minutes until the job
timeout, and the cancelled job kept no result bundle. A watchdog now runs
beside the CI tasks. After 10 minutes without test output it samples
xcodebuild and the test processes into build/ci-logs/hang and sends SIGINT
to xcodebuild. The interrupted xcodebuild still writes the result bundle,
which marks the running test as canceled, so the step fails early and names
the hung test. The xcresult bundle is also uploaded when the job is
cancelled.
@onevcat
onevcat merged commit ceec23f into main Oct 4, 2026
1 check passed
@onevcat
onevcat deleted the ci/test-hang-watchdog branch October 4, 2026 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant