[core] Fix driver SIGABRT + core dump when SIGTERM arrives during a slow graceful shutdown

[Ti1mmy]

Description

On POSIX, Ray registers SIGTERM with abseil's failure signal handler:

ray/src/ray/util/logging.cc

Line 437 in 2a70ac4

std::vector<int> installed_signals({SIGSEGV, SIGILL, SIGFPE, SIGABRT, SIGTERM});

Here, logging.cc sets only call_previous_handler and writerfn on the options struct, leaves every other option at its default — alarm_on_failure_secs defaults to 3 seconds — and immediately passes it to absl::InstallFailureSignalHandler:

ray/src/ray/util/logging.cc

Lines 506 to 509 in 2a70ac4

	absl::FailureSignalHandlerOptions options;
	options.call_previous_handler = call_previous_handler;
	options.writerfn = WriteFailureMessage;
	absl::InstallFailureSignalHandler(options);

So when SIGTERM arrives, abseil's handler runs first and arms a 3-second abort timer before chaining to the previous handler (absl/debugging/failure_signal_handler.cc#L405-L409):

if (fsh_options.alarm_on_failure_secs > 0) {
  alarm(0);  // Cancel any existing alarms.
  signal(SIGALRM, ImmediateAbortSignalHandler);
  alarm(static_cast<unsigned int>(fsh_options.alarm_on_failure_secs));
}

The chained handler is Ray's Python SIGTERM handler, which recovers into a graceful sys.exit() that tears down the driver-owned local node. A Jupyter notebook kernel restart typically takes more than 3 seconds to clean up (the kernel's Ray daemons received the same process-group SIGTERM, so the driver stalls waiting on them while they die). So before the cleanup finishes, the alarm goes off and sends the ABORT signal — and that SIGABRT is what ends up creating the multi-GB core dump in the notebook's working directory on every restart.

Fix: cancel the pending alarm (signal.alarm(0)) before invoking any handler installed via ray._private.utils.set_sigterm_handler(). Every handler installed through that function recovers into a graceful shutdown, so this applies to exactly the affected code paths; abseil's abort-timer protection for genuine crash signals (SIGSEGV, etc.) is unaffected.

Includes a regression test: a driver whose exit outlives the 3s timer must exit gracefully with code 15 instead of being killed by SIGABRT.

Related issues

Fixes #64829

Additional information

Verification (the fix is pure Python; validated by applying this exact patch to the released 2.56.0 wheel):

• Repro from the issue: exit -6 (SIGABRT) 3.0s after SIGTERM before the fix → clean exit 15 after (3/3 runs)
• pytest python/ray/tests/test_ray_shutdown.py -k test_sigterm_during_slow_graceful_exit_does_not_abort: fails (-6) before, passes after
• pre-commit run --files python/ray/_private/utils.py python/ray/tests/test_ray_shutdown.py: all hooks pass

Duplicate-work check: no linked PRs on #64829 and no open PRs found for this area.

AI assistance disclosure: prepared with AI assistance (Claude Code); the submitter reviewed every changed line and ran the repro and tests locally.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request wraps the SIGTERM handler in Ray to cancel any active SIGALRM timer (using signal.alarm(0)), preventing abseil's failure signal handler from aborting the process during a slow graceful shutdown. A regression test has also been added. The reviewer correctly identified a potential TypeError if the provided sigterm_handler is not callable (such as signal.SIG_DFL or signal.SIG_IGN), and suggested adding a check for callability and verifying the existence of signal.alarm for better compatibility.

[gemini-code-assist]

If sigterm_handler is not a callable (for example, if it is signal.SIG_DFL or signal.SIG_IGN), wrapping it in sigterm_handler_wrapper and calling sigterm_handler(signum, frame) will raise a TypeError when the signal is received. To prevent this, we should check if sigterm_handler is callable before wrapping it. Additionally, checking hasattr(signal, "alarm") ensures compatibility with non-standard environments where signal.alarm might not be available.

        if callable(sigterm_handler):

            def sigterm_handler_wrapper(signum, frame):
                # abseil's failure signal handler (Ray registers SIGTERM with it)
                # has already armed a SIGALRM timer that SIGABRTs the process
                # after 3s. Cancel it, since handlers installed here recover into
                # a graceful shutdown that may legitimately take longer.
                if hasattr(signal, "alarm"):
                    signal.alarm(0)
                sigterm_handler(signum, frame)

            signal.signal(signal.SIGTERM, sigterm_handler_wrapper)
        else:
            signal.signal(signal.SIGTERM, sigterm_handler)

[Copilot]

Pull request overview

This PR addresses a SIGTERM shutdown edge case where Ray’s SIGTERM handler recovers into a graceful Python exit while Abseil’s failure-signal handler has already armed a 3s SIGALRM abort timer, leading to SIGABRT + large core dumps during slow graceful shutdowns (e.g., Jupyter kernel restarts).

Changes:

• Cancel any pending POSIX alarm() in Ray’s SIGTERM handler path (via set_sigterm_handler) to prevent Abseil’s abort timer from firing during graceful shutdown.
• Add a regression test that reproduces the slow-graceful-exit-over-3s scenario and asserts the driver exits cleanly with SIGTERM (15), not SIGABRT.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
python/ray/_private/utils.py	Wraps the POSIX SIGTERM handler to call signal.alarm(0) before executing the existing graceful handler.
python/ray/tests/test_ray_shutdown.py	Adds a regression test to ensure SIGTERM during a slow graceful exit does not result in SIGABRT/core dump.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.