Tools: ros2: tear down rclpy cleanly in the AP_DDS tests - #33894
Open
peterbarker wants to merge 1 commit into
Open
Tools: ros2: tear down rclpy cleanly in the AP_DDS tests#33894peterbarker wants to merge 1 commit into
peterbarker wants to merge 1 commit into
Conversation
peterbarker
force-pushed
the
pr-claude2/dds-test-teardown
branch
from
July 30, 2026 23:32
5fbe203 to
80ac7ee
Compare
Every test in ardupilot_dds_tests tore down with nothing but rclpy.shutdown(), which shuts the context down while that context's node, subscriptions and DDS participant are all still alive, and left its rclpy.spin() thread unjoined. colcon runs the whole package in one interpreter, so those zombie endpoints accumulated from test to test. They race with Fast-DDS intra-process delivery, and the process intermittently died with SIGSEGV on an internal DDS thread: History::get_change_nts StatefulReader::NotifyChanges StatefulReader::change_received StatefulReader::processDataMsg StatefulWriter::deliver_sample_to_intraprocesses FlowControllerImpl<...>::run Both CI occurrences were in the 22nd of the 23 tests, i.e. at maximum accumulation, and killed the whole pytest run with exit code -11. rclpy.spin() cannot be torn down cleanly: it uses a process-wide global executor bound to whichever context was current when the first test span, and the only way to make it return is to shut the context down. So spin on a private SingleThreadedExecutor via a ros_node() context manager, which shuts the executor down, joins the spin thread and destroys the node while the context is still valid. Reproduced by repeating the time tests in one pytest process (pytest --count=40). Over 6 shards x 4 loops, with identical harness and only this change differing: 4 of 24 loops segfaulted before, 0 of 24 after, with all 80 tests passing in every loop. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
peterbarker
force-pushed
the
pr-claude2/dds-test-teardown
branch
from
July 31, 2026 04:05
80ac7ee to
8152eb1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes segfault recently starting in colcon tests
Classification & Testing (check all that apply and add your own)
Description
Every test in ardupilot_dds_tests tore down with nothing but rclpy.shutdown(), which shuts the context down while that context's node, subscriptions and DDS participant are all still alive, and left its rclpy.spin() thread unjoined. colcon runs the whole package in one interpreter, so those zombie endpoints accumulated from test to test.
They race with Fast-DDS intra-process delivery, and the process intermittently died with SIGSEGV on an internal DDS thread:
History::get_change_nts
StatefulReader::NotifyChanges
StatefulReader::change_received
StatefulReader::processDataMsg
StatefulWriter::deliver_sample_to_intraprocesses
FlowControllerImpl<...>::run
Both CI occurrences were in the 22nd of the 23 tests, i.e. at maximum accumulation, and killed the whole pytest run with exit code -11.
rclpy.spin() cannot be torn down cleanly: it uses a process-wide global executor bound to whichever context was current when the first test span, and the only way to make it return is to shut the context down. So spin on a private SingleThreadedExecutor via a ros_node() context manager, which shuts the executor down, joins the spin thread and destroys the node while the context is still valid.
Reproduced by repeating the time tests in one pytest process (pytest --count=40). Over 6 shards x 4 loops, with identical harness and only this change differing: 4 of 24 loops segfaulted before, 0 of 24 after, with all 80 tests passing in every loop.