Skip to content

Dds test fix - #33682

Open
khancyr wants to merge 1 commit into
ArduPilot:masterfrom
khancyr:dds_test_fix
Open

Dds test fix#33682
khancyr wants to merge 1 commit into
ArduPilot:masterfrom
khancyr:dds_test_fix

Conversation

@khancyr

@khancyr khancyr commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Fix test_joy_msg_received.py's process_climb runs while True: with no timeout, only breaking once climbing is detected.
The other patch is override by #33894

Classification & Testing (check all that apply and add your own)

  • Checked by a human programmer
  • Non-functional change
  • No-binary change
  • Infrastructure change (e.g. unit tests, helper scripts)
  • Automated test(s) verify changes (e.g. unit test, autotest)
  • Tested manually, description below (e.g. SITL)
  • Tested on hardware
  • Logs attached
  • Logs available on request

Description

Tested locally

@github-actions github-actions Bot added the Python label Jul 9, 2026
@Ryanf55

Ryanf55 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

This is now the 4th or 5th attempt. What evidence do you have that threads were still running and left running or shut down in the wrong order?

@khancyr

khancyr commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

From global analysis. The issue is segfault on python layer.
I did a comparaison between the new container and the old one, python code (pip packages and build one's are the same beside some colcon one's but the patch version change and changelog show nothing important), other package are working
DDS build is working fine
Only colcon dds_test is failing: randomly and on different tests, but always with segfault.

Local test didn't permit to reproduce the issue, but show really inconsistent timing for the tests.

the log analysis : https://github.com/ArduPilot/ardupilot/actions/runs/29001587714/job/86063191686#step:12:1257 show that finalize_launch_service is where the issue happens.
From there, we don't have a lot of culprit.
Looking at how we use the thread we can see that they are all using (or likely)
self.ros_spin_thread = threading.Thread(target=lambda node: rclpy.spin(node), args=(self,))

so they are blocking on spin() and are unmanaged thread (out of context of ros and pytest)
When we call rclpy.shutdown(), it should unblock the spin (according to the doc)
As the thread is a background one's, it could be still running when we go to another test makeing spin() working again and triggering the segfault
Therefore waiting for the thread truely stop on the test seems the right behavior there, so we manage the lifetime of the node and thread correctly.

I cannot garantee that is the true issue but at least this one exists to me.

@peterbarker

Copy link
Copy Markdown
Contributor

Ping @Ryanf55

@khancyr
khancyr force-pushed the dds_test_fix branch 2 times, most recently from 2b1b473 to 7d997ad Compare July 20, 2026 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants