Steps:
- run
luxonis/rae-ros-robot:humble docker image
- run
bringup.launch.py
Expected: nodes are successfully started
Actual: VPU has crashed
Randomly caught it several times in the past week. But in general, RAE can't easily recover from this state. Only hard shutdown helps to resurrect VPU again. Here's what I caught in logs when it happened:
ros2 launch rae_bringup robot.launch.py
[INFO] [launch]: All log files can be found below /root/.ros/log/2023-10-21-19-12-13-121861-rae-901
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container-1]: process started with pid [915]
[INFO] [robot_state_publisher-2]: process started with pid [917]
[INFO] [ekf_node-3]: process started with pid [919]
[INFO] [ros2_control_node-4]: process started with pid [921]
[INFO] [spawner-5]: process started with pid [923]
[INFO] [spawner-6]: process started with pid [925]
[INFO] [mic_node-7]: process started with pid [927]
[INFO] [speakers_node-8]: process started with pid [929]
[INFO] [battery_status.py-9]: process started with pid [931]
[component_container-1] [INFO] [1697915536.759803476] [rae_container]: Load Library: /underlay_ws/install/depthai_ros_driver/lib/libdepthai_ros_driver.so
[robot_state_publisher-2] [INFO] [1697915537.039387815] [robot_state_publisher]: got segment base_footprint
[robot_state_publisher-2] [INFO] [1697915537.046802611] [robot_state_publisher]: got segment base_link
[robot_state_publisher-2] [INFO] [1697915537.046892902] [robot_state_publisher]: got segment base_link_back
[robot_state_publisher-2] [INFO] [1697915537.046931986] [robot_state_publisher]: got segment chassis
[robot_state_publisher-2] [INFO] [1697915537.046968277] [robot_state_publisher]: got segment left_caster_wheel
[robot_state_publisher-2] [INFO] [1697915537.047004403] [robot_state_publisher]: got segment left_wheel
[robot_state_publisher-2] [INFO] [1697915537.047040319] [robot_state_publisher]: got segment rae_imu_frame
[robot_state_publisher-2] [INFO] [1697915537.047076486] [robot_state_publisher]: got segment rae_left_back_camera_frame
[robot_state_publisher-2] [INFO] [1697915537.047113361] [robot_state_publisher]: got segment rae_left_back_camera_optical_frame
[robot_state_publisher-2] [INFO] [1697915537.047152111] [robot_state_publisher]: got segment rae_left_camera_frame
[robot_state_publisher-2] [INFO] [1697915537.047189069] [robot_state_publisher]: got segment rae_left_camera_optical_frame
[robot_state_publisher-2] [INFO] [1697915537.047226486] [robot_state_publisher]: got segment rae_rgb_camera_frame
[robot_state_publisher-2] [INFO] [1697915537.047473194] [robot_state_publisher]: got segment rae_rgb_camera_optical_frame
[robot_state_publisher-2] [INFO] [1697915537.054898324] [robot_state_publisher]: got segment rae_right_back_camera_frame
[robot_state_publisher-2] [INFO] [1697915537.054986074] [robot_state_publisher]: got segment rae_right_back_camera_optical_frame
[robot_state_publisher-2] [INFO] [1697915537.055050741] [robot_state_publisher]: got segment rae_right_camera_frame
[robot_state_publisher-2] [INFO] [1697915537.055092282] [robot_state_publisher]: got segment rae_right_camera_optical_frame
[robot_state_publisher-2] [INFO] [1697915537.055156282] [robot_state_publisher]: got segment right_caster_wheel
[robot_state_publisher-2] [INFO] [1697915537.055196574] [robot_state_publisher]: got segment right_wheel
[ros2_control_node-4] [INFO] [1697915537.299739972] [resource_manager]: Loading hardware 'RAE'
[ekf_node-3] X acceleration is being measured from IMU; X velocity control input is disabled
[ekf_node-3] use_control is set to true, but control_config has only false values. No control term will be used.
[ros2_control_node-4] [INFO] [1697915537.981454801] [resource_manager]: Initialize hardware 'RAE'
[ros2_control_node-4] [INFO] [1697915537.990737432] [resource_manager]: Successful initialization of hardware 'RAE'
[ros2_control_node-4] [INFO] [1697915537.991079349] [resource_manager]: 'configure' hardware 'RAE'
[ros2_control_node-4] [INFO] [1697915538.000984146] [resource_manager]: Successful 'configure' of hardware 'RAE'
[ros2_control_node-4] [INFO] [1697915538.008744693] [resource_manager]: 'activate' hardware 'RAE'
[ros2_control_node-4] [INFO] [1697915538.015841280] [resource_manager]: Successful 'activate' of hardware 'RAE'
[ros2_control_node-4] [INFO] [1697915538.132318475] [controller_manager]: update rate is 100 Hz
[ros2_control_node-4] [INFO] [1697915538.146974192] [controller_manager]: RT kernel is recommended for better performance
[component_container-1] [INFO] [1697915538.277393936] [rae_container]: Found class: rclcpp_components::NodeFactoryTemplate<depthai_ros_driver::Camera>
[component_container-1] [INFO] [1697915538.277898311] [rae_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<depthai_ros_driver::Camera>
[mic_node-7] [INFO] [1697915538.366772781] [mic_node]: Mic node running!
[speakers_node-8] [INFO] [1697915538.377270120] [speakers_node]: Speakers node running!
[component_container-1] [INFO] [1697915538.409901557] [rae]: No ip/mxid specified, connecting to the next available device.
[component_container-1] [2023-10-21 19:12:18.440] [depthai] [warning] USB protocol not available - If running in a container, make sure that the following is set: "-v /dev/bus/usb:/dev/bus/usb --device-cgroup-rule='c 189:* rmw'"
[ros2_control_node-4] [INFO] [1697915540.690126724] [controller_manager]: Loading controller 'joint_state_broadcaster'
[ros2_control_node-4] [INFO] [1697915540.934130615] [controller_manager]: Loading controller 'diff_controller'
[spawner-5] [INFO] [1697915541.254460005] [spawner_diff_controller]: Loaded diff_controller
[ros2_control_node-4] [INFO] [1697915541.274109266] [controller_manager]: Configuring controller 'diff_controller'
[spawner-6] [INFO] [1697915541.291942735] [spawner_joint_state_broadcaster]: Loaded joint_state_broadcaster
[ros2_control_node-4] [INFO] [1697915541.325846754] [controller_manager]: Configuring controller 'joint_state_broadcaster'
[ros2_control_node-4] [INFO] [1697915541.326277421] [joint_state_broadcaster]: 'joints' or 'interfaces' parameter is empty. All available state interfaces will be published
[spawner-5] [INFO] [1697915541.398116378] [spawner_diff_controller]: Configured and activated diff_controller
[spawner-6] [INFO] [1697915541.449936283] [spawner_joint_state_broadcaster]: Configured and activated joint_state_broadcaster
[INFO] [spawner-5]: process has finished cleanly [pid 923]
[INFO] [spawner-6]: process has finished cleanly [pid 925]
[component_container-1] [INFO] [1697915545.540972979] [rae]: Camera with MXID: xlinkserver and Name: 127.0.0.1 connected!
[component_container-1] [INFO] [1697915545.541175062] [rae]: PoE camera detected. Consider enabling low bandwidth for specific image topics (see readme).
[component_container-1] [INFO] [1697915545.573147329] [rae]: Device type: RAE
[component_container-1] [INFO] [1697915545.862981941] [rae]: Pipeline type: rae
[component_container-1] [INFO] [1697915546.948649755] [rae]: Finished setting up pipeline.
[component_container-1] [INFO] [1697915547.680929336] [rae]: Camera ready!
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/rae' in container '/rae_container'
[component_container-1] [INFO] [1697915547.686688089] [rae_container]: Load Library: /ws/install/rae_hw/lib/librae_hw.so
[component_container-1] [INFO] [1697915547.755684291] [rae_container]: Found class: rclcpp_components::NodeFactoryTemplate<rae_hw::BatteryNode>
[component_container-1] [INFO] [1697915547.755879708] [rae_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<rae_hw::BatteryNode>
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/battery_node' in container 'rae_container'
[component_container-1] [INFO] [1697915547.825451743] [battery_node]: Battery node running!
[component_container-1] [INFO] [1697915547.841370918] [rae_container]: Found class: rclcpp_components::NodeFactoryTemplate<rae_hw::BatteryNode>
[component_container-1] [INFO] [1697915547.841543918] [rae_container]: Found class: rclcpp_components::NodeFactoryTemplate<rae_hw::LCDNode>
[component_container-1] [INFO] [1697915547.841583751] [rae_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<rae_hw::LCDNode>
[component_container-1] [ERROR] [1697915548.034614974] [rae_container]: Component constructor threw an exception: Statically typed parameter 'default_logo_path' must be initialized.
[ERROR] [launch_ros.actions.load_composable_nodes]: Failed to load node 'lcd_node' of type 'rae_hw::LCDNode' in container 'rae_container': Component constructor threw an exception: Statically typed parameter 'default_logo_path' must be initialized.
[component_container-1] [INFO] [1697915548.052709400] [rae_container]: Found class: rclcpp_components::NodeFactoryTemplate<rae_hw::BatteryNode>
[component_container-1] [INFO] [1697915548.052865067] [rae_container]: Found class: rclcpp_components::NodeFactoryTemplate<rae_hw::LCDNode>
[component_container-1] [INFO] [1697915548.053017484] [rae_container]: Found class: rclcpp_components::NodeFactoryTemplate<rae_hw::LEDNode>
[component_container-1] [INFO] [1697915548.053061109] [rae_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<rae_hw::LEDNode>
[component_container-1] [INFO] [1697915548.164287664] [led_node]: LED node running!
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/led_node' in container 'rae_container'
[component_container-1] [INFO] [1697915548.329528997] [battery_node]: Power supply status changed to [Charging] after 0 h 0 min 0 secs.
[component_container-1] [4B928816838860C6] [127.0.0.1] [1697915548.551] [system] [critical] VPU crash dump found, saved to: /tmp/gate_fw_vje7sd3t/485e53f7b26b4c79c89d13afc817c7c279630434c15fb52103ffa5c792c86074/crashDump.json
[INFO] [launch.user]: Resetting PWM.
[INFO] [busybox devmem 0x20320180 32 0x00000000-10]: process started with pid [1045]
[INFO] [busybox devmem 0x20320180 32 0x00000000-10]: process has finished cleanly [pid 1045]
[component_container-1] [INFO] [1697915553.828914946] [battery_node]: Power supply status changed to [Discharging] after 0 h 0 min 5 secs.
[component_container-1] [4B928816838860C6] [127.0.0.1] [1697915558.548] [StereoDepth(12)] [error] Node threw exception CallVpu error: -1
Key takes:
[component_container-1] [4B928816838860C6] [127.0.0.1] [1697915548.551] [system] [critical] VPU crash dump found, saved to: /tmp/gate_fw_vje7sd3t/485e53f7b26b4c79c89d13afc817c7c279630434c15fb52103ffa5c792c86074/crashDump.json
[component_container-1] [4B928816838860C6] [127.0.0.1] [1697915558.548] [StereoDepth(12)] [error] Node threw exception CallVpu error: -1
Attaching the crash dump.
Note that I tried to reload the container and even restart RAE, but every further run caused the same error that there are no available devices:
root@rae:/ws# ros2 launch rae_bringup robot.launch.py
[INFO] [launch]: All log files can be found below /root/.ros/log/2023-10-21-19-18-37-837447-rae-1103
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container-1]: process started with pid [1117]
[INFO] [robot_state_publisher-2]: process started with pid [1119]
[INFO] [ekf_node-3]: process started with pid [1121]
[INFO] [ros2_control_node-4]: process started with pid [1123]
[INFO] [spawner-5]: process started with pid [1125]
[INFO] [spawner-6]: process started with pid [1127]
[INFO] [mic_node-7]: process started with pid [1129]
[INFO] [speakers_node-8]: process started with pid [1133]
[INFO] [battery_status.py-9]: process started with pid [1142]
[component_container-1] [INFO] [1697915920.665089864] [rae_container]: Load Library: /ws/install/rae_hw/lib/librae_hw.so
[robot_state_publisher-2] [INFO] [1697915920.780842500] [robot_state_publisher]: got segment base_footprint
[robot_state_publisher-2] [INFO] [1697915920.788166407] [robot_state_publisher]: got segment base_link
[robot_state_publisher-2] [INFO] [1697915920.795050689] [robot_state_publisher]: got segment base_link_back
[robot_state_publisher-2] [INFO] [1697915920.795129314] [robot_state_publisher]: got segment chassis
[robot_state_publisher-2] [INFO] [1697915920.795167981] [robot_state_publisher]: got segment left_caster_wheel
[robot_state_publisher-2] [INFO] [1697915920.799579600] [robot_state_publisher]: got segment left_wheel
[robot_state_publisher-2] [INFO] [1697915920.799822350] [robot_state_publisher]: got segment rae_imu_frame
[robot_state_publisher-2] [INFO] [1697915920.799864183] [robot_state_publisher]: got segment rae_left_back_camera_frame
[robot_state_publisher-2] [INFO] [1697915920.799902850] [robot_state_publisher]: got segment rae_left_back_camera_optical_frame
[robot_state_publisher-2] [INFO] [1697915920.799941724] [robot_state_publisher]: got segment rae_left_camera_frame
[robot_state_publisher-2] [INFO] [1697915920.799979599] [robot_state_publisher]: got segment rae_left_camera_optical_frame
[robot_state_publisher-2] [INFO] [1697915920.800016516] [robot_state_publisher]: got segment rae_rgb_camera_frame
[robot_state_publisher-2] [INFO] [1697915920.800811723] [robot_state_publisher]: got segment rae_rgb_camera_optical_frame
[robot_state_publisher-2] [INFO] [1697915920.800916598] [robot_state_publisher]: got segment rae_right_back_camera_frame
[robot_state_publisher-2] [INFO] [1697915920.800957223] [robot_state_publisher]: got segment rae_right_back_camera_optical_frame
[robot_state_publisher-2] [INFO] [1697915920.800995181] [robot_state_publisher]: got segment rae_right_camera_frame
[robot_state_publisher-2] [INFO] [1697915920.801032723] [robot_state_publisher]: got segment rae_right_camera_optical_frame
[robot_state_publisher-2] [INFO] [1697915920.801070265] [robot_state_publisher]: got segment right_caster_wheel
[robot_state_publisher-2] [INFO] [1697915920.801107348] [robot_state_publisher]: got segment right_wheel
[ekf_node-3] X acceleration is being measured from IMU; X velocity control input is disabled
[ekf_node-3] use_control is set to true, but control_config has only false values. No control term will be used.
[ros2_control_node-4] [INFO] [1697915920.987014140] [resource_manager]: Loading hardware 'RAE'
[ros2_control_node-4] [INFO] [1697915921.551319424] [resource_manager]: Initialize hardware 'RAE'
[ros2_control_node-4] [INFO] [1697915921.555050211] [resource_manager]: Successful initialization of hardware 'RAE'
[ros2_control_node-4] [INFO] [1697915921.555403461] [resource_manager]: 'configure' hardware 'RAE'
[ros2_control_node-4] [INFO] [1697915921.564000699] [resource_manager]: Successful 'configure' of hardware 'RAE'
[ros2_control_node-4] [INFO] [1697915921.564155574] [resource_manager]: 'activate' hardware 'RAE'
[ros2_control_node-4] [INFO] [1697915921.564192532] [resource_manager]: Successful 'activate' of hardware 'RAE'
[ros2_control_node-4] [INFO] [1697915921.665503563] [controller_manager]: update rate is 100 Hz
[ros2_control_node-4] [INFO] [1697915921.691775486] [controller_manager]: RT kernel is recommended for better performance
[component_container-1] [INFO] [1697915921.767732550] [rae_container]: Found class: rclcpp_components::NodeFactoryTemplate<rae_hw::BatteryNode>
[component_container-1] [INFO] [1697915921.773583459] [rae_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<rae_hw::BatteryNode>
[mic_node-7] [INFO] [1697915921.790000562] [mic_node]: Mic node running!
[component_container-1] [INFO] [1697915921.941992483] [battery_node]: Battery node running!
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/battery_node' in container 'rae_container'
[component_container-1] [INFO] [1697915921.944774521] [rae_container]: Load Library: /underlay_ws/install/depthai_ros_driver/lib/libdepthai_ros_driver.so
[component_container-1] [INFO] [1697915922.228020682] [rae_container]: Found class: rclcpp_components::NodeFactoryTemplate<depthai_ros_driver::Camera>
[component_container-1] [INFO] [1697915922.229131097] [rae_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<depthai_ros_driver::Camera>
[speakers_node-8] [INFO] [1697915922.275798368] [speakers_node]: Speakers node running!
[component_container-1] [INFO] [1697915922.341636987] [rae]: No ip/mxid specified, connecting to the next available device.
[component_container-1] [2023-10-21 19:18:42.353] [depthai] [warning] USB protocol not available - If running in a container, make sure that the following is set: "-v /dev/bus/usb:/dev/bus/usb --device-cgroup-rule='c 189:* rmw'"
[ros2_control_node-4] [INFO] [1697915923.732574240] [controller_manager]: Loading controller 'joint_state_broadcaster'
[spawner-6] [INFO] [1697915924.302596847] [spawner_joint_state_broadcaster]: Loaded joint_state_broadcaster
[ros2_control_node-4] [INFO] [1697915924.338333965] [controller_manager]: Configuring controller 'joint_state_broadcaster'
[ros2_control_node-4] [INFO] [1697915924.338870840] [joint_state_broadcaster]: 'joints' or 'interfaces' parameter is empty. All available state interfaces will be published
[spawner-6] [INFO] [1697915924.428230386] [spawner_joint_state_broadcaster]: Configured and activated joint_state_broadcaster
[INFO] [spawner-6]: process has finished cleanly [pid 1127]
[ros2_control_node-4] [INFO] [1697915925.100175812] [controller_manager]: Loading controller 'diff_controller'
[spawner-5] [INFO] [1697915925.216702405] [spawner_diff_controller]: Loaded diff_controller
[ros2_control_node-4] [INFO] [1697915925.225400351] [controller_manager]: Configuring controller 'diff_controller'
[spawner-5] [INFO] [1697915925.288347850] [spawner_diff_controller]: Configured and activated diff_controller
[INFO] [spawner-5]: process has finished cleanly [pid 1125]
[component_container-1] [ERROR] [1697915926.491951889] [rae]: No available devices (3 connected, but in use)
[component_container-1] [INFO] [1697915926.492175013] [rae]: No ip/mxid specified, connecting to the next available device.
[component_container-1] [ERROR] [1697915930.623728570] [rae]: No available devices (3 connected, but in use)
[component_container-1] [INFO] [1697915930.623943945] [rae]: No ip/mxid specified, connecting to the next available device.
[component_container-1] [ERROR] [1697915934.711731984] [rae]: No available devices (3 connected, but in use)
[component_container-1] [INFO] [1697915934.711957650] [rae]: No ip/mxid specified, connecting to the next available device.
[INFO] [launch.user]: Resetting PWM.
[INFO] [busybox devmem 0x20320180 32 0x00000000-10]: process started with pid [1232]
[INFO] [busybox devmem 0x20320180 32 0x00000000-10]: process has finished cleanly [pid 1232]
[component_container-1] [ERROR] [1697915938.803749733] [rae]: No available devices (3 connected, but in use)
[component_container-1] [INFO] [1697915938.803962149] [rae]: No ip/mxid specified, connecting to the next available device.
[component_container-1] [ERROR] [1697915942.931719689] [rae]: No available devices (3 connected, but in use)
[component_container-1] [INFO] [1697915942.931935689] [rae]: No ip/mxid specified, connecting to the next available device.
[component_container-1] [ERROR] [1697915947.019724000] [rae]: No available devices (3 connected, but in use)
[component_container-1] [INFO] [1697915947.019950500] [rae]: No ip/mxid specified, connecting to the next available device.
[component_container-1] [ERROR] [1697915951.143768853] [rae]: No available devices (3 connected, but in use)
[component_container-1] [INFO] [1697915951.144122394] [rae]: No ip/mxid specified, connecting to the next available device.
[component_container-1] [ERROR] [1697915955.219725240] [rae]: No available devices (3 connected, but in use)
[component_container-1] [INFO] [1697915955.219937240] [rae]: No ip/mxid specified, connecting to the next available device.
[component_container-1] [ERROR] [1697915959.327722800] [rae]: No available devices (3 connected, but in use)
[component_container-1] [INFO] [1697915959.327946675] [rae]: No ip/mxid specified, connecting to the next available device.
[component_container-1] [ERROR] [1697915963.459735336] [rae]: No available devices (3 connected, but in use)
[component_container-1] [INFO] [1697915963.459943961] [rae]: No ip/mxid specified, connecting to the next available device.
[component_container-1] [ERROR] [1697915967.575717488] [rae]: No available devices (3 connected, but in use)
[component_container-1] [INFO] [1697915967.575947988] [rae]: No ip/mxid specified, connecting to the next available device.
Steps:
luxonis/rae-ros-robot:humbledocker imagebringup.launch.pyExpected: nodes are successfully started
Actual: VPU has crashed
Randomly caught it several times in the past week. But in general, RAE can't easily recover from this state. Only hard shutdown helps to resurrect VPU again. Here's what I caught in logs when it happened:
Key takes:
Attaching the crash dump.
Note that I tried to reload the container and even restart RAE, but every further run caused the same error that there are no available devices: