X_LINK_DEVICE_NOT_FOUND with Depthai ROS on successive launches

asperry

I am using DepthAI ROS to interact with an OAK-D PRO POE. The ROS nodes are working just fine, but I am running into an issue where it fails to connect to the camera if I try to launch the node again shortly after I kill a previously running instance.

Steps to reproduce:

ros2 launch depthai_ros_driver rgbd_pcl.launch.py
Wait for everything to start up normally
Use CTRL+C to kill the running ROS node (which is standard for ROS)
Repeat step 1 shortly after the terminal is available from the CTRL+C

In this scenario, I get a Failed to connect to device, error message: X_LINK_DEVICE_NOT_FOUND message on step 4.

I have tried specifying the camera's static IP using the camera.i_ip ROS parameter and the only thing that changes is that the error message changes to No devices detected!

I have done the above steps while having a ping command running in another terminal and the pinging runs just fine at all times.

The error does not happen if I wait some amount of time (a minute or so) before step 4 so this appears to only happen on quick successive ros2 launch calls. I've also noticed that using ros2 node list shows the /oak nodes for some time after the CTRL+C has finished. Waiting for these nodes to stop appearing in ros2 node list seems to indicate that I have waited long enough for the next ros2 launch call to succeed.

I've read this related post where the author described a very similar problem, but adding DEPTHAI_WATCHDOG_INITIAL_DELAY=60000 DEPTHAI_BOOTUP_TIMEOUT=60000 does not solve the problem for me.

jakaskerl

asperry
Update the bootloader to the latest version: https://docs.luxonis.com/projects/api/en/latest/components/bootloader/#bootloader

Other than that, it's expected that the device (POE) takes some time to reboot. That unfortunately can not be solved.

cc @Luxonis-Adam If there is some specific issue related to ROS that would cause that.

asperry

As some additional information, I've noticed that most of the time just using CTRL+C results in the following error

^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT)
[component_container-1] [INFO] [1715795814.547732352] [rclcpp]: signal_handler(signum=2)
[ERROR] [component_container-1]: process has died [pid 31153, exit code -11, cmd '/opt/ros/humble/lib/rclcpp_components/component_container --ros-args --log-level info --ros-args -r __node:=oak_container -r __ns:=/'].

If I call the /oak/stop_camera ROS service before using CRTL+C, then the process finishes cleanly as follows

^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT)
[component_container-1] [INFO] [1715796342.479880771] [rclcpp]: signal_handler(signum=2)
[INFO] [component_container-1]: process has finished cleanly [pid 34437]

and I am able to restart the node immediately afterward without any issue connecting to the camera.

However, sometimes the /oak/stop_camera service hangs up (seemingly forever) and when this happens it is as if the ROS node has become frozen because the SIGINT form the CTRL+C does not kill the node and the signal is elevated all the way to a SIGKILL

^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT)
[component_container-1] [INFO] [1715795919.817004394] [rclcpp]: signal_handler(signum=2)
[ERROR] [component_container-1]: process[component_container-1] failed to terminate '5' seconds after receiving 'SIGINT', escalating to 'SIGTERM'
[INFO] [component_container-1]: sending signal 'SIGTERM' to process[component_container-1]
[component_container-1] [INFO] [1715795924.949782261] [rclcpp]: signal_handler(signum=15)
[ERROR] [component_container-1]: process[component_container-1] failed to terminate '10.0' seconds after receiving 'SIGTERM', escalating to 'SIGKILL'
[INFO] [component_container-1]: sending signal 'SIGKILL' to process[component_container-1]
[ERROR] [component_container-1]: process has died [pid 32385, exit code -9, cmd '/opt/ros/humble/lib/rclcpp_components/component_container --ros-args --log-level info --ros-args -r __node:=oak_container -r __ns:=/'].

In this scenario, restarting the node results in the failure to connect to the camera I described above.

asperry

jakaskerl it's expected that the device (POE) takes some time to reboot. That unfortunately can not be solved.

The problem isn't that reboot takes time, it's that reboot is being triggered at all when it should not be. The fact that I can restart the ROS node immediately after stopping it with CTRL+C and have it connect to the camera just fine if I was able to stop the camera with /oak/stop_camera beforehand is clear evidence that I should be able to stop the ROS node without causing a camera reboot. To me, there is clearly a bug in depthai_ros or depthai that is causing a camera reboot when it should not be. The node should not be crashing when I use CTRL+C to stop it. That is fundamental expected behavior of ROS nodes.

Luxonis-Adam

Hey, as mentioned above this is firmware related, ROS-related parts shouldn't cause this specific issue.

asperry

Update: I've updated to the latest bootloader version 0.0.28 and the problem was not resolved.

@Luxonis-Adam I would argue that the hanging of the /oak/stop_camera service is certainly a problem of the ROS-related parts. Arguably, if something goes wrong in the firmware when attempting to stop the camera as a result of a call to the /oak/stop_camera service, at the very least, the ROS node acting as the service server should identify this and return a result that stopping the camera did not succeed instead of hanging indefinitely.

asperry

Can I get a response to my update here? Updating the bootloader did not solve the problem. Are you able to reproduce the problem yourselves?

mcowan

@jakaskerl I have been able to reproduce the issue, and I have a solution.

Problem: When a ros node is launched from a launch file, if it is terminated with cntl-c, the node will tear down and it uses the Camera's default destructor. The root problem is that the camera, if connected, has an open XLink connection over TCPIP. The default destructor does not call stop, which leaves the ports open and at the whims of the operating system to shutdown correctly.

Solution:
Modify the destructor here: luxonis/depthai-rosblob/humble/depthai_ros_driver/src/camera.cpp#L19

Camera::~Camera() {
RCLCPP_INFO(this->get_logger(), "Camera Destuctor!");
stop();
RCLCPP_INFO(this->get_logger(), "Camera Stopped!");
}

obviously the debug information is optional, but this change allowed me to re-launch the nodes immediately after a cntl-c and reconnect in under 5 seconds; vs 75 seconds.

jakaskerl

mcowan
Much thanks!

cc @Luxonis-Adam thoughts?

Luxonis-Adam

Hi, thanks for the investigation @mcowan , this change will be added in upcoming releases.

mcowan

Luxonis-Adam Is there an ETA for the next release?

Luxonis-Adam

Hi, it should arrive next week