Crash when starting multiple OAK-D-PRO-POE cameras with ROS

AlessandroPalleschi

Hello,

We are currently experiencing some problems when starting multiple OAK-D-PRO-POE cameras using the jazzy ros2 version of the depthai-core library (Version: 2.31.0-1noble.20260121.183544)

More specifically, we are currently running 6 POE cameras connected to the same UniFI Pro Max 16 switch. We are doing a stress test on the system using a staggering launch approach, starting a camera every 5 seconds.

For the cameras we are implementing a custom trigger-based capture mechanism patching the RGB pipeline to allow publishing streams on request. I have attached the modified camera.cpp, rgb.cpp and img_pub.cpp we are currently using.

patched-code.zip

11kB

As we run cameras in this custom mode, we are noticing frequent startup errors like these:

PRIMARY_FAILURE: [component_container-2] [31m[ERROR][1772005256.235094480] [left_robot.rpc_camera]: Camera startup attempt 1/3 failed: Device already closed or disconnected: Input/output error[0m

This then triggers the call to the stop function where we get:

[component_container-2] [0m[INFO] [1772005256.235175890] [left_robot.rpc_camera]: Stopping camera.[0m

[component_container-2] [0m[INFO] [1772005256.235194341] [left_robot.rpc_camera]: Closing Queues[0m

[component_container-2] [0m[INFO] [1772005256.235336831] [left_robot.rpc_camera]: Closed Queues[0m

[component_container-2] [0m[INFO] [1772005256.235350301] [left_robot.rpc_camera]: Clearing Nodes[0m

[component_container-2] [0m[INFO] [1772005256.235416271] [left_robot.rpc_camera]: Cleared Nodes[0m

[component_container-2] [0m[INFO] [1772005256.235427521] [left_robot.rpc_camera]: Resetting device[0m

[component_container-2] [14442C1051C5F2D600] [198.18.0.6] [1772005257.959] [host] [warning] Monitor thread (device: 14442C1051C5F2D600 [198.18.0.6]) - ping was missed, closing the device connection

[ERROR][component_container-2]: process has died [pid 2505189, exit code -11, cmd '/opt/ros/jazzy/lib/rclcpp_components/component_container --ros-args --log-level info --ros-args -r __node:=rpc_camera_container -r __ns:=/left_robot'].

Any ideas on why we get the initial error in the first place and what is causing the crash during the stop routine?

Thanks a lot for the help!

Best

Alessandro @ WittyMachines

OskarSonc

Hey AlessandroPalleschi - the initial Device already closed or disconnected error is most likely the DepthAI watchdog path on PoE under startup load, which matches the ping was missed, closing the device connection message. The -11 crash during stop looks like a teardown race in the custom trigger flow, where callbacks/services can still run while queues/device are being reset. I’d try serializing start/stop/restart with a mutex and making shutdown strictly ordered (stop service, remove callbacks, close/reset queues, then reset device/pipeline), and as a stress-test check increase DEPTHAI_WATCHDOG and DEPTHAI_WATCHDOG_INITIAL_DELAY to confirm watchdog sensitivity. As a quick test you could also increase the stagger from 5 seconds to 10 seconds to see whether startup stability improves.

Thanks,
Oskar

AlessandroPalleschi

Thanks for the prompt response! We are going to try this solution and let you know the results.

In the meantime, here also a crashdump from one of the cameras when the crash happens:

crash-dump.zip

7kB

OskarSonc

Hm, I checked it and it points to an on-device crash: RTEMS_FATAL_SOURCE_EXCEPTION with illegal instruction in thread PlgCPython (DepthAI commit f6083e8ec0afa948497100b39ec967a901ad57c4, device 14442C10F1D1F2D600).
Tht indicates the failure is happening in the Script runtime on the camera side, which fits your custom trigger pipeline and also explains why the host later sees missed ping/disconnect and then can crash during stop teardown.