Vast Performance difference between amd64 and arm64 docker containers

DionysiosLefkaditis · Apr 5, 2023

Hello all!

I am not a ROS expert so let me know if you need more information than what I provide...

I have been having some issues with running the arm64 ROS2 containers on a Jetson Xavier NX (same on a Jetson Nano). Running ros2 launch depthai_examples stereo_inertial_node.launch.py launches rviz with no visible camera images, no pointcloud visualisation and no mapping. Rviz reports that image_raw and pointcloud messages are recieved. It also reports only 2fps. The cpu useage is ~50%, which I interprete as computational resources are not the issue here. Please see screenshot below:

If I run the same thing on my laptop, it appears fine with 31fps:

This is on the latest humble release (2.7.0). In the previous releases things were even worse as rviz was not receiving pointcloud messages even if they were present (but low frequency) using ros2 topic hz /oak/points.

Looking forward to your responses...

Lluxonis-Sachin · Apr 5, 2023

Can you share the logs with export DEPTHAI_LEVEL=debug set

DionysiosLefkaditis · Apr 6, 2023

Sure! Thank you for your quick response! I am not sure how I can upload a text file here (it only allows for images), so I am pasting the logs below:

#                                                                                                                                                                                                                    
 
]2;root@638b95b3ceb0:/]1;/
➜  / e eepe  xxpo x  eexport     e  export DEPTHAI_LEVEL=debugexport DEPTHAI_LEVEL=debug

]2;export DEPTHAI_LEVEL=debug ]1;export#                                                                                                                                                                                                                    
 
]2;root@638b95b3ceb0:/]1;/
➜  / export DEPTHAI_LEVEL=debug                          DDISPLAY=:0 ros2 depthai_e depthai_el depthai_ea depthai_eu depthai_en depthai_ec depthai_eh depthai_examples stereo_inertial_node.launch.py 

]2;DISPLAY=:0 ros2 launch depthai_examples stereo_inertial_node.launch.py]1;ros2[INFO] [launch]: All log files can be found below /root/.ros/log/2023-04-06-09-57-44-543336-638b95b3ceb0-160
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [robot_state_publisher-1]: process started with pid [174]
[INFO] [stereo_inertial_node-2]: process started with pid [176]
[INFO] [component_container-3]: process started with pid [178]
[INFO] [rviz2-4]: process started with pid [180]
[robot_state_publisher-1] [INFO] [1680775065.710750367] [oak_state_publisher]: got segment oak-d-base-frame
[robot_state_publisher-1] [INFO] [1680775065.711103296] [oak_state_publisher]: got segment oak-d_frame
[robot_state_publisher-1] [INFO] [1680775065.711165728] [oak_state_publisher]: got segment oak_imu_frame
[robot_state_publisher-1] [INFO] [1680775065.711205728] [oak_state_publisher]: got segment oak_left_camera_frame
[robot_state_publisher-1] [INFO] [1680775065.711244864] [oak_state_publisher]: got segment oak_left_camera_optical_frame
[robot_state_publisher-1] [INFO] [1680775065.711284865] [oak_state_publisher]: got segment oak_model_origin
[robot_state_publisher-1] [INFO] [1680775065.711323041] [oak_state_publisher]: got segment oak_rgb_camera_frame
[robot_state_publisher-1] [INFO] [1680775065.711357889] [oak_state_publisher]: got segment oak_rgb_camera_optical_frame
[robot_state_publisher-1] [INFO] [1680775065.711395265] [oak_state_publisher]: got segment oak_right_camera_frame
[robot_state_publisher-1] [INFO] [1680775065.711434465] [oak_state_publisher]: got segment oak_right_camera_optical_frame
[rviz2-4] QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
[component_container-3] [INFO] [1680775065.864250289] [container]: Load Library: /opt/ros/humble/lib/libdepth_image_proc.so
[stereo_inertial_node-2] [2023-04-06 09:57:45.974] [debug] Library information - version: 2.20.2, commit:  from , build: 2023-02-02 20:42:56 +0000
[stereo_inertial_node-2] [2023-04-06 09:57:45.979] [debug] Initialize - finished
[stereo_inertial_node-2] 1280 720 1280 720
[component_container-3] [INFO] [1680775066.125360208] [container]: Found class: rclcpp_components::NodeFactoryTemplate<depth_image_proc::ConvertMetricNode>
[component_container-3] [INFO] [1680775066.125512304] [container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<depth_image_proc::ConvertMetricNode>
[stereo_inertial_node-2] [2023-04-06 09:57:46.172] [debug] Resources - Archive 'depthai-bootloader-fwp-0.0.24.tar.xz' open: 13ms, archive read: 179ms
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/convert_metric_node' in container '/container'
[component_container-3] [INFO] [1680775066.202436506] [container]: Found class: rclcpp_components::NodeFactoryTemplate<depth_image_proc::ConvertMetricNode>
[component_container-3] [INFO] [1680775066.202576826] [container]: Found class: rclcpp_components::NodeFactoryTemplate<depth_image_proc::CropForemostNode>
[component_container-3] [INFO] [1680775066.202687866] [container]: Found class: rclcpp_components::NodeFactoryTemplate<depth_image_proc::DisparityNode>
[component_container-3] [INFO] [1680775066.202736122] [container]: Found class: rclcpp_components::NodeFactoryTemplate<depth_image_proc::PointCloudXyzNode>
[component_container-3] [INFO] [1680775066.202771130] [container]: Found class: rclcpp_components::NodeFactoryTemplate<depth_image_proc::PointCloudXyzRadialNode>
[component_container-3] [INFO] [1680775066.202810363] [container]: Found class: rclcpp_components::NodeFactoryTemplate<depth_image_proc::PointCloudXyziNode>
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/point_cloud_xyzrgb_node' in container '/container'
[component_container-3] [INFO] [1680775066.202847067] [container]: Found class: rclcpp_components::NodeFactoryTemplate<depth_image_proc::PointCloudXyziRadialNode>
[component_container-3] [INFO] [1680775066.202881851] [container]: Found class: rclcpp_components::NodeFactoryTemplate<depth_image_proc::PointCloudXyzrgbNode>
[component_container-3] [INFO] [1680775066.202917307] [container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<depth_image_proc::PointCloudXyzrgbNode>
[rviz2-4] [INFO] [1680775066.268835020] [rviz2]: Stereo is NOT SUPPORTED
[rviz2-4] [INFO] [1680775066.269095661] [rviz2]: OpenGl version: 4.5 (GLSL 4.5)
[stereo_inertial_node-2] Listing available devices...
[stereo_inertial_node-2] Device Mx ID: 1944301021C01C1300
[rviz2-4] [INFO] [1680775066.385098317] [rviz2]: Stereo is NOT SUPPORTED
[stereo_inertial_node-2] [2023-04-06 09:57:46.530] [debug] Device - OpenVINO version: 2022.1
[stereo_inertial_node-2] [2023-04-06 09:57:46.531] [debug] Device - BoardConfig: {"camera":[],"emmc":null,"gpio":[],"logDevicePrints":null,"logPath":null,"logSizeMax":null,"logVerbosity":null,"network":{"mtu":0,"xlinkTcpNoDelay":true},"nonExclusiveMode":false,"pcieInternalClock":null,"sysctl":[],"uart":[],"usb":{"flashBootedPid":63037,"flashBootedVid":999,"maxSpeed":4,"pid":63035,"vid":999},"usb3PhyInternalClock":null,"watchdogInitialDelayMs":null,"watchdogTimeoutMs":null} 
[stereo_inertial_node-2] libnop:
[stereo_inertial_node-2] 0000: b9 10 b9 05 81 e7 03 81 3b f6 81 e7 03 81 3d f6 04 b9 02 00 01 ba 00 be be bb 00 bb 00 be be be
[stereo_inertial_node-2] 0020: be be be be 00 bb 00
[component_container-3] [WARN] [1680775066.655656803] [point_cloud_xyzrgb_node]: New subscription discovered on topic '/stereo/points', requesting incompatible QoS. No messages will be sent to it. Last incompatible policy: RELIABILITY_QOS_POLICY
[component_container-3] [WARN] [1680775066.657740967] [point_cloud_xyzrgb_node]: New subscription discovered on topic '/stereo/points', requesting incompatible QoS. No messages will be sent to it. Last incompatible policy: RELIABILITY_QOS_POLICY
[stereo_inertial_node-2] [2023-04-06 09:57:46.930] [debug] Resources - Archive 'depthai-device-fwp-8c3d6ac1c77b0bf7f9ea6fd4d962af37663d2fbd.tar.xz' open: 8ms, archive read: 938ms
[stereo_inertial_node-2] [2023-04-06 09:57:47.696] [debug] Searching for booted device: DeviceInfo(name=1.2.1, mxid=1944301021C01C1300, X_LINK_BOOTED, X_LINK_USB_VSC, X_LINK_MYRIAD_X, X_LINK_SUCCESS), name used as hint only
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [0.834] [system] [info] Memory Usage - DDR: 0.12 / 340.42 MiB, CMX: 2.05 / 2.50 MiB, LeonOS Heap: 7.25 / 77.32 MiB, LeonRT Heap: 2.89 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [0.835] [system] [info] Temperatures - Average: 35.83 °C, CSS: 35.83 °C, MSS 36.06 °C, UPA: 35.83 °C, DSS: 35.59 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [0.835] [system] [info] Cpu Usage - LeonOS 70.08%, LeonRT: 1.81%
[stereo_inertial_node-2] [2023-04-06 09:57:48.561] [debug] Schema dump: {"connections":[{"node1Id":5,"node1Output":"out","node1OutputGroup":"","node2Id":6,"node2Input":"in","node2InputGroup":""},{"node1Id":3,"node1Output":"depth","node1OutputGroup":"","node2Id":4,"node2Input":"in","node2InputGroup":""},{"node1Id":2,"node1Output":"out","node1OutputGroup":"","node2Id":3,"node2Input":"right","node2InputGroup":""},{"node1Id":1,"node1Output":"out","node1OutputGroup":"","node2Id":3,"node2Input":"left","node2InputGroup":""},{"node1Id":9,"node1Output":"out","node1OutputGroup":"","node2Id":10,"node2Input":"in","node2InputGroup":""},{"node1Id":9,"node1Output":"passthrough","node1OutputGroup":"","node2Id":11,"node2Input":"in","node2InputGroup":""},{"node1Id":3,"node1Output":"depth","node1OutputGroup":"","node2Id":9,"node2Input":"inputDepth","node2InputGroup":""},{"node1Id":7,"node1Output":"preview","node1OutputGroup":"","node2Id":9,"node2Input":"in","node2InputGroup":""},{"node1Id":7,"node1Output":"isp","node1OutputGroup":"","node2Id":8,"node2Input":"in","node2InputGroup":""},{"node1Id":0,"node1Output":"out","node1OutputGroup":"","node2Id":1,"node2Input":"inputControl","node2InputGroup":""},{"node1Id":0,"node1Output":"out","node1OutputGroup":"","node2Id":2,"node2Input":"inputControl","node2InputGroup":""}],"globalProperties":{"calibData":null,"cameraTuningBlobSize":null,"cameraTuningBlobUri":"","leonCssFrequencyHz":700000000.0,"leonMssFrequencyHz":700000000.0,"pipelineName":null,"pipelineVersion":null,"xlinkChunkSize":-1},"nodes":[[0,{"id":0,"ioInfo":[[["","out"],{"blocking":false,"group":"","id":45,"name":"out","queueSize":8,"type":0,"waitForMessage":false}]],"name":"XLinkIn","properties":[185,3,189,7,99,111,110,116,114,111,108,130,0,0,80,0,8]}],[1,{"id":1,"ioInfo":[[["","out"],{"blocking":false,"group":"","id":44,"name":"out","queueSize":8,"type":0,"waitForMessage":false}],[["","raw"],{"blocking":false,"group":"","id":43,"name":"raw","queueSize":8,"type":0,"waitForMessage":false}],[["","frameEvent"],{"blocking":false,"group":"","id":42,"name":"frameEvent","queueSize":8,"type":0,"waitForMessage":false}],[["","inputControl"],{"blocking":true,"group":"","id":41,"name":"inputControl","queueSize":8,"type":3,"waitForMessage":false}]],"name":"MonoCamera","properties":[185,8,185,27,0,3,0,0,0,185,3,0,0,0,185,5,0,0,0,0,0,185,5,0,0,0,0,0,0,0,0,0,0,0,0,185,3,0,0,0,185,3,0,0,0,0,0,0,0,0,0,0,0,0,0,1,189,0,255,0,136,0,0,240,65,3,3]}],[2,{"id":2,"ioInfo":[[["","out"],{"blocking":false,"group":"","id":40,"name":"out","queueSize":8,"type":0,"waitForMessage":false}],[["","raw"],{"blocking":false,"group":"","id":39,"name":"raw","queueSize":8,"type":0,"waitForMessage":false}],[["","frameEvent"],{"blocking":false,"group":"","id":38,"name":"frameEvent","queueSize":8,"type":0,"waitForMessage":false}],[["","inputControl"],{"blocking":true,"group":"","id":37,"name":"inputControl","queueSize":8,"type":3,"waitForMessage":false}]],"name":"MonoCamera","properties":[185,8,185,27,0,3,0,0,0,185,3,0,0,0,185,5,0,0,0,0,0,185,5,0,0,0,0,0,0,0,0,0,0,0,0,185,3,0,0,0,185,3,0,0,0,0,0,0,0,0,0,0,0,0,0,2,189,0,255,0,136,0,0,240,65,3,3]}],[3,{"id":3,"ioInfo":[[["","depth"],{"blocking":false,"group":"","id":36,"name":"depth","queueSize":8,"type":0,"waitForMessage":false}],[["","rectifiedLeft"],{"blocking":false,"group":"","id":34,"name":"rectifiedLeft","queueSize":8,"type":0,"waitForMessage":false}],[["","syncedLeft"],{"blocking":false,"group":"","id":35,"name":"syncedLeft","queueSize":8,"type":0,"waitForMessage":false}],[["","syncedRight"],{"blocking":false,"group":"","id":31,"name":"syncedRight","queueSize":8,"type":0,"waitForMessage":false}],[["","rectifiedRight"],{"blocking":false,"group":"","id":30,"name":"rectifiedRight","queueSize":8,"type":0,"waitForMessage":false}],[["","outConfig"],{"blocking":false,"group":"","id":28,"name":"outConfig","queueSize":8,"type":0,"waitForMessage":false}],[["","debugExtDispLrCheckIt2"],{"blocking":false,"group":"","id":27,"name":"debugExtDispLrCheckIt2","queueSize":8,"type":0,"waitForMessage":false}],[["","debugDispCostDump"],{"blocking":false,"group":"","id":29,"name":"debugDispCostDump","queueSize":8,"type":0,"waitForMessage":false}],[["","confidenceMap"],{"blocking":false,"group":"","id":26,"name":"confidenceMap","queueSize":8,"type":0,"waitForMessage":false}],[["","debugDispLrCheckIt2"],{"blocking":false,"group":"","id":25,"name":"debugDispLrCheckIt2","queueSize":8,"type":0,"waitForMessage":false}],[["","debugDispLrCheckIt1"],{"blocking":false,"group":"","id":32,"name":"debugDispLrCheckIt1","queueSize":8,"type":0,"waitForMessage":false}],[["","debugExtDispLrCheckIt1"],{"blocking":false,"group":"","id":24,"name":"debugExtDispLrCheckIt1","queueSize":8,"type":0,"waitForMessage":false}],[["","disparity"],{"blocking":false,"group":"","id":33,"name":"disparity","queueSize":8,"type":0,"waitForMessage":false}],[["","inputConfig"],{"blocking":false,"group":"","id":23,"name":"inputConfig","queueSize":4,"type":3,"waitForMessage":false}],[["","left"],{"blocking":false,"group":"","id":22,"name":"left","queueSize":8,"type":3,"waitForMessage":true}],[["","right"],{"blocking":false,"group":"","id":21,"name":"right","queueSize":8,"type":3,"waitForMessage":true}]],"name":"StereoDepth","properties":[185,18,185,5,185,9,0,2,136,0,0,122,68,1,0,1,5,3,0,185,7,5,0,185,5,0,2,136,0,0,0,63,0,1,185,4,0,3,136,205,204,204,62,0,185,2,0,134,255,255,0,0,185,2,0,50,185,2,1,0,185,4,255,0,1,0,185,5,1,0,0,128,200,185,3,0,2,127,185,5,1,128,250,129,244,1,128,250,129,244,1,0,1,0,190,190,190,190,1,185,5,189,0,189,0,190,16,16,0,3,255,255,1,190,190,190]}],[4,{"id":4,"ioInfo":[[["","in"],{"blocking":true,"group":"","id":20,"name":"in","queueSize":8,"type":3,"waitForMessage":true}]],"name":"XLinkOut","properties":[185,3,136,0,0,128,191,189,5,100,101,112,116,104,0]}],[5,{"id":5,"ioInfo":[[["","out"],{"blocking":false,"group":"","id":19,"name":"out","queueSize":8,"type":0,"waitForMessage":false}]],"name":"IMU","properties":[185,4,186,2,185,5,0,0,0,129,244,1,20,185,5,0,0,0,129,144,1,21,5,20,0]}],[6,{"id":6,"ioInfo":[[["","in"],{"blocking":true,"group":"","id":18,"name":"in","queueSize":8,"type":3,"waitForMessage":true}]],"name":"XLinkOut","properties":[185,3,136,0,0,128,191,189,3,105,109,117,0]}],[7,{"id":7,"ioInfo":[[["","video"],{"blocking":false,"group":"","id":17,"name":"video","queueSize":8,"type":0,"waitForMessage":false}],[["","still"],{"blocking":false,"group":"","id":15,"name":"still","queueSize":8,"type":0,"waitForMessage":false}],[["","isp"],{"blocking":false,"group":"","id":14,"name":"isp","queueSize":8,"type":0,"waitForMessage":false}],[["","preview"],{"blocking":false,"group":"","id":16,"name":"preview","queueSize":8,"type":0,"waitForMessage":false}],[["","raw"],{"blocking":false,"group":"","id":13,"name":"raw","queueSize":8,"type":0,"waitForMessage":false}],[["","frameEvent"],{"blocking":false,"group":"","id":12,"name":"frameEvent","queueSize":8,"type":0,"waitForMessage":false}],[["","inputConfig"],{"blocking":false,"group":"","id":11,"name":"inputConfig","queueSize":8,"type":3,"waitForMessage":false}],[["","inputControl"],{"blocking":true,"group":"","id":10,"name":"inputControl","queueSize":8,"type":3,"waitForMessage":false}]],"name":"ColorCamera","properties":[185,24,185,27,0,3,0,0,0,185,3,0,0,0,185,5,0,0,0,0,0,185,5,0,0,0,0,0,0,0,0,0,0,0,0,185,3,0,0,0,185,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,189,0,255,0,0,0,129,160,1,129,160,1,255,255,255,255,0,136,0,0,240,65,136,0,0,128,191,136,0,0,128,191,1,185,4,2,3,2,3,3,3,4,4,4]}],[8,{"id":8,"ioInfo":[[["","in"],{"blocking":true,"group":"","id":9,"name":"in","queueSize":8,"type":3,"waitForMessage":true}]],"name":"XLinkOut","properties":[185,3,136,0,0,128,191,189,3,114,103,98,0]}],[9,{"id":9,"ioInfo":[[["","out"],{"blocking":false,"group":"","id":8,"name":"out","queueSize":8,"type":0,"waitForMessage":false}],[["","passthroughDepth"],{"blocking":false,"group":"","id":7,"name":"passthroughDepth","queueSize":8,"type":0,"waitForMessage":false}],[["","passthrough"],{"blocking":false,"group":"","id":6,"name":"passthrough","queueSize":8,"type":0,"waitForMessage":false}],[["","boundingBoxMapping"],{"blocking":false,"group":"","id":5,"name":"boundingBoxMapping","queueSize":8,"type":0,"waitForMessage":false}],[["","in"],{"blocking":false,"group":"","id":4,"name":"in","queueSize":5,"type":3,"waitForMessage":true}],[["","inputDepth"],{"blocking":false,"group":"","id":3,"name":"inputDepth","queueSize":4,"type":3,"waitForMessage":true}]],"name":"SpatialDetectionNetwork","properties":[185,9,130,128,188,185,0,189,12,97,115,115,101,116,58,95,95,98,108,111,98,8,0,0,185,7,0,136,0,0,0,63,80,4,186,12,136,0,0,32,65,136,0,0,96,65,136,0,0,184,65,136,0,0,216,65,136,0,0,20,66,136,0,0,104,66,136,0,0,162,66,136,0,0,164,66,136,0,0,7,67,136,0,0,41,67,136,0,0,172,67,136,0,128,159,67,187,2,189,6,115,105,100,101,49,51,188,12,3,0,0,0,4,0,0,0,5,0,0,0,189,6,115,105,100,101,50,54,188,12,1,0,0,0,2,0,0,0,3,0,0,0,136,0,0,0,63,136,0,0,0,63,185,2,100,129,16,39,0]}],[10,{"id":10,"ioInfo":[[["","in"],{"blocking":true,"group":"","id":2,"name":"in","queueSize":8,"type":3,"waitForMessage":true}]],"name":"XLinkOut","properties":[185,3,136,0,0,128,191,189,10,100,101,116,101,99,116,105,111,110,115,0]}],[11,{"id":11,"ioInfo":[[["","in"],{"blocking":true,"group":"","id":1,"name":"in","queueSize":8,"type":3,"waitForMessage":true}]],"name":"XLinkOut","properties":[185,3,136,0,0,128,191,189,7,112,114,101,118,105,101,119,0]}]]}
[stereo_inertial_node-2] [2023-04-06 09:57:48.561] [debug] Asset map dump: {"map":{"/node/9/__blob":{"alignment":64,"offset":0,"size":12172416}}}
[rviz2-4] [INFO] [1680775068.592826737] [rviz2]: Stereo is NOT SUPPORTED
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [1.020] [MonoCamera(2)] [info] Using board socket: 2, id: 2
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [1.020] [MonoCamera(1)] [info] Using board socket: 1, id: 1
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [1.024] [system] [info] SpatialLocationCalculator shave buffer size '29696'B
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [1.024] [system] [info] SIPP (Signal Image Processing Pipeline) internal buffer size '16384'B
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [1.059] [system] [info] ImageManip internal buffer size '258560'B, shave buffer size '34816'B
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [1.059] [system] [info] NeuralNetwork allocated resources: shaves: [0-9] cmx slices: [0-9] 
[stereo_inertial_node-2] ColorCamera allocated resources: no shaves; cmx slices: [13-15] 
[stereo_inertial_node-2] MonoCamera allocated resources: no shaves; cmx slices: [13-15] 
[stereo_inertial_node-2] StereoDepth allocated resources: shaves: [12-12] cmx slices: [10-12] 
[stereo_inertial_node-2] ImageManip allocated resources: shaves: [15-15] no cmx slices. 
[stereo_inertial_node-2] SpatialLocationCalculator allocated resources: shaves: [14-14] no cmx slices. 
[stereo_inertial_node-2] 
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [1.587] [StereoDepth(3)] [info] Using focal length from calibration intrinsics '791.1147'
[stereo_inertial_node-2] Device USB status: [1944301021C01C1300] [1.2.1] [1.602] [SpatialDetectionNetwork(9)] [info] Needed resources: shaves: 6, ddr: 5191680 
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [1.602] [SpatialDetectionNetwork(9)] [warning] Network compiled for 6 shaves, maximum available 10, compiling for 5 shaves likely will yield in better performance
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [1.616] [SpatialDetectionNetwork(9)] [info] Inference thread count: 1, number of shaves allocated per thread: 6, number of Neural Compute Engines (NCE) allocated per thread: 2
[stereo_inertial_node-2] SUPER
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [1.837] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.69 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [1.837] [system] [info] Temperatures - Average: 37.59 °C, CSS: 38.18 °C, MSS 37.24 °C, UPA: 38.18 °C, DSS: 36.77 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [1.837] [system] [info] Cpu Usage - LeonOS 27.07%, LeonRT: 15.79%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [2.839] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [2.839] [system] [info] Temperatures - Average: 41.26 °C, CSS: 41.43 °C, MSS 41.89 °C, UPA: 41.66 °C, DSS: 40.04 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [2.840] [system] [info] Cpu Usage - LeonOS 69.49%, LeonRT: 32.18%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [3.841] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [3.841] [system] [info] Temperatures - Average: 42.12 °C, CSS: 42.35 °C, MSS 43.04 °C, UPA: 42.58 °C, DSS: 40.51 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [3.841] [system] [info] Cpu Usage - LeonOS 69.39%, LeonRT: 33.50%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [4.843] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [4.843] [system] [info] Temperatures - Average: 43.10 °C, CSS: 43.73 °C, MSS 43.50 °C, UPA: 44.18 °C, DSS: 40.97 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [4.843] [system] [info] Cpu Usage - LeonOS 68.80%, LeonRT: 34.12%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [5.845] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [5.845] [system] [info] Temperatures - Average: 43.33 °C, CSS: 43.50 °C, MSS 42.81 °C, UPA: 45.09 °C, DSS: 41.89 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [5.845] [system] [info] Cpu Usage - LeonOS 69.07%, LeonRT: 33.71%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [6.847] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [6.847] [system] [info] Temperatures - Average: 43.95 °C, CSS: 43.96 °C, MSS 44.18 °C, UPA: 45.32 °C, DSS: 42.35 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [6.847] [system] [info] Cpu Usage - LeonOS 68.36%, LeonRT: 34.71%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [7.849] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [7.849] [system] [info] Temperatures - Average: 44.01 °C, CSS: 43.96 °C, MSS 43.96 °C, UPA: 44.87 °C, DSS: 43.27 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [7.849] [system] [info] Cpu Usage - LeonOS 68.16%, LeonRT: 33.37%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [8.851] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [8.851] [system] [info] Temperatures - Average: 44.41 °C, CSS: 44.41 °C, MSS 44.87 °C, UPA: 45.09 °C, DSS: 43.27 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [8.851] [system] [info] Cpu Usage - LeonOS 68.66%, LeonRT: 32.50%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [9.852] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [9.852] [system] [info] Temperatures - Average: 44.87 °C, CSS: 45.32 °C, MSS 45.09 °C, UPA: 45.77 °C, DSS: 43.27 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [9.852] [system] [info] Cpu Usage - LeonOS 68.34%, LeonRT: 32.64%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [10.854] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.43 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [10.854] [system] [info] Temperatures - Average: 45.26 °C, CSS: 45.32 °C, MSS 45.32 °C, UPA: 46.23 °C, DSS: 44.18 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [10.854] [system] [info] Cpu Usage - LeonOS 68.93%, LeonRT: 32.40%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [11.856] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [11.856] [system] [info] Temperatures - Average: 45.21 °C, CSS: 45.09 °C, MSS 44.64 °C, UPA: 46.90 °C, DSS: 44.18 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [11.856] [system] [info] Cpu Usage - LeonOS 68.09%, LeonRT: 32.53%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [12.858] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [12.858] [system] [info] Temperatures - Average: 45.66 °C, CSS: 44.87 °C, MSS 46.68 °C, UPA: 46.68 °C, DSS: 44.41 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [12.858] [system] [info] Cpu Usage - LeonOS 69.08%, LeonRT: 32.75%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [13.860] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [13.860] [system] [info] Temperatures - Average: 45.26 °C, CSS: 45.55 °C, MSS 44.64 °C, UPA: 46.90 °C, DSS: 43.96 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [13.860] [system] [info] Cpu Usage - LeonOS 68.52%, LeonRT: 32.96%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [14.862] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [14.862] [system] [info] Temperatures - Average: 46.05 °C, CSS: 45.77 °C, MSS 46.68 °C, UPA: 47.35 °C, DSS: 44.41 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [14.862] [system] [info] Cpu Usage - LeonOS 67.49%, LeonRT: 32.58%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [15.864] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [15.864] [system] [info] Temperatures - Average: 45.94 °C, CSS: 46.00 °C, MSS 46.45 °C, UPA: 47.58 °C, DSS: 43.73 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [15.864] [system] [info] Cpu Usage - LeonOS 68.61%, LeonRT: 33.22%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [16.866] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [16.866] [system] [info] Temperatures - Average: 46.22 °C, CSS: 46.68 °C, MSS 46.23 °C, UPA: 47.35 °C, DSS: 44.64 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [16.866] [system] [info] Cpu Usage - LeonOS 68.07%, LeonRT: 33.16%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [17.868] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [17.868] [system] [info] Temperatures - Average: 46.34 °C, CSS: 46.90 °C, MSS 46.23 °C, UPA: 47.13 °C, DSS: 45.09 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [17.868] [system] [info] Cpu Usage - LeonOS 68.07%, LeonRT: 33.19%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [18.871] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [18.871] [system] [info] Temperatures - Average: 46.96 °C, CSS: 47.13 °C, MSS 46.90 °C, UPA: 48.03 °C, DSS: 45.77 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [18.871] [system] [info] Cpu Usage - LeonOS 68.71%, LeonRT: 32.70%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [19.873] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [19.873] [system] [info] Temperatures - Average: 46.39 °C, CSS: 46.90 °C, MSS 46.45 °C, UPA: 47.80 °C, DSS: 44.41 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [19.873] [system] [info] Cpu Usage - LeonOS 68.45%, LeonRT: 32.86%
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [20.875] [system] [info] Memory Usage - DDR: 115.19 / 340.42 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 48.94 / 77.32 MiB, LeonRT Heap: 5.42 / 41.23 MiB
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [20.875] [system] [info] Temperatures - Average: 46.68 °C, CSS: 46.45 °C, MSS 47.13 °C, UPA: 47.80 °C, DSS: 45.32 °C
[stereo_inertial_node-2] [1944301021C01C1300] [1.2.1] [20.875] [system] [info] Cpu Usage - LeonOS 68.11%, LeonRT: 32.20%
^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT)
[rviz2-4] [INFO] [1680775088.671095022] [rclcpp]: signal_handler(signum=2)
[INFO] [robot_state_publisher-1]: process has finished cleanly [pid 174]
[INFO] [component_container-3]: process has finished cleanly [pid 178]
[ERROR] [stereo_inertial_node-2]: process has died [pid 176, exit code -6, cmd '/ws/install/lib/depthai_examples/stereo_inertial_node --ros-args --params-file /tmp/launch_params_46zknjpa --params-file /tmp/launch_params_xag3dzrw --params-file /tmp/launch_params_cm54ypcy --params-file /tmp/launch_params_820fnpkj --params-file /tmp/launch_params_5vloq8f4 --params-file /tmp/launch_params_xh1t0udh --params-file /tmp/launch_params_9psk2ib1 --params-file /tmp/launch_params_joo6x05z --params-file /tmp/launch_params_ed9r5p4u --params-file /tmp/launch_params_y5e86b3q --params-file /tmp/launch_params_f6yxw2co --params-file /tmp/launch_params_c2jmtw07 --params-file /tmp/launch_params_a_pnf3l4 --params-file /tmp/launch_params_14g28ggf --params-file /tmp/launch_params_90_kn6v1 --params-file /tmp/launch_params_ki_cpheg --params-file /tmp/launch_params_ftbsi8lt --params-file /tmp/launch_params_ymu_ibqa --params-file /tmp/launch_params_1mqhtg33 --params-file /tmp/launch_params_xvwty3fv --params-file /tmp/launch_params_57k4o5hi --params-file /tmp/launch_params_2g9xj4bc --params-file /tmp/launch_params_v05wfqc8 --params-file /tmp/launch_params_e0amwhbi --params-file /tmp/launch_params_ji9si1w2 --params-file /tmp/launch_params_ff3h0lyf --params-file /tmp/launch_params_l6sb9cxs --params-file /tmp/launch_params_7d_wbnq4 --params-file /tmp/launch_params_71aq9zzw --params-file /tmp/launch_params_hlc84adi --params-file /tmp/launch_params_ur43gxra --params-file /tmp/launch_params_pcj599gl --params-file /tmp/launch_params_cp6zd9xm --params-file /tmp/launch_params_fdwpm32v'].
[component_container-3] [INFO] [1680775088.671086542] [rclcpp]: signal_handler(signum=2)
[robot_state_publisher-1] [INFO] [1680775088.673197652] [rclcpp]: signal_handler(signum=2)
[stereo_inertial_node-2] [INFO] [1680775088.673496244] [rclcpp]: signal_handler(signum=2)
[stereo_inertial_node-2] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[stereo_inertial_node-2]   what():  failed to create guard condition: the given context is not valid, either rcl_init() was not called or rcl_shutdown() was called., at ./src/rcl/guard_condition.c:67
[stereo_inertial_node-2] [2023-04-06 09:58:08.684] [error] Callback with id: 1 throwed an exception: could not count subscribers: rcl node's context is invalid, at ./src/rcl/node.c:428
[stereo_inertial_node-2] [2023-04-06 09:58:08.692] [error] Callback with id: 1 throwed an exception: Invalid topic name: topic name must not contain characters other than alphanumerics, '_', '~', '{', or '}':
[stereo_inertial_node-2]   '���U'
[stereo_inertial_node-2]    ^
[stereo_inertial_node-2] 
[stereo_inertial_node-2] [2023-04-06 09:58:08.693] [error] Callback with id: 1 throwed an exception: could not count subscribers: rcl node's context is invalid, at ./src/rcl/node.c:428
[stereo_inertial_node-2] [2023-04-06 09:58:08.693] [error] Callback with id: 1 throwed an exception: could not count subscribers: rcl node's context is invalid, at ./src/rcl/node.c:428
[stereo_inertial_node-2] [2023-04-06 09:58:08.694] [error] Callback with id: 1 throwed an exception: could not count subscribers: rcl node's context is invalid, at ./src/rcl/node.c:428
[INFO] [rviz2-4]: process has finished cleanly [pid 180]

DionysiosLefkaditis · Apr 12, 2023

Hello again!

This is just to let you know that I have tried humble release v2.7.1 and I am still getting the same behaviour… The problem is also aparent when running ros2 launch depthai_ros_driver rgbdpcl.launch.py , when I try to ros2 topic echo /oak/points or ros2 topic hz /oak/points from another terminal inside the container, I get no data back. The other topics (like /oak/rgb/image_raw, /oak/imu/data) behave as expected…

Please, let me know how I can help to resolve the issue faster…

Thanks in advance!

erik · Apr 12, 2023

@luxonis-Sachin / @Luxonis-Adam what do you guys think could be the issue here?

Bbrianlmerritt · Apr 12, 2023

It might well be computational at 50% if one or two cores are maxed out, and probably also worth checking memory when it is running.

Have you tried connecting the Xavier and launching just the camera side and having RVIZ launched on a PC/Host?

Luxonis-Adam · Apr 12, 2023

Hi, PCL generation is done on CPU, which might be troublesome for Jetson boards, you can investigate resource usage with htop to see if that's the bottleneck. If that's the case, maybe try lowering output image sizes for rgb and depth. Another bottleneck could be DDS related, you can find information on DDS tuning here.

DionysiosLefkaditis · Apr 12, 2023

Hmm, I was hoping that pointclouds are computed onboard the camera… Nevertheless, I think the cpu is not maxing out:

As I had mentioned before, the average cpu is around ~50% and it is balanced among the cores, also RAM looks pretty healthy.

I will look into lowering image and depth resolution and also DDS tuning as you suggested. I think the Jetson boards are an important platform to support properly as I would expect it to be the goto board for lightweight/edge robotics and computer vision…

I will report my findings asap… On your side, can you confirm that you get similar performance on an arm64 board?

Thanks!

erik · Apr 12, 2023

DionysiosLefkaditis not yet, but we have on-device pointcloud generation on our roadmap.

DionysiosLefkaditis · Apr 17, 2023

Luxonis-Adam

Hello again!

I have been trying to lower the image resolution of rgb and depth, but I am not able to achieve it. I have tried through rqt_reconfigure and through arguments while launching from terminal, but they don't seem to get picked up (no warnings either).
Could you please provide me with an example of how to do this?

Thanks in advance!

DionysiosLefkaditis · Apr 22, 2023

Hello all!

I would like to give you an update and close the subject:
Apparently, the whole thing works just fine if you Xforward Rviz to another host with (ip is obviously just an example):
DISPLAY=192.168.0.2:0 ros2 launch depthai_examples stereo_inertial_node.launch.py

So, there is something weird going on on the Jetson Xavier and it is not an issue of the depthai-ros container. Also, for the record, the Xavier produces pointclouds at 25Hz at 50% cpu, which in my view is plenty of horsepower for this kind of workload…

I would still be interested to see how I can limit depth resolution and throttle framerates to further optimise cpu load. Can you please give an example?

Thank you so much for the support…

erik · Apr 22, 2023

Hi DionysiosLefkaditis ,
To downscale the depth before converting it to pointcloud I'd suggest using decimation filter, documentation here: https://docs.luxonis.com/projects/api/en/latest/tutorials/configuring-stereo-depth/#decimation-filter-for-pointcloud

Thanks, Erik

EricC · Aug 27, 2023

@DionysiosLefkaditis

Thanks for sharing this. I was having the exact same issue on arm64 (Raspberry pi) running Ubuntu 22.04. I tried running Rviz2 on a remote machine, but had the same problem. After researching further, the topic publishing latency was terrible and I encountered these two articles on DDS tuning on Linux and followed the recommendations indicated.

https://docs.ros.org/en/humble/How-To-Guides/DDS-tuning.html#cross-vendor-tuning
sudo sysctl net.ipv4.ipfrag_time=3
sudo sysctl net.ipv4.ipfrag_high_thresh=134217728

https://fast-dds.docs.eprosima.com/en/1.5.0/advanced.html
sudo sysctl -w net.ipv4.udp_mem="102400 873800 16777216"
sudo sysctl -w net.core.netdev_max_backlog="30000"
sudo sysctl -w net.core.rmem_max="16777216"
sudo sysctl -w net.core.wmem_max="16777216"

After doing so and running Rviz2 remotely, the camera and point cloud topic data began flowing and I was able to visualize the data in Rviz2. However, there is still about a 3 second delay, but at least it's working.