Hi BradSteiner
Means there was an issue retrieving a frame from the RGB queue - mostly caused by queue filling up, crashing the device pipeline.
Thanks,
Jaka
Hi BradSteiner
Means there was an issue retrieving a frame from the RGB queue - mostly caused by queue filling up, crashing the device pipeline.
Thanks,
Jaka
What could cause that? Sometimes it seems to run fine, but I was seeing it a lot yesterday. It would only run for a few minutes before crashing due to that error. Is it the USB cable? Are the PoE version more reliable?
BradSteiner
Could be USB cable (lower bandwidth causing a block), POE (even lower bandwidth + ETH stack on CPU).. What does your pipeline look like? Is your host side loop fast enough to read the queues? If you set all queues to non-blocking, does the issue disappear.
Thanks,
Jaka
Hi, we are still seeing issue even with the PoE version. It looks like this:
[component_container-1] [ERROR] [1742415245.811915967] [oak]: No data on logger queue!
[component_container-1] [ERROR] [1742415245.812111873] [oak]: Camera diagnostics error: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'sys_logger_queue' (X_LINK_ERROR)'
We are launching with this command:
ros2 launch depthai_ros_driver camera.launch.py params_file:=camera_config.yaml
where camera_config.yaml looks like this:
/oak:
ros__parameters:
camera:
i_pipeline_type: rgbd
i_nn_type: none
i_floodlight_brightness: 1000 # 1500 is max
i_laser_dot_brightness: 1200 # 1200 is max
rgb:
i_fps: 10.0
i_resolution: '4K' #'720P' #'4K'
i_set_isp_scale: false
stereo:
i_width: 1280 #512 1280
i_height: 720 #288 720
# i_resolution: '720P'
i_extended_disp: true
i_subpixel: false
i_depth_preset: HIGH_DENSITY
i_enable_decimation_filter: true
i_enable_temporal_filter: false
i_enable_speckle_filter: true
i_enable_spatial_filter: true
i_lr_check: true
left:
i_publish_topic: true
Hi, could you check camera CPU usage? You can get that either via subscribing to diagnostics topic or enabling DEBUG logs via running export DEPTHAI_DEBUG=1
. You can also disable diagnostics queue just in case by setting
pipeline_gen:
i_enable_diagnostics: false
Sorry for the late reply. Here is a snapshot of the camera usage.
status:
- level: "\0"
name: 'oak: sys_logger'
message: System Information
hardware_id: /oak_18443010616C980F00_OAK-D-PRO-POE
values:
- key: Leon CSS CPU Usage
value: '99.859'
- key: Leon MSS CPU Usage
value: '9.071'
- key: Ddr Memory Usage
value: '229.081'
- key: Ddr Memory Total
value: '333.263'
- key: Cmx Memory Usage
value: '2.10254'
- key: Cmx Memory Total
value: '2621440'
- key: Leon CSS Memory Usage
value: '68.5048'
- key: Leon CSS Memory Total
value: '81.6971'
- key: Leon MSS Memory Usage
value: '4.67026'
- key: Leon MSS Memory Total
value: '39.8734'
- key: Average Chip Temperature
value: '54.6903'
- key: Leon CSS Chip Temperature
value: '56.7999'
- key: Leon MSS Chip Temperature
value: '53.3338'
- key: UPA Chip Temperature
value: '54.8573'
- key: DSS Chip Temperature
value: '53.7703'
---
Values are pretty consistent across messages
jakaskerl This is using the ROS Humble driver. Any idea on how this gets dealt with there?
Hi, it seems that CPU usage is too high (99%), this can cause connection issues as mentioned in the documentation. To improve it, you would need to adjust your pipeline accordingly (lower resolutions, reduce FPS, limit 3A algorithms cc @jakaskerl what else could might help here)
More information:
https://docs.luxonis.com/hardware/platform/deploy/poe-deployment-guide#PoE%20deployment%20guide-Runtime-Debugging
https://docs.luxonis.com/software/depthai/debugging#cpu-usage
Our framerate is already fairly low at 10 FPS. Is it expected to see such high CPU usage at this framerate?
Luxonis-Adam Even with the following configuration
/oak:
ros__parameters:
camera:
i_pipeline_type: rgbd
i_nn_type: none
i_floodlight_brightness: 800 # 800 is max
i_laser_dot_brightness: 1200 # 1200 is max
rgb:
i_low_bandwidth: true
left:
i_low_bandwidth: true
right:
i_low_bandwidth: true
stereo:
i_low_bandwidth: true
We do see an increased framerate (~20 fps) at the expense of poor depth images.
However, the CPU utilization seems to be still quite high. Here is the /diagnostics
output:
status:
- level: "\0"
name: 'oak: sys_logger'
message: System Information
hardware_id: /oak_18443010516DE7F400_OAK-D-PRO-POE
values:
- key: Leon CSS CPU Usage
value: '98.693'
- key: Leon MSS CPU Usage
value: '17.734'
- key: Ddr Memory Usage
value: '183.424'
- key: Ddr Memory Total
value: '333.263'
- key: Cmx Memory Usage
value: '2.37988'
- key: Cmx Memory Total
value: '2621440'
- key: Leon CSS Memory Usage
value: '67.9544'
- key: Leon CSS Memory Total
value: '81.6971'
- key: Leon MSS Memory Usage
value: '5.65635'
- key: Leon MSS Memory Total
value: '39.8734'
- key: Average Chip Temperature
value: '50.031'
- key: Leon CSS Chip Temperature
value: '50.9173'
- key: Leon MSS Chip Temperature
value: '49.3651'
- key: UPA Chip Temperature
value: '50.0317'
- key: DSS Chip Temperature
value: '49.8097'
---
Is this expected behavior using the ROS node?
Hi, sorry for the delay, I think with the default resolutions and fps it is to be expected since the camera has to run the network stack as well, you can try decreasing those parameters and/or ISP3A algorithms to see if it improves things much for you, you could maybe also try some other encoder options. cc @jakaskerl