Couldn't read data from stream: 'sys_logger_queue' (X_LINK_ERROR)'

jakaskerl

Hi BradSteiner
Means there was an issue retrieving a frame from the RGB queue - mostly caused by queue filling up, crashing the device pipeline.

Thanks,
Jaka

BradSteiner

What could cause that? Sometimes it seems to run fine, but I was seeing it a lot yesterday. It would only run for a few minutes before crashing due to that error. Is it the USB cable? Are the PoE version more reliable?

jakaskerl

BradSteiner
Could be USB cable (lower bandwidth causing a block), POE (even lower bandwidth + ETH stack on CPU).. What does your pipeline look like? Is your host side loop fast enough to read the queues? If you set all queues to non-blocking, does the issue disappear.

Thanks,
Jaka

CamDuarte

jakaskerl This is using the ROS Humble driver. Any idea on how this gets dealt with there?

BradSteiner

Hi, we are still seeing issue even with the PoE version. It looks like this:

[component_container-1] [ERROR] [1742415245.811915967] [oak]: No data on logger queue!
[component_container-1] [ERROR] [1742415245.812111873] [oak]: Camera diagnostics error: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'sys_logger_queue' (X_LINK_ERROR)'

We are launching with this command:
ros2 launch depthai_ros_driver camera.launch.py params_file:=camera_config.yaml

where camera_config.yaml looks like this:

/oak:
  ros__parameters:
    camera:        
      i_pipeline_type: rgbd
      i_nn_type: none
      i_floodlight_brightness: 1000  # 1500 is max
      i_laser_dot_brightness: 1200  # 1200 is max
    rgb:
      i_fps: 10.0                                                                                                                                                                  
      i_resolution: '4K' #'720P' #'4K'       
      i_set_isp_scale: false

    stereo:
      i_width: 1280 #512 1280
      i_height: 720 #288 720
      # i_resolution: '720P'
      i_extended_disp: true
      i_subpixel: false
      i_depth_preset: HIGH_DENSITY
      i_enable_decimation_filter: true
      i_enable_temporal_filter: false
      i_enable_speckle_filter: true
      i_enable_spatial_filter: true
      i_lr_check: true

    left:
      i_publish_topic: true

jakaskerl

BradSteiner cc @Luxonis-Adam

Luxonis-Adam

Hi, could you check camera CPU usage? You can get that either via subscribing to diagnostics topic or enabling DEBUG logs via running export DEPTHAI_DEBUG=1. You can also disable diagnostics queue just in case by setting

pipeline_gen:
    i_enable_diagnostics: false

CamDuarte

Sorry for the late reply. Here is a snapshot of the camera usage.

status:
- level: "\0"
  name: 'oak: sys_logger'
  message: System Information
  hardware_id: /oak_18443010616C980F00_OAK-D-PRO-POE
  values:
  - key: Leon CSS CPU Usage
    value: '99.859'
  - key: Leon MSS CPU Usage
    value: '9.071'
  - key: Ddr Memory Usage
    value: '229.081'
  - key: Ddr Memory Total
    value: '333.263'
  - key: Cmx Memory Usage
    value: '2.10254'
  - key: Cmx Memory Total
    value: '2621440'
  - key: Leon CSS Memory Usage
    value: '68.5048'
  - key: Leon CSS Memory Total
    value: '81.6971'
  - key: Leon MSS Memory Usage
    value: '4.67026'
  - key: Leon MSS Memory Total
    value: '39.8734'
  - key: Average Chip Temperature
    value: '54.6903'
  - key: Leon CSS Chip Temperature
    value: '56.7999'
  - key: Leon MSS Chip Temperature
    value: '53.3338'
  - key: UPA Chip Temperature
    value: '54.8573'
  - key: DSS Chip Temperature
    value: '53.7703'
---

Values are pretty consistent across messages

Luxonis-Adam

Hi, it seems that CPU usage is too high (99%), this can cause connection issues as mentioned in the documentation. To improve it, you would need to adjust your pipeline accordingly (lower resolutions, reduce FPS, limit 3A algorithms cc @jakaskerl what else could might help here)
More information:
https://docs.luxonis.com/hardware/platform/deploy/poe-deployment-guide#PoE%20deployment%20guide-Runtime-Debugging
https://docs.luxonis.com/software/depthai/debugging#cpu-usage

CamDuarte

Luxonis-Adam Even with the following configuration

/oak:
  ros__parameters:
    camera:        
      i_pipeline_type: rgbd
      i_nn_type: none
      i_floodlight_brightness: 800  # 800 is max
      i_laser_dot_brightness: 1200  # 1200 is max
    rgb:
      i_low_bandwidth: true
    left:
      i_low_bandwidth: true
    right:
      i_low_bandwidth: true
    stereo:
      i_low_bandwidth: true

We do see an increased framerate (~20 fps) at the expense of poor depth images.

However, the CPU utilization seems to be still quite high. Here is the /diagnostics output:

status:
- level: "\0"
  name: 'oak: sys_logger'
  message: System Information
  hardware_id: /oak_18443010516DE7F400_OAK-D-PRO-POE
  values:
  - key: Leon CSS CPU Usage
    value: '98.693'
  - key: Leon MSS CPU Usage
    value: '17.734'
  - key: Ddr Memory Usage
    value: '183.424'
  - key: Ddr Memory Total
    value: '333.263'
  - key: Cmx Memory Usage
    value: '2.37988'
  - key: Cmx Memory Total
    value: '2621440'
  - key: Leon CSS Memory Usage
    value: '67.9544'
  - key: Leon CSS Memory Total
    value: '81.6971'
  - key: Leon MSS Memory Usage
    value: '5.65635'
  - key: Leon MSS Memory Total
    value: '39.8734'
  - key: Average Chip Temperature
    value: '50.031'
  - key: Leon CSS Chip Temperature
    value: '50.9173'
  - key: Leon MSS Chip Temperature
    value: '49.3651'
  - key: UPA Chip Temperature
    value: '50.0317'
  - key: DSS Chip Temperature
    value: '49.8097'
---

Is this expected behavior using the ROS node?

BradSteiner

Our framerate is already fairly low at 10 FPS. Is it expected to see such high CPU usage at this framerate?

Luxonis-Adam

Hi, sorry for the delay, I think with the default resolutions and fps it is to be expected since the camera has to run the network stack as well, you can try decreasing those parameters and/or ISP3A algorithms to see if it improves things much for you, you could maybe also try some other encoder options. cc @jakaskerl