Camera diagnostics error 'sys_logger_queue' (X_LINK_ERROR)'

MariosAntakis

Hello!

I have lately acquired a LUXONIS OAK-D Pro W camera and I am frequently getting the following error:

$$
[ERROR] [1787576071.126707272]: No data on logger queue!
[ERROR] [1787576071.127410650]:
$$

crash-dump.zip

6kB

crash-dump.zip

6kB

$$
Camera diagnostics error: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'sys_logger_queue' (X_LINK_ERROR)'
$$

While the connections seems to be good suddenly this error occurs and no communication is possible. This error might occur for example after a change in the config .yaml of the launch file.

My setup incudes a LUXONIS OAK-D Pro W camera and I am using DepthAIv2 along with DepthAI-Ros-Driver (Noetic branch with some minor changes) for ROS Melodic. The power of the OAK is through an external power source using the Y-adapter.

At a launch with no problem I got these metrics which seem to be pretty good.

$$
[14442C1091F0CFD200] [3.7] [106.019] [system] [info] Memory Usage - DDR: 173.24 / 332.98 MiB, CMX: 2.10 / 2.50 MiB, LeonOS Heap: 51.53 / 81.56 MiB, LeonRT Heap: 4.65 / 39.75 MiB / NOC ddr: 6
30 MB/s
[14442C1091F0CFD200] [3.7] [106.019] [system] [info] Temperatures - Average: 46.45C, CSS: 47.35C, MSS 46.23C, UPA: 46.68C, DSS: 45.55C
[14442C1091F0CFD200] [3.7] [106.019] [system] [info] Cpu Usage - LeonOS 42.42%, LeonRT: 32.46%
[14442C1091F0CFD200] [3.7] [106.019] [StereoDepth(9)] [debug] Executing spatial filter
[14442C1091F0CFD200] [3.7] [106.024] [system] [warning] PRINT:LeonCss: 106.019 27550.493 WD:1978 %:42.4,32.5 DDR:173.24 CMX:2.10 Heap:51.53,4.65 NOC:630 TX:12402368 Temp:47,46,46,45
$$

Oskar-Luxonis

Hey MariosAntakis - the crash dump shows that the device firmware crashes first; the sys_logger_queue X_LINK error is a result of that crash.

Could you share the launch file and YAML that cause the issue, plus the output of python3 -c "import depthai as dai; print(dai.__version__)" from the ROS environment, If you are using old version please update to newest first. Please also confirm if the same problem occurs with an unmodified depthai_ros launch.

Thanks,
Oskar

MariosAntakis

launch-configs.zip

4kB

I share with you my config file rgbd.yaml along with the launch file rgbdpcl.launch and the camera.launch which is called inside it. I have installe depthai-core from source (branch v2stable) and I am still using ROS Melodic since the whole software stack of the robot is built on it.

If I understand correctly no DEPTHAIv3 combatibility exists with ROS Melodic

Oskar-Luxonis

Hey MariosAntakis - we tested the equivalent RGBD pipeline on an OAK-D Pro W with DepthAI v2.32.0.0 and could not reproduce the crash.

Could you please send the output of this command from the same ROS Melodic environment that runs the camera?

python3 -c "import depthai as dai; print(dai.__version__)"

Also, as a diagnostic test, please set stereo_i_align_depth: false and check if the crash still occurs. The crash dump points to the device warp path, which depth-to-RGB alignment uses.

Thanks,
Oskar

MariosAntakis

The output is the following:

python3 -c "import depthai as dai; print(dai.__version__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'depthai'

However I have installed depthai-core from source (branch v2stable) so I cannot understand why it might be a problem.

It is a bit odd problem. I tried setting stereo_i_align_depth: false and it worked. Then I changed the camera_i_pipeline_type from RGBD to RGB and back to RGBD and the error showed up again.

MariosAntakis

I also tried using depthai-core (branch v2_stable) and depthai-ros (branch noetic) with a ROS Noetic docker image which does not require any change in the packages code and I still face this error after some changes in the config file (by change I mean that I change a parameter and then I launch the node. After I Ctrl-C the node and I change some other parameter etc.) I have checked that no running processes of this particular node remain active after Ctrl-C.

In order to make the camera run effectively (after all these consecutive changes and the appearance of the error) the solution I have found so far is restarting roscore.

Oskar-Luxonis

Hey MariosAntakis - the missing Python module is expected: depthai-core is the C++ library used by the ROS driver, not the Python package.

Since stereo_i_align_depth: false works, the failure appears related to the depth-to-RGB alignment path. Please keep this setting disabled for now and also set:

camera_i_restart_on_diagnostics_error: false

Could you send the exact revisions you built?

cd /path/to/depthai-core && git rev-parse HEAD
cd /path/to/depthai-ros && git rev-parse HEAD

Thanks,
Oskar

MariosAntakis

Hey

stereo_i_align_depth: false do yoy have any possible explanation regarding this issue?
And the output you asked for:
cd /path/to/depthai-core && git rev-parse HEAD outputs bd34eb5637640f23250170acc37f08ef11fc2c5f and cd /path/to/depthai-ros && git rev-parse HEAD outputs e0945a05d18bf0383cbed7187bedf0a09dfeb07a. However for depthai-ros I have forked and created a new branch with minimal chnages in order for it to be successfully built in ROS melodic. (These changes seem to not be the problem since in ROS Noetic which does not require these changes I face a similar problem).

The Python module is mandatory and should be installed using pip?

Oskar-Luxonis

Hey

Your depthai-core revision is v2.32.0, which matches our test version.

stereo_i_align_depth: true uses the device warp path to transform stereo depth into the RGB-camera view. Your crash occurs in that warp path (PlgWarpHW), so disabling alignment avoids the failing operation. This can relate to the calibration data or to a v2.32 issue in this specific configuration.

Could you set this temporarily and share the created file?

camera_i_calibration_dump: true

The driver should save the calibration file to /tmp/<MXID>_calibration.json when it starts.

Python DepthAI is not required for depthai_ros; install it only if you need to run Python examples or diagnostics.

Thanks,
Oskar

MariosAntakis

calib-dump.zip

2kB

with camera_i_restart_on_diagnostics_error: true and stereo_i_align_depth: false

MariosAntakis

I do not know if it is somehow related but after some runs with the same config file I get this StereoDepth(8) Error

[ INFO] [1787751000.222247076]: Pipeline type: RGBD
[ INFO] [1787751001.547930576]: Finished setting up pipeline.
[14442C1091F0CFD200] [3.7] [2.725] [StereoDepth(8)] [error] Custom disparity/depth width must be multiple of 16.
[ INFO] [1787751002.466103879]: Camera ready!
[ERROR] [1787751005.460237380]: No data on logger queue!
[ERROR] [1787751005.460895973]: Camera diagnostics error: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'sys_logger_queu
e' (X_LINK_ERROR)'

no matter if stereo_i_align_depth: false or stereo_i_align_depth: true

It appears because of the stereo_i_enable_decimation_filter: true parameter. For some reason during consecutive launches I see the following output from first to last launch (the last one is the one where the error occurs). My monos are 720P and the RGB 1080P in this specific try:

Decimation filter enabled with decimation factor 2. Previous width and height: 1280 x 720, after decimation: 640 x 360
Decimation filter enabled with decimation factor 2. Previous width and height: 640 x 360, after decimation: 320 x 180
Decimation filter enabled with decimation factor 2. Previous width and height: 320 x 180, after decimation: 160 x 90
Decimation filter enabled with decimation factor 2. Previous width and height: 160 x 90, after decimation: 80 x 45
Decimation filter enabled with decimation factor 2. Previous width and height: 80 x 45, after decimation: 40 x 22

It looks like some parameters are saved in the ROS parameter server and are used for every camera launch unless roscore is restarted every time.

Should I explicitly set stereo_i_height and stereo_i_width parameters? Or add clear_params="true" in the launch file for the \oak node? If I do this I observe that if stereo_i_align_depth: true the stereo output is aligned in resolution with the RGB Image. For example if RGB resolution is 1080P and stereo 720P then stereo is resized to 1080P. So I suspect that if I set the parameters by hand in the config file I should also apply the above resize?

Oskar-Luxonis

MariosAntakis - we reviewed the driver source and your logs. This is a ROS 1 driver issue, not a camera calibration issue.

With the decimation filter enabled, the driver calculates the reduced output size, then writes that reduced width and height back to the ROS parameter server. On the next launch it reads those values and decimates them again:

1280×720 → 640×360 → 320×180 → 160×90 → 80×45 → 40×22

The final width is invalid because custom StereoDepth output width must be divisible by 16. The later sys_logger_queue XLink error is a result of the invalid device pipeline.

Do not set fixed stereo_i_width and stereo_i_height values. With depth alignment enabled, the driver normally gets these from the RGB resolution.

As a temporary workaround, clear the saved camera parameters before every launch:

rosparam delete /oak

Then launch the camera normally.

This is the legacy, deprecated DepthAI v2 / ROS 1 driver path. ROS Melodic is also outside the supported ROS versions for this driver. We recommend moving to a supported current ROS and DepthAI stack when this is possible for your robot software.

MariosAntakis

Thanks a lot for the reply @Oskar-Luxonis !

MariosAntakis

In left, right mono and the rgb camera should I only set the desired resolution and the width and height are configured based on that value? Or should I set both resolution and based on that manually change the width, height params?

Also based on the versions I use does it make sense for my RGB camera resolution to fall to 720P while I explicitly set it at 1080P in the config file?

Oskar-Luxonis

Hey MariosAntakis - Set both resolution and matching width/height values. For example, use 720P with 1280×720, and 1080P with 1920×1080.

For RGB, also set:

rgb_i_set_isp_scale: false

Do not set fixed stereo_i_width or stereo_i_height while depth alignment is enabled.

With rgb_i_resolution: 1080P and rgb_i_width/height: 1920×1080, RGB should not fall to 720P. Before each test, run:

rosparam delete /oak

This clears the stale values from previous launches.

MariosAntakis

Will this rgb_i_set_isp_scale: false be a problem in case we also want to perfrom NN inference? Otherwise as I understand it does not affect the functionality of the camera and just stops ISP from resizing RGB images to lower resolution.

Is this parameter deprecated as I cannot find it in the DepthAIv3 ROS driver params?

Oskar-Luxonis

No, this should not prevent NN inference. It only stops the legacy driver from automatically scaling RGB down with the ISP.

The NN input is resized to the model input size separately. The main trade-off is higher device processing and USB bandwidth because RGB stays at 1080p.

This is a legacy DepthAI v2 / ROS 1 parameter. It is not used by the v3 ROS driver, which uses a different camera-output and resize configuration.