OAK-D S2 PoE; "ping was missed" error; LeonOS usage increase.

John194 · 2023-09-07T10:19:27+00:00

Hello, I'm experiencing a couple of problems. "ping was missed" error within 1 hour of usage. Rare sudden LeonOS usage increase. Problem 1: "ping was mi...

erik

Hi John194 ,
Thank you for the detailed report, we will investigate this issue further. In the meantime, could you try:

Increasing watchdog delay
Decrease 3A FPS, to reduce CPU usage. I suspect the "ping missed" issue is caused due to high CPU usage.

Thanks, Erik

John194

erik

When I set the watchdog variable to 60s in the docker entry point and ran my program, ping was still missed.

[system] [info] Memory Usage - DDR: 49.19 / 337.18 MiB, CMX: 2.41 / 2.50 MiB, LeonOS Heap: 66.16 / 80.03 MiB, LeonRT Heap: 4.99 / 41.06 MiB
[system] [info] Temperatures - Average: 56.15C, CSS: 57.01C, MSS 55.51C, UPA: 55.94C, DSS: 56.15C
[system] [info] Cpu Usage - LeonOS 95.65%, LeonRT: 21.50%
[host] [warning] Monitor thread (device: 194430108198721300 [169.254.1.222]) - ping was missed, closing the device connection
[host] [debug] Timesync thread exception caught: Couldn't read data from stream: '__timesync' (X_LINK_ERROR)
[host] [debug] Log thread exception caught: Couldn't read data from stream: '__log' (X_LINK_ERROR)
[WARN] [depthai_camera]: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'right' (X_LINK_ERROR)'
[host] [debug] Device about to be closed...
[host] [debug] Device closed, 1972
[host] [warning] Watchdog initial delay set to 60000ms

John194

Thanks for the reply. I forgot to mention that I have already set 3A FPS to 13, while the camera FPS is set to 30. I don't want 3A FPS any lower for my application. I didn't try testing the Watchdog delay and will report back soon.

themarpe

@John194 do you mind setting the following variable:

DEPTHAI_WATCHDOG=30000

The one mentioned in documentation only affects the inital boot and not the interval pings. (CC: @erik )

Note: in case of clean exit of the host program, the device might not be available again for cca 30s with above set.

John194

themarpe

Having set DEPTHAI_WATCHDOG=30000 , the camera died within a few seconds. Logs didn't mention a missed ping and no crash dump. Will try to make an MRE later.

[host] [debug] Timesync thread exception caught: Couldn't read data from stream: '__timesync' (X_LINK_ERROR)
[host] [debug] Log thread exception caught: Couldn't read data from stream: '__log' (X_LINK_ERROR)
[depthai_camera]: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'left' (X_LINK_ERROR)'
[host] [debug] Device about to be closed...
[host] [debug] Device closed, 14997
[host] [warning] Using a custom watchdog value of 30000ms
[system] [warning] Watchdog was capped from 30000ms to 4500ms

themarpe

@John194

Also, if you can provide a MRE, that would help us pin down the issue in case it is crash related

John194

themarpe

I made the MRE here.

John194

themarpe

After further testing with my mentioned OAK-D S2 PoE, a new OAK-D S2 PoE, and OAK-D PoE, here are some observations:

Sometimes "ping was missed" occurs a few times within 10 minutes, other times it can go on for hours.
"ping was missed" was very frequent for all cameras on bootloader v0.0.19. On v0.0.26 it happens more rarely;
The new OAK-D S2 PoE and OAK-D PoE "ping was missed" seem less frequent than the old OAK-D S2 PoE;
Testing on a Laptop without running docker, "ping was missed" seems to happen less frequently than on a Jetson with docker;
"ping was missed" happens on Jetson Orin Nano too;
Changing network cables to shielded ones did not help;
Turning off the RGB camera doesn't help;
Turning off left and right cameras with disparity seems to throw "ping was missed" less often;

themarpe

@John194 does this happen under a specific pipeline or do many examples showcase this behavior?

If its the former, an MRE of the pipeline would be great to have to pin down the root cause.

In the latter, what is your networking setup like, as we've saw this playing a role in this case as well

John194

themarpe

This happens with many examples, but they have some similarities. I already posted the code in my previous message, it is pretty simple.

What should I provide about my network setup?

erik

Hi John194 ,
Here are a few options for the ping was missed issue:
https://docs.luxonis.com/en/latest/pages/troubleshooting/#ping-was-missed-closing-the-device-connection