X_LINK_ERROR, camera disconnects after one hour ish

DavidShaer

Hi,

I connected three cameras and run it on the Jetson using the a script that based on oak-reconnect.py.

Once in an hour ish I'm getting camera disconnection.

Hardware configuration used for this test:

Nvida Jetson AGX Orin (64 GB memory, running in 50W power mode)
1x Oak-D Pro Camera module
2x Oak-1 Lite Camera modules
Cameras were connected to Jetson using the following USB hub: Insignia™ 4-Port USB 3.0 Powered Hub Metallic Gray NS-PH3A4AP - Best Buy
Oak-D Pro was connected to the hub using original cable
The other cameras were connected using USB 3 "Super" speed cables. They did not come with cables in the box.
I tried swapping the cables between the different modules, and received the same type of error.

Error snippet is as follows:

[19443010C1F19D2E00] [1.4.1.3] [1711396014.090] [host] [debug] Device about to be closed...

[19443010C1F19D2E00] [1.4.1.3] [1711396014.093] [host] [debug] Shutdown OK

[18443010B134660F00] [1.4.1.2] [3530.546] [system] [info] Memory Usage - DDR: 29.36 / 333.46 MiB, CMX: 2.39 / 2.50 MiB, LeonOS Heap: 21.67 / 82.56 MiB, LeonRT Heap: 3.47 / 40.68 MiB / NOC ddr: 487 MB/s

[18443010B134660F00] [1.4.1.2] [3530.546] [system] [info] Temperatures - Average: 49.86C, CSS: 51.36C, MSS 49.37C, UPA: 49.14C, DSS: 49.59C

[18443010B134660F00] [1.4.1.2] [3530.546] [system] [info] Cpu Usage - LeonOS 18.40%, LeonRT: 2.98%

[19443010C1F19D2E00] [1.4.1.3] [1711396014.229] [host] [debug] Watchdog thread exception caught: Couldn't write data to stream: '__watchdog' (X_LINK_ERROR)

[19443010C1F19D2E00] [1.4.1.3] [1711396014.242] [host] [debug] Log thread exception caught: Couldn't read data from stream: '__log' (X_LINK_ERROR)

[19443010C1F19D2E00] [1.4.1.3] [1711396014.242] [host] [debug] Timesync thread exception caught: Couldn't read data from stream: '__timesync' (X_LINK_ERROR)

I have the full log file and the test script but for some reason the files uploading doesn't work for me.

I will be happy for your assistance,

Thanks,

David

jakaskerl

Hi @DavidShaer
The script is meant to be used with a single device. How are you handling the multiple device instances? Please also check the resource usage for the script during runtime.

Thanks,
Jaka

DavidShaer

Hi,
In order to connect multiple cameras - I used getAllAvailableDevices function.
I uploaded the script and the log with the resource usage here

Thanks
David

erik

Hi @DavidShaer ,
You are likely seeing this issue due to insufficient power. The USB3 hub you are using is powered by 5V/4A power supply, which is likely at the limit of supplying power to 3 cameras.
While average power usage is around 5w for these cameras, there are random spikes of 2w (so up to 7w). Maybe once every hour all 3 of the cameras have this power spike at the exact same time, hub can't provide enough current, and cameras (maybe only one) brownout.
Have you already tried:

Using a different (more powerful) usb3 hub?
Connecting 1 or 2 cameras directly to the computer, instead of all via usb3 hub?

DavidShaer

Hi @erik

So in order to check if we have a power related issue I did couple of changes as following:

a. connected the OAK-D camera directly to the Jetson (with the original cable and Y-adapter with power)

b. connected the other two OAK1-lite cameras via USB-Hub with power (the Hub specifications described in the previous message^)

c. reducing the Frame Rate to 5 Fps

And I ran two experiments:

Connecting all three cameras: and got disconnections of the left camera after about two hours, relevant logs attached here
Connecting two cameras (one OAK-D and one OAK1-lite cameras): and got disconnections of the left camera after about 30 minutes, relevant logs attached here

The script I ran is the same as the previous one I shared, but with one difference of the FPS setting, relevant script attached here

Note:
We notice in the logs "ping was missed" message, maybe it is related.

Thanks,

David

erik

Hi @DavidShaer ,
Could you share some images/video of your setup? I'd also suggest checking "Connection drop" tab in USB connection debugging docs. Also, are you using the latest depthai version (2.25)?
Thanks, Erik

DavidShaer

Hi @erik,

We run couple of tests as follows:

Test #1 (the previous test we sent in the last message):

Top camera (OAK-D) was connected via Y-Adapter directly to the Jetson, and the left and right cameras (OAK1-lite) we're connected via the USB-Hub and without Y-Adapters.
We ran couple of connections tests from the connection debugging docs you suggested, we think the results are valid and you can see them here.
In addition we checked the crash report and we didn't saw any relevant reports.

Test #2 (new test: connecting all three cameras via Y-Adapter and power supply):

all three cameras we're connected directly to the Jetson, in this test we saw disconnections after about two hours ish.
relevant log attached here
sides cameras connected via USB-Hub, and the top camera was connected directly to the Jetson, in this test we saw disconnections after about two hour also.
relevant log attached here

You can see some images of the Setup here
depthai version: 2.25
We used original Luxonis Cables and Power suppliers.

We'll be happy for your help.
Thanks,
David

jakaskerl

Hi @DavidShaer
This seems to be a bigger problem, likely related to Jetsons only. The syslog and dmesg give no valuable info so we will try to debug this in depth.

Will report back when we get an idea of what the cause is.

Thanks,
Jaka

DavidShaer

jakaskerl Thanks 🙂

Hector

@jakaskerl

Hi, I have a similar issue with the Orin nx 16GB module(Jetpack 5.1.2) with FFC-4P, where the device disconnects randomly (sometimes it works for 4-5 hours and sometimes 10 minutes).

I have also tried using UBS2, and it still disconnects randomly.

But when I run the same code with Xavier NX, everything works as expected.

So, if there are any updates regarding this issue, please let us know … it will be very helpful.

Thanks

Hector

@jakaskerl Are you guys actively looking at this issue? Any updates will help us decide on that platform.

jakaskerl

Hector
We tested this multiple times, everything points to the issue being on jetson's USB drivers. The port stops giving power, looks to the OAK device as a standard USB disconnect, nothing too suspicious.

PoE devices work as expected. We will try with Jetpack6 as well to see if there is any difference in driver setup but I'm doubtful whether it will work.

Thanks,
Jaka

Hector

Please try with Jetpack 6 that would be very helpful. Are there any timelines you can provide for this?

jakaskerl

Hector
Over the weekend.

Thanks,
Jaka

jakaskerl

HI Hector
Sorry for the long pause, forgot to reply here. The Jetpack6 did not fix the issue.

Thanks,
Jaka