X_LINK_DEVICE_NOT_FOUND on OAK-D PoE

RudolfLovrencic

I have problem with connecting to OAK-D PoE camera. I have it on the same LAN as the host and depthai_viewerpicks it up (I can select it in the GUI). When I select it, after about 20 seconds, I get:

[2024-06-12T06:49:56Z ERROR re_viewer::depthai::depthai] Error: Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND, Try to connect the device to a different port.

I get the same problem when trying to run RGB Preview demo in C++.
dai::Device device{pipeline, dai::DeviceInfo{"169.254.1.222"}}; throws an exception with the same error message as above.

I am running Arch Linux, depthai-core v.2.6.0, bootloader v0.0.28. I have successfuly updated the bootloader from v0.0.26 to v0.0.28using device_manager.py.

I have tried prolonging the timeout as suggested in the docs (DEPTHAI_WATCHDOG_INITIAL_DELAY=60000 DEPTHAI_BOOTUP_TIMEOUT=60000) but nothing changed, except I waited a bit longer 🙂

Finally, here is the log when running with XLINK_LEVEL=debug.

jakaskerl

Hi @RudolfLovrencic
Does the error present itself right-away even on RGB Preview script? If not, does the script work properly until it crashes? How are you powering the device?

Thanks,
Jaka

RudolfLovrencic

Hey, thanks for a prompt response!

Camera is powered by Netgear GS305P switch. The behaviour is identical in all cases: RGB Preview python, RGB Preview C++ and depthai_viewer. Attached logs are from the C++ RGB Preview script and line dai::Device device{pipeline, dai::DeviceInfo{"169.254.1.222"}}; throws (that's the only line I have changed from the original script since it originaly contained dai::UsbSpeed::SUPER and my usecase uses PoE device instaed of USB).

jakaskerl

Hi @RudolfLovrencic
Can you run with DEPTHAI_DEBUG=1. Do you get an image before the crash?

Thanks,
Jaka

RudolfLovrencic

No image since I do not get past the device creation.

Here are the logs when running C++ app with DEPTHAI_DEBUG=1. They are the same for depthai-viewer.

[2024-06-13 11:53:20.035] [depthai] [info] DEPTHAI_DEBUG enabled, lowered DEPTHAI_LEVEL to 'debug'
[2024-06-13 11:53:20.035] [depthai] [debug] Library information - version: 2.26.0, commit: 90487456cb92d15bcf0129a2f54c332250f1df31 from 2024-05-25 08:30:10 +0200, build: 2024-06-12 06:34:27 +0000, libusb enabled: true
[2024-06-13 11:53:20.039] [depthai] [debug] Initialize - finished
[2024-06-13 11:53:20.043] [host] [debug] Found an actual device by given DeviceInfo: DeviceInfo(name=169.254.1.222, mxid=1944301071DD732700, X_LINK_BOOTLOADER, X_LINK_TCP_IP, X_LINK_MYRIAD_X, X_LINK_SUCCESS)
[2024-06-13 11:53:20.043] [host] [debug] Device - OpenVINO version: universal
[1944301071DD732700] [169.254.1.222] [1718272400.043] [host] [debug] Device - BoardConfig: {"camera":[],"emmc":null,"gpio":[],"imu":null,"logDevicePrints":true,"logPath":null,"logSizeMax":null,"logVerbosity":null,"network":{"mtu":0,"xlinkTcpNoDelay":true},"nonExclusiveMode":false,"pcieInternalClock":null,"sysctl":[],"uart":[],"usb":{"flashBootedPid":63037,"flashBootedVid":999,"manufacturer":"","maxSpeed":4,"pid":63035,"productName":"","vid":999},"usb3PhyInternalClock":null,"uvc":null,"watchdogInitialDelayMs":null,"watchdogTimeoutMs":null}
libnop:
0000: b9 12 b9 07 81 e7 03 81 3b f6 81 e7 03 81 3d f6 04 bd 00 bd 00 b9 02 00 01 ba 00 be be bb 00 bb
0020: 00 be be be be be be 01 00 bb 00 be be
[2024-06-13 11:53:20.094] [depthai] [debug] Resources - Archive 'depthai-bootloader-fwp-0.0.28.tar.xz' open: 4ms, archive read: 54ms
[2024-06-13 11:53:20.343] [depthai] [debug] Resources - Archive 'depthai-device-fwp-24a3b465b979de3f69410cd225914d8bd029f3ba.tar.xz' open: 4ms, archive read: 302ms
[2024-06-13 11:53:20.353] [depthai] [debug] Searching for booted device: DeviceInfo(name=169.254.1.222, mxid=1944301071DD732700, X_LINK_BOOTLOADER, X_LINK_TCP_IP, X_LINK_MYRIAD_X, X_LINK_SUCCESS), name used as hint only
[2024-06-13 11:53:20.555] [depthai] [debug] Connected bootloader version 0.0.28
[1944301071DD732700] [169.254.1.222] [1718272403.115] [host] [debug] Booting FW with Bootloader. Version 0.0.28, Time taken: 2559ms
[2024-06-13 11:53:23.115] [depthai] [debug] DeviceBootloader about to be closed...
[2024-06-13 11:53:23.116] [depthai] [debug] XLinkResetRemote of linkId: (0)
[2024-06-13 11:53:23.811] [depthai] [debug] DeviceBootloader closed, 695
[2024-06-13 11:53:23.822] [depthai] [debug] Searching for booted device: DeviceInfo(name=169.254.1.222, mxid=1944301071DD732700, X_LINK_BOOTED, X_LINK_TCP_IP, X_LINK_MYRIAD_X, X_LINK_SUCCESS), name used as hint only
terminate called after throwing an instance of 'std::runtime_error'
  what():  Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND

jakaskerl

Hi @RudolfLovrencic
Do you have any other POE switch you could try? Failed to find device after booting usually means the device powered off before TCP connection is established.
Check the ETH cable, switch and perhaps the firewall on port 11491 for TCP.

Thanks,
Jaka

RudolfLovrencic

@jakaskerl
I have tried different 3 different ETH cables. I do not have another switch to try, but I'd say it's unlikely it's causing the problem. I have opened the firewall:

$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
11491/tcp                  ALLOW       Anywhere
11491/tcp (v6)             ALLOW       Anywhere (v6)

The problem persists.

jakaskerl

Hi RudolfLovrencic

Any chance you can check the network traffic of the device (wireshark or something similar)? The communication should go through unless blocked by some firewall.
Could you also check if you get a different response by connecting the device to DHCP server (a router) so the device gets assigned IP instead of the fallback one. Just in-case the PC is sending traffic through the wrong interface as you probably have more than one.
Update bootloader (factory bootloader option).
Supplying power via M8 connector.

Thanks,
Jaka

RudolfLovrencic

Hey, @jakaskerl, sorry for the delay!
I've inspected some traffic with tshark. Not sure if this is relevant, but in the first 20 packets, I notice ICMP gets Destination unreachable (Port unreachable):

    1 0.000000000 169.254.1.10 → 169.254.1.222 UDP 46 58035 → 11491 Len=4
    2 0.002595666 Luxonis_37:c0:9e → Broadcast    ARP 60 Who has 169.254.1.10? Tell 169.254.1.222
    3 0.002609137 NeostarTechn_ae:03:c5 → Luxonis_37:c0:9e ARP 42 169.254.1.10 is at 00:24:32:ae:03:c5
    4 0.003379046 169.254.1.222 → 169.254.1.10 UDP 82 11491 → 58035 Len=40
    5 0.337722271 169.254.1.10 → 169.254.1.222 UDP 46 42927 → 11491 Len=4
    6 0.338000293 169.254.1.10 → 169.254.255.255 UDP 46 42927 → 11491 Len=4
    7 0.338879671 169.254.1.222 → 169.254.1.10 UDP 82 11491 → 42927 Len=40
    8 0.339070036 169.254.1.10 → 169.254.1.222 TCP 74 34332 → 11490 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM TSval=2632473107 TSecr=0 WS=128
    9 0.339296089 169.254.1.222 → 169.254.1.10 UDP 82 11491 → 42927 Len=40
   10 0.339332367 169.254.1.10 → 169.254.1.222 ICMP 110 Destination unreachable (Port unreachable)
   11 0.340631083 169.254.1.222 → 169.254.1.10 TCP 74 11490 → 34332 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=64 SACK_PERM TSval=523191573 TSecr=2632473107
   12 0.340677812 169.254.1.10 → 169.254.1.222 TCP 66 34332 → 11490 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=2632473109 TSecr=523191573
   13 0.341764965 169.254.1.10 → 169.254.1.222 TCP 150 34332 → 11490 [PSH, ACK] Seq=1 Ack=1 Win=64256 Len=84 TSval=2632473110 TSecr=523191573
   14 0.342940837 169.254.1.222 → 169.254.1.10 TCP 150 11490 → 34332 [PSH, ACK] Seq=1 Ack=85 Win=65664 Len=84 TSval=523191575 TSecr=2632473110
   15 0.342991402 169.254.1.10 → 169.254.1.222 TCP 66 34332 → 11490 [ACK] Seq=85 Ack=85 Win=64256 Len=0 TSval=2632473111 TSecr=523191575
   16 0.343557252 169.254.1.10 → 169.254.1.222 TCP 150 34332 → 11490 [PSH, ACK] Seq=85 Ack=85 Win=64256 Len=84 TSval=2632473112 TSecr=523191575
   17 0.343787963 169.254.1.222 → 169.254.1.10 TCP 150 11490 → 34332 [PSH, ACK] Seq=85 Ack=85 Win=65664 Len=84 TSval=523191576 TSecr=2632473111
   18 0.343924139 169.254.1.10 → 169.254.1.222 TCP 150 34332 → 11490 [PSH, ACK] Seq=169 Ack=169 Win=64256 Len=84 TSval=2632473112 TSecr=523191576
   19 0.344769853 169.254.1.10 → 169.254.1.222 TCP 150 34332 → 11490 [PSH, ACK] Seq=253 Ack=169 Win=64256 Len=84 TSval=2632473113 TSecr=523191576
   20 0.345040757 169.254.1.222 → 169.254.1.10 TCP 66 11490 → 34332 [ACK] Seq=169 Ack=253 Win=65600 Len=0 TSval=523191577 TSecr=2632473112

I've captured ~11k packets and they can be found here: capture.pcapng (26.3MB).

jakaskerl

Hi @RudolfLovrencic
Hard to say if that is the problem since the TCP connection is established afterwards. Did you try the other methods I described above (namely the M8 connector powering)?

Thanks,
Jaka

RudolfLovrencic

@jakaskerl Do I have to open the enclosure for that? And in the docs, I only see Powering from USB.

RudolfLovrencic

@jakaskerl Solved. Here under Debugging > Multiple interfaces lies the solution. Thanks for your time!