Found 2 devices -- Failed to find device after booting

The_Real_Enrico_Pallazzo

Heya! (Jaka)

I've tried to run the depthai-experiments // gen2-multiple-devices to test the 2x S2 PoE cam setup and I get this error:

Found 2 devices

Exception in thread Thread-1 (worker):

Traceback (most recent call last):

File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner

self.run()

File "/usr/lib/python3.11/threading.py", line 975, in run

self._target(*self._args, **self._kwargs)

File "/home/deeplearning/Desktop/Creating/CompVi/OAK/depthai-experiments/gen2-multiple-devices/multi-device-mobilenet.py", line 45, in worker

device = stack.enter_context(dai.Device(openvino_version, device_info, usb2_mode))

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

RuntimeError: Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND

Seems like a weird error. All I want to do tbh….at this point….is to be able to access a camera and get the raw video….maybe in h.264 for then to do inference on the host machine….

The docs are surprisingly…well..vague…although there's some helpful images around, like the one on API Diagram of Host <-> Cam but there's no good docs on a raw easy way to connect and start messing with our (PoE) cams.

Any suggestions?

jakaskerl

Hi The_Real_Enrico_Pallazzo
Do normal (single device) examples work as expected? Have you tried the cam_test.py script?
If normal operation is as expected, I suggest implementing a check of the device state to see if each of the devices can be connected to.
There is one solution by Penguinmon here.

Hope this helps,
Jaka

The_Real_Enrico_Pallazzo

jakaskerl

Both cams have been tested with the depthai_demo.py over the local network and were found to be working all OK.
(PC <--> switch <--> PoE Cams)

I've also tired using the extra flags for testing if that might work (it was needed on my Orin Nano…but not on PC for regular test)… DEPTHAI_WATCHDOG_INITIAL_DELAY=60000 DEPTHAI_BOOTUP_TIMEOUT=60000 python3 script.py

So so far…am considering writing from scratch a basic connectivity script to access the cameras…since the Pipeline docs are kind of …non-existing.

Is there a newer version you folks will be releasing anytime soon?

jakaskerl

Hi The_Real_Enrico_Pallazzo
Any script/example written with API or SDK should run on OAK-s2-POE. The problem you are experiencing likely can be resolved through the solution I have sent above.

If you need additional docs, there is always the python api reference which should include the information on usage for pretty much any function available.

Then there's https://docs.luxonis.com/projects/hardware/en/latest/pages/guides/getting-started-with-poe/#getting-started-with-oak-poe-devices

Thanks,
Jaka

The_Real_Enrico_Pallazzo

Tried some debugging with the solution of Penguinmon…and printed out something weird…

While it did continue to one device …somehow magically….and then failed (1 out of 30 tries so far)….I'm starting to think both cameras are defaulting to same I.P address? Is that possible?

The_Real_Enrico_Pallazzo

The_Real_Enrico_Pallazzo
Additional debugging.
Tried NMAP on the network:
~/Desktop/Creating/CompVi/OAK/gen2-multiple-devices  nmap -T5 -sn 169.254.1.0-255

Starting Nmap 7.93 ( https://nmap.org ) at 2023-10-02 17:44 CEST

Nmap scan report for DeepLearningRig (169.254.1.10)

Host is up (0.000040s latency).

Nmap scan report for 169.254.1.222

Host is up (0.00059s latency).

Nmap done: 256 IP addresses (2 hosts up) scanned in 2.15 seconds

Tried removing a single cam and pinging it…
~/Desktop/Creating/CompVi/OAK/gen2-multiple-devices  ping 169.254.1.222  ✔ │ OAK 

PING 169.254.1.222 (169.254.1.222) 56(84) bytes of data.

64 bytes from 169.254.1.222: icmp_seq=1 ttl=64 time=0.167 ms

64 bytes from 169.254.1.222: icmp_seq=2 ttl=64 time=0.133 ms

64 bytes from 169.254.1.222: icmp_seq=3 ttl=64 time=0.130 ms

64 bytes from 169.254.1.222: icmp_seq=4 ttl=64 time=0.134 ms

--- 169.254.1.222 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3050ms

rtt min/avg/max/mdev = 0.130/0.141/0.167/0.015 ms

This was the result for both cams (inter-change-ably connected)…..so both work….but when both plugged into switch only one gets discovered by NMAP….which is weird…since the programming main.py from gen2-multiple-devices was stating it found both?

I really do think both might be trying to jump on the same IP default address for them…the 169.254.1.222

jakaskerl

Hi The_Real_Enrico_Pallazzo
If the switch is not connected to a router, this will happen, yes.
Docs here.

Thanks,
Jaka

The_Real_Enrico_Pallazzo

jakaskerl
Is there a way to hard-code into bootloader a new / different I.P to give to my cameras….i.e. overwrite which generic IP it is set up with?

I understand now the reason the script detected 2 cameras is not via IP….but the MxID…

The only reference to the MAC / MxID I found is this C++ file:
GitHub MxID C++

jakaskerl

Hi The_Real_Enrico_Pallazzo
You can use device_manager.py GUI (or this script) to set the cameras to some arbitrary IP address. Then it should work since IP config gets stored on the device. Just make sure to set the IP according to the netmask.

Thanks,
Jaka

Penguinmon

The_Real_Enrico_Pallazzo
Like jakaskerl said, you can set the IP for each device.

I've also had luck using dnsmasq to run a dhcp server on a pi. The pi (or other device) would have to be connected to the switch that the cameras are connected to. This way you can just plug in any camera without having to worry about its IP.

The_Real_Enrico_Pallazzo

jakaskerl
Have tried Penguinmon's suggestion with DNSMASQ and now both cameras get a proper IP….but also for some reason get picked up on the "default" .222 one.

Well regardless, when running a script…now it splits into one working and one "broken" camera…alternating when re-running script.

The output for error is:
Found 2 devices

[2023-10-07 18:35:45.699] [warning] Watchdog initial delay set to 60000ms

[2023-10-07 18:35:46.700] [warning] Watchdog initial delay set to 60000ms

Exception in thread Thread-2:

Traceback (most recent call last):

File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner

self.run()

File "/usr/lib/python3.8/threading.py", line 870, in run

self._target(*self._args, **self._kwargs)

File "main.py", line 31, in worker

device: dai.Device = stack.enter_context(dai.Device(openvino_version, dev_info, False))

depthai.XLinkWriteError: Couldn't write data to stream: '__bootloader' (X_LINK_ERROR)

=== Connected to 1944301061BA5B2700

>>> MXID: 1944301061BA5B2700

>>> Cameras: RGB LEFT RIGHT

>>> USB speed: UNKNOWN

Thoughts?

Edit: Both cameras work with depthai_demo.py when given option to select [0] or device [1]
Edit2: current NMAP shows this
nmap -T5 -sn 169.254.1.0-255

Starting Nmap 7.80 ( https://nmap.org ) at 2023-10-07 18:41 CEST

Nmap scan report for OrinNano (169.254.1.10)

Host is up (0.00010s latency).

Nmap scan report for 169.254.1.213

Host is up (0.0010s latency).

Nmap scan report for 169.254.1.220

Host is up (0.0011s latency).

Nmap scan report for 169.254.1.222

Host is up (0.00095s latency).

Nmap done: 256 IP addresses (4 hosts up) scanned in 1.77 seconds

The_Real_Enrico_Pallazzo

Penguinmon
I gave up with the custom DCHP server, it was half-working but still issues. Straight up ordered a Mikrotik hEX S Ethernet-Router to resolve the DCHP IP allocation…… before it arrives I might try your trick with dnsmasq on the Orin Nano to see if that gets me anywhere.

In short I'll try custom on-device (Orin) DCHP with this:
PC <--> PoE switch <--> PoE Cams

Then if failed I'll hopefully resolve it with the new router
PC <--> router <--> PoE switch <--> PoE Cams

jakaskerl

Hi The_Real_Enrico_Pallazzo
The script still acts as though both devices are on the same IP. I think that when dnsmasq assigns new IP to the device, the old (fallback) one is retained as well, resulting in two devices on different IPs also having an overlapping IP which is the one DepthAI picks up.
Are you explicitly specifying the IPs of the devices when connecting to them?
Few things to try:

arp -a when one device is connected to see if both IPs point to the same mac address
ping the same device at both IPs to see if it responds

Also, why not just set the static IP to the device using device_manager.py?

Thanks,
Jaka