• DepthAI-v2

  • Found 2 devices -- Failed to find device after booting

Heya! (Jaka)

I've tried to run the depthai-experiments // gen2-multiple-devices to test the 2x S2 PoE cam setup and I get this error:

Found 2 devices

Exception in thread Thread-1 (worker):

Traceback (most recent call last):

File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner

self.run()

File "/usr/lib/python3.11/threading.py", line 975, in run

self._target(*self._args, **self._kwargs)

File "/home/deeplearning/Desktop/Creating/CompVi/OAK/depthai-experiments/gen2-multiple-devices/multi-device-mobilenet.py", line 45, in worker

device = stack.enter_context(dai.Device(openvino_version, device_info, usb2_mode))

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

RuntimeError: Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND

Seems like a weird error. All I want to do tbh….at this point….is to be able to access a camera and get the raw video….maybe in h.264 for then to do inference on the host machine….

The docs are surprisingly…well..vague…although there's some helpful images around, like the one on API Diagram of Host <-> Cam but there's no good docs on a raw easy way to connect and start messing with our (PoE) cams.

Any suggestions?

    Hi The_Real_Enrico_Pallazzo
    Do normal (single device) examples work as expected? Have you tried the cam_test.py script?
    If normal operation is as expected, I suggest implementing a check of the device state to see if each of the devices can be connected to.
    There is one solution by Penguinmon here.

    Hope this helps,
    Jaka

        jakaskerl

        Both cams have been tested with the depthai_demo.py over the local network and were found to be working all OK.
        (PC <--> switch <--> PoE Cams)

        I've also tired using the extra flags for testing if that might work (it was needed on my Orin Nano…but not on PC for regular test)… DEPTHAI_WATCHDOG_INITIAL_DELAY=60000 DEPTHAI_BOOTUP_TIMEOUT=60000 python3 script.py

        So so far…am considering writing from scratch a basic connectivity script to access the cameras…since the Pipeline docs are kind of …non-existing.

        Is there a newer version you folks will be releasing anytime soon?

            Hi The_Real_Enrico_Pallazzo
            Any script/example written with API or SDK should run on OAK-s2-POE. The problem you are experiencing likely can be resolved through the solution I have sent above.

            If you need additional docs, there is always the python api reference which should include the information on usage for pretty much any function available.

            Then there's https://docs.luxonis.com/projects/hardware/en/latest/pages/guides/getting-started-with-poe/#getting-started-with-oak-poe-devices

            Thanks,
            Jaka

              Tried some debugging with the solution of Penguinmon…and printed out something weird…

              While it did continue to one device …somehow magically….and then failed (1 out of 30 tries so far)….I'm starting to think both cameras are defaulting to same I.P address? Is that possible?

                The_Real_Enrico_Pallazzo
                Additional debugging.
                Tried NMAP on the network:
                ~/Desktop/Creating/CompVi/OAK/gen2-multiple-devices  nmap -T5 -sn 169.254.1.0-255

                Starting Nmap 7.93 ( https://nmap.org ) at 2023-10-02 17:44 CEST

                Nmap scan report for DeepLearningRig (169.254.1.10)

                Host is up (0.000040s latency).

                Nmap scan report for 169.254.1.222

                Host is up (0.00059s latency).

                Nmap done: 256 IP addresses (2 hosts up) scanned in 2.15 seconds

                Tried removing a single cam and pinging it…
                ~/Desktop/Creating/CompVi/OAK/gen2-multiple-devices  ping 169.254.1.222  ✔ │ OAK 

                PING 169.254.1.222 (169.254.1.222) 56(84) bytes of data.

                64 bytes from 169.254.1.222: icmp_seq=1 ttl=64 time=0.167 ms

                64 bytes from 169.254.1.222: icmp_seq=2 ttl=64 time=0.133 ms

                64 bytes from 169.254.1.222: icmp_seq=3 ttl=64 time=0.130 ms

                64 bytes from 169.254.1.222: icmp_seq=4 ttl=64 time=0.134 ms

                --- 169.254.1.222 ping statistics ---
                4 packets transmitted, 4 received, 0% packet loss, time 3050ms

                rtt min/avg/max/mdev = 0.130/0.141/0.167/0.015 ms

                This was the result for both cams (inter-change-ably connected)…..so both work….but when both plugged into switch only one gets discovered by NMAP….which is weird…since the programming main.py from gen2-multiple-devices was stating it found both?

                I really do think both might be trying to jump on the same IP default address for them…the 169.254.1.222

                    jakaskerl
                    Is there a way to hard-code into bootloader a new / different I.P to give to my cameras….i.e. overwrite which generic IP it is set up with?

                    I understand now the reason the script detected 2 cameras is not via IP….but the MxID…

                    The only reference to the MAC / MxID I found is this C++ file:
                    GitHub MxID C++

                        The_Real_Enrico_Pallazzo
                        Like jakaskerl said, you can set the IP for each device.

                        I've also had luck using dnsmasq to run a dhcp server on a pi. The pi (or other device) would have to be connected to the switch that the cameras are connected to. This way you can just plug in any camera without having to worry about its IP.

                            Penguinmon
                            I gave up with the custom DCHP server, it was half-working but still issues. Straight up ordered a Mikrotik hEX S Ethernet-Router to resolve the DCHP IP allocation…… before it arrives I might try your trick with dnsmasq on the Orin Nano to see if that gets me anywhere.

                            In short I'll try custom on-device (Orin) DCHP with this:
                            PC <--> PoE switch <--> PoE Cams

                            Then if failed I'll hopefully resolve it with the new router
                            PC <--> router <--> PoE switch <--> PoE Cams

                              jakaskerl
                              Have tried Penguinmon's suggestion with DNSMASQ and now both cameras get a proper IP….but also for some reason get picked up on the "default" .222 one.

                              Well regardless, when running a script…now it splits into one working and one "broken" camera…alternating when re-running script.

                              The output for error is:
                              Found 2 devices

                              [2023-10-07 18:35:45.699] [warning] Watchdog initial delay set to 60000ms

                              [2023-10-07 18:35:46.700] [warning] Watchdog initial delay set to 60000ms

                              Exception in thread Thread-2:

                              Traceback (most recent call last):

                              File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner

                              self.run()

                              File "/usr/lib/python3.8/threading.py", line 870, in run

                              self._target(*self._args, **self._kwargs)

                              File "main.py", line 31, in worker

                              device: dai.Device = stack.enter_context(dai.Device(openvino_version, dev_info, False))

                              depthai.XLinkWriteError: Couldn't write data to stream: '__bootloader' (X_LINK_ERROR)

                              === Connected to 1944301061BA5B2700

                              >>> MXID: 1944301061BA5B2700

                              >>> Cameras: RGB LEFT RIGHT

                              >>> USB speed: UNKNOWN

                              Thoughts?

                              Edit: Both cameras work with depthai_demo.py when given option to select [0] or device [1]
                              Edit2: current NMAP shows this
                              nmap -T5 -sn 169.254.1.0-255

                              Starting Nmap 7.80 ( https://nmap.org ) at 2023-10-07 18:41 CEST

                              Nmap scan report for OrinNano (169.254.1.10)

                              Host is up (0.00010s latency).

                              Nmap scan report for 169.254.1.213

                              Host is up (0.0010s latency).

                              Nmap scan report for 169.254.1.220

                              Host is up (0.0011s latency).

                              Nmap scan report for 169.254.1.222

                              Host is up (0.00095s latency).

                              Nmap done: 256 IP addresses (4 hosts up) scanned in 1.77 seconds

                                  Hi The_Real_Enrico_Pallazzo
                                  The script still acts as though both devices are on the same IP. I think that when dnsmasq assigns new IP to the device, the old (fallback) one is retained as well, resulting in two devices on different IPs also having an overlapping IP which is the one DepthAI picks up.
                                  Are you explicitly specifying the IPs of the devices when connecting to them?
                                  Few things to try:

                                  • arp -a when one device is connected to see if both IPs point to the same mac address
                                  • ping the same device at both IPs to see if it responds

                                  Also, why not just set the static IP to the device using device_manager.py?

                                  Thanks,
                                  Jaka