Solved my issue. Connected via a different computer (x86-64 based instead of my ARM based Jetson Orin Nano) and proceeded to use the device_manager.py from depthai-python repo on github. Flashing both cams to latest firmware and then re-trying to connect on the Jetson now works!
I've been trying to run 2x OAK-D's with the depthai-experiments gen2-multiple-devices example found on github and it keeps failing.
I've tried stand-alone script to run each individual cam and both work… so technically can run 2 terminals and have both feeds up at the same time… (one cam is at .251 and other at .252 IP address
Edited single-specified_IP-cam from (https://docs.luxonis.com/projects/hardware/en/latest/pages/guides/getting-started-with-poe/)
import cv2
import depthai as dai
pipeline = dai.Pipeline()
camRgb = pipeline.createColorCamera()
xoutRgb = pipeline.createXLinkOut()
xoutRgb.setStreamName("rgb")
camRgb.preview.link(xoutRgb.input)
device_info = dai.DeviceInfo("192.168.88.251")
# device_info = depthai.DeviceInfo("14442C108144F1D000") # MXID
# device_info = depthai.DeviceInfo("3.3.3") # USB port name
with dai.Device(pipeline, device_info) as device:
qRgb = device.getOutputQueue(name="rgb", maxSize=4, blocking=False)
while True:
cv2.imshow("rgb", qRgb.get().getCvFrame())
if cv2.waitKey(1) == ord('q'):
break
however when I run the main.py of the gen2-multi-cam example (luxonis/depthai-experimentstree/master/gen2-multiple-devices) I get the following error:
NMAP command shows the cams IP are correct:
(VENV) jetty@jetty-desktop ~/Desktop nmap -T5 -sn 192.168.88.0-255
Starting Nmap 7.80 ( https://nmap.org ) at 2024-02-07 11:58 CET
Nmap scan report for _gateway (192.168.88.1)
Host is up (0.00065s latency).
Nmap scan report for 192.168.88.251
Host is up (0.0010s latency).
Nmap scan report for 192.168.88.252
Host is up (0.00090s latency).
Nmap scan report for jetty-desktop (192.168.88.253)
Host is up (0.00014s latency).
Nmap done: 256 IP addresses (4 hosts up) scanned in 1.78 seconds
Craftonix
It's better that you just go ahead and do that tiny test to see if you can replicate your problem (by fiddling with the DHCP Server settings and i.e. setting expiration to 5min).
Judging by this post: (https://superuser.com/questions/1217093/how-to-prevent-network-outages-caused-by-renewal-of-dhcp-leasing) …
…And other random bits on the net…..just troubleshoot m8, do the tweak, if it breaks the same way as after many hours, you've discovered your issue, if not, keep debugging.
Good luck!
Craftonix
I don't see a reason why they would have coded in a probable error like this one in regards to networking issues…this is Python, not Rust.
In essence, if the DHCP was assigning same IP that's all good and predictable, mine does as well, always a .213 and .220 IP allocation for the 2 cams. However, one, they are not static allocated via MAC address, and two, they don't have an infinite "license" allocated to their IP.
What I'm pointing out is, this "might" be the issue causing your disconnects. From what you've described the situation as, going from working for several hours to sudden drop, this is what came to mind.
Good luck with your debugging, still need to even get to your level with the connections issues I'm having ^^.
I was having some issues with my PoE…multi-cam config. After a lot of learning, one thing is somewhat popping out in my mind in regards to your problem. The DHCP server has an allocation time for the IP of the cameras (else they default to .222 ending domain). If said time allocation is exceeded, IP may be dropped or renewed (perhaps to new IP)
So you might want to check your DHCP settings for the following (the beneath is standard):
default-lease-time 600;
max-lease-time 7200;
jakaskerl
Have tried Penguinmon's suggestion with DNSMASQ and now both cameras get a proper IP….but also for some reason get picked up on the "default" .222 one.
Well regardless, when running a script…now it splits into one working and one "broken" camera…alternating when re-running script.
The output for error is:
Found 2 devices
[2023-10-07 18:35:45.699] [warning] Watchdog initial delay set to 60000ms
[2023-10-07 18:35:46.700] [warning] Watchdog initial delay set to 60000ms
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "main.py", line 31, in worker
device: dai.Device = stack.enter_context(dai.Device(openvino_version, dev_info, False))
depthai.XLinkWriteError: Couldn't write data to stream: '__bootloader' (X_LINK_ERROR)
=== Connected to 1944301061BA5B2700
>>> MXID: 1944301061BA5B2700
>>> Cameras: RGB LEFT RIGHT
>>> USB speed: UNKNOWN
Thoughts?
Edit: Both cameras work with depthai_demo.py when given option to select [0] or device [1]
Edit2: current NMAP shows this
nmap -T5 -sn 169.254.1.0-255
Starting Nmap 7.80 ( https://nmap.org ) at 2023-10-07 18:41 CEST
Nmap scan report for OrinNano (169.254.1.10)
Host is up (0.00010s latency).
Nmap scan report for 169.254.1.213
Host is up (0.0010s latency).
Nmap scan report for 169.254.1.220
Host is up (0.0011s latency).
Nmap scan report for 169.254.1.222
Host is up (0.00095s latency).
Nmap done: 256 IP addresses (4 hosts up) scanned in 1.77 seconds
Penguinmon
I've tried all the docs and the --experiments/gen2 to no avail….except one time a single cam magically worked….out of 40+ tries….
I'll be trying your dnsmasq tomorrow….maybe it works.
Penguinmon
If you could share how you managed to successfully connect to your multiple cams I'd appreciate it.
And if you can, share what your setup is ….I.e. from PC to some PoE Routers to cams…or different?
You can most easily paste the code here:
https://paste.modprog.de/
Penguinmon
I gave up with the custom DCHP server, it was half-working but still issues. Straight up ordered a Mikrotik hEX S Ethernet-Router to resolve the DCHP IP allocation…… before it arrives I might try your trick with dnsmasq on the Orin Nano to see if that gets me anywhere.
In short I'll try custom on-device (Orin) DCHP with this:
PC <--> PoE switch <--> PoE Cams
Then if failed I'll hopefully resolve it with the new router
PC <--> router <--> PoE switch <--> PoE Cams
jakaskerl
Is there a way to hard-code into bootloader a new / different I.P to give to my cameras….i.e. overwrite which generic IP it is set up with?
I understand now the reason the script detected 2 cameras is not via IP….but the MxID…
The only reference to the MAC / MxID I found is this C++ file:
GitHub MxID C++
The_Real_Enrico_Pallazzo
Additional debugging.
Tried NMAP on the network:
~/Desktop/Creating/CompVi/OAK/gen2-multiple-devices nmap -T5 -sn 169.254.1.0-255
Starting Nmap 7.93 ( https://nmap.org ) at 2023-10-02 17:44 CEST
Nmap scan report for DeepLearningRig (169.254.1.10)
Host is up (0.000040s latency).
Nmap scan report for 169.254.1.222
Host is up (0.00059s latency).
Nmap done: 256 IP addresses (2 hosts up) scanned in 2.15 seconds
Tried removing a single cam and pinging it…
~/Desktop/Creating/CompVi/OAK/gen2-multiple-devices ping 169.254.1.222 
│ OAK
PING 169.254.1.222 (169.254.1.222) 56(84) bytes of data.
64 bytes from 169.254.1.222: icmp_seq=1 ttl=64 time=0.167 ms
64 bytes from 169.254.1.222: icmp_seq=2 ttl=64 time=0.133 ms
64 bytes from 169.254.1.222: icmp_seq=3 ttl=64 time=0.130 ms
64 bytes from 169.254.1.222: icmp_seq=4 ttl=64 time=0.134 ms
--- 169.254.1.222 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3050ms
rtt min/avg/max/mdev = 0.130/0.141/0.167/0.015 ms
This was the result for both cams (inter-change-ably connected)…..so both work….but when both plugged into switch only one gets discovered by NMAP….which is weird…since the programming main.py from gen2-multiple-devices was stating it found both?
I really do think both might be trying to jump on the same IP default address for them…the 169.254.1.222
Tried some debugging with the solution of Penguinmon…and printed out something weird…
While it did continue to one device …somehow magically….and then failed (1 out of 30 tries so far)….I'm starting to think both cameras are defaulting to same I.P address? Is that possible?
Penguinmon
Could you share your code?
I've been trying to skirt-around using the DEPTHAI_WATCHDOG_INITIAL_DELAY=60000 DEPTHAI_BOOTUP_TIMEOUT=60000 python3 script.py flags you can pass on but that also didn't work.
Much appreciated.
Both cams have been tested with the depthai_demo.py over the local network and were found to be working all OK.
(PC <--> switch <--> PoE Cams)
I've also tired using the extra flags for testing if that might work (it was needed on my Orin Nano…but not on PC for regular test)… DEPTHAI_WATCHDOG_INITIAL_DELAY=60000 DEPTHAI_BOOTUP_TIMEOUT=60000 python3 script.py
So so far…am considering writing from scratch a basic connectivity script to access the cameras…since the Pipeline docs are kind of …non-existing.
Is there a newer version you folks will be releasing anytime soon?
Heya! (Jaka)
I've tried to run the depthai-experiments // gen2-multiple-devices to test the 2x S2 PoE cam setup and I get this error:
Found 2 devices
Exception in thread Thread-1 (worker):
Traceback (most recent call last):
File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
self.run()
File "/usr/lib/python3.11/threading.py", line 975, in run
self._target(*self._args, **self._kwargs)
File "/home/deeplearning/Desktop/Creating/CompVi/OAK/depthai-experiments/gen2-multiple-devices/multi-device-mobilenet.py", line 45, in worker
device = stack.enter_context(dai.Device(openvino_version, device_info, usb2_mode))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND
Seems like a weird error. All I want to do tbh….at this point….is to be able to access a camera and get the raw video….maybe in h.264 for then to do inference on the host machine….
The docs are surprisingly…well..vague…although there's some helpful images around, like the one on API Diagram of Host <-> Cam but there's no good docs on a raw easy way to connect and start messing with our (PoE) cams.
Any suggestions?
Just gotten 2x Oak-D S2 PoE cams and have an Orin Nano…..will be testing the connectivity / power-on directly for the direct connection, however have a PoE switch ready for later. From what I understand, you'd still need a form of splitter to run the 2 cat6a cables, one for each cam…to that single port on the Orin….so might as well get yourself a switch.
As to the capabilities of running inference on 6(max) videos, 3 each from every camera…. I'll have to test that…will post back here.
Good luck!
If I understood everything correctly, the camera itself is doing the encoding of the multi-video-raw-steams for us, which then only need to be decoded at the recipient point (in my case the Orin Nano)….also the encoding will be depending on the flags we use…or…well, we can probably access it with a custom script to give it instructions on which/how the 3-lens videos are to be encoded. Correct?
One last thing would be, will the AV1 codec be supported in the future on the cameras?
Edit: Apparently some hardware designs have been released (according to Quora):
The AOM has developed a reference design for AV1 hardware codecs that can be used by manufacturers to develop their own AV1-compatible hardware.
So am guessing it's more of an when?, not an if….although it's still very fresh….alongside the other 2 codec standards EVC and VVC
jakaskerl Thank you for your input. As I am still waiting for the 2x Oak-D S2 PoE cameras to arrive, am not sure what exactly consists of the "stream" of said camera.
Is the stream from a single camera combined on the camera (on the fly) from the 3 different lenses..into one? Or does it stream 3 perspectives simultaneously? Or is it 2 stereos combined and colored lens separately --> 2 videos?
Will have to do some calculations after your input…since I'm starting to think the 2 video cameras might overwhelm the little Orin Nano… (if it's 6 total videos…or 4 total….or just 2 videos)
Heya folks!
Looking to figure out how to do this dance of streaming video from the OAK-D S2 PoE camera via a PoE Switch directly to the Nvidia Jetson Orin Nano (DevKit).
I've been researching which codex/protocols are the fastest, but in-so-far only got that…well…they are kinda ancient.
What gives best 1080p or atleast 720p performance? Inference would mostly be run on the Orin Nano, with the exception (maybe) of hand-gesture-recognition on the camera itself.
I do plan to save the video for later training as well, the Orin has a fresh new Samsung 2TB 990 Pro as main (OS) drive.
How did you folks set up your cameras?
Edit: So far, while researching codecs, while the H.265 seems best for higher-resolution (4k) video…it does come at a cost. H.264 is "faster" but with a pay-off of more disk-space used.
Is there a data-spreadsheet to compare performance and benchmark it against other types/modes of streaming, i.e. the Streaming_API for ZED cameras?
