Sometimes can't connect when using multiple cameras

Penguinmon

Hi, When I try connecting to multiple cameras I can usually connect to them all, but sometimes I can't connect to a few of them. I've noticed it happening more often when I quickly restart my program. I'm using OAK-1 PoE cameras, and I still have this issue even after updating the bootloader and depthai versions.

I am running each device in its own thread, and from what I can tell, sometimes some of the threads just get stuck trying to connect to a camera. I made sure to include:
os.environ["DEPTHAI_WATCHDOG_INITIAL_DELAY"] = "60000"

os.environ["DEPTHAI_BOOTUP_TIMEOUT"] = "60000"

jakaskerl

Hi Penguinmon

Penguinmon I've noticed it happening more often when I quickly restart my program.

How quickly are we talking about? Do you notice any improvements with changed watchdog delay as opposed to the standard configuration?
Could you add a MRE of how you are connecting to the cameras on different threads?

Thanks,
Jaka

Penguinmon

Quickly like if I stop the program, wait a few seconds, then restart it.

I just tried testing it in one of my projects using three cameras:
I ran it twenty times using docker compose restart, and there were only two times where I couldn't connect to all of the cameras (one different camera each time)
I didn't notice any difference with the watchdog delay in this case. I just made a habit of putting it in my projects because there are some cases where I need it.

I'll see if I can get an MRE.

Penguinmon

I think checking to see if the device is in the bootloader state first fixed it.
# Check every second for new devices
while True:
device_infos = dai.Device.getAllAvailableDevices()
for device_info in device_infos:
if device_info.state != dai.XLinkDeviceState.X_LINK_BOOTLOADER:
continue
# Start thread here
sleep(1)

The_Real_Enrico_Pallazzo

Penguinmon
Could you share your code?

I've been trying to skirt-around using the DEPTHAI_WATCHDOG_INITIAL_DELAY=60000 DEPTHAI_BOOTUP_TIMEOUT=60000 python3 script.py flags you can pass on but that also didn't work.

Much appreciated.

Penguinmon

The_Real_Enrico_Pallazzo
Do you still need help? Were you just wanting to see how I connect to the devices?

The_Real_Enrico_Pallazzo

Penguinmon
If you could share how you managed to successfully connect to your multiple cams I'd appreciate it.

And if you can, share what your setup is ….I.e. from PC to some PoE Routers to cams…or different?

You can most easily paste the code here:
https://paste.modprog.de/

Penguinmon

The_Real_Enrico_Pallazzo
I'm not sure if it's the best way to do it, but this is how I've been connecting to multiple devices: https://paste.modprog.de/adSu

There are also some depthai examples that I looked at previously:
https://docs.luxonis.com/projects/api/en/latest/samples/mixed/multiple_devices/#multiple-devices
https://github.com/luxonis/depthai-experiments/blob/master/gen2-multiple-devices/main.py

Setups:
Pi (running a dhcp server) <--> Unmanaged PoE switch <--> PoE Cams
PC <--> Managed PoE switch <--> PoE Cams

The_Real_Enrico_Pallazzo

Penguinmon
I've tried all the docs and the --experiments/gen2 to no avail….except one time a single cam magically worked….out of 40+ tries….

I'll be trying your dnsmasq tomorrow….maybe it works.