Unable to ping multiple cameras

Kristoffer

Hi!

We are having issues when connecting multiple (4) cameras (PoE). Sometimes we are able to ping some of the cameras and often it varies which we are able to ping. Each camera has been configured to use a static IP address.

We have verified the PoE switch is able to deliver enough power, we have flashed the cameras to use the latest firmware, we have also tried multiple network cards.

Any ideas of what could cause the instability?

Camera: Oak 1 PoE
OS: Windows 11

Thanks!

erik

Hi @Kristoffer ,
We have seen that usually the culprit of such problem is the mask/gateway. Were those 2 fields configured correctly when flashing the static IP?

Kristoffer

Mask should be OK, but is the gateway also required?

jakaskerl

Hi Kristoffer
Could you post a screenshot of the device manager? Just so we can verify.

Thanks,
Jaka

Kristoffer

erik

Hi @Kristoffer ,
Mask should likely be 255.255.0.0, and if you want to access internet from the device you should set correct gateway as well.

Kristoffer

Hi,

255.255.255.0 (most common) should be fine for our configuration and we do not need access to the internet from the cameras. Anything else that could cause the problem?

Kristoffer

We are still having problems connecting to the cameras. We are running 4 x OAK1-PoE, Windows 11 and have configured each camera to use static IP.

Often we are unable to ping and which camera varies, sometimes we are able to ping, but not start the camera. Often we need to retry multiple times and eventually all 4 cameras will run. After struggling for a long time (not getting the cameras to run) a reboot of the host PC does something, but we still need to try for a long time to get the cameras running.

Are there some specific setting in Windows we need to change? Do you have examples of how you run multiple cameras and which hardware? Do we need to fill in the MAC address field and what value should we use and for what purpose?

Thanks!

jakaskerl

Hi Kristoffer
Could you try with this script:

https://discuss.luxonis.com/d/2241-sometimes-cant-connect-when-using-multiple-cameras/3

Let me know if it works.

Thanks,
Jaka

Kristoffer

Hi!

Yes, but it is still not working reliably. Sometimes the cameras with their assigned IP becomes invisible. Eg. A camera may be configured to 192.168.10.60 and will show up when pinging and it is also possible to connect normally (at least for a few tries). Then it suddenly becomes invisible from ping and dai.Device.getAllAvailableDevices(). However, when changing the IP of the network interface to be 169.254.1.50, all cameras show up again (all with the default IP as name, even if all cameras are configured to a unique static IP).

I believe there is something wrong with the implementation of device_info = dai.DeviceInfo(IP) and dai.Device(device_info). Connecting using MXID works more reliably.

jakaskerl

Hi Kristoffer
Are you running the latest bootloader 0.0.26? From the screenshot above it looks like you are using *24. Would you mind updating first, since many issues like that were fixed in the patch.

Thanks,
Jaka

Kristoffer

Hi!

I have now updated to the latest bootloader and also use a hybrid of your script to wait for all devices to be in state X_LINK_BOOTLOADER. I then try to connect using IP, but the problem remains. It may start up all four cameras without any problems the first time, but when the camera/script is shut down, it has a really hard time connecting to the cameras using the IP even if all the devices are found using dai.Device.getAllAvailableDevices() and waiting for all devices to be in the X_LINK_BOOTLOADER state. It has to retry multiple times before the cameras start running and eventually it will stop finding some cameras.

Kristoffer

We still have huge problems starting multiple devices (4) and it looks to me that it is something weird happening when trying to start the script over remote desktop. The devices do usually start much more easily when running the script locally, but we can not do that as some of our customers are located far away.

The devices are of the type OAK-1-PoE and we are running Windows 11.

We have tried the following:

Starting devices using threads
Starting devices sequentially
We have used both our own code and your example code for starting multiple devices
Using a separate network card for each device
Using a switch instead of separate network cards
Creating a routing table in Windows
Connecting using IP (static IP on device)
Connecting using MXID
Increasing the watchdog timeout to 60 seconds
Using the famous if device_info.state != dai.XLinkDeviceState.X_LINK_BOOTLOADER
Deactivating IPv6
Deactivating Wifi
Updating depthai
Updating the bootloader to the latest version (multiple times)

Can you do some testing and see if you get the same problems when connecting over remote desktop? The devices are configured with static IP's.

jakaskerl

Hi Kristoffer
Ok, thanks. Will try to debug this on Monday as I currently don't have access to a win11 machine.

jakaskerl

Hi Kristoffer
Do you have any MRE I could run locally that would reproduce the issue you are experiencing? I will try to run through remote desktop.

Thanks,
Jaka

Kristoffer

Hi!

The problems start already before the script is run it seems. And we are using Windows 10 apparently, not 11 as stated earlier.

The problem is mainly that Windows have problems finding the four devices between reboots. When running the device manager from the depthai utilities only 3 or maybe 2 devices will show. We are not even able to ping the missing devices.

Lets say we have device1, device2, device3 and device4. After booting only device1,device2 and device4 are showing (not device3). The next time we boot device1 could be the one missing, but device3 and the rest is found. Then suddenly after connecting to the devices only two will show.

jakaskerl

Hi Kristoffer
Can you confirm you are only getting this issue when running on windows?
UDP Device discovery (port 11491) is likely the cause here, any way you could check the network traffic?

Thanks,
Jaka

Kristoffer

The firewall is now turned completely off and we still experience the same problems. This is a problem happening when using Windows (10 in our case) and when running multiple devices (4 in our case).

Windows is not able to detect all cameras from the device_manager utility and this happens right after Windows has booted usually. Right now it is only detecting 2 devices, but sometimes it will detect all 4 or 3. Trying to connect to the devices usually makes it even worse when we shut them down and then try to reconnect after waiting for the devices for at least 10 seconds to boot up of course.

Are you able to test running at least 4 devices (OaK-1-PoE) running Windows 10? This has been going on for weeks now and nothing has changed. I installed Depthai 24 earlier this week without any change.

We really need this fixed and have even sent an email to support explaining how critical this is for us without even getting a reply.

Thanks

jakaskerl

Kristoffer Are you able to test running at least 4 devices (OaK-1-PoE) running Windows 10?

Yes, I am. On two separate machines.

I ran https://docs.luxonis.com/projects/api/en/latest/samples/mixed/multiple_devices/ 10 times with 10/10 success rate. I'm unsure what the issue is on your side. I have had problems with autodiscovery when devices shared the same IP, but you stated this is not the case for you. Do you have a DHCP server connected to the switch that would assign each device its own unique IP (I had them freshly unboxed, didn't update bootloader, didn't flash static IP, but I connected them via switch to DHCP)?

I did have issue when restarting the script directly after, but if I waited 10 seconds (giving devices time to properly reboot), the issue was gone.

Thanks,
Jaka

Kristoffer

HI!

Great! Then I at least know it should be possible which is a step forward. We have tried using both static IP and a configured DHCP server. Do you use the internal Windows DHCP server or a third party program?

May you also share which hardware components you are using on the machines and exactly which Windows version you are using? Maybe it is a hardware issue. I did try the prebuilt Luxonis Raspberry Pi image and were able to run the devices successfully multiple times in a row however.

I am also interested in how you installed Depthai and its dependencies. There is an MSI installer and Chocolatey is also mentioned as an alternative. We used Conda and downloaded Depthai using the following environment.yml file:

name: depthai_24
channels: # Channels are listed in prioritized order, do not change!
- conda-forge
- defaults
dependencies:
- python>=3.12,<3.13
- numpy
- opencv
# Pysimplegui and pillow required by DepthAI utilities (eg. device manager)
- pysimplegui
- pillow
- pip
- pip:
# Install prebuilt wheel (win_amd64) from:
- https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local/depthai/depthai-2.24.0.0.dev0+3f39768ab74394fa629ad6b4bfbf4d9a9f99eb6c-cp312-cp312-win_amd64.whl
variables:
DEPTHAI_WATCHDOG: 4500
DEPTHAI_PROTOCOL: tcpip

Thanks!

jakaskerl

Hi Kristoffer

Kristoffer DEPTHAI_PROTOCOL: tcpip

Don't use environment variables. Especially not the protocol. The device discovery is UDP, which might be blocked after using this setting.

Kristoffer Do you use the internal Windows DHCP server or a third party program?

DHCP on a router connected to the switch.

Kristoffer I am also interested in how you installed Depthai and its dependencies.

Installed via installer (so dependencies are installed). Then went to some custom directory, pulled depthai-python, did install_requirements.py inside examples/ and then inside /utilities as well. Ran python examples/mixed/multiple_devices.py.

EDIT: also used python3.11 since 3.12 is not currently supported unless on the python312 branch.

Thanks,
Jaka