Multiple Oak PoE Cameras Not Working

Sunny_Yeung

Hello,

I am new to using PoE cameras and connected two Oak PoE cameras (one Oak-D, one Oak-1) into a PoE switch. When I ran the gen-2-multiple-devices code on https://github.com/luxonis/depthai-experiments, I received the following output.

C:\Users\syeung\Documents\DepthAI\depthai-experiments-master\gen2-multiple-devices> python main.py
Found 2 devices
Traceback (most recent call last):
File "C:\Users\syeung\Documents\DepthAI\depthai-experiments-master\gen2-multiple-devices\main.py", line 45, in <module>
device = stack.enter_context(dai.Device(openvino_version, device_info, usb2_mode))
RuntimeError: Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND

The switch I used was the TRENDnet TPE-LG80.

A few things that were tried was:

Reflashing the cameras with their factory settings, then installing the newest network bootloader via the device_manager.py file found in https://github.com/luxonis/depthai-python/tree/main/utilities
Tried different switches
Tried different PoE cameras

None of these seemed to help the issue

Any insights with this issue would be greatly appreciated!

erik

Hi Sunny_Yeung ,
Could you please proivde a full MRE, and its details, so we help debug this issue asap?
Thanks, Erik

Sunny_Yeung

Hi erik,

Sorry for the late reply,

I couldn't get the pipeline graphing tool working, but I have all the other relevant information.
Attached to this post is the MRE, and screenshots of the device manager for the Oak 1 PoE and Oak D PoE devices.

The version of DepthAI being ran is 2.17.3.1, and the cameras are an Oak 1 PoE and an Oak D PoE.

import depthai as dai
import contextlib

def getPipeline(device_type):
    pipeline = dai.Pipeline()

    cam_rgb = pipeline.create(dai.node.ColorCamera)
    cam_rgb.setPreviewSize(600, 300)

    cam_rgb.setBoardSocket(dai.CameraBoardSocket.RGB)
    cam_rgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
    cam_rgb.setInterleaved(False)

    # Create output
    xout_rgb = pipeline.create(dai.node.XLinkOut)
    xout_rgb.setStreamName("rgb")
    cam_rgb.preview.link(xout_rgb.input)

    return pipeline

q_rgb_list = []

with contextlib.ExitStack() as stack:
    device_infos = dai.Device.getAllAvailableDevices()

    for device_info in device_infos:
        openvino_version = dai.OpenVINO.Version.VERSION_2021_4
        usb2_mode = False
        device = stack.enter_context(dai.Device(openvino_version, device_info, usb2_mode))
`

erik

Hi Sunny_Yeung ,
Could you try out the last option on the POE troubleshooting docs (longer delays). I have reproduced this issue, and it only happens if the device isn't able to connect within 30sec (default timeout). I have reported it to the FW team in hopes to fix this issue.
Thanks, Erik

Sunny_Yeung

erik

The troubleshooting guide tells me to run script.py, but I can't seem to find the file location. Where would the default install be?

I tried setting the initial delay and bootup timeout already, and it seems to be giving the same error.

Thank you

erik

Hi Sunny_Yeung ,
script.py is just any script you want, in your case main.py. Could you copy the full log that shows how you set both watchdogs and still got that error?
Thanks, Erik

Sunny_Yeung

Hi erik ,

Here is the corresponding terminal output

PS C:\Users\syeung\Documents\DepthAI\depthai-experiments-master\gen2-multiple-devices> set DEPTHAI_WATCHDOG_INITIAL_DELAY=60000 PS C:\Users\syeung\Documents\DepthAI\depthai-experiments-master\gen2-multiple-devices> set DEPTHAI_BOOTUP_TIMEOUT=60000 PS C:\Users\syeung\Documents\DepthAI\depthai-experiments-master\gen2-multiple-devices> python main.py Found 2 devices Traceback (most recent call last): File "C:\Users\syeung\Documents\DepthAI\depthai-experiments-master\gen2-multiple-devices\main.py", line 45, in <module> device = stack.enter_context(dai.Device(openvino_version, device_info, usb2_mode)) RuntimeError: Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND PS C:\Users\syeung\Documents\DepthAI\depthai-experiments-master\gen2-multiple-devices>
Thanks

erik

HI Sunny_Yeung ,
I can reproduce the issue; it's usually when you have mutliple POE cameras. I refactored the code (PR here) so it now connects to multiple devices in parallel, could you give it a try?
Thanks, Erik

Sunny_Yeung

Hi erik

I tried running the code and found that I still got the same error. Just for clarity's sake, I will post the logs and the code that I ran below.

Logs:
Found 2 devices Exception in thread Thread-1 (worker): Traceback (most recent call last): File "C:\Users\syeung\AppData\Local\Programs\Python\Python310\lib\threading.py", line 1016, in _bootstrap_inner self.run() File "C:\Users\syeung\AppData\Local\Programs\Python\Python310\lib\threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "c:\Users\syeung\Downloads\multipleCamerasTest.py", line 31, in worker device: dai.Device = stack.enter_context(dai.Device(openvino_version, dev_info, False)) RuntimeError: Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND

Code Ran:

import threading
import contextlib
import cv2
import time

def getPipeline(stereo):
    pipeline = dai.Pipeline()

    # Define a source - color camera
    cam_rgb = pipeline.create(dai.node.ColorCamera)
    # For the demo, just set a larger RGB preview size for OAK-D
    if stereo:
        cam_rgb.setPreviewSize(600, 300)
    else:
        cam_rgb.setPreviewSize(300, 300)
    cam_rgb.setBoardSocket(dai.CameraBoardSocket.RGB)
    cam_rgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
    cam_rgb.setInterleaved(False)

    # Create output
    xout_rgb = pipeline.create(dai.node.XLinkOut)
    xout_rgb.setStreamName("rgb")
    cam_rgb.preview.link(xout_rgb.input)
    return pipeline

def worker(dev_info, stack, dic):
    openvino_version = dai.OpenVINO.Version.VERSION_2021_4
    device: dai.Device = stack.enter_context(dai.Device(openvino_version, dev_info, False))

    # Note: currently on POE, DeviceInfo.getMxId() and Device.getMxId() are different!
    print("=== Connected to " + dev_info.getMxId())
    mxid = device.getMxId()
    cameras = device.getConnectedCameras()
    usb_speed = device.getUsbSpeed()
    print("   >>> MXID:", mxid)
    print("   >>> Cameras:", *[c.name for c in cameras])
    print("   >>> USB speed:", usb_speed.name)

    device.startPipeline(getPipeline(len(cameras)==3))
    dic["rgb-" + mxid] = device.getOutputQueue(name="rgb")

device_infos = dai.Device.getAllAvailableDevices()
print(f'Found {len(device_infos)} devices')

with contextlib.ExitStack() as stack:
    queues = {}
    threads = []
    for dev in device_infos:
        time.sleep(1) # Currently required due to XLink race issues
        thread = threading.Thread(target=worker, args=(dev, stack, queues))
        thread.start()
        threads.append(thread)

    for t in threads:
        t.join() # Wait for all threads to finish

    while True:
        for name, queue in queues.items():
            if queue.has():
                cv2.imshow(name, queue.get().getCvFrame())
        if cv2.waitKey(1) == ord('q'):
            break

print('Devices closed')`

erik

Hi Sunny_Yeung ,
Could you also try the new xlink_race_improvements branch on depthai-python, which should fix some xlink issues (that you are having), especially for the POE devices?
Thanks, Erik