• Hardware
  • Issues shutting down and restarting connection to PoE S-2 Camera

I am having a lot of trouble with the PoE Camera. Reconnecting to it after running my program.

I have a program based on the example provided with connects to the camera and saves the information is retrreives on a continuous basis. We have both USB and PoE OAK-D S-2 Cameras.

Everything works fine with the USB but the PoE version is giving problems.

I have a small network set up with the Camera, a PoE Switch and a connection to my PC. These are the only devices on this network and I disconnect the PC from other networks while trying to diagnose this. There is no DHCP and the PoE Camera reverts to it's default ip address of 169.254.1.222.

I power up all the equipment and start the program running, after a show while it connects to the camera and everything works fine. I can tell from the LEDs on the switch when this connection is working because they start flickering quickly. I then exit by clicking on the image and pressing 'q'. After a few seconds the python program stops and gives the message 'Exiting cleanly'. This should indicate that the context manager created by the line 'with dai.Device(pipeline, device_info, maxUsbSpeed=dai.UsbSpeed.HIGH) as device:' has exited cleanly.

However restarting this program (by running python main.py) fails with the following message

===========================================================

PS C:\code\cadds-visual-tyre-handler\source\recorder> python .\main.py

StereoDepth config options:

Left-Right check:   True

Extended disparity: False

Subpixel:           True

Median filtering:   MedianFilter.KERNEL_7x7

C:\code\cadds-visual-tyre-handler\source\recorder\main.py:39: DeprecationWarning: LEFT is deprecated, use CAM_B or address camera by name instead.

monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT)

C:\code\cadds-visual-tyre-handler\source\recorder\main.py:43: DeprecationWarning: RIGHT is deprecated, use CAM_C or address camera by name instead.

monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT)

C:\code\cadds-visual-tyre-handler\source\recorder\main.py:89: DeprecationWarning: RGB is deprecated, use CAM_A or address camera by name instead.

stereo.setDepthAlign(dai.CameraBoardSocket.RGB)

[2023-07-13 14:35:35.502] [depthai] [warning] Monitor thread (device: 19443010B1937F1300 [169.254.1.222]) - ping was missed, closing the device connection

Traceback (most recent call last):

File "C:\code\cadds-visual-tyre-handler\source\recorder\main.py", line 154, in <module>

with dai.Device(pipeline, device_info, maxUsbSpeed=dai.UsbSpeed.HIGH) as device:

depthai.XLinkWriteError: Couldn't write data to stream: '__bootloader' (X_LINK_ERROR)

PS C:\code\cadds-visual-tyre-handler\source\recorder>

=======================================================================================

The error message may sometimes be different, but this one is very common.

depthai version ==2.22.0.0

Operating System Windows 10

Python Version 3.10.7

=====main.py source code===================================

#!/usr/bin/env python3

from queue import Full

import time

from sys import maxsize

import cv2

import depthai as dai

import os

import shutil

import zipfile

import pickle

from multiprocessing import Queue

from tyre_detector.saver import Saver, SaveRecord, SaverCommand

import sys

import time

COLOR = True

SAVE_FREQUENCY = 0

lrcheck = True # Better handling for occlusions

extended = False # Closer-in minimum depth, disparity range is doubled

subpixel = True # Better accuracy for longer distance, fractional disparity 32-levels

# Options: MEDIAN_OFF, KERNEL_3x3, KERNEL_5x5, KERNEL_7x7

median = dai.StereoDepthProperties.MedianFilter.KERNEL_7x7

print("StereoDepth config options:")

print(" Left-Right check: ", lrcheck)

print(" Extended disparity:", extended)

print(" Subpixel: ", subpixel)

print(" Median filtering: ", median)

pipeline = dai.Pipeline()

monoLeft = pipeline.create(dai.node.MonoCamera)

monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)

monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT)

monoRight = pipeline.create(dai.node.MonoCamera)

monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)

monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT)

stereo = pipeline.createStereoDepth()

stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)

stereo.initialConfig.setMedianFilter(median)

stereo.setLeftRightCheck(lrcheck)

stereo.setExtendedDisparity(extended)

stereo.setSubpixel(subpixel)

monoLeft.out.link(stereo.left)

monoRight.out.link(stereo.right)

config = stereo.initialConfig.get()

config.postProcessing.speckleFilter.enable = False

config.postProcessing.speckleFilter.speckleRange = 50

config.postProcessing.temporalFilter.enable = True

config.postProcessing.spatialFilter.enable = True

config.postProcessing.spatialFilter.holeFillingRadius = 2

config.postProcessing.spatialFilter.numIterations = 1

config.postProcessing.thresholdFilter.minRange = 400

config.postProcessing.thresholdFilter.maxRange = 200000

config.postProcessing.decimationFilter.decimationFactor = 1

stereo.initialConfig.set(config)

xout_depth = pipeline.createXLinkOut()

xout_depth.setStreamName("depth")

stereo.depth.link(xout_depth.input)

xout_colorize = pipeline.createXLinkOut()

xout_colorize.setStreamName("colorize")

xout_rect_left = pipeline.createXLinkOut()

xout_rect_left.setStreamName("rectified_left")

xout_rect_right = pipeline.createXLinkOut()

xout_rect_right.setStreamName("rectified_right")

if COLOR:

camRgb = pipeline.create(dai.node.ColorCamera)

camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)

camRgb.setIspScale(1, 3)

camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.RGB)

camRgb.initialControl.setManualFocus(130)

stereo.setDepthAlign(dai.CameraBoardSocket.RGB)

camRgb.isp.link(xout_colorize.input)

else:

stereo.rectifiedRight.link(xout_colorize.input)

stereo.rectifiedLeft.link(xout_rect_left.input)

stereo.rectifiedRight.link(xout_rect_right.input)

class HostSync:

def __init__(self):

    self.arrays = {}

def add_msg(self, name, msg):

    if not name in self.arrays:

        self.arrays[name] = []

    # Add msg to array

    self.arrays[name].append({"msg": msg, "seq": msg.getSequenceNum()})

    synced = {}

    for name, arr in self.arrays.items():

        for i, obj in enumerate(arr):

            if msg.getSequenceNum() == obj["seq"]:

                synced[name] = obj["msg"]

                break

    # If there are 4 (all) synced msgs, remove all old msgs

    # and return synced msgs

    if len(synced) == 4:  # color, depth, rectified_left, rectified_right

        # Remove old msgs

        for name, arr in self.arrays.items():

            for i, obj in enumerate(arr):

                if obj["seq"] < msg.getSequenceNum():

                    arr.remove(obj)

                else:

                    break

        return synced

    return False

if name == 'main':

device_info = dai.DeviceInfo('169.254.1.222')

with dai.Device(pipeline, device_info, maxUsbSpeed=dai.UsbSpeed.HIGH) as device:

    device.setIrLaserDotProjectorBrightness(1200)

    qs = []

    qs.append(device.getOutputQueue("depth", maxSize=1, blocking=False))

    qs.append(device.getOutputQueue("colorize", maxSize=1, blocking=False))

    qs.append(device.getOutputQueue("rectified_left", maxSize=1, blocking=False))

    qs.append(device.getOutputQueue("rectified_right", maxSize=1, blocking=False))

    serial_no = device.getMxId()

    sync = HostSync()

    depth_vis, color = None, None

    image_queue = Queue()

    saver = Saver(image_queue, os.path.join(os.path.dirname(__file__), 'data'))

    saver.start()

    previous_save_time = 0.0

    any_output = False

    while True:

        for q in qs:

            new_msg = q.tryGet()

            if new_msg is not None:

                msgs = sync.add_msg(q.getName(), new_msg)

                if msgs:

                    depth = msgs["depth"].getFrame()

                    color = msgs["colorize"].getCvFrame()

                    cv2.imshow("color", color)

                    any_output = True

        now = time.time() \* 1000.0

        if any_output and now > previous_save_time + SAVE_FREQUENCY:

            previous_save_time = now

            try:

                image_queue.put(

                    SaverCommand(

                        save_record=SaveRecord(color, depth, now, serial_no, None, None, None, None),

                    exit_now=False,

                    max_telemetry_size=None),

                    block=False

                )

            except Full:

                print("Queue was reported full")

        key = cv2.waitKey(1)

        if key == ord("q"):

            print("Attempting to terminate saver")

            image_queue.put(

                SaverCommand(

                    save_record=None,

                    exit_now=True,

                    max_telemetry_size=None)

            )

            print("Waiting to join saver")

            saver.join()

            print("Saver joined")

            break

print("Waiting to shutdown")

time.sleep(5)

print("Exiting cleanly")

====================================================================================================================

Any help would be much appreciated

    Hi mark_squarecode
    The device needs some time to reboot as it takes watchdog a few second to kick in after disconnecting from the host. I assume the device connects fine if there is some time inbetween the separate runs? Can you confirm this is the case?
    Also, you don't need to set usbSpeed since you are running over ethernet.

    Thanks,
    Jaka

      8 days later

      jakaskerl

      Sorry for taking a while to get back to you.

      I have tried waiting for 60 seconds after stopping the program but that still does not work.

      I am on depthai version 2.22.0.0, os is Windows.

      The Error I get is

      [2023-07-21 14:30:00.791] [depthai] [warning] Monitor thread (device: 19443010B1937F1300 [169.254.1.222]) - ping was missed, closing the device connection

      Traceback (most recent call last):

      File "C:\code\cadds-visual-tyre-handler\source\recorder\main.py", line 153, in <module>

      with dai.Device(pipeline, device_info) as device:

      depthai.XLinkWriteError: Couldn't write data to stream: '__bootloader' (X_LINK_ERROR)

      Also if I don't set usbSpeed I get a warning

      C:\code\cadds-visual-tyre-handler\source\recorder\main.py:153: DeprecationWarning: Use constructor taking 'UsbSpeed' instead

      with dai.Device(pipeline, device_info) as device:

        Hi mark_squarecode
        Try flashing the latest bootloader using device_manager.py. Let me know if it solves the problem.

        Thanks,
        Jaka

        5 days later

        I have done this It upgraded from version 0.0.21 to version 0.0.26 but that appears to have made things worse. I can no longer get it to work at all (ie power cycling doesn't fix anything).

        The error I get now is
        [19443010B1937F1300] [169.254.1.222] [1690352930.317] [host] [warning] Using a custom watchdog value of 60000ms

        [2023-07-26 14:29:02.667] [depthai] [warning] Monitor thread (device: 19443010B1937F1300 [169.254.1.222]) - ping was missed, closing the device connection

        Traceback (most recent call last):

        File "C:\code\cadds-visual-tyre-handler\source\recorder\main.py", line 118, in <module>

        with dai.Device(pipeline, device_info, maxUsbSpeed=dai.UsbSpeed.HIGH) as device:

        depthai.XLinkWriteError: Couldn't write data to stream: '__bootloader' (X_LINK_ERROR)

        It took less than 60 seconds for this error to come up so it isn't the watchdog tripping.Any ideas? I can still connect the device manager to it (and ping it). One of my colleague suggested I should do a factory reset. To do this I ran this from the depthai-python source code repo. Latest main branch as of commit 236e7ec1b8ac2c7573d980f45efe90e9b3df812f

        • erik replied to this.

          Hi mark_squarecode ,
          This seems like a networking issue. Could you perhaps try with a different host computer / switch? As you have already tried with longer watchdog timeout (60s), it seems your network setup is a bit more "special".

            erik
            So today I have worked through quite a few hardware combinations as per this request.
            For our production device we are planning to deliver to our client is a
            + Microsoft Surface Go 3 Tablet running Windows 11
            + USB - C to Ethernet adaptor
            + Tp-Link POE desktop switch (TL-SF1005P)
            + OAK S-2 POE Camera

            All of this will be vehicle mounted, so we want as little gear as possible. Also we are focused on the PoE Camera instead of the USB Camera because Ethernet cables can be longer and clip in and unlike USB cables won't slip out.

            I didn't test on that tablet, instead I have two laptop I use for development. One is Windows only the other dual boots Linux Ubuntu and Windows. Both laptops are Dell Vostro 7590's

            I have three USB Network devices, a Comsol USB-A to ethernet adapter (it's a cheap one), a Bonelk USB-C to Ethernet adapter. A standard Dell Dock.

            I replaced the network cable between the PC/ USB to ethernet and the switch at the start of today's test. That did not appear to make a difference.

            No matter what combination of hardware I tried on Windows it did not work and always gave the same message as above.

            At one I tried to reset to the Factory Defaults in the device_mamanger.py program (under Windows). This did not work, it gave a message to the effect that it had tried to reboot the device but couldn't. The device then disappeared from the network. Eventually after power cycling it, it came back but the firmware had not been downgraded.

            I will frequently use 'ping 169.252.1.222' to check if it is on the network. I find that I frequently have to power cycle the switch and then the PoE camera by unplugging it to get it responding to pings.

            The good news is that under Linux I was able to get it to work particular the Dell Dock seemed to work better than the Comsol USB adapter. But they were both able to work.

            In the case of the Dell Dock I could confirm that I can start the program up. Then received images from the Camera. Then start it up again and receive more images from the Camera. However after this second successful run when I quit the program the Camera appeared to crash and I could no longer ping it. I was eventually able to get to to respond to pings by power cycling it.

            We are considering buying a second PoE Camera incase we simply have a defective unit. (We have a USB Camera as well that we can develop the software with).

            • erik replied to this.

              Hi mark_squarecode , what's the bootloader version on this device? Having it worked with Dell Dock means there was indeed some problem with hardware on the host side. But it's strange that you still have issues. How long after you quit the program did you try again? as camera needs about 10sec to restart/reboot, so you can connect to it again.

              6 days later

              Sorry for taking a while to get back, client wanted to borrow the Camera to work on the housing design.

              Got it back this morning and have managed to get a firm diagnosis.

              Currently at version 0.0.26 of the bootloader firmware.

              Everything works fine (with a few glitches) if I run it under Linux but the same program fails under Windows with the given message. (The program is this case being the code listed at the start of the original post in this thread).

              This behaviour indicates a bug which is specific to running your software under Windows. The specific error message indicates a ping is being timed out too aggressively when running under Windows.

              I have worked from home today and have been running the Camera off my home Ethernet. I can connect to the home Ethernet via Wifi or a USB Ethernet device which I can plug into either the PoE Switch which powers the Camera or my normal home Ethernet Switch. Which option I use doesn't matter Linux works 100%, Windows fails every time.

              Just to be clear it's the same computer which I can boot into either Windows or Linux. The version of the depthai library being used is 2.22.0.0 installed from pypi

              I have noticed some other issues.
              - If I isolate the Camera from the DHCP Server and try to get it to use the link local address (169.254.1.222) under Linux this is unreliable. This may have worked better when using the Dell Dock, but I don't have that with me today.
              - I use ping to test if the Camera is available. When trying under Windows and getting that error message, it is like the Camera would crash and not respond to pings. I had to power cycle it to get it back, sometimes multiple times.

              Hope that is clear, I am glad I now have a clearly reproducible failure. If there is anything I can do on our end to help fix this, eg getting more logs from the library I am happy to do that. We also have a client we are building a hardware and software package for who are pushing us for this to be delivered now and this problem is the only thing holding us back from doing that.

                mark_squarecode

                Acting on a hunch that it will have worked in older versions I tried that. Version 2.17.4.0 appears to connect to the camera occasionally under Windows and doesn't given the same missing ping error message.

                Notable using 2.17.4.0 under Windows it is taking much longer to connect to the camera 50 to 60 seconds, vs 16 to 24 seconds using 2.22.0.0 under Linux

                • erik replied to this.

                  Hi mark_squarecode ,
                  It would be great to reproduce this locally as well, but I tried with windows machine here, but there are some differences.
                  Regarding the older versions - do you perhaps know what is the latest (older) depthai version where it still works? As we can look into FW changes that occurred after that version to potentially pin down the connectivity issue.
                  Thank you for your help.
                  Erik

                  I can confirm with version 2.17.4.0 I can connect to the Camera and with version 2.18.0.0 I cannot connect and reliably get the same "missing ping error" message.

                  2.17.4.0 has it's own issues though so downgrading is not an option for us and we need to concentrate on getting the most recent version working for us.

                    Hi mark_squarecode
                    That's great, thanks. Could you please create an issue on depthai-core so our firmware team can look into it. Maybe link to this discussion as well so we have a bit more context.

                    Thanks again for your help,
                    Jaka

                      7 days later
                      2 months later

                      This issue has been solved by upgrading all the network equipment to gigabit. Gigabit PoE switch plus gigabit ethernet adaptor. I am guessing the ping packet is not received because the network link is overwhelmed with traffic.

                      Leaving comment here for any future users searching for this issue on google