X_LINK_ERROR - Couldn't read data from stream: 'video' in long term testing

stevefs

Hello
We are running 4 OAK-D USB
DepthAI version 2.19.1.0
OS - Windows 10
Cable length 1.5 m, 4 x 1 m cable plus 4 port USB 3.0 hub with 0.5 m cable
We are only using the RGB camera, frame rate is 20 fps, 5 fps x 4 cameras
The complete error is RuntimeError: Communication exception - possible device error/misconfiguration.
Original message 'Couldn't read data from stream: 'video' (X_LINK_ERROR)'
This error comes up in long term testing after running for 23 hours.
This is concerning as we plan to run in industrial production.
Can you suggest some code to catch the exception and initialize the pipeline again?
Have you heard of similar problems in long term testing with OAK-D USB? If so what was the solution?

Best Regards

erik

Hi stevefs ,
We have heard about such sporadic crashes, usually due to powering issues. Do you externally power each OAK-D via barrel jack connector? But if such errors do occur, some had success with try/catch, waiting a few seconds and reconnecting to the device (eg. by specifying its mxid). Thoughts?
Thanks, Erik

erik

Hi stevefs ,
I just checked, these cables are indeed rated for 1gbps, but it might be that the particular one was defective and/or didn't get a full contact with the connector, as all 8 lines need to have contact for the 1gbps to work (and only 4 lines to get 100mbps). Did you try reconnecting it?
Thanks, Erik

stevefs

erik Hi Erik
Current test has been running 26 hours.
We are not externally powering each OAK-D via the barrel jack connector, so we can try that.

We have code like this to create the pipeline.

mxid = device_info.getMxId()
device = stack.enter_context(dai.Device(getPipeline()))
q_rgb = device.getOutputQueue(name="video", maxSize=1, blocking=False)

def getPipeline():
    # Create pipeline
    pipeline = dai.Pipeline()

    # Define source and output
    camRgb = pipeline.create(dai.node.ColorCamera)
    xoutVideo = pipeline.create(dai.node.XLinkOut)

    xoutVideo.setStreamName("video")

    # Properties
    camRgb.setBoardSocket(dai.CameraBoardSocket.RGB)
    camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
    camRgb.setVideoSize(1920, 1080)

    xoutVideo.input.setBlocking(False)
    xoutVideo.input.setQueueSize(1)

    # Linking
    camRgb.video.link(xoutVideo.input)
    return pipeline

Should we be recreating the pipeline or?

stevefs

erik Thanks for checking on this. We did try reconnecting the cable which tells me that this particular cable was defective.

erik

Hi stevefs , we apologize for the inconvenience, we will thoroughly check the quality of these cables. Please inform us on support@luxonis.com so we can send you a replacement cable.
Thanks, Erik

erik

Hi stevefs , pipeline doesn't need to be recreated, you just need to pass it to the Device object again - to initialize the device and send the pipeline to it. Let us know how that goes.
THanks, Erik

stevefs

erik Hi Erik

Thanks for your help

        device_infos = dai.Device.getAllAvailableDevices()
        for device_info in device_infos:
            mxid = device_info.getMxId()
            device = stack.enter_context(dai.Device(getPipeline()))
            print("=== Connected to " + mxid)
            q_rgb = device.getOutputQueue(name="video", maxSize=1, blocking=False)
            q_rgb_list.append((q_rgb, index, M, maxWidth, maxHeight))

I took a look at the Device API. Given our code, what API call do I make to pass the pipeline to the Device object again to initialize the device and send the pipeline to it?

The code faults at inRgb = qRgb.get(), so I will put the try catch exception handling at that point.

stevefs

erik Hi Erik
I have tried some different approaches with no success, so I would appreciate some more guidance on what approaches you have tested and work.

Best Regards

theDesertMoon

Hopefully this helps. I had a similar issue a while ago and Erik helped me with similar advice. This is what I did.

needtoreinitialize starts as 'True'.

I catch the exceptions from my overarching loop and set the needtoreinitialize variable back to True. Then at the top of that loop I check to make sure I don't need to re-initialize the camera (PoE).

` found, device_info = dai.Device.getDeviceByMxId(self.OakID)

needtoreinitialize = True
device = qPreview = None
recoveryattempts = 0
while needtoreinitialize:
  try:
    device = dai.Device(pipeline, device_info)
    preview = device.getOutputQueue(name=rgb_str, maxSize=2, blocking=False)
    qDet = device.getOutputQueue(name=nns_str, maxSize=2, blocking=False)
    needtoreinitialize = False
    recoveryattempts = 0
  except Exception as exception:
    exc_type, exc_obj, exc_tb = sys.exc_info()
    print(str(exc_type)+" "+str(exc_tb.tb_lineno))
    self.oaklogging.error("<b>"+self.camlogident+"_Oakcam has encountered an error, reconnection attempt: "+str(recoveryattempts)+"</b>")
    recoveryattempts += 1`

stevefs

theDesertMoon Hi theDesertMoon
Thanks for posting.
Since it takes 24 hours to get an X Link error, I was planning to simulate by quickly unplugging and replugging the USB cable. In your experience were you able to recover from such a test or is it too extreme?
Best Regards

theDesertMoon

stevefs Nope, not too extreme at all. 🙂

About 5 seconds after I reconnected the camera to the switch, I was receiving video from the camera again.