Segmentation Fault in Multiple Camera code

John-R

Hello everyone.
I am currently running a code on an Intel NUC device that works 20% of the time, but fails 80% of the time without warning. I based this code on: Gen 2 multiple devices

Hardware

Intel NUC
2 Luxonis cameras
- Connected via USB
- Powerbrick plugged

Code

Here is the code I use to instantiate a pipeline:

def getPipeline(device_type):
    # Start defining a pipeline
    pipeline = dai.Pipeline()

    # Define a source - color camera
    cam_rgb = pipeline.createColorCamera()

    cam_rgb.setPreviewSize(1920, 1080)

    cam_rgb.setBoardSocket(dai.CameraBoardSocket.RGB)
    cam_rgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
    cam_rgb.setInterleaved(False)

    # Create output

    xout_rgb = pipeline.createXLinkOut()
    xout_rgb.setStreamName("rgb")
    cam_rgb.preview.link(xout_rgb.input)

    return pipeline

I use the following code to find which cameras are connected, and then attempt to connect to them. Note I have added a retry block. Since we also kept on getting could not write to stream errors.

device_infos = dai.Device.getAllAvailableDevices()
if len(device_infos) < 2:
    print("Not enough devices found, re-running to get all devices... (Required is 2 cameras)")
    while len(device_infos) < 2:
        device_infos = dai.Device.getAllAvailableDevices()   
else:
    print("Found", len(device_infos), "devices:")

# print out devices
for device_info in device_infos:
    print("\t" + device_info.getMxId())

for device_info in device_infos:
    print("=== Connecting to " + device_info.getMxId())

    openvino_version = dai.OpenVINO.Version.VERSION_2021_4
    usb2_mode = False
    device = None # instantiate device variable

    # Attempt to instantiate the device
    for i in range(0,10):
        try:
            device = stack.enter_context(dai.Device(openvino_version, device_info, usb2_mode))
            break
        except Exception as e:
            print(e)
            print("Retrying...")
            continue
    else:
        print("Maximum retries exceeded.")
        continue

    mxid = device.getMxId()
    cameras = device.getConnectedCameras()
    usb_speed = device.getUsbSpeed()
    print("   Connected!")
    print("   >>> MXID:", mxid)
    print("   >>> Cameras:", *[c.name for c in cameras])
    print("   >>> USB speed:", usb_speed.name)

    device_type = "unknown"
    if   len(cameras) == 1: device_type = "OAK-1"
    elif len(cameras) == 3: device_type = "OAK-D"
    # If USB speed is UNKNOWN, assume it's a POE device
    if usb_speed == dai.UsbSpeed.UNKNOWN: device_type += "-POE"
    # Get a customized pipeline based on identified device type
    pipeline = getPipeline(device_type)
    print("   >>> Loading pipeline for:", device_type)
    device.startPipeline(pipeline)
    # Output queue will be used to get the rgb frames from the output defined above
    q_rgb = device.getOutputQueue(name="rgb", maxSize=4, blocking=False)
    stream_name = str(device_info.getMxId()).replace(".", "_") + "-" + device_type + ".jpeg"

    q_rgb_list.append((q_rgb, stream_name))

Here is the code that constantly loops to get images every second or so.

while True:
    for q_rgb, stream_name in q_rgb_list:
        in_rgb = q_rgb.tryGet()
        if in_rgb is not None:
            cv2.imwrite(stream_name, in_rgb.getCvFrame())
            print(stream_name + " image saved!")
        time.sleep(5)

    ## some codes for uploading to Azure Storage
    time.sleep(60)

Errors

The errors start happening when and after the code attempts to instantiate the pipeline in the device = getPipeline( line. They usually happen when attempting to connect to the 2nd device. But they can also happen suddenly within the while True loop. The errors are sudden, and tend to be different every time:

These exceptions could be caught by an except statement, and will not stop the retry loop:

Couldn't write data to stream: '__rpc_main' (X_LINK_ERROR)

These exceptions could NOT be addressed by the try-except statements, and will immediately terminate the program:

*** stack smashing detected ***: terminated
Segmentation fault
Segmentation fault (Address not mapped to object[(nil)])
Aborted
Aborted (Signal sent by tkill() 385200 0)

We hope you could help us in troubleshooting these. Thank you very much.

erik

Hello John-R ,
we apologize for the inconvenience, could you provide additional information so we can debug this easier; what's the USB speed (being printed to console)? What depthai version are you using (latest?)? Are you using an USB hub or are both cameras connected directly to the NUC? Do you wait about 10seconds before trying to run this demo again? As that's about how long it takes for the watchdog to kick in and dispose resources on the device.
Thanks, Erik

John-R

Hello!

We managed to fix this issue by changing the wires. We were using some 2.5m long USB Cables since the cameras were on the ceiling, while our NUC device was on the table. When we plugged the original wires in, as well as elevated the device with some shelves, the segmentation error disappeared. The code has been running without a hitch for about a week now.

Do you have any recommendation as to the maximum length of the wire this setup can handle? Or are there settings we can adjust on the device to prevent this problem? Our team found the following recently: Forcing USB2 mode. Would this be an appropriate fix? Are there other things we must consider to prevent this kind of error in the future?

Thank you very much.

erik

Hello John-R , with USB3, it's about 1.5m max length, above that you might have issues. Forcing to USB2 mode would be a great solution, as you can go to about _5m. For longer distances, you might want to check out OAK POE, as POE can work up to 100m. Someone on discord also mentioned that active USB hubs work great for this issue as well.
Thanks, Erik