• DepthAI

  • setResizeThumbnail crash with 4K ColorCamera

Hi everyone!

I'm trying to change my pipeline ColorCamera resolution from 1080P to 4K to have bigger images. Also, I also need to keep the full FOV of the sensor, so I opted in using an ImageManip node with a setResizeThumbnail. While this works properly with 1080P images, with 4K images the code crashes with these logs:

[system] [critical] Fatal error. Please report to developers. Log: 'ResourceLocker' '358'

[host] [warning] Device crashed, but no crash dump could be extracted.

I've noticed that everything works if I resize the image into a bigger one, for instance a resize.initialConfig.setResizeThumbnail(800, 800, 0, 0, 0) does not give any problem (but the size is then too big for my NN).

Can you provide any help please?

Here is the MRE using an Openvino neural network. I'm using Depthai 2.25.0 and an OAK-1 device.

import os
import depthai as dai
import blobconverter
import cv2

OUTPUT_SIZE = 384

pipeline = dai.Pipeline()

cam = pipeline.create(dai.node.ColorCamera)
cam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_4_K)
cam.setInterleaved(False)
cam.setPreviewSize(3840,2160) #4K

resize = pipeline.create(dai.node.ImageManip)
resize.initialConfig.setResizeThumbnail(OUTPUT_SIZE, OUTPUT_SIZE, 0, 0, 0) #<- This one causes crash, if OUTPUT_SIZE = 800 it works (but not size for NN)
resize.initialConfig.setKeepAspectRatio(False)
resize.setMaxOutputFrameSize(OUTPUT_SIZE * OUTPUT_SIZE * 3)
cam.preview.link(resize.inputImage)

x_out = pipeline.create(dai.node.XLinkOut)
x_out.setStreamName("image")
resize.out.link(x_out.input)

detection_nn = pipeline.create(dai.node.MobileNetDetectionNetwork)
detection_nn.setConfidenceThreshold(0.7)
detection_nn.setBlobPath(blobconverter.from_zoo(name="person-detection-0201"))
resize.out.link(detection_nn.input)
    
x_out1 = pipeline.create(dai.node.XLinkOut)
x_out1.setStreamName("det")
detection_nn.out.link(x_out1.input)

with dai.Device(pipeline) as device:
    qImg = device.getOutputQueue(name="image", maxSize=3, blocking=False)
    yoloQ = device.getOutputQueue(name="det", maxSize=3, blocking=False)
    cv2.namedWindow("Image", cv2.WINDOW_NORMAL)
    while True:
        img = qImg.get()
        gs = img.getCvFrame()
        dets = yoloQ.get().detections
        cv2.imshow("Image", gs)
        if cv2.waitKey(1) == ord('q'):  # Exit when 'q' is pressed
            break

Here is the pipeline graph

Thank you for your help!

Best regards

Simone

    Hi @sbellini
    You are essentially running out of memory when using large images.

    sbellini I'm trying to change my pipeline ColorCamera resolution from 1080P to 4K to have bigger images.

    Why do this? The resolution has no effect on the NN input.

    sbellini Also, I also need to keep the full FOV of the sensor, so I opted in using an ImageManip node with a setResizeThumbnail.

    You only get full FOV with isp output at 12MP. https://docs.luxonis.com/projects/api/en/latest/tutorials/maximize_fov/#maximizing-fov. You get the same FOV by using 1080p resolution.

    Thoughts?

          Dear Jaka,
          Thank you for your reply.

          jakaskerl You are essentially running out of memory when using large images.

          I don't get why it runs out of memory with a setResizeThumbnail(384,384) and not a setResizeThumbnail(800,800) (which would generate bigger images).

          jakaskerl Why do this? The resolution has no effect on the NN input.

          The MRE I provided is the first step of a bigger pipeline in which a script node receives object coordinates from a MobileNetDetectionNetwork and extracts the object from the original image, so extracting them from a 4K image would give me objects with 4 times the size.

          jakaskerl You only get full FOV with isp output at 12MP.

          Sorry, I probably expressed myself badly, I meant to say that I cannot use resize.initialConfig.setResize(384,384)and I need to use resize.initialConfig.setResizeThumbnail(384,384)because I need to keep the FOV of the 1080p (or 4K) image.
          Thank you again for your help

          Kind regards

          Simone

                  sbellini I don't get why it runs out of memory with a setResizeThumbnail(384,384) and not a setResizeThumbnail(800,800) (which would generate bigger images).

                  It's a problem of 4K image being copied to internal buffer (in memory) before being resized.

                  sbellini The MRE I provided is the first step of a bigger pipeline in which a script node receives object coordinates from a MobileNetDetectionNetwork and extracts the object from the original image, so extracting them from a 4K image would give me objects with 4 times the size.

                  Then do preview.setKeepAspectRatio(False) which will keep the FOV and downscale only.

                  Just like it's done here: https://docs.luxonis.com/projects/api/en/latest/samples/MobileNet/rgb_mobilenet_4k/

                  Thanks,
                  Jaka

                        Dear Jaka

                        Thank you for your quick reply!

                        jakaskerl Then do preview.setKeepAspectRatio(False) which will keep the FOV and downscale only.

                        Yes, I didn't think of that, I could do that (maybe with some tuning of the Neural network), thank you!

                        Just for my understanding, what's the reason why some setResizeThumbnail make the pipeline crash?

                        After some tests changing OUTPUT_SIZE here's what I've found:

                        OUTPUT_SIZE = 360, 361, 362, 363, 364 OK

                        OUTPUT_SIZE = 365 to 392 crash (Fatal error. Please report to developers. Log: 'ResourceLocker' '358')

                        OUTPUT_SIZE = 393, 394 OK

                        OUTPUT_SIZE = 395, 396, 397, 398 crash

                        OUTPUT_SIZE = 399, 400, 401, 402 OK

                        I don't get why some works and some not, I would expect them to take almost the same amount of memory (especially with very similar sizes like 398 and 399).

                        Thank you again for your help

                        Best regards

                        Simone

                          Hi @sbellini
                          Don't know for sure, I assume it has something to do with downscaling optimizations. Similar to "scaling has to be a factor of 32"; it because the algorithm is much more efficient on certain sizes due to how binary systems work.

                          Thanks,
                          Jaka