Cropping Still Image - unexpected behavior

ChristophGG

Hi all,
I am just starting to experiment with some features of depthai. In order to speed up transfer and reduce computation time, i would like to do some image manipulation such as cropping on the still image. But the cropping leads to unexpected sizes or even crashes. Here is what I've tried. Am I getting something wrong?

Version: 2.17.3.1
OAK-D Lite

Thanks,
Christoph

import depthai as dai
import cv2

pipeline = dai.Pipeline()

cam = pipeline.createColorCamera()
cam.setBoardSocket(dai.CameraBoardSocket.RGB)
cam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_13_MP)

cam.setIspScale(1,4)

w0,h0 = cam.getStillWidth(), cam.getStillHeight()

still_manip = pipeline.create(dai.node.ImageManip)
still_manip.setMaxOutputFrameSize(3*w0*h0)
still_manip.inputConfig.setWaitForMessage(True)
cam.still.link(still_manip.inputImage)

xout_still = pipeline.createXLinkOut()
xout_still.setStreamName("still")
still_manip.out.link(xout_still.input)

control_in = pipeline.create(dai.node.XLinkIn)
control_in.setStreamName('control')
control_in.out.link(cam.inputControl)

still_config_in = pipeline.create(dai.node.XLinkIn)
still_config_in.setStreamName('still_config')
still_config_in.out.link(still_manip.inputConfig)


device = dai.Device(pipeline)

stream_still = device.getOutputQueue(name="still", maxSize=1, blocking=False)
control_queue = device.getInputQueue('control')
still_config_queue = device.getInputQueue('still_config')

print("orig size")
print(w0,h0)                    # --> 1048 780

#rect = (0.0, 0.0, 1, 1)         # --> crop size: 1024 780 (why not original size?)
#rect = (0.0, 0.0, 0.5, 0.5)    # --> crop size: 512 390, but not from 0,0 but some arbitrary point.
                                # Also image is stretched by a factor of (approximately?) 2
rect = (0.0, 0.0, 0.8, 0.8)     # crashes at frame = stream_still.get().getCvFrame() with
                                # color.simd_helpers.hpp:108: error: (-215:Assertion failed) sz.width % 2 == 0 && sz.height % 3 == 0
print(rect)

cfg = dai.ImageManipConfig()
cfg.setCropRect(*rect)
still_config_queue.send(cfg)

ctrl = dai.CameraControl()
ctrl.setCaptureStill(True)
control_queue.send(ctrl)

frame = stream_still.get().getCvFrame()
print("crop size")
print(*frame.shape[1::-1])

cv2.imshow("frame", frame)
cv2.waitKey(0)

erik

Hi ChristophGG ,
I believe that's because YUV420 requires sz.width % 2 == 0 && sz.height % 3 == 0, so it crashes when you try to convert it to bgr frame on the host. If you try with normal OAK-D which uses 12MP (IMX378) color sensor, it won't crash, as it will results in frame of size 794x608.

You can also add cfg.setFrameType(dai.RawImgFrame.Type.BGR888p) to your code in order for ImageManip to convert YUV420 -> BGR, which will make the code workable again (output size 819x624). Thoughts on such approach?
Thanks, Erik

erik

I also wanted to mention you did a great job of creating an error report, thanks for that!

ChristophGG

Hi erik

Great! Thanks for the quick reply. I changed the format to BGR as you suggested, and that solved the previous crash. However, there still seem to be some other parity related issue. I tried cropping to different sizes and it turns out that for those sizes that are divisible by 4 it works fine, but if they have remainder 2 the image is somehow scaled by a factor of two first (I wanted to attach an image to demonstrate, but i get an error there as well...). If they are odd, i get a crash

[1844301021FF8D1200] [2.1.2.4] [514.200] [system] [critical] Fatal error. Please report to developers. Log: 'ResourceLocker' '358'

any thoughts on that?

For example with a crop rect = (0,0,1,y) for the following y
1.0: 780%4 = 0 (ok)
0.95: 741%4 = 1 (crash)
0.9: 702%4=2 (doubled)
0.8: 624%4=0 (ok)

erik

Hi ChristophGG ,
Interesting, I haven't yet seen this issue. I think it would be best to opne an Issue at depthai-core so our firmware engineers can look into it.
Thanks, Erik