Please help with crop parameters

Thor

I want to be able to specify a crop region, a rectangle of a certain size positioned anywhere in the frame
I thought I using setCropRect, but this
cfg.setCropRect(0.2,0.2,0.5,0.5)
gives this error:
[184430107119670F00] [1.1] [16.913] [ColorCamera(0)] [warning] Crop rectangle out of bounds: x,y (384, 216), w,h (1920, 1080). Using center crop

Any help?

jakaskerl

Hi @Thor
MRE please. It works as expected with https://docs.luxonis.com/projects/api/en/latest/samples/StereoDepth/depth_crop_control/.

Thanks,
Jaka

Thor

Your example uses 2 nodes. I want to send an ImageManipConfig message directly to cam.inputConfig, it should be possible as described in the docs
try this MRE:
#!/usr/bin/env python3
import depthai as dai
pipeline = dai.Pipeline()
cam = pipeline.create(dai.node.ColorCamera)
configIn = pipeline.create(dai.node.XLinkIn)
configIn.setStreamName('config')
xout = pipeline.create(dai.node.XLinkOut)
xout.setStreamName('out')
cam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
configIn.out.link(cam.inputConfig)
cam.video.link(xout.input)
with dai.Device(pipeline) as device:
while True:
q = device.getOutputQueue(xout.getStreamName(), maxSize=4, blocking=False)
f = q.get()
configQueue = device.getInputQueue(configIn.getStreamName())
cfg = dai.ImageManipConfig()
cfg.setCropRect(0.2,0.2,0.5,0.5)
configQueue.send(cfg)

I get:
[184430107119670F00] [1.1] [1.834] [ColorCamera(0)] [warning] Crop rectangle out of bounds: x,y (384, 216), w,h (1920, 1080). Using center crop

jakaskerl

Hi @Thor
The only thing you can use it for is moving the crop when using setVideoSize or setPreviewSize which will normally centercrop the image. This is so you can move the frame around.

#!/usr/bin/env python3
import depthai as dai
import cv2

# Create a pipeline
pipeline = dai.Pipeline()

# Setup Color Camera
cam = pipeline.create(dai.node.ColorCamera)
cam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
cam.setInterleaved(False)
cam.setVideoSize(640, 480)


# Setup input stream for configurations
configIn = pipeline.create(dai.node.XLinkIn)
configIn.setStreamName('config')
configIn.out.link(cam.inputConfig)

# Setup output stream for video
xout = pipeline.create(dai.node.XLinkOut)
xout.setStreamName('out')
cam.video.link(xout.input)

# Initialize the device and start processing frames
with dai.Device(pipeline) as device:
    i = 0
    while True:
        i+=0.01
        if i >= 0.5:
            i = 0
        # Get frames from the output queue
        q = device.getOutputQueue(name='out', maxSize=4, blocking=False)
        f = q.get()  # Fetch the frame (not used further in this snippet)

        # Prepare and send a crop configuration
        configQueue = device.getInputQueue(name='config')
        cfg = dai.ImageManipConfig()
        cfg.setCropRect(i, 0.2, 0.3, 0.3)
        configQueue.send(cfg)

        # Get the frame from the output queue
        frame = q.get().getCvFrame()
        cv2.imshow('video', frame)

        # Break the loop when 'q' is pressed
        if cv2.waitKey(1) == ord('q'):
            break

Thanks,
Jaka

Thor

@jakaskerl @erik But it only works with the video out. IT DOES NOT WORK WITH THE PREVIEW OUT (try it out on your snippet), so it is useless for our needs

Any chance you can "fix" this shortly?

Thor

@jakaskerl @erik to better specify:

This doesn't work
cam.setVideoSize(1920, 1080)
cam.setPreviewSize(640, 480)
cam.preview.link(xout.input)
this works, even removing setPreviewSize
cam.setVideoSize(640, 480)
cam.setPreviewSize(640, 480)
cam.preview.link(xout.input)

So it seems that video and preview must be set to the same size. This leaves the isp out to be used for full res out. Can live with that for now

Thor

@jakaskerl @erik Just to make sure I don't misunderstand
if I set the sensor to THE_1080P, and setVideoSize to 640,640 => sensor frame will be center cropped to 640x640, with no resizing nor image distortion. If I don't touch Preview parameters, the Preview out will be = to Video out.
Correct?
I want to make sure I get no image distortion

jakaskerl

Hi @Thor

Thor if I set the sensor to THE_1080P, and setVideoSize to 640,640 => sensor frame will be center cropped to 640x640, with no resizing nor image distortion. If I don't touch Preview parameters, the Preview out will be = to Video out.
Correct?

Yep.

Thor I want to make sure I get no image distortion

If you don't undistort the image, you will get raw, distorted image. Video and preview have nothing to do with that.

Thanks,
Jaka

erik

Hi @Thor ,

setVideoSize to 640,640 => sensor frame will be center cropped to 640x640,

Just to clarify, it will try to downscale to your desired video size. If aspect ratio doesn't match, it will crop some of the pixel to match specifically your requested size.
ColorCamera doesn't do undistortion, but Camera node does have it integrated:
https://docs.luxonis.com/projects/api/en/latest/samples/Camera/camera_undistort
So to get undistorted image, you should rather use Camera node.
I hope this helps.
Thanks, Erik

Thor

@Erik. Thanks for the explanation
So, there is no way to truly crop a frame (without downscaling) at the video/preview level, while keeping the ISP out = selected sensor resolution?

Also, you told me once not to use the Camera node, and use the ColorCamera instead. I don't remember exactly why, but there was something not working. I guess now the Camera node is the preferred option?

erik

Hi @Thor ,
To crop on sensor level, you can use setSensorCrop
https://docs.luxonis.com/projects/api/en/latest/references/cpp/?highlight=setSensorCrop#_CPPv4N3dai4node11ColorCamera13setSensorCropEff

You can only select one corner, not both (hardware limitation). You could also just use ImageManip to fully crop, but that takes computational resources.

WRT Camera / ColorCamera nodes - depends on the feature needed, some things still aren't supported for Camera node.

Thor

@Erik Thank you Erik. We tried the crop at the sensor level and cropping at a later stage with ImageManip.
Using one additional imageManip brings down frame rate, also if I remember correctly uses shaves/cmx we had to use an NN model with reduced shaves. Cropping at the sensor level has similar issues, we also tried a number of alternatives but in the end we got nowhere.
For our application we need a full frame view to show and use as a reference where the user can move a rectangle around to define a ROI for the preview out and using isp scale to control downsizing to include a wider region if needed in the 640x413 preview out to NN
Something you might consider to support for the next version, hopefully
For now we'll stick with the limitations we've got, I started this thread because the documentation about setCropRect didn't match our findings and I just wanted to better understand how it works. Thanks for all the support

wendbb

@erik @jakaskerl
I am also having issue with using setCropRect
I want to set input to yolo nn with input size of 512 by 512 but specific region of the input image.
float xmin = 0.6f; float ymin = 0.144444444f; float xmax = 1.0f; float ymax = 0.855555556f; imageManip->initialConfig.setCropRect(xmin, ymin, xmax, ymax);
but the setup fail without log.
this image shows how I want the image to nn get cropped.

could you comment on this?

wendbb

I had to setPreview() before setCropRect call. this is resolved for me. thank you

erik

That's great to hear, thanks for the update🙂