Reveal more of the rectified / undistorted image with depthai v2

richardrl

Hi, in depthai v2, how do we show more of the rectified image?

code for showing the image from Oak1W in depthai v2 below:

https://www.dropbox.com/scl/fi/or9jm9aycjgov00gbq78j/oak_stock_show_video.py?rlkey=yaqywce1opawh6vrz3v3pg4bd&dl=0

What I want is to show more of the rectified image, even if it requires black pixels. For example in V3 here: https://docs.luxonis.com/software-v3/depthai/examples/camera/camera_multiple_outputs/

the letterbox'd setting is one such way.

Then, if I am able to get letterbox'd - how do I get proper intrinsics for the letterbox'd image?

Intrinsics are returned for a 3840 x 2160 image. They are not adaptive based on imagemanip nodes or anything else.

Since I assume the letterbox'ing does some scaling internally, I doubt these are correct for the undistorted image with letterbox processing.

richardrl

Example image. We have lost tons of the field of view, using the default camRgb.setMeshSource(dai.CameraProperties.WarpMeshSource.CALIBRATION)

undistortion

jakaskerl

richardrl camRgb.setCalibrationAlpha()

Thanks
Jaka

richardrl

What is the standard alpha setting to get all visible pixels in the image?

Does 1.0 correspond to the provided intrinsics?

richardrl

@jakaskerl To clarify, can you provide a way to get the exact pinhole intrinsics matrix after changing camRgb.setCalibrationAlpha? I believe the intrinsics from this:
calibData = device.readCalibration2()
intrinsics_matrix = np.array(calibData.getCameraIntrinsics(dai.CameraBoardSocket.CAM_A))

Is always constant and does not change when alpha is changed.

jakaskerl

richardrl
The intrinsics are meant for the undistorted image. If you change the alpha, these intrinsics will not be correct anymore.

Thanks,
Jaka

richardrl

But we need them. Ie to overlay 3D objects over the 2D image.

Are you guys calling opencv functions under the hood? The opencv functions return the matrices

If you can explain how to get the uncropped unmodified isp image and distortion parameters (which I assume goes into an opencv function) then i may be able to get what i need from opencv

By default the isp image is cropped and i cannot access the original full image for processing @jakaskerl

jakaskerl

richardrl
We will expose the alpha like it was done on the V2. It should automatically update the intrinsics with the new optimal matrix (kind of like it is done in opencv).

You need to use ISP output @ 12MP, the eeprom calibration should give you access to intrinsics and distortion coefficients which you can use to manually undistort it. luxonis/depthai-pythonblob/main/examples/ColorCamera/rgb_undistort.py (v2 code)

Thanks,
Jaka