OAK-D-PRO different undistort results between stereoDepth and cv2.undistort

I tried tu use cv2.undistort() with the calibration results from oliver-batchelor/multical

The intrinsics matrices are very close, but the distortion coefficients are very different.

(On the left, the output of stereodepth.rectifiedLeft, using calibration parameters stored on the device. On the right, cv2.undistort() using calibration parameters from multical)

Hi @jakaskerl, thanks for the insight !

I need to be able to stereo rectify the color stereo pair, I would like to have the same output as the stereoDepth rectified streams, but in color

FYI I have two different setups with two FFC-4Ps:

• Two OV9282 monochrome cameras (wide)
• Two IMX296 with fisheye lenses

Trying to replicate the output of the stereoDepth rectified streams, I modified the get_mesh() function given here

def get_mesh(cam_name, resolution, fisheye, name_to_socket):
    left_socket = get_socket_from_name("left", name_to_socket)
    right_socket = get_socket_from_name("right", name_to_socket)

    calib = dai.Device().readCalibration()
    left_K = np.array(
        calib.getCameraIntrinsics(left_socket, resolution[0], resolution[1])
    )
    left_D = np.array(calib.getDistortionCoefficients(left_socket))
    # R1 = np.array(calib.getStereoLeftRectificationRotation())

    right_K = np.array(
        calib.getCameraIntrinsics(right_socket, resolution[0], resolution[1])
    )
    right_D = np.array(calib.getDistortionCoefficients(right_socket))
    # R2 = np.array(calib.getStereoRightRectificationRotation())

    extrinsics = np.array(calib.getCameraExtrinsics(left_socket, right_socket))

    if fisheye:
        R1, R2, P1, P2, Q = cv2.fisheye.stereoRectify(
            left_K,
            left_D,
            right_K,
            right_D,
            resolution,
            extrinsics[:3, :3],
            extrinsics[:3, 3],
            flags=0,
        )

        mapXL, mapYL = cv2.fisheye.initUndistortRectifyMap(
            left_K, left_D, R1, P1, resolution, cv2.CV_32FC1
        )
        mapXR, mapYR = cv2.fisheye.initUndistortRectifyMap(
            right_K, right_D, R2, P2, resolution, cv2.CV_32FC1
        )
    else:
        R1, R2, P1, P2, Q, _, _ = cv2.stereoRectify(
            left_K,
            left_D,
            right_K,
            right_D,
            resolution,
            extrinsics[:3, :3],
            extrinsics[:3, 3],
            flags=0,
        )

        mapXL, mapYL = cv2.initUndistortRectifyMap(
            left_K, left_D, R1, P1, resolution, cv2.CV_32FC1
        )
        mapXR, mapYR = cv2.initUndistortRectifyMap(
            right_K, right_D, R2, P2, resolution, cv2.CV_32FC1
        )

    mapX = mapXL if cam_name == "left" else mapXR
    mapY = mapYL if cam_name == "left" else mapYR

    meshCellSize = 16
    mesh0 = []
    # Creates subsampled mesh which will be loaded on to device to undistort the image
    for y in range(mapX.shape[0] + 1):  # iterating over height of the image
        if y % meshCellSize == 0:
            rowLeft = []
            for x in range(mapX.shape[1]):  # iterating over width of the image
                if x % meshCellSize == 0:
                    if y == mapX.shape[0] and x == mapX.shape[1]:
                        rowLeft.append(mapX[y - 1, x - 1])
                        rowLeft.append(mapY[y - 1, x - 1])
                    elif y == mapX.shape[0]:
                        rowLeft.append(mapX[y - 1, x])
                        rowLeft.append(mapY[y - 1, x])
                    elif x == mapX.shape[1]:
                        rowLeft.append(mapX[y, x - 1])
                        rowLeft.append(mapY[y, x - 1])
                    else:
                        rowLeft.append(mapX[y, x])
                        rowLeft.append(mapY[y, x])
            if (mapX.shape[1] % meshCellSize) % 2 != 0:
                rowLeft.append(0)
                rowLeft.append(0)

            mesh0.append(rowLeft)

    mesh0 = np.array(mesh0)
    meshWidth = mesh0.shape[1] // 2
    meshHeight = mesh0.shape[0]
    mesh0.resize(meshWidth * meshHeight, 2)

    mesh = list(map(tuple, mesh0))

    return mesh, meshWidth, meshHeight

I think I am on the right track, as on the OV9282 setup I get very close results:

From stereoDepth node

With get_mesh()

You can see the image is a little bit more zoomed in coming out of the stereoDepth node, and the epilines scores are better (0.03% vs 0.1%)

This would be acceptable to me.

However, when checking the same things with the IMX296 color cameras with fisheye lenses, I get:

From stereoDepth node:

From get_mesh()

As you can see, both images are weirdly translated, and have large distortion, especially the image on the right.

The two setups have been calibrated with the same procedure, but they differ in two ways

• The IMX296 sensors output a 1440x1080 image that I resize to (1280, 800) using an ImageManip node, while the OV9282 are natively (1280,800)

    def create_manipRescale(pipeline, resolution):
        manipRescale = pipeline.createImageManip()
        manipRescale.initialConfig.setResize(resolution[0], resolution[1])
        manipRescale.setMaxOutputFrameSize(resolution[0] * resolution[1] * 3)
        return manipRescale
  

• The lenses mounted on the IMX296 have a larger FOV than the OV9282W (below examples without undistort)

  

  

A few other questions while I am at it:

• Can https://docs.luxonis.com/projects/api/en/latest/samples/Camera/camera_undistort/#undistort-camera-stream be used to do what I want to do ? Is it mature or still experimental ? Right now I get black images with the IMX296 sensors when trying to get any output (video, preview, isp) from this node
• Would it be possible to directly get the meshes computed by the stereoDepth node ? It would be perfect, as I would just need to instantiate a StereoDepth node, get the mesh and set it as the warp mesh of my rectify ImageManip node.

Sorry, that's a lot, thanks for your time!

Antoine