• DepthAI-v2Hardware

  • OAK-D Pro W PoE camera get intrinsics matrix/distortion coeffs in OpenCV style

I'm trying to get the OAK RGB camera's camera intrinsics matrix and the distortion coefficients vector to match the OpenCV style of a 3x3 matrix and a 1x5 vector, respectively. My goal is to use this information to compute pose estimation with ArUco markings. While I do get a 3x3 matrix for the camera intrinsics, I get [k1,k2,p1,p2,k3,k4,k5,k6,s1,s2,s3,s4,τx,τy] instead of just [k1,k2,p1,p2,k3] in the OpenCV style obtained from cv2.calibrateCamera(). Additionally, I have read that the distortion coefficients obtained from the OAK have undergone further rectification than just fisheye undistortion so the values may be wrong - I am getting highly erroneous pose estimation results.

My code is shown below:

`def get_OAK_rgb_matrices():
device = dai.Device()
rgb_camsocket = dai.CameraBoardSocket.CAM_A
calib_data = device.readCalibration()

# OpenCV reference: https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html
# depthai reference: https://docs.luxonis.com/projects/api/en/latest/references/python/#depthai.CalibrationHandler.getDistortionCoefficients

camera_intrinsics_matrix = np.array(calib_data.getCameraIntrinsics(rgb_camsocket))
distortion_coeffs = np.array(calib_data.getDistortionCoefficients(rgb_camsocket)[:5]).reshape((1,5))

return camera_intrinsics_matrix, distortion_coeffs`

jakaskerl

I obtain the camera intrinsics and distortion coeffs via this script:

import cv2
import numpy as np
import depthai as dai
import yaml
import argparse

def save_coefficients(mtx, dist, path):
    """ Save the camera matrix and the distortion coefficients to given path/file. """
    cv_file = cv2.FileStorage(path, cv2.FILE_STORAGE_WRITE)
    cv_file.write("K", mtx)
    cv_file.write("D", dist)
    # note you *release* you don't close() a FileStorage object
    cv_file.release()

def get_OAK_rgb_matrices():
    device = dai.Device()
    rgb_camsocket = dai.CameraBoardSocket.CAM_A
    calib_data = device.readCalibration()

    # OpenCV reference: https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html
    # depthai reference: https://docs.luxonis.com/projects/api/en/latest/references/python/#depthai.CalibrationHandler.getDistortionCoefficients

    camera_intrinsics_matrix = np.array(calib_data.getCameraIntrinsics(rgb_camsocket, 1920, 1080))
    distortion_coeffs = np.array(calib_data.getDistortionCoefficients(rgb_camsocket)[:5]).reshape((1,5))

    return camera_intrinsics_matrix, distortion_coeffs

def load_coefficients(path):
    """ Loads camera matrix and distortion coefficients. """
    # FILE_STORAGE_READ
    cv_file = cv2.FileStorage(path, cv2.FILE_STORAGE_READ)

    # note we also have to specify the type to retrieve other wise we only get a
    # FileNode object back instead of a matrix
    camera_matrix = cv_file.getNode("K").mat()
    dist_matrix = cv_file.getNode("D").mat()

    cv_file.release()
    return [camera_matrix, dist_matrix]


if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Get OAK Camera Calibration')
    parser.add_argument('--save_file', type=str, default="oak_calibration.yml", required=False, help='YML file to save OAK calibration matrices')

    args = parser.parse_args()
    mtx, dist = get_OAK_rgb_matrices()
    save_coefficients(mtx, dist, args.save_file)

The main issues I encounter when I try to get xyz readings are that I'm getting 5-6mm of xy deviation even at 10-15cm distances away from the reference aruco marker and the z reading is completely off by 10-20cm no matter where

    Additionally, is there a depthai function akin to rs2_deproject_pixel_to_point() from intel realsense?

    6 days later

    Hi, I was able to figure out the problem - I was already getting distortion rectified frames from the RGB camera and mistakenly undistorting again during reprojection. Thanks!

    One final thing - is the final camera intrinsic matrix for the undistorted virtual rectified camera the same as the one obtained from (device.readCalibration()).getCameraIntrinsics() or should I obtain it from some other method to sanity check?

      Hi @ZhifeiShen

      ZhifeiShen One final thing - is the final camera intrinsic matrix for the undistorted virtual rectified camera the same as the one obtained from (device.readCalibration()).getCameraIntrinsics() or should I obtain it from some other method to sanity check?

      Yes yes, should be the same.