Incorrect Frame size for Intrinsics

crs287

I am wanting to use the Intrinsics and Distortion parameters for a given camera in my OAK-D PoE sensor. I need these in order to extrinsic sensor calibration between this camera and my LiDAR. However while working on this, I noticed that the intrinsic matrix parameters didn't seem to line up with the actual image size that I got from the following code:

import cv2
import numpy as np
import time
from datetime import datetime, timezone

import depthai as dai


# OAK-D Thread
def oak_d_thread(pipeline, data_dir):
    # 1. Capture "monotonic now" (time since host PC boot, unaffected by system clock changes)
    monotonic_ref = dai.Clock.now()        # Returns datetime.timedelta
    # 2. Capture "UTC now" at the same instant
    utc_ref = datetime.now(timezone.utc) # A naive datetime in UTC
    # 3. Compute offset:  (UTC time) - (monotonic time)
    offset = utc_ref - monotonic_ref

    frame_cnt = 0
    with dai.Device(pipeline) as device:
        q_rgb = device.getOutputQueue("rgb")
        while True:
            in_rgb = q_rgb.tryGet()
            if in_rgb is not None:
                if frame_cnt == 0:
                    frame_cnt += 1
                    continue
                frame = in_rgb.getCvFrame()
                cam_deltatime = in_rgb.getTimestamp()
                cam_time = cam_deltatime + offset
                print(f"Camera: Timestamp {cam_time}")

                # Save Camera Image
                cv2.imwrite(f"{data_dir}/oakd_cam/img_{cam_time.timestamp()}.jpg", frame)
                frame_cnt += 1

            # if frame_cnt > 50:
            #     break

# Main Program
if __name__ == "__main__":
    data_dir = '/lidar_camera_calibration/data/'
    
    pipeline = dai.Pipeline()
    cam_rgb = pipeline.createColorCamera()
    cam_rgb.setFps(2)
    cam_rgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
    cam_rgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1200_P)
    cam_rgb.setPreviewSize(1024, 512)
    cam_rgb.setInterleaved(False)
    cam_rgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.RGB)
    xout_rgb = pipeline.createXLinkOut()
    xout_rgb.setStreamName("rgb")
    cam_rgb.preview.link(xout_rgb.input)

    # Threads for both devices
    oak_d_thread(pipeline, data_dir)

    print("Finished processing Camera!")

crs287

Also this is the code I used to get the intrinsic and distortion parameters:

import depthai as dai

pipeline = dai.Pipeline()
cam_rgb = pipeline.createColorCamera()
cam_rgb.setFps(2)
cam_rgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
cam_rgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1200_P)
cam_rgb.setPreviewSize(1024, 512)
cam_rgb.setInterleaved(False)
cam_rgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.RGB)
xout_rgb = pipeline.createXLinkOut()
xout_rgb.setStreamName("rgb")
cam_rgb.preview.link(xout_rgb.input)

with dai.Device(pipeline) as device:
    calib_data = device.readCalibration2()
    # print(calib_data)
    
    ## Get Extrinsics between cameras (Camera A is the origin in this case)
    extrinsics_AtoB = calib_data.getCameraExtrinsics(dai.CameraBoardSocket.CAM_A,dai.CameraBoardSocket.CAM_B) # [cm]
    print("Extrinsics from A to B:",extrinsics_AtoB)

    ## Get Intrinsics for a given camera [fx,0,cx; 0,fy,cy; 0,0,1]
    intrinsics = calib_data.getCameraIntrinsics(dai.CameraBoardSocket.CAM_A) # [pixels]
    print("Camera A intrinsics:",intrinsics)

    ## Distortion Coefficients: "k1","k2","p1","p2","k3","k4","k5","k6","s1","s2","s3","s4","τx","τy"
    dist_coeff = calib_data.getDistortionCoefficients(dai.CameraBoardSocket.CAM_A)
    print("Camera A distortion coefficients:",dist_coeff)

crs287

I think I found my answer. I need to instead get my intrinsics for my specified camera image size: e.g. calibdata.getCameraIntrinsics(dai.CameraBoardSocket.CAMA, 1024, 512)

My new question is then, is the image that I saved from my above code already undistorted? Or do I need to use these distortion parameters to undistort it?

jakaskerl

crs287 My new question is then, is the image that I saved from my above code already undistorted? Or do I need to use these distortion parameters to undistort it?

ColorCamera node doesn't automatically undistort frames. They are still distorted.

crs287 Should I instead be saving the "isp" image since that is supposedly undistorted? Then I can use that with my intrinsic and extrinsic calibration code (assuming the distortion parameters are meant for the "isp" image)?

Use isp, yes. I think max resolution for you is 1200p anyway so it shouldn't make a difference though.

Thanks,
Jaka

crs287

Should I instead be saving the "isp" image since that is supposedly undistorted? Then I can use that with my intrinsic and extrinsic calibration code (assuming the distortion parameters are meant for the "isp" image)?