Wondering if anyone has any advice on how to achieve better calibration results?

There is visible distortion after running through the calibrate.py script. See included image. Is this just a natural limitation of the Wide FOV cameras?

I am using a CharuCo target board with squares that are 3.75 cm in side length, so that is one area for potential improvement, but looking for advice/solutions.

Thank you!

You can see the distortion along the sides. This is after a "successful" calibration that was uploaded to the device EEPROM.

    henry0987
    Can you show some results from the calibration procedure (dataset images, coverage)? The cameras are Wide and are more prone to distortion, but this image shows awful distortion. What code are you using for undistortion?

    Thanks,
    Jaka

        jakaskerl

        Yes, here are the dataset images and coverage: https://drive.google.com/drive/folders/1-Dfev8BYhDRghQULCtfdSrdIJXzf2YiD?usp=sharing

        For undistortion, I am using Foxglove which I believe is using OpenCV under the hood.

        Two things:

        • I have performed the calibration several times, and for some reason, the distortion always seems to be most noticeable in the bottom left.
        • I have performed the calibration on multiple Oak-D W Pros, and I get similar results, meaning likely not a hardware issue. Most likely I am somehow performing the calibration incorrectly, but I get good coverage and the calibration is "successful" (low reprojection and epipolar error), so really not sure what the issue is.

            jakaskerl

            Hi @jakaskerl, yes. This code does seem to help with the distortion, although it creates a circular artifact in the image which is also an issue (image included).

            I do need to integrate my Oak camera into ROS (i.e. calibrated using CameraInfo messages), so do you have any advice on how I can fix that? Is there perhaps another calibration script that is designed specifically to work with Wide FOV cameras? Also, were you able to view my calibrationd dataset and coverage? Should I be performing the calibration differently?

            Thanks.

            @Luxonis-Adam @erik

              • erik

              • Post #7

              Hi @henry0987 , are you using the latest depthai version? I've seen these circles in ~ 2023 quite a lot, IIRC it was because of camera model that was used to generate mesh for undistortion. I believe that after we skipped some parameters (of camera distortion model) we didn't get these strange artifacts anymore.

              Hi @erik, the circles come from when I viewed the image with the code provided via the gist a few comments up.

              My bigger issue is that when I calibrate the camera via calibrate.py and view in Foxglove (which uses the ROS CameraInfo message to calibrate), I still see noticeable distortion in the image. I am using the Oak-D Pro W PoE.

              Has this happened before? Have you had success calibrating this camera before and seeing a good undistortion? Wondering what I can do differently, or if there is a specific procedure/branch I can use? Or, is there just some limit to how well this Wide FOV camera can be calibrated?

              Hi @erik, thank you. When viewing with the Camera node, I now see the image with the correct undistortion.

              Follow up: is there source code for how this undistortion is actually done? For my use case, I would just like to get the raw images from the device, then do this undistortion procedure off-device. Is this possible?

              Just to follow up, the answer is yes. Solution: simply use OpenCV's undistort() function.

              #!/usr/bin/env python3

import depthai as dai
import cv2
import numpy as np

# Create the pipeline and Camera object
pipeline = dai.Pipeline()
cam_rgb = pipeline.create(dai.node.Camera)

cam_rgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
cam_rgb.setSize((1280, 800))

# Create the XLinkOut object
video_rgb_out_x = pipeline.create(dai.node.XLinkOut)
video_rgb_out_x.setStreamName("isp_rgb")
cam_rgb.isp.link(video_rgb_out_x.input)

with dai.Device(pipeline) as device:
    video_rgb_queue = device.getOutputQueue(name="isp_rgb", maxSize=1, blocking=False)

    calibData = device.readCalibration()
    intrinsics_rgb = np.array(calibData.getCameraIntrinsics(dai.CameraBoardSocket.CAM_A))
    distortion_rgb = np.array(calibData.getDistortionCoefficients(dai.CameraBoardSocket.CAM_A))
    
    while True:
        if video_rgb_queue.has():

            # convert to opencv mat
            cv_mat = video_rgb_queue.get().getCvFrame()
            cv2.imshow("original isp_rgb", cv_mat)

            # undistort
            cv_mat_undistorted = cv2.undistort(cv_mat, intrinsics_rgb, distortion_rgb)
            cv2.imshow("undistorted", cv_mat_undistorted)

        if cv2.waitKey(1) == ord('q'):
            break

              For some reason, Foxglove is not performing the undistortion correctly. No idea why. The solution is to simply do it yourself.

              Note: I also passed in the CameraInfo topic to ROS2's image_proc package, and it also correctly undistorted the image. This means the issue appears to be Foxglove's undistortion procedure, whatever that is. If anyone has any information on why this is happening, I would be very interested to know.