Hi, I need to undistort multiple wide angle color cameras on a OAK-FFC device. I can see there is a script for [stereo camera calibration](https://docs.luxonis.com/en/latest/pages/calibration/), but I can't find one for rgb camera calibration like [this](https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html). The [example](https://github.com/luxonis/depthai-python/blob/develop/examples/ColorCamera/rgb_undistort.py) from depthai for camera undistortion during runtime only addresses one camera use case. Let's assume all the cameras have different intrinsics and distortion coefficients. What is the workflow to calibrate each of them, flash them on the device and use them for ImageManip in the pipeline? Thanks Lincoln

Hi @"lincolnxlw"#p10018 , I just created docs PR on how to do that: https://github.com/luxonis/depthai-python/pull/861 I hope this helps!

I also tried to run the calibration for OAK-1-LITE-W with this [json](https://github.com/luxonis/depthai-boards/blob/main/boards/OAK-1-LITE-W.json) file, and it returned the same error expecting stereo_config.

Hi @"erik"#p10200, I just created an issue: https://github.com/luxonis/depthai/issues/1081 Thanks Lincoln

Hi @"lincolnxlw"#p10363 , could you try using the latest depthai `develop` branch?

Hi @"erik"#p10433 I will! I usually use [depthai-library](https://hub.docker.com/r/luxonis/depthai-library) from docker for testing. But if I need to use depthai develop branch, does that mean I need to modify this [docker file](https://github.com/luxonis/depthai-python/blob/main/ci/Dockerfile) and build with that branch? Or I can find the tag in depthai-library that already built with the develop branch?

The workflow to undistort multiple color cameras on devices like OAK-FFC 3P/4P

lincolnxlw

Hi,

I need to undistort multiple wide angle color cameras on a OAK-FFC device. I can see there is a script for stereo camera calibration, but I can't find one for rgb camera calibration like this. The example from depthai for camera undistortion during runtime only addresses one camera use case.

Let's assume all the cameras have different intrinsics and distortion coefficients. What is the workflow to calibrate each of them, flash them on the device and use them for ImageManip in the pipeline?

Thanks

Lincoln

erik

Hi lincolnxlw ,
I just created docs PR on how to do that: luxonis/depthai-python861
I hope this helps!

lincolnxlw

erik

Thanks for the reply! As far as I can see, the PR doesn't include how to do the calibration when there is no calibration file yet, how to flash the calibration to the device and how to do this process for multiple cameras on the same device. Will those being added to the PR later?

Thanks

Lincoln

erik

Hi lincolnxlw ,
Sorry about that, the PR only shows how to undistort frames (after the device is calibrated). Calibration docs here: https://docs.luxonis.com/projects/hardware/en/latest/pages/guides/calibration/

lincolnxlw

Hi erik,

Thanks for the response. Because I only use my cameras on the board as color cameras, I created a json with only one color camera for using OAK-FFC 3P just to do a quick test.

{

"board_config":

{

    "name": "OAK-FFC-3P-1-CAM",

    "revision": "R1M0E1",

    "cameras":{

        "CAM_A": {

            "name": "rgb",

            "hfov": 127,

            "type": "color"

        }

    }

}

}

After running the calibration script with

python calibrate.py -s 3 -db -brd /config/OAK-FFC-3P-1-CAM.json

It gives me error regarding no stereo config found:

File "/depthai/calibrate.py", line 699, in capture_images_sync

leftStereo = self.board_config['cameras'][self.board_config['stereo_config']['left_cam']]['name']

KeyError: 'stereo_config'

But in my case I am not using the cameras as depth cameras, is it possible to disable this error and only run rgb camera calibration for all the cameras connected to the board?

Thanks

Lincoln

lincolnxlw

I also tried to run the calibration for OAK-1-LITE-W with this json file, and it returned the same error expecting stereo_config.

erik

Hi lincolnxlw ,
Could you create an issue on depthai github repo, so you get notified when this is fixed?

lincolnxlw

Hi erik,

I just created an issue:

luxonis/depthai1081

Thanks

Lincoln

lincolnxlw

Hi erik

Thanks to your suggestion, the folks from the repo suggested a temporary fix. And I am able to do the calibration process. When I try to undistort the camera with Camera node according to the example from your PR, I am able to get a full undistorted image

However, there are some features are not currently available in Camera node yet, so when I try to undistort with ColorCamera node following this example. I am only getting the "zoom in" version of the undistorted image.

What I need to modify in order to get the full undistorted image using ColorCamera node?

Thanks

Lincoln

erik

Hi lincolnxlw , could you try using the latest depthai develop branch?

lincolnxlw

Hi erik

I will! I usually use depthai-library from docker for testing. But if I need to use depthai develop branch, does that mean I need to modify this docker file and build with that branch? Or I can find the tag in depthai-library that already built with the develop branch?

erik

Hi lincolnxlw ,
I am not very familiar with docker, but I believe you can just use this tag: https://hub.docker.com/r/luxonis/depthai-library/tags?page=1&name=946a1843dbf9cf86a9c819505d03e0b456400604
Which is linked to latest develop of depthai-python library.

lincolnxlw

Hi erik

Thanks! How do I know which branch the tag is linked to? I was looking into this.

jakaskerl

Hi lincolnxlw
The tag name is the same as the commit id. See https://github.com/luxonis/depthai-python/commits/develop.

Thanks,
Jaka

lincolnxlw

Hi jakaskerl

I see. That's very helpful. Thanks!

Lincoln

lincolnxlw

Hi @erik,

Just want to give you an update with the "zoom-in" issue. Change to the latest develop branch didn't make a difference. But it can be fixed by adding a scaling factor to the camera intrinsics array while creating the maps for undistortion.

Here is the updated function getMesh function for this example

def getMesh(calibData: dai.CalibrationHandler, ispSize, focal_len_scaling_factor: float = 1.0):
    M1 = np.array(calibData.getCameraIntrinsics(camSocket, ispSize[0], ispSize[1]))
    d1 = np.array(calibData.getDistortionCoefficients(camSocket))
    R1 = np.identity(3)
    M1_new = M1.copy()
    M1_new[0][0] *= focal_len_scaling_factor
    M1_new[1][1] *= focal_len_scaling_factor
    mapX, mapY = cv2.initUndistortRectifyMap(M1, d1, R1, M1_new, ispSize, cv2.CV_32FC1)

    meshCellSize = 16
    mesh0 = []
    # Creates subsampled mesh which will be loaded on to device to undistort the image
    for y in range(mapX.shape[0] + 1): # iterating over height of the image
        if y % meshCellSize == 0:
            rowLeft = []
            for x in range(mapX.shape[1]): # iterating over width of the image
                if x % meshCellSize == 0:
                    if y == mapX.shape[0] and x == mapX.shape[1]:
                        rowLeft.append(mapX[y - 1, x - 1])
                        rowLeft.append(mapY[y - 1, x - 1])
                    elif y == mapX.shape[0]:
                        rowLeft.append(mapX[y - 1, x])
                        rowLeft.append(mapY[y - 1, x])
                    elif x == mapX.shape[1]:
                        rowLeft.append(mapX[y, x - 1])
                        rowLeft.append(mapY[y, x - 1])
                    else:
                        rowLeft.append(mapX[y, x])
                        rowLeft.append(mapY[y, x])
            if (mapX.shape[1] % meshCellSize) % 2 != 0:
                rowLeft.append(0)
                rowLeft.append(0)

            mesh0.append(rowLeft)

    mesh0 = np.array(mesh0)
    meshWidth = mesh0.shape[1] // 2
    meshHeight = mesh0.shape[0]
    mesh0.resize(meshWidth * meshHeight, 2)

    mesh = list(map(tuple, mesh0))

    return mesh, meshWidth, meshHeight

And here are the undistorted images with scaling factor=1.0, 0.8 and 0.6. Smaller the factor, the more "zoom out" the image is.

Thanks

Lincoln