Hey! I recently discovered that the factory calibration of my OAK-D Lite camera had a principal point about 40px to the left, which I find to be excessive, so I tried calibrating my camera using calibrate.py from github. Unfortunately I can't get the epipolar error for the RGB camera below 1. It's usually at about 4-5. I'm using the recommended aruco board pattern on my 27 inch monitor, which is pretty flat and high-fidelity and have tried many configurations of shooting the shots, aiming to fill the camera frame and look from many directions at the monitor. Still, the error is quite high. Interestingly, the error for the left and right camera is quite low, so I can't really figure out how to get it to perform better. Has anyone had similar problems? What else can I do? Thank you! `` `INTRINSIC CALIBRATION` `Reprojection error of rgb: 5.034222658668403` `` `INTRINSIC CALIBRATION` `Reprojection error of left: 0.15107010555635988` `` `INTRINSIC CALIBRATION` `Reprojection error of right: 0.16599770162246127` `` `Reprojection error is 0.1601330748860618` `` `Original intrinsics ....` `L [[457.10106012 0. 328.89856635]` ` [ 0. 456.91268888 225.50472026]` ` [ 0. 0. 1. ]]` `R: [[458.20856262 0. 307.18176888]` ` [ 0. 458.06629417 234.9234571 ]` ` [ 0. 0. 1. ]]` `Intrinsics from the getOptimalNewCameraMatrix/Original ....` `L: [[458.20856262 0. 307.18176888]` ` [ 0. 458.06629417 234.9234571 ]` ` [ 0. 0. 1. ]]` `R: [[458.20856262 0. 307.18176888]` ` [ 0. 458.06629417 234.9234571 ]` ` [ 0. 0. 1. ]]` `Average Epipolar Error is : 0.14776542348769103` `` `Reprojection error is 0.42267471973251036` `` `Original intrinsics ....` `L [[458.20856262 0. 307.18176888]` ` [ 0. 458.06629417 174.9234571 ]` ` [ 0. 0. 1. ]]` `R: [[498.7213913 0. 306.59984722]` ` [ 0. 497.02051934 180.05594615]` ` [ 0. 0. 1. ]]` `Intrinsics from the getOptimalNewCameraMatrix/Original ....` `L: [[498.7213913 0. 306.59984722]` ` [ 0. 497.02051934 180.05594615]` ` [ 0. 0. 1. ]]` `R: [[498.7213913 0. 306.59984722]` ` [ 0. 497.02051934 180.05594615]` ` [ 0. 0. 1. ]]` `Average Epipolar Error is : 0.3633410286610644` `Reprojection error threshold -> 3` `rgb Reprojection Error: 5.034223` `/home/evg/depthai/calibrate.py:1029: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)` ` calibration_handler.setDistortionCoefficients(stringToCam[camera], cam_info['dist_coeff'])` `Reprojection error threshold -> 1.0` `left Reprojection Error: 0.151070` `/home/evg/depthai/calibrate.py:1068: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)` ` calibration_handler.setCameraExtrinsics(stringToCam[camera], stringToCam[cam_info['extrinsics']['to_cam']], cam_info['extrinsics']['rotation_matrix'], cam_info['extrinsics']['translation'], specTranslation)` `Reprojection error threshold -> 1.0` `right Reprojection Error: 0.165998` `['high Reprojection Error']` `py: DONE.`

When calibrating using the old set of scripts you'd get a box overlayed on your image showing a suggestion of how to capture a photo of the board next, in the the newer scripts you instead get an occupancy image showing areas where you have markers detected so that you can get good coverage. This is an example of the polygons you'd see using the old scripts: [upl-image-preview url=https://discuss.luxonis.com/assets/files/2024-02-13/1707863926-568743-image.png] If you see those polygons over your images you're using the older set of scripts, running the following will get you up to date: ``` git clone https://github.com/luxonis/depthai.git cd depthai git submodule update --init --recursive python3 install_requirements.py ```

Hi @"jakaskerl"#1112, I'm currently utilizing the new calibration script. Here's the full command I'm executing: `python3 calibrate.py -s 2.3 -ms 1.7 -nx 11 -ny 8 -brd bw1098obc -ih` However, I'm encountering several issues: 1. During calibration, when it calculates the epipolar error, either the color camera or the mono camera's error tends to be significant. 2. Even if I succeed in overriding the **EEPROM**, the calibration data doesn't seem accurate (in my estimation). 3. There are slight offsets in the camera matrix data, as shown below (This is the factory calibration): ``` "intrinsicMatrix": [ [2978.385009765625, 0.0, 1843.81689453125], [0.0, 2978.385009765625, 1079.8861083984375], [0.0, 0.0, 1.0] ], "height": 2160, "width": 3840, "specHfovDeg": 68.7938003540039 ``` The **cx** value of **1843.81689453125** should be closer to **1920**, and the same applies to **cy**. I presume the distortion coefficients should correct for that offset. However, I'm uncertain about the reason behind such a significant offset. Any clarification on this matter would be greatly appreciated.

Hi @"vkmNuller"#2162 1. Yes there should be a visible difference between epipolar error of color and mono camera, which is due to transformation from RGB -> mono camera, which needs to be done, since for calibration of intrinsic parameters only mono frames can be used. And with the transformation, the image properties (brightness mostly) changes. And also the image in the RGB sensor has smaller FOV and image looks much closer than in mono stereo cameras, which leads to cameras being more dependent on pixel detection error. For that reason in user calibration script there are values of epipolar error of rgb and mono cameras. 2. Yes, it is normal for intrinsic parameters to be different than the values of normal intrinsic matrix (which should be around size_of_image/2), which happens due to corrections of distortion coefficients. In case you are not sure if the intrinsic matrix is correct, it can be tested by computing the depth from camera ans checking if depth is correct. For more information, you can send us the dataset you captured and we can look into it. Thanks, Jaka

High Epipolar Error when calibrating OAK-D Lite camera

EvgeniyPopov

Hey! I recently discovered that the factory calibration of my OAK-D Lite camera had a principal point about 40px to the left, which I find to be excessive, so I tried calibrating my camera using calibrate.py from github. Unfortunately I can't get the epipolar error for the RGB camera below 1. It's usually at about 4-5. I'm using the recommended aruco board pattern on my 27 inch monitor, which is pretty flat and high-fidelity and have tried many configurations of shooting the shots, aiming to fill the camera frame and look from many directions at the monitor. Still, the error is quite high. Interestingly, the error for the left and right camera is quite low, so I can't really figure out how to get it to perform better.

Has anyone had similar problems? What else can I do?

Thank you!

<------------Calibrating rgb ------------>

INTRINSIC CALIBRATION

Reprojection error of rgb: 5.034222658668403

<------------Calibrating left ------------>

INTRINSIC CALIBRATION

Reprojection error of left: 0.15107010555635988

<------------Calibrating right ------------>

INTRINSIC CALIBRATION

Reprojection error of right: 0.16599770162246127

<-------------Extrinsics calibration of left and right ------------>

Reprojection error is 0.1601330748860618

<-------------Epipolar error of left and right ------------>

Original intrinsics ....

L [[457.10106012 0. 328.89856635]

[ 0. 456.91268888 225.50472026]

[ 0. 0. 1. ]]

R: [[458.20856262 0. 307.18176888]

[ 0. 458.06629417 234.9234571 ]

[ 0. 0. 1. ]]

Intrinsics from the getOptimalNewCameraMatrix/Original ....

L: [[458.20856262 0. 307.18176888]

[ 0. 458.06629417 234.9234571 ]

[ 0. 0. 1. ]]

R: [[458.20856262 0. 307.18176888]

[ 0. 458.06629417 234.9234571 ]

[ 0. 0. 1. ]]

Average Epipolar Error is : 0.14776542348769103

<-------------Extrinsics calibration of right and rgb ------------>

Reprojection error is 0.42267471973251036

<-------------Epipolar error of right and rgb ------------>

Original intrinsics ....

L [[458.20856262 0. 307.18176888]

[ 0. 458.06629417 174.9234571 ]

[ 0. 0. 1. ]]

R: [[498.7213913 0. 306.59984722]

[ 0. 497.02051934 180.05594615]

[ 0. 0. 1. ]]

Intrinsics from the getOptimalNewCameraMatrix/Original ....

L: [[498.7213913 0. 306.59984722]

[ 0. 497.02051934 180.05594615]

[ 0. 0. 1. ]]

R: [[498.7213913 0. 306.59984722]

[ 0. 497.02051934 180.05594615]

[ 0. 0. 1. ]]

Average Epipolar Error is : 0.3633410286610644

Reprojection error threshold -> 3

rgb Reprojection Error: 5.034223

/home/evg/depthai/calibrate.py:1029: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)

calibration_handler.setDistortionCoefficients(stringToCam[camera], cam_info['dist_coeff'])

Reprojection error threshold -> 1.0

left Reprojection Error: 0.151070

/home/evg/depthai/calibrate.py:1068: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)

calibration_handler.setCameraExtrinsics(stringToCam[camera], stringToCam[cam_info['extrinsics']['to_cam']], cam_info['extrinsics']['rotation_matrix'], cam_info['extrinsics']['translation'], specTranslation)

Reprojection error threshold -> 1.0

right Reprojection Error: 0.165998

['high Reprojection Error']

py: DONE.

jakaskerl

Hi @EvgeniyPopov
Are you using the new calibration procedure (the one without polygons on the screen)? The calibration is pretty sensitive to any inaccuracies like the square size or bad lighting. Are you following the guide?

Thanks,
Jaka

EvgeniyPopov

I am using the one with the 11x8 aruco board, can you reference what you're referring to?

Edit: I used this guide to start: https://docs.luxonis.com/en/latest/pages/calibration/

kneave

When calibrating using the old set of scripts you'd get a box overlayed on your image showing a suggestion of how to capture a photo of the board next, in the the newer scripts you instead get an occupancy image showing areas where you have markers detected so that you can get good coverage.

This is an example of the polygons you'd see using the old scripts:

If you see those polygons over your images you're using the older set of scripts, running the following will get you up to date:

git clone https://github.com/luxonis/depthai.git
cd depthai
git submodule update --init --recursive
python3 install_requirements.py

EvgeniyPopov

I'm running the new scripts in that case and get a pretty good occupancy. I tried with 13 and 39 images.

jakaskerl

Hi @EvgeniyPopov
Could you paste the command used when running the calibration? Also any images/coverage would help.

Thanks,
Jaka

vkmNuller

Hi @jakaskerl,

I'm currently utilizing the new calibration script. Here's the full command I'm executing:

python3 calibrate.py -s 2.3 -ms 1.7 -nx 11 -ny 8 -brd bw1098obc -ih

However, I'm encountering several issues:

During calibration, when it calculates the epipolar error, either the color camera or the mono camera's error tends to be significant.
Even if I succeed in overriding the EEPROM, the calibration data doesn't seem accurate (in my estimation).
There are slight offsets in the camera matrix data, as shown below (This is the factory calibration):

"intrinsicMatrix": [
    [2978.385009765625, 0.0, 1843.81689453125],
    [0.0, 2978.385009765625, 1079.8861083984375],
    [0.0, 0.0, 1.0]
],
"height": 2160,
"width": 3840,
"specHfovDeg": 68.7938003540039

The cx value of 1843.81689453125 should be closer to 1920, and the same applies to cy. I presume the distortion coefficients should correct for that offset. However, I'm uncertain about the reason behind such a significant offset. Any clarification on this matter would be greatly appreciated.

jakaskerl

Hi @vkmNuller

Yes there should be a visible difference between epipolar error of color and mono camera, which is due to transformation from RGB -> mono camera, which needs to be done, since for calibration of intrinsic parameters only mono frames can be used. And with the transformation, the image properties (brightness mostly) changes. And also the image in the RGB sensor has smaller FOV and image looks much closer than in mono stereo cameras, which leads to cameras being more dependent on pixel detection error. For that reason in user calibration script there are values of epipolar error of rgb and mono cameras.
Yes, it is normal for intrinsic parameters to be different than the values of normal intrinsic matrix (which should be around size_of_image/2), which happens due to corrections of distortion coefficients. In case you are not sure if the intrinsic matrix is correct, it can be tested by computing the depth from camera ans checking if depth is correct. For more information, you can send us the dataset you captured and we can look into it.

Thanks,
Jaka

vkmNuller

Hi jakaskerl,
It's a lot clearer for me now, but I still don't understand why I'm encountering one problem. My idea is to detect an ArUco marker using OpenCV, but the issue I'm facing is that the detection is giving me the marker with an offset:

DAI: [ 5.04137533e-02 -5.04137533e-02  7.06905365e+01]
CV: [ 1.86120775e+00 -3.23760908e-02  7.06905365e+01]
Diff: [-1.810794   -0.01803766  0.        ]

In this example, I'm using OpenCV's depth to calculate the x & y in DAI's data, and it's odd that the closest I get to 0,0 is 36 pixels away from (0,0) by the x-axis.
This is the calibration I'm using for this test:

        [
            0,
            {
                "cameraType": 0,
                "distortionCoeff": [
                    -4.282538414001465,
                    15.699326515197754,
                    0.0014291888801380992,
                    0.0007043222431093454,
                    -26.29835319519043,
                    -4.37771463394165,
                    15.943224906921387,
                    -26.34567642211914,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0
                ],
                "extrinsics": {
                    "rotationMatrix": [
                        [
                            0.0,
                            0.0,
                            0.0
                        ],
                        [
                            0.0,
                            0.0,
                            0.0
                        ],
                        [
                            0.0,
                            0.0,
                            0.0
                        ]
                    ],
                    "specTranslation": {
                        "x": -0.0,
                        "y": -0.0,
                        "z": -0.0
                    },
                    "toCameraSocket": -1,
                    "translation": {
                        "x": 0.0,
                        "y": 0.0,
                        "z": 0.0
                    }
                },
                "height": 2160,
                "intrinsicMatrix": [
                    [
                        2978.385009765625,
                        0.0,
                        1843.81689453125
                    ],
                    [
                        0.0,
                        2978.385009765625,
                        1079.8861083984375
                    ],
                    [
                        0.0,
                        0.0,
                        1.0
                    ]
                ],
                "lensPosition": 0,
                "specHfovDeg": 68.7938003540039,
                "width": 3840
            }
        ]

Could you provide some insights or ideas on why I'm experiencing this offset?

jakaskerl

Hi @vkmNuller
How are you detecting aruco markers on DAI? Perhaps it could be due to the fact that depth is aligned to left mono camera?

Thanks,
Jaka

vkmNuller

Hi jakaskerl,
I do not detect the ArUco marker on DAI's method, i use DepthAI's ROI example ( luxonis/depthai-experimentsblob/master/gen2-calc-spatials-on-host/calc.py , a bit modified), and click in the middle of the ArUco marker.
Also i use the depth from OpenCV's detection so the calculations are as close as possible (only for the test).
So does this really make any changes ?

Perhaps it could be due to the fact that depth is aligned to left mono camera?

The way i get the ArUco position is peaty standard:

        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        corners, ids, _ = self.detector.detectMarkers(gray)

        _, positions, _ = cv2.aruco.estimatePoseSingleMarkers(corners, 3.3, self.cameraMatrix, self.distortionCoeffs)

And the way i calculate the "ROI":

        position = np.array([
            averageDepth * np.tan(angleX),
            averageDepth * np.tan(angleY),
            averageDepth
        ])

Where averageDepth = ArUco's z

jakaskerl

Hi @vkmNuller
That I understand. What confuses me is how you calculate depth? OpenCV's stereoRectify? Where does the difference come from then? What are you using to calculate this part: CV: [ 1.86120775e+00 -3.23760908e-02 7.06905365e+01]? SolvePnP maybe or some other complex CV method?

Thanks,
Jaka

vkmNuller

Hi jakaskerl,

What confuses me is how you calculate depth?

The standard way i get depth is by depthai
In this test i use what cv2.aruco.estimatePoseSingleMarkers outputs

Where does the difference come from then?

This is the question i'm asking 😃

What are you using to calculate this part: CV: [ 1.86120775e+00 -3.23760908e-02 7.06905365e+01]. SolvePnP maybe or some other complex CV method?

cv2.aruco.estimatePoseSingleMarkers gives the center of the the ArUco marker in Camera Space.
I think cv2.aruco.estimatePoseSingleMarkers uses some sort of SolvePnP

jakaskerl

Hi @vkmNuller
Well the CV method is more prone to errors due to image distortions right? This algorithm simply runs on the undistorted RGB image? Actual stereo depth image should be much more accurate than that. Are you also using stereo.setDepthAlign perhaps?

Thanks,
Jaka