High Epipolar Error when calibrating OAK-D Lite camera

Hi @EvgeniyPopov
Are you using the new calibration procedure (the one without polygons on the screen)? The calibration is pretty sensitive to any inaccuracies like the square size or bad lighting. Are you following the guide?

Thanks,
Jaka

When calibrating using the old set of scripts you'd get a box overlayed on your image showing a suggestion of how to capture a photo of the board next, in the the newer scripts you instead get an occupancy image showing areas where you have markers detected so that you can get good coverage.

This is an example of the polygons you'd see using the old scripts:

If you see those polygons over your images you're using the older set of scripts, running the following will get you up to date:

git clone https://github.com/luxonis/depthai.git
cd depthai
git submodule update --init --recursive
python3 install_requirements.py

Hi @EvgeniyPopov
Could you paste the command used when running the calibration? Also any images/coverage would help.

Thanks,
Jaka

Hi @vkmNuller

1. Yes there should be a visible difference between epipolar error of color and mono camera, which is due to transformation from RGB -> mono camera, which needs to be done, since for calibration of intrinsic parameters only mono frames can be used. And with the transformation, the image properties (brightness mostly) changes. And also the image in the RGB sensor has smaller FOV and image looks much closer than in mono stereo cameras, which leads to cameras being more dependent on pixel detection error. For that reason in user calibration script there are values of epipolar error of rgb and mono cameras.
2. Yes, it is normal for intrinsic parameters to be different than the values of normal intrinsic matrix (which should be around size_of_image/2), which happens due to corrections of distortion coefficients. In case you are not sure if the intrinsic matrix is correct, it can be tested by computing the depth from camera ans checking if depth is correct. For more information, you can send us the dataset you captured and we can look into it.

Thanks,
Jaka

Hi jakaskerl,
It's a lot clearer for me now, but I still don't understand why I'm encountering one problem. My idea is to detect an ArUco marker using OpenCV, but the issue I'm facing is that the detection is giving me the marker with an offset:

DAI: [ 5.04137533e-02 -5.04137533e-02  7.06905365e+01]
CV: [ 1.86120775e+00 -3.23760908e-02  7.06905365e+01]
Diff: [-1.810794   -0.01803766  0.        ] 

In this example, I'm using OpenCV's depth to calculate the x & y in DAI's data, and it's odd that the closest I get to 0,0 is 36 pixels away from (0,0) by the x-axis.
This is the calibration I'm using for this test:

        [
            0,
            {
                "cameraType": 0,
                "distortionCoeff": [
                    -4.282538414001465,
                    15.699326515197754,
                    0.0014291888801380992,
                    0.0007043222431093454,
                    -26.29835319519043,
                    -4.37771463394165,
                    15.943224906921387,
                    -26.34567642211914,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0
                ],
                "extrinsics": {
                    "rotationMatrix": [
                        [
                            0.0,
                            0.0,
                            0.0
                        ],
                        [
                            0.0,
                            0.0,
                            0.0
                        ],
                        [
                            0.0,
                            0.0,
                            0.0
                        ]
                    ],
                    "specTranslation": {
                        "x": -0.0,
                        "y": -0.0,
                        "z": -0.0
                    },
                    "toCameraSocket": -1,
                    "translation": {
                        "x": 0.0,
                        "y": 0.0,
                        "z": 0.0
                    }
                },
                "height": 2160,
                "intrinsicMatrix": [
                    [
                        2978.385009765625,
                        0.0,
                        1843.81689453125
                    ],
                    [
                        0.0,
                        2978.385009765625,
                        1079.8861083984375
                    ],
                    [
                        0.0,
                        0.0,
                        1.0
                    ]
                ],
                "lensPosition": 0,
                "specHfovDeg": 68.7938003540039,
                "width": 3840
            }
        ] 

Could you provide some insights or ideas on why I'm experiencing this offset?

Hi @vkmNuller
How are you detecting aruco markers on DAI? Perhaps it could be due to the fact that depth is aligned to left mono camera?

Thanks,
Jaka

Hi @vkmNuller
That I understand. What confuses me is how you calculate depth? OpenCV's stereoRectify? Where does the difference come from then? What are you using to calculate this part: CV: [ 1.86120775e+00 -3.23760908e-02 7.06905365e+01]? SolvePnP maybe or some other complex CV method?

Thanks,
Jaka

Hi jakaskerl,

What confuses me is how you calculate depth?

The standard way i get depth is by depthai
In this test i use what cv2.aruco.estimatePoseSingleMarkers outputs

Where does the difference come from then?

This is the question i'm asking

What are you using to calculate this part: CV: [ 1.86120775e+00 -3.23760908e-02 7.06905365e+01]. SolvePnP maybe or some other complex CV method?

cv2.aruco.estimatePoseSingleMarkers gives the center of the the ArUco marker in Camera Space.
I think cv2.aruco.estimatePoseSingleMarkers uses some sort of SolvePnP

Hi @vkmNuller
Well the CV method is more prone to errors due to image distortions right? This algorithm simply runs on the undistorted RGB image? Actual stereo depth image should be much more accurate than that. Are you also using stereo.setDepthAlign perhaps?

Thanks,
Jaka