Help Fix High Epipolar Error

ShreyasK

I am working on camera pair of 3 cameras where 2 are OV9782-FF and one in between is OV9782-W wide angle lens of 127 fov. So the 2 stereo camera pairs are placed normally but rgb is placed 180 degree roll as seen in the attached image. Cameras are connected to OAK_FFC_4P Board, Middle wide angle camera connected on CAM_A socket and other two on CAMB and CAM_C.

I am sharing the json containing the configurations:

https://drive.google.com/file/d/1RWpEvvNUlenjYJSSa2eXDMWXyD_ISpcc/view?usp=sharing

I am using charuco24inch_13x7.pdf for calibration on 27-inch monitor which has each block to be of 4.44cm.(python3 calibrate.py -s 4.440 -nx 13 -ny 7 -brd OAK-FFC-4P-3Cam.json)

The epipolar error photo for your reference:

Command Line Output:

Starting image processing

<------------Calibrating right ------------>

INTRINSIC CALIBRATION

Reprojection error of right: 0.2880745752764004

<------------Calibrating left ------------>

INTRINSIC CALIBRATION

Reprojection error of left: 0.2727957762410266

<------------Calibrating middle ------------>

INTRINSIC CALIBRATION

Reprojection error of middle: 0.30279732329162556

<-------------Extrinsics calibration of right and left ------------>

Reprojection error is 0.29644268614224706

<-------------Epipolar error of right and left ------------>

Original intrinsics ....

L [[399.53672362 0. 325.88392518]

[ 0. 399.64585442 203.29043559]

[ 0. 0. 1. ]]

R: [[392.66001003 0. 298.92308488]

[ 0. 392.80459967 201.00279038]

[ 0. 0. 1. ]]

Intrinsics from the getOptimalNewCameraMatrix/Original ....

L: [[392.66001003 0. 298.92308488]

[ 0. 392.80459967 201.00279038]

[ 0. 0. 1. ]]

R: [[392.66001003 0. 298.92308488]

[ 0. 392.80459967 201.00279038]

[ 0. 0. 1. ]]

Average Epipolar Error is : 0.22796665257621102

Displaying Stereo Pair for visual inspection. Press the [ESC] key to exit.

<-------------Extrinsics calibration of left and middle ------------>

Reprojection error is 0.4714245988764749

<-------------Epipolar error of left and middle ------------>

Original intrinsics ....

L [[392.66001003 0. 298.92308488]

[ 0. 392.80459967 201.00279038]

[ 0. 0. 1. ]]

R: [[283.07918457 0. 312.22032188]

[ 0. 283.16246496 197.31399282]

[ 0. 0. 1. ]]

Intrinsics from the getOptimalNewCameraMatrix/Original ....

L: [[283.07918457 0. 312.22032188]

[ 0. 283.16246496 197.31399282]

[ 0. 0. 1. ]]

R: [[283.07918457 0. 312.22032188]

[ 0. 283.16246496 197.31399282]

[ 0. 0. 1. ]]

Average Epipolar Error is : 16.532479417584668

Displaying Stereo Pair for visual inspection. Press the [ESC] key to exit.

Reprojection error threshold -> 1.0

right Reprojection Error: 0.288075

Reprojection error threshold -> 1.0

left Reprojection Error: 0.272796

Reprojection error threshold -> 1.1111111111111112

middle Reprojection Error: 0.302797

['high epipolar error between left and middle']

py: DONE.

ShreyasK

The Main Goal was to use align depth output for obstacle avoidance application.
Initially I tried connecting only CAM_A and calibrating it first, then connected CAM_B and CAM_C and calibrated only CAM_B and CAM_C. Basic Stereo depth and CAM_A undistorted RGB output. But when align depth is tried to be used, it ends up with error of no extrinsic between selected camera sockets.
CAM_A will be used for RGB camera feed with Camera feed rotated.
CAM_B and CAM_C will be used as stereo and for depth estimation with Greyscale output from each camera.

OskarSonc

Hey @ShreyasK - You’re running into a calibration issue between the wide RGB (CAM_A) and the left mono, not a tool problem. The stereo pair itself looks good; only the left and middle epipolar error is huge, so there’s no valid extrinsic for align-depth to use with CAM_A. In your JSON the baselines are basically zero (numbers are mm!) and the 180° roll is on the wrong camera, so the solver starts from a bad prior and drifts. I’d update the JSON with realistic center-to-center distances (in mm), put the 180° roll on the actually rotated wide sensor (CAM_A) or just zero all rotations, and make sure the resolutions match your runtime (e.g. 1280×800 for OV9782). Then rerun a single 3-camera calibration with plenty of Charuco views where the board is clearly visible in all three cams, flash that calibration, and afterwards stereo.setDepthAlign(dai.CameraBoardSocket.CAM_A) should work.

Thanks,
Oskar

ShreyasK

Please check this hope things are right:

{
    "board_config": {
        "name": "OAK-FFC-4P",
        "cameras": {
            "CAM_C": {
                "name": "right",
                "hfov": 80,
                "type": "mono",
                "extrinsics": {
                    "to_cam": "CAM_B",
                    "specTranslation": {
                        "x": 60.07,
                        "y": 0,
                        "z": 0
                    },
                    "rotation": {
                        "r": 0,
                        "p": 0,
                        "y": 0
                    }
                }
            },
            "CAM_B": {
                "name": "left",
                "hfov": 80,
                "type": "mono"
            },
            "CAM_A": {
                "name": "middle",
                "hfov": 127,
                "type": "color",
                "extrinsics": {
                    "to_cam": "CAM_B",
                    "specTranslation": {
                        "x": -30.04,
                        "y": 0,
                        "z": 0
                    },
                    "rotation": {
                        "r": -3.142,
                        "p": 0,
                        "y": 0
                    }
                }                   
            }
        },
        "stereo_config": {
            "left_cam": "CAM_B",
            "right_cam": "CAM_C"
        }
    }
}

Distances are measured from the centre of lenses. So Baseline distance between two stereo ff cameras is 6cm and RGB camera placed at exact middle at 3cm distance from each other.

With regard to runtime related, you meant to say that current calibration process need to be done on the resolution that will be used in real world, right? As current my application usecase will be of 640*400 and current calibration is going on with that same resolution.

I tried calibrating using above json file and still ended up with same error again:

<------------Calibrating right ------------>
INTRINSIC CALIBRATION
Reprojection error of right: 0.2680925121082819
<------------Calibrating left ------------>
INTRINSIC CALIBRATION
Reprojection error of left: 0.21555168733528138
<------------Calibrating middle ------------>
INTRINSIC CALIBRATION
Reprojection error of middle: 0.25423129503488584
<-------------Extrinsics calibration of right and left ------------>
Reprojection error is 0.25054940915687046
<-------------Epipolar error of right and left ------------>
Original intrinsics ....
L [[399.88851517   0.         326.15706581]
 [  0.         399.90694417 203.15031341]
 [  0.           0.           1.        ]]
R: [[392.00665481   0.         299.91846282]
 [  0.         392.15825958 200.85930848]
 [  0.           0.           1.        ]]
Intrinsics from the getOptimalNewCameraMatrix/Original ....
L: [[392.00665481   0.         299.91846282]
 [  0.         392.15825958 200.85930848]
 [  0.           0.           1.        ]]
R: [[392.00665481   0.         299.91846282]
 [  0.         392.15825958 200.85930848]
 [  0.           0.           1.        ]]
Average Epipolar Error is : 0.20798287362059525
Displaying Stereo Pair for visual inspection. Press the [ESC] key to exit.
<-------------Extrinsics calibration of middle and left ------------>
Reprojection error is 0.5008017676991678
<-------------Epipolar error of middle and left ------------>
Original intrinsics ....
L [[283.15027068   0.         312.23280212]
 [  0.         283.22513803 197.34991088]
 [  0.           0.           1.        ]]
R: [[392.00665481   0.         299.91846282]
 [  0.         392.15825958 200.85930848]
 [  0.           0.           1.        ]]
Intrinsics from the getOptimalNewCameraMatrix/Original ....
L: [[392.00665481   0.         299.91846282]
 [  0.         392.15825958 200.85930848]
 [  0.           0.           1.        ]]
R: [[392.00665481   0.         299.91846282]
 [  0.         392.15825958 200.85930848]
 [  0.           0.           1.        ]]
Average Epipolar Error is : 22.224551723984234
Displaying Stereo Pair for visual inspection. Press the [ESC] key to exit.
Reprojection error threshold -> 1.0
right Reprojection Error: 0.268093
Reprojection error threshold -> 1.0
left Reprojection Error: 0.215552
Reprojection error threshold -> 1.1111111111111112
middle Reprojection Error: 0.254231
['high epipolar error between middle and left']
py: DONE.

Few other questions:

Like using 127Degree feed of view camera with 80 degree field of view camera is a problem?
End of the day when the application need to run then I need to do undistortion (By cropping method) and use it with align depth so does this calibration process need to be done in different way?

OskarSonc

Hey @ShreyasK - JSON and baselines look fine now, and calibrating at 640×400 (same as runtime) is correct. Since B and C epipolar error is good and only A and B is bad, this still points to calibration data quality – you need plenty of frames where the Charuco board is large and clearly visible in both CAM_A and CAM_B.

Can you confirm that all three cameras look normal in a simple preview (no obvious distortion or focus issues), and that you followed the steps from our manual calibration guide (board distance, angles, coverage, number of frames, etc.)?