problems with stereo calibration and bad depth estimation

adrivg

Hardware used:

2 camera module OAK-FFC-48MP
1 OAK-FFC 4P
charuco_28inch_15x8

Hardware mounting:

Config board:

OAK-FFC-4P.json

I have been trying to calibrate a stereo pair for depth calculation (disparity). However, I cannot get the epipolar error lower than 1.06. I also have noticed that the reprojection error between cameras differ a lot (0.76 against 1.24)

I am using the charuco from the official github in a flat monitor of 27inch (so I adapted the checker size to 3.8cm). I have followed the recommendations from the website but the error is still high.

This is the command I am running to make de calibration: python3 calibrate.py -s 3.8 -nx 15 -ny 8 -brd OAK-FFC-4P.json

Anyway, I flashed the EEPROM and tried to run depth estimation but the results are really bad

¿Do you have any idea/recomendation about what I can do to improve the calibration?

I attach the images used for the calibration and the script I ran for the disparity calculation

https://drive.google.com/drive/folders/1CVXPqhDcZUiZ2mwpr0SuMn01XbnaRlPP?usp=sharing

Kind regards

Adrian

jakaskerl

adrivg
Try switching the left and right when performing stereo.

# Linking
monoLeft.out.link(depth.right)
monoRight.out.link(depth.left)
depth.disparity.link(xout.input)

or switch sockets for left and right.

Thanks,
Jaka

adrivg

Thanks for the quick answer. It seems to get better results. However, the person standing in this picture is around 180cm from the camera but the working distance I need is around 70cm. How can I fix that?. From the picture it seems that the FOV is around 20/30º when it should be 45º.

By the way, I am using RGB cameras for stereo, does it affect?

Left = pipeline.create(dai.node.ColorCamera)

Right = pipeline.create(dai.node.ColorCamera)

depth = pipeline.create(dai.node.StereoDepth)

xout = pipeline.create(dai.node.XLinkOut)

Left.video.link(depth.right)

Right.video.link(depth.left)

depth.disparity.link(xout.input)

Thanks,

Adrian

jakaskerl

adrivg
One thing I notice is that lenses are out of focus. This will have an affect on the end result (error).
Second thing that is the problem is that the sensors are rolling shutter which is a problem for moving tagrets -- but should be fine if cameras are completely stationary and the image does not move.

Can you add the calibration dump from the camera?

Thanks,
Jaka

adrivg

I added the calibration dump to the same folder (https://drive.google.com/drive/folders/1CVXPqhDcZUiZ2mwpr0SuMn01XbnaRlPP?usp=sharing).

Regarding to " the lens are out of focus". How could I fix that?

Thanks,

Adrian

jakaskerl

adrivg
Hey, the extrisics in the calibration dump say -5cm from one camera to the other, but you have specified 2.9cm. Make sure the calibration was actually applied.

Thanks,
Jaka

adrivg

Hi!

You were right, The calibration I used to flash the EEPROM was with 5cm between cameras ( I moved the cameras to that distance) but the disparity is almost as bad.

I attatch another full configuration (images used, dump file, reprojection errors, ffc json conf and disparity error). I have tried many different configurations but I cannot get a lower epipolar error of 2.5 ( I think it is too much). Watching the images, do you have any recommendation or maybe any conf that I could try?

https://drive.google.com/drive/folders/1sGeHljsd4kI0ILADPgCvLULOnQ3uTewE?usp=drive_link

Kind regards,

Adrian

jakaskerl

adrivg

Do calibration on THE_4_K resolution. luxonis/depthaiblob/main/calibrate.py#L53
Make sure you update the depthai repo with --recurse-submodules. It seems you have some older repo versions.

Thanks,
Jaka

adrivg

I updated the repo and changed the resolution to THE4K:

'LCM48' : dai.ColorCameraProperties.SensorResolution.THE_4_K

The reprojection error was reduced but the epipolar error is still above 2.

This is the new calibration with the calibrate.py I am using:

https://drive.google.com/drive/folders/1pACZ2lLaIlww9yqOEIQOuYEM_tQTt4-k?usp=drive_link

jakaskerl

adrivg
Don't point the camera at the bright window, it throws off the AE.
Set lens position manually with -rlp {position 0-255}
'-rlp', '--rgbLensPosition', nargs='*', action=ParseKwargs, required=False, default={} , help="Set the manual lens position of the camera for calibration. Example -rlp rgb=135 night=135")

Thanks
Jaka

adrivg

I will try it and give you a respond.

What do you mean by "set lens position manually"?. I do not understand what that number between 0-255 refers to.

Thanks

Adrian

jakaskerl

adrivg
by setting the -rlp argument (when running the calibrate.py), you can set the value of lens position (same value that is changed using autofocus). It can be 0-255, 0 being close to the glass, 255 being the farthest away.

It will change the focus of the image.

Thanks,
Jaka

adrivg

I tried to calibrate again. I used python3 calibrate.py -s 3.9 -nx 15 -ny 8 -brd OAK-FFC-4P.json -dst calib_3_7_C -ep 2 -rlp right=135 left=135.

I tested different rlp values to focus a distance of 90/100 cm from the cameras and the example values were the ones that gave me the best results. The reprojection error was reduced a bit but the epipolar error is still too high (1.06)

I send you the setup for the new calibration: https://drive.google.com/drive/folders/1wITNra_xuafv5RCj7222uhPM0IfPHbqV?usp=drive_link

jakaskerl

adrivg adrivg
The calibration looks fine now.
What script are you using to create stereo depth? Can you show left and right rectified frames and the depth created from them?

Thanks,
Jaka

adrivg

I added the script to the same drive under disparity.py (https://drive.google.com/drive/folders/1wITNra_xuafv5RCj7222uhPM0IfPHbqV). It is based on https://docs.luxonis.com/software/depthai/examples/stereo_depth_video/

Distance from the camera is around 130cm

PD the image's titles are set wrongly (left and rectifiedLeft should be right and rectified right and vice versa)

jakaskerl

adrivg
Cameras are not on the same horizontal axis.. Stereo matching won't work in this case.

it is also not accounted for in the extrinsics (nether translation, nor rotation).

Please keep them on the same axis.

Thanks,
Jaka

adrivg

the distribution is the one shown in the harware mounting, They should be in the same horizontal... I am going to check if there is any slack

jakaskerl

adrivg
We checked the dump and recreated depth.. works ok. I think the only issue now is that sockets are switched.

Though I would advise you to correct the camera positions 🙂

Thanks,
Jaka

adrivg

I swapped the sockets based on your previous comment:

Try switching the left and right when performing stereo.

Linking
monoLeft.out.link(depth.right)
monoRight.out.link(depth.left)
depth.disparity.link(xout.input)
or switch sockets for left and right.

Thanks,
Jaka

Should I change them back?
I am trying to correct camera positions, not sure why I have such slack using OAK-FFC 4P

jakaskerl

adrivg
I know, I was suggesting you try switching it in case it improves the depth. Since it did not solve the issue, yes, revert it back.

Thanks,
Jaka