Hardware used: 2 camera module OAK-FFC-48MP 1 OAK-FFC 4P charuco_28inch_15x8 Hardware mounting: Config board: OAK-FFC-4P.json I have been trying to calibrate a stereo pair for depth calculation (disparity). However, I cannot get the epipolar error lower than 1.06. I also have noticed that the reprojection error between cameras differ a lot (0.76 against 1.24) I am using the charuco from the official github in a flat monitor of 27inch (so I adapted the checker size to 3.8cm). I have followed the recommendations from the website but the error is still high. This is the command I am running to make de calibration: python3 calibrate.py -s 3.8 -nx 15 -ny 8 -brd OAK-FFC-4P.json Anyway, I flashed the EEPROM and tried to run depth estimation but the results are really bad ¿Do you have any idea/recomendation about what I can do to improve the calibration? I attach the images used for the calibration and the script I ran for the disparity calculation https://drive.google.com/drive/folders/1CVXPqhDcZUiZ2mwpr0SuMn01XbnaRlPP?usp=sharing Kind regards Adrian

adrivg Try switching the left and right when performing stereo. # Linking monoLeft.out.link(depth.right) monoRight.out.link(depth.left) depth.disparity.link(xout.input) if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); } or switch sockets for left and right. Thanks, Jaka

Thanks for the quick answer. It seems to get better results. However, the person standing in this picture is around 180cm from the camera but the working distance I need is around 70cm. How can I fix that?. From the picture it seems that the FOV is around 20/30º when it should be 45º. By the way, I am using RGB cameras for stereo, does it affect? Left = pipeline.create(dai.node.ColorCamera) Right = pipeline.create(dai.node.ColorCamera) depth = pipeline.create(dai.node.StereoDepth) xout = pipeline.create(dai.node.XLinkOut) Left.video.link(depth.right) Right.video.link(depth.left) depth.disparity.link(xout.input) Thanks, Adrian

adrivg One thing I notice is that lenses are out of focus. This will have an affect on the end result (error). Second thing that is the problem is that the sensors are rolling shutter which is a problem for moving tagrets -- but should be fine if cameras are completely stationary and the image does not move. Can you add the calibration dump from the camera? Thanks, Jaka

I added the calibration dump to the same folder ( https://drive.google.com/drive/folders/1CVXPqhDcZUiZ2mwpr0SuMn01XbnaRlPP?usp=sharing ). Regarding to " the lens are out of focus". How could I fix that? Thanks, Adrian

adrivg Hey, the extrisics in the calibration dump say -5cm from one camera to the other, but you have specified 2.9cm. Make sure the calibration was actually applied. Thanks, Jaka

Hi! You were right, The calibration I used to flash the EEPROM was with 5cm between cameras ( I moved the cameras to that distance) but the disparity is almost as bad. I attatch another full configuration (images used, dump file, reprojection errors, ffc json conf and disparity error). I have tried many different configurations but I cannot get a lower epipolar error of 2.5 ( I think it is too much). Watching the images, do you have any recommendation or maybe any conf that I could try? https://drive.google.com/drive/folders/1sGeHljsd4kI0ILADPgCvLULOnQ3uTewE?usp=drive_link Kind regards, Adrian

adrivg Do calibration on THE_4_K resolution. luxonis/depthai blob/main/calibrate.py#L53 Make sure you update the depthai repo with --recurse-submodules. It seems you have some older repo versions. Thanks, Jaka

problems with stereo calibration and bad depth estimation

adrivg

I updated the repo and changed the resolution to THE4K:

'LCM48' : dai.ColorCameraProperties.SensorResolution.THE_4_K

The reprojection error was reduced but the epipolar error is still above 2.

This is the new calibration with the calibrate.py I am using:

https://drive.google.com/drive/folders/1pACZ2lLaIlww9yqOEIQOuYEM_tQTt4-k?usp=drive_link

jakaskerl

adrivg
Don't point the camera at the bright window, it throws off the AE.
Set lens position manually with -rlp {position 0-255}
'-rlp', '--rgbLensPosition', nargs='*', action=ParseKwargs, required=False, default={} , help="Set the manual lens position of the camera for calibration. Example -rlp rgb=135 night=135")

Thanks
Jaka

adrivg

I will try it and give you a respond.

What do you mean by "set lens position manually"?. I do not understand what that number between 0-255 refers to.

Thanks

Adrian

jakaskerl

adrivg
by setting the -rlp argument (when running the calibrate.py), you can set the value of lens position (same value that is changed using autofocus). It can be 0-255, 0 being close to the glass, 255 being the farthest away.

It will change the focus of the image.

Thanks,
Jaka

adrivg

I tried to calibrate again. I used python3 calibrate.py -s 3.9 -nx 15 -ny 8 -brd OAK-FFC-4P.json -dst calib_3_7_C -ep 2 -rlp right=135 left=135.

I tested different rlp values to focus a distance of 90/100 cm from the cameras and the example values were the ones that gave me the best results. The reprojection error was reduced a bit but the epipolar error is still too high (1.06)

I send you the setup for the new calibration: https://drive.google.com/drive/folders/1wITNra_xuafv5RCj7222uhPM0IfPHbqV?usp=drive_link

jakaskerl

adrivg adrivg
The calibration looks fine now.
What script are you using to create stereo depth? Can you show left and right rectified frames and the depth created from them?

Thanks,
Jaka

adrivg

I added the script to the same drive under disparity.py (https://drive.google.com/drive/folders/1wITNra_xuafv5RCj7222uhPM0IfPHbqV). It is based on https://docs.luxonis.com/software/depthai/examples/stereo_depth_video/

Distance from the camera is around 130cm

PD the image's titles are set wrongly (left and rectifiedLeft should be right and rectified right and vice versa)

jakaskerl

adrivg
Cameras are not on the same horizontal axis.. Stereo matching won't work in this case.

it is also not accounted for in the extrinsics (nether translation, nor rotation).

Please keep them on the same axis.

Thanks,
Jaka

adrivg

the distribution is the one shown in the harware mounting, They should be in the same horizontal... I am going to check if there is any slack

jakaskerl

adrivg
We checked the dump and recreated depth.. works ok. I think the only issue now is that sockets are switched.

Though I would advise you to correct the camera positions 🙂

Thanks,
Jaka

adrivg

I swapped the sockets based on your previous comment:

Try switching the left and right when performing stereo.

Linking
monoLeft.out.link(depth.right)
monoRight.out.link(depth.left)
depth.disparity.link(xout.input)
or switch sockets for left and right.

Thanks,
Jaka

Should I change them back?
I am trying to correct camera positions, not sure why I have such slack using OAK-FFC 4P

jakaskerl

adrivg
I know, I was suggesting you try switching it in case it improves the depth. Since it did not solve the issue, yes, revert it back.

Thanks,
Jaka

adrivg

Hi Jakaskerl,

I realised that I had the cameras swapped on OAK-FFC-4P.json.

I also discovered that there is an inconsistency when you add the parameter -rlp.
As you can see from the images below. When you are capturing the images, you get the name mapping your camId (the socket name). However, when you set the calibration, you get the name mapping the cam_info(from json file)

If you change the mapping when capturing the images from camIdmapping to cam_infoshould work fine.

That being said, I have redefined the workflow and uploaded it here There are different calibration folders name as calib_Xbeing X the horizontal distance between cameras

I have added more info than before:

The stereo pair images rectified (I am getting high errors in some pictures and I cannot figure out why. Can you take a look?)
I have also realised that the fov has changed (When I run disparity.py the images are zoomed in compared to when I run cam_test.py from the depthai_python repository. It also seems that the autofocus is still set even after using -rlp when calibrating

These are the depth and disparity results for calib_5. The epipolar error I am getting is still too high (2.81) and there is a big difference between right and left reprojection error (almost double, I am not sure if that is normal)

Thank you in advance
Adrián

jakaskerl

adrivg
Change left.setResolution(dai.ColorCameraProperties.SensorResolution.THE_4_K) and scale it using Isp scale rather than setVideoSize. I think there is a bug and the wrong intrinsics are taken.

Thanks,
Jaka

adrivg

I have changed those attributes:

However, when I run te script the windows are frozen:

At least, the left frame where it hangs up seems to be fixed.
Any idea?

adrivg

I set setIspScale to (1,3) to meet the restrictions and now it works:

PD: Do you have any module/script for color correction?