Reverse Rectification

DanielScho

Hello everyone,

We had a slight hiccup in our data collection process. We recorded a rather large dataset (left/right rectified frames, RGB and depth) with an OAK-D Pro, but unfortunately the distortion correction seems to have been disabled. This leads to very distorted depth information and obviously also very unrealistic point clouds with a large curvature away from the camera. We fixed that since then, by setting enableDistortionCorrection(False), but would be delighted if there was a way to repair the already collected dataset (on-site collection is rather work intensive).

Would it be possible to invert the rectification operation, such that we get the original unrectified (and in our case distorted) monocamera frames so that we can just put them through the complete stereo-pipeline again (which itself then performs distortion correction and again rectifies the frames)? The transformation matrices for rectification seem to be retrievable by the device API, so it should be possible to calculate the inverse transform?

I already tried "stereoNode.setRectification(False)" previously, but that seemed to disable both rectification and distortion correction. Which makes sense, as I understand that these two operations are not independent of each other and their order of application would make a difference.

Best regards,

Daniel

erik

Hi @DanielScho ,
Undistortion isn't strictly necessary for normal FoV cameras, and its actually disabled by default, only enabled for wide fov cameras. So you can just set rectification to false, and stream the rectified frames to the StereoDepth node. If that doesn't work as expected, perhaps you can share some images?
Thanks, Erik

DanielScho

erik

Hi @erik,

I captured some new images by pointing the camera at a parallel empty white wall. Once with activated Distortion Correction, once without. The colors of the depth frames are normalized to a distance range of 1.6 to 1.9 meters, with actual depth values averaged as overlay. I would expect the depth map to be more or less constant, but you can clearly see the curvature away from the camera at the left and right sides. What you can also see is that the distortion correction seems to have a positive effect on the quality of the depth map and is correcting it at least partially. The Problem is obviously still there but much less pronounced. That's the reason we wanted to apply it to our already captured data.

Rectified right mono and depth frame without distortion correction:

With distortion correction:

My initial assumption was some kind of problem with the camera calibration. The positive effect from the distortion correction is then only some lucky coincident?

Thanks,
Daniel

erik

Hi Daniel,
Besides the missing distortion correction, it might also be that the dot projector is messing up disparity matching, as it's too strong. I'd rather suggest testing this with a noise pattern instead, just to eliminate the potential of the dot projector causing this issue.

DanielScho

erik

Hi @erik,

I took similar images from a wooden floor at around 1.5m distance. The left image is again the rectified right mono frame, middle the depth output without distortion correction and right with correction.

With deactivated dot projector:

With projector at 0.1 brightness value:

The projector obviously helps with the artifacts created from the vertical lines of the floor tiles, but there appears to be no influence on the distortion itself. This is also not caused by the capture environment, as we have seen the same distortion with images taken at an industrial site.

Thanks,
Daniel

erik

Hi @DanielScho ,
Thank you for the detailed report. It might be possible to un-rectify the frames, by following stereo-on-host demo, but you'd want to create reverse rectification matrix (H1/H2) and then doing warping for unrectification. Note that you might lose some FOV/details, as rectification can crop/zoom/rotate the original frame.

DanielScho

erik

Ok, thanks. I will take a look at that.

Just another short question:
Is such a high distortion normal for OAK-D Pro cameras? Or is this just a device-specific problem because we have a second OAK-D Pro that appears to be similar. Just to make sure it not a software problem on our side, I can see the same distortion with luxonis/depthai-viewer

Thanks,
Daniel

erik

DanielScho Hi,
I don't think it's such a high distortion, but even tiny distortion can throw off stereo matching algorithm, so you will end up with large discrepancies in the depth values (as you have visualized). But yes, not all lenses are identical, so cameras will have different amount of distortion.