Does depth/disparity correspond to raw or rectified images?

asperry

The StereoDepth node rectifies the left and right images before performing stereo matching. Does that mean that the depth/disparity at a given pixel (x,y) actually corresponds to the pixel (x,y) in the rectified image (left or right depending on which camera the depth/disparity is aligned to) and not the raw image?

In other words, is it technically wrong to use the depth/disparity map with the raw image (because distortion, etc. would put each depth at a slightly different place)?

jakaskerl

Hi @asperry
Yes, the depth map corresponds to the rectified images and not raw frame. The reason for it is, that in case of using unrectified images, the actual stereo matching would not correspond to the projection into planar plain (x,y) but the projection into the spherical plane (because of curvature of the lens), which would result in wrong depth map (parabolical profile while facing the straight white wall).

Thanks,
Jaka

asperry

Thanks for your reply @jakaskerl! As a follow-up question, then the RGB-aligned depth map would correspond to the rectified (and not raw) RGB frame? If so, then why does this launch file from depthai-ros not rectify the RGB by default?

jakaskerl

cc @Luxonis-Adam

Luxonis-Adam

Hi, this behavior is mostly due to legacy reasons, in the future that launch file will be changed/deprecated in favor of dedicated Pointcloud DAI node.

asperry

Interesting, are you saying that there are plans to compute the point cloud directly on the camera instead of using the ROS pointcloud nodes?

jakaskerl

asperry
Yep.