• DepthAI

  • Depth Alignment to RGB function

Hi,

We are trying to run our own model to get a disparity/depth map (which is aligned to the left mono cam), which we want to align with the rgb image. Could you share the functions that are used to do this?

To clarify, we wanted to run our own depth estimation model, using the rectified left and right stereo pair images.
So each frame, we extract rectified left and right from the camera, as well as rgb.
We run our depth estimation model, which outputs a disparity as seen from the left camera's point of view.
We would like to align this output to the rgb image, which in the pipeline can be done by using the stereo.setDepthAlign(rgbCamSocket) function. However, we are trying to do this outside of the pipeline. We were wondering what kind of intrinsics/matrices/operations you do internally to do this alignment, as I cannot find it in the source code

Hi @erik

I tried to run this script on our camera, but I can still see a small shift / misalignment between the two. (first image)
If I run our pipeline that uses the align depth to rgb function, this shift is not there (second image), any idea what could be happening there?

As we are using the OV9782 sensors, I had to change the resolution of the rgb cam to THE800P. I basically changed all resolutions to 800P (and obtaining intrinsics from the camera with that format), and commented out the isp scaling factor for the rgb camera, to match output sizes for all images. Other than that I run the code exactly as showed in the example

Hi @Tsjarly ,
Yeah the code used above isn't perfect, but a decent demo. On FW we do it differently, as it's optimized for parallel execution (we rectify depth + color, then only horizontally shift depth points, instead of projection).

5 days later

Hi @erik ,
Okay, fair enough. Do you have any indication when the DepthAlign node will be exposed? Will this be together with the release of the SR ToF camera, or sooner/later?

@erik I am also doing something similar, I am getting stereo (left, right) images and calculating depth on host (not spatial on host), do you have any example/method which aligns the RGB stream with Depth Calculated on Host. Regards!

In my case Im just getting streams from cameras, depth and detection is happening on host, I just want to align my Depth (which is calculated on host) should be aligned with RGB from camera. Regards!

Hi @Tsjarly and @SamiUddin .
The plan is to release it together with SR PoE (cam that has ToF), so in 2 weeks. We might also write a python script that demos the alignment on the host, but I can't promise that atm.
Thanks, Erik

erik Thanks Erik I will must integrate in my project and give you feedback