To clarify, we wanted to run our own depth estimation model, using the rectified left and right stereo pair images.
So each frame, we extract rectified left and right from the camera, as well as rgb.
We run our depth estimation model, which outputs a disparity as seen from the left camera's point of view.
We would like to align this output to the rgb image, which in the pipeline can be done by using the stereo.setDepthAlign(rgbCamSocket) function. However, we are trying to do this outside of the pipeline. We were wondering what kind of intrinsics/matrices/operations you do internally to do this alignment, as I cannot find it in the source code