Hi @jakaskerl
I'm still confused by your use of the term "rectified" for the RGB image. Rectification typically refers to computing a 3x3 rotation matrix that causes the image plane from two or more cameras to be on the same plane. Therefore, implied in the use of the word "rectified" would be another camera to rectify with.
In the case of the RGB camera, it does not take part in the stereo matching process and, besides that, there does not seem to be a rectification matrix for the RGB camera in the calibration on EEPROM indicating that rectification for the RGB image is not available natively from the camera calibration.
So are you saying that we need to compute a rectification matrix ourselves for the RGB image? Or are you using the term "rectified" loosely here for the RGB camera to just indicate "undistorted"?