Obtaining Q-matrix from stereo camera (Oak-d Pro)

Tsjarly

Hi,

I was wondering whether it is possible to obtain the q-matrix from the stereo camera pair. We want to try a non-default model that computes / estimates the disparity.
The first question is can we simply obtain the disparity-to-depth matrix that is used within the camera to do some computations on our side.
The second question is, would it be possible to add our own model to estimate disparity from the left-and-right streams to run on the cameras themselves, and still output the resulting depth from the camera streams, and if possible, how would one go about that?

jakaskerl

Tsjarly I was wondering whether it is possible to obtain the q-matrix from the stereo camera pair. We want to try a non-default model that computes / estimates the disparity.
The first question is can we simply obtain the disparity-to-depth matrix that is used within the camera to do some computations on our side.

We don't expose the matrix, no. But you can make it using the calibration data, if you find a way to calculate c_x' yourself.

Tsjarly The second question is, would it be possible to add our own model to estimate disparity from the left-and-right streams to run on the cameras themselves, and still output the resulting depth from the camera streams, and if possible, how would one go about that?

No, I don't think that is possible. You can run it on host, but it will be slower. This is because it is all run on StereoDepth node, which means you can't split the computation unless you seriously alter the FW.

luxonis/depthai-experimentstree/master/gen2-triangulation

Thanks,
Jaka

Tsjarly

jakaskerl

I'm not sure how to compute cx', but I suppose we could use cv2.stereoRectify() function, which outputs Q matrix. Inputs are intrinsics and distortion coefficients from both cameras, as well as the rotation and translation matrix from one monocam to the other, which all can be obtained from device.readCalibration() if I'm not mistaken.

Is there a particular reason why the Q matrix isn't exposed directly?

jakaskerl

Hi @Tsjarly

Tsjarly Is there a particular reason why the Q matrix isn't exposed directly?

Not sure, tbh.

jakaskerl if you find a way to calculate c_x' yourself.

C_x' should also be accessible from the calibration. It's the first row, last column of the second intrinsic matrix. So the whole matrix should be accessible, just not via API, but manually through the calibration on the EEPROM.

Thanks,
Jaka