• DepthAI-v2
  • StereoDepth: Why does `setDepthAlign` change the left-right frame ?

Hello !

I'm working with StereoDepth on the Oak-D Lite and I just find a strange behavior of depthai

  • if I set the disparity/depth align to RECTIFIED_LEFT (stereo.setDepthAlign(depthai.RawStereoDepthConfig.AlgorithmControl.DepthAlign.RECTIFIED_LEFT), I got the frame left like image below:

  • if I set the disparity/depth align to RGB (stereo.setDepthAlign(depthai.CameraBoardSocket.RGB)), I got the frame left like image below:

If you look at the image border, you will see a huge difference. Therefore, i dont understand the behavior of setDepthAlign. Should it align the Depth/Disparity only ? Why is the frame left / right changed too ?

Thanks for your support !

    Hi AnhTuNguyen
    This looks strange indeed. Can you add some minimal code so I can reproduce it locally?

    Thanks,
    Jaka

    Note that this isn't left frame, but rectified left frame, which would get changed if we change the depth alignment.

      Hi AnhTuNguyen ,
      I believe rectification changes based on the selected aligned sensor, to preserve operations, otherwise another warp operation would need to be executed.

        erik
        Hello !

        Thanks. I used my oak-d lite camera to do some records, and I believe that the calibration of this camera is not good. When did a test by visualizing the rectifiedLeft-rectifedRight (aligned with RGB), I observe a huge vertical shift (1-2 pixels between left-right).
        To be able to correct these records, I want to understand your calib.json file and your alignement process so that I can correct the records that I made.

        This is the calibration file. In the calibration, i found the last part (rectified rotation). Can I use those metrics to revert the rectification please ?
        {
        "batchName": "",
        "batchTime": 0,
        "boardConf": "",
        "boardCustom": "",
        "boardName": "OAK-D-LITE",
        "boardOptions": 0,
        "boardRev": "R1M1E3",
        "cameraData": [
        [
        2,
        {
        "cameraType": 0,
        "distortionCoeff": [
        -12.104032516479492,
        77.64612579345703,
        0.001263035461306572,
        0.0022347013000398874,
        -77.21680450439453,
        -12.094476699829102,
        77.58488464355469,
        -77.23764038085938,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0
        ],
        "extrinsics": {
        "rotationMatrix": [
        [
        0.9999452233314514,
        0.0073130750097334385,
        0.007484255824238062
        ],
        [
        -0.007313414476811886,
        0.9999732375144958,
        1.7986509192269295e-05
        ],
        [
        -0.007483924273401499,
        -7.272099173860624e-05,
        0.9999719858169556
        ]
        ],
        "specTranslation": {
        "x": 3.75,
        "y": 0.0,
        "z": 0.0
        },
        "toCameraSocket": 0,
        "translation": {
        "x": 3.7195262908935547,
        "y": -0.08316565304994583,
        "z": 0.026069268584251404
        }
        },
        "height": 480,
        "intrinsicMatrix": [
        [
        453.2582092285156,
        0.0,
        336.7252197265625
        ],
        [
        0.0,
        453.2582092285156,
        258.0137939453125
        ],
        [
        0.0,
        0.0,
        1.0
        ]
        ],
        "lensPosition": 0,
        "specHfovDeg": 72.9000015258789,
        "width": 640
        }
        ],
        [
        1,
        {
        "cameraType": 0,
        "distortionCoeff": [
        -9.763245582580566,
        37.59429168701172,
        -0.0005994563107378781,
        6.300376116996631e-05,
        -37.508628845214844,
        -9.76201057434082,
        37.57674026489258,
        -37.51167297363281,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0
        ],
        "extrinsics": {
        "rotationMatrix": [
        [
        0.9999633431434631,
        -0.0022503056097775698,
        -0.008261280134320259
        ],
        [
        0.002276295330375433,
        0.9999924898147583,
        0.0031379053834825754
        ],
        [
        0.008254156447947025,
        -0.0031565953977406025,
        0.9999609589576721
        ]
        ],
        "specTranslation": {
        "x": -7.5,
        "y": 0.0,
        "z": 0.0
        },
        "toCameraSocket": 2,
        "translation": {
        "x": -7.500237464904785,
        "y": 0.01437666080892086,
        "z": -0.12574531137943268
        }
        },
        "height": 480,
        "intrinsicMatrix": [
        [
        457.1339111328125,
        0.0,
        323.1533508300781
        ],
        [
        0.0,
        457.1339111328125,
        249.11166381835938
        ],
        [
        0.0,
        0.0,
        1.0
        ]
        ],
        "lensPosition": 0,
        "specHfovDeg": 72.9000015258789,
        "width": 640
        }
        ],
        [
        0,
        {
        "cameraType": 0,
        "distortionCoeff": [
        -4.017131328582764,
        7.194274425506592,
        0.0006911047967150807,
        -0.0003079548478126526,
        -2.7309975624084473,
        -4.1296186447143555,
        7.666990280151367,
        -3.326430320739746,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0
        ],
        "extrinsics": {
        "rotationMatrix": [
        [
        0.0,
        0.0,
        0.0
        ],
        [
        0.0,
        0.0,
        0.0
        ],
        [
        0.0,
        0.0,
        0.0
        ]
        ],
        "specTranslation": {
        "x": -0.0,
        "y": -0.0,
        "z": -0.0
        },
        "toCameraSocket": -1,
        "translation": {
        "x": 0.0,
        "y": 0.0,
        "z": 0.0
        }
        },
        "height": 2160,
        "intrinsicMatrix": [
        [
        3003.086181640625,
        0.0,
        1863.3856201171875
        ],
        [
        0.0,
        3003.086181640625,
        1119.2449951171875
        ],
        [
        0.0,
        0.0,
        1.0
        ]
        ],
        "lensPosition": 0,
        "specHfovDeg": 68.7938003540039,
        "width": 3840
        }
        ]
        ],
        "hardwareConf": "",
        "imuExtrinsics": {
        "rotationMatrix": [
        [
        0.0,
        0.0,
        0.0
        ],
        [
        0.0,
        0.0,
        0.0
        ],
        [
        0.0,
        0.0,
        0.0
        ]
        ],
        "specTranslation": {
        "x": 0.0,
        "y": 0.0,
        "z": 0.0
        },
        "toCameraSocket": -1,
        "translation": {
        "x": 0.0,
        "y": 0.0,
        "z": 0.0
        }
        },
        "miscellaneousData": [],
        "productName": "",
        "stereoRectificationData": {
        "leftCameraSocket": 1,
        "rectifiedRotationLeft": [
        [
        0.9992857575416565,
        0.03516098111867905,
        0.013843906112015247
        ],
        [
        -0.0351836197078228,
        0.9993798732757568,
        0.0013948403066024184
        ],
        [
        -0.01378627773374319,
        -0.0018809227040037513,
        0.9999032020568848
        ]
        ],
        "rectifiedRotationRight": [
        [
        0.9990556240081787,
        0.03747883066534996,
        0.021980637684464455
        ],
        [
        -0.03744276240468025,
        0.9992966651916504,
        -0.0020502633415162563
        ],
        [
        -0.02204201929271221,
        0.0012253113090991974,
        0.9997562766075134
        ]
        ],
        "rightCameraSocket": 2
        },
        "version": 6
        }

        • erik replied to this.