Stereo rectified streams have rectification issues on sdk 2.21

Mmrvladimir · May 9, 2023

I switched to the new sdk version 2.21.2 and now the rectified streams of the depth node look like they have an undistortion error.

This is how the rectified streams look:

I tried recalibrating the camera, and got calibration numbers that look reasonable and that used to work before:

"distortionCoeff": [
                    -8.933034896850586,
                    77.59778594970703,
                    -0.0017588274786248803,
                    0.0015303785912692547,
                    -85.58924102783203,
                    -9.001920700073242,
                    77.50651550292969,
                    -85.17313385009766,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0
                ],
                "extrinsics": {
                    "rotationMatrix": [
                        [
                            0.9999570250511169,
                            -0.009076806716620922,
                            0.0019016165751963854
                        ],
                        [
                            0.008980450220406055,
                            0.9989150762557983,
                            0.04569495469331741
                        ],
                        [
                            -0.002314317738637328,
                            -0.045675911009311676,
                            0.998953640460968
                        ]
                    ],
                    "specTranslation": {
                        "x": -7.5,
                        "y": 0.0,
                        "z": 0.0
                    },
                    "toCameraSocket": 2,
                    "translation": {
                        "x": -7.489298343658447,
                        "y": -0.08138903975486755,
                        "z": 0.0009566023363731802
                    }
                },
                "height": 800,
                "intrinsicMatrix": [
                    [
                        805.7501831054688,
                        0.0,
                        641.7357788085938
                    ],
                    [
                        0.0,
                        805.892578125,
                        397.3677978515625
                    ],
                    [
                        0.0,
                        0.0,
                        1.0
                    ]
                ],

Is this a bug in the SDK or did I do something wrong? I have tried running calibrate.py and depthai_demo.py in the latest depthai github repo and got the same error.

I am using the OAK-D camera.

erik · May 10, 2023

Hi mrvladimir ,
Could you try using latest develop branch as well?

On depthai-python repo checkout to develop
Execute python script examples/install_requirements.py

Thanks, Erik

Mmrvladimir · May 10, 2023

I tried the develop branch on the depthai repo, still same issue:

/depthai# git log
commit d47c1882b9ef868dd6d67c733f57f374a949f79c (HEAD -> origin/develop, origin/d
evelop)
Merge: 856b26a5 9aadf3e1

On the depthai-python repo commit da2b14f85c6946a283079219170796a94d6366ca I got this error when running examples/install_requirements.py:

ERROR: No matching distribution found for depthai==2.21.2.0.dev+da2b14f85c6946a283079219170796a94d6366ca

Mmrvladimir · May 15, 2023

I tried on devel branch again for depthai-python. This time the rectificaiton worked.

Mmrvladimir · May 24, 2023

This bug in version 2.21.2 is blocking us from using it. When will the new SDK with the fix from develop branch be released?

jakaskerl · May 25, 2023

@erik

erik · May 25, 2023

Hi @mrvladimir ,
Which error/fix are you referring to?

Mmrvladimir · May 25, 2023

Hi @erik ,

I'm referring to the error in this post. The rectified images from the depth node are improperly rectified, as you can see in picture in the first post in this thread.

The same camera calibration, which used to work on sdk 2.20, produces the badly undistorted images above in sdk 2.21, and that is fixed if I use the devel branch.

erik · May 25, 2023

Hi mrvladimir ,
Thanks for the info, yes I think we pushed some checks to develop which auto-applies device-calibrated meshes for wide FOV cameras.

Mmrvladimir · May 25, 2023

This was not on a wide angle camera. It was on OAK-D.

Also do you have an estimate when there will be a new SDK release with the fix?

Mmrvladimir · May 30, 2023

Just to add more to the issue, here are two images of the same scene with the same exact calibration rectified by SDK 2.21.2 and by the devel branch:

(red lines added in GIMP to show curvature)

2.21.2:

devel branch:

Clearly something is off with the rectification in sdk 2.21.2. @erik when can we expect a fix with a stable release version? We can't use the devel branch in our code because there are other stability issues with the devel branch.

erik · Jun 1, 2023

Hi mrvladimir ,
Sorry about the my confusion, but by devel branch do you mean develop branch? If that's the case, then that's great news as it will be released soon to mainline. What are the stability issues on the develop branch - is there a separate ticket for that?

Mmrvladimir · Jun 1, 2023

Hi @erik ,

Yes, I mean the develop branch. Any hint of when soon is? Within the next week or within the next month?

I thought that a random commit on the develop branch shouldn't be expected to be stable. I assume the develop branch is an in-progress branch. I didn't report the issues for it because I assumed you're all aware of them, for example building with it throws c++ compile warnings.

Thanks,

Vladimir

erik · Jun 2, 2023

Hi mrvladimir ,
I'd say this month, I doubt next week already. Usually, it's stable, but really long-term tests / stress tests happen on PR from develop to main, so until then we can't be 100% it's stable. And yes, please do report any issues on develop as well, it helps us find them sooner rather than later.
Thanks, Erik