Hi, I have a setup of two OVO 9282 cameras with an Oak FFC 3P. I tried calibrating the camera with the calibration.py file, and the following were the results, which didn't include distortion coefficients. Also, on pressing esc at the end, no calibration results were saved in a file. My question is: why is the calibration result incomplete with no distortion coefficients and rectification matrix? Is there no distortion in these cameras? Why is this file not saving calibration data? shivam157@ubuntu:~/depthai$ python3 calibrate.py -s 3.8 --board OAK-FFC-3P -nx 13 -ny 7 -dbg if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); } Using dataset path: dataset Starting image processing <------------Calibrating right ------------> INTRINSIC CALIBRATION Reprojection error of right: 0.1450288774499517 <------------Calibrating left ------------> INTRINSIC CALIBRATION Reprojection error of left: 0.18671330931332783 <-------------Extrinsics calibration of right and left ------------> Reprojection error is 0.1806323559925981 <-------------Epipolar error of right and left ------------> Original intrinsics .... L [[909.05184795 0. 644.25179619] [ 0. 909.33572655 383.70089729] [ 0. 0. 1. ]] R: [[906.07261965 0. 625.2752514 ] [ 0. 906.30579217 399.46458918] [ 0. 0. 1. ]] Intrinsics from the getOptimalNewCameraMatrix/Original .... L: [[906.07261965 0. 625.2752514 ] [ 0. 906.30579217 399.46458918] [ 0. 0. 1. ]] R: [[906.07261965 0. 625.2752514 ] [ 0. 906.30579217 399.46458918] [ 0. 0. 1. ]] Average Epipolar Error is : 0.1324884918914444 Displaying Stereo Pair for visual inspection. Press the [ESC] key to exit. Reprojection error threshold -> 1.1111111111111112 right Reprojection Error: 0.145029 /home/shivam157/depthai/calibrate.py:1029: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.) calibration_handler.setDistortionCoefficients(stringToCam[camera], cam_info['dist_coeff']) /home/shivam157/depthai/calibrate.py:1068: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.) calibration_handler.setCameraExtrinsics(stringToCam[camera], stringToCam[cam_info['extrinsics']['to_cam']], cam_info['extrinsics']['rotation_matrix'], cam_info['extrinsics']['translation'], specTranslation) Reprojection error threshold -> 1.1111111111111112 left Reprojection Error: 0.186713 [] py: DONE. shivam157@ubuntu:~/depthai$ ``` The following is the board file: { "board_config": { "name": "OAK-FFC-3P", "revision": "R1M0E1", "cameras":{ "CAM_C": { "name": "right", "hfov": 71.86, "type": "mono", "extrinsics": { "to_cam": "CAM_B", "specTranslation": { "x": 14.5, "y": 0, "z": 0 }, "rotation":{ "r": 0, "p": 0, "y": 0 } } }, "CAM_B": { "name": "left", "hfov": 71.86, "type": "mono" } }, "stereo_config":{ "left_cam": "CAM_B", "right_cam": "CAM_C" } } if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); } } I think this is related to numpy version. I am not sure if I downgrade numpy that it will support Opencv 4.9. It looks like the error is in the distortion coefficient and extrinsic parameters lines of code. What should I do?

I am not 100% sure but the number of markers are different in some images. This might be the problem?

Hi @ShivamSharma And if you tried to load the calibration file with https://docs.luxonis.com/software/depthai/examples/calibration_reader/ ? Thanks, Jaka

When I run the script given in the link you sent, I get the following output: venvshivam157@ubuntu:~/depthai-python$ python3 get_calib.py RGB Camera Default intrinsics... [[3133.269775390625, 0.0, 1922.1737060546875], [0.0, 3133.713623046875, 1102.6722412109375], [0.0, 0.0, 1.0]] 3840 2160 RGB Camera Default intrinsics... [[3133.269775390625, 0.0, 1922.1737060546875], [0.0, 3133.713623046875, 1102.6722412109375], [0.0, 0.0, 1.0]] 3840 2160 RGB Camera resized intrinsics... 3840 x 2160 [[3.13326978e+03 0.00000000e+00 1.92217371e+03] [0.00000000e+00 3.13371362e+03 1.10267224e+03] [0.00000000e+00 0.00000000e+00 1.00000000e+00]] RGB Camera resized intrinsics... 4056 x 3040 [[3.30951611e+03 0.00000000e+00 2.03029590e+03] [0.00000000e+00 3.30998486e+03 1.54394751e+03] [0.00000000e+00 0.00000000e+00 1.00000000e+00]] LEFT Camera Default intrinsics... [[693.9341430664062, 0.0, 664.17138671875], [0.0, 694.2032470703125, 406.8681640625], [0.0, 0.0, 1.0]] 1280 800 LEFT Camera resized intrinsics... 1280 x 720 [[693.93414307 0. 664.17138672] [ 0. 694.20324707 366.86816406] [ 0. 0. 1. ]] RIGHT Camera resized intrinsics... 1280 x 720 [[690.59875488 0. 619.99499512] [ 0. 690.54821777 276.21847534] [ 0. 0. 1. ]] LEFT Distortion Coefficients... k1: -0.14615201950073242 k2: 5.131624221801758 p1: -0.0004437704337760806 p2: -0.0003282237739767879 k3: 0.816307544708252 k4: 0.22460998594760895 k5: 5.021973609924316 k6: 2.6677396297454834 s1: 0.0 s2: 0.0 s3: 0.0 s4: 0.0 τx: 0.0 τy: 0.0 RIGHT Distortion Coefficients... k1: 1.1050113439559937 k2: 1.1275875568389893 p1: 0.00031773868249729276 p2: 0.0002416888455627486 k3: 0.12704245746135712 k4: 1.4751837253570557 k5: 1.46196711063385 k6: 0.46535491943359375 s1: 0.0 s2: 0.0 s3: 0.0 s4: 0.0 τx: 0.0 τy: 0.0 RGB FOV 68.7938003540039, Mono FOV 110.0 LEFT Camera stereo rectification matrix... [[ 9.90124752e-01 7.40214346e-03 -3.64586040e+01] [-7.32702106e-03 9.95746611e-01 -8.60322599e+01] [-8.12898310e-06 3.72185724e-06 1.00401435e+00]] RIGHT Camera stereo rectification matrix... [[ 9.94906762e-01 7.44132256e-03 4.97689224e+00] [-7.36240841e-03 1.00101704e+00 2.47433011e+00] [-8.16824369e-06 3.74155680e-06 1.00401153e+00]] Transformation matrix of where left Camera is W.R.T right Camera's optical center [[ 9.99970078e-01 2.70099915e-03 7.25283753e-03 5.21282959e+00] [-2.66335462e-03 9.99982953e-01 -5.19499322e-03 -1.26599353e-02] [-7.26674590e-03 5.17552113e-03 9.99960184e-01 8.40604864e-03] [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 1.00000000e+00]] Transformation matrix of where left Camera is W.R.T RGB Camera's optical center [[ 9.99992728e-01 -2.86172348e-04 3.81197175e-03 7.94537354e+00] [ 2.56973435e-04 9.99970675e-01 7.65808392e-03 -3.24345136e+00] [-3.81405186e-03 -7.65704829e-03 9.99963403e-01 -5.16706049e-01] [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 1.00000000e+00]] venvshivam157@ubuntu:~/depthai-python$ Is this calibration data from my camera? I am using two Mono OV9282 camera with each 75 degree FOV. I don't have any rgb camera. I only used calibration.py file. I don't think I flashed the device with the current setup. My baseline is 14.5 cm in x

Hi ShivamSharma Looks like the flash didn't succeed. What is -dbg parameter? Are you using the latest calibrate script? Also try disabling rgb with -dsb rgb . Thanks, Jaka

Thank you for your reply! I am using the latest branch of the depthAI library. The dbg parameter is to make the calibration file use the dataset. I will use the dsb tag with calibration file. Do you have more debugging steps?

Hi @ShivamSharma ShivamSharma The dbg parameter is to make the calibration file use the dataset. Use -m process flag to only run the processing stage and not the image acquisition. The -dbg flag prevents the flashing part of the script. Thanks, Jaka

I used the -m process and -dsb rgb tags you mentioned and the calibration.py file flashed the device and the calibration result was saved in the resources folder. I will check how accurate it is by getting disparity and will let you know the result.

ShivamSharma Yes, they normally large due to distortion correction being a rational function and not only a polynomial one as seen here https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html

jakaskerl I think you didn’t see my updated reply

No distortion coefficient and rectification matrix generated by calibration.py

jakaskerl

ShivamSharma
Could you send over the dataset you have used (should be stored in the dataset folder)? Also left and right (not rectified) image for scene where the depth doesn't look ok, and a calibration dump. We'll try to reproduce the results to see what is wrong.

Thanks,
Jaka

ShivamSharma

Link to dataset

The following are the raw images:
Link to left and right image

The following is the link to calibration dump:
Link

jakaskerl

Hi @ShivamSharma
The coverage during calibration is very poor. In order to properly calibrate the cameras, make sure to cover the full FOV of the camera.

Board config file might not be flashed properly, since the calibration_dump actually has three cameras (one seems to be duplicated.

It should look more like this:

{
    "batchName": "",
    "batchTime": 1679712500,
    "boardConf": "IR-C00M05-00",
    "boardCustom": "",
    "boardName": "DM1090",
    "boardOptions": 0,
    "boardRev": "R3M0E3",
    "cameraData": [
        [
            2,
            {
                "cameraType": 0,
                "distortionCoeff": [
                    -2.396423578262329,
                    -10.477461814880371,
                    -0.0004928055568598211,
                    0.0011145062744617462,
                    26.935871124267578,
                    -2.4506843090057373,
                    -10.170122146606445,
                    26.494205474853516,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0
                ],
                "extrinsics": {
                    "rotationMatrix": [
                        [
                            0.997919499874115,
                            0.05123051628470421,
                            -0.03914157673716545
                        ],
                        [
                            -0.051841288805007935,
                            0.9985463619232178,
                            -0.014751287177205086
                        ],
                        [
                            0.03832896426320076,
                            0.016749747097492218,
                            0.9991247653961182
                        ]
                    ],
                    "specTranslation": {
                        "x": -14.5,
                        "y": -0.0,
                        "z": -0.0
                    },
                    "toCameraSocket": 1,
                    "translation": {
                        "x": -14.075519561767578,
                        "y": 0.5210347771644592,
                        "z": -0.29575979709625244
                    }
                },
                "height": 800,
                "intrinsicMatrix": [
                    [
                        909.0518188476563,
                        0.0,
                        644.2517700195313
                    ],
                    [
                        0.0,
                        909.3357543945313,
                        383.7008972167969
                    ],
                    [
                        0.0,
                        0.0,
                        1.0
                    ]
                ],
                "lensPosition": 0,
                "specHfovDeg": 75.0,
                "width": 1280
            }
        ],
        [
            1,
            {
                "cameraType": 0,
                "distortionCoeff": [
                    -3.9288971424102783,
                    -3.196507692337036,
                    -0.00022527640976477414,
                    0.0015712999738752842,
                    19.68087387084961,
                    -3.980436325073242,
                    -2.887248992919922,
                    19.199928283691406,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0
                ],
                "extrinsics": {
                    "rotationMatrix": [
                    [
                        0.0,
                        0.0,
                        0.0
                    ],
                    [
                        0.0,
                        0.0,
                        0.0
                    ],
                    [
                        0.0,
                        0.0,
                        0.0
                    ]
                    ],
                    "specTranslation": {
                    "x": -0.0,
                    "y": -0.0,
                    "z": -0.0
                    },
                    "toCameraSocket": -1,
                    "translation": {
                    "x": 0.0,
                    "y": 0.0,
                    "z": 0.0
                    }
                },
                "height": 800,
                "intrinsicMatrix": [
                    [
                        906.0726318359375,
                        0.0,
                        625.2752685546875
                    ],
                    [
                        0.0,
                        906.3057861328125,
                        399.464599609375
                    ],
                    [
                        0.0,
                        0.0,
                        1.0
                    ]
                ],
                "lensPosition": 0,
                "specHfovDeg": 75.0,
                "width": 1280
            }
        ]
    ],
    "deviceName": "",
    "hardwareConf": "F0-FV00-BC000",
    "housingExtrinsics": {
        "rotationMatrix": [],
        "specTranslation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "toCameraSocket": -1,
        "translation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        }
    },
    "imuExtrinsics": {
        "rotationMatrix": [
            [
                0.0,
                0.0,
                0.0
            ],
            [
                0.0,
                0.0,
                0.0
            ],
            [
                0.0,
                0.0,
                0.0
            ]
        ],
        "specTranslation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "toCameraSocket": -1,
        "translation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        }
    },
    "miscellaneousData": [],
    "productName": "OAK-FFC-3P",
    "stereoEnableDistortionCorrection": false,
    "stereoRectificationData": {
        "leftCameraSocket": 1,
        "rectifiedRotationLeft": [
            [
                0.9990953207015991,
                -0.03698360174894333,
                0.020993344485759735
            ],
            [
                0.036815181374549866,
                0.9992871880531311,
                0.008353326469659805
            ],
            [
                -0.02128731645643711,
                -0.007572895381599665,
                0.999744713306427
            ]
        ],
        "rectifiedRotationRight": [
            [
                0.9997386932373047,
                0.014605958014726639,
                -0.017585638910531998
            ],
            [
                -0.014745572581887245,
                0.9998605847358704,
                -0.007835760712623596
            ],
            [
                0.017468737438321114,
                0.008093023672699928,
                0.9998146295547485
            ]
        ],
        "rightCameraSocket": 2
    },
    "stereoUseSpecTranslation": true,
    "version": 7,
    "verticalCameraSocket": -1
}

Thanks,
Jaka

ShivamSharma

Okay, I will work on using the full FOV. But why are there 3 cameras in the calibration file? Is it because of poor calibration or bad board file. I shared my board file in the replies above. Is everything correct in my board file? Also, if calibration is incorrect then why does it give better disparity outside of ROS 2?

Edit:
It is difficult to cover the Full FOV because charuco board goes out of frame if I move the other camera to border. The cable length is also limited so I can't go far from the screen . The following are the result outside of ROS 2 with 10cm baseline

There is still old data in the saved calib file from previous calibration. Still three sockets even though I am using two.

Please help me get good depth, fps and synchronization in ROS 2. I have created another discussion for fps and synchronization but I think these two issues are related. Thank you for your help so far!

I also want to discuss hard coding by Luxonis in ROS 2 driver. I see some hard coding of focal length and baseline in the code.

The following is the depth I am getting in ROS 2 after calibration and hardcoding the correct focal length and baseline.

jakaskerl

Hi @ShivamSharma
You don't need to keep the whole charuco in the frame, it's fine if some squares are missed. It's more important that the whole FOV is covered.

And I think it should work if you omit the -dsb rgb, the script should still work if correct (2 camera) calib file is provided.

As for ROS, @DaniloPejovic is looking into it.

Thanks,
Jaka

ShivamSharma

I omitted the -dsb rgb flag and made sure to cover all the fields of view of both cameras. I am attaching the dataset again. If you think calibration is still wrong, let me know so I can do it again. I will also attach a link to the board file I used and the calibration result. Could it be possible that I am using the wrong values from the calibration file? I have double-checked that I am using the values for the right and left sockets. I will update the answer with the ROS2 results. There are still three sockets in the calibration result.

Link

Update: Still bad depth in ROS 2 after re-calibration, and still 3 sockets in the calibration result. One socket that doesn't have a camera attached has data from previous calibrations.

I also uploaded the right and left calibration files in ROS 2 to the link above.

jakaskerl

Hi @ShivamSharma
Could be a bug in the calibration script, would have to check that...
Could you run it with -nic so no existing calibration is used? That would make sure the cameras are not restacked.

Thanks,
Jaka

ShivamSharma

When I use -nic I get the following error:

Flashing Calibration data into 
EEPROM VERSION being flashed is  -> 7
EEPROM VERSION being flashed is  -> 7
Device closed in exception..
No PROTECTED permissions to override protected EEPROM fields
Traceback (most recent call last):
  File "/home/shivam157/depthai/calibrate.py", line 1124, in calibrate
    self.device.flashCalibration2(calibration_handler)
RuntimeError: No PROTECTED permissions to override protected EEPROM fields

Also, now I also get the following error when I run cam_test.py:

shivam157@ubuntu:~/depthai-python/utilities$ python3 cam_test.py --cameras left,m right,m
DepthAI version: 2.21.2.0
DepthAI path: /home/shivam157/.local/lib/python3.10/site-packages/depthai.cpython-310-aarch64-linux-gnu.so
Traceback (most recent call last):
  File "/home/shivam157/depthai-python/utilities/cam_test.py", line 186, in <module>
    '1012': dai.ColorCameraProperties.SensorResolution.THE_1352X1012,
AttributeError: type object 'depthai.SensorResolution' has no attribute 'THE_1352X1012'

I was not getting this error before. After flashing without
Update: Not getting the second error above probably because I flashed without -nic.

@jakaskerl A correct calibration flash will most probably solve the bad depth in ROS2. I might just need to switch sockets in config. I checked the camera information topics and saw that the projection matrix was wrong. So if the baseline is being calculated from the projection matrix, then the depth will be bad. This is the calibration it is getting from device. Please fix the calibration.py file so that I can flash it, or tell me how to not use the device calibration data and use the external files in the camera. yaml. Right now, if I use the external files, then both of the calibration sources are publishing to the same info topic.

Btw, this time the saved calibration file with -nic looks correct, it is just not flashed because of the error I pasted above.
The calibration file saved:


{
    "batchName": "",
    "batchTime": 0,
    "boardConf": "",
    "boardCustom": "",
    "boardName": "Custom FFC",
    "boardOptions": 0,
    "boardRev": "R1M0E3",
    "cameraData": [
        [
            1,
            {
                "cameraType": 0,
                "distortionCoeff": [
                    -1.8108996152877808,
                    0.5892674922943115,
                    0.00020837810006923974,
                    0.0008147237240336835,
                    -0.4694308340549469,
                    -1.8531672954559326,
                    0.7104603052139282,
                    -0.5547073483467102,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0
                ],
                "extrinsics": {
                    "rotationMatrix": [],
                    "specTranslation": {
                        "x": 0.0,
                        "y": 0.0,
                        "z": 0.0
                    },
                    "toCameraSocket": -1,
                    "translation": {
                        "x": 0.0,
                        "y": 0.0,
                        "z": 0.0
                    }
                },
                "height": 800,
                "intrinsicMatrix": [
                    [
                        906.6533813476563,
                        0.0,
                        625.2329711914063
                    ],
                    [
                        0.0,
                        906.753662109375,
                        399.9358825683594
                    ],
                    [
                        0.0,
                        0.0,
                        1.0
                    ]
                ],
                "lensPosition": 0,
                "specHfovDeg": 75.0,
                "width": 1280
            }
        ],
        [
            2,
            {
                "cameraType": 0,
                "distortionCoeff": [
                    -1.1144567728042603,
                    0.4457603096961975,
                    -0.00030109440558589995,
                    0.000643925741314888,
                    1.2683645486831665,
                    -1.1590780019760132,
                    0.5456246733665466,
                    1.2029682397842407,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0
                ],
                "extrinsics": {
                    "rotationMatrix": [
                        [
                            0.9946930408477783,
                            0.10237327218055725,
                            -0.010269088670611382
                        ],
                        [
                            -0.1015079990029335,
                            0.9927471876144409,
                            0.06441415846347809
                        ],
                        [
                            0.016788896173238754,
                            -0.06302992254495621,
                            0.9978703856468201
                        ]
                    ],
                    "specTranslation": {
                        "x": 9.0,
                        "y": 0.0,
                        "z": 0.0
                    },
                    "toCameraSocket": 1,
                    "translation": {
                        "x": -8.774820327758789,
                        "y": 0.4384150207042694,
                        "z": -0.33474013209342957
                    }
                },
                "height": 800,
                "intrinsicMatrix": [
                    [
                        910.39697265625,
                        0.0,
                        646.8530883789063
                    ],
                    [
                        0.0,
                        910.701904296875,
                        386.07025146484375
                    ],
                    [
                        0.0,
                        0.0,
                        1.0
                    ]
                ],
                "lensPosition": 0,
                "specHfovDeg": 75.0,
                "width": 1280
            }
        ]
    ],
    "hardwareConf": "",
    "imuExtrinsics": {
        "rotationMatrix": [],
        "specTranslation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "toCameraSocket": -1,
        "translation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        }
    },
    "miscellaneousData": [],
    "productName": "",
    "stereoRectificationData": {
        "leftCameraSocket": 1,
        "rectifiedRotationLeft": [
            [
                0.9980300664901733,
                -0.049864426255226135,
                0.03807265684008598
            ],
            [
                0.051083534955978394,
                0.9981896281242371,
                -0.03174852952361107
            ],
            [
                -0.036420609802007675,
                0.03363087400794029,
                0.9987704753875732
            ]
        ],
        "rectifiedRotationRight": [
            [
                0.9984344244003296,
                0.05026911944150925,
                0.024530746042728424
            ],
            [
                -0.05104481801390648,
                0.998180627822876,
                0.03209204599261284
            ],
            [
                -0.02287287451326847,
                -0.03329396992921829,
                0.9991838335990906
            ]
        ],
        "rightCameraSocket": 2
    },
    "version": 7
}

jakaskerl

ShivamSharma DepthAI version: 2.21.2.0

Update depthai to latest version. Use install_requirements.py.

ShivamSharma Update: Not getting the second error above probably because I flashed without -nic.

-nic will try to flash protected fields in eeprom. Will be fixed on our side.

ShivamSharma Btw, this time the saved calibration file with -nic looks correct, it is just not flashed because of the error I pasted above.

Great! You can try to swap the calibration values (not the board config/board name/etc.. fields) on the non working config and try the calibration by loading it first: https://docs.luxonis.com/software/depthai/examples/calibration_load/.

Thanks,
Jaka

ShivamSharma

jakaskerl

I did python3 install_requirements.py on the develop branch, but still, the version is same and this script installed numpy 2 which was just released so I will have to downgrade numpy now.

(venv) shivam157@ubuntu:~/depthai$ python3 install_requirements.py
pip 24.0 from /home/shivam157/depthai-python/venv/lib/python3.10/site-packages/pip (python 3.10)
Requirement already satisfied: pip in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (24.0)
Collecting pip
  Using cached pip-24.1-py3-none-any.whl.metadata (3.6 kB)
Using cached pip-24.1-py3-none-any.whl (1.8 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 24.0
    Uninstalling pip-24.0:
      Successfully uninstalled pip-24.0
Successfully installed pip-24.1
WARNING: Skipping opencv-python as it is not installed.
Found existing installation: opencv-contrib-python 4.5.5.62
Uninstalling opencv-contrib-python-4.5.5.62:
  Successfully uninstalled opencv-contrib-python-4.5.5.62
Found existing installation: depthai 2.24.0.0
Uninstalling depthai-2.24.0.0:
  Successfully uninstalled depthai-2.24.0.0
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple, https://artifacts.luxonis.com/artifactory/luxonis-depthai-data-local/wheels/, https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local/
Ignoring pyqt5: markers 'platform_machine != "armv6l" and platform_machine != "armv7l" and platform_machine != "aarch64" and platform_machine != "arm64"' don't match your environment
Requirement already satisfied: requests==2.26.0 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from -r requirements.txt (line 1)) (2.26.0)
Requirement already satisfied: numpy>=1.21.4 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from -r requirements.txt (line 3)) (1.26.4)
Collecting numpy>=1.21.4 (from -r requirements.txt (line 3))
  Using cached numpy-2.0.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (62 kB)
Collecting opencv-contrib-python==4.5.5.62 (from -r requirements.txt (line 4))
  Using cached opencv_contrib_python-4.5.5.62-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (18 kB)
Requirement already satisfied: depthai-sdk==1.9.4 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from -r requirements.txt (line 5)) (1.9.4)
Collecting depthai==2.21.2.0 (from -r requirements.txt (line 9))
  Using cached depthai-2.21.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (8.7 kB)
Requirement already satisfied: Qt.py in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from -r requirements.txt (line 10)) (1.4.1)
Requirement already satisfied: scipy in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from -r requirements.txt (line 11)) (1.13.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from requests==2.26.0->-r requirements.txt (line 1)) (1.26.18)
Requirement already satisfied: certifi>=2017.4.17 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from requests==2.26.0->-r requirements.txt (line 1)) (2024.6.2)
Requirement already satisfied: charset-normalizer~=2.0.0 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from requests==2.26.0->-r requirements.txt (line 1)) (2.0.12)
Requirement already satisfied: idna<4,>=2.5 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from requests==2.26.0->-r requirements.txt (line 1)) (3.7)
Requirement already satisfied: blobconverter>=1.2.8 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from depthai-sdk==1.9.4->-r requirements.txt (line 5)) (1.4.3)
Requirement already satisfied: pytube>=12.1.0 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from depthai-sdk==1.9.4->-r requirements.txt (line 5)) (15.0.0)
Requirement already satisfied: PyTurboJPEG==1.6.4 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from depthai-sdk==1.9.4->-r requirements.txt (line 5)) (1.6.4)
Requirement already satisfied: marshmallow==3.17.0 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from depthai-sdk==1.9.4->-r requirements.txt (line 5)) (3.17.0)
Requirement already satisfied: distinctipy in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from depthai-sdk==1.9.4->-r requirements.txt (line 5)) (1.3.4)
Requirement already satisfied: xmltodict in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from depthai-sdk==1.9.4->-r requirements.txt (line 5)) (0.13.0)
Requirement already satisfied: packaging>=17.0 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from marshmallow==3.17.0->depthai-sdk==1.9.4->-r requirements.txt (line 5)) (24.0)
Requirement already satisfied: types-PySide2 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from Qt.py->-r requirements.txt (line 10)) (5.15.2.1.7)
Requirement already satisfied: PyYAML in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from blobconverter>=1.2.8->depthai-sdk==1.9.4->-r requirements.txt (line 5)) (6.0.1)
Using cached opencv_contrib_python-4.5.5.62-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (44.8 MB)
Using cached depthai-2.21.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (10.3 MB)
Using cached numpy-2.0.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (13.9 MB)
Installing collected packages: numpy, depthai, opencv-contrib-python
  Attempting uninstall: numpy
    Found existing installation: numpy 1.26.4
    Uninstalling numpy-1.26.4:
      Successfully uninstalled numpy-1.26.4
Successfully installed depthai-2.21.2.0 numpy-2.0.0 opencv-contrib-python-4.5.5.62
Ignoring open3d: markers 'platform_machine != "armv6l" and platform_machine != "armv7l" and python_version < "3.9" and platform_machine != "aarch64"' don't match your environment
Requirement already satisfied: ffmpy3==0.2.4 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from -r requirements-optional.txt (line 2)) (0.2.4)
Requirement already satisfied: pyusb==1.2.1 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from -r requirements-optional.txt (line 3)) (1.2.1)
Requirement already satisfied: sentry-sdk==1.5.1 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from -r requirements-optional.txt (line 4)) (1.5.1)
Requirement already satisfied: mcap>=0.0.10 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from -r requirements-optional.txt (line 7)) (1.1.1)
Requirement already satisfied: mcap-ros1-support==0.0.8 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from -r requirements-optional.txt (line 8)) (0.0.8)
Requirement already satisfied: rosbags==0.9.11 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from -r requirements-optional.txt (line 9)) (0.9.11)
Requirement already satisfied: urllib3>=1.10.0 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from sentry-sdk==1.5.1->-r requirements-optional.txt (line 4)) (1.26.18)
Requirement already satisfied: certifi in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from sentry-sdk==1.5.1->-r requirements-optional.txt (line 4)) (2024.6.2)
Requirement already satisfied: lz4 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from rosbags==0.9.11->-r requirements-optional.txt (line 9)) (4.3.3)
Requirement already satisfied: numpy in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from rosbags==0.9.11->-r requirements-optional.txt (line 9)) (2.0.0)
Requirement already satisfied: ruamel.yaml in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from rosbags==0.9.11->-r requirements-optional.txt (line 9)) (0.18.6)
Requirement already satisfied: zstandard in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from rosbags==0.9.11->-r requirements-optional.txt (line 9)) (0.22.0)
Requirement already satisfied: ruamel.yaml.clib>=0.2.7 in /home/shivam157/depthai-python/venv/lib/python3.10/site-packages (from ruamel.yaml->rosbags==0.9.11->-r requirements-optional.txt (line 9)) (0.2.8)
(venv) shivam157@ubuntu:~/depthai$ python3 calibrate.py -s 3.8 -brd OAK-FFC-3P.json -nx 13 -ny 7 -m process -nic

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last):  File "/home/shivam157/depthai/calibrate.py", line 17, in <module>
    import cv2
  File "/home/shivam157/depthai-python/venv/lib/python3.10/site-packages/cv2/__init__.py", line 8, in <module>
    from .cv2 import *
AttributeError: _ARRAY_API not found
Traceback (most recent call last):
  File "/home/shivam157/depthai/calibrate.py", line 17, in <module>
    import cv2
  File "/home/shivam157/depthai-python/venv/lib/python3.10/site-packages/cv2/__init__.py", line 8, in <module>
    from .cv2 import *
ImportError: numpy.core.multiarray failed to import
(venv) shivam157@ubuntu:~/depthai$

Also, how to fix the following after installing OpenCV again to work with Numpy2:

Installing collected packages: opencv-python
Successfully installed opencv-python-4.10.0.84
(venv) shivam157@ubuntu:~/depthai$ python3 calibrate.py -s 3.8 -brd OAK-FFC-3P.json -nx 13 -ny 7 -m process -nic
Traceback (most recent call last):
  File "/home/shivam157/depthai/calibrate.py", line 1207, in <module>
    Main().run()
  File "/home/shivam157/depthai/calibrate.py", line 351, in __init__
    self.aruco_dictionary = cv2.aruco.Dictionary_get(
AttributeError: module 'cv2.aruco' has no attribute 'Dictionary_get'. Did you mean: 'Dictionary'?
(venv) shivam157@ubuntu:~/depthai$

Update:
The develop branch install_requirements.py uses the 2.21.20 depth AI version. So I edited it to 2.24.0.0, removed Numpy 2 because it was giving an error, and installed Numpy 1.21.4. You should fix this file as well.

Edit 1:

Still, after loading the calibration from the script in depthai-python, the flashed calibration is incorrect when I run the calibration reader file. How will loading calibration help me? The problem is in ROS 2 calibration info and depth.

jakaskerl

Hi ShivamSharma

ShivamSharma You should fix this file as well.

True. Should be fixed with the next SDK version. Though by install_requirements.py I meant the depthai-python one. Sorry for the confusion.

ShivamSharma Still, after loading the calibration from the script in depthai-python, the flashed calibration is incorrect when I run the calibration reader file. How will loading calibration help me? The problem is in ROS 2 calibration info and depth.

The idea is to check with a loaded calibration (not flashed) if the depth looks correct without changing the sockets. Then, once everything looks ok, you can flash the calibration and the ROS counterpart should work as well without any socket shenanigans.

Thanks,
Jaka

ShivamSharma

jakaskerl I checked the loaded calibration from the calibration file I got when I used -nic and switched the calib values, but no window opened when I ran the calibration_load.py script.

When I paste the incorrectly flashed calibration for the loading, then I can see a window, but the depth is pretty bad. Now even the Depthai library script depth_preview,py is giving bad depth. I switched left and right in it, but I still have bad depth. I didn't move the camera setup, so I don't know what changed. The cameras are clamped.

How do I debug this?

jakaskerl

Hi @ShivamSharma
Make sure these fields are flashed correctly (example for FFC-4P):

"batchName": "",
"batchTime": 1666854146,
"boardConf": "IR-C00M00-00",
"boardCustom": "",
"boardName": "DD2090",
"boardOptions": 3,
"boardRev": "R5M1E5",
"cameraData": [],
"deviceName": "",
"hardwareConf": "F0-FV00-BC001",

Thanks,
Jaka

ShivamSharma

jakaskerl

But the -nic option gives empty values for those fields. I don't understand. I copied and pasted with and without the value
of these fields from the flashed calib file, but both times no window opened.

jakaskerl

ShivamSharma
Can I see the full calibration file with the protected fields please?

Thanks,
Jaka

ShivamSharma

The following is the calibration file with the -nic option :

{
    "batchName": "",
    "batchTime": 0,
    "boardConf": "",
    "boardCustom": "",
    "boardName": "Custom FFC",
    "boardOptions": 0,
    "boardRev": "R1M0E3",
    "cameraData": [
        [
            1,
            {
                "cameraType": 0,
                "distortionCoeff": [
                    -1.8108996152877808,
                    0.5892674922943115,
                    0.00020837810006923974,
                    0.0008147237240336835,
                    -0.4694308340549469,
                    -1.8531672954559326,
                    0.7104603052139282,
                    -0.5547073483467102,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0
                ],
                "extrinsics": {
                    "rotationMatrix": [],
                    "specTranslation": {
                        "x": 0.0,
                        "y": 0.0,
                        "z": 0.0
                    },
                    "toCameraSocket": -1,
                    "translation": {
                        "x": 0.0,
                        "y": 0.0,
                        "z": 0.0
                    }
                },
                "height": 800,
                "intrinsicMatrix": [
                    [
                        906.6533813476563,
                        0.0,
                        625.2329711914063
                    ],
                    [
                        0.0,
                        906.753662109375,
                        399.9358825683594
                    ],
                    [
                        0.0,
                        0.0,
                        1.0
                    ]
                ],
                "lensPosition": 0,
                "specHfovDeg": 75.0,
                "width": 1280
            }
        ],
        [
            2,
            {
                "cameraType": 0,
                "distortionCoeff": [
                    -1.1144567728042603,
                    0.4457603096961975,
                    -0.00030109440558589995,
                    0.000643925741314888,
                    1.2683645486831665,
                    -1.1590780019760132,
                    0.5456246733665466,
                    1.2029682397842407,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0
                ],
                "extrinsics": {
                    "rotationMatrix": [
                        [
                            0.9946930408477783,
                            0.10237327218055725,
                            -0.010269088670611382
                        ],
                        [
                            -0.1015079990029335,
                            0.9927471876144409,
                            0.06441415846347809
                        ],
                        [
                            0.016788896173238754,
                            -0.06302992254495621,
                            0.9978703856468201
                        ]
                    ],
                    "specTranslation": {
                        "x": 9.0,
                        "y": 0.0,
                        "z": 0.0
                    },
                    "toCameraSocket": 1,
                    "translation": {
                        "x": -8.774820327758789,
                        "y": 0.4384150207042694,
                        "z": -0.33474013209342957
                    }
                },
                "height": 800,
                "intrinsicMatrix": [
                    [
                        910.39697265625,
                        0.0,
                        646.8530883789063
                    ],
                    [
                        0.0,
                        910.701904296875,
                        386.07025146484375
                    ],
                    [
                        0.0,
                        0.0,
                        1.0
                    ]
                ],
                "lensPosition": 0,
                "specHfovDeg": 75.0,
                "width": 1280
            }
        ]
    ],
    "deviceName": "",
    "hardwareConf": "",
    "housingExtrinsics": {
        "rotationMatrix": [],
        "specTranslation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "toCameraSocket": -1,
        "translation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        }
    },
    "imuExtrinsics": {
        "rotationMatrix": [],
        "specTranslation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "toCameraSocket": -1,
        "translation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        }
    },
    "miscellaneousData": [],
    "productName": "",
    "stereoEnableDistortionCorrection": false,
    "stereoRectificationData": {
        "leftCameraSocket": 1,
        "rectifiedRotationLeft": [
            [
                0.9980300664901733,
                -0.049864426255226135,
                0.03807265684008598
            ],
            [
                0.051083534955978394,
                0.9981896281242371,
                -0.03174852952361107
            ],
            [
                -0.036420609802007675,
                0.03363087400794029,
                0.9987704753875732
            ]
        ],
        "rectifiedRotationRight": [
            [
                0.9984344244003296,
                0.05026911944150925,
                0.024530746042728424
            ],
            [
                -0.05104481801390648,
                0.998180627822876,
                0.03209204599261284
            ],
            [
                -0.02287287451326847,
                -0.03329396992921829,
                0.9991838335990906
            ]
        ],
        "rightCameraSocket": 2
    },
    "stereoUseSpecTranslation": true,
    "version": 7,
    "verticalCameraSocket": -1
}

jakaskerl

Hi ShivamSharma
I tried this .json file with calibration_load.py and experienced no issues (albeit the depth was bad, which was expected since the device is different). Do other examples work as expected (correct Opencv version, correct depthai version,..). I think the issue might be somewhere else.

Thanks,
Jaka

ShivamSharma

Okay, I tried the load calibration again today, and I saw the depth window open with the same file I pasted above. The depth was bad, even though the calibration is correct as far as I know. The following is the depth:

What should I do now?

Edit1:
The depth is still bad after switching left and right in the calibration load file. I don't think I changed the camera setup. Should I calibrate again just to be sure? A few days ago, switching left and right in the depth_preview file gave the correct depth, but that is not the case now.

Edit2:
I calibrated the camera again, and now depth_preview is better, but I keep seeing blue-colored shadows of the outline of objects. Why is this happening? Is this how it is supposed to work around the discontinuity in depth?

Edit3:

After switching left and right in the calibration_load.py and using the -nic calibration data, I got good depth. It looks like the following:

Do you still want me to switch numerical calibration values instead to make it work with ROS 2?

Edit 4:

I did what you asked. I switched the numeric calibration values in the -nic calibration file but got bad depth with and without switching the cameras in calibration_load.py. I have pasted the original and the modified below:

Original:

{
    "batchName": "",
    "batchTime": 0,
    "boardConf": "",
    "boardCustom": "",
    "boardName": "Custom FFC",
    "boardOptions": 0,
    "boardRev": "R1M0E3",
    "cameraData": [
        [
            1,
            {
                "cameraType": 0,
                "distortionCoeff": [
                    -0.7255496382713318,
                    5.1313700675964355,
                    -0.0002011057222262025,
                    0.0008841821108944714,
                    -6.754741668701172,
                    -0.7626256942749023,
                    5.142012119293213,
                    -6.697120189666748,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0
                ],
                "extrinsics": {
                    "rotationMatrix": [],
                    "specTranslation": {
                        "x": 0.0,
                        "y": 0.0,
                        "z": 0.0
                    },
                    "toCameraSocket": -1,
                    "translation": {
                        "x": 0.0,
                        "y": 0.0,
                        "z": 0.0
                    }
                },
                "height": 800,
                "intrinsicMatrix": [
                    [
                        907.2047119140625,
                        0.0,
                        625.1134033203125
                    ],
                    [
                        0.0,
                        907.6209716796875,
                        397.6976013183594
                    ],
                    [
                        0.0,
                        0.0,
                        1.0
                    ]
                ],
                "lensPosition": 0,
                "specHfovDeg": 75.0,
                "width": 1280
            }
        ],
        [
            2,
            {
                "cameraType": 0,
                "distortionCoeff": [
                    -0.11288045346736908,
                    0.9440451264381409,
                    -0.0003182076325174421,
                    0.0005265712388791144,
                    -2.1388356685638428,
                    -0.15885838866233826,
                    1.0026153326034546,
                    -2.1278512477874756,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0
                ],
                "extrinsics": {
                    "rotationMatrix": [
                        [
                            0.9948962926864624,
                            0.10089320689439774,
                            -0.0013819887535646558
                        ],
                        [
                            -0.10070427507162094,
                            0.9937053918838501,
                            0.04907328262925148
                        ],
                        [
                            0.006324450485408306,
                            -0.04868365451693535,
                            0.9987941980361938
                        ]
                    ],
                    "specTranslation": {
                        "x": 9.0,
                        "y": 0.0,
                        "z": 0.0
                    },
                    "toCameraSocket": 1,
                    "translation": {
                        "x": -8.728154182434082,
                        "y": 0.45325255393981934,
                        "z": -0.024646950885653496
                    }
                },
                "height": 800,
                "intrinsicMatrix": [
                    [
                        907.8340454101563,
                        0.0,
                        646.8483276367188
                    ],
                    [
                        0.0,
                        908.2755737304688,
                        384.77850341796875
                    ],
                    [
                        0.0,
                        0.0,
                        1.0
                    ]
                ],
                "lensPosition": 0,
                "specHfovDeg": 75.0,
                "width": 1280
            }
        ]
    ],
    "deviceName": "",
    "hardwareConf": "",
    "housingExtrinsics": {
        "rotationMatrix": [],
        "specTranslation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "toCameraSocket": -1,
        "translation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        }
    },
    "imuExtrinsics": {
        "rotationMatrix": [],
        "specTranslation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "toCameraSocket": -1,
        "translation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        }
    },
    "miscellaneousData": [],
    "productName": "",
    "stereoEnableDistortionCorrection": false,
    "stereoRectificationData": {
        "leftCameraSocket": 1,
        "rectifiedRotationLeft": [
            [
                0.9986503720283508,
                -0.05185985937714577,
                0.0028200335800647736
            ],
            [
                0.05191338434815407,
                0.9983528256416321,
                -0.024425296112895012
            ],
            [
                -0.0015486960764974356,
                0.024538729339838028,
                0.9996976852416992
            ]
        ],
        "rectifiedRotationRight": [
            [
                0.99879390001297,
                0.049086324870586395,
                -0.0011084240395575762
            ],
            [
                -0.04904444143176079,
                0.9984954595565796,
                0.024524861946702003
            ],
            [
                0.0023105917498469353,
                -0.02444092184305191,
                0.9996985793113708
            ]
        ],
        "rightCameraSocket": 2
    },
    "stereoUseSpecTranslation": true,
    "version": 7,
    "verticalCameraSocket": -1
}

Modified:


{
    "batchName": "",
    "batchTime": 0,
    "boardConf": "",
    "boardCustom": "",
    "boardName": "Custom FFC",
    "boardOptions": 0,
    "boardRev": "R1M0E3",
    "cameraData": [
        [
            1,
            {
                "cameraType": 0,
                "distortionCoeff": [
                    -0.11288045346736908,
                    0.9440451264381409,
                    -0.0003182076325174421,
                    0.0005265712388791144,
                    -2.1388356685638428,
                    -0.15885838866233826,
                    1.0026153326034546,
                    -2.1278512477874756,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0
                ],
                "extrinsics": {
                    "rotationMatrix": [
                        [
                            0.9948962926864624,
                            0.10089320689439774,
                            -0.0013819887535646558
                        ],
                        [
                            -0.10070427507162094,
                            0.9937053918838501,
                            0.04907328262925148
                        ],
                        [
                            0.006324450485408306,
                            -0.04868365451693535,
                            0.9987941980361938
                        ]
                    ],
                    "specTranslation": {
                        "x": 9.0,
                        "y": 0.0,
                        "z": 0.0
                    },
                    "toCameraSocket": 2,
                    "translation": {
                        "x": -8.728154182434082,
                        "y": 0.45325255393981934,
                        "z": -0.024646950885653496
                    }
                },
                "height": 800,
                "intrinsicMatrix": [
                    [
                        907.8340454101563,
                        0.0,
                        646.8483276367188
                    ],
                    [
                        0.0,
                        908.2755737304688,
                        384.77850341796875
                    ],
                    [
                        0.0,
                        0.0,
                        1.0
                    ]
                ],
                "lensPosition": 0,
                "specHfovDeg": 75.0,
                "width": 1280
            }
        ],
        [
            2,
            {
                "cameraType": 0,
                "distortionCoeff": [
                    -0.7255496382713318,
                    5.1313700675964355,
                    -0.0002011057222262025,
                    0.0008841821108944714,
                    -6.754741668701172,
                    -0.7626256942749023,
                    5.142012119293213,
                    -6.697120189666748,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0
                ],
                "extrinsics": {
                    "rotationMatrix": [],
                    "specTranslation": {
                        "x": 0.0,
                        "y": 0.0,
                        "z": 0.0
                    },
                    "toCameraSocket": -1,
                    "translation": {
                        "x": 0.0,
                        "y": 0.0,
                        "z": 0.0
                    }
                },
                "height": 800,
                "intrinsicMatrix": [
                    [
                        907.2047119140625,
                        0.0,
                        625.1134033203125
                    ],
                    [
                        0.0,
                        907.6209716796875,
                        397.6976013183594
                    ],
                    [
                        0.0,
                        0.0,
                        1.0
                    ]
                ],
                "lensPosition": 0,
                "specHfovDeg": 75.0,
                "width": 1280
            }
        ]
    ],
    "deviceName": "",
    "hardwareConf": "",
    "housingExtrinsics": {
        "rotationMatrix": [],
        "specTranslation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "toCameraSocket": -1,
        "translation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        }
    },
    "imuExtrinsics": {
        "rotationMatrix": [],
        "specTranslation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "toCameraSocket": -1,
        "translation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        }
    },
    "miscellaneousData": [],
    "productName": "",
    "stereoEnableDistortionCorrection": false,
    "stereoRectificationData": {
        "leftCameraSocket": 1,
        "rectifiedRotationLeft": [
           [
                0.99879390001297,
                0.049086324870586395,
                -0.0011084240395575762
            ],
            [
                -0.04904444143176079,
                0.9984954595565796,
                0.024524861946702003
            ],
            [
                0.0023105917498469353,
                -0.02444092184305191,
                0.9996985793113708
            ]
        ],
        "rectifiedRotationRight": [
           [
                0.9986503720283508,
                -0.05185985937714577,
                0.0028200335800647736
            ],
            [
                0.05191338434815407,
                0.9983528256416321,
                -0.024425296112895012
            ],
            [
                -0.0015486960764974356,
                0.024538729339838028,
                0.9996976852416992
            ]
        ],
        "rightCameraSocket": 2
    },
    "stereoUseSpecTranslation": true,
    "version": 7,
    "verticalCameraSocket": -1
}

Edit :
@jakaskerl I tried the ROS 1 noetic container, and the depth was good. But the fps was around 10, and it was unsynchronized even after setting synchronization to true. So the bug is in the ROS 2 driver, and the hardware is faulty?

I can't use the ROS 1 topic with RTabmap because the camera information is giving a negative baseline, even though the depth in rviz is correct. Rtabmap works with positive baseline. Rtabmap assumes the left camera is the reference, but I see in the rostopic info that it is switched.

Also, the image in ROS 1 topic has switched left and right. I move my right arm and see left side move in the image not like a mirror.

jakaskerl

ShivamSharma

ShivamSharma ut I keep seeing blue-colored shadows of the outline of objects. Why is this happening?

This is normal and is a result of binocular occlusion, where one camera sees something the other does not (because the object is blocking it). This means the stereo depth can't be calculated there since stereo matching fails.

ShivamSharma Do you still want me to switch numerical calibration values instead to make it work with ROS 2?

No, if depth in the preview looks good without changing the code in any way, then leave it as it is. It is a problem of ROS.
cc @Luxonis-Adam

Thanks,
Jaka

Luxonis-Adam

Hi, DepthAI assumes different stereo cam order than OpenCV, you can toggle this behavior using a ROS parameter, as described here. Regarding socket order, it has some default parameters which can be overriden as well using parameters (cam_name_i_board_socket_id). Stereo node also has predefined socket numbers which can be overriden using parameters.

« Previous Page Next Page »