getImuToCameraExtrinsics error

Dong_Liu

both calibData.getImuToCameraExtrinsics(dai::CameraBoardSocket::RIGHT, false) and calibData.getImuToCameraExtrinsics(dai::CameraBoardSocket::RIGHT, true) got

terminate called after throwing an instance of 'std::runtime_error'
what(): IMU calibration data is not available on device yet.
Is this going to be supported?

Thanks!

luxonis-Sachin

Dong_Liu We don't have IMU calibration inPlace during factory calibration. Hence it is empty. We will be adding IMU calibration support in the future.

Dong_Liu

getCameraToImuExtrinsics() same result

erik

Hello @Dong_Liu , so the error means that your device hasn't been calibrated. You can calibrate your device following this tutorial. Just curious, which device do you have (and where did you buy it from)?

Dong_Liu

Hi erik, the device is OAK-D, my co-worker purchased it. I'll ask him where did he ordered it.

But do you mean the device hasn't been calibrated for the IMU to LEFT, RIGHT and RGB cameras or it hasn't been calibrated at all? Because I can get other calibration info, such as camera's intrinsics, left right camera extrinsics etc just fine.

Thanks!

erik

I'm actually not sure about the IMU calibration. Asking Sachin for help here. He's from US so will probably respond later.

Dong_Liu

Hi erik, another question. The distortion coeff I got from

calibData.getDistortionCoefficients(dai::CameraBoardSocket::RIGHT);

is [9.6685 26.4033 0.000109294 -0.00131915 177.747 9.71128 24.7958 178.397 0 0 0 0 0 0]

Are these right values? they seem too big. And the images captured on the host side seem have no distortion. Actually if I use these value to undistort the images, I got some fisheye results.

Thanks!

luxonis-Sachin

Dong_Liu Those are fine. Since the equation is the ratio the big numbers don't make any difference.
We use a rational model from OpenCV and they provide large numbers instead of small floats. Hope this answers your doubt.

Dong_Liu

luxonis-Sachin Thanks for the info. Is there any instructions to calibrate IMU? I could not find how to calibrate imu from the video and calibrate.py on luxonis web site.

Thanks,

Dong_Liu

luxonis-Sachin Yes, I tried it again. The distort coefficients are correct. I didn't realize I need to use the whole vector to call cv.undistot(), I just used the first 4 elements, so it gave me wrong result.

Thanks!

luxonis-Sachin

Dong_Liu We don't have IMU calibration support yet since we had to prioritize over some CV features.
But if more users or use-cases request we can increase the priority on this I suppose.

Dong_Liu

luxonis-Sachin We want to use OAK-D to do VIO, it is important to have IMU to camera calibrated.

I'm trying to use kalibr to do it myself. But it is quite an old project. I managed to run it under docker. But the test pattern file and some config are hosted in a private google drive now it is closed 🙁

luxonis-Sachin

Dong_Liu you should be able to generate one using this command I suppose ?
Another reference from google groups here

Another Option would be ViCalib.

Dong_Liu

luxonis-Sachin Thanks! I was able to print out the AprilTag pattern. Now I'll need to hunt down the config yaml file. I think I saw it somehwere.

I'll report back if the calibration works or not.

By the way, that python freeze emacs when open it 🙁 Because it contains a VERY long line.

Thanks again.

luxonis-Sachin

Dong_Liu

By the way, that python freeze emacs when open it 🙁 Because it contains a VERY long line.

In Kalibr ?

Dong_Liu

luxonis-Sachin No, I tried to open the file using emacs, the emacs was frozen. It was actually running, because it is in python mode, just took forever to parse the very long line. So I had to kill it.

Dong_Liu

Another question. I set the update rate of IMU to 400HZ. But when I recorded the ros bag, there were only slightly more imu messages than image messages. Wii it be a problem?

$ rosbag info _{/foo/2021-09-29-13-49-27.bag}
path: /home/dliu/foo/2021-09-29-13-49-27.bag
version: 2.0
duration: 1:08s (68s)
start: Sep 29 2021 13:49:27.96 (1632937767.96)
end: Sep 29 2021 13:50:36.62 (1632937836.62)
size: 1006.9 MB
messages: 6195
compression: none [1030/1030 chunks]
types: sensor_msgs/Image [060021388200f6f0f447d0fcd9c64743]
sensor_msgs/Imu [6a62c6daae103f4ff57a132d6f95cec2]
topics: /cam0/image_raw 2059 msgs : sensor_msgs/Image
/cam1/image_raw 2059 msgs : sensor_msgs/Image
/imu0 2077 msgs : sensor_msgs/Imu

luxonis-Sachin

Dong_Liu Can you check the rostopic hz "topicname" and see the publishing rate on ros ?

Dong_Liu

luxonis-Sachin which topic? and who publishes it? My code only publish /cam{0|1}/image_raw and /imu0. The depthai ros examples stereo node also doesn't publish anything with hz?

Dong_Liu

Here is calibrated result of imu with the stereo camera set to 640x400 resolution
cam0:
T_cam_imu:

[0.03887680448008979, -0.9992016387989338, -0.009202124479843001, -0.0006451104397754213]
[0.99846239120464, 0.0384805735529119, 0.03990111287861405, -0.0028735167146291523]
[-0.03951515435032274, -0.010739202976226664, 0.9991612593050765, 0.000392439569927346]
[0.0, 0.0, 0.0, 1.0]
cam_overlaps: [1]
camera_model: pinhole
distortion_coeffs: [0.026474033567193075, -0.05969556492611341, -0.002817745414182798,
0.004924235269650544]
distortion_model: radtan
intrinsics: [400.0064170380642, 397.2638167232537, 329.5418926134828, 197.04540739777443]
resolution: [640, 400]
rostopic: /cam0/image_raw
timeshift_cam_imu: 0.0008590827955647041
cam1:
T_cam_imu:
[0.026880135407322037, -0.999628330381034, 0.004545263480889773, -0.07358932785004997]
[0.9989752544416461, 0.027027644643332516, 0.036303545808673066, -0.0017805337977785335]
[-0.036412900649807046, 0.003564761515223131, 0.999330472437225, -0.002939578961411555]
[0.0, 0.0, 0.0, 1.0]
T_cn_cnm1:
[0.9998334535948544, -0.011446106146888528, 0.014214490024609772, -0.07298279375678171]
[0.011496829079291606, 0.9999278124509344, -0.0034918201857753366, 0.0011015625375971258]
[-0.014173496170921451, 0.003654660197938216, 0.9998928719943602, -0.0033306182332533537]
[0.0, 0.0, 0.0, 1.0]
cam_overlaps: [0]
camera_model: pinhole
distortion_coeffs: [0.016727023328150355, -0.04497168337727961, -0.002975051937495909,
0.005200871168133805]
distortion_model: radtan
intrinsics: [399.1877435307988, 395.58676406318335, 331.88450604453317, 194.85854223018865]
resolution: [640, 400]
rostopic: /cam1/image_raw
timeshift_cam_imu: -0.0009904323881395447

The eprom calibration for the left camera's intrinsics are

421.241 0 317.216
0 421.378 202.988
0 0 1

Little bit different. But eprom left to right extrinsics are

0.999711 -0.0122152 0.0207223 -7.458
0.0123016 0.999916 -0.00405039 0.0796994
-0.0206711 0.00430414 0.999777 0.082101

The number at the end of first row -7.458 seems wrong. That number is the distance of two camera if I understand correctly. 7.458 meters is way to large. I think it should be 0.07458 meters, which is close to value I got 0.07359

Thanks!

luxonis-Sachin

Dong_Liu I would suggest ignoring the camera intrinsic from this and use the IMU extrinsics and other information for the usecase without affecting the EEPROM would be best.
As of Why. We calibrate the camera at full resolution in the factory so that it gets scaled well when used at different resolutions for stereo. So it might not be a good idea to mess with calibrated stereo I think.
Thoughts ?