IMU and camera coordinate frames?

Shaadee

Hi, just wondering if there is any information on IMU and cameras coordinate frames avaulable? and their relations wrt each other? (short of trying to figure it out)
Thanks

luxonis-Sachin

Shaadee Sorry you couldn't find this on the web. I am assuming you are asking for OAK-D. Let me know if you need it for different modules. We will update this on the web too.

luxonis-Sachin

as of the camera coordinates. it can be found in this example.

Shaadee

Thanks for the quick reply. Sorry I completly forgot to mention that I am asking about the WiFi verion of OAK-D (https://docs.luxonis.com/projects/hardware/en/latest/pages/DM1098OBC.html)

I couldn't find info on the IMU transforms for this board and unfortunately the calibration strored on its EEPROM doesn't have info on IMU either. I have dumped the caliberation info stored in EEPROM and copied to the end of this post.

I then used Kalibr, a calibration tool, and calibrated the board. I have copied the output from that calibration at the end of this post also. It includes all the information that I needed. I did notice however that the coordinate frame markings on the PCB don't match the IMU coordinate frame (there is a 90 degrees rotation around the Z-Axis- output from the Kalibr )! Usually board coordinates match the IMU (accelerometer) coordinates but this is not the case here, unless I am missing something, so not sure what the board marking refers to. Any pointers?

What I can't understand also is that looking at the rotation parameters from the Kalibr output for the camera-imu suggests that there is a 90 degrees rotation around the Z-axis, but from looking at camera images and accelerometer output I expect something like the picture below (board viewed from the heatsink side - back of the cameras) which suggest there should be a 180 degree rotation between the cameras and IMU. Any idea what I am missing here? (and sorry about a very long question 🙂

EEPROM Calibration data:
{ "boardName": "DM1098OAKD_WIFI", "boardRev": "R0M0E0", "cameraData": [ [ 2, { "cameraType": 0, "distortionCoeff": [ -5.259305477142334, 17.186681747436523, 0.0001805080974008888, -0.0008773330482654274, -16.840999603271484, -5.310172080993652, 17.374258041381836, -17.015090942382813, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ], "extrinsics": { "rotationMatrix": [ [ 0.99930739402771, -0.037021756172180176, 0.0037524327635765076 ], [ 0.03705490007996559, 0.999271035194397, -0.009185632690787315 ], [ -0.003409628989174962, 0.009318316355347633, 0.9999507665634155 ] ], "specTranslation": { "x": 3.750000238418579, "y": 0.0, "z": 0.0 }, "toCameraSocket": 0, "translation": { "x": 3.7746572494506836, "y": -0.06653548777103424, "z": -0.018930474296212196 } }, "height": 800, "intrinsicMatrix": [ [ 856.5543212890625, 0.0, 638.7073974609375 ], [ 0.0, 857.0066528320313, 407.7195129394531 ], [ 0.0, 0.0, 1.0 ] ], "lensPosition": 0, "specHfovDeg": 71.86000061035156, "width": 1280 } ], [ 1, { "cameraType": 0, "distortionCoeff": [ -6.351046085357666, 21.93744659423828, -0.0008764712256379426, -0.0009463315946049988, -24.2668514251709, -6.391645908355713, 22.08717918395996, -24.405529022216797, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ], "extrinsics": { "rotationMatrix": [ [ 0.9999349117279053, 0.004014940001070499, -0.010680022649466991 ], [ -0.004196074791252613, 0.9998468160629272, -0.01699218526482582 ], [ 0.01061016321182251, 0.017035892233252525, 0.999798595905304 ] ], "specTranslation": { "x": -7.500000476837158, "y": 0.0, "z": 0.0 }, "toCameraSocket": 2, "translation": { "x": -7.540456295013428, "y": -0.08471102267503738, "z": -0.012086952105164528 } }, "height": 800, "intrinsicMatrix": [ [ 851.7982788085938, 0.0, 636.7968139648438 ], [ 0.0, 852.2836303710938, 391.3769836425781 ], [ 0.0, 0.0, 1.0 ] ], "lensPosition": 0, "specHfovDeg": 71.86000061035156, "width": 1280 } ], [ 0, { "cameraType": 0, "distortionCoeff": [ 1.2997827529907227, -9.799908638000488, 0.0010331615339964628, 0.0005897152586840093, 139.3450469970703, 0.9742816686630249, -7.792778968811035, 134.64906311035156, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ], "extrinsics": { "rotationMatrix": [], "specTranslation": { "x": 0.0, "y": 0.0, "z": 0.0 }, "toCameraSocket": -1, "translation": { "x": 0.0, "y": 0.0, "z": 0.0 } }, "height": 1080, "intrinsicMatrix": [ [ 1482.8480224609375, 0.0, 963.8943481445313 ], [ 0.0, 1480.971923828125, 547.895751953125 ], [ 0.0, 0.0, 1.0 ] ], "lensPosition": 135, "specHfovDeg": 68.7938003540039, "width": 1920 } ] ], "imuExtrinsics": { "rotationMatrix": [], "specTranslation": { "x": 0.0, "y": 0.0, "z": 0.0 }, "toCameraSocket": -1, "translation": { "x": 0.0, "y": 0.0, "z": 0.0 } }, "miscellaneousData": [], "stereoRectificationData": { "leftCameraSocket": 1, "rectifiedRotationLeft": [ [ 0.9998403787612915, 0.0152737470343709, -0.009267695248126984 ], [ -0.015351753681898117, 0.9998468160629272, -0.008405081927776337 ], [ 0.009137898683547974, 0.008546016179025173, 0.9999217391014099 ] ], "rectifiedRotationRight": [ [ 0.9999356269836426, 0.011233480647206306, 0.0016028438694775105 ], [ -0.011246662586927414, 0.9999008774757385, 0.008467010222375393 ], [ -0.001507570967078209, -0.008484491147100925, 0.9999628663063049 ] ], "rightCameraSocket": 2 }, "version": 5 }

Kalibr Calibration output:

`Calibration results

Normalized Residuals

Reprojection error (cam0): mean 2.12060189795, median 1.48929995146, std: 1.89710118874
Reprojection error (cam1): mean 2.19886439886, median 1.51845668685, std: 2.01286747347
Gyroscope error (imu0): mean 8.11606922003, median 4.26710198746, std: 18.30840149
Accelerometer error (imu0): mean 8.47127610343, median 6.55178887856, std: 6.48271229279

Residuals

Reprojection error (cam0) [px]: mean 2.12060189795, median 1.48929995146, std: 1.89710118874
Reprojection error (cam1) [px]: mean 2.19886439886, median 1.51845668685, std: 2.01286747347
Gyroscope error (imu0) [rad/s]: mean 0.00619285937826, median 0.00325595578902, std: 0.0139699838425
Accelerometer error (imu0) [m/s^2]: mean 0.154718837242, median 0.119661447079, std: 0.118399836797

Transformation (cam0):

T_ci: (imu0 to cam0):
[[ 0.00123937 -0.99999265 0.00362736 0.04628487]
[ 0.99989731 0.00129102 0.01427216 0.02213576]
[-0.01427674 0.0036093 0.99989157 0.00541469]
[ 0. 0. 0. 1. ]]

T_ic: (cam0 to imu0):
[[ 0.00123937 0.99989731 -0.01427674 -0.02211355]
[-0.99999265 0.00129102 0.0036093 0.04623641]
[ 0.00362736 0.01427216 0.99989157 -0.00589792]
[ 0. 0. 0. 1. ]]

timeshift cam0 to imu0: (t_imu = t_cam + shift)
0.00754363609056

Transformation (cam1):

T_ci: (imu0 to cam1):
[[ 0.00600151 -0.99997661 -0.00328096 -0.0287159 ]
[ 0.99998112 0.00600579 -0.00129631 0.02211846]
[ 0.00131599 -0.00327311 0.99999378 0.0059507 ]
[ 0. 0. 0. 1. ]]

T_ic: (cam1 to imu0):
[[ 0.00600151 0.99998112 0.00131599 -0.02195354]
[-0.99997661 0.00600579 -0.00327311 -0.02882859]
[-0.00328096 -0.00129631 0.99999378 -0.0060162 ]
[ 0. 0. 0. 1. ]]

timeshift cam1 to imu0: (t_imu = t_cam + shift)
0.00742263606222

Baselines:

Baseline (cam0 to cam1):
[[ 0.9999648 0.00466307 -0.00697549 -0.0750646 ]
[-0.0047711 0.99986769 -0.01555097 0.00029066]
[ 0.00690206 0.0155837 0.99985474 -0.00012763]
[ 0. 0. 0. 1. ]]
baseline norm: 0.0750652671003 [m]

Gravity vector in target coords: [m/s^2]
[ 0.08546133 -9.77169456 0.82164753]

Calibration configuration

cam0

Camera model: pinhole
Focal length: [851.3498657713043, 851.7570181885181]
Principal point: [637.9843721932358, 390.38580129191007]
Distortion model: radtan
Distortion coefficients: [0.08213822656691498, -0.1661844004051627, -0.0011847924664171747, -0.0003649500443862518]
Type: aprilgrid
Tags:
Rows: 6
Cols: 6
Size: 0.02553086 [m]
Spacing 0.007659258 [m]

cam1

Camera model: pinhole
Focal length: [857.5234996166471, 857.1292929273301]
Principal point: [636.6089851586235, 404.33112957504926]
Distortion model: radtan
Distortion coefficients: [0.07025450356160097, -0.14698705024056916, -0.0005644109332028175, -0.0013258243506964478]
Type: aprilgrid
Tags:
Rows: 6
Cols: 6
Size: 0.02553086 [m]
Spacing 0.007659258 [m]

IMU configuration

IMU0:

Model: calibrated
Update rate: 600
Accelerometer:
Noise density: 0.000745622029494
Noise density (discrete): 0.0182639351324
Random walk: 0.00110616053066
Gyroscope:
Noise density: 3.1150846373e-05
Noise density (discrete): 0.000763036786697
Random walk: 1.56105578628e-05
T_i_b
[[ 1. 0. 0. 0.]
[ 0. 1. 0. 0.]
[ 0. 0. 1. 0.]
[ 0. 0. 0. 1.]]
time offset with respect to IMU0: 0.0 `

luxonis-Sachin

Shaadee you couldn't find the IMU calibration in the EEPROM because we are not calibrating that as of now.

And as of Kalibr calibration you have done I am seeing lot of reprojection error of 2+ for cameras which should be less than 1. And as for gyro and Accel also looks like the error is pretty high and your results might be wrong in terms of that also.

Shaadee

Thanks for your comments.
Regarding Kalibr output, I do realize that the errors are more than I like to see, but I highly doubt that these errors result in large extrinsics errors like 90 degrees error in rotations! We can see that even with rather high calibration errors, distances between cameras and IMU are estimated to be very close to what I can measure with calipers on the PCB.

In any case, can you comment on the picture in my earlier post on coordinate frames orientations of cameras and IMU and what the coordinate shown on the PCB relates to?
Thanks

David

Hi Shaadee

Please see the attached screen for the distances between cameras and IMU.

Shaadee

Thanks for this. What I really like to know is the orientation of the coordinate frames of camera, IMU and the board itself.
Can anyone from Luxonis help?
Also I can't find OAK-D-IOT-75 on the site anymore!! Has this been removed?

luxonis-Sachin

Shaadee Here is an image representing the optical frames of the camera. And in the image shared by @David you can see the IMU orientation with x in down, y to the right and z pointing outwards from the PCB. Does this addess the question ?

David

Shaadee
You can find the OAK-D-IoT-75 on github here..
If you didn't find it on the store, I think we removed it because we will assemble those into enclosures in these days so they are headed back to our CM for reassembly.

WRT IMU orientation I am suspecting that the orientation determined by the silk screen is correct:

Shaadee

Hi, it looks like the optical frames (at least for the left and right cameras) match to what I had drawn in my previous message. What is confusing is the IMU coordinates. Yes I have seen the picture and marking on the PCB but I believe they do not match the actual accelerometer values that are read from the board. I think there is a 90 degree rotation around the Z-axis difference! ie accelerometer values suggest that Y-axis is up towards the cameras and not to the side as shown on the board! Can you check for this please (I have a OAK-D-IOT-75 board)
Thanks

luxonis-Sachin

Shaadee Thanks for the feedback. @David can you cross check how IMU is mounted and second confirm it here ?

koonpl

I asked about setting IMU's static calibration record or the Sensor Orientation Record here, oak-d-bno085-imu-coordinate-frame. Unfortunately, this capabiility is not currently exposed in the IMU node. I was also told the static calibration record has not been set. Therefore, the BNO085 chip is still reporting the default configuration reference of East, North, Up, This means when it is level and the Y axis is pointing towards magnetic north the rotation vector will be ll zeros. The image of the OAK-D circuit board shows the chip is rotated 90 degrees clockwise from the default orientation so it is in physical South, East, Up configuration. I am basing this off of the dot in the image above. Is there a wish list for depthai features? Can adding the functions to set the configuration records on the BNO085 to the list?

GergelySzabolcs

koonpl

You can open a feature request here.
At the moment we are focusing on higher priority tasks.

Note that transformation can also be applied on the host side.

koonpl

Thanks for the link and answering my questions. I really think being able to call the tare functions will make the IMU more usable. With the tare, one can place the camera in a know orientation then zero its axes so it is aligned with the larger system to which it is attached. I will add the feature requests to the list.

GergelySzabolcs

koonpl

I agree that it would be nice, I briefly tried to enable it but it didn't work, the IMU became unresponsice, will have to debug it once I get there task-wise, until then I suggest applying the transformation of the coordinate system on the host.