Hi, it looks like the optical frames (at least for the left and right cameras) match to what I had drawn in my previous message. What is confusing is the IMU coordinates. Yes I have seen the picture and marking on the PCB but I believe they do not match the actual accelerometer values that are read from the board. I think there is a 90 degree rotation around the Z-axis difference! ie accelerometer values suggest that Y-axis is up towards the cameras and not to the side as shown on the board! Can you check for this please (I have a OAK-D-IOT-75 board)
Thanks

Thanks for this. What I really like to know is the orientation of the coordinate frames of camera, IMU and the board itself.
Can anyone from Luxonis help?
Also I can't find OAK-D-IOT-75 on the site anymore!! Has this been removed?

GergelySzabolcs The default coordinate frame reference is used.

That doesn't seem to be the case for the OAK-D-IOT-75 board. By monitoring accelerometer output and rotating the board, it seems to me that there is a 90 degrees rotation about the Z axis compared to the default which is also printed on the PCB. Can anyone confirm this or otherwise?

Thanks for your comments.
Regarding Kalibr output, I do realize that the errors are more than I like to see, but I highly doubt that these errors result in large extrinsics errors like 90 degrees error in rotations! We can see that even with rather high calibration errors, distances between cameras and IMU are estimated to be very close to what I can measure with calipers on the PCB.

In any case, can you comment on the picture in my earlier post on coordinate frames orientations of cameras and IMU and what the coordinate shown on the PCB relates to?
Thanks

Thanks for the quick reply. Sorry I completly forgot to mention that I am asking about the WiFi verion of OAK-D (https://docs.luxonis.com/projects/hardware/en/latest/pages/DM1098OBC.html)

I couldn't find info on the IMU transforms for this board and unfortunately the calibration strored on its EEPROM doesn't have info on IMU either. I have dumped the caliberation info stored in EEPROM and copied to the end of this post.

I then used Kalibr, a calibration tool, and calibrated the board. I have copied the output from that calibration at the end of this post also. It includes all the information that I needed. I did notice however that the coordinate frame markings on the PCB don't match the IMU coordinate frame (there is a 90 degrees rotation around the Z-Axis- output from the Kalibr )! Usually board coordinates match the IMU (accelerometer) coordinates but this is not the case here, unless I am missing something, so not sure what the board marking refers to. Any pointers?

What I can't understand also is that looking at the rotation parameters from the Kalibr output for the camera-imu suggests that there is a 90 degrees rotation around the Z-axis, but from looking at camera images and accelerometer output I expect something like the picture below (board viewed from the heatsink side - back of the cameras) which suggest there should be a 180 degree rotation between the cameras and IMU. Any idea what I am missing here? (and sorry about a very long question

EEPROM Calibration data:
{
"boardName": "DM1098OAKD_WIFI",
"boardRev": "R0M0E0",
"cameraData": [
[
2,
{
"cameraType": 0,
"distortionCoeff": [
-5.259305477142334,
17.186681747436523,
0.0001805080974008888,
-0.0008773330482654274,
-16.840999603271484,
-5.310172080993652,
17.374258041381836,
-17.015090942382813,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0
],
"extrinsics": {
"rotationMatrix": [
[
0.99930739402771,
-0.037021756172180176,
0.0037524327635765076
],
[
0.03705490007996559,
0.999271035194397,
-0.009185632690787315
],
[
-0.003409628989174962,
0.009318316355347633,
0.9999507665634155
]
],
"specTranslation": {
"x": 3.750000238418579,
"y": 0.0,
"z": 0.0
},
"toCameraSocket": 0,
"translation": {
"x": 3.7746572494506836,
"y": -0.06653548777103424,
"z": -0.018930474296212196
}
},
"height": 800,
"intrinsicMatrix": [
[
856.5543212890625,
0.0,
638.7073974609375
],
[
0.0,
857.0066528320313,
407.7195129394531
],
[
0.0,
0.0,
1.0
]
],
"lensPosition": 0,
"specHfovDeg": 71.86000061035156,
"width": 1280
}
],
[
1,
{
"cameraType": 0,
"distortionCoeff": [
-6.351046085357666,
21.93744659423828,
-0.0008764712256379426,
-0.0009463315946049988,
-24.2668514251709,
-6.391645908355713,
22.08717918395996,
-24.405529022216797,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0
],
"extrinsics": {
"rotationMatrix": [
[
0.9999349117279053,
0.004014940001070499,
-0.010680022649466991
],
[
-0.004196074791252613,
0.9998468160629272,
-0.01699218526482582
],
[
0.01061016321182251,
0.017035892233252525,
0.999798595905304
]
],
"specTranslation": {
"x": -7.500000476837158,
"y": 0.0,
"z": 0.0
},
"toCameraSocket": 2,
"translation": {
"x": -7.540456295013428,
"y": -0.08471102267503738,
"z": -0.012086952105164528
}
},
"height": 800,
"intrinsicMatrix": [
[
851.7982788085938,
0.0,
636.7968139648438
],
[
0.0,
852.2836303710938,
391.3769836425781
],
[
0.0,
0.0,
1.0
]
],
"lensPosition": 0,
"specHfovDeg": 71.86000061035156,
"width": 1280
}
],
[
0,
{
"cameraType": 0,
"distortionCoeff": [
1.2997827529907227,
-9.799908638000488,
0.0010331615339964628,
0.0005897152586840093,
139.3450469970703,
0.9742816686630249,
-7.792778968811035,
134.64906311035156,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0
],
"extrinsics": {
"rotationMatrix": [],
"specTranslation": {
"x": 0.0,
"y": 0.0,
"z": 0.0
},
"toCameraSocket": -1,
"translation": {
"x": 0.0,
"y": 0.0,
"z": 0.0
}
},
"height": 1080,
"intrinsicMatrix": [
[
1482.8480224609375,
0.0,
963.8943481445313
],
[
0.0,
1480.971923828125,
547.895751953125
],
[
0.0,
0.0,
1.0
]
],
"lensPosition": 135,
"specHfovDeg": 68.7938003540039,
"width": 1920
}
]
],
"imuExtrinsics": {
"rotationMatrix": [],
"specTranslation": {
"x": 0.0,
"y": 0.0,
"z": 0.0
},
"toCameraSocket": -1,
"translation": {
"x": 0.0,
"y": 0.0,
"z": 0.0
}
},
"miscellaneousData": [],
"stereoRectificationData": {
"leftCameraSocket": 1,
"rectifiedRotationLeft": [
[
0.9998403787612915,
0.0152737470343709,
-0.009267695248126984
],
[
-0.015351753681898117,
0.9998468160629272,
-0.008405081927776337
],
[
0.009137898683547974,
0.008546016179025173,
0.9999217391014099
]
],
"rectifiedRotationRight": [
[
0.9999356269836426,
0.011233480647206306,
0.0016028438694775105
],
[
-0.011246662586927414,
0.9999008774757385,
0.008467010222375393
],
[
-0.001507570967078209,
-0.008484491147100925,
0.9999628663063049
]
],
"rightCameraSocket": 2
},
"version": 5
}

Kalibr Calibration output:

`Calibration results

Normalized Residuals

Reprojection error (cam0): mean 2.12060189795, median 1.48929995146, std: 1.89710118874
Reprojection error (cam1): mean 2.19886439886, median 1.51845668685, std: 2.01286747347
Gyroscope error (imu0): mean 8.11606922003, median 4.26710198746, std: 18.30840149
Accelerometer error (imu0): mean 8.47127610343, median 6.55178887856, std: 6.48271229279

Residuals

Reprojection error (cam0) [px]: mean 2.12060189795, median 1.48929995146, std: 1.89710118874
Reprojection error (cam1) [px]: mean 2.19886439886, median 1.51845668685, std: 2.01286747347
Gyroscope error (imu0) [rad/s]: mean 0.00619285937826, median 0.00325595578902, std: 0.0139699838425
Accelerometer error (imu0) [m/s^2]: mean 0.154718837242, median 0.119661447079, std: 0.118399836797

Transformation (cam0):

T_ci: (imu0 to cam0):
[[ 0.00123937 -0.99999265 0.00362736 0.04628487]
[ 0.99989731 0.00129102 0.01427216 0.02213576]
[-0.01427674 0.0036093 0.99989157 0.00541469]
[ 0. 0. 0. 1. ]]

T_ic: (cam0 to imu0):
[[ 0.00123937 0.99989731 -0.01427674 -0.02211355]
[-0.99999265 0.00129102 0.0036093 0.04623641]
[ 0.00362736 0.01427216 0.99989157 -0.00589792]
[ 0. 0. 0. 1. ]]

timeshift cam0 to imu0: (t_imu = t_cam + shift)
0.00754363609056

Transformation (cam1):

T_ci: (imu0 to cam1):
[[ 0.00600151 -0.99997661 -0.00328096 -0.0287159 ]
[ 0.99998112 0.00600579 -0.00129631 0.02211846]
[ 0.00131599 -0.00327311 0.99999378 0.0059507 ]
[ 0. 0. 0. 1. ]]

T_ic: (cam1 to imu0):
[[ 0.00600151 0.99998112 0.00131599 -0.02195354]
[-0.99997661 0.00600579 -0.00327311 -0.02882859]
[-0.00328096 -0.00129631 0.99999378 -0.0060162 ]
[ 0. 0. 0. 1. ]]

timeshift cam1 to imu0: (t_imu = t_cam + shift)
0.00742263606222

Baselines:

Baseline (cam0 to cam1):
[[ 0.9999648 0.00466307 -0.00697549 -0.0750646 ]
[-0.0047711 0.99986769 -0.01555097 0.00029066]
[ 0.00690206 0.0155837 0.99985474 -0.00012763]
[ 0. 0. 0. 1. ]]
baseline norm: 0.0750652671003 [m]


Gravity vector in target coords: [m/s^2]
[ 0.08546133 -9.77169456 0.82164753]

Calibration configuration

cam0

Camera model: pinhole
Focal length: [851.3498657713043, 851.7570181885181]
Principal point: [637.9843721932358, 390.38580129191007]
Distortion model: radtan
Distortion coefficients: [0.08213822656691498, -0.1661844004051627, -0.0011847924664171747, -0.0003649500443862518]
Type: aprilgrid
Tags:
Rows: 6
Cols: 6
Size: 0.02553086 [m]
Spacing 0.007659258 [m]

cam1

Camera model: pinhole
Focal length: [857.5234996166471, 857.1292929273301]
Principal point: [636.6089851586235, 404.33112957504926]
Distortion model: radtan
Distortion coefficients: [0.07025450356160097, -0.14698705024056916, -0.0005644109332028175, -0.0013258243506964478]
Type: aprilgrid
Tags:
Rows: 6
Cols: 6
Size: 0.02553086 [m]
Spacing 0.007659258 [m]

IMU configuration

IMU0:

Model: calibrated
Update rate: 600
Accelerometer:
Noise density: 0.000745622029494
Noise density (discrete): 0.0182639351324
Random walk: 0.00110616053066
Gyroscope:
Noise density: 3.1150846373e-05
Noise density (discrete): 0.000763036786697
Random walk: 1.56105578628e-05
T_i_b
[[ 1. 0. 0. 0.]
[ 0. 1. 0. 0.]
[ 0. 0. 1. 0.]
[ 0. 0. 0. 1.]]
time offset with respect to IMU0: 0.0 `

Hi, just wondering if there is any information on IMU and cameras coordinate frames avaulable? and their relations wrt each other? (short of trying to figure it out)
Thanks

Thanks for the reply. I can see the reasoning now, however in my logs the sequence numbers don't follow a 500/400 ratio- They are more like 256/245 for whatever reason. Anyway I only use the timestamps now for the IMU and to get rid of the 1-2ms jitter between sample times of accel and gyro, I use the timestamp of accel and linear-interpolate gyro values with timestamps just before and after the accel timestamp to get a gyro value at the exact time of the accel sample time. That's of course assuming that the gyro values change linearly with time. I hope this makes sense.
Thanks

Hi all,
I like to get synchronised IMU data and stereo images from the OAK-D hardware.
I am using 30fps left and right for the stereo images at 640x400 and 500Hz for the IMU. I am checking timestamps on both stereo images, accel and gyro in an IMU packet on the host side and ensure that they line up. I have noticed that even though accel and gyro timestamps are pretty close (within a few miliseconds) their sequence numbers are quite different sometimes by 20 counts or so- which should result in a timestamp diffence of 40ms or so. For instance I might receive a very first IMU packet, with sequence number of 20 for accel and seuence number of 0 for gyro but their respective timestamps are within 1 milisecond!
Do sequence numbers in IMU suppose to line up with timestamps?
Also any pointers as to the best method for ensuing synchronisation of gyro, accel, left and right images other than checking and comparing timestamp associated with each one of them?

Thanks for clarification.