Hi,recently I wanted to convert the point cloud generated by Oak-D-Pro from the camera coordinate system to the world coordinate system with the rotation vector information output by the IMU, but I was using the Windows system and did not calibrate the IMU.I don't know how wrong the IMU data is now, but in fact, the coordinate transformation results are not ideal for me. Pls what should I do, I'm about to collapse.Please help me.

Hi @gifvgk ,
What exactly do you wish to accomplish? What exactly is wrong with the output? Your post doesn't help us much on debugging.

How to calibrate IMU with Oak D Pro on Windows system

@gifvgk, erik is asking you want does your Camera-IMU calibration data look like. You've given no data for which to work from. It would be very helpful if you could provide a ROSBAG or similar, or show plots where the IMU data is bad?

Sorry for not describing my problem in detail before.I took the quaternion of the IMU at a certain moment in time and converted it into a rotation matrix.

q=np.array([-0.019775,0.479309,0.003967,0.877441])
def quaternion_to_rotation_matrix(q): # x, y ,z ,w
rot_matrix = np.array(
[[1.0 - 2 * (q[1] * q[1] + q[2] * q[2]), 2 * (q[0] * q[1] - q[3] * q[2]), 2 * (q[3] * q[1] + q[0] * q[2])],
[2 * (q[0] * q[1] + q[3] * q[2]), 1.0 - 2 * (q[0] * q[0] + q[2] * q[2]), 2 * (q[1] * q[2] - q[3] * q[0])],
[2 * (q[0] * q[2] - q[3] * q[1]), 2 * (q[1] * q[2] + q[3] * q[0]), 1.0 - 2 * (q[0] * q[0] + q[1] * q[1])]],
dtype=q.dtype)
return rot_matrix

R = np.eye(4)
R[:3, :3] = quaternion_to_rotation_matrix(q)
print(R)

As a result, it was：

[[ 0.54049429 -0.02591829 0.84097384 0. ]

[-0.01199505 0.99918642 0.03850563 0. ]

[-0.84128763 -0.03089995 0.53974366 0. ]

[ 0. 0. 0. 1. ]]

I think the resulting rotation matrix should be an orthogonal identity matrix, but obviously, R is not.An orthogonal identity matrix can convert the point cloud to the world coordinate system.That's where I'm confused

    gifvgk

    Why should the rotation matrix be Identity? An identity rotation matrix means it has no attitude? (e.g. 0 roll, 0 pitch, 0 yaw)

      Sorry, I may have made a mistake with the transform relationship. My camera and IMU are not jointly calibrated, and I shouldn't use the IMU's rotation matrix directly on the camera coordinate system transformation.Maybe that's the key.

      Even if you jointly calibrated the Camera-IMU sensor system, the rotation matrix between them should never be perfectly at identity (it can be close but never exactly).

      What should be Identity (exactly) is if you have transformation matrices between all the sensors and you form a loop, then it should be identity. Example:

      Let:

      • T_LCam_IMU: Left camera to IMU 4x4 Transformation matrix
      • T_RCam_IMU: Right camera to IMU 4x4 Transformation matrix
      • T_LCam_RCam: Left camera to Right Camera 4x4 Transformation matrix

      If you did

      inv(T_LCam_IMU) * T_LCam_RCam *  T_RCam_IMU -> Should be Identity 4x4

      Thank you very much for your patience.The problem is that my oak d pro currently only has a joint calibration program on ubuntu, there is no windows, my programs are all on windows. Unable to get the conversion matrix for camera and IMU.