Parameters for VIO using the IMU and the RGB camera

Hi Jaka and Erik, Thanks for the provided info.

On 2nd question, I appreciate the trust you put in a stranger to understand the github https://github.com/luxonis/depthai-hardware and filter out the required information… however, I am unable to see the forest through the trees here:

• First of all, can you confirm that the camera model is https://github.com/luxonis/depthai-hardware/tree/master/DM9098_OAK-D-Pro/3D_Models for the OAKD pro camera I have? (BTW some links (at least one) in the readme are broken…)

• How can I find the different files for the different IMU's Since the orientation is not the same there should be different files for this? Which branch or tag do I need for the BMI270?

• 
  How do I know what component is the IMU? And since they are just blocks how can I extract the orientation from it?

• In the post https://discuss.luxonis.com/d/383-imu-and-camera-coordinate-frames/8 @David provided a clear image :

  

  How can I get a similar image for the version with the BMI270 and the one with the BNO080085?

thanks in advance and I apologize for my lack of knowledge and skill for this.

YD

@erik thanks for this!

Indeed I seem to have the BMI270.
I also found this https://github.com/luxonis/depthai-boards/blob/adding_IMU_tmp/boards/OAK-D-PRO.json#L54-L86

But it seems not to correspond with the provided CAD…

Can you confirm the optical frames of all the cameras have the z -axis going out of the camera and the y -axis pointing down forming a right-hand reference frame, similar with :

In this case, it seems that the json file is giving a -7.5 cm(? since no unit is provided) in the x direction from the left camera to the right camera. However, using the drawing provided by you and the assumptions the optical frames are similar to the image provided above it should be positive 7.5 cm?

Moreover, by plotting the acceleration while moving the camera around I noticed the IMU reference frame has the x pointing upwards, the pointing towards the left camera and the z axis pointing forward (same direction as the z-axis of the optical frame. Making it a left-hand reference frame as opposed to the normal right-hand reference frames mostly used in engineering. Moreover while turning the camera around these principle axes the angular velocity is also captured as if it is a left-hand reference frame. This means that one cannot express the transformation between imu and the cameras using RPY angles only.
This also means that the translation to the left camera should be positive and not negative as shown in the json file.

Can you confirm that the IMU has a left hand reference frame as I measured?
Can you confirm the cameras are using a right hand reference frame?

Can somebody please provide the actual transformation and reference frames of the 3 cameras and the IMU? I have been searching for weeks now and I have not come close to finding the actual values.

Also, the functions of the API provided before do not yield results.

@erik and TomMunnecke ,

Chatgpt is correct and says the same as me. You cannot transfer it using only RPY angles (as provided in the .json file):

If your IMU has a left-hand coordinate system with coordinates $(x,y,z)$(x,y,z), you can convert it to a right-hand system with the transformation: $(x,y,−z)$(x,y,−z).

…
Once you have converted the coordinate systems, you can apply the relative extrinsic parameters (rotation and translation) to align the IMU and the camera.

…

Procedure

1. Convert the IMU coordinates to the right-hand system.

2. Construct the rotation matrix for pan, roll, and yaw.

3. Apply the rotation and translation to align the IMU with the camera using the extrinsic matrix.

Moreover, it also means you have to switch the sign of the angular velocity for the x and y directions. (Which Chatgpt did not mention, but I am sure it will give this information if you ask it specifically.)

But this is not the actual issue.
The actual issue is that if the BMI270 is a left-hand reference frame the information provided in the .json file is not enough to calculate the transformation since you need to know that it would be a left-hand reference frame, which is not mentioned in the .json file, which makes me doubt all the information in the .sjon file.

And since you just confirmed that the IMU uses a left-hand reference frame it means that somebody has to add some additional information to the .json file, and maybe check again all the information.

I hope you have confirmed this with a person with the actual knowledge to answer the question?
(I appologise if I am doubting the information you are providing, but your previous answer did not give me a lot of confidence in your abilities for this particular question. )