Calculating world coordinates from Oak-D (Pro) sensor data

Aingu7ae

Hello

I'd like to translate the pixel coordinates in the camera frame (Oak-D-Pro) to the coordinates in my room. I searched the web for inspiration and learned that quaternions might be the way to go but I'm quite not there to put the pieces together. I saw procedures where markers where placed at specific locations in the room to create a correspondence with the pixels in the camera frame. But since the Oak-D-Pro has an IMU unit, a sense of depth and a laser dot projector, I wonder if placing markers in the room can be avoided. I imagine that the only coordinates I need to know is the x-y-z position of the camera in the room and that the remaining necessary parameters for the calculations could be obtained from the IMU, FOV, depth etc.

It would be super nice and cool and generous if some bright minds could walk me, and potential other seekers, through the process of calibration and calculation. At some point in my journey I'd like to track objects in the camera frame and assign them real world coordinates.

erik

Hello Aingu7ae ,
That's exactly what OAK cameras are for, fusing the AI results with depth map! An example can be found here - so you can achieve this with about 100 LoC. DepthAI library will abstract all of these calculations and this will run in the firmware. You could also calculate spatial coordiantes on the host (instead of the device) as on the demo here - which also showcases the logic behind it. Thoughts?
Thanks, Erik

lss5fc0d

For orientation, I've had some success turning the ROTATION_VECTOR quaternion from the IMU into a rotation matrix, and applying that to the [y,x,z] coordinates given by YoloSpatialDetectionNetwork. This transforms from the camera's reference frame to the a frame where either x or y is North, and z is up and down relative to gravity. Using the pyquaternion library, something like:

camera_xyz=np.array([y,x,z]) #perhaps the IMU sensor is on sideways, or I have a minus sign wrong somewhere?
quat=Quaternion(rotation_vector_from_imu)
rot=quat.rotation_matrix
map_xyz=rot@camera_xyz

erik

Hi lss5fc0d , I have never used that library, but IMO such an approach would be appropriate, but it depends what the reference coordinate system you are wanting to use (eg where is 0/0 point?).

Aingu7ae

Hello erik & lss5fc0d, thanks for your feedback! I gotta work through your suggestions and the examples first and I have a hunch that I'll come back with more specific questions. Kind regrads