Multi Camera Pointcloud Fusion Code

drakeramses

Hello,

I hope everyone is doing well. I am not very good at coding and I needed help with this project. I have 2 Oak D Pro W cameras and I have installed DepthAI v3 in Python. I have a structure where two cameras are fixed onto. I want to generate a pointcloud that combines the data from both cameras but I am struggling to do so. Is there any example code for this already made?

I understand the process of time stamp matching and the process of actually getting the extrinsics matrix but I do not know how to make a viewer that combines both cameras' outputs. I really need help on this and would appreciate any help.

Thank you!

jakaskerl

Hi drakeramses
You get points for both devices on your host machine. You can just display them with Open3D - considering the calibration is good enough, you will have combined pointclouds representing the 3D object.
If you want to merge the pointclouds then I suggest ICP (voxel downsampling, outlier removal, normals, and point-to-plane)

Thanks
Jaka

drakeramses

I appreciate the response. So what I am understanding is this:

1) Do calibration for both. Maybe set camera A as the world view and then calculate extrinsics for Camera A relative to Camera A's world view and Camera B's relative to Camera A's world view

2) Instantiate two different pointcloud pipeline streams. Camera A and Camera B

3) Next with our calibration matrices, transform the pointcloud into x, y, z per frame for both pointclouds.

4) Display on Open3D

The only question I have is how do I time sync both frames?

jakaskerl

drakeramses
Do calibration (you only need the extrinsic matrix between the cameras really - or cam_to_world for both). Then apply the transformation for points on the local pointclouds. Correct.

The pcl messages have timestamps which are based on the host timestamp (monotonic time since host boot). https://docs.luxonis.com/hardware/platform/deploy/frame-sync/. Sync those together (find best match).

Thanks,
Jaka