Multi-device point-cloud alignment

saiborg

I was demoing rgbd-pointcloud-fusion and came across this misalignment issue. Even after running multi-cam calibration and reseting the alignment, the capture from all 4 cameras are not aligned on the center object. How do I go about fixing this issue?

erik

saiborg To me, it seems like a problem with the multi-camera calibration... Pasting from gpt4 below, I would especially look into the last option.
-------
Misalignment in stereo camera setups can be caused by various factors such as errors in calibration, synchronization issues, or mechanical inaccuracies. To address this issue, you can take the following steps:

Double-check calibration: Ensure that you are using a high-quality calibration pattern (such as a chessboard or circle grid) with high contrast and sharp edges. Make sure to take multiple images from different angles and distances to cover the entire field of view of both cameras. Use a reliable calibration algorithm (like OpenCV's stereo calibration or MATLAB's camera calibration toolbox) to obtain accurate intrinsic and extrinsic parameters.
Improve the calibration setup: If possible, use a larger calibration pattern to increase the accuracy of the calibration. Additionally, ensure the calibration environment has stable lighting conditions and no reflections on the pattern.
Verify camera synchronization: Ensure that the cameras are synchronized, meaning they are capturing images at the same time. Mismatched timestamps can cause misalignment in the point clouds. If your cameras support hardware synchronization, use it for better performance.
Check for mechanical inaccuracies: Inspect the camera rig for any mechanical issues or inaccuracies in the mounting. Make sure the cameras are rigidly mounted, and there is no relative movement between them during the operation.
Refine point cloud registration: After obtaining the initial extrinsic parameters, you can further refine the alignment by using a point cloud registration algorithm such as Iterative Closest Point (ICP) or Generalized-ICP. These algorithms will minimize the distance between corresponding points in the two point clouds, improving the alignment

wamj45

@erik
I am also facing the same issue where my two Oak-D Pros' point clouds are not aligning correctly.
One question I have is one image sufficient to calibrate the camera? In my experience usually 15-30 images are bare minimum. Or am I using the multi-cam-calibration incorrectly?

erik

Hi @wamj45 ,
15-30 images is usually for intrinsic/image distortion calibration. For extrinsics between 2 cameras you don't need that many images, and if you don't need such high level of accuracy (which you typically don't for pointcloud fusion), 1 img should be enough. And I believe the multi cam calibration script will estimate extrinsics only from 1 pair of images.

wamj45

hmm… my cameras were not calibrated with calibration marker being visible in both views. Could that be it?
I moved the calibration target between image captures

jakaskerl

wamj45
Both extrinsic and intrinsic calibrations need markers in all camera frames, otherwise it has no clue where the rest of the cameras are / how each camera behaves distortion-wise.

And you need a calibrated intrinsics before you can perform the extrinsic calibration, otherwise the distortions will shift the real world positions resulting in bad matching.

Thanks,
Jaka