Saving Point Clouds for Future analysis

michaelsalerno

Dear Luxonis Team,

Thank you for all the support you have provided me and I appreciate your continued consistent assistance with the issues and questions I have.

Currently, I am trying to record a 3D point cloud so in the future I may do further analysis. How can I save a point cloud streamed from my Oak-D camera, then replay the recording, than allowing me to export the final point cloud generated. I have been trying to work with rerun and depthai, but I am unsure of the next steps I should take. Is there a code that I can run from the luxonis repository that will allow me to do this? I want the point cloud to also have depth estimates associated with it, giving each point an xyz coordinate. Please let me know if I can do this, or if using applications such as COLMap and openMVG are more streamlined alternatives.

Do you know how I will be able to accomplish this?

jakaskerl

Hi @michaelsalerno
No examples yet, but how about saving the device generated pointclouds with o3d. I tested it and it gives you the ability to replay the pointclouds which have the RGBD data.

# Define the number of frames and delay between captures
num_frames = 100
delay_between_frames = 0.1  # in seconds

# Start capturing frames
for i in range(num_frames):
    print(f"Capturing frame {i}")
    frame = sensor.capture_frame(True)

    # Assuming the frame has a valid depth image and camera intrinsic parameters
    pcd = frame.create_point_cloud()

    # Save to a PLY file
    filename = f"captured_frame_{i:04d}.ply"
    o3d.io.write_point_cloud(filename, pcd)

    # Wait before capturing the next frame
    time.sleep(delay_between_frames)

print(f"Finished capturing {num_frames} frames")

Replay:

import open3d as o3d
import glob
import time

# Define the delay between frames for the replay
replay_delay = 0.1  # in seconds

# Load the sequence of point cloud filenames
pcl_files = sorted(glob.glob('captured_frame_*.ply'))

# Create a visualizer
vis = o3d.visualization.Visualizer()
vis.create_window()

for i, pcl_file in enumerate(pcl_files):
    print(f"Replaying frame {i}")
    pcd = o3d.io.read_point_cloud(pcl_file)

    # Update the point cloud for the visualizer
    if i == 0:
        vis.add_geometry(pcd)
    else:
        vis.update_geometry(pcd)

    vis.poll_events()
    vis.update_renderer()
    time.sleep(replay_delay)

vis.destroy_window()
print("Finished replaying the point cloud video")

Thoughts?

michaelsalerno

@jakaskerl

I tried running the code in both an active and deactivated virtual environment and both returned the same error code. I placed the code in a file that is just named rerun code, but that is just one location I have a virtual environment. How would I resolve this issue? Is there a requirements.txt I need to run prior? Also, when activated correctly does it display in a viewing window?

(venv) C:\Users\sale4088\AppData\Local\Programs\rerun code>python recordT.py

Capturing frame 0

Traceback (most recent call last):

File "C:\Users\sale4088\AppData\Local\Programs\rerun code\recordT.py", line 8, in <module>

frame = sensor.capture_frame(True)

NameError: name 'sensor' is not defined

jakaskerl

Hi @michaelsalerno
The example was not plug n play.

    inPointCloud = inMessage["pcl"]
    while True:
            # if we have message 
            points = inPointCloud.getPoints().astype(np.float64)
            pcd.points = o3d.utility.Vector3dVector(points)
            colors = (cvRGBFrame.reshape(-1, 3) / 255.0).astype(np.float64)
            pcd.colors = o3d.utility.Vector3dVector(colors)
            # save the point cloud to a file
            o3d.io.write_point_cloud("pointcloud.pcd", pcd)

jakaskerl

jakaskerl
This is for saving a single frame. Hope it makes sense:

with dai.Device(pipeline) as device:
    isRunning = True
    def key_callback(vis, action, mods):
        global isRunning
        if action == 0:
            isRunning = False

    q = device.getOutputQueue(name="out", maxSize=4, blocking=False)
    pcd = o3d.geometry.PointCloud()
    coordinateFrame = o3d.geometry.TriangleMesh.create_coordinate_frame(size=1000, origin=[0,0,0])

    first = True
    fpsCounter = FPSCounter()
    while isRunning:
        inMessage = q.get()
        inColor = inMessage["rgb"]
        inPointCloud = inMessage["pcl"]
        cvColorFrame = inColor.getCvFrame()
        # Convert the frame to RGB
        cvRGBFrame = cv2.cvtColor(cvColorFrame, cv2.COLOR_BGR2RGB)
        fps = fpsCounter.tick()
        # Display the FPS on the frame
        cv2.putText(cvColorFrame, f"FPS: {fps:.2f}", (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 255), 2)
        cv2.imshow("color", cvColorFrame)
        key = cv2.waitKey(1)
        if key == ord('q'):
            break
        if inPointCloud:
            t_before = time.time()
            points = inPointCloud.getPoints().astype(np.float64)
            pcd.points = o3d.utility.Vector3dVector(points)
            colors = (cvRGBFrame.reshape(-1, 3) / 255.0).astype(np.float64)
            pcd.colors = o3d.utility.Vector3dVector(colors)
            # save the point cloud to a file
            o3d.io.write_point_cloud("pointcloud.pcd", pcd)

Thanks,
Jaka

michaelsalerno

@jakaskerl

Thank you for the support, if the codes are not plug n play than where should I be placing the codes that you provided earlier? I have open3d downloaded, as well as the Main and ML libraries. How should they be executed?