Getting PCD format point cloud data

hexbabe

Hey all,

Following the point cloud demo tutorial, I initialized the OakCamera and invoked the point cloud method. However, my goal is to get the raw point cloud data in a format such as PCD— not directly visualize it.

Is this possible with the SDK right now?

jakaskerl

Hi hexbabe
The pointcould is not in PCD format, but you can access the points by using a custom callback function:

def cb(pcl):
    print(pcl.points)

with OakCamera() as oak:
    color = oak.camera('color')
    stereo = oak.create_stereo()
    stereo.config_stereo(align=color)
    pcl = oak.create_pointcloud(stereo=stereo, colorize=color)
    oak.callback(pcl, callback=cb)
    oak.start(blocking=True)

Thanks,
Jaka

hexbabe

Ahh retrieving the packet works now. Thank you!

Unfortunately, the points array is too massive for our internal gRPC message to send to another service. Is there a way to lower the number of points generated by the DepthAI SDK?

jakaskerl

Hi hexbabe
You can edit the resolutions of the input cameras. This way stereo will run at lower resolution, resulting in less overall points.

Thanks,
Jaka

hexbabe

jakaskerl

Thanks so much for the help so far!

It seems like when I set the resolution for my color and stereo component cameras e.g. oak.camera('color', fps=self.frame_rate, resolution=self.color_resolution), I get the following error[ColorCamera(1)] [warning] Unsupported resolution set for detected camera IMX378/214, needs 1080_P / 4_K / 12_MP. Defaulting to 1080_P

What is the proper way of setting the cameras' resolution?

FYI, self.color_resolution = dai.ColorCameraProperties.SensorResolution.THE_720_P

hexbabe

I'm initializing the stereo component like this in a similar way:
oak.stereo(fps=self.frame_rate, left=mono_left, right=mono_right, resolution=self.depth_resolution)

where self.depth_resolution = dai.MonoCameraProperties.SensorResolution.THE_720_P

Edit: in fact, after some investigating, I think color actually does set resolution correctly— stereo is the one throwing the above error.

Could something like:stereo.node.setOutputSize(*color.node.getPreviewSize()) work? I'm worried it might not keep alignment between the two images.

jakaskerl

Hi hexbabe
Lowest an IMX378 can go is 1080p; since the point cloud is colorized by default, it is set to the resolution of the color camera.
To lower that, you need to apply isp scaling. Eg: color.config_color_camera(isp_scale=(2, 3)) with a 1080p sensor resolution will give you 720p --> 720x1280x3 point cloud.

Thanks,
Jaka

hexbabe

jakaskerl

Gotcha, is there no other way to change the output size of depth maps? When I do

stereo.config_stereo(align=color) # ensures alignment and output resolution are same

stereo.node.setOutputSize(*color.node.getPreviewSize())

The depth and color outputs are the same size. However, when I don't initialize my color node, and I don't set the alignment camera, I can't seem to resize the stereo depth output the same way (.setOutputSize doesn't actually change the depth output size for some reason). Is there an alternative way to set output sizes for depth out?

This is getting slightly off topic— let me know if I should make a separate thread in the future.

jakaskerl

Hi hexbabe
When you align to color, the stereo will always get upscaled to that resolution, that's why the stereo.node.setOutputSize(*color.node.getPreviewSize()) doesn't actually do anything in that case.

If you wish to resize the depth when not aligning to color, you do that by setting the mono camera resolution or by specifying the resolution when creating the stereo stereo = oak.create_stereo(resolution="720p") - which essentially does the same thing.
This will in turn change the depth resolution as well as the end point cloud resolution (since you are not aligning it to color).
The moment you set the alignment to color, the stereo will be upscaled to color resolution.

Thanks,
Jaka

Autovetus

hexbabe Hey mate !

Did you manage to get the point cloud properly ? how did yours look like ?

There is something wrong with mine

jakaskerl

Autovetus
What script are you using? Looks like either wrong intrinsics are used or the pointcloud generation is not correct.

Thanks,
Jaka

Autovetus

Hi jakaskerl !

We used the instructions from point cloud demo.

now the code looks like this :

with OakCamera() as oak:

color = oak.camera('color', resolution=OakCamera.Resolution.UHD)

stereo = oak.create_stereo()

stereo.config_stereo(align=color, 

                     confidence_threshold=240, 

                     median_filter=OakCamera.MedianFilter.KERNEL_7x7, 

                     lr_check=True, 

                     resolution=OakCamera.Resolution.HIGH,

                     sync=True)

pcl = oak.create_pointcloud(depth_input=stereo, colorize=color)

oak.visualize(pcl, visualizer='depthai-viewer')

oak.start(blocking=True)

jakaskerl

Autovetus
Code shouldn't work. Which SDK version are you using? Anyway, I changed the code a bit, and it seems to work, but the visualization is bad because the viewer puts origin too far away from the view.

with OakCamera() as oak:

    color = oak.camera('color', resolution="1080p")

    stereo = oak.create_stereo()

    stereo.config_stereo(align=color, confidence=200, median=7)
    pcl = oak.create_pointcloud(depth_input=stereo, colorize=color)

    oak.visualize(pcl, visualizer="depthai-viewer")

    oak.start(blocking=True)

The pointcloud looks ok though.

Thanks,
Jaka

PawelDuda

Thank you for improving the code.
We use SDK 1.15.0 and Python 3.11 on Ubuntu.
What is the best version combination for professional purposes?

Proposed code works better however the next visible issue is disappearing video content in the depthai-viewer after around 10 seconds. No visible logs in the console. Where I can find more logs from SDK?

jakaskerl

PawelDuda
Use latest depthai == 2.28
1.15 SDK is ok
Use latest viewer == 0.2.4

Thanks,
Jaka