• DepthAI

  • Low FPS When Aligning Point Cloud to RGB

Hi,

I’m sending RGB images at THE_1080_P and point clouds at THE_400_P from an OAK-D Pro to a Raspberry 5 over a USB 3 connection. If I’m not aligning the point cloud to the image, I’m achieving 26 FPS. But when I do align them, I only get 6 FPS. Can someone please explain why aligning the point cloud to the image slows things down so much and/or how to achieve higher frame rates?

Here is the code I’m using:

import depthai as dai
import time

pipeline = dai.Pipeline()

camRgb = pipeline.create(dai.node.ColorCamera)
monoLeft = pipeline.create(dai.node.MonoCamera)
monoRight = pipeline.create(dai.node.MonoCamera)
depth = pipeline.create(dai.node.StereoDepth)
pointcloud = pipeline.create(dai.node.PointCloud)
sync = pipeline.create(dai.node.Sync)
xOut = pipeline.create(dai.node.XLinkOut)
xOut.input.setBlocking(False)

camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
camRgb.setFps(30)

monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoLeft.setCamera("left")
monoLeft.setFps(30)

monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoRight.setCamera("right")
monoRight.setFps(30)

depth.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)
depth.initialConfig.setMedianFilter(dai.MedianFilter.KERNEL_7x7)
depth.setLeftRightCheck(True)
depth.setExtendedDisparity(False)
depth.setSubpixel(False)
depth.setDepthAlign(dai.CameraBoardSocket.CAM_A)

monoLeft.out.link(depth.left)
monoRight.out.link(depth.right)
depth.depth.link(pointcloud.inputDepth)
camRgb.isp.link(sync.inputs["rgb"])
pointcloud.outputPointCloud.link(sync.inputs["pcl"])
pointcloud.initialConfig.setSparse(False)
sync.out.link(xOut.input)
xOut.setStreamName("out")

with dai.Device(pipeline) as device:
    q = device.getOutputQueue(name="out", maxSize=4, blocking=False)
    startTime = time.time()
    for i in range(1, 100):
        inMessage = q.get()
    print(1/((time.time() - startTime)/100))

Thank you in advance for any help.

Cheers,
Tim

    Timm
    AFAIK, the depth map gets upscaled to RGB resolution when you align to the color camera. This is very likely the cause of the low FPS.
    You could try adding a decimation filter to lower the depth resolution.

    Thanks,
    Jaka

    • Timm replied to this.

        jakaskerl

        I'd like to give this a try, but adding a decimation filter like so

        configuration = depth.initialConfig.get()
configuration.postProcessing.decimationFilter.decimationFactor = 2 
depth.initialConfig.set(configuration)

        doesn't work in DepthAI 2.29 and 2.30 if I'm also aligning to the color camera. Are there alternative ways to add this filter?

        Thanks,
        Tim

            Timm
            It should work if you manually specify the filter order and set the median filter to be last. Also I think a fix was pushed to develop branch already.

            Thanks,
            Jaka

            • Timm replied to this.