How can I efficiently align depth to RGB using OAK4?

rohan

Hello,

I'm using an OAK4-D PRO W device to build a pipeline that aligns stereo depth map to the color frame. I was following this example for ImageAlign node and I noticed a significant lag in the color display even when I'm using 1200p resolution for color camera. The display is normal when I lower the resolution, but I want to use 1200p for color camera and 800p for mono cameras. How can I improve the performance of alignment? Could you please share any guide or documentation if available? The StereoDepth node has a method- setPostProcessingHardwareResources(). This would not work for RVC4 devices, would it?

P.S. I've also noticed a lag in the display when I add an ImageManip node to the pipeline. This lag is only seen on the RVC4 device. It works fine on RVC2 devices.

Any insights are appreciated. Thank you.

jakaskerl

rohan

rohan The StereoDepth node has a method. This would not work for RVC4 devices, would it?

This should still work afaik. And it is more performant generally, unless the alignment has to upscale the resolution. Likely not the case anymore.

Thanks,
Jaka

rohan

jakaskerl

Hey Jaka, I edited my question. I was referring to the setPostProcessingHardwareResources method. Sorry for the confusion. This would not work on RVC4 devices, right?

Moreover, I observed the following-

The StereoDepth class has setDepthAlign method, but it doesn't seem to set the camera for alignment. The depthAlign attribute from StereoDepthConfig does not change when this method is used.
When I directly set the alignment using algorithmControl from the StereoDepthConfig, it is not producing a depth map that is the same size of the color image. I'm using a color size of 1920x1200, and mono size of 1280x800. The depth map I'm getting is of 320x200 size. There is no lag, but the sizes do not match. Is this the default output size of the StereoDepth node?
The setOutputSize method on StereoDepth class is not available for RVC4 devices. So I must use an ImageManip node is resize the depth map. But this node is also causing a lag.

rohan

Hi @jakaskerl,

Did you get a chance to look into this? I appreciate any insights you can share.

Thank you.

jakaskerl

rohan
The most efficient way currently is to use the imageAlign - https://docs.luxonis.com/software-v3/depthai/examples/image_align/depth_align/. And run it on device (setRunOnHost = False). If you are experiencing lag, make sure all nodes are running on the device.

You can also post some minimal repro code and we can take a quick look.

Thanks,
Jaka

rohan

import depthai as dai
import numpy as np
import cv2
from datetime import timedelta

ALIGN_NODE = True

DEFAULT_FPS = 15
COLOR_WIDTH = 1280
COLOR_HEIGHT = 800
MONO_WIDTH = 1280
MONO_HEIGHT = 800

pipeline = dai.Pipeline()

colorNode = pipeline.create(dai.node.Camera).build(boardSocket=dai.CameraBoardSocket.CAM_A)
colorOut = colorNode.requestOutput((COLOR_WIDTH, COLOR_HEIGHT), fps=DEFAULT_FPS)

monoLeft = pipeline.create(dai.node.Camera).build(boardSocket=dai.CameraBoardSocket.CAM_B)
monoLeftOut = monoLeft.requestOutput((MONO_WIDTH, MONO_HEIGHT), fps=DEFAULT_FPS)

monoRight = pipeline.create(dai.node.Camera).build(boardSocket=dai.CameraBoardSocket.CAM_C)
monoRightOut = monoRight.requestOutput((MONO_WIDTH, MONO_HEIGHT), fps=DEFAULT_FPS)

stereoNode = pipeline.create(dai.node.StereoDepth)
monoLeftOut.link(stereoNode.left)
monoRightOut.link(stereoNode.right)

sync = pipeline.create(dai.node.Sync)
sync.setSyncThreshold(timedelta(milliseconds=80))
colorOut.link(sync.inputs["color"])
if ALIGN_NODE is True:
    alignNode = pipeline.create(dai.node.ImageAlign)
    alignNode.setRunOnHost(False)
    stereoNode.depth.link(alignNode.input)
    colorOut.link(alignNode.inputAlignTo)
    alignNode.outputAligned.link(sync.inputs["depth"])
else:
    stereoNode.depth.link(sync.inputs["depth"])
syncQueue = sync.out.createOutputQueue()


with pipeline:
    pipeline.start()

    cv2.namedWindow("color", cv2.WINDOW_NORMAL)
    cv2.resizeWindow("color", 640, 400)
    cv2.namedWindow("depth", cv2.WINDOW_NORMAL)
    cv2.resizeWindow("depth", 640, 400)

    while pipeline.isRunning():
        syncGroup = syncQueue.tryGet()
        if syncGroup is None:
            continue
        
        assert isinstance(syncGroup, dai.MessageGroup)
        if syncGroup.getNumMessages() == 0:
            continue

        print(f"MSG Group timestamp: {syncGroup.getTimestamp()}")
        for name, payload in syncGroup:
            assert isinstance(payload, dai.ImgFrame)
            print(f"\t[{name}] size: [{payload.getWidth()} x {payload.getHeight()}]")

            if name == "color":
                cv2.imshow(name, payload.getCvFrame())
            elif name == "depth":
                depthMap = payload.getFrame()
                normalizedMap = (depthMap * (255 / stereoNode.initialConfig.getMaxDisparity())).astype(np.uint8)
                colorizedDepth = cv2.applyColorMap(normalizedMap, cv2.COLORMAP_JET)
                cv2.imshow(name, colorizedDepth)

        key = cv2.waitKey(1)
        if key == ord('q'):
            pipeline.stop()
            break

Hey @jakaskerl,

Here's an MRE for the problem. You can toggle the ALIGN_NODE flag to compare the effect of ImageAlign node on the pipeline. You can lower the resolution of the color output and see that the lag is eliminated. Please let me know if you need anything else.

I tested this code on OAK-4-PRO-AF and OAK-4-PRO-W devices.

Thank you.

jakaskerl

Hey rohan
Just letting you know I've reproed the issue last week and passed it to the development team to take a look. Likely related to how long you are waiting on pools.. But it needs further inspection.

Thanks,
Jaka