How can I efficiently align depth to RGB using OAK4?

rohan

Hello,

I'm using an OAK4-D PRO W device to build a pipeline that aligns stereo depth map to the color frame. I was following this example for ImageAlign node and I noticed a significant lag in the color display even when I'm using 1200p resolution for color camera. The display is normal when I lower the resolution, but I want to use 1200p for color camera and 800p for mono cameras. How can I improve the performance of alignment? Could you please share any guide or documentation if available? The StereoDepth node has a method- setPostProcessingHardwareResources(). This would not work for RVC4 devices, would it?

P.S. I've also noticed a lag in the display when I add an ImageManip node to the pipeline. This lag is only seen on the RVC4 device. It works fine on RVC2 devices.

Any insights are appreciated. Thank you.

jakaskerl

rohan

rohan The StereoDepth node has a method. This would not work for RVC4 devices, would it?

This should still work afaik. And it is more performant generally, unless the alignment has to upscale the resolution. Likely not the case anymore.

Thanks,
Jaka

rohan

jakaskerl

Hey Jaka, I edited my question. I was referring to the setPostProcessingHardwareResources method. Sorry for the confusion. This would not work on RVC4 devices, right?

Moreover, I observed the following-

The StereoDepth class has setDepthAlign method, but it doesn't seem to set the camera for alignment. The depthAlign attribute from StereoDepthConfig does not change when this method is used.
When I directly set the alignment using algorithmControl from the StereoDepthConfig, it is not producing a depth map that is the same size of the color image. I'm using a color size of 1920x1200, and mono size of 1280x800. The depth map I'm getting is of 320x200 size. There is no lag, but the sizes do not match. Is this the default output size of the StereoDepth node?
The setOutputSize method on StereoDepth class is not available for RVC4 devices. So I must use an ImageManip node is resize the depth map. But this node is also causing a lag.