OAK-D Pro PoE Depth + RGB latency

random849

Hi,

I'm running an oak-d pro over poe, RGB output on it's own is fine (~100ms) latency and depth alone is ok, but when i enable both depth and rgb output the latency becomes very high (675ms).

Is this expected? I've run the experimental test scripts:

poe_test.py:

mxid: 18443010618AE9F400 (OK)

speed: 1000 (OK)

full duplex: 1 (OK)

boot mode: 3 (OK)

oak_bandwidth_test.py

Downlink 892.6 mbps

Uplink 221.3 mbps

latency_test.py

Average latency 1.77 ms, Std: 0.1

and they all look ok. It's connected to a desktop via a single switch.

Attached is an MRE, in the current state I get:

Mean depth latency in ms: 97.56883333333333

Mean frame rate: 19.432195182464387

with the rgb output uncommented I get:

Mean depth latency in ms: 670.4786401869159

Mean frame rate: 17.065788134779535

Thanks in advance!

random849

MRE: (won't let me upload as a file?)

random849

update: making the left and right camera objects monochrome and setting the resolutions down to 400p seems to solve it (latency of 150ms with both depth and rgb enabled) but as soon as I add an XLinkOut node for the rgb stream (even without subscribing to it) I go back to 560ms latency

jakaskerl

Hi @random849
I think the issue might be that by setting the depthAlign, you are essentially upscaling the depth to 1080p, which is not very efficient.

You can check the processing time of each node by using env variable DEPTHAI_LEVEL=trace. https://docs.luxonis.com/software/depthai/debugging/#Debugging%20with%20environmental%20variable%20%2C%5Bobject%20Object%5D

Thanks,
Jaka

random849

Hi Jaka,

Thanks for getting back to me, it does indeed appear to be that depth align setting. Turning that off brings the latency down to 210ms.

-> Does luxonis have a recommended configuration for using the Pro-PoE as an RGBD camera? I just want a >20fps depth image at the best possible resolution, with an >20fps aligned RGB feed at the best possible resolution. Within a latency of <400ms

-> Same thing for the short range camera, a depth map + aligned RGB at >20fps, some reasonable latency and the best resolution that the camera can output with those timing specs.

-> I've enabled the DEPTHAI_LEVEL=trace env var as well as self.device.setLogLevel(dai.LogLevel.TRACE) and self.device.setLogOutputLevel(dai.LogLevel.TRACE) but I'm still not seeing all the pipeline components in the terminal output? I.E. I can't spot that the depth alignment is the expensive operation from the trace data? is there a way of telling that, or increasing the profiling further? (outputs are attached)

-> If there isn't a recommended config, how do you recommend doing on camera alignment without killing the latency? Should I downsample the rgb, do an alignment to that and then upsample the depth on the host to match the native RGB output? Are there any alignment functions I can call on the host side?

Thanks,

Jasper

jakaskerl

random849 Does luxonis have a recommended configuration for using the Pro-PoE as an RGBD camera? I just want a >20fps depth image at the best possible resolution, with an >20fps aligned RGB feed at the best possible resolution. Within a latency of <400ms

You have 1000Mbps bandwidth (divided by 8 and fps > 20, that is 6.25MBps). Size for both RGB and Depth should be the same (W, H).
In case of video output and subpixel depth that is (1.5bytes/pixel NV12 + 2byte/pixel UINT16) 3.5B/pix

This leaves you with
approximately 1,780 pixels in width and 1,002 pixels in height

In practice, the bandwidth is lower and there is some overhead.

random849
You are reaching POE bandwidth limits.
Convert the stream to preview or video and lower the size.

Also, it's bad practice to use mono cameras as ColorCamera node. Use MonoCamera instead.

Thanks
Jaka

random849

Update:

Did some more testing and depthAlign is not the only culprit, demo attached below.

This demo is ~600ms latency because I added an rgb output (the rgb_queue) object. I'm not reading from it, just creating it as a non blocking queue. DepthAlign is turned off

At this point I can't even seem to get a non aligned RGB + D feed off the camera at a reasonable rate? Is this something luxonis supports or is this just outside the hardware processing specs?