Identifying Disparity vs Confidence Maps

AndrewHatstat

Say I set up a pipeline on OAK-D that uses stereo depth node. I want to stream both the disparity and confidence images. On the host side I can create two queues linked to two XLinkOut nodes and retrieve each stream of images separately. This works, but having two asynchronous queues to read images from is not as convenient for collecting synchronized frames as having one queue. If I combine the disparity and confidence streams into one XLinkOut channel with one host queue I get both images from the queue, but I cannot tell which image is from which source. Both the disparity and confidence image has the same instance number (2), size, type (18), and "category" (0) whatever that is. Is there any way to identify which image is which?

erik

Hi AndrewHatstat ,
Even if you only have 1 stream/queue, this won't eliminate the problem, you would still need to sync those up. So I would suggest having 2 queues and do software msg syncing based on sequence number..

But since these 2 should be produced at the same time intervals, you could do something like this:

# while loop
disparityImgFrame = dispQ.tryGet()
if disparityImgFrame is not None:
    confImgFrame  = confQ.get() # Wait a few miliseconds

Thoughts?
Thanks, Erik

AndrewHatstat

Thanks @erik. I thought that might be the case. I created a pretty elaborate host side engine that assembles frames from multiple queues (e.g. rgb, mono left, mono right, disparity, confidence) based on the sequence ID. The goal is to create a collection of frames with same ID to pass onto the next image processing stage. This frame assembly step is complicated because it has to deal with both missing frames and time skew between queues. I found that under heavy load, the sequence numbers between queues can be off by as many as ten frames, even when the queue size is set to 1/non-blocking. I believe for the solution you propose you will get two frames, but it is not guaranteed that the sequence IDs will always match. I thought that using a single queue might reduce the skew between channels, but it seems that there is no way to determine which frame is which when using a single queue. It would be nice if I could ask for a collection of synchronized frames, and get them grouped together in one queue. Do you have a suggestion for grouping frames on the host side by sequence id that guarantees the sequence IDs match?

erik

Hi AndrewHatstat

off by as many as ten frames, even when the queue size is set to 1/non-blocking

Non-blocking means there might be frame drops. You should likely also evaluate blocking queue modes, so you won't have frame drops.

Do you have a suggestion for grouping frames on the host side by sequence id that guarantees the sequence IDs match?

depthai api doesn't provide such functionality, but in depthai-sdk there's an oak.sync() option, example here. Thoughts?
Thanks, Erik

AndrewHatstat

That is interesting. Unfortunately I'm using c++, but I can inspect the implementation. Regarding my original question, can you confirm there is no way to identify a disparity vs confidence frame that came from a single queue?