Problems with detections timestamp/age after a reconnect (depthai 3.1.0)

SharadMaheshwari

Hi,
We've been using rcv2 devices (oak 1) with depthai 3.1.0 and have been having problems with the age of detections/timestamps after a temporary disconnect.
With USB devices, we unfortunately have disconnects randomly, and depthai tries to reconnect, which is great.

But from what I see in the object detections pipeline, the detections are lagging (old) once the connection is established again.

My process of checking this:
1. We send a custom image sequence from another camera to oak devices (that's a requirement for us) at 35fps
2. We can get "self.detectionOutQueue.tryGet()" for each image that we send. I use "getSequenceNum()" method to get the sequence number for this detection
3. Internally, I also maintain a sequence number list for every image that we send
4. Before any disconnect, the difference between the detection's sequence number and my internal sequence number is 3. I already have code to account for that
5. After there is a temporary disconnect and a connection to the device is established again, this difference jumps to 14. Any subsequent reconnects don't change this difference (always 14)
6. The difference changes to 3 again if I have a fresh machine start.

Based on the detection's "age" we ask our robots to actuate. Unfortunately, all our actuation is delayed if we have a reconnect.
It looks like this is coming from the depthai library when it handles disconnects. Can someone help with this urgently please?

OskarSonc

Hey @SharadMaheshwari – this typically isn’t “extra latency” in the device, but a backlog of frames building up while USB is disconnected. During the drop, the host can keep queuing input frames, and when the device reconnects DepthAI replays those, so detections come out 14 frames “old” instead of 3. To avoid that, make your input and detection queues small and non-blocking so old frames get dropped, e.g. setQueueSize(1) + setBlocking(False) / maxSize=1, blocking=False, flush any pending detections right after reconnect, and if you use trySend, don’t advance your own sequence counter when the send fails. After reconnect you can also re-baseline your offset (take the first detection’s getSequenceNum() as the new reference) so your “expected gap” goes back to around 3.
Also worth trying a newer DepthAI build (3.2.x) as some auto-reconnect/queue behaviors were improved after 3.1.0.

Thanks,
Oskar

SharadMaheshwari

Hi Oskar,
Thanks for you response. Based on your answer, I found something interesting.
We have a pipeline with the following:
1. Frame input queue to the neural network: We feed the images with this. We already have blocking set to False and maxSize set to 1 for this.
2. Detections output queue from the neural network: We get detections with this. We did not set block or maxSize explicitly (because we thought it should effectively only depend on the input queue)

As you know, when the device is disconnected, and we connect again, the sequence difference jumps from 3 to 14. But logically, if the input queue has maxSize =1 and blocking = False, it should not happen because there should be no accumulation in detections output queue if the oak device is disconnected for a period

Surprisingly, when I set maxSize=1 and blocking = False for detections output queue as well, the behavior changed. After a disconnect and reconnect, the sequence number difference starts from 16, and in about 30-40 cycles, it goes down slowly to 3 again.

Question 1: Why would setting maxSize and blocking for the frame input queue not be enough?
Question 2: To us, it feels like there's an internal queue of around 16 that is used while the device is disconnected and reconnected again (even if we explicitly fix it with maxSize). Do you have some ideas?

Question 3: Additionally, I'm trying to understand why we have this sequence number difference of 3 in the first place.
Since we maintain a sequence number for each frame we feed, we should get the same sequence number with getSequenceNum() api. Do you have some thoughts on this as well?

Thanks,
Sharad

OskarSonc

Hey @SharadMaheshwari
For Q1: limiting only the frame input queue isn’t enough because there are several buffers after that point: NN input, the NN’s own frame pool, the device-side XLink ring, and finally the host OutputQueue (which by default is quite large and blocking). If the host isn’t reading while the device is happily producing detections again, those downstream queues can still accumulate old results even though the input side is “latest only”, which is why setting maxSize/blocking on the detection side immediately changed the behavior.

For Q2: the “starts at 16 and then slowly walks back to 3” strongly suggests the total internal buffering depth is around that size – in-flight frames plus those XLink/NN pools. After reconnect, those buffered frames are replayed, so you begin 16 behind and then drain down to the steady 3 as the pipeline stabilizes. You can reduce this effect by keeping all queues small + non-blocking (input, NN input, XLinkOut, host output) and by not sending new frames when trySend fails, or even fully rebuilding the pipeline on reconnect so it starts from an empty state.

For Q3: the “3 frame” gap itself I think is normal – it’s just your steady-state pipeline depth, how many frames are in flight between your send and the NN output at 35 FPS. As long as that offset is stable, it just reflects inference + transport latency.

Hope this helps,
Oskar