Object detection from MobileNetSSD intermittently drop for some frames

msharad19

Hi,
I'm using MobileNetSSD for object detection on OAK D Lite. As objects move in the frames, they are detected well. But every couple of frames, there is usually one random frame where there are no detections at all. This keeps on happening every Nth frame (where N visually looks like it is between 10-20).
Can you please help? I am running OAK D Lite with USB3 (SUPER speed)

Thank you,
Sharad

erik

Hello msharad19 , could you share the code you are using? It might be that NN isn't fast enough, so it's dropping some frames. In such case, you could:

Look at host/device side syncing . This would discard other frames that don't have detections associated with them.
Limit camera FPS so this doesn't happen. From your testing it seems you would need to lower it just a bit.
Have blocking input queues (docs here) on NN node. This would block camera frames until NN can run inference on older ones. This is a bit more complex approach, but can be useful in some cases.

Thoughts?
Thanks, Erik

msharad19

I still need some time to investigate due to other urgent aspects to work on. I will circle back and confirm as soon as I can. Thanks, Erik!

msharad19

Hi Eric!
I tried the first option (host-nn-sync.py) because reducing fps did not work
This script works as expected in isolation. But when I tried using this logic in our system, we have a problem that detIn.getSequenceNum() is always 0 (detections sequence number does not increase). This was surprising because we are getting newer detections for new frames (lagging slightly behind the frames, unfortunately). Are there any pointers for me to consider here?

To give a little more clarity, my main problem is that detection boxes lag behind frames (fixed distance lag) for a moving object. Using passthrough frames did not change this and I suspected that periodic detection drops might somehow be the reason. I'm not sure about it anymore. Can you suggest some ideas here please?

I'm not running detections on the frames grabbed from RGB camera directly. In my pipeline, I grab the RGB frame, morph it based on some requirements and send it back to the VPU for inference

If nothing helps, I can share some parts of the code.

Thanks,
Sharad

erik

Hi msharad19 ,
Regarding " morph it based on some requirements" - after resending it back to the OAK device, do you also set the original timestamp? If you construct a new ImgFrame (and you don't set the seq num), it's expected that sequence number will be 0. I think it would be fasest if you could provide a full MRE so we can take a look.
Thanks, Erik

msharad19

Hi @erik
In the process of preparing a full MRE, I realised that I did not set the timestamp and sequence number for the morphed imageFrame. Setting it and then using host-nn-sync example fixed the issue. Thank you so much for pointing it out. The issue is resolved now 🙂

But I was wondering why is it that the detections are slightly lagging behind image frames. I tried it at 30fps, and the difference between them was 4-5 frames, at 15 fps, it was 3-4 frames. And at 2fps, it was still 1 frame. It's interesting to know that there's no accumulation of this lag with time. Even if I run the system for longer, the difference remains constant. Would that not mean that nn inference rate is NOT slower than the incoming frame rate? (because the lag doesn't increase with time). I don't understand what causes this, and was wondering if you could help 🙂

Thanks,
Sharad

erik

Hi msharad19 ,
So you are lowering the FPS, which means there's more delay between frame. Each inference will take fixed amount of time (from your results I would imagine about 150ms-200ms), so lowering FPS will result in less frequent frames, and more "in-sync". It's not that NN inference is slower, it's just that frames are slower. THoughts?
THanks, Erik

msharad19

Hi Erik,
That makes sense. Thank you so much for your help. This solved the only issue we faced with OAK-D 🙂

Thanks again,
Sharad