Object tracking for slow neural networks

ThomasBr76

Hi,

I'm trying to use a yolov8 medium neural network for object detection and a tracker to evaluate the object's movements. The neural network is really slow (we have 2fps on XLinkOut3 referencing the graph below) so I want to use zero term tracking to track the objects between 2 inferences and not run the neural network detection on every frame.

Is there an example somewhere to achieve that ?

How is XLinkOut3 influenced by the neural network ?

This is how the pipeline is currently looking, is there something wrong with it ?

jakaskerl

ThomasBr76 so I want to use zero term tracking to track the objects between 2 inferences and not run the neural network detection on every frame.

Then perhaps you are looking for short term tracking: https://docs.luxonis.com/projects/api/en/latest/components/nodes/object_tracker/#short-term-tracking. The examples we have shouldn't be any different when using a different tracker type.

ThomasBr76 How is XLinkOut3 influenced by the neural network ?
The NN will block the ColorCamera from producing frames faster than NN can process them, which means the XLINKOUT(3) will have the same FPS as the NN.

ThomasBr76 This is how the pipeline is currently looking, is there something wrong with it ?

Looks fine. Consider using a smaller model though.

Thanks,
Jaka

ThomasBr76

jakaskerl

Hi Jaka,

Thanks for your answer !

We tried using short term tracking but we didn't manage to have more fps. How can the tracker be beneficial, if the NN blocks the rgb camera from producing new frames, which could potentially be used by the tracker ?

Using a smaller network increases the fps but the detection is less accurate.

jakaskerl

Hi @ThomasBr76
I think you can use detectionNetwork.input.setBlocking(False) then link the color camera frames directly to tracker's tracking input. Afaik this shouldn't block the color camera node.

Thanks,
Jaka

ThomasBr76

Yes I already tried that but it didn't help unfortunately

jakaskerl

Hi @ThomasBr76
Looks like there is a FW issue locking the object tracker to NN node FPS :/
I've forwarded this issue to the team.

Thanks,
Jaka

ThomasBr76

jakaskerl

Hi Jaka,

Thanks for reporting the issue 🙂 Could you let me know if there is a fix or an update please ?