Configuration message latency for real-time apps

nmvrs

This has been previously discussed for CameraControl and the ImageManip node (typically for cropping):

There seems to be a time delay before the application of config messages from the host to the device(s). Minimizing this latency is critical for high rate real-time applications. Or applications that need to distribute processing between the devices and host, and/or fuse multiple devices' outputs.

The latest on this based on the prior discussions above is that it is not possible to overcome this delay, and this is not a high priority issue.

With the higher computational capacity OAK4 now available, it makes sense to aggregate all these issues and revisit this:

Any plans to work on minimizing the delay ?
Would the same delay apply if the host sends another type of message? NNData messages can be send from the host (or created via scripts), but would they be matched with the latest image on the queue? Or will this suffer from the same delay?
It was mentioned that ImageManip suffers from a "recompile" delay. Assuming this is on top of the config message application, would a specialized crop node with minimal latency (similar to the audio implementation) make sense?

jakaskerl

Hi nmvrs
My prediction would be that it is much faster on OAK4 since the whole system is on a single chip - to make sure, we'd have to test this. Can you give me specific use-cases/examples for which you want the latency to be as minimal as possible? Eg. CameraControl (host) -> Camera node (effect visible on device), ...

Thanks
Jaka

nmvrs

It is very similar to the use cases described in the aforementioned discussions/issues.

The use case is about tracking an object based on inference done on the edge, as well as extra processing on the host. The host owns an improved estimate that needs to be communicated to the edge. But since the object can move fast, the rate needs to be high (60+Hz) and the overall system's accuracy and robustness improves when the cropped image precisely contains the object.

I will test using the scripts provided by the other discussions/issues with an OAK4 and re-visit this once I have something more concrete to share.