Set up streaming pipeline for real-time image processing

BenjaminKiefer

Hi!

I was curious to learn some best practise on how to set up a streaming pipeline with 30FPS Fullhd image processing. In particular, I want to run a neural network on a powerful hardware (4090GPU, i9 processor,…).

I need the video stream and corresponding IMU data. For that, I started working with that example:

https://github.com/luxonis/depthai-experiments/blob/master/gen2-syncing/imu-frames-timestamp-sync.py

However, when introducing several image manipulation operations on the host side, the FPS goes down dramatically (like 5fps only). When doing the exact same logic on a ZED2 camera, I don't have any problems maintaining 30fps.

I wonder how I'd need to set up the oak to also provide that. Do I need to include threading or such?

I'd be thankful for any advice!

Thanks!

jakaskerl

Hi BenjaminKiefer
Are you sure it's the image manipulation operations that are causing the FPS drop?
Host side performance should be the same regardless of which device you are using. Also what kind of image manipulations are you performing and what libraries are you using to achieve them?

Thanks,
Jaka

BenjaminKiefer

jakaskerl

Essentially, I compare the raw script from above with one, where I included a neural network and some resizing and stabilization (feature-based) in the same main loop, so no threading or such. Obviously, that slows down the the loop, but it seems like it does slow it down more than just the time that is introduced by these operations.

I feel like the operations "compete" with the loading of the Oak, causing a significant performance drop. Again, with the ZED2, these included operations did only slow down the loop by the amount of time introduced by the operations.

In more detail:

I'm using YOLOv7, several cv2-operations on the rgbframe that I get from the script as given above.

May it be the syncing or should I use a more updated script?
In theory, I'd not need to sync if the frames come in fast enough for my application

Thanks!

jakaskerl

Hi BenjaminKiefer
I'm not really sure what might be causing the problem here. From what you have described, it seems as though the ZED2 operations use a different process which doesn't affect the main loop. When using the OAK, everything (apart from the operations run on the device) is executed in the same thread which might eventually be causing the drop you are experiencing.

2 tests that come to mind:

try removing the msgSync feature to see if the syncing is causing the low FPS; many times that is the case
if you are able to do so, spread the cv2 manipulations across different threads to see if it improves performance.

Alternatively you can use the DEPTHAI_LEVEL=TRACE env variable to check how long an operation takes on the OAK; or use the python to time the host operations to pinpoint the source of the delays.

Thoughts?
Jaka

BenjaminKiefer

Hi Jaka,

thanks for your response!

I put the Oak camera loop into a separate thread, which improved things massively (I think you have some kind of buffer in place which causes the high latency). It's still not optimal, and I'm even thinking of running in it in a separate process and stream to Python over socket (just to continue to stream via socket to Unity), but this seems very complicated. If the UVC feature would work together with the IMU, that would simplify things by a bit, but probably I can't load just the IMU via your SDK and the video feed via UVC, could I?

Removing the syncing didn't improve things by a lot.

On another note: At some point, I experienced the Xlink error (I think this one: [BUG] error message: X_LINK_DEVICE_NOT_FOUND · Issue #518 · luxonis/depthai (github.com)). This only appeared after using the camera for a couple of hours (on/off). I think it's critical from a reliability aspect since I'm looking to deploy a dozen of these cameras in a remote location without having access to it (unplugging and plugging in again is not an option).

The cable seems to be fine (USB-4). Are there any other best practises to avoid that?

In any way, thanks again!

-Benjamin

jakaskerl

Hi BenjaminKiefer
Sorry for the late reply,
I don't think streaming IMU and UVC together can be done (not easily anyway) as of now. Are you running into any latency when streaming to unity? Usually if you intend to have multiple streams, this would quickly saturate your network bandwidth which would introduce delays and possibly crashes as well.

BenjaminKiefer At some point, I experienced the Xlink error

Does this error pop up when the device is booted; or does it occur when the device is booting up (when you start a script)? In any case, for more devices I would always recommend setting up a watchdog service of some kind which will check whether the camera is correctly being used and be able to restart it if need be.

Thoughts?
Jaka

BenjaminKiefer

Yes, latency issues.

I'm running it in a separate thread now, but won't get more than 8FPS, probably because my NN&CV operations cost a lot.

It happened randomly in the middle of running a script. I'm afraid that this will happen again in a production setting. What workaround would there be to restart it?

Thanks!

jakaskerl

BenjaminKiefer
You could try timing the script to see which operations take the most time. That way you will make sure whether it's the host that bottlenecks.

I'll ask at the office tomorrow since it seems to be a reocurring problem and I don't know the cause behind it.

Thanks,
Jaka