[Android] OAK-1 and OAK-D Low frame rate

Brac

Hi !
I'm actually working on a project where I need to plug an OAK-1 camera to an Android device.
I took inspiration from this projet : https://github.com/ibaiGorordo/depthai-android-jni-example

My goal is to get the YUV420 directly from the ISP in 4K at a frame rate of at least 20FPS (using the C++ API), I saw on the documentation that these numbers were normally largely attainable.

But my problem is that when I connect the OAK-1 to an Android device (Note 9 and others...) and try to get the image from the ISP the frame rate is only 4 FPS, even with a SUPER usb speed.
In fact the same code run around 28FPS on my Windows machine.

I wanted to know if this problem was known and if someone managed to overcome it ?
Or if someone has an idea of where to explore on the side of depthai-core or its dependencies to try to speed up this frame rate.

Any help would be appreciated 😀

erik

Hello Brac ,
I believe that is because two things:

Our Android implementation (depthai-android-jni-example) isn't the most performant implementation in regards of copying some messages around, so there's the first place of improvement
4k ISP frames is A LOT. Eg. RPi 4B can't even handle 20FPS 1080P frames, just to give you an idea of low fps due on low-end computers (CPU). I would suggest lowering resolution and/or encoding (example here) the stream (mjpeg/h264/h265) to reduce bandwidth needed.

Thoughts?
Thanks, Erik

Brac

Hello @erik ,
Thanks for this quick answer ! and sorry for this late return...

For the first point I known that this example was not the most performant, I just linked it to put some context on what I was doing.
In fact, I started from this example by removing everything that could have "side effects" on performance and just keeping a callback on the ISP output in the C++ code.

Indeed 4K is a lot, but the objective was to have the best possible resolution without loss of data during the transfer. That's why the resolution and encoding was not necessarily the first choice ^^

By lowering the resolution to 1080P and by using the video encoder (MJEPG) I can reach about 15 to 20FPS, but a the expense of a loss of precision on the output of the detection algorithm that will consume the image.

Thank you for taking the time to answer my question 😃

I'll let you know if I find an effective implementation to this problme 👍️

erik

Hi Brac ,
Interesting, thanks for the feedback! I would have thought android phones were more performant and there wouldn't be an issue. Are you also displaying/decoding the frames (15/20FPS) or just reading them and discarding them?

Brac

Now I use the 1080P video pipeline encoded in MJPEG.

I create a cv::Mat from the Java side and send the native address to the native side where I fill the Mat.
(it seems more efficient than passing an array of integers as in the example).

If I display the video feedback using a Bitmap (like in the example) I get around 14 to 18FPS
And if I display it through OpenGL (ES2.0) I get around 18 to 22FPS which is starting to be intresting ^^

I am still surprised because I use a device with a Qualcomn XR1 and I would have expected more performance.

Work in progress ^^

erik

Hi Brac ,
This is looking really promising!! If your code is opensource we would love to link it from our depthai-android example repo. Otherwise I can also just mention the details (eg OpenGL for faster live stream) in our repo.
Thanks, Erik