I haven't worked with the onboard processing capabilities of the Oak-D much so I'm trying to get a grasp on what to expect from the hardware on the device.

My plan was to do the neural network inferencing on the host device and use a laptop or something with a GPU like a Jetson. However, if the oak-d can handle the segmentation task then I would be able to use much less power hungry hardware for the rest of the system and free up a lot of cooling capacity. This will eventually be going on a mobile autonomous vehicle so unfortunately I don't have the luxury of a machine with a lot of compute.

What I have in mind is a U-Net with 3 levels on the encoder side and 3 levels on the decoder side. The input image (the rgb image from the oak-d) would be 600x400 and the output image is the same size. All I really need is about 10fps for this to be acceptable. Does anybody know of an example of something similar being done? Also is there a way to try to calculate how much compute a model would require so that I can compare to what is available on the oak-d?

Hi @silo ,
I don't think we've tried running U-NET directly on-device, but I guess it's tool large. I'd suggest checking these docs:
https://docs.luxonis.com/hardware/platform/rvc/rvc2/#Robotics%20Vision%20Core%202%20(RVC2)-RVC2%20NN%20Performance
Paper that talks about using OAK to run obj det., and only send crops to Jetson, where they ran U-NET:
https://www.preprints.org/manuscript/202410.0903/v1

Thoughts?
Thanks, Erik

Hi @erik,

I had assumed that a network that large would probably just be too much for an oak-d but wanted to double check if somebody had possibly found a way to make it work. I have seen examples of people attempting to use Topformer on the oak-d hardware with varying degrees of usability, but wasn't able to find anybody that had put together a custom U-Net or CNN for segmentation on the device.

I appreciate your effort looking up that article. I think the application is somewhat different to mine but there may be some useful bits in there that might apply if I end up settling on a jetson.

Thank you for the information.