Vision Language Model on Luxonis Gen 4 Hardware

RakshithSingh

Hi, I would like to know if it is possible to run a vision language model like Paligemma inside the compute unit of the new generation luxonis cameras. We have a few applications that make use of a vision language model to describe the scene so having the ability to run the model inside the camera can avoid adding additional expensive compute units. Thank you for your help!

jakaskerl

RakshithSingh
There are plans to do it for RVC4, but it's low priority currently. Can't say if Paligemma specifically can be converted. Apparently they have an ONNX model for it so this increases the chances that all layers are supported. I suggest you try with following https://rvc4.docs.luxonis.com/software/ai-inference/conversion/ and converting them model. We intend to do it in the future as well.

Thanks
Jaka