art project background-removal and motion estimation.

CorLangerak

Hi I am a young artist in the Netherlands. Currently I am working on an interactive video installation.
I would like to mask-out only the moving people from the background in realtime, and paste them onto a video that plays in a loop. I have bought this amazing device in the hope that I would be able to do this. I think the kit could be great for artists and I am very happy to be able to interact with, though I am quite new to coding and don't know my way around very well yet.
I would really appreciate if someone could give me an insight or thought on how to achieve this.
Would this be possible or is it too difficult of a task.
Again all help is greatly appreciated!
I hope this my explanation is clear enough 🙂

dhruvsheth-ai

So this isn't a very tought task, but you might require some editing.
You can use some tutorial such as -
https://docs.opencv.org/3.4/d1/dc5/tutorial_background_subtraction.html#:_{:text=OpenCV%3A%20How%20to%20Use%20Background%20Subtraction%20Methods&text=Background%20subtraction%20}(BS)%20is%20a,scene)%20by%20using%20static%20cameras.

But instead of the video frame, you might require tweaking code to use OAK-D as cam input. This produces output as -

Note: It captures all objects which are in motion. Not only moving people.

CorLangerak

dhruvsheth-ai Thanks for the reply! I had tried it, but the results are very noisey and it detects every thing that moves. I am wondering if I wouldn't get a better result by using an instance segmentation model like: Mask R CNN on regions containing motion.

Is there an instance segmentation like Mask R-CNN for the OAK-D?
I have looked at deepvlab, but its not very accurate...
How could I convert such a model?

mfocka

dhruvsheth-ai, quick question about this, is there a way that we can convert this process (MOG2) to a blob (using PyTorch or something) and run it on the VPU of the camera?
I have tried this, but faced some issues with keeping hidden states BETWEEN forward passes, instead of one hidden state for each forward pass, i.e., updating the background model.

Thanks 🙂

Brandon

So CorLangerak I am not familiar with any instance segmentation models that could be run on-device. As instance-segmentation is usually a bit heavier. That said, a lot of these models are being optimized over time and then becoming tractable.

The PINTO0309 modelzoo is a good one to look through to see what is the latest:
https://github.com/PINTO0309/PINTO_model_zoo

Katsuya updates it so often that I am not up to speed on the latest.