Spatial for custom net - doable?

tarocal

Hello,

I have been using Spatial Detection Network module with pipeline.createMobileNetSpatialDetectionNetwork()

It has been amazing. But I wanted to use the same funtionality for my custom nets. The model is something like this. This has been difficult. Is it because it is not yet possible? or is this not something you will be rolling out?

Something like facenet, which is very similar to mobilenet is easy to adapt to this form. But using custom nets, and finding the location of the detected object's heatmap concentration is not working. Any fixes or examples?

The error seems to be from the ROI.

[14442C10B12849D200] [16.722] [SpatialDetectionNetwork(1)] [error] ROI x:0.7597127 y:0.902489 width:0 height:0 is not a valid rectangle.

... gets posted.

Also, there are only 28 detections always (for Face or Mobile networks), why is that?

P.S: trying with the following:
VM Workstation 15+, Ubuntu 20.04, OAK-D, Depathai 2.1.0.0, model: posenet_0005

Brandon

Hi tarocal ,

Great question. This is possible and supported. So from what I'm seeing is that it's just the format of output from your network is different than the format of input that the spatial location calculator is expecting.

So the Spatial Location Calculator needs input in the format defined here:
https://docs.luxonis.com/projects/api/en/latest/components/nodes/spatial_location_calculator/

And it looks like the ROI returned by your neural network are simply in a different format.

So what would be required here is to process the bounding box on the host, convert the metadata results to the format that the Spatial Detection Network requires, and then feed that into the Spatial Detection Network. I think we have an example of processing custom networks on the host.

I'm pinging Erik offline as he knows more about this than me.

We can probably help write an example of this as well.

Thanks,
Brandon

erik

Hello tarocal ,
As Brandon mentioned, the problem is the output of your network - apparently, it's not the same as the usual MobileNet output.

option would be to decode the output on the host and sending SpatialLocationCalculatorConfig back to the device
option would be to decode the NN output in script node (similar example here) and create SpatialLocationCalculatorConfig on the device and feed that into SpatialLocationCalculator.

Demo of sending SpatialLocationCalculatorConfig to device here.
Hopefully, this explanation was clear🙂
Thanks, Erik