Semantic segmentation on OAK1

Letty

Hi all, I just received the OAK1 and I'm impressed so far!

I'm asking here since I am very new to the subject. I would like to try to run one of the OpenVINO models with semantic segmentation on the OAK1, but it's not very clear to me how to include this use this sort of network in DepthAI and how to parse the output since all the examples are based more on object detection.

I downloaded and compiled the models, if someone could give me some pointers on how to process a semantic segmentation network with multiple classes I would be very grateful!

Brandon

Hi Letty ,

Sorry about the delay. So you can actually use the following example to see how to run Semantic segmentation on depthai/OAK:
https://github.com/luxonis/depthai-experiments/tree/master/gen2-deeplabv3_person

So we need to make an example with more than one class. But I -think- it should not be hard to do the equivalent of this for more than one class.

Thoughts?

Thanks,
Brandon

Letty

Hi Brandon,

thank you very much for your reply! By looking at that example I managed to run another network with multiple classes.
I have one question on how the network results are read: in the examples there are uses of

getFirstLayerFp16()

for object detection networks and

output_layers = in_nn.getAllLayerNames()
layer1 = in_nn.getLayerInt32(output_layers[0])

for the example that you reported here, but I can't find these methods in the documentation.
Are these always the same for object detection and semantic segmentation, or is there a rule to know which ones to use with different networks?
Thanks again for finding the time to reply!

Brandon

Hi Letty ,

Thanks! Great questions. I actually don't know either. I haven't dug this deep yet unfortunately.

I am asking the team and will circle back. And will see about getting all of these documented. In the meantime, I -think- that these are actually poached directly from OpenVINO network parsing, as I think we tried to be compatible with OpenVINO output parsing where possible. (And then in this case, I wouldn't be surprised if OpenVINO inherited what Deeplabv3+ did, but I'm not sure).

Thanks and will circle back.

-Brandon

themarpe

Hi Letty

These are used to get the outputs available from inference. For example: https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/intel/age-gender-recognition-retail-0013/description/age-gender-recognition-retail-0013.md

We can see that there are 2 outputs: 'boxes' and 'labels'. These will be in format as specified when converting the IR to blob. (usually same as the network datatype, by default FP16)

Usually a getFirstLayer*() is called when there is only one output. If there are multiple we can get by name.

GergelySzabolcs

Letty
Hello.
I added to the deeplab example utility functions, how to query info about the output layers of the neural network.
See latest changes here : https://github.com/luxonis/depthai-experiments/commit/9e43eaff812363c2c2f504abd4f1033e3c37d5a4

Letty

That's great, thanks a lot to everybody!

Brandon

Very welcome! Excited to see how it all works for you!

MarxMelencio

Hi @Letty and @Brandon!
Hoping for some guidance re: OpenVINO semantic segmentation network with multiple labels and depth visualization.
Is it a matter of entering class_colors and class_binary numpy arrays for all colors and classes in lines 31 and 39 of depthai-experiments/gen2-deeplabv3_depth/main.py ?
Please advise. Thanks!!

Brandon

Hi MarxMelencio ,

Great to hear from you again!

Yes, we're more than happy to help. Actually @erik and @Luxonis-Lukasz are already planning on adding a multi-class example that I think should help here.

I'm thinking we could probably have something out next week.

@Luxonis-Lukasz - I know you're already planning on adding this to depthai_demo.py, but I think we should make it both as an experiment and in depthai_demo.py. That way folks who want to use it alone with simpler code can do so simply in the example/experiment. And those who just want to try it can do so in depthai_demo.py.

Thanks,
Brandon

MarxMelencio

Brandon
Cool!! Yeah, and it's great to be here again! I'll keep posted for the example. Thanks!

Brandon

Perfect - thanks! I think we should be able to have it next week.

Malcs

I'm trying to do segmentation in C++ using the above example as a guide but it's not working. I wonder if someone in the know could offer some guidance as I can't find any documentation that helps.

I already have everything working with the OAK-1 for mobilenet and YOLO detection so my setup must be ok.

Looking at what I might have to change from, say my mobilenet code, I am guessing that firstly the creation of the pipeline has to be different?

Instead of:
auto nn = pipeline.create<dai::node::MobileNetDetectionNetwork>();

Should I use the following?
auto nn = pipeline.create<dai::node::NeuralNetwork>()

If not, what should I use instead?

If this is correct then what do I do to capture the segmentation results?

I am currently doing this:

auto q_nn = device.getOutputQueue("nn", 4, false);
auto nndata = q_nn->get<dai::NNData>();
CIntVector res;
if (nndata)
   res = nndata->getFirstLayerInt32();

The problem is this seems to hang indefinitely and no NNData is returned.

Btw. I'm using road-segmentation-adas-0001_openvino_2021.4_6shave.blob for this.

Clearly I am missing something so any pointers would be much appreciated.

erik

Hi Malcs ,

I have already replied to the other forum post regarding C++ semantic segmentation demo, and we should likely continue the conversation there. Just to quickly answer your questions:

Should I use the following?

Yep, as it's not SSD/Yolo output/model, you would need to use NeuralNetwork node.

If this is correct then what do I do to capture the segmentation results?

Looks correct, but really depends on the pipeline. I would start with python demo and convert it 1:1 to C++.
Thanks, Erik

Malcs

Apologies - I wasn't sure where was best to post. Will continue over on that other thread.

Thanks Erik!