Hi all, I just received the OAK1 and I'm impressed so far!

I'm asking here since I am very new to the subject. I would like to try to run one of the OpenVINO models with semantic segmentation on the OAK1, but it's not very clear to me how to include this use this sort of network in DepthAI and how to parse the output since all the examples are based more on object detection.

I downloaded and compiled the models, if someone could give me some pointers on how to process a semantic segmentation network with multiple classes I would be very grateful!

    Hi Brandon,

    thank you very much for your reply! By looking at that example I managed to run another network with multiple classes.
    I have one question on how the network results are read: in the examples there are uses of

    getFirstLayerFp16() 

    for object detection networks and

    output_layers = in_nn.getAllLayerNames()
    layer1 = in_nn.getLayerInt32(output_layers[0]) 

    for the example that you reported here, but I can't find these methods in the documentation.
    Are these always the same for object detection and semantic segmentation, or is there a rule to know which ones to use with different networks?
    Thanks again for finding the time to reply!

      Hi Letty ,

      Thanks! Great questions. I actually don't know either. I haven't dug this deep yet unfortunately.

      I am asking the team and will circle back. And will see about getting all of these documented. In the meantime, I -think- that these are actually poached directly from OpenVINO network parsing, as I think we tried to be compatible with OpenVINO output parsing where possible. (And then in this case, I wouldn't be surprised if OpenVINO inherited what Deeplabv3+ did, but I'm not sure).

      Thanks and will circle back.

      -Brandon

      Hi Letty

      These are used to get the outputs available from inference. For example: https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/intel/age-gender-recognition-retail-0013/description/age-gender-recognition-retail-0013.md

      We can see that there are 2 outputs: 'boxes' and 'labels'. These will be in format as specified when converting the IR to blob. (usually same as the network datatype, by default FP16)

      Usually a getFirstLayer*() is called when there is only one output. If there are multiple we can get by name.

      That's great, thanks a lot to everybody!

      Very welcome! Excited to see how it all works for you!

      5 months later

      Hi @Letty and @Brandon!
      Hoping for some guidance re: OpenVINO semantic segmentation network with multiple labels and depth visualization.
      Is it a matter of entering class_colors and class_binary numpy arrays for all colors and classes in lines 31 and 39 of depthai-experiments/gen2-deeplabv3_depth/main.py ?
      Please advise. Thanks!!

        Hi MarxMelencio ,

        Great to hear from you again!

        Yes, we're more than happy to help. Actually @erik and @Luxonis-Lukasz are already planning on adding a multi-class example that I think should help here.

        I'm thinking we could probably have something out next week.

        @Luxonis-Lukasz - I know you're already planning on adding this to depthai_demo.py, but I think we should make it both as an experiment and in depthai_demo.py. That way folks who want to use it alone with simpler code can do so simply in the example/experiment. And those who just want to try it can do so in depthai_demo.py.

        Thanks,
        Brandon

          Perfect - thanks! I think we should be able to have it next week.

          a year later

          I'm trying to do segmentation in C++ using the above example as a guide but it's not working. I wonder if someone in the know could offer some guidance as I can't find any documentation that helps.

          I already have everything working with the OAK-1 for mobilenet and YOLO detection so my setup must be ok.

          Looking at what I might have to change from, say my mobilenet code, I am guessing that firstly the creation of the pipeline has to be different?

          Instead of:
          auto nn = pipeline.create<dai::node::MobileNetDetectionNetwork>();

          Should I use the following?
          auto nn = pipeline.create<dai::node::NeuralNetwork>()

          If not, what should I use instead?

          If this is correct then what do I do to capture the segmentation results?

          I am currently doing this:

          auto q_nn = device.getOutputQueue("nn", 4, false);
          auto nndata = q_nn->get<dai::NNData>();
          CIntVector res;
          if (nndata)
             res = nndata->getFirstLayerInt32();

          The problem is this seems to hang indefinitely and no NNData is returned.

          Btw. I'm using road-segmentation-adas-0001_openvino_2021.4_6shave.blob for this.

          Clearly I am missing something so any pointers would be much appreciated.

          • erik replied to this.

            Hi Malcs ,

            I have already replied to the other forum post regarding C++ semantic segmentation demo, and we should likely continue the conversation there. Just to quickly answer your questions:

            Should I use the following?

            Yep, as it's not SSD/Yolo output/model, you would need to use NeuralNetwork node.

            If this is correct then what do I do to capture the segmentation results?

            Looks correct, but really depends on the pipeline. I would start with python demo and convert it 1:1 to C++.
            Thanks, Erik

            Apologies - I wasn't sure where was best to post. Will continue over on that other thread.

            Thanks Erik!