How to use YOLOv5n-seg for object segmentation with OAK1/OAK-D?

Hi @daniqsilva
I'm not really too familiar with decoding but try this (looks sensible to me):

import numpy as np
import cv2

def upsample_and_combine(outputs, img_width, img_height, num_classes):
    # Upsample each output to the original image size
    upsampled_outputs = [cv2.resize(output, (img_width, img_height), interpolation=cv2.INTER_NEAREST) for output in outputs]
    
    # Combine the upsampled outputs
    # This example simply averages the outputs, but other techniques could be applied
    combined_output = np.mean(upsampled_outputs, axis=0)
    
    return combined_output

def decode_segmentation(output, img_width, img_height, num_classes):
    class_predictions = np.argmax(output, axis=-1)
    segmentation_map = np.zeros((output.shape[0], output.shape[1], 3), dtype=np.uint8)
    colors = np.random.randint(0, 255, size=(num_classes, 3), dtype=np.uint8)
    for class_id in range(num_classes):
        segmentation_map[class_predictions == class_id] = colors[class_id]
    segmentation_map_resized = cv2.resize(segmentation_map, (img_width, img_height), interpolation=cv2.INTER_NEAREST)
    return segmentation_map_resized

# Assuming outputs is a list of the three outputs from YOLOv5-seg, each being a (H, W, C) tensor
outputs = [output1, output2, output3]  # Placeholder for actual model outputs
num_classes = 20  # Example number of classes
img_width, img_height = 1280, 720  # Example dimensions

# Process the outputs
combined_output = upsample_and_combine(outputs, img_width, img_height, num_classes)
segmentation_map = decode_segmentation(combined_output, img_width, img_height, num_classes)

# segmentation_map now holds the final segmentation result

Thanks,
Jaka

Hi @daniqsilva,

apologies for the delay in our response. Currently our tools support the conversion of only object detection models. We are also actively working on supporting instance segmentation Yolo models. However, we don't have any ETA yet. Could you please elaborate on how you converted the model using our tools? Did you edit the model in some way?

Best,
Jan

Hi @daniqsilva,

I see; it's interesting that the conversion didn't fail.

I apologize, but as I said, currently, http://tools.luxonis.com doesn't support the conversion of segmentation models for any Yolo version. We are actively working on supporting it. As soon as we add it, we'll let you know. Until then, unfortunately, you will have to convert the model partially manually with the help of our blobconverter or use a different segmentation model.

I am very sorry for the unclear description of the conversion options and that I can't help you more now.

Best,
Jan

Hi @daniqsilva @JanCuhel ,

Did you find an alternate approach to run instance segmentation models in the OAK devices? Is there any other model architecture that is supported by oak (which can be loaded with a model trained on custom data)? I also want to get the depth (z - axis) value along with the center (X, Y) points and segmentation masks array. By when could the on device support for yolo segment models be included @JanCuhel ?

daniqsilva

Does that also give us the depth calibrated distance?

Hi!

Maybe it will help:

"The only approach that I've found was this one.">>> Here use tensorflow 1.15 and below version 2 is no longer supported in google colab. Last TF version (2.16) it is not possible to convert with openvino and blobconverter. It's an eternal labyrinth

Pytorch is the only option for Mobilenet and depthai.

The ideal is yolo5
It works better than mobilenet, it is quick to train and convert with tool.luxonis

Hi @daniqsilva , 1000 thanks!!! excellent work, thanks for sharing!!!