YOLO model for classification

MahdiMseddi

Dear all

I am trying to test a yolo model for classification. I trained the model and converted it to blob format.
When I try to implement the model and extract the final class prediction I cannot access the information.
In fact, the softmax layer is followed by another layer to just show the results but cannot give the probability of each class. I tried getfirstlayerint16 and all related function found in tutorials with other models but it shows only the last layer.
can you check on how to extract the probability information precisely on yolo-cls ?

Thank you very much

jakaskerl

Hi MahdiMseddi
Check this example which shows how to do a similar thing on host. Or alternatively, this one, that does so on the device itself.

Hope this helps,
Jaka

MahdiMseddi

Thank you for your reply. I checked these tutorials but the applied models are different from the yolo-cls which I find issues to extract the final output probablities. In my case I am not using detection, I am only classifying.

the final layer called output0 shows always the same values even though the model is working well before the conversion.
If it is not possible to use yolo classification model as a classifier, do you have a tutorial to convert mobilenet/resnet from temsorflow format to blob format ?

Thank you very much in advance

jakaskerl

Hi MahdiMseddi
What does .getAllLayerNames() return? Or are you using the YoloDetectionNetwork node?

Thanks,
Jaka

MahdiMseddi

getAllLayerManes is returning fixed values ( [0,1] since am classifying 2 classes )eventhough the model works fine before the conversion.
Could this be a conversion problem ? and how to solve it ?

THank you

MahdiMseddi

I tried to convert the model using this tool: DepthAI Tools (luxonis.com) but it doesn't work.
so I am converting it to openvino or onnx and then pass it to : BlobConverter | Convert model to MyriadX blob (luxonis.com)
That is why probably the problem is related to conversion

jakaskerl

@Matija

Matija

Hey @MahdiMseddi ,

Yes, you are correct. We do not support classification models at tools.luxonis.com. But from the graph it looks like the model is correctly exported. My guess would be that the final layer output0 does nothing and is just an output layer. However, you need to be careful when converting the model from ONNX to blob. Since it's from Ultralytics, my guess would be that ONNX expect BGR images in range 0-1, but on OAK we feed BGR images with range 0-255. This means that you need to specify `--scale 255` model optimizer flag when converting ONNX to blob. This should be possible under advanced options.

The output0 should return probabilities of each class. Number at index 0 will be probability for class 0.

MahdiMseddi

Matija

Thank you very much for your reply.
I already tried --scale_values=[255,255,255] and --scale=255 but the probabilities are still fixed and the model loses its performance after the conversion.
I also tried to train other tf model like in OpenVINO + OpenCV-OAK - Tensorflow Image Classification.ipynb - Colaboratory (google.com) but the conversion is not working in the notebook ( errors while converting)

I need to implement a classification model . When I am using yolov8-cls it is converted but loses the performance and when I try to convert the tf models the conversion fails.
Thank you again for your help and I will be grateful if you can give me some hints on how to proceed to make one of the solutions work.

Matija

MahdiMseddi

Could you try checking the name in the ONNX for the Softmax layer and specify `--output layer_name` flag to model optimizer as well? Just in case that output0 layer does something with it

Matija

MahdiMseddi

In case Yolo-Cls expect RGB images and you are feeding BGR from OAK you should also specify the `--reverse_input_channels`.

martin181818

Matija

I will check that and add both params

Thank you

YWei

Hi guys martin181818 Matija

I got a similar problem, which is my ResNet18 image classification trained with PyTorch only shows 2 classes (while there are 7 classes in total). And these 2 detected classes always come with a confidence of 100%

Now I am wondering if I should go for another classification model, or tweak my current model's architecture a bit. Any thoughts please?

Cheers,

Austin

Matija

YWei

Hey,

ResNet18 should work fine. Are you sure:

1/ Your export process is correct? If you use ImageNet normalization PyTorch model will expect normalized images, while images on OAK get passed in unnormalized manner [0-255]. Some more info here and here.

2/ You are correctly post-processing the outputs? Usually Softmax is not a part of the model itself, so you might have to apply it to the outputs you get from the model.

YWei

Thank you so much Matija

Your advice is really on spot. I followed your first point and my model just works way better.

furthermore, may I ask more about your second opinion. Usually how could we really extract the detected results from an image classification model when running OAK camera? Are there any examples I could have a look?

The examples about running a classifier with OAK I can find just process like this way:

## I skip the part of creating node ##

def to_tensor_result(packet):
    return {
        tensor.name: np.array(packet.getLayerFp16(tensor.name)).reshape(tensor.dims)
        for tensor in packet.getRaw().tensors
    }


def softmax(x):
    """Compute softmax values for each sets of scores in x."""
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum(axis=0)


## Here I just jump to the part of retrieving result ##

def to_planar(arr: np.ndarray, shape: tuple) -> np.ndarray:
        resized = cv2.resize(arr, shape)
        return resized.transpose(2, 0, 1).flatten()

        inDet = qDet.tryGet()

        if inDet is not None:
            data = softmax(inDet.getFirstLayerFp16())
            result_conf = np.max(data)
            if result_conf > 0.2:
                result = {
                    "name": labels[np.argmax(data)],
                    "conf": round(100 * result_conf, 2)
                }
            else:
                result = None

Really love to know more about your opinions 🙂

Regards,

Austin

erik

Hi YWei ,
We have a few examples with classification models, and usually you do postprocessing on host/Script node (like softmax);

I hope these help!

YWei

erik

Thank you Erik, I will definitely read them through!

YWei

Hi Majita Matija and Erik erik

Thank you for your explanation, and I tried more today but didn't get any good.

Not very familiar with those concepts mentioned. You said we probably can't use a softmax to retrieve my classified results, so what solutions else I could use to achieve that accurately? i did a bit study then and it seemed like softmax is just a math way to extract the results without really changing any calculated numbers I guess?

Now I have a trained ResNet18 and YOLOV5s-cls model, both trained with PyTorch. To verify that my trained models can actually work well, I ran them on my host to do classification inferences respectively today (so without an OAK D device). They both worked better than on my OAK D, and a part of code is like below:

    # Inference with YOLOv5
    with torch.no_grad():
        results = model(img_tensor)
        
    # Convert the logits to percentages using softmax
    probs = F.softmax(results, dim=1)
        
    # Get class with highest confidence
    confidences, class_idx = probs.squeeze(0).max(0)
    label = f'{class_names[class_idx]} {confidences:.2f}'

Is there anything wrong with the configuration or set-up on OAK D(such as the process of resizing images, colour channel order, etc) or I should try another version of model (e.g., YOLOv6 or higher)? I'm very confused now…

Cheers,

Austin

Matija

To add to that, if you use a basic ResNet18 from torchvision for example, it will not contain a Softmax operation. You can take the ONNX and inspect the last node in netron.app.

In those cases you have to use the softmax like in the example you've shared. Feel free to check out other examples shared by Erik as well.

Best,
Matija

erik

Hi YWei ,
I believe just the softmax layer of the NN can't be executed on the accelerated HW (no support), so it would need to be run elsewhere - perhaps on the host computer, or on-deivce inside the Script node.

YWei

Hi Erik erik

Can you be more detailed please (not very familiar to these…)? Since Oak cannot do anything with Softmax by itself, so this operation is basically done by our computer/host (as most of other examples did)?

And that means we cannot really do softmax with a device in a standalone mode?

Am I understanding your opinion correctly?

Cheers,

Austin