Dear all

I am trying to test a yolo model for classification. I trained the model and converted it to blob format.
When I try to implement the model and extract the final class prediction I cannot access the information.
In fact, the softmax layer is followed by another layer to just show the results but cannot give the probability of each class. I tried getfirstlayerint16 and all related function found in tutorials with other models but it shows only the last layer.
can you check on how to extract the probability information precisely on yolo-cls ?

Thank you very much

    Thank you for your reply. I checked these tutorials but the applied models are different from the yolo-cls which I find issues to extract the final output probablities. In my case I am not using detection, I am only classifying.

    the final layer called output0 shows always the same values even though the model is working well before the conversion.
    If it is not possible to use yolo classification model as a classifier, do you have a tutorial to convert mobilenet/resnet from temsorflow format to blob format ?

    Thank you very much in advance

      Hi MahdiMseddi
      What does .getAllLayerNames() return? Or are you using the YoloDetectionNetwork node?

      Thanks,
      Jaka

        getAllLayerManes is returning fixed values ( [0,1] since am classifying 2 classes )eventhough the model works fine before the conversion.
        Could this be a conversion problem ? and how to solve it ?

        THank you

        Hey @MahdiMseddi ,

        Yes, you are correct. We do not support classification models at tools.luxonis.com. But from the graph it looks like the model is correctly exported. My guess would be that the final layer output0 does nothing and is just an output layer. However, you need to be careful when converting the model from ONNX to blob. Since it's from Ultralytics, my guess would be that ONNX expect BGR images in range 0-1, but on OAK we feed BGR images with range 0-255. This means that you need to specify `--scale 255` model optimizer flag when converting ONNX to blob. This should be possible under advanced options.

        The output0 should return probabilities of each class. Number at index 0 will be probability for class 0.

          Matija

          Thank you very much for your reply.
          I already tried --scale_values=[255,255,255] and --scale=255 but the probabilities are still fixed and the model loses its performance after the conversion.
          I also tried to train other tf model like in OpenVINO + OpenCV-OAK - Tensorflow Image Classification.ipynb - Colaboratory (google.com) but the conversion is not working in the notebook ( errors while converting)

          I need to implement a classification model . When I am using yolov8-cls it is converted but loses the performance and when I try to convert the tf models the conversion fails.
          Thank you again for your help and I will be grateful if you can give me some hints on how to proceed to make one of the solutions work.

              MahdiMseddi

              Could you try checking the name in the ONNX for the Softmax layer and specify `--output layer_name` flag to model optimizer as well? Just in case that output0 layer does something with it

                MahdiMseddi

                In case Yolo-Cls expect RGB images and you are feeding BGR from OAK you should also specify the `--reverse_input_channels`.

                    4 months later

                    Hi guys martin181818 Matija

                    I got a similar problem, which is my ResNet18 image classification trained with PyTorch only shows 2 classes (while there are 7 classes in total). And these 2 detected classes always come with a confidence of 100%

                    Now I am wondering if I should go for another classification model, or tweak my current model's architecture a bit. Any thoughts please?

                    Cheers,

                    Austin

                          YWei

                          Hey,

                          ResNet18 should work fine. Are you sure:

                          1/ Your export process is correct? If you use ImageNet normalization PyTorch model will expect normalized images, while images on OAK get passed in unnormalized manner [0-255]. Some more info here and here.

                          2/ You are correctly post-processing the outputs? Usually Softmax is not a part of the model itself, so you might have to apply it to the outputs you get from the model.

                          • YWei replied to this.

                              Thank you so much Matija

                              Your advice is really on spot. I followed your first point and my model just works way better.

                              furthermore, may I ask more about your second opinion. Usually how could we really extract the detected results from an image classification model when running OAK camera? Are there any examples I could have a look?

                              The examples about running a classifier with OAK I can find just process like this way:

                              ## I skip the part of creating node ##

def to_tensor_result(packet):
    return {
        tensor.name: np.array(packet.getLayerFp16(tensor.name)).reshape(tensor.dims)
        for tensor in packet.getRaw().tensors
    }


def softmax(x):
    """Compute softmax values for each sets of scores in x."""
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum(axis=0)


## Here I just jump to the part of retrieving result ##

def to_planar(arr: np.ndarray, shape: tuple) -> np.ndarray:
        resized = cv2.resize(arr, shape)
        return resized.transpose(2, 0, 1).flatten()

        inDet = qDet.tryGet()

        if inDet is not None:
            data = softmax(inDet.getFirstLayerFp16())
            result_conf = np.max(data)
            if result_conf > 0.2:
                result = {
                    "name": labels[np.argmax(data)],
                    "conf": round(100 * result_conf, 2)
                }
            else:
                result = None

                              Really love to know more about your opinions 🙂

                              Regards,

                              Austin

                              • erik replied to this.

                                  erik

                                  Thank you Erik, I will definitely read them through!

                                    To add to that, if you use a basic ResNet18 from torchvision for example, it will not contain a Softmax operation. You can take the ONNX and inspect the last node in netron.app.

                                    In those cases you have to use the softmax like in the example you've shared. Feel free to check out other examples shared by Erik as well.

                                    Best,
                                    Matija

                                    • YWei replied to this.

                                      Hi Majita Matija and Erik erik

                                      Thank you for your explanation, and I tried more today but didn't get any good.

                                      Not very familiar with those concepts mentioned. You said we probably can't use a softmax to retrieve my classified results, so what solutions else I could use to achieve that accurately? i did a bit study then and it seemed like softmax is just a math way to extract the results without really changing any calculated numbers I guess?

                                      Now I have a trained ResNet18 and YOLOV5s-cls model, both trained with PyTorch. To verify that my trained models can actually work well, I ran them on my host to do classification inferences respectively today (so without an OAK D device). They both worked better than on my OAK D, and a part of code is like below:

                                          # Inference with YOLOv5
    with torch.no_grad():
        results = model(img_tensor)
        
    # Convert the logits to percentages using softmax
    probs = F.softmax(results, dim=1)
        
    # Get class with highest confidence
    confidences, class_idx = probs.squeeze(0).max(0)
    label = f'{class_names[class_idx]} {confidences:.2f}'

                                      Is there anything wrong with the configuration or set-up on OAK D(such as the process of resizing images, colour channel order, etc) or I should try another version of model (e.g., YOLOv6 or higher)? I'm very confused now…

                                      Cheers,

                                      Austin

                                      • erik replied to this.

                                            Hi YWei ,
                                            I believe just the softmax layer of the NN can't be executed on the accelerated HW (no support), so it would need to be run elsewhere - perhaps on the host computer, or on-deivce inside the Script node.

                                            • YWei replied to this.