Quantized int8 Yolov11 does not detect any boxes

ChuongNguyen

Hi, I tried using Luxonis Hub to convert a Pytorch/ONNX model, so I tried the easiest solution with a Official Checkpoint Yolov11s.pth from Ultralytics.

The model can be converted correctly with fp16 option, and I can see the model detects boxes. However, when I tried to use Int8 or Mixed Precision, although the models can be converted, it does not detect any boxes.

I tried different calibration dataset, and different precisions option, but none is working expect fp16.

KlemenSkrlj

Hi @ChuongNguyen ,
I just now tried to reproduce your issue and I wasn't able to. So I'll list down the steps I took and you go through them and see if any differ from yours:

I downloaded the YOLOv11-n checkpoint from Ultralytics here
I went to HubAI, to the "Quick Conversion" page and uploaded the model. Here I selected:
- Platform: RVC4
- Input shape: 640x640 (matches the original but you can choose whichever works for you)
- Target Precision: INT8 (Standard) - you can go with some other if you want higher accuracy but a bit slower model
- Quantization Data: GENERAL - this is a subset of COCO images which is suitable for this general model
- SNPE Version: 2.32.6 - this is important, we had a bug in HubAI last week where the default was set to 2.23 which affected specifically YOLOv11 models where they were producing wrong predictions. So worth double checking that your model was exported with 2.32.6

After convervision finished I used this model with generic-example to quickly test it out and I got expected results.

Let me know if following this solves the issue for you.
Best,
Klemen

ChuongNguyen

Thank so much for your reply.

I tried again, if I choose the Int8 accuracy as your demo, it works.
the int8int16 mix precision (as shown in the attached image) does not work, no matter what I tried
What is the Maximum Quantized Images should I pick. I were trying with 20, 1000 or leave it empty, but none is working.

KlemenSkrlj

Hi @ChuongNguyen ,
we indeed were able to replicate bad results when INT8INT16 mix was used. These accuracy focused profiles are a but experimental and they don't always yield the inteeded result (we'll make sure that this is clearer in the UI). But we are now internally investigating what can be done about this and we hope to share more information soon.

What is the Maximum Quantized Images should I pick. I were trying with 20, 1000 or leave it empty, but none is working.

You can leave this field empty and it will take all the images from the selected dataset (which is 1024 in this case). General rule of thumb is that the more images you have in quantization set the better the quantization (assuming these images are representative of what model will see in the real deployment).

Hope this gives a bit more context.
Klemen