Yolov11n model performance on OAK-D-CM4 POE camera

GurdeepakSidhu

Hi Luxonis Team,

I’ve trained a YOLOv11n model on 414×416 images using my GPU VM, and the performance during inference on the VM is very good — it correctly distinguishes between "can" and "empty can" classes.

However, after converting the model to a blob using tools.luxonis.com and deploying it to an OAK-D-CM4 POE camera, the model no longer performs reliably. Specifically, it tends to classify both "can" and "empty can" as just "can", ignoring the distinction it made correctly during testing on the host.

What could be the issue here? Am I missing something in the conversion or deployment process?

Your help in this regard would be appreciated.

Thank you and best regards.

KlemenSkrlj

Hi @GurdeepakSidhu
To make the model runnable on the device there is some compression that happens (which is handled by tools.luxonis.com) so some drop in performance is expected. But this final accuracy dependes also on the quality of the base model. One thing I would suggest you try to compare the models on the same images between the original versus the one deployed on the camera. Easiest way to do this would be to save frames that get fed into the camera during inference (output of the .passthrough queue on the neural network node) and run the original model on these same images while also saving the output from the model ran on the device. This way you can have 1:1 comparison.
For improving the model usually adding more images to the dataset improves the the most. So collecting more images from where the model is deployed and finetuning on that.
Let me know how it goes,
Klemen

GurdeepakSidhu

KlemenSkrlj

Thank you for your help, and appreciate the insight you have shared.

I just wanted to clarify that based on your suggestion I need to save the raw frame coming from this detectionNetwork.passthrough.link(xoutRgb.input) and then test this frame for inference with the model on the GPU VM right? Then I can compare the output from the inference between the GPU VM and the OAK camera.

Please do let me know if I am understanding this correctly, and I will proceed with the test you suggested.

Thank you and have a great day.

Best regards,

KlemenSkrlj

Yeah correct. I suggest you save the frame you get from the xoutRgb node and you run inference with the GPU model on this exact image. And to compare 1:1 you also save the visualization that you get on this same frame from the OAK camera (running the .blob file). There is some (rather small) expected difference between these two but if object is detected with high confidence with the GPU model on this frame then it generally should be detected with the OAK pipeline as well.

GurdeepakSidhu

KlemenSkrlj

Thank you for the clarification.

I will proceed with this testing and get back to you on how the results look.

I am grateful for your help and appreciate it.

Have a great day and best regards,

GurdeepakSidhu

Hi Klemen,

I hope you are doing well.

I performed the test you suggested of comparing the inference results of the model (.pt) running on the GPU VM versus model running on the OAK-D-CM4 POE camera on the frame output of the xoutRgb node.

First here is the passthrough output from the Yolo detection node, the previewSize is set to 416,416

Passthrough Output

Incomplete Can

Empty Can

Yolo Detection Inference Output

Incomplete Can

Empty Can

We can see from the output above that the Yolo model deployed on the camera is predicting incorrectly.

GurdeepakSidhu

Below is the output of running the model in the GPU VM on the passthrough images.

Yolo model on GPU VM

Incomplete Can

Empty Can

It seems that even the Yolo model deployed on the GPU VM is not predicting correctly on the passthrough images from the camera.

I am not sure why this is the case, what would you suggest as next steps?

Since if a run a full size image on the same model on GPU VM, and use model.predict(imgsz=416). It gives me the following result.

Raw Image

Inferred Image

As you can see if I use one of the raw images I have (similar to the size of the image used for training) I am getting good confidence for predicting the different classes and it is predicting correctly.

However when I inferred using the passthrough images it predicting inaccurately.

How can I address this issue or any tweaks or improvements I can make?

Thank you and I look forward to hearing from you.

Take care and best regards,

KlemenSkrlj

Thanks for the detailed evaluation. Based on the results I would say your model is overfitted to some specific dataset which doesn't capture what the model sees when actually deployed. One difference is definitely the aspect ratio. To acount for this you can try to set cam_rgb.setPreviewKeepAspectRatio(True) which will create a center crop and keep the aspect ratio. If this doesn't do the trick I think the next step would be to collect more data from the scene where the system will be used, annotate this data and finetune the model with it. Because if model has seen only images where can is perfectly positioned during training it might struggle when there is other stuff in the scene (e.g. the white dividers) and thus not have good enough predictions.

GurdeepakSidhu

KlemenSkrlj

Thank you so much for your feedback and this is appreciated.

I will be testing today with setting cam_rgb.setPreviewKeepAspectRatio(True) . I will get back to you one how the predictions look after apply the setting suggested.

Thank you and have a great day.

Best regards,

GurdeepakSidhu

I wasn't able to test this today, but will be testing with the client on Wednesday afternoon.

But I do have a question in regards to collecting data based on the camera and conveyor setup. Should we be collection annotating images from the passthrough output which 416x416 (either with setPreviewKeepAspectRatio(True) or setPreviewKeepAspectRatio(False)), or we should be collecting annotating images coming from the camRgb video output?

We wanted to make sure that we are doing the data collection correctly, so that we can better improve the model performance.

Thank you and appreciate your help.

Best regards,

KlemenSkrlj

The .passthrough is what the model gets as in input so I'd say you want to collect more images like that. And similarly for aspect ratio: if your intention is to run the model with a pipeline that keeps aspect ratio then use that for collection as well. Esentially you want to expand the dataset with images that better represent actual real world deployment.

GurdeepakSidhu

KlemenSkrlj

Thank you so much for this feedback, we will adjust our data collection setup to taken into consideration collecting .passthrough images of real world deployments. We are in process to set this up with our client.

On an aside how do I correctly draw yolo detection bounding box results on the video frames collected from camRgb when setPreviewKeepAspectRatio is set to True? Previously I had setPreviewKeepAspectRatio to false and was able to drew the bounding box on the frame from .video without an issue using this code,

def frameNorm(frame, bbox):

normVals = np.full(len(bbox), frame.shape[0])

normVals[::2] = frame.shape[1]

return (np.clip(np.array(bbox), 0, 1) * normVals).astype(int)

However this doesn't translate when setPreviewKeepAspectRatio is set to True,

For an example this how the bounding box looks

Thank you for your help, I look forward to hearing from you.

Have a great day!

jakaskerl

GurdeepakSidhu
I'd keep using the keepAspectRatio to false. Otherwise you might loose some information from the sides - https://docs.luxonis.com/software-v3/depthai/tutorials/resolution-techniques/. It's also easier to rescale since the width scaling factor is full_width/squeezed_width.
For cropped image, you have to undo what the preview crop does: downscale from original (video) -> crop to aspect ratio

Thanks,
Jaka

KlemenSkrlj

But note that in that case you also need to use "squished" images in training otherwise model with not work as good when deployed.

GurdeepakSidhu

KlemenSkrlj

Thank you so much for this pointers, we will be working on training the model with squished images, so that we don't have to be concerned with missing information from the sides due to cropping as result of keepAspectRatio to true.

On an aside, we tested running the model with setPreviewkeepAspectRatio to true, and indeed we see better performance for the yolo detection model on the camera. The model was able to predict empty cans accurately, while previously it predicted empty can as can when keepAspectRatio was set to False. I was wondering what would be the drawback of drawing the bounding box on the frame from .video using the approach below:

Also finally, my understanding is that it is better to train a model with squished images and have KeepAspectRatio false on the preview from camera before it pass to the Yolo Detection node. The reason for this is that if we KeepAspectRatio True, it is possible to miss out some important information.

Thank you and I look forward to hearing back from you.

Best regards,

KlemenSkrlj

Glad you see some improvements already and I am sure you'll get a nice model once you train your model further.
Regarding the keep aspect ratio: This is mainly up to you how you want to design your solution. Both options are valid, it just depends what you want to achieve:

setKeepAspectRation(False): The benefit is that yu effectively keep the original FOV because it just naively resizes the whole frame into smaller shape. The drawback is that you need to also train on such squeeshed images as well becuse the objects you want to detect also visually "deform" and your original model is not "used to that"
setKeepAspectRatio(True): Here the benefit is that you can train on "normal" images, probably don't have to change the training code. But as Jaka pointed out you do loose some FOV because the resize also crops the frame in (becuase video and preview aspect ratio are not the same).
So if you are fine with a bit smaller FOV then go with the second option otherwise go with the first but make some adjustments in the training code (usually this resize happens in the augmentation step in the data loader).

GurdeepakSidhu

KlemenSkrlj

Thank you so much for the clarification, and this is quite helpful.

In case of training images that are squished to the size of the input frame from the Yolo detection node, how crucial it is to train using images collected from the conveyor setup? Currently most of the images we have for training the model is not coming from the setup where the camera is deployed. Would this be an issue?

Thank you and I am grateful for your help.

Best regards,

KlemenSkrlj

In both cases (squeeshed images or not) you would ideally want to train on the images that the model will see in production. But this can be an iterative process: You first training images you already have, check the performance, if that si good then great, if it is not then you can collect more data and include it into the next training round.

GurdeepakSidhu

KlemenSkrlj

Thank you so much, I will keep that in mind as we continue with the model development.

I am grateful for your help, and will follow up with any other question I have.

Have a great day and best regards,