Hello, I tried training a model using Ultralytics to generate a YOLOv5n6u model using PyTorch. The image size is 640. When I toss the 'model.pt' file to the DepthAI Tools converter ( http://tools.luxonis.com/ ) to generate the blob, it returns that there is an "Error while loading model". The model file is 8 MB and above the limit for attachment. However, is there an issue with my current approach? Is it possible to share my .pt model file for testing to make sure that it's possible to convert into a blob for use on an OAK-D-S2 device? Thank you!

Hi @"DiproChakraborty"#1178 , After checking, I was able to get `YOLOv8n` running on the device. Steps: 1. Use tools.luxonis.com to convert to xml/bin/json 2. Use [gen2-yolo/device-decoding/main.py](https://github.com/luxonis/depthai-experiments/blob/master/gen2-yolo/device-decoding/main.py) script and specify the downloaded json path as the `--config` arg 3. Make sure to use latest blobconverter (1.4.1) which supports **openvino 2022.1**. We just released it an hour ago, previous version only supported 2021.4 version, whcih didn't work with ultralytics models. We also created [PR for SDK here](https://github.com/luxonis/depthai-experiments/blob/master/gen2-yolo/device-decoding/main.py), so it will work out of the box. After some testing, with the YOLOv8n we got: - `640x352`: 19FPS - `512x288`: 27FPS I chose these input sizes to be as close to 19:6 aspect ratio (like the camera) as possible. I hope this helps.

Hi @"DiproChakraborty"#p7485 , I tried the same approach (ultralytics hub -> tools.luxonis.com -> deploy to OAK) a few weeks back and it worked as expected. Could you share your .pt file (via gdrive or similar) so we can take a look at the conversion? Thanks, Erik

If you get an error when loading a model into the DepthAI tool converter, it could be due to various reasons, including an incorrect model format, errors in the model file, or problems with the DepthAI tool converter. If your .pt model file exceeds the attachment size limit, you can try compressing it to a smaller size. You can use model compression techniques such as Quantization Aware Training or Knowledge Distillation to reduce the size of your model without losing accuracy.

Hi @"erik"#175 , Here is a GDrive link to the .pt file. Not sure what's going wrong with the conversion. I would also add that the model file does not exceed the attachment size limit for the DepthAI tools converter, only for the support forum. https://drive.google.com/file/d/1A27lbfDpraTTur48NKLfbuKJV9hx7-Bh/view?usp=share_link

Hi @"DiproChakraborty"#p7513 I believe "detection only" refers to the model being able to make object detections - as opposed to detection, segmentation and classification of objects ([source](https://github.com/ultralytics/ultralytics/tree/522f1937edb0728f1fb1ec6398ae56d7f476c3a4)). If detection is what all you need then yes, it can support SD program. Hope this helps, Jaka

Converting YOLOv5n6u PyTorch Weights to Blob

erik

Hi DiproChakraborty ,
Looks like an issue with architecture. I'd suggest going with newer version (od ultralytics hub), or looking at/using our YoloV5 notebook.
Thanks, Erik

DiproChakraborty

Hi @erik ,

I was able to use a YOLOv8 model and convert it correctly using the conversion tool. It says on DepthAI Tools that YOLOv8 supports 'detection only' -- what is this referring to? Can I use a YOLOv8 model blob to support a Spatial Detection program, for example? What are the limitations of the YOLOv8 'detection only' compared to YOLOv5?

Thank you!
Dipro

jakaskerl

Hi DiproChakraborty
I believe "detection only" refers to the model being able to make object detections - as opposed to detection, segmentation and classification of objects (source).
If detection is what all you need then yes, it can support SD program.

Hope this helps,
Jaka

DiproChakraborty

Hi @erik ,

After converting the YOLOv8 model using the DepthAI tool, I tried loading the blob into a demo SpatialDetection "spatial_tiny_yolo.py" script to see if the model would load correctly. However, the script crashes on loading the nnBlobPath, returning IndexError: map::at: key not found . Is this an issue with trying to adjust the 'spatial_yolo.py' demo script, or an architecture issue with converting YOLOv8 to the blob using the DepthAI tool?

I've attached the YOLOv8 model (which converts using the DepthAI tool) as 'model3.pt'
The converted zip file returned is attached as 'model3.zip'
The Spatial Detection code file is attached as 'spatial_model3.py'
All 3 can be found within the GDrive.

The ultimate goal is just to be able to run a YOLO model for object detection and do spatial detection.

https://drive.google.com/drive/folders/1xgy6brpU2lhBzERSS0KlAZ5d6LnZC4ol?usp=share_link

Please advise,
Dipro

erik

Hi DiproChakraborty ,
Seems like the pt is in incorrect format;

Downloading C:\Users\Erik\.cache\blobconverter\model3_openvino_2021.4_6shave.blob...
{
    "exit_code": 1,
    "message": "Command failed with exit code 1, command: /opt/intel/openvino2021_4/deployment_tools/inference_engine/lib/intel64/myriad_compile -m /tmp/blobconverter/6323bdb603fe4574931f2cba620be919/model3/FP16/model3.xml -o /tmp/blobconverter/6323bdb603fe4574931f2cba620be919/model3/FP16/model3.blob -c /tmp/blobconverter/6323bdb603fe4574931f2cba620be919/myriad_compile_config.txt -ip U8",
    "stderr": "Unknown model format! Cannot find reader for model format: xml and read the model: /tmp/blobconverter/6323bdb603fe4574931f2cba620be919/model3/FP16/model3.xml. Please check that reader library exists in your PATH.\n",
    "stdout": "Inference Engine: \n\tIE version ......... 2021.4.2\n\tBuild ........... 2021.4.2-3974-e2a469a3450-releases/2021/4\n"
}

How did you get the .pt model - from ultralytics hub?

DiproChakraborty

Hi erik - That’s correct, I got the YOLOv8 .pt weights from Ultralytics Hub. The converter tool did return a zip file and processed, but was that converted model also not in the right format? The zip file post-depthAI tool model was also in the GDrive.

Is there any way to salvage these weights / the converted model within the zip file? I was hoping the .pt model could be converted into a usable format for the OAK camera.

Thanks!

EDIT: Another thing that is strange is that when I download the zip file of the exported model from DepthAI Tools, the name of the blob I get is 'model3_openvino_2022.1_6shave.blob', not 'model3_openvino_2021.4_6shave.blob'. Hopefully the correct blob is being used.

DiproChakraborty

As a followup, if there is any model off Ultralytics Hub that can be correctly converted to a version that can be deployed on the OAK-D and run spatial detection, that would be much appreciated. I tried converting my previous .pt weights to an onnx model and then using various Luxonis tools, but none of them seem to allow the model to work. My use case is running object detection on custom objects and returning spatial information back.

Thank you,
Dipro

erik

Hi DiproChakraborty ,
I know I used YoloV5s on ultralytics hub together with tools.luxonis.com and it worked as expected. Could you perhaps try the same? It might be that yolov8 is in some different format, I haven't tried it yet.
Thanks, Erik

DiproChakraborty

Hi,

I tried using some of the YOLOv5 models (like YOLOv5n6u at the top of the thread), but the DepthAI tools converter returned 'Error while loading model'. Could you share which specific YOLOv5 model you used that worked? I attach a picture of all available YOLO models on Ultralytics Hub

erik

Hi DiproChakraborty , I used Yolov5s.

DiproChakraborty

Hi,

How did you choose or find YOLOv5s in this set? When I add a custom dataset, it only lets me choose between the previously attached models for training on a custom dataset. Is it still possible for you to use YOLOv5s on a custom dataset? I'm wondering if something changed on Ultralytics Hub or if there are specific permissions for you to access YOLOv5s for custom dataset training? Are you able to see the other models as well?

As you can see below, I don't see an option to choose YOLOv5s?

erik

Hi DiproChakraborty , Looks like they updated the ultralytics hub, previously I was able to choose that, not anymore. Perhaps you could try with other models.

DiproChakraborty

Hi,

Correct, the Ultralytics Hub seems to be updated. None of the YOLOv5 models seem to be able to be converted using the DepthAI Tool now. The only model that seems to be able to be converted with the DepthAI Tool is "YOLOv8n". However, as you previously tried, it seemed like that pt was in the incorrect format and could not find the reader for model format xml and read the model. So currently, it seems that no YOLO models on Ultralytics Hub can make it all the way down to a deployable OAK-D model. Is there any way this can be corrected?

Is there any way to adjust the blob converter or OpenVINO converter en route to the MyriadX blob converter? I can see that these models work in PyTorch, but they cannot be converted to be deployed to the OAK-D currently.

Alternatively, is there to correct the converter or allow the VOLOv8 model to work on the OAK-D?

As another option, is there a way to allow any of the Ultralytics Hub models YOLOv5 (or v8) to convert with the DepthAI Tool?

erik

Hi DiproChakraborty ,
Yes, you can also go with the "manual path" - using OpenVINO tools. Docs here: https://docs.luxonis.com/en/latest/pages/model_conversion/
I hope this helps.

DiproChakraborty

Hi @erik ,

I tried using the online Blobconverter app to convert the Ultralytics HUB model 'YOLOv5su'. It was able to return a YOLOv5su blob. Does this blob work for you for spatial detection?

YOLOv5su Blob: https://drive.google.com/drive/folders/1tU07tntcYj6dhEAo6f78LZZWbT-GvXUQ?usp=share_link

erik

DiproChakraborty Could you follow the tutorial here? https://github.com/luxonis/depthai-experiments/tree/master/gen2-yolo/device-decoding#yolo-with-depthai-sdk-tutorial

DiproChakraborty

Hi @erik ,

The tutorial seems to ask for multiple files in addition to the .blob file. When I submit an ONNX model to BlobConverter, I only get returned the .blob file. As a reminder, I used Ultralytics Hub to train a YOLOv5su model, then downloaded the ONNX weights from Ultralytics Hub. I then converted the ONNX weights to the Blob Converter using http://blobconverter.luxonis.com/ . I could not use DepthAI Tools to convert the trained model in PyTorch, because it does not convert YOLOv5su models correctly and returns an error.

When I try following the tutorial and running the model, I get an error as follows:

python main_api.py -m dome_handle_yv5su_try2.blob

returns the following error:

{'classes': 1, 'coordinates': 4, 'anchors': [1], 'anchor_masks': {}, 'iou_threshold': 0.5, 'confidence_threshold': 0.5}

[18443010E157E40F00] [20.4] [2.005] [DetectionNetwork(1)] [error] Mask is not defined for output layer with width '3549'. Define at pipeline build time using: 'setAnchorMasks' for 'side3549'.

Despite the fact that I have the PyTorch/ONNX model, I'm not sure there is a way to convert it into a usable OAK-D model? Is there something further I need to modify with the model or the JSON to allow it to run? I do not need any anchors for this YOLO model.

erik

Hi DiproChakraborty ,
For YOLO models you need to supply anchors and masks, and since you didn't supply these, it throws an error. Using tools.luxonis.com will generate correct anchors/masks from .pt model.
Thanks, Erik

erik

Hi @DiproChakraborty ,
After checking, I was able to get YOLOv8n running on the device. Steps:

Use tools.luxonis.com to convert to xml/bin/json
Use gen2-yolo/device-decoding/main.py script and specify the downloaded json path as the --config arg
Make sure to use latest blobconverter (1.4.1) which supports openvino 2022.1. We just released it an hour ago, previous version only supported 2021.4 version, whcih didn't work with ultralytics models. We also created PR for SDK here, so it will work out of the box.

After some testing, with the YOLOv8n we got:

640x352: 19FPS
512x288: 27FPS

I chose these input sizes to be as close to 19:6 aspect ratio (like the camera) as possible.

I hope this helps.

DiproChakraborty

Hi @erik ,

Yes, I am also able to get YOLOv8n running on the device! I was also able to configure the spatial detection YOLO script to return X Y Z values of the YOLOv8n object detection objects as well.

Thank you for all your help 🙂 Phew!