Questions on pretrained weights & external YOLO support in luxonis-train

shail

Hi team, I had few questions
1. Is there any way to load custom pretrained weights from external YOLO models (e.g., Ultralytics YOLOv8 or any other version) into the luxonis-train framework?
2. The default pretrained weights for EfficientRep backbone appear to be trained on COCO — is that correct? Are there weights pretrained on other datasets available?
3. If we want to use a full YOLOv8 architecture or any other version architecture(not just the head), is that something supported in any way?

Oskar-Luxonis

cc @KlemenSkrlj

KlemenSkrlj

Hi @shail ,

Is there any way to load custom pretrained weights from external YOLO models (e.g., Ultralytics YOLOv8 or any other version) into the luxonis-train framework?

Not directly. The predefined detection network in luxonis-train is a custom architecture, inspired by YOLOv6 with some of our custom improvements. If you would like to load pretrained weights from YOLOv8 (or any other model that is not directly supported) you would need to register custom nodes and losses through our Registry pattern (documentation here and relevant tutorial here).

The default pretrained weights for EfficientRep backbone appear to be trained on COCO — is that correct? Are there weights pretrained on other datasets available?

Currently only COCO pretrained weights are available.

If we want to use a full YOLOv8 architecture or any other version architecture(not just the head), is that something supported in any way?

Here answer would be similar to the answer for the 1st question. Luxonis-train is designed in a way that is very extendable through the Registry pattern. So tehnically you can implement a completely custom architecture, losses, metrics, etc.

The main reason why Ultralytics architectures are not implemented out of the box is that these models are under the A-GPL license which we found is usually not what our customers want due to specific commercial use requirements. But we still allow anyone to register the architecture by themselves if they are fine with that.
And we also support direct conversion flow for Ultralytics pytorch checkpoints through our Hub conversion (documentation here) - this is if you already have a model trained with Ultralytics package, outside of luxonis-train.

Best,
Klemen

shail

Thanks @KlemenSkrlj . This was really helpful. We will follow the documentation for registering custom nodes. I also have few questions regarding results after training. I will share them here.

1. When we run evaluation on our test set, the confusion matrix produced by luxonis-train doesn't reconcile with the number of test samples. For example, with 100 test images, the cell counts (true positives, false positives, false negatives, etc.) sum to less than 100, some items are silently being skipped and never counted into any cell.

2. For our object detection models, we'd like to generate these curves - a precision-vs-confidence curve, a recall-vs-confidence curve, and a precision-recall curve - all swept across confidence thresholds so we can see how precision/recall trade off. Does luxonis-train support generating or logging any of these curves for detection out of the box, or expose the raw scored test-set predictions (boxes + confidence scores + matched ground truth) that we'd need to compute them ourselves in a post-training step? Just trying to confirm what's available before we build anything on our side.

Thanks

KlemenSkrlj

shail
1/ This sounds like a bug in our side.
2/ We don't support this yet out of the box. But it should be possible to implement as new metrics. There is one caveat currently which is that the head is responsible for postprocessing results before the data comes to the attached modules (e.g. metrics and visualizers). This means that the predictions that you would get right now into your eg. PrecisionConfCurve metric class wouldn't be raw. So these would need to be handled a bit differently.

For both of the points we would love to collaborate with you. Luxonis-train is a fully opensource project so I would encourage you to make PRs with features or fixes. Or an Issue with a MRE (e.g. for the first point you raised about wrong conf matrix results). This is mainly because Github is better suited for this kind of reports.
I encourage you to become contributors to the package 🙂

shail

Thanks @KlemenSkrlj . Next time when we run an inference, I will check the confusion matrix again and make a PR. And regarding point 2 I will make a feature request when we have some downtime. Thanks again for your time. Helpful as always.