Controlling Class-Specific Augmentation to Address Class Imbalance

shail

Hi Luxonis team,

I’m working on an object detection task, and I was wondering about the augmentation options.

Is it possible to control the augmentation in a way that we can augment underrepresented classes more heavily than others? For example, some mechanism like weights or class-specific augmentation so that we can help address class imbalance during training.

Would like to know if this is supported or if there are recommended approaches.

KlemenSkrlj

Hi @shail
I assume you are asking this in the contex of LuxonisTrain right? If that is the case then unfortunately we don't have this feature in. We use Albumentations library for augmentations with our custom extension that also supports BatchTransforms (e.g. Mosaic4 or MixUp). But when doing the transforms we don't take into account any class distributions. We'll make a feature request on this on our side but at the moment I can't promise any date that this will become available.
So currently what you can do is ideally collect more data for the underrepresented class. Or a more naive approach that could also work is to bake in augmentations to the training dataset. What I mean by this is that you augment specific images (e.g. of the underrepresented class) and save them directly to the training dataset. During training they would still be included in the augmentation step (so essentially augmented twice) but you would have expanded the amount of images that this class is represented on.
Hope this helps,
Klemen

shail

Ok, no problem. Thanks for clearing that up, Klemen.

Following up on this, I wanted to ask one more thing. Is there a way to print a confusion matrix when working with object detection and multiple classes? From my understanding, we can either call test or infer after training, but neither of these commands seems to print the confusion matrix.

I was wondering if it’s possible to do that, or if I’m missing another command.

KlemenSkrlj

We do have ConfusionMatrix metric supported. So if you add it in the config (e.g. here) and call the test it should be showing up in the printout (and also shown during the .
The infer method is for when you want to infer on data for which you don't have ground truth (e.g. infer on some extra images or video "from the wild") and since you don't have the GT the metrics are not computed.

shail

Hey, sorry, yes I know about this config but I wanted to know how to define the metrics while using a predefined model. I am specifically using detection light model. My config is similar to this

model:

name: detection_light

predefined_model:

name: DetectionModel

variant: light

params:

  loss_params:

    iou_type: "siou"

    iou_loss_weight: 20

    class_loss_weight: 8

KlemenSkrlj

You are correct, my mistake. We do compute ConfusionMetrix (it is enabled by default on the predefined model here) and we do push it to the Tracker (e.g. to Tensorboard or MLFlow as a table) but you are right, we currently don't show it in the output logs (in the terminal). There isn't really any reason for not doing this, it is just a bug that you have found.
We'll address this issue shortly (if you give me your GH username I can ping you inside it). Thanks for the report.

As a side node: Here are all the available params that are exposed on the predefined models. So if you would want any other metric you can just expand the metrics list under the params.metrics on the predefined model

shail

Thanks Klemen, I appreciate it. So if I understand correctly, I can still view the matrix in tensorboard for that run?
Also here is my github username - shail10. I appreciate the support.

KlemenSkrlj

Correct, it should be under the "Text" tab in Tensorboard.

KlemenSkrlj

Hey @shail , I just wanted to let you know that we made a PR on LuxonisTrain which introduces this feature. Feel free to check it out.

shail

Thanks Klemen. I appreciate it.