Help Needed: Deploying MobileNetSSD-based Object Detection Model on OAK-D-CM4

KlemenSkrlj

Which model did you end up training in LuxonisTrain? Could you share the training config (.yaml file)? We can try it out on our "sample" dataset and compare detections between pytorch and .blob.
But generally the models should be convertable to a good degree and you should see the detections with blob most of the time, especially if you lower the confidence threshold. Maybe try to increase the IoU threshold as well? Olso on this: are you using ParsingNeuralNetwork node for NN inference and parsing? Just want to make sure its not in issue with incorrect parsing.

nileena_thomas

@KlemenSkrlj This is the yaml file I used -

model:
  name: can_detection_01aug2025_augmented
  predefined_model:
    name: DetectionModel
    params:
      variant: light
      loss_params:
        iou_type: "siou"
        n_warmup_epochs: 0 # No assigner warmup
        iou_loss_weight: 20 # Should be 2.5 * accumulate_grad_batches for best results
        class_loss_weight: 8 # Should be 1 * accumulate_grad_batches for best results

loader:
  params:
    dataset_name: can_defect_dataset_v9

trainer:
  preprocessing:
    train_image_size: [416, 416]
    keep_aspect_ratio: true
    normalize:
      active: true
      params:
        mean: [0., 0., 0.]
        std: [1, 1, 1]
    augmentations:
      - name: Affine
        params:
          scale: [0.7, 1.7]
          rotate: 20
          shear: 5
          p: 0.3
      - name: HorizontalFlip
        params:
          p: 0.3
      - name: ColorJitter
        params:
          brightness: [0.8, 1.2]
          contrast: [0.8, 1.2]
          saturation: [0.8, 1.2]
          hue: 0
          p: 0.2

  batch_size: 8
  epochs: &epochs 400
  accumulate_grad_batches: 8 # For best results, always accumulate gradients to effectively use 64 batch size
  n_workers: 8
  validation_interval: 10
  n_log_images: 8

  callbacks:
    - name: EMACallback
      params:
        decay: 0.9999
        use_dynamic_decay: True
        decay_tau: 2000

  training_strategy: # Fine tuning params
    name: "TripleLRSGDStrategy"
    params:
      warmup_epochs: 2
      warmup_bias_lr: 0.05
      warmup_momentum: 0.5
      lr: 0.0032
      lre: 0.000384
      momentum: 0.843
      weight_decay: 0.00036
      nesterov: True

nileena_thomas

@KlemenSkrlj IOU is the standard 0.45
Parsing seems to be working fine overall, as many detections are coming through correctly. My colleague handled that part, so I’m not fully across all the details on that side.

KlemenSkrlj

Nothing seems particularly out of the ordinary from the model config perspective. You are using the light predefined model and we've used that one before in many different projects of our own without issues when converting to RVC2.
One thing you can do for debugging purposes is feed the same image in pytorch/ONNX and in your depthai script with the .blob model. You would expect the predictions to be very similar and the confidences to be similar as well.
This way you can test this out on a set of images and check if there is some pattern showing for images where detections are consistent between model formats and where there aren't.

nileena_thomas

Hi @KlemenSkrlj ,

Thank you for your response. Feeding the same image to both the .blob model and the DepthAI script is a good idea — I’ll keep this approach in mind.

I realized that after augmenting one particular class heavily, the other class became underrepresented. Addressing the class imbalance and then testing the retrained model led to better performance.

I had started working with the annotate feature but got sidetracked by these performance issues. I’ll be getting back to it soon. Thanks for sharing the links to Annotate — quick question: does the annotate feature also generate labels for each image file? Are there any tutorials or guides you’d recommend to help me get started with it?

KlemenSkrlj

It does generate the anotations for each image and the output is a new LDF instance. At the moment we don't have yet a tutorial for it outside of the documentation inside the repository. But we will be doing a tutorial for it as well soon and adding it in the ai-tutorials repository.

nileena_thomas

Hi @KlemenSkrlj ,

Sounds good. Are there any tutorials that point towards working with LDF instances ?

KlemenSkrlj

Yes, feel free to explore a bit through them. They are mainly focused on our stack (LDF, LuxonisTrain, Datadreamer, etc.) + some integrations with other libraries like Ultralytics for example.

nileena_thomas

Thanks @KlemenSkrlj . You have been very helpful

nileena_thomas

@KlemenSkrlj ,

Quick question — I’ve tried segmentation models before, but they were a bit heavy and had to run separately outside the camera for processing. Since I see segmentation in the Luxonis Train framework, I wanted to check if it’s possible to train and run a segmentation + regression model directly on the OAK camera that’s fast enough for live, real-time results.

KlemenSkrlj

We didn't really experiment with such model yet. But you should be able to define such model and train it through LuxonisTrain. You would need to spec out the architecture manually from nodes. This means you need to specify specific nodes (like we have for complex model config), you can check out which ones are inside the segmentation predefined model here. And then you add a new classification/regression head on the backbone (in addition to existing segmentation one).
For benchmarking the architecture in advance you can train for 1 epoch, use ArchiveOnTrainEnd callback, use the ONNX Archive to convert to RVC2 Archive and then use Modelconverter benchmark module to benchmark it. For reference here are the benchmarks of the segmentation predefined models and I don't expect adding a small classification head should lower the FPS by a lot.

« Previous Page