Help Needed: Deploying MobileNetSSD-based Object Detection Model on OAK-D-CM4

KlemenSkrlj

Which model did you end up training in LuxonisTrain? Could you share the training config (.yaml file)? We can try it out on our "sample" dataset and compare detections between pytorch and .blob.
But generally the models should be convertable to a good degree and you should see the detections with blob most of the time, especially if you lower the confidence threshold. Maybe try to increase the IoU threshold as well? Olso on this: are you using ParsingNeuralNetwork node for NN inference and parsing? Just want to make sure its not in issue with incorrect parsing.

nileena_thomas

@KlemenSkrlj This is the yaml file I used -

model:
  name: can_detection_01aug2025_augmented
  predefined_model:
    name: DetectionModel
    params:
      variant: light
      loss_params:
        iou_type: "siou"
        n_warmup_epochs: 0 # No assigner warmup
        iou_loss_weight: 20 # Should be 2.5 * accumulate_grad_batches for best results
        class_loss_weight: 8 # Should be 1 * accumulate_grad_batches for best results

loader:
  params:
    dataset_name: can_defect_dataset_v9

trainer:
  preprocessing:
    train_image_size: [416, 416]
    keep_aspect_ratio: true
    normalize:
      active: true
      params:
        mean: [0., 0., 0.]
        std: [1, 1, 1]
    augmentations:
      - name: Affine
        params:
          scale: [0.7, 1.7]
          rotate: 20
          shear: 5
          p: 0.3
      - name: HorizontalFlip
        params:
          p: 0.3
      - name: ColorJitter
        params:
          brightness: [0.8, 1.2]
          contrast: [0.8, 1.2]
          saturation: [0.8, 1.2]
          hue: 0
          p: 0.2

  batch_size: 8
  epochs: &epochs 400
  accumulate_grad_batches: 8 # For best results, always accumulate gradients to effectively use 64 batch size
  n_workers: 8
  validation_interval: 10
  n_log_images: 8

  callbacks:
    - name: EMACallback
      params:
        decay: 0.9999
        use_dynamic_decay: True
        decay_tau: 2000

  training_strategy: # Fine tuning params
    name: "TripleLRSGDStrategy"
    params:
      warmup_epochs: 2
      warmup_bias_lr: 0.05
      warmup_momentum: 0.5
      lr: 0.0032
      lre: 0.000384
      momentum: 0.843
      weight_decay: 0.00036
      nesterov: True

nileena_thomas

@KlemenSkrlj IOU is the standard 0.45
Parsing seems to be working fine overall, as many detections are coming through correctly. My colleague handled that part, so I’m not fully across all the details on that side.

KlemenSkrlj

Nothing seems particularly out of the ordinary from the model config perspective. You are using the light predefined model and we've used that one before in many different projects of our own without issues when converting to RVC2.
One thing you can do for debugging purposes is feed the same image in pytorch/ONNX and in your depthai script with the .blob model. You would expect the predictions to be very similar and the confidences to be similar as well.
This way you can test this out on a set of images and check if there is some pattern showing for images where detections are consistent between model formats and where there aren't.

nileena_thomas

Hi @KlemenSkrlj ,

Thank you for your response. Feeding the same image to both the .blob model and the DepthAI script is a good idea — I’ll keep this approach in mind.

I realized that after augmenting one particular class heavily, the other class became underrepresented. Addressing the class imbalance and then testing the retrained model led to better performance.

I had started working with the annotate feature but got sidetracked by these performance issues. I’ll be getting back to it soon. Thanks for sharing the links to Annotate — quick question: does the annotate feature also generate labels for each image file? Are there any tutorials or guides you’d recommend to help me get started with it?

KlemenSkrlj

It does generate the anotations for each image and the output is a new LDF instance. At the moment we don't have yet a tutorial for it outside of the documentation inside the repository. But we will be doing a tutorial for it as well soon and adding it in the ai-tutorials repository.

nileena_thomas

Hi @KlemenSkrlj ,

Sounds good. Are there any tutorials that point towards working with LDF instances ?

KlemenSkrlj

Yes, feel free to explore a bit through them. They are mainly focused on our stack (LDF, LuxonisTrain, Datadreamer, etc.) + some integrations with other libraries like Ultralytics for example.

nileena_thomas

Thanks @KlemenSkrlj . You have been very helpful

nileena_thomas

@KlemenSkrlj ,

Quick question — I’ve tried segmentation models before, but they were a bit heavy and had to run separately outside the camera for processing. Since I see segmentation in the Luxonis Train framework, I wanted to check if it’s possible to train and run a segmentation + regression model directly on the OAK camera that’s fast enough for live, real-time results.

KlemenSkrlj

We didn't really experiment with such model yet. But you should be able to define such model and train it through LuxonisTrain. You would need to spec out the architecture manually from nodes. This means you need to specify specific nodes (like we have for complex model config), you can check out which ones are inside the segmentation predefined model here. And then you add a new classification/regression head on the backbone (in addition to existing segmentation one).
For benchmarking the architecture in advance you can train for 1 epoch, use ArchiveOnTrainEnd callback, use the ONNX Archive to convert to RVC2 Archive and then use Modelconverter benchmark module to benchmark it. For reference here are the benchmarks of the segmentation predefined models and I don't expect adding a small classification head should lower the FPS by a lot.

nileena_thomas

@KlemenSkrlj , I’m trying to run luxonis_train train using 4 GPUs on my VM, but I keep running into this error:[rank0]: RuntimeError: No backend type associated with device type cpu

Could you advise on the correct way to configure Luxonis Train for multi-GPU training? Any tips for avoiding this CPU backend issue would be greatly appreciated.

Thank you,
Nileena

KlemenSkrlj

Hi @nileena_thomas ,
Could you share the full error log, I'm mainly interested in which step of the code does this happen. Also would you be able to share a bit more information about your multi-gpu setup? Are all GPUs in a single node and you get this error trying to run in DDP mode? If that is the case and the error is triggered by the metric computation could you give it a go with this branch as it might be the fix for issue. But please let me know if that is or isn't the case.
Thanks,
Klemen

nileena_thomas

Hi Klemen,

Thanks for your message.

Here are the details you asked for:

Multi-GPU setup: 4 × NVIDIA T4 GPUs within a single VM (n1-highmem-64, 64 vCPUs, 416 GB RAM).
DDP setup: Running with strategy: "ddp", accelerator: "gpu", and devices: [0,1,2,3]
Error: The crash occurs right after the sanity check. The last log entries show validation loss and metrics (all zeros), followed by this error

I’ve attached the relevant config and log snippet below for reference. I am yet to look into the branch you suggested, I will give it a try and get back to you.

nileena_thomas

This is the yaml file @KlemenSkrlj

model:

  name: can_detection_14oct

  predefined_model:

    name: DetectionModel

    params:

      variant: light

      loss_params:

        iou_type: "siou"

        n_warmup_epochs: 0 # No assigner warmup

        iou_loss_weight: 20 # Should be 2.5 \* accumulate_grad_batches for best results

        class_loss_weight: 8 # Should be 1 \* accumulate_grad_batches for best results

loader:

  params:

    dataset_name: can_defect_dataset_v12

trainer:

  preprocessing:

    train_image_size: [416, 416]

    keep_aspect_ratio: true

    normalize:

      active: true

      params:

        mean: [0., 0., 0.]

        std: [1, 1, 1]

    augmentations:

      - name: Affine

        params:

          scale: [0.7, 1.7]

          rotate: 20

          shear: 5

          p: 0.3

      - name: HorizontalFlip

        params:

          p: 0.3

      - name: VerticalFlip

        params:

          p: 0.3

      - name: ColorJitter

        params:

          brightness: [0.8, 1.2]

          contrast: [0.8, 1.2]

          saturation: [0.8, 1.2]

          hue: 0

          p: 0.2

  batch_size: 8

  epochs: &epochs 350

  accumulate_grad_batches: 8 # For best results, always accumulate gradients to effectively use 64 batch size

  n_workers: 8

  validation_interval: 10

  n_log_images: 8

  accelerator: "gpu"

  devices: [0,1,2,3]

  strategy: "ddp"

  precision: "16-mixed"

  callbacks:

    - name: EMACallback

      params:

        decay: 0.9999

        use_dynamic_decay: True

        decay_tau: 2000

  training_strategy: # Fine tuning params

    name: "TripleLRSGDStrategy"

    params:

      warmup_epochs: 2

      warmup_bias_lr: 0.05

      warmup_momentum: 0.5

      lr: 0.0032

      lre: 0.000384

      momentum: 0.843

      weight_decay: 0.00036

      nesterov: True

nileena_thomas

@KlemenSkrlj This is the log file. I am unable to attach it directly here , hence pasting it -

──────────────────────────────────────────────────────────── Validation ─────────────────────────────────────────────────────────────
Loss: 28.90247917175293
Metrics:
                 EfficientBBoxHead                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Name                                   ┃ Value   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ MeanAveragePrecision                   │ 0.00000 │
│ map_50                                 │ 0.00000 │
│ map_75                                 │ 0.00000 │
│ map_small                              │ 0.00000 │
│ map_medium                             │ 0.00000 │
│ map_large                              │ 0.00000 │
│ mar_1                                  │ 0.00000 │
│ mar_10                                 │ 0.00000 │
│ mar_100                                │ 0.00000 │
│ mar_small                              │ 0.00000 │
│ mar_medium                             │ 0.00000 │
│ mar_large                              │ 0.00000 │
│ f1_small                               │ nan     │
│ f1_medium                              │ nan     │
│ f1_large                               │ nan     │
│ map_per_class_blood                    │ 0.00000 │
│ map_per_class_bruise                   │ 0.00000 │
│ map_per_class_can                      │ 0.00000 │
│ map_per_class_dark_flakes              │ 0.00000 │
│ map_per_class_defective_appearance     │ 0.00000 │
│ map_per_class_empty_can                │ 0.00000 │
│ map_per_class_incomplete               │ 0.00000 │
│ map_per_class_pin_bone                 │ 0.00000 │
│ map_per_class_scorching                │ 0.00000 │
│ map_per_class_skin_and_scales          │ 0.00000 │
│ mar_100_per_class_blood                │ 0.00000 │
│ mar_100_per_class_bruise               │ 0.00000 │
│ mar_100_per_class_can                  │ 0.00000 │
│ mar_100_per_class_dark_flakes          │ 0.00000 │
│ mar_100_per_class_defective_appearance │ 0.00000 │
│ mar_100_per_class_empty_can            │ 0.00000 │
│ mar_100_per_class_incomplete           │ 0.00000 │
│ mar_100_per_class_pin_bone             │ 0.00000 │
│ mar_100_per_class_scorching            │ 0.00000 │
│ mar_100_per_class_skin_and_scales      │ 0.00000 │
│ mcc                                    │ 0.00000 │
└────────────────────────────────────────┴─────────┘
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 | print_results:186
2025-10-15 12:25:59 [INFO] Validation main metric (EfficientBBoxHead/MeanAveragePrecision): 0.0000 | _print_results:746
2025-10-15 12:25:59 [WARNING] Logged images (6) != expected (8). Possible reasons: class imbalance or a small number of images in the split. | _evaluation_epoch_end:720
2025-10-15 12:25:59 [ERROR] Encountered an exception during training. | _train:287
Traceback (most recent call last):

  File "/home/nileena/.conda/envs/luxonis-train/bin/luxonis_train", line 8, in <module>
    sys.exit(app.meta())
    │   │    │   └ <property object at 0x7693670d9f80>
    │   │    └ App(help='Luxonis Train CLI', alias=(), version=<function <lambda> at 0x769368326f80>, version_flags=('--version',), help_fla...
    │   └ <built-in function exit>
    └ <module 'sys' (built-in)>
  File "/home/nileena/.local/lib/python3.10/site-packages/cyclopts/core.py", line 1257, in __call__
    return command(*bound.args, **bound.kwargs)
           │        │     │       │     └ <property object at 0x769367309710>
           │        │     │       └ <BoundArguments (tokens=('train', '--config', 'can_detection_14oct.yaml'))>
           │        │     └ <property object at 0x7693673096c0>
           │        └ <BoundArguments (tokens=('train', '--config', 'can_detection_14oct.yaml'))>
           └ <function launcher at 0x769366e388b0>
  File "/home/nileena/.conda/envs/luxonis-train/lib/python3.10/site-packages/luxonis_train/__main__.py", line 363, in launcher
    app(tokens)
    │   └ ('train', '--config', 'can_detection_14oct.yaml')
    └ App(help='Luxonis Train CLI', alias=(), version=<function <lambda> at 0x769368326f80>, version_flags=('--version',), help_fla...
  File "/home/nileena/.local/lib/python3.10/site-packages/cyclopts/core.py", line 1257, in __call__
    return command(*bound.args, **bound.kwargs)
           │        │     │       │     └ <property object at 0x769367309710>
           │        │     │       └ <BoundArguments (config='can_detection_14oct.yaml')>
           │        │     └ <property object at 0x7693673096c0>
           │        └ <BoundArguments (config='can_detection_14oct.yaml')>
           └ <function train at 0x7693670c48b0>
  File "/home/nileena/.conda/envs/luxonis-train/lib/python3.10/site-packages/luxonis_train/__main__.py", line 62, in train
    create_model(config, opts, debug).train(weights=weights)
    │            │       │     │                    └ None
    │            │       │     └ False
    │            │       └ None
    │            └ 'can_detection_14oct.yaml'
    └ <function create_model at 0x7693670c4820>
  File "/home/nileena/.conda/envs/luxonis-train/lib/python3.10/site-packages/luxonis_train/core/core.py", line 354, in train
    self._train(
    │    └ <function LuxonisModel._train at 0x769128114c10>
    └ <luxonis_train.core.core.LuxonisModel object at 0x76936830abc0>
> File "/home/nileena/.conda/envs/luxonis-train/lib/python3.10/site-packages/luxonis_train/core/core.py", line 285, in _train
    self.pl_trainer.fit(*args, ckpt_path=resume, **kwargs)
    │    │          │    │               │         └ {}
    │    │          │    │               └ None
    │    │          │    └ (LuxonisLightningModule(
    │    │          │        (nodes): Nodes(
    │    │          │          (EfficientRep): EfficientRep(
    │    │          │            (repvgg_encoder): RepVGGBlock(
    │    │          │              (no...
    │    │          └ <function Trainer.fit at 0x7691374cdc60>
    │    └ <lightning.pytorch.trainer.trainer.Trainer object at 0x769128113a00>
    └ <luxonis_train.core.core.LuxonisModel object at 0x76936830abc0>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 561, in fit
    call._call_and_handle_interrupt(
    │    └ <function _call_and_handle_interrupt at 0x7691375b5c60>
    └ <module 'lightning.pytorch.trainer.call' from '/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/ca...
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 47, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
           │       │                        │            │             │          └ {}
           │       │                        │            │             └ <lightning.pytorch.trainer.trainer.Trainer object at 0x769128113a00>
           │       │                        │            └ (LuxonisLightningModule(
           │       │                        │                (nodes): Nodes(
           │       │                        │                  (EfficientRep): EfficientRep(
           │       │                        │                    (repvgg_encoder): RepVGGBlock(
           │       │                        │                      (no...
           │       │                        └ <bound method Trainer._fit_impl of <lightning.pytorch.trainer.trainer.Trainer object at 0x769128113a00>>
           │       └ <property object at 0x7691374d4360>
           └ <lightning.pytorch.trainer.trainer.Trainer object at 0x769128113a00>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/strategies/launchers/subprocess_script.py", line 105, in launch
    return function(*args, **kwargs)
           │         │       └ {}
           │         └ (LuxonisLightningModule(
           │             (nodes): Nodes(
           │               (EfficientRep): EfficientRep(
           │                 (repvgg_encoder): RepVGGBlock(
           │                   (no...
           └ <bound method Trainer._fit_impl of <lightning.pytorch.trainer.trainer.Trainer object at 0x769128113a00>>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 599, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
    │    │    │                └ None
    │    │    └ LuxonisLightningModule(
    │    │        (nodes): Nodes(
    │    │          (EfficientRep): EfficientRep(
    │    │            (repvgg_encoder): RepVGGBlock(
    │    │              (non...
    │    └ <function Trainer._run at 0x7691374ce4d0>
    └ <lightning.pytorch.trainer.trainer.Trainer object at 0x769128113a00>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1012, in _run
    results = self._run_stage()
              │    └ <function Trainer._run_stage at 0x7691374ce5f0>
              └ <lightning.pytorch.trainer.trainer.Trainer object at 0x769128113a00>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1054, in _run_stage
    self._run_sanity_check()
    │    └ <function Trainer._run_sanity_check at 0x7691374ce680>
    └ <lightning.pytorch.trainer.trainer.Trainer object at 0x769128113a00>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1083, in _run_sanity_check
    val_loop.run()
    │        └ <function _no_grad_context.<locals>._decorator at 0x769137453be0>
    └ <lightning.pytorch.loops.evaluation_loop._EvaluationLoop object at 0x769128111600>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 179, in _decorator
    return loop_run(self, *args, **kwargs)
           │        │      │       └ {}
           │        │      └ ()
           │        └ <lightning.pytorch.loops.evaluation_loop._EvaluationLoop object at 0x769128111600>
           └ <function _EvaluationLoop.run at 0x769137453b50>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 152, in run
    return self.on_run_end()
           │    └ <function _EvaluationLoop.on_run_end at 0x7691374640d0>
           └ <lightning.pytorch.loops.evaluation_loop._EvaluationLoop object at 0x769128111600>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 295, in on_run_end
    self._on_evaluation_epoch_end()
    │    └ <function _EvaluationLoop._on_evaluation_epoch_end at 0x7691374644c0>
    └ <lightning.pytorch.loops.evaluation_loop._EvaluationLoop object at 0x769128111600>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 377, in _on_evaluation_epoch_end
    trainer._logger_connector.on_epoch_end()
    │       │                 └ <function _LoggerConnector.on_epoch_end at 0x7691374531c0>
    │       └ <lightning.pytorch.trainer.connectors.logger_connector.logger_connector._LoggerConnector object at 0x769366e276d0>
    └ <lightning.pytorch.trainer.trainer.Trainer object at 0x769128113a00>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/logger_connector/logger_connector.py", line 197, in on_epoch_end
    metrics = self.metrics
              │    └ <property object at 0x76913744f150>
              └ <lightning.pytorch.trainer.connectors.logger_connector.logger_connector._LoggerConnector object at 0x769366e276d0>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/logger_connector/logger_connector.py", line 236, in metrics
    return self.trainer._results.metrics(on_step)
           │    │       │                └ False
           │    │       └ <property object at 0x7691374d5a80>
           │    └ <lightning.pytorch.trainer.trainer.Trainer object at 0x769128113a00>
           └ <lightning.pytorch.trainer.connectors.logger_connector.logger_connector._LoggerConnector object at 0x769366e276d0>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 476, in metrics
    value = self._get_cache(result_metric, on_step)
            │    │          │              └ False
            │    │          └ _ResultMetric('val/metric/EfficientBBoxHead/MeanAveragePrecision', value=0.0, cumulated_batch_size=1)
            │    └ <staticmethod(<function _ResultCollection._get_cache at 0x7691374524d0>)>
            └ {False, {'on_validation_epoch_end.val/loss/EfficientBBoxHead/AdaptiveDetectionLoss': _ResultMetric('val/loss/EfficientBBoxHea...
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 440, in _get_cache
    result_metric.compute()
    │             └ <function _ResultMetric.compute at 0x768fd8254f70>
    └ _ResultMetric('val/metric/EfficientBBoxHead/MeanAveragePrecision', value=0.0, cumulated_batch_size=1)
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 289, in wrapped_func
    self._computed = compute(*args, **kwargs)
    │    │           │        │       └ {}
    │    │           │        └ ()
    │    │           └ <bound method _ResultMetric.compute of _ResultMetric('val/metric/EfficientBBoxHead/MeanAveragePrecision', value=0.0, cumulate...
    │    └ None
    └ _ResultMetric('val/metric/EfficientBBoxHead/MeanAveragePrecision', value=0.0, cumulated_batch_size=1)
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 249, in compute
    value = self.meta.sync(self.value.clone())  # `clone` because `sync` is in-place
            │    │    │    │    │     └ <method 'clone' of 'torch._C.TensorBase' objects>
            │    │    │    │    └ tensor(0.)
            │    │    │    └ _ResultMetric('val/metric/EfficientBBoxHead/MeanAveragePrecision', value=0.0, cumulated_batch_size=1)
            │    │    └ <property object at 0x76913744ec50>
            │    └ _Metadata(fx='on_validation_epoch_end', name='val/metric/EfficientBBoxHead/MeanAveragePrecision', prog_bar=False, logger=True...
            └ _ResultMetric('val/metric/EfficientBBoxHead/MeanAveragePrecision', value=0.0, cumulated_batch_size=1)
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/strategies/ddp.py", line 342, in reduce
    return _sync_ddp_if_available(tensor, group, reduce_op=reduce_op)
           │                      │       │                └ 'mean'
           │                      │       └ None
           │                      └ tensor(0.)
           └ <function _sync_ddp_if_available at 0x769139b7f910>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/fabric/utilities/distributed.py", line 179, in _sync_ddp_if_available
    return _sync_ddp(result, group=group, reduce_op=reduce_op)
           │         │             │                └ 'mean'
           │         │             └ None
           │         └ tensor(0.)
           └ <function _sync_ddp at 0x769139b7f9a0>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/fabric/utilities/distributed.py", line 229, in _sync_ddp
    torch.distributed.all_reduce(result, op=op, group=group, async_op=False)
    │     │           │          │          │         └ <torch.distributed.distributed_c10d.ProcessGroup object at 0x768e02ce32b0>
    │     │           │          │          └ <RedOpType.AVG: 1>
    │     │           │          └ tensor(0.)
    │     │           └ <function all_reduce at 0x7692dc5583a0>
    │     └ <module 'torch.distributed' from '/home/nileena/.local/lib/python3.10/site-packages/torch/distributed/__init__.py'>
    └ <module 'torch' from '/home/nileena/.local/lib/python3.10/site-packages/torch/__init__.py'>
  File "/home/nileena/.local/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
    return func(*args, **kwargs)
           │     │       └ {'op': <RedOpType.AVG: 1>, 'group': <torch.distributed.distributed_c10d.ProcessGroup object at 0x768e02ce32b0>, 'async_op': F...
           │     └ (tensor(0.),)
           └ <function all_reduce at 0x7692dc558310>
  File "/home/nileena/.local/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2810, in all_reduce
    work = group.allreduce([tensor], opts)
           │     │          │        └ <torch.distributed.distributed_c10d.AllreduceOptions object at 0x768e028f90f0>
           │     │          └ tensor(0.)
           │     └ <instancemethod allreduce at 0x7692ed7f5ea0>
           └ <torch.distributed.distributed_c10d.ProcessGroup object at 0x768e02ce32b0>

RuntimeError: No backend type associated with device type cpu
2025-10-15 12:25:59 [ERROR] Encountered an exception during training. | _train:287
Traceback (most recent call last):

  File "/home/nileena/.conda/envs/luxonis-train/bin/luxonis_train", line 8, in <module>
    sys.exit(app.meta())
    │   │    │   └ <property object at 0x7fe20df5e020>
    │   │    └ App(help='Luxonis Train CLI', alias=(), version=<function <lambda> at 0x7fe20f14ef80>, version_flags=('--version',), help_fla...
    │   └ <built-in function exit>
    └ <module 'sys' (built-in)>
  File "/home/nileena/.local/lib/python3.10/site-packages/cyclopts/core.py", line 1257, in __call__
    return command(*bound.args, **bound.kwargs)
           │        │     │       │     └ <property object at 0x7fe20e18d800>
           │        │     │       └ <BoundArguments (tokens=('train', '--config', 'can_detection_14oct.yaml'))>
           │        │     └ <property object at 0x7fe20e18d7b0>
           │        └ <BoundArguments (tokens=('train', '--config', 'can_detection_14oct.yaml'))>
           └ <function launcher at 0x7fe20dc708b0>
  File "/home/nileena/.conda/envs/luxonis-train/lib/python3.10/site-packages/luxonis_train/__main__.py", line 363, in launcher
    app(tokens)
    │   └ ('train', '--config', 'can_detection_14oct.yaml')
    └ App(help='Luxonis Train CLI', alias=(), version=<function <lambda> at 0x7fe20f14ef80>, version_flags=('--version',), help_fla...
  File "/home/nileena/.local/lib/python3.10/site-packages/cyclopts/core.py", line 1257, in __call__
    return command(*bound.args, **bound.kwargs)
           │        │     │       │     └ <property object at 0x7fe20e18d800>
           │        │     │       └ <BoundArguments (config='can_detection_14oct.yaml')>
           │        │     └ <property object at 0x7fe20e18d7b0>
           │        └ <BoundArguments (config='can_detection_14oct.yaml')>
           └ <function train at 0x7fe20df488b0>
  File "/home/nileena/.conda/envs/luxonis-train/lib/python3.10/site-packages/luxonis_train/__main__.py", line 62, in train
    create_model(config, opts, debug).train(weights=weights)
    │            │       │     │                    └ None
    │            │       │     └ False
    │            │       └ None
    │            └ 'can_detection_14oct.yaml'
    └ <function create_model at 0x7fe20df48820>
  File "/home/nileena/.conda/envs/luxonis-train/lib/python3.10/site-packages/luxonis_train/core/core.py", line 354, in train
    self._train(
    │    └ <function LuxonisModel._train at 0x7fdfcf54cc10>
    └ <luxonis_train.core.core.LuxonisModel object at 0x7fe20f132bf0>
> File "/home/nileena/.conda/envs/luxonis-train/lib/python3.10/site-packages/luxonis_train/core/core.py", line 285, in _train
    self.pl_trainer.fit(*args, ckpt_path=resume, **kwargs)
    │    │          │    │               │         └ {}
    │    │          │    │               └ None
    │    │          │    └ (LuxonisLightningModule(
    │    │          │        (nodes): Nodes(
    │    │          │          (EfficientRep): EfficientRep(
    │    │          │            (repvgg_encoder): RepVGGBlock(
    │    │          │              (no...
    │    │          └ <function Trainer.fit at 0x7fdfde329c60>
    │    └ <lightning.pytorch.trainer.trainer.Trainer object at 0x7fdfcf54be50>
    └ <luxonis_train.core.core.LuxonisModel object at 0x7fe20f132bf0>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 561, in fit
    call._call_and_handle_interrupt(
    │    └ <function _call_and_handle_interrupt at 0x7fdfde411c60>
    └ <module 'lightning.pytorch.trainer.call' from '/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/ca...
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 47, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
           │       │                        │            │             │          └ {}
           │       │                        │            │             └ <lightning.pytorch.trainer.trainer.Trainer object at 0x7fdfcf54be50>
           │       │                        │            └ (LuxonisLightningModule(
           │       │                        │                (nodes): Nodes(
           │       │                        │                  (EfficientRep): EfficientRep(
           │       │                        │                    (repvgg_encoder): RepVGGBlock(
           │       │                        │                      (no...
           │       │                        └ <bound method Trainer._fit_impl of <lightning.pytorch.trainer.trainer.Trainer object at 0x7fdfcf54be50>>
           │       └ <property object at 0x7fdfde3304f0>
           └ <lightning.pytorch.trainer.trainer.Trainer object at 0x7fdfcf54be50>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/strategies/launchers/subprocess_script.py", line 105, in launch
    return function(*args, **kwargs)
           │         │       └ {}
           │         └ (LuxonisLightningModule(
           │             (nodes): Nodes(
           │               (EfficientRep): EfficientRep(
           │                 (repvgg_encoder): RepVGGBlock(
           │                   (no...
           └ <bound method Trainer._fit_impl of <lightning.pytorch.trainer.trainer.Trainer object at 0x7fdfcf54be50>>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 599, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
    │    │    │                └ None
    │    │    └ LuxonisLightningModule(
    │    │        (nodes): Nodes(
    │    │          (EfficientRep): EfficientRep(
    │    │            (repvgg_encoder): RepVGGBlock(
    │    │              (non...
    │    └ <function Trainer._run at 0x7fdfde32a4d0>
    └ <lightning.pytorch.trainer.trainer.Trainer object at 0x7fdfcf54be50>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1012, in _run
    results = self._run_stage()
              │    └ <function Trainer._run_stage at 0x7fdfde32a5f0>
              └ <lightning.pytorch.trainer.trainer.Trainer object at 0x7fdfcf54be50>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1054, in _run_stage
    self._run_sanity_check()
    │    └ <function Trainer._run_sanity_check at 0x7fdfde32a680>
    └ <lightning.pytorch.trainer.trainer.Trainer object at 0x7fdfcf54be50>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1083, in _run_sanity_check
    val_loop.run()
    │        └ <function _no_grad_context.<locals>._decorator at 0x7fdfde2afbe0>
    └ <lightning.pytorch.loops.evaluation_loop._EvaluationLoop object at 0x7fdfcf5496c0>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 179, in _decorator
    return loop_run(self, *args, **kwargs)
           │        │      │       └ {}
           │        │      └ ()
           │        └ <lightning.pytorch.loops.evaluation_loop._EvaluationLoop object at 0x7fdfcf5496c0>
           └ <function _EvaluationLoop.run at 0x7fdfde2afb50>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 152, in run
    return self.on_run_end()
           │    └ <function _EvaluationLoop.on_run_end at 0x7fdfde2c00d0>
           └ <lightning.pytorch.loops.evaluation_loop._EvaluationLoop object at 0x7fdfcf5496c0>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 295, in on_run_end
    self._on_evaluation_epoch_end()
    │    └ <function _EvaluationLoop._on_evaluation_epoch_end at 0x7fdfde2c04c0>
    └ <lightning.pytorch.loops.evaluation_loop._EvaluationLoop object at 0x7fdfcf5496c0>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 377, in _on_evaluation_epoch_end
    trainer._logger_connector.on_epoch_end()
    │       │                 └ <function _LoggerConnector.on_epoch_end at 0x7fdfde2af1c0>
    │       └ <lightning.pytorch.trainer.connectors.logger_connector.logger_connector._LoggerConnector object at 0x7fe20dc5f6d0>
    └ <lightning.pytorch.trainer.trainer.Trainer object at 0x7fdfcf54be50>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/logger_connector/logger_connector.py", line 197, in on_epoch_end
    metrics = self.metrics
              │    └ <property object at 0x7fdfde2a7740>
              └ <lightning.pytorch.trainer.connectors.logger_connector.logger_connector._LoggerConnector object at 0x7fe20dc5f6d0>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/logger_connector/logger_connector.py", line 236, in metrics
    return self.trainer._results.metrics(on_step)
           │    │       │                └ False
           │    │       └ <property object at 0x7fdfde331c10>
           │    └ <lightning.pytorch.trainer.trainer.Trainer object at 0x7fdfcf54be50>
           └ <lightning.pytorch.trainer.connectors.logger_connector.logger_connector._LoggerConnector object at 0x7fe20dc5f6d0>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 476, in metrics
    value = self._get_cache(result_metric, on_step)
            │    │          │              └ False
            │    │          └ _ResultMetric('val/metric/EfficientBBoxHead/MeanAveragePrecision', value=0.0, cumulated_batch_size=1)
            │    └ <staticmethod(<function _ResultCollection._get_cache at 0x7fdfde2ae4d0>)>
            └ {False, {'on_validation_epoch_end.val/loss/EfficientBBoxHead/AdaptiveDetectionLoss': _ResultMetric('val/loss/EfficientBBoxHea...
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 440, in _get_cache
    result_metric.compute()
    │             └ <function _ResultMetric.compute at 0x7fde19f3cf70>
    └ _ResultMetric('val/metric/EfficientBBoxHead/MeanAveragePrecision', value=0.0, cumulated_batch_size=1)
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 289, in wrapped_func
    self._computed = compute(*args, **kwargs)
    │    │           │        │       └ {}
    │    │           │        └ ()
    │    │           └ <bound method _ResultMetric.compute of _ResultMetric('val/metric/EfficientBBoxHead/MeanAveragePrecision', value=0.0, cumulate...
    │    └ None
    └ _ResultMetric('val/metric/EfficientBBoxHead/MeanAveragePrecision', value=0.0, cumulated_batch_size=1)
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/logger_connector/result.py", line 249, in compute
    value = self.meta.sync(self.value.clone())  # `clone` because `sync` is in-place
            │    │    │    │    │     └ <method 'clone' of 'torch._C.TensorBase' objects>
            │    │    │    │    └ tensor(0.)
            │    │    │    └ _ResultMetric('val/metric/EfficientBBoxHead/MeanAveragePrecision', value=0.0, cumulated_batch_size=1)
            │    │    └ <property object at 0x7fdfde2a6ca0>
            │    └ _Metadata(fx='on_validation_epoch_end', name='val/metric/EfficientBBoxHead/MeanAveragePrecision', prog_bar=False, logger=True...
            └ _ResultMetric('val/metric/EfficientBBoxHead/MeanAveragePrecision', value=0.0, cumulated_batch_size=1)
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/pytorch/strategies/ddp.py", line 342, in reduce
    return _sync_ddp_if_available(tensor, group, reduce_op=reduce_op)
           │                      │       │                └ 'mean'
           │                      │       └ None
           │                      └ tensor(0.)
           └ <function _sync_ddp_if_available at 0x7fdfe0a73910>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/fabric/utilities/distributed.py", line 179, in _sync_ddp_if_available
    return _sync_ddp(result, group=group, reduce_op=reduce_op)
           │         │             │                └ 'mean'
           │         │             └ None
           │         └ tensor(0.)
           └ <function _sync_ddp at 0x7fdfe0a739a0>
  File "/home/nileena/.local/lib/python3.10/site-packages/lightning/fabric/utilities/distributed.py", line 229, in _sync_ddp
    torch.distributed.all_reduce(result, op=op, group=group, async_op=False)
    │     │           │          │          │         └ <torch.distributed.distributed_c10d.ProcessGroup object at 0x7fde0b15a1b0>
    │     │           │          │          └ <RedOpType.AVG: 1>
    │     │           │          └ tensor(0.)
    │     │           └ <function all_reduce at 0x7fe18334c3a0>
    │     └ <module 'torch.distributed' from '/home/nileena/.local/lib/python3.10/site-packages/torch/distributed/__init__.py'>
    └ <module 'torch' from '/home/nileena/.local/lib/python3.10/site-packages/torch/__init__.py'>
  File "/home/nileena/.local/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
    return func(*args, **kwargs)
           │     │       └ {'op': <RedOpType.AVG: 1>, 'group': <torch.distributed.distributed_c10d.ProcessGroup object at 0x7fde0b15a1b0>, 'async_op': F...
           │     └ (tensor(0.),)
           └ <function all_reduce at 0x7fe18334c310>
  File "/home/nileena/.local/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2810, in all_reduce
    work = group.allreduce([tensor], opts)
           │     │          │        └ <torch.distributed.distributed_c10d.AllreduceOptions object at 0x7fddd8745330>
           │     │          └ tensor(0.)
           │     └ <instancemethod allreduce at 0x7fe1945e9e70>
           └ <torch.distributed.distributed_c10d.ProcessGroup object at 0x7fde0b15a1b0>

RuntimeError: No backend type associated with device type cpu

KlemenSkrlj

Thanks for the info. So it looks like it is something we have seen before and hopefully should be solved in the branch I linked to. If you can confirm this that would be great so we can mainline the fix for the next release.

nileena_thomas

@KlemenSkrlj I am still running into the same error. This is what i did -
I cloned the git branch you specifed, installed luxonis-train from that git folder and tried to run from it. Looks like it is still failing with similar error. is there anything I am missing or should do differently?

KlemenSkrlj

I see. Yeah the fix in the branch was not thoroughly tested yet. We'll do some internal tests and come up with a correct fix. We should have some more comments on this in the comming days.

nileena_thomas

@KlemenSkrlj okay thank you

« Previous Page Next Page »