Multiple oakd pro w enumaration issue

Behnaz

I have a Nvidia PC with debian12 running 4 oakd pro w cameras. The system used to be working fine for 9 weeks until it started crashing on systemd. System can only work on one condition : running manual run script after re plugged power on oakds any warm reboot shows the camera unavailable. all four camera’s sitting on one bus and on failure on lsusb -t I get them 480M and on working mode it’s 5000M.

To investigate the problem these are the steps I did so far:

What I’ve tested

Hardware power/USB permutations, teamed up with our electronic engineer.

Each camera is powered on it’s own power injector. The cases we tested:

Used a startech powered hub, exchanged cables to shorter one, test with direct camera connection to pc.

In all cases, a warm reboot causes the system to hang waiting for frames. Only unplug/re-plug camera power restores operation.

Cross-machine check

Full system installed on a new PC(Nvidia to Nuc) → same behaviour.

New PC tested on machine → same behaviour.

Software changes made

lowered my resolution and fps, currently on 720 and 30fps.

Udev: removed conflicting rules, consolidated into one, test for simplest udev rules for both booted and unbooted mode, set autosuspend off on reboot.

Enumeration: added extra wait time to reduce race conditions. an script for soft re-enumeration.

systemd: added pre-start script to allow enumeration to settle. Restricted systemd to be launched after udev rules are checked.

DepthAI init: switched to explicit MXID targeting; added bounded retry.

Pipeline: improved graceful termination.

Errors about device discovery are gone (thanks to MXID+retry), but it’s only a mask!

Runtime still stalls on warm reboot: “waiting for new frame.”

I also flashed the bootloader on the cameras. For few try it showed all cameras available and keep consistent on reboot.

As soon as I activated systemd I am back to square one.

It’s like 13 days I am working on it. Any advice is appreciated.

jakaskerl

Behnaz

Behnaz I also flashed the bootloader on the cameras. For few try it showed all cameras available and keep consistent on reboot.

You shouldn't flash bootloader to USB devices. Not sure how you managed that since they have so little flash memory.

Can you show the important parts of dmesg (for systemd service, and both 2485 and f3b6 or whatever the numbers are)?
This only happens if you restart the host with devices plugged in? Same thing on power cut as well as (software)sudo reboot?

Thanks,
Jaka

Behnaz

Hi Jaka, I used this python script to flash bootloader. https://docs.luxonis.com/software/depthai/examples/flash_bootloader/

You’re right, the behaviour happens on both `sudo reboot` and after a cold power cut if the devices are plugged in and also simply stop and restart the manual run on the same boot. It originally started after the cameras stopped being detected a few hours into runtime, and now it’s reproducible every restart.

Here are the trimmed `dmesg` logs after a reboot with both OAKs plugged in (originally they are four, I unplugged two for testing purposes ) . As for today I am working only with two cameras connected. So in working condition:

Sep 05 11:09:29 kernel: usb 3-1.1.1: New USB device found, idVendor=03e7, idProduct=f63c  Product: Luxonis Bootloader  Serial: 18443010E13C3BF500

Sep 05 11:09:29 kernel: usb 3-1.3.1: New USB device found, idVendor=03e7, idProduct=f63c  Product: Luxonis Bootloader  Serial: 14442C1061B4D8D600

and in run time

Sep 05 11:10:22 kernel: usb 4-1.1.1: New USB device found, idVendor=03e7, idProduct=f63b  Product: Luxonis Device  Serial: 18443010E13C3BF500

Sep 05 11:10:22 kernel: usb 4-1.3.1: New USB device found, idVendor=03e7, idProduct=f63b  Product: Luxonis Device  Serial: 14442C1061B4D8D600

lsusb -t
/:  Bus 001.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/1p, 480M

/:  Bus 002.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/4p, 10000M

/:  Bus 003.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/12p, 480M

    |__ Port 001: Dev 002, If 0, Class=Hub, Driver=hub/4p, 480M

        |__ Port 001: Dev 005, If 0, Class=Hub, Driver=hub/1p, 480M

        |__ Port 003: Dev 007, If 0, Class=Hub, Driver=hub/1p, 480M

    |__ Port 002: Dev 003, If 0, Class=Hub, Driver=hub/2p, 480M

        |__ Port 002: Dev 006, If 0, Class=Communications, Driver=cdc_acm, 12M

        |__ Port 002: Dev 006, If 1, Class=CDC Data, Driver=cdc_acm, 12M

    |__ Port 010: Dev 004, If 0, Class=Wireless, Driver=btusb, 12M

    |__ Port 010: Dev 004, If 1, Class=Wireless, Driver=btusb, 12M

/:  Bus 004.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/4p, 10000M

    |__ Port 001: Dev 002, If 0, Class=Hub, Driver=hub/4p, 10000M

        |__ Port 001: Dev 004, If 0, Class=Hub, Driver=hub/1p, 5000M

            |__ Port 001: Dev 006, If 0, Class=Vendor Specific Class, Driver=usbfs, 5000M

        |__ Port 003: Dev 005, If 0, Class=Hub, Driver=hub/1p, 5000M

            |__ Port 001: Dev 007, If 0, Class=Vendor Specific Class, Driver=usbfs, 5000M

    |__ Port 004: Dev 003, If 0, Class=Hub, Driver=hub/2p, 10000M

/:  Bus 005.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/2p, 480M

/:  Bus 006.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/4p, 10000M

dmesg around a manual stop/start (same boot)

After I stop and start the app manually, devices bounce between bootloader and runtime and end up stuck in bootloader on USB2:

12:15:47 kernel: usb 3-1.1.1: New USB device found, idVendor=03e7, idProduct=f63c  Product: Luxonis Bootloader

12:15:47 kernel: usb 3-1.3.1: New USB device found, idVendor=03e7, idProduct=f63c  Product: Luxonis Bootloader

12:15:58 kernel: usb 4-1.1.1: New USB device found, idVendor=03e7, idProduct=f63b  Product: Luxonis Device

12:15:58 kernel: usb 4-1.3.1: New USB device found, idVendor=03e7, idProduct=f63b  Product: Luxonis Device

12:16:13 kernel: usb 3-1.1.1: New USB device found, idVendor=03e7, idProduct=f63c  Product: Luxonis Bootloader

12:16:31 kernel: usb 3-1.3.1: New USB device found, idVendor=03e7, idProduct=f63c  Product: Luxonis Bootloader

/:  Bus 003.Port 001: Dev 002, Class=Hub, 480M

    |__ Port 001: Dev 014, If 0, Class=Vendor Specific Class, Driver=[none], 480M

/:  Bus 003.Port 001: Dev 002, Class=Hub, 480M

    |__ Port 003: Dev 007, If 0, Class=Hub, 480M

        |__ Port 001: Dev 015, If 0, Class=Vendor Specific Class, Driver=[none], 480M

jakaskerl

Behnaz
The non-working devices don't print the serial (dmesg output) or was it just cut off from the logs?

Thanks,
Jaka