“Device crashed but did not reboot” → stuck in X_LINK_BOOTED

joshbarclay

Hi Luxonis team,

We’re testing an OAK-D Wide on a Raspberry Pi 5 with a USB3 power splitter connected to a 5V 2-3A power block. The camera starts and runs fine for a while with depthai.UsbSpeed.SUPER — but after some time (typically 30–90 minutes), it crashes and gets stuck in a X_LINK_BOOTED state.
{

"mxId": "14442C10D1ECD4D200",

"name": "1.3",

"isOn": false,

"platform": "X_LINK_MYRIAD_X",

"protocol": "X_LINK_USB_VSC",

"state": "X_LINK_BOOTED",

"status": "X_LINK_SUCCESS"

}

From our app logs:

[host] [error] Device likely crashed but did not reboot in time to get the crash dump

At this point, the camera is still listed on the USB bus Bus 001 Device 051: ID 03e7:2485 Intel Movidius MyriadX, but calling our self._init() method (which re-instantiates the pipeline with depthai.Device(...)) doesn’t bring it back — we have to physically reconnect the camera to recover.

Strangely, when I replicate this locally on my dev machine (no power splitter), I can turn the camera back on using the same self._init() method. So I’m confused why the exact same crash leaves the camera in a dead state on one setup but not the other.

I’ve read that not calling device.close() can leave the OAK-D in a bad state. Could that be the issue here? Should we be explicitly calling .close() before instantiating a new depthai.Device(...)?

Also I'd like to understand what's causing the camera to crash after running for 30+ minutes. When we were previously working out power supply issues, the camera would fall off the USB bus if it was undervolted, but that doesn't seem to be the issue here. If it was an issue with the pipeline, I would expect it to fail much sooner. Is the camera entering some sort of sleep mode?

Thanks for any guidance you can offer!
Josh

jakaskerl

joshbarclay
Could you share a MRE for this? It is possible for the device to be stuck in BOOTED state if the connection was abruptly ended (mostly with a KILL signal). Generally the with open() context manager should close the device gracefully but this doesn't seem to be the case for you.

Also what happens on dmesg when this issue happens?

Thanks,
Jaka

lovro

Hey Josh,

Thanks for the detailed info — that really helps.

Not sure what’s going on just yet, but I’ll check with the team and get back to you after we’ve looked into it more. That X_LINK_BOOTED thing and needing to unplug definitely sounds like something we should dig into.

Appreciate you waiting — we’ll follow up soon.

joshbarclay

I've done some more testing and observed that on my local machine the pipeline runs until my machine goes to sleep and then the Oak-D goes to sleep as well (or crashes). It seems like the camera we have deployed is exhibiting similar behavior, with the exception that it's getting stuck in the BOOTED state. It's running on a kiosk and when the kiosk is powered on, the Oak-D pipeline initializes and runs as long as the kiosk is running. Maybe this isn't best practice and it should be turned off when the kiosk is idle.

Below is my attempt at an MRE. Hopefully this isn't too simplified, the actual pipeline is using face-detection-retail-0004 to detect and crop user's faces. We have three output queues that can be polled by our application to get a user's distance, height, and eye visibility.

import depthai as dai
import cv2
import numpy as np
import time
import logging

# Set up logging
logging.basicConfig(level=logging.INFO)
log = logging.getLogger(__name__)

class OakDMinimal:
    def __init__(self):
        self.pipeline = None
        self.device = None
        self._init_pipeline()

    def _init_pipeline(self):
        """Initialize the Oak-D pipeline with minimal components."""
        try:
            self.pipeline = dai.Pipeline()
            
            # Create RGB Camera node
            camRgb = self.pipeline.create(dai.node.ColorCamera)
            camRgb.setFps(15)
            # Use CAM_A instead of deprecated RGB
            camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
            camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
            camRgb.setIspScale(3, 4)
            camRgb.setInterleaved(False)
            camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)
            
            # Create Stereo node for depth
            monoLeft = self.pipeline.create(dai.node.MonoCamera)
            monoRight = self.pipeline.create(dai.node.MonoCamera)
            stereo = self.pipeline.create(dai.node.StereoDepth)
            
            # Configure mono cameras
            for mono in [monoLeft, monoRight]:
                mono.setFps(15)
                mono.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
            monoLeft.setCamera("left")
            monoRight.setCamera("right")
            
            # Configure stereo
            stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.DEFAULT)
            stereo.setLeftRightCheck(True)
            stereo.setSubpixel(True)
            stereo.setDepthAlign(dai.CameraBoardSocket.CAM_A)
            
            # Link mono cameras to stereo
            monoLeft.out.link(stereo.left)
            monoRight.out.link(stereo.right)
            
            # Create output nodes
            xoutRgb = self.pipeline.create(dai.node.XLinkOut)
            xoutDepth = self.pipeline.create(dai.node.XLinkOut)
            xoutRgb.setStreamName("rgb")
            xoutDepth.setStreamName("depth")
            
            # Link outputs
            camRgb.isp.link(xoutRgb.input)
            stereo.depth.link(xoutDepth.input)
            
            # Initialize device with USB SUPER speed
            log.info("Initializing device with USB 3.0 (SUPER) speed...")
            self.device = dai.Device(self.pipeline, maxUsbSpeed=dai.UsbSpeed.SUPER)
            
            # Log USB speed after initialization
            usb_speed = self.device.getUsbSpeed()
            log.info(f"Device initialized with USB speed: {usb_speed.name}")
            
            # Get output queues
            self.qRgb = self.device.getOutputQueue(name="rgb", maxSize=1, blocking=False)
            self.qDepth = self.device.getOutputQueue(name="depth", maxSize=1, blocking=False)
            
            log.info("Pipeline initialized successfully")
            
        except Exception as e:
            log.error(f"Failed to initialize pipeline: {str(e)}")
            raise

    def run(self):
        """Run the pipeline continuously and monitor for issues."""
        try:
            log.info("Starting pipeline...")
            start_time = time.time()
            frame_count = 0
            
            while True:
                if self.device.isClosed():
                    log.error("Device connection lost!")
                    break
                    
                # Get frames
                rgbFrame = self.qRgb.get()
                depthFrame = self.qDepth.get()
                
                if rgbFrame is None or depthFrame is None:
                    log.warning("Received empty frame")
                    continue
                
                # Process frames (minimal processing to simulate load)
                rgb = rgbFrame.getCvFrame()
                depth = depthFrame.getFrame()
                
                # Log device stats every 30 seconds
                frame_count += 1
                if frame_count % 450 == 0:  # At 15 FPS, this is roughly every 30 seconds
                    self._log_device_stats()
                
                # Optional: display frames
                cv2.imshow("RGB", rgb)
                cv2.imshow("Depth", depth)
                
                if cv2.waitKey(1) == ord('q'):
                    break
                    
        except Exception as e:
            log.error(f"Error during pipeline execution: {str(e)}")
        finally:
            self.cleanup()

    def _log_device_stats(self):
        """Log device statistics."""
        try:
            temp = self.device.getChipTemperature().average
            css_cpu = self.device.getLeonCssCpuUsage().average
            mss_cpu = self.device.getLeonMssCpuUsage().average
            
            log.info(f"Device Stats - Temp: {temp:.1f}°C, CSS CPU: {css_cpu:.1f}%, MSS CPU: {mss_cpu:.1f}%, USB: {usb_speed.name}")
            
        except Exception as e:
            log.error(f"Failed to get device stats: {str(e)}")

    def cleanup(self):
        """Clean up resources."""
        if self.device is not None and not self.device.isClosed():
            self.device.close()
        cv2.destroyAllWindows()

if __name__ == "__main__":
    try:
        oak = OakDMinimal()
        oak.run()
    except Exception as e:
        log.error(f"Application error: {str(e)}")

Here is the relevant output from dmesg:

[Thu May  1 19:09:01 2025] usb 1-3: USB disconnect, device number 49
[Thu May  1 19:09:02 2025] usb 2-3: new SuperSpeed USB device number 35 using xhci_hcd
[Thu May  1 19:09:02 2025] usb 2-3: New USB device found, idVendor=03e7, idProduct=f63b, bcdDevice= 1.00
[Thu May  1 19:09:02 2025] usb 2-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[Thu May  1 19:09:02 2025] usb 2-3: Product: Luxonis Device
[Thu May  1 19:09:02 2025] usb 2-3: Manufacturer: Intel Corporation
[Thu May  1 19:09:02 2025] usb 2-3: SerialNumber: 14442C10D1ECD4D200
[Thu May  1 19:09:02 2025] usb 2-3: USB disconnect, device number 35
[Thu May  1 19:09:02 2025] usb 1-3: new high-speed USB device number 50 using xhci_hcd
[Thu May  1 19:09:02 2025] usb 1-3: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
[Thu May  1 19:09:02 2025] usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[Thu May  1 19:09:02 2025] usb 1-3: Product: Movidius MyriadX
[Thu May  1 19:09:02 2025] usb 1-3: Manufacturer: Movidius Ltd.
[Thu May  1 19:09:02 2025] usb 1-3: SerialNumber: 03e72485
[Thu May  1 19:09:05 2025] usb 1-3: USB disconnect, device number 50
[Thu May  1 19:09:06 2025] usb 2-3: new SuperSpeed USB device number 36 using xhci_hcd
[Thu May  1 19:09:06 2025] usb 2-3: New USB device found, idVendor=03e7, idProduct=f63b, bcdDevice= 1.00
[Thu May  1 19:09:06 2025] usb 2-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[Thu May  1 19:09:06 2025] usb 2-3: Product: Luxonis Device
[Thu May  1 19:09:06 2025] usb 2-3: Manufacturer: Intel Corporation
[Thu May  1 19:09:06 2025] usb 2-3: SerialNumber: 14442C10D1ECD4D200
[Thu May  1 19:09:07 2025] usb 2-3: USB disconnect, device number 36
[Thu May  1 19:09:07 2025] usb 1-3: new high-speed USB device number 51 using xhci_hcd
[Thu May  1 19:09:07 2025] usb 1-3: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
[Thu May  1 19:09:07 2025] usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[Thu May  1 19:09:07 2025] usb 1-3: Product: Movidius MyriadX
[Thu May  1 19:09:07 2025] usb 1-3: Manufacturer: Movidius Ltd.
[Thu May  1 19:09:07 2025] usb 1-3: SerialNumber: 03e72485
[Thu May  1 21:24:41 2025] systemd-journald[296]: Data hash table of /var/log/journal/6bd4c45ff3444a0ca280f96ef5c0d07a/system.journal has a fill level at 75.0 (174763 of 233016 items, 109051904 file size, 623 bytes per hash table item), suggesting rotation.
[Thu May  1 21:24:41 2025] systemd-journald[296]: /var/log/journal/6bd4c45ff3444a0ca280f96ef5c0d07a/system.journal: Journal header limits reached or header out-of-date, rotating.
[Thu May  1 17:47:49 2025] usb 1-3: Manufacturer: Movidius Ltd.
[Thu May  1 17:47:49 2025] usb 1-3: SerialNumber: 03e72485
[Thu May  1 18:47:19 2025] systemd-journald[296]: Data hash table of /var/log/journal/6bd4c45ff3444a0ca280f96ef5c0d07a/system.journal has a fill level at 75.0 (174765 of 233016 items, 109051904 file size, 623 bytes per hash table item), suggesting rotation.
[Thu May  1 18:47:19 2025] systemd-journald[296]: /var/log/journal/6bd4c45ff3444a0ca280f96ef5c0d07a/system.journal: Journal header limits reached or header out-of-date, rotating.

jakaskerl

joshbarclay
Maybe try checking if the process keeps running in the background. Something like
ps aux | grep python or something similar.

Thanks
Jaka