• DepthAI-v2
  • “Device crashed but did not reboot” → stuck in X_LINK_BOOTED

Hi Luxonis team,

We’re testing an OAK-D Wide on a Raspberry Pi 5 with a USB3 power splitter connected to a 5V 2-3A power block. The camera starts and runs fine for a while with depthai.UsbSpeed.SUPER — but after some time (typically 30–90 minutes), it crashes and gets stuck in a X_LINK_BOOTED state.
{

    "mxId": "14442C10D1ECD4D200",

    "name": "1.3",

    "isOn": false,

    "platform": "X_LINK_MYRIAD_X",

    "protocol": "X_LINK_USB_VSC",

    "state": "X_LINK_BOOTED",

    "status": "X_LINK_SUCCESS"

}

From our app logs:

[host] [error] Device likely crashed but did not reboot in time to get the crash dump

At this point, the camera is still listed on the USB bus Bus 001 Device 051: ID 03e7:2485 Intel Movidius MyriadX, but calling our self._init() method (which re-instantiates the pipeline with depthai.Device(...)) doesn’t bring it back — we have to physically reconnect the camera to recover.

Strangely, when I replicate this locally on my dev machine (no power splitter), I can turn the camera back on using the same self._init() method. So I’m confused why the exact same crash leaves the camera in a dead state on one setup but not the other.

I’ve read that not calling device.close() can leave the OAK-D in a bad state. Could that be the issue here? Should we be explicitly calling .close() before instantiating a new depthai.Device(...)?

Also I'd like to understand what's causing the camera to crash after running for 30+ minutes. When we were previously working out power supply issues, the camera would fall off the USB bus if it was undervolted, but that doesn't seem to be the issue here. If it was an issue with the pipeline, I would expect it to fail much sooner. Is the camera entering some sort of sleep mode?

Thanks for any guidance you can offer!
Josh

    Hey Josh,

    Thanks for the detailed info — that really helps.

    Not sure what’s going on just yet, but I’ll check with the team and get back to you after we’ve looked into it more. That X_LINK_BOOTED thing and needing to unplug definitely sounds like something we should dig into.

    Appreciate you waiting — we’ll follow up soon.

    joshbarclay
    Could you share a MRE for this? It is possible for the device to be stuck in BOOTED state if the connection was abruptly ended (mostly with a KILL signal). Generally the with open() context manager should close the device gracefully but this doesn't seem to be the case for you.

    Also what happens on dmesg when this issue happens?

    Thanks,
    Jaka

    I've done some more testing and observed that on my local machine the pipeline runs until my machine goes to sleep and then the Oak-D goes to sleep as well (or crashes). It seems like the camera we have deployed is exhibiting similar behavior, with the exception that it's getting stuck in the BOOTED state. It's running on a kiosk and when the kiosk is powered on, the Oak-D pipeline initializes and runs as long as the kiosk is running. Maybe this isn't best practice and it should be turned off when the kiosk is idle.

    Below is my attempt at an MRE. Hopefully this isn't too simplified, the actual pipeline is using face-detection-retail-0004 to detect and crop user's faces. We have three output queues that can be polled by our application to get a user's distance, height, and eye visibility.

    import depthai as dai
    import cv2
    import numpy as np
    import time
    import logging
    
    # Set up logging
    logging.basicConfig(level=logging.INFO)
    log = logging.getLogger(__name__)
    
    class OakDMinimal:
        def __init__(self):
            self.pipeline = None
            self.device = None
            self._init_pipeline()
    
        def _init_pipeline(self):
            """Initialize the Oak-D pipeline with minimal components."""
            try:
                self.pipeline = dai.Pipeline()
                
                # Create RGB Camera node
                camRgb = self.pipeline.create(dai.node.ColorCamera)
                camRgb.setFps(15)
                # Use CAM_A instead of deprecated RGB
                camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
                camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
                camRgb.setIspScale(3, 4)
                camRgb.setInterleaved(False)
                camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)
                
                # Create Stereo node for depth
                monoLeft = self.pipeline.create(dai.node.MonoCamera)
                monoRight = self.pipeline.create(dai.node.MonoCamera)
                stereo = self.pipeline.create(dai.node.StereoDepth)
                
                # Configure mono cameras
                for mono in [monoLeft, monoRight]:
                    mono.setFps(15)
                    mono.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
                monoLeft.setCamera("left")
                monoRight.setCamera("right")
                
                # Configure stereo
                stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.DEFAULT)
                stereo.setLeftRightCheck(True)
                stereo.setSubpixel(True)
                stereo.setDepthAlign(dai.CameraBoardSocket.CAM_A)
                
                # Link mono cameras to stereo
                monoLeft.out.link(stereo.left)
                monoRight.out.link(stereo.right)
                
                # Create output nodes
                xoutRgb = self.pipeline.create(dai.node.XLinkOut)
                xoutDepth = self.pipeline.create(dai.node.XLinkOut)
                xoutRgb.setStreamName("rgb")
                xoutDepth.setStreamName("depth")
                
                # Link outputs
                camRgb.isp.link(xoutRgb.input)
                stereo.depth.link(xoutDepth.input)
                
                # Initialize device with USB SUPER speed
                log.info("Initializing device with USB 3.0 (SUPER) speed...")
                self.device = dai.Device(self.pipeline, maxUsbSpeed=dai.UsbSpeed.SUPER)
                
                # Log USB speed after initialization
                usb_speed = self.device.getUsbSpeed()
                log.info(f"Device initialized with USB speed: {usb_speed.name}")
                
                # Get output queues
                self.qRgb = self.device.getOutputQueue(name="rgb", maxSize=1, blocking=False)
                self.qDepth = self.device.getOutputQueue(name="depth", maxSize=1, blocking=False)
                
                log.info("Pipeline initialized successfully")
                
            except Exception as e:
                log.error(f"Failed to initialize pipeline: {str(e)}")
                raise
    
        def run(self):
            """Run the pipeline continuously and monitor for issues."""
            try:
                log.info("Starting pipeline...")
                start_time = time.time()
                frame_count = 0
                
                while True:
                    if self.device.isClosed():
                        log.error("Device connection lost!")
                        break
                        
                    # Get frames
                    rgbFrame = self.qRgb.get()
                    depthFrame = self.qDepth.get()
                    
                    if rgbFrame is None or depthFrame is None:
                        log.warning("Received empty frame")
                        continue
                    
                    # Process frames (minimal processing to simulate load)
                    rgb = rgbFrame.getCvFrame()
                    depth = depthFrame.getFrame()
                    
                    # Log device stats every 30 seconds
                    frame_count += 1
                    if frame_count % 450 == 0:  # At 15 FPS, this is roughly every 30 seconds
                        self._log_device_stats()
                    
                    # Optional: display frames
                    cv2.imshow("RGB", rgb)
                    cv2.imshow("Depth", depth)
                    
                    if cv2.waitKey(1) == ord('q'):
                        break
                        
            except Exception as e:
                log.error(f"Error during pipeline execution: {str(e)}")
            finally:
                self.cleanup()
    
        def _log_device_stats(self):
            """Log device statistics."""
            try:
                temp = self.device.getChipTemperature().average
                css_cpu = self.device.getLeonCssCpuUsage().average
                mss_cpu = self.device.getLeonMssCpuUsage().average
                
                log.info(f"Device Stats - Temp: {temp:.1f}°C, CSS CPU: {css_cpu:.1f}%, MSS CPU: {mss_cpu:.1f}%, USB: {usb_speed.name}")
                
            except Exception as e:
                log.error(f"Failed to get device stats: {str(e)}")
    
        def cleanup(self):
            """Clean up resources."""
            if self.device is not None and not self.device.isClosed():
                self.device.close()
            cv2.destroyAllWindows()
    
    if __name__ == "__main__":
        try:
            oak = OakDMinimal()
            oak.run()
        except Exception as e:
            log.error(f"Application error: {str(e)}") 

    Here is the relevant output from dmesg:

    [Thu May  1 19:09:01 2025] usb 1-3: USB disconnect, device number 49
    [Thu May  1 19:09:02 2025] usb 2-3: new SuperSpeed USB device number 35 using xhci_hcd
    [Thu May  1 19:09:02 2025] usb 2-3: New USB device found, idVendor=03e7, idProduct=f63b, bcdDevice= 1.00
    [Thu May  1 19:09:02 2025] usb 2-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
    [Thu May  1 19:09:02 2025] usb 2-3: Product: Luxonis Device
    [Thu May  1 19:09:02 2025] usb 2-3: Manufacturer: Intel Corporation
    [Thu May  1 19:09:02 2025] usb 2-3: SerialNumber: 14442C10D1ECD4D200
    [Thu May  1 19:09:02 2025] usb 2-3: USB disconnect, device number 35
    [Thu May  1 19:09:02 2025] usb 1-3: new high-speed USB device number 50 using xhci_hcd
    [Thu May  1 19:09:02 2025] usb 1-3: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
    [Thu May  1 19:09:02 2025] usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
    [Thu May  1 19:09:02 2025] usb 1-3: Product: Movidius MyriadX
    [Thu May  1 19:09:02 2025] usb 1-3: Manufacturer: Movidius Ltd.
    [Thu May  1 19:09:02 2025] usb 1-3: SerialNumber: 03e72485
    [Thu May  1 19:09:05 2025] usb 1-3: USB disconnect, device number 50
    [Thu May  1 19:09:06 2025] usb 2-3: new SuperSpeed USB device number 36 using xhci_hcd
    [Thu May  1 19:09:06 2025] usb 2-3: New USB device found, idVendor=03e7, idProduct=f63b, bcdDevice= 1.00
    [Thu May  1 19:09:06 2025] usb 2-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
    [Thu May  1 19:09:06 2025] usb 2-3: Product: Luxonis Device
    [Thu May  1 19:09:06 2025] usb 2-3: Manufacturer: Intel Corporation
    [Thu May  1 19:09:06 2025] usb 2-3: SerialNumber: 14442C10D1ECD4D200
    [Thu May  1 19:09:07 2025] usb 2-3: USB disconnect, device number 36
    [Thu May  1 19:09:07 2025] usb 1-3: new high-speed USB device number 51 using xhci_hcd
    [Thu May  1 19:09:07 2025] usb 1-3: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
    [Thu May  1 19:09:07 2025] usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
    [Thu May  1 19:09:07 2025] usb 1-3: Product: Movidius MyriadX
    [Thu May  1 19:09:07 2025] usb 1-3: Manufacturer: Movidius Ltd.
    [Thu May  1 19:09:07 2025] usb 1-3: SerialNumber: 03e72485
    [Thu May  1 21:24:41 2025] systemd-journald[296]: Data hash table of /var/log/journal/6bd4c45ff3444a0ca280f96ef5c0d07a/system.journal has a fill level at 75.0 (174763 of 233016 items, 109051904 file size, 623 bytes per hash table item), suggesting rotation.
    [Thu May  1 21:24:41 2025] systemd-journald[296]: /var/log/journal/6bd4c45ff3444a0ca280f96ef5c0d07a/system.journal: Journal header limits reached or header out-of-date, rotating.
    [Thu May  1 17:47:49 2025] usb 1-3: Manufacturer: Movidius Ltd.
    [Thu May  1 17:47:49 2025] usb 1-3: SerialNumber: 03e72485
    [Thu May  1 18:47:19 2025] systemd-journald[296]: Data hash table of /var/log/journal/6bd4c45ff3444a0ca280f96ef5c0d07a/system.journal has a fill level at 75.0 (174765 of 233016 items, 109051904 file size, 623 bytes per hash table item), suggesting rotation.
    [Thu May  1 18:47:19 2025] systemd-journald[296]: /var/log/journal/6bd4c45ff3444a0ca280f96ef5c0d07a/system.journal: Journal header limits reached or header out-of-date, rotating.

      joshbarclay
      Maybe try checking if the process keeps running in the background. Something like
      ps aux | grep python or something similar.

      Thanks
      Jaka

        7 days later

        jakaskerl

        It shows the app.server.py is running (which we use to expose the Oak-D)

        root     1162449 66.6  0.0  11404  7808 ?        Rs   10:37   0:00 /opt/service/.venv/bin/python /opt/service/.venv/bin/uvicorn src.server:app --host 0.0.0.0 --port 3332
        root     1515088  5.0  0.7 594532 57944 ?        Ssl  May01 494:42 /opt/service/.venv/bin/python /opt/service/.venv/bin/uvicorn app.server:router --host 0.0.0.0 --port 3330
        root     1516033  0.3  4.6 1648768 369616 ?      Ssl  May01  31:15 python -m app.server

        app.server.py

        from fastapi import BackgroundTasks, FastAPI, HTTPException, Response, status
        from app.camera.oakd_camera import OakDCamera 
        
        class OakDAPI(FastAPI):
            """
            A thin wrapper class around `FastAPI` which allows an `OakD_Camera` instance to be instantiated
            and accessed in a type-safe way. The camera instance is global to all requests reaching the
            server, assuming a single process server.
            """
        
            def __init__(self, *args, logger, **kwargs):
                super().__init__(*args, **kwargs)
                self.router.route_class = make_error_handler_route_class(
                    logger=logger, get_context=self.inspect
                )
                self._oakd_camera: Optional[OakDCamera] = None
        
            def init_camera(self):
                self._oakd_camera = OakDCamera.create_camera()
        
            @property
            def oakd_camera(self):
                if self._oakd_camera is None:
                    raise Exception("`oakd_camera` is None; call `init_camera` first")
                return self._oakd_camera
        
            def inspect(self):
                return self.oakd_camera.inspect()
        
        
        app = OakDAPI(logger=log)
        
        @app.on_event("startup")
        async def app_startup():
            app.init_camera()
        
        @app.on_event("shutdown")
        async def app_shutdown():
            app.oakd_camera.off()
        
        def jsonResponse(body):
            """
            Helper method that formats a python object as JSON and returns the response
            """
            json_str = json.dumps(body, indent=4, default=str)
            return Response(content=json_str, media_type="application/json")
        
        @app.get("/health/check")
        def read_healthcheck():
            return jsonResponse({"ready": app.oakd_camera.is_on})
        
        @app.get("/health/inspect")
        def call_inspect():
            return jsonResponse(app.oakd_camera.inspect())
        
        
        @app.post("/off")
        def turn_off(background_tasks: BackgroundTasks):
            # The camera takes about 4 seconds to turn off so we do it in the background to not block the
            # caller.
            background_tasks.add_task(app.oakd_camera.off)
            return Response(status_code=status.HTTP_202_ACCEPTED)
        
        @app.post("/on")
        def turn_on(background_tasks: BackgroundTasks):
            # The camera takes about 4 seconds to turn on so we do it in the background to not block the
            # caller.
            background_tasks.add_task(app.oakd_camera.on)
            return Response(status_code=status.HTTP_202_ACCEPTED)
        
        @app.get("/distance")
        def estimate_distance():
            app.assert_camera_is_on()
            res = app.oakd_camera.estimate_distance()
            return jsonResponse({"distance": res})
        
        
        def get_app():
            return
        
        
        if __name__ == "__main__":
            """Launched with `poetry run start` at root level"""
            import uvicorn
        
            uvicorn.run("app.server:app", host="0.0.0.0", port=PORT, reload=False)

        Perhaps the OakDCamera.on() method might be throwing an exception or block silently? I can try to add some more logging here.

        Is there a way to profile the hardware for memory leak or check the internal state of the Oak-D?

          5 days later

          joshbarclay
          You can check the state of OAK-D with print(dai.Device.getAllConnectedDevices()). This will give you the state of the device (booted/unbooted). Keep in mind that if you use multiprocessing, it's likely the device won't close correctly. You need to make sure you always keep the device context. Not sure how FastAPI handles this.

          Thanks,
          Jaka