Device crashes and connection drops/retries

MichaelKarlesky

Very happy to provide these things, jakaskerl

The Luxonis Discuss interface won’t allow me to attach the crash logs or even a zip archive. JSON files are prohibited? Here is a Github gist that contains two crash logs instead.

Below are two methods. The first is pipeline set up. A depthai.Device configured with camera IP address is passed in. The second is a method called in a thread to perform the desired human detection. This code is targeting an OAK-D SR PoE camera with static IP.

I am new to DepthAI. I was provided demo code by another team member and have been reworking it into a production-worth application. I'm only in the early stages of doing that. I've gotten far enough that human detection seems to be working, but we’re only logging events rather than using them for anything. And, I can run the application for long stretches of time to exercise stability and reliability. I’ve got a ways to go on all fronts.

Thank you for any assistance.

def _create_depthai_pipeline(self, device: dai.Device) -> dai.Pipeline:
    """Create and configure the DepthAI pipeline"""
    pipeline = dai.Pipeline(device)

    # Set up ToF sensor
    tof = pipeline.create(dai.node.ToF).build()
    
    # Set up RGB Camera
    camera_node = pipeline.create(dai.node.Camera).build(
        boardSocket=dai.CameraBoardSocket.CAM_C, # RGB_SOCKET
        sensorFps=self._fps
    )
    camera_full = camera_node.requestFullResolutionOutput(type=dai.ImgFrame.Type.BGR888p)
    camera_nn = camera_node.requestOutput(
        size=(416, 416),
        resizeMode=dai.ImgResizeMode.LETTERBOX,
        type=dai.ImgFrame.Type.BGR888p
    )

    # Set up image processing for ToF alignment
    tof_resize = pipeline.create(dai.node.ImageManip)
    tof_resize.initialConfig.setOutputSize(416, 416, dai.ImageManipConfig.ResizeMode.LETTERBOX)

    # Set up detection network
    self._detection_network = pipeline.create(dai.node.DetectionNetwork)
    self._detection_network.setFromModelZoo(
        dai.NNModelDescription(model="yolov6-nano", platform="RVC2"),
        useCached=True
    )

    # Create sync and align nodes
    sync = pipeline.create(dai.node.Sync)
    sync.setSyncThreshold(timedelta(seconds=0.5 / self._fps))
    align = pipeline.create(dai.node.ImageAlign)

    # Linking
    camera_nn.link(self._detection_network.input)
    tof.depth.link(align.input)
    camera_full.link(align.inputAlignTo)
    align.outputAligned.link(tof_resize.inputImage)
    
    self._detection_network.passthrough.link(sync.inputs["rgb"])
    tof_resize.out.link(sync.inputs["depth_aligned"])
    sync.inputs["rgb"].setBlocking(False)

    # Create output queues
    self._detection_queue = self._detection_network.out.createOutputQueue()
    self._sync_queue = sync.out.createOutputQueue()

    return pipeline

def _detection_loop(self) -> None:
    """Main detection processing loop running in separate thread"""
    self.get_logger().info("Human detection loop starting...")
    
    try:
        label_map = self._detection_network.getClasses()
    except Exception as e:
        self.get_logger().fatal(f"Fatal error starting detection loop: {e}")
        return

    while self._detection_running:
        try:
            # Get detections and synchronized depth data
            input_detections = self._detection_queue.get()
            message_group = self._sync_queue.get()
            
            if input_detections is None:
                continue

            detections = input_detections.detections
            
            # Get the aligned depth frame
            frame_depth = message_group["depth_aligned"]

            if frame_depth is None:
                continue

            depth_array = frame_depth.getFrame()
            frame_shape = depth_array.shape
            
            # Process detections
            detection_results = []
            alert_level = None
            
            # Process each detection
            for detection in detections:
                # Only process person detections
                if label_map[detection.label] != 'person':
                    continue

                if self._verbose_logging:
                    self.get_logger().info(f"Person detected with confidence: {detection.confidence:.3f}")

                # Only process person detections with sufficient confidence
                if detection.confidence < self._confidence_threshold:
                    continue

                try:
                    # Convert normalized bbox to pixel coordinates
                    bbox = self._frame_norm(
                        frame_shape, 
                        (
                            detection.xmin, detection.ymin, 
                            detection.xmax, detection.ymax
                        )
                    )
                    
                    # Calculate median depth in the bounding box ROI
                    median_depth_mm = self._median_depth_roi(depth_array, bbox)                                    

                    # Only proceed if meaningul depth data
                    if median_depth_mm is None:
                        if self._verbose_logging:
                            self.get_logger().warning("Person detected but no valid depth data in ROI")
                        continue

                    # Convert to meters for readability
                    distance_m = median_depth_mm / 1000.0
                                                                
                    # Determine alert level and log accordingly
                    if median_depth_mm < self._fault_threshold_mm and alert_level != "CRITICAL":
                        alert_level = "CRITICAL"
                        self.get_logger().error(f"Person very close - {distance_m:.2f}m")
                    elif median_depth_mm < self._pause_threshold_mm and alert_level != "WARNING":
                        alert_level = "WARNING"
                        self.get_logger().warning(f"Person approaching - {distance_m:.2f}m")
                    elif alert_level != "SAFE":
                        alert_level = "SAFE"
                        self.get_logger().notice(f"Person at safe distance - {distance_m:.2f}m")

                    if self._verbose_logging:
                        self.get_logger().info(f"Person detected at distance {distance_m:.2f}m with confidence: {detection.confidence:.3f}")

                    # Store detection result for image publishing
                    detection_result = {
                        'distance_mm': median_depth_mm,
                        'distance_m': distance_m,
                        'confidence': detection.confidence,
                        'bbox': {
                            'xmin': detection.xmin,
                            'ymin': detection.ymin,
                            'xmax': detection.xmax,
                            'ymax': detection.ymax
                        }
                    }
                    detection_results.append(detection_result)
                                                                
                except Exception as e:
                    self.get_logger().error(
                        f"Error processing person detection but trying again: {e}",
                        throttle_duration_sec=5.0
                    )
                                                    
        except Exception as e:
            self.get_logger().fatal(f"Error in detection loop: {e}")

    self.get_logger().info("Detection loop ended")

jakaskerl

Hi MichaelKarlesky
This is not MRE so I can't repro it. Can you try running it with DEPTHAI_LEVEL=DEBUG? It should print the CPU and RAM usage.

Thanks,
Jaka

MichaelKarlesky

I will run our code with DEPTHAI_LEVEL=DEBUG and report back.

If you can outline for me your definition of MRE in our case, I will gladly try to provide executable code. This code is running in a ROS2 node. You would need a specific version of ROS2 Humble and perhaps one other customization in order to build the workspace and run the node. If there's a different approach to provide an MRE that would be helpful here, please let me know.

jakaskerl

MichaelKarlesky
cc @Luxonis-Adam for ROS related issues -

MichaelKarlesky

@jakaskerl @Luxonis-Adam To be clear, we're not using any Luxonis ROS integration. We are using the v3-rc4 DepthAI package with straight Python calls, but those calls are wrapped in a ROS2 node. Of course, it's entirely possible that something about ROS2 or our ROS2 node is contributing to the communication problems and device crashes, but, for all intents and purposes, our object detection is executing like a basic Python application that happens to be running inside a ROS2 node.

jakaskerl

MichaelKarlesky
Ah, okay. Then can you extract the running python code and give it as a reproducible example (one that I can paste into a file and run). I would like to test the stability of it myself.

Thanks,
Jaka

MichaelKarlesky

I am working on creating the requested MRE. In the meantime, below is a capture of logging with DEPTHAI_LEVEL=DEBUG.

Today, I simply ran the ROS2 node that contains object detection as is in order to capture debug logs. Pipeline creation repeatedly failed with this exception:

ImageManip out of memory; tried to allocate '225472'B out of '142336'B available

The logs below were captured up until this exception.

I tried several things on the host side including killing processes and rebooting the computer. Nothing seemed to have any impact. Is the memory error referring to the host or to the camera?

[2025-09-11 02:48:58.695] [depthai] [info] DEPTHAI_DEBUG enabled, lowered DEPTHAI_LEVEL to 'debug'
[2025-09-11 02:48:58.695] [depthai] [info] Environment variable DEPTHAI_INSTALL_SIGNAL_HANDLER is not set. Using default value: ''
[2025-09-11 02:48:58.695] [depthai] [info] Environment variable DEPTHAI_LIBUSB_ANDROID_JAVAVM is not set. Using default value: ''
[2025-09-11 02:48:58.695] [depthai] [debug] Python bindings - version: 3.0.0 from  build: 2025-09-07 22:07:32 +0000
[2025-09-11 02:48:58.695] [depthai] [debug] Library information - version: 3.0.0, commit:  from , build: 2025-09-07 22:07:45 +0000, libusb enabled: true
[2025-09-11 02:48:58.708] [depthai] [info] Environment variable DEPTHAI_PROFILING is not set. Using default value: ''
[2025-09-11 02:48:58.708] [depthai] [info] Environment variable XLINK_LEVEL is not set. Using default value: ''
[2025-09-11 02:48:58.708] [depthai] [debug] Initialize - finished
[2025-09-11 02:48:58.862] [host] [debug] Found an actual device by given DeviceInfo: DeviceInfo(name=10.4.8.125, deviceId=19443010B165705A00, X_LINK_BOOTLOADER, X_LINK_TCP_IP, X_LINK_MYRIAD_X, X_LINK_SUCCESS)
[2025-09-11 02:48:58.862] [host] [debug] Device - OpenVINO version: universal
[2025-09-11 02:48:58.862] [depthai] [info] Environment variable DEPTHAI_WATCHDOG is not set. Using default value: ''
[2025-09-11 02:48:58.862] [depthai] [info] Environment variable DEPTHAI_WATCHDOG_INITIAL_DELAY is not set. Using default value: ''
[19443010B165705A00] [10.4.8.125] [1757558938.863] [host] [debug] Device - BoardConfig: {"camera":[],"defaultImuExtr":{},"emmc":null,"gpio":[],"imu":null,"logDevicePrints":true,"logPath":null,"logSizeMax":null,"logVerbosity":null,"network":{"mtu":0,"xlinkTcpNoDelay":true},"nonExclusiveMode":false,"pcieInternalClock":null,"sysctl":[],"uart":[],"usb":{"flashBootedPid":63037,"flashBootedVid":999,"manufacturer":"","maxSpeed":4,"pid":63035,"productName":"","vid":999},"usb3PhyInternalClock":null,"uvc":null,"watchdogInitialDelayMs":null,"watchdogTimeoutMs":null}
libnop:
0000: b9 13 b9 07 81 e7 03 81 3b f6 81 e7 03 81 3d f6 04 bd 00 bd 00 b9 02 00 01 ba 00 be be bb 00 bb
0020: 00 be be be be be be 01 00 bb 00 be be bb 00
[2025-09-11 02:48:58.874] [depthai] [debug] Resources - Archive 'depthai-bootloader-fwp-0.0.28.tar.xz' open: 3ms, archive read: 174ms
[2025-09-11 02:48:59.568] [depthai] [debug] Resources - Archive 'depthai-device-fwp-03a9cc7382bd8755768dc858cb62398643029e6e.tar.xz' open: 3ms, archive read: 868ms
[2025-09-11 02:48:59.568] [depthai] [info] Environment variable DEPTHAI_DEVICE_BINARY is not set. Using default value: ''
[2025-09-11 02:48:59.616] [depthai] [info] Environment variable DEPTHAI_CONNECT_TIMEOUT is not set. Using default value: ''
[2025-09-11 02:48:59.616] [depthai] [info] Environment variable DEPTHAI_BOOTUP_TIMEOUT is not set. Using default value: ''
[2025-09-11 02:48:59.616] [depthai] [debug] Searching for booted device: DeviceInfo(name=10.4.8.125, deviceId=19443010B165705A00, X_LINK_BOOTLOADER, X_LINK_TCP_IP, X_LINK_MYRIAD_X, X_LINK_SUCCESS), name used as hint only
[2025-09-11 02:48:59.632] [depthai] [debug] Connected bootloader version 0.0.28
[19443010B165705A00] [10.4.8.125] [1757558940.652] [host] [debug] Booting FW with Bootloader. Version 0.0.28, Time taken: 1019ms
[2025-09-11 02:49:00.652] [depthai] [debug] DeviceBootloader about to be closed...
[2025-09-11 02:49:00.652] [depthai] [debug] XLinkResetRemote of linkId: (0)
[2025-09-11 02:49:01.690] [depthai] [debug] DeviceBootloader closed, 1038
[2025-09-11 02:49:01.695] [depthai] [debug] Searching for booted device: DeviceInfo(name=10.4.8.125, deviceId=19443010B165705A00, X_LINK_BOOTED, X_LINK_TCP_IP, X_LINK_MYRIAD_X, X_LINK_SUCCESS), name used as hint only
[2025-09-11 02:49:06.450] [depthai] [info] Environment variable DEPTHAI_CRASHDUMP is not set. Using default value: ''
[19443010B165705A00] [10.4.8.125] [5.232] [system] [warning] PRINT:LeonCss: [OK]: INIT_NOC: noc probes timer create
[OK]: INIT_NOC: noc probes timer start
[OK]: INIT_NOC: ddr bandwith profiling!
GPIO boot mode 0x3, interface SPI_MASTER_EFF
Setting aons(0..4) back to boot from flash (offset = 0)
====ENABLE WATCHDOG====1
initial keepalive, countdown: 7
PLL0: 700000 AUX_IO0: 24000 AUX_IO1: 24000 MCFG: 24000 MECFG: 24000
Board init ret 3
DepthAI Firmware - version: 03a9cc7382bd8755768dc858cb62398643029e6e
eeprom configuration version: 55AA0007
Reading VERSION 7 --- ->
eeprom configuration load from user area, status: 0
Reading VERSION 7 --- ->
Found a device/board entry matching the eeprom data - Board: OAK-D SR POE (R4M3E4) Device: OAK-D-SR-POE
Board options: 00001901
Overriding GPIO bootstrap with boot mode from eeprom boardOptions. Boot mode: 0x3
--> brdInit ...
brdInitAuxDevices: Error: SC = 27: io_initialize expander_cam_gpios_1 [OK]
[human_detector-1]
spi_N25Q_init: Flash JEDEC ID: 20 bb 19
Initialized pin: 43, mode: 7, level: 0, pad: 0x00000010
Initialized pin: 59, mode: 7, level: 1, pad: 0x00000010
Initialized pin: 51, mode: 7, level: 1, pad: 0x00000010
Initialized pin: 61, mode: 7, level: 0, pad: 0x00000050
Initialized pin: 60, mode: 7, level: 0, pad: 0x00000010
Initialized pin: 30, mode: 7, level: 1, pad: 0x00000010
inited hal ---
def: 53 46 45 34 10 06 08 34 10 53 14 45 46 08 06
val: 16 11 45 15 15 06 08 15 15 16 14 45 11 08 06
Opening bus for IR driver: 1
Failed to probe IR driver LM3644
Opening bus for IR driver: 2
LM3644 detected, ID = 0x02
===== IR write bus 2: 0x07 = 0x89
===== IR write bus 2: 0x02 = 0x01
===== IR write bus 2: 0x03 = 0x00
===== IR write bus 2: 0x04 = 0x00
===== IR write bus 2: 0x05 = 0x00
===== IR write bus 2: 0x06 = 0x00
===== IR write bus 2: 0x07 = 0x09
===== IR write bus 2: 0x08 = 0x1a
===== IR write bus 2: 0x09 = 0x08
===== IR write bus 2: 0x01 = 0x24
Opening bus for IR driver: 3
Failed to probe IR driver LM3644
Closing EEPROm!
MyriaX board configuration
pll0 frequency: 700000, ref0 frequency: 24000
Setup network with bootloader config
OsDrvCprSysDeviceAction() for PCIe failed: 13
nexus0: <RTEMS Nexus device>
pcib0: <MyriadX PCIe RC> on nexus0
PCIE_REGS_PCIE_CFG_ADR = 0x60100
PCIE_REGS_PCIE_PLL_CNTRL_ADR = 0x1
pcib0: Link training complete, PCIe downstream link is UP
pcib0: myriad_pcie_err_interrupt: SMLH LINK UP
pcib0: myriad_pcie_err_interrupt: SMLH LINK DOWN
pcib0: myriad_pcie_err_interrupt: RDLH LINK UP
pcib0: myriad_pcie_err_interrupt: RDLH LINK DOWN
pcib0: myriad_pcie_err_interrupt: CORE RST
pcib0: myriad_pcie_err_interrupt: CORE RSTN
pci0: <PCI bus> on pcib0
memcpy_dma_init: got DMA id 62
re0: <Realtek PCIe GBE Family Controller> mem 0x1000000-0x1000fff,0x1004000-0x1007fff at device 0.0 on pci0
re0: Using Memory Mapping!
re0: Using 1 MSI message
Overriden invalid mac address, mac: 44:A9:2C:34:4F:96
re0: version:1.95.00
re0: Ethernet address: 0x83f531b0
[human_detector-1]
This product is covered by one or more of the following patents:
US6,570,884, US6,115,776, and US6,327,625.
info: re0: Ethernet address: 44:a9:2c:34:4f:96
myriad_mmc2: <MyriadX MMC> on nexus0
myriad_mmc2: Memory resource: rid 0 tag 0x0 handle 0x33000000 size 0x1
myriad_mmc2: Found that the memory was NULL - releasing.
myriad_mmc2: Error getting voltage regulator resource.
myriad_mmc2: Found that the memory was NULL - releasing.
myriad_mmc2: Error getting card detect resource.
myriad_mmc2: Memory resource: rid 3 tag 0x0 handle 0x20280000 size 0x1
myriad_mmc2: Memory resource: rid 4 tag 0x0 handle 0x83d7fa38 size 0x1
myriad_mmc2: IRQ resource: start 0xa001b: irq = 27, priority = 10
myriad_mmc2: Doing host configuration ... 5
myriad_mmc2: CFG 0x0 = 0xffb23840
myriad_mmc2: CFG 0x4 = 0x1c008be
myriad_mmc2: CFG 0x24 = 0xb00a621
myriad_mmc2: CFG 0x2c = 0xa000
myriad_mmc2: Polling CFG 0x40 for 0x40
myriad_mmc2: Polling complete, value became 0x4c
myriad_mmc2: Base clock = 200000000 Hz
myriad_mmc2: Re-tuning mode is 1
myriad_mmc2: Re-Tuning timer count is 1 seconds
myriad_mmc2: SDR50 needs tuning.
myriad_mmc2: Supports Vdd=3.3
myriad_mmc2: Supports Vdd=3.0
myriad_mmc2: Supports Vdd=1.8
myriad_mmc2: Supports 8-bit bus width
myriad_mmc2: Supports High Speed
myriad_mmc2: Supports DDR50, DDR52 @ 1.8V
myriad_mmc2: Supports HS200 @ 1.8V
myriad_mmc2: Supports Driver Type A
myriad_mmc2: Supports Driver Type C
myriad_mmc2: Supports Driver Type D
myriad_mmc2: Data timeout counter value = 0xc
myriad_mmc2: Data timeout clock frequency = 48000 kHz, timeout count = 0xc
myriad_mmc2: No voltage regulator.
myriad_mmc2: Card detect task up!
myriad_mmc2: Card attached!
mmc0: <MMC/SD bus> on myriad_mmc2
myriad_mmc2: Updating card mode to SD
myriad_mmc2: Updating card mode to MMC
Probed device vendor id: 0x0, device id 0x0
mmc0: No information for this mode.
mmcsd0: 16GB <MMCHC DF4016 0.1 SN D03F6D1D MFG 05/2021 by 69 0x0000> at mmc0 200.0MHz/8bit/1-block
mmcsd0boot0: 4MB partition 1 at mmcsd0
mmcsd1boot1: 4MB partition 2 at mmcsd0
mmcsd0rpmb: 4MB partition 3 at mmcsd0
net.inet.tcp.delayed_ack: 1 -> 0
info: lo0: link state changed to UP
ifconfig: inet6: bad value
ifcommand setting static ip: 10.4.8.125 mask: 255.255.255.0, gateway: 10.4.8.1
=== Network ready!
Is booted from flash by bootloader: 0
Started device discovery service!
Waiting for broadcast message..
=== Enumerating on socket: Cam_A / RGB / Center
        skipped OV7251
        skipped AR0234
  >> Registered camera ToF33D (s5k33d) as /dev/Camera_0
camera socket: 0, name: tof
        config - w: 1280, h: 3848, type: TOF
Adding socket 0: cam 22. Sen name: S5K33D
=== Enumerating on socket: Cam_B / Left
  >> Registered camera TG161B (ov9282) as /dev/Camera_1
camera socket: 1, name: left
        config - w: 1280, h: 720, type: COLOR
        config - w: 1280, h: 800, type: COLOR
Adding socket 1: cam 5. Sen name: OV9782
=== Enumerating on socket: Cam_C / Right
  >> Registered camera TG161B (ov9282) as /dev/Camera_2
camera socket: 2, name: right
        config - w: 1280, h: 720, type: COLOR
        config - w: 1280, h: 800, type: COLOR
Adding socket 2: cam 5. Sen name: OV9782
Found a device/board entry matching the eeprom data - Board: OAK-D SR POE (R4M3E4) Device: OAK-D-SR-POE
Error [-3] : Device not found error. It occurs when the device chip id is incorrectly read
SH2 Reset.
IMU version: BNO086
Initializing XLink...
info: re0: link state changed to DOWN
initial keepalive, countdown: 6
initial keepalive, countdown: 5
info: re0: link state changed to UP
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,LINKSTATE>
        ether 44:a9:2c:34:4f:96
        inet 10.4.8.125 netmask 0xffffff00 broadcast 10.4.8.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
Received device discovery request, sending back - mxid: 19443010B165705A00, state: 1
Received device discovery request, sending back - mxid: 19443010B165705A00, state: 1
Received device discovery request, sending back - mxid: 19443010B165705A00, state: 1
Done!
Reading..........Reading VERSION 7 --- ->
Found a device/board entry matching the eeprom data - Board: OAK-D SR POE (R4M3E4) Device: OAK-D-SR-POE
Temperature: Driver registered.
Temperature: Initialized driver.
Temperature: Sensor opened: CSS.
Temperature: Sensor opened: MSS.
Temperature: Sensor opened: UPA.
Temperature: Sensor opened: DSS.
[19443010B165705A00] [10.4.8.125] [5.243] [system] [info] Memory Usage - DDR: 0.12 / 333.26 MiB, CMX: 2.04 / 2.50 MiB, LeonOS Heap: 25.06 / 81.17 MiB, LeonRT Heap: 2.89 / 39.23 MiB / NOC ddr: 0 MB/s
[19443010B165705A00] [10.4.8.125] [5.243] [system] [info] Temperatures - Average: 37.47C, CSS: 39.11C, MSS 37.01C, UPA: 36.77C, DSS: 37.01C
[19443010B165705A00] [10.4.8.125] [5.243] [system] [info] Cpu Usage - LeonOS 40.29%, LeonRT: 0.58%
[INFO] [1757558946.467387370] [human_detector]: Camera found and set up successfully
[19443010B165705A00] [10.4.8.125] [5.253] [system] [warning] PRINT:LeonCss: I: [Timesync] [   8303904] [XLin] startSync:186     Timesync | Callback not set
[INFO] [1757558946.470054619] [human_detector]: Creating DepthAI pipeline...
[2025-09-11 02:49:06.483] [depthai] [info] Trying to get API key from environment variable DEPTHAI_HUB_API_KEY
[2025-09-11 02:49:06.483] [depthai] [info] Environment variable DEPTHAI_HUB_API_KEY is not set. Using default value: ''
[2025-09-11 02:49:06.483] [depthai] [info] API key not provided
[2025-09-11 02:49:06.483] [depthai] [info] Trying to get cache directory from environment variable DEPTHAI_ZOO_CACHE_PATH
[2025-09-11 02:49:06.483] [depthai] [info] Environment variable DEPTHAI_ZOO_CACHE_PATH is not set. Using default value: '.depthai_cached_models'
[2025-09-11 02:49:06.483] [depthai] [debug] Cache directory has read permissions: true, has write permissions: true
[2025-09-11 02:49:06.483] [depthai] [info] Cache directory has write permissions, creating a .locks folder
[2025-09-11 02:49:06.483] [depthai] [info] Locking model cache directory: .depthai_cached_models/.locks/98fa008f114a052fc110a400b9f2526e541e7c72.lock
[2025-09-11 02:49:06.483] [depthai] [info] Model cache directory locked: .depthai_cached_models/.locks/98fa008f114a052fc110a400b9f2526e541e7c72.lock
[2025-09-11 02:49:06.484] [depthai] [info] Environment variable DEPTHAI_ZOO_INTERNET_CHECK is not set. Using default value: 'true'
[2025-09-11 02:49:06.484] [depthai] [info] Environment variable DEPTHAI_ZOO_INTERNET_CHECK_TIMEOUT is not set. Using default value: '1000'
[2025-09-11 02:49:06.684] [depthai] [info] Model is cached: true | Metadata present: true | Use cached model: true | Perform internet check: true | Internet is available: true | useCached: true | has folder lock: true
[19443010B165705A00] [10.4.8.125] [5.506] [system] [warning] PRINT:LeonCss: Received device discovery request, sending back - mxid: 19443010B165705A00, state: 1
[19443010B165705A00] [10.4.8.125] [5.517] [system] [warning] PRINT:LeonCss: Received device discovery request, sending back - mxid: 19443010B165705A00, state: 1
[19443010B165705A00] [10.4.8.125] [6.045] [system] [warning] PRINT:LeonCss: initial keepalive, countdown: 4
[2025-09-11 02:49:07.312] [depthai] [info] Using cached model located at /home/optimusclean/optimus_clean_ws/.depthai_cached_models/98fa008f114a052fc110a400b9f2526e541e7c72/yolov6n_coco_416x416.rvc2.tar.xz
[19443010B165705A00] [10.4.8.125] [6.244] [system] [info] Memory Usage - DDR: 0.12 / 333.26 MiB, CMX: 2.04 / 2.50 MiB, LeonOS Heap: 25.54 / 81.17 MiB, LeonRT Heap: 2.89 / 39.23 MiB / NOC ddr: 6 MB/s
[19443010B165705A00] [10.4.8.125] [6.244] [system] [info] Temperatures - Average: 37.36C, CSS: 39.11C, MSS 37.01C, UPA: 36.77C, DSS: 36.53C
[19443010B165705A00] [10.4.8.125] [6.244] [system] [info] Cpu Usage - LeonOS 3.36%, LeonRT: 0.05%
[19443010B165705A00] [10.4.8.125] [6.507] [system] [warning] PRINT:LeonCss: Received device discovery request, sending back - mxid: 19443010B165705A00, state: 1
Received device discovery request, sending back - mxid: 19443010B165705A00, state: 1
[19443010B165705A00] [10.4.8.125] [7.245] [system] [info] Memory Usage - DDR: 0.12 / 333.26 MiB, CMX: 2.04 / 2.50 MiB, LeonOS Heap: 25.54 / 81.17 MiB, LeonRT Heap: 2.89 / 39.23 MiB / NOC ddr: 2 MB/s
[19443010B165705A00] [10.4.8.125] [7.245] [system] [info] Temperatures - Average: 37.30C, CSS: 38.88C, MSS 36.53C, UPA: 37.01C, DSS: 36.77C
[19443010B165705A00] [10.4.8.125] [7.245] [system] [info] Cpu Usage - LeonOS 1.06%, LeonRT: 0.05%
[19443010B165705A00] [10.4.8.125] [7.508] [system] [warning] PRINT:LeonCss: Received device discovery request, sending back - mxid: 19443010B165705A00, state: 1
Received device discovery request, sending back - mxid: 19443010B165705A00, state: 1
[19443010B165705A00] [10.4.8.125] [8.246] [system] [info] Memory Usage - DDR: 0.12 / 333.26 MiB, CMX: 2.04 / 2.50 MiB, LeonOS Heap: 25.54 / 81.17 MiB, LeonRT Heap: 2.89 / 39.23 MiB / NOC ddr: 1 MB/s
[19443010B165705A00] [10.4.8.125] [8.246] [system] [info] Temperatures - Average: 37.59C, CSS: 39.11C, MSS 37.48C, UPA: 37.01C, DSS: 36.77C
[19443010B165705A00] [10.4.8.125] [8.246] [system] [info] Cpu Usage - LeonOS 0.98%, LeonRT: 0.05%
[19443010B165705A00] [10.4.8.125] [8.509] [system] [warning] PRINT:LeonCss: Received device discovery request, sending back - mxid: 19443010B165705A00, state: 1
Received device discovery request, sending back - mxid: 19443010B165705A00, state: 1
[19443010B165705A00] [10.4.8.125] [9.247] [system] [info] Memory Usage - DDR: 0.12 / 333.26 MiB, CMX: 2.04 / 2.50 MiB, LeonOS Heap: 25.54 / 81.17 MiB, LeonRT Heap: 2.89 / 39.23 MiB / NOC ddr: 1 MB/s
[19443010B165705A00] [10.4.8.125] [9.247] [system] [info] Temperatures - Average: 37.36C, CSS: 39.11C, MSS 37.24C, UPA: 36.77C, DSS: 36.30C
[19443010B165705A00] [10.4.8.125] [9.247] [system] [info] Cpu Usage - LeonOS 0.61%, LeonRT: 0.05%
[19443010B165705A00] [10.4.8.125] [9.510] [system] [warning] PRINT:LeonCss: Received device discovery request, sending back - mxid: 19443010B165705A00, state: 1
Received device discovery request, sending back - mxid: 19443010B165705A00, state: 1
[19443010B165705A00] [10.4.8.125] [10.248] [system] [info] Memory Usage - DDR: 0.12 / 333.26 MiB, CMX: 2.04 / 2.50 MiB, LeonOS Heap: 25.54 / 81.17 MiB, LeonRT Heap: 2.89 / 39.23 MiB / NOC ddr: 1 MB/s
[19443010B165705A00] [10.4.8.125] [10.248] [system] [info] Temperatures - Average: 37.47C, CSS: 39.58C, MSS 36.30C, UPA: 36.30C, DSS: 37.71C
[19443010B165705A00] [10.4.8.125] [10.248] [system] [info] Cpu Usage - LeonOS 0.59%, LeonRT: 0.05%
[19443010B165705A00] [10.4.8.125] [10.511] [system] [warning] PRINT:LeonCss: Received device discovery request, sending back - mxid: 19443010B165705A00, state: 1
Received device discovery request, sending back - mxid: 19443010B165705A00, state: 1
[INFO] [1757558952.006409385] [human_detector]: Pipeline created successfully
[INFO] [1757558952.007451740] [human_detector]: Human detector configured successfully
[lifecycle_manager-2] [INFO] [1757558952.009946409] [lifecycle_manager]: Activating human_detector
[INFO] [1757558952.012763677] [human_detector]: Activating human detector...
[INFO] [1757558952.014102069] [human_detector]: Starting DepthAI pipeline...
[2025-09-11 02:49:12.014] [depthai] [info] Environment variable DEPTHAI_RECORD is not set. Using default value: ''
[2025-09-11 02:49:12.014] [depthai] [info] Environment variable DEPTHAI_REPLAY is not set. Using default value: ''
[19443010B165705A00] [10.4.8.125] [1757558952.060] [host] [debug] Schema dump: {"connections":[{"node1Id":14,"node1Output":"out","node1OutputGroup":"","node2Id":10,"node2Input":"input","node2InputGroup":""},{"node1Id":1,"node1Output":"depth","node1OutputGroup":"","node2Id":11,"node2Input":"in","node2InputGroup":""},{"node1Id":4,"node1Output":"0","node1OutputGroup":"dynamicOutputs","node2Id":10,"node2Input":"inputAlignTo","node2InputGroup":""},{"node1Id":4,"node1Output":"1","node1OutputGroup":"dynamicOutputs","node2Id":7,"node2Input":"in","node2InputGroup":""},{"node1Id":5,"node1Output":"out","node1OutputGroup":"","node2Id":9,"node2Input":"depth_aligned","node2InputGroup":"inputs"},{"node1Id":7,"node1Output":"passthrough","node1OutputGroup":"","node2Id":9,"node2Input":"rgb","node2InputGroup":"inputs"},{"node1Id":7,"node1Output":"out","node1OutputGroup":"","node2Id":8,"node2Input":"in","node2InputGroup":""},{"node1Id":8,"node1Output":"out","node1OutputGroup":"","node2Id":15,"node2Input":"in","node2InputGroup":""},{"node1Id":9,"node1Output":"out","node1OutputGroup":"","node2Id":17,"node2Input":"in","node2InputGroup":""},{"node1Id":10,"node1Output":"outputAligned","node1OutputGroup":"","node2Id":5,"node2Input":"inputImage","node2InputGroup":""}],"globalProperties":{"calibData":null,"cameraTuningBlobSize":null,"cameraTuningBlobUri":"","eepromId":0,"leonCssFrequencyHz":700000000.0,"leonMssFrequencyHz":700000000.0,"pipelineName":null,"pipelineVersion":null,"sippBufferSize":18432,"sippDmaBufferSize":16384,"xlinkChunkSize":-1},"nodes":[[15,{"alias":"","id":15,"ioInfo":[[["","in"],{"blocking":true,"group":"","id":28,"name":"in","queueSize":3,"type":3,"waitForMessage":false}]],"logLevel":1,"name":"XLinkOut","parentId":-1,"properties":[185,3,136,0,0,128,191,189,9,95,95,120,95,56,95,111,117,116,0]}],[11,{"alias":"","id":11,"ioInfo":[[["","in"],{"blocking":true,"group":"","id":26,"name":"in","queueSize":3,"type":3,"waitForMessage":false}]],"logLevel":1,"name":"XLinkOut","parentId":-1,"properties":[185,3,136,0,0,128,191,189,12,95,95,120,95,49,95,95,100,101,112,116,104,0]}],[10,{"alias":"","id":10,"ioInfo":[[["","outputAligned"],{"blocking":false,"group":"","id":24,"name":"outputAligned","queueSize":8,"type":0,"waitForMessage":false}],[["","passthroughInput"],{"blocking":false,"group":"","id":25,"name":"passthroughInput","queueSize":8,"type":0,"waitForMessage":false}],[["","inputAlignTo"],{"blocking":false,"group":"","id":23,"name":"inputAlignTo","queueSize":1,"type":3,"waitForMessage":true}],[["","input"],{"blocking":false,"group":"","id":22,"name":"input","queueSize":4,"type":3,"waitForMessage":false}],[["","inputConfig"],{"blocking":false,"group":"","id":21,"name":"inputConfig","queueSize":4,"type":3,"waitForMessage":false}]],"logLevel":1,"name":"ImageAlign","parentId":-1,"properties":[185,8,185,1,0,4,0,0,188,0,255,1,2]}],[9,{"alias":"","id":9,"ioInfo":[[["","out"],{"blocking":false,"group":"","id":20,"name":"out","queueSize":8,"type":0,"waitForMessage":false}],[["inputs","rgb"],{"blocking":false,"group":"inputs","id":19,"name":"rgb","queueSize":10,"type":3,"waitForMessage":false}],[["inputs","depth_aligned"],{"blocking":false,"group":"inputs","id":18,"name":"depth_aligned","queueSize":10,"type":3,"waitForMessage":false}]],"logLevel":1,"name":"Sync","parentId":-1,"properties":[185,2,134,120,81,254,0,255]}],[8,{"alias":"detectionParser","id":8,"ioInfo":[[["","out"],{"blocking":false,"group":"","id":17,"name":"out","queueSize":8,"type":0,"waitForMessage":false}],[["","in"],{"blocking":true,"group":"","id":16,"name":"in","queueSize":1,"type":3,"waitForMessage":true}]],"logLevel":1,"name":"DetectionParser","parentId":6,"properties":[185,3,8,187,1,189,6,105,109,97,103,101,115,185,10,133,33,67,1,4,188,16,160,1,0,0,160,1,0,0,3,0,0,0,1,0,0,0,188,0,189,6,105,109,97,103,101,115,0,0,136,0,0,128,63,136,0,0,0,0,185,10,0,189,6,121,111,108,111,118,54,136,0,0,0,63,80,186,80,189,6,112,101,114,115,111,110,189,7,98,105,99,121,99,108,101,189,3,99,97,114,189,10,109,111,116,111,114,99,121,99,108,101,189,8,97,105,114,112,108,97,110,101,189,3,98,117,115,189,5,116,114,97,105,110,189,5,116,114,117,99,107,189,4,98,111,97,116,189,13,116,114,97,102,102,105,99,32,108,105,103,104,116,189,12,102,105,114,101,32,104,121,100,114,97,110,116,189,9,115,116,111,112,32,115,105,103,110,189,13,112,97,114,107,105,110,103,32,109,101,116,101,114,189,5,98,101,110,99,104,189,4,98,105,114,100,189,3,99,97,116,189,3,100,111,103,189,5,104,111,114,115,101,189,5,115,104,101,101,112,189,3,99,111,119,189,8,101,108,101,112,104,97,110,116,189,4,98,101,97,114,189,5,122,101,98,114,97,189,7,103,105,114,97,102,102,101,189,8,98,97,99,107,112,97,99,107,189,8,117,109,98,114,101,108,108,97,189,7,104,97,110,100,98,97,103,189,3,116,105,101,189,8,115,117,105,116,99,97,115,101,189,7,102,114,105,115,98,101,101,189,4,115,107,105,115,189,9,115,110,111,119,98,111,97,114,100,189,11,115,112,111,114,116,115,32,98,97,108,108,189,4,107,105,116,101,189,12,98,97,115,101,98,97,108,108,32,98,97,116,189,14,98,97,115,101,98,97,108,108,32,103,108,111,118,101,189,10,115,107,97,116,101,98,111,97,114,100,189,9,115,117,114,102,98,111,97,114,100,189,13,116,101,110,110,105,115,32,114,97,99,107,101,116,189,6,98,111,116,116,108,101,189,10,119,105,110,101,32,103,108,97,115,115,189,3,99,117,112,189,4,102,111,114,107,189,5,107,110,105,102,101,189,5,115,112,111,111,110,189,4,98,111,119,108,189,6,98,97,110,97,110,97,189,5,97,112,112,108,101,189,8,115,97,110,100,119,105,99,104,189,6,111,114,97,110,103,101,189,8,98,114,111,99,99,111,108,105,189,6,99,97,114,114,111,116,189,7,104,111,116,32,100,111,103,189,5,112,105,122,122,97,189,5,100,111,110,117,116,189,4,99,97,107,101,189,5,99,104,97,105,114,189,5,99,111,117,99,104,189,12,112,111,116,116,101,100,32,112,108,97,110,116,189,3,98,101,100,189,12,100,105,110,105,110,103,32,116,97,98,108,101,189,6,116,111,105,108,101,116,189,2,116,118,189,6,108,97,112,116,111,112,189,5,109,111,117,115,101,189,6,114,101,109,111,116,101,189,8,107,101,121,98,111,97,114,100,189,10,99,101,108,108,32,112,104,111,110,101,189,9,109,105,99,114,111,119,97,118,101,189,4,111,118,101,110,189,7,116,111,97,115,116,101,114,189,4,115,105,110,107,189,12,114,101,102,114,105,103,101,114,97,116,111,114,189,4,98,111,111,107,189,5,99,108,111,99,107,189,4,118,97,115,101,189,8,115,99,105,115,115,111,114,115,189,10,116,101,100,100,121,32,98,101,97,114,189,10,104,97,105,114,32,100,114,105,101,114,189,10,116,111,111,116,104,98,114,117,115,104,4,186,0,187,0,186,0,136,0,0,0,63]}],[7,{"alias":"neuralNetwork","id":7,"ioInfo":[[["","passthrough"],{"blocking":false,"group":"","id":15,"name":"passthrough","queueSize":8,"type":0,"waitForMessage":false}],[["","out"],{"blocking":false,"group":"","id":14,"name":"out","queueSize":8,"type":0,"waitForMessage":false}],[["","in"],{"blocking":true,"group":"","id":13,"name":"in","queueSize":3,"type":3,"waitForMessage":true}]],"logLevel":1,"name":"NeuralNetwork","parentId":6,"properties":[185,10,0,130,170,152,132,0,189,12,97,115,115,101,116,58,95,95,98,108,111,98,189,0,8,0,0,0,189,0,187,0]}],[5,{"alias":"","id":5,"ioInfo":[[["","out"],{"blocking":false,"group":"","id":12,"name":"out","queueSize":8,"type":0,"waitForMessage":false}],[["","inputImage"],{"blocking":true,"group":"","id":11,"name":"inputImage","queueSize":3,"type":3,"waitForMessage":false}],[["","inputConfig"],{"blocking":true,"group":"","id":10,"name":"inputConfig","queueSize":3,"type":3,"waitForMessage":false}]],"logLevel":1,"name":"ImageManip","parentId":-1,"properties":[185,5,185,4,185,11,186,0,129,160,1,129,160,1,1,2,0,0,0,0,0,0,33,0,0,134,0,0,16,0,4,0,0]}],[17,{"alias":"","id":17,"ioInfo":[[["","in"],{"blocking":true,"group":"","id":29,"name":"in","queueSize":3,"type":3,"waitForMessage":false}]],"logLevel":1,"name":"XLinkOut","parentId":-1,"properties":[185,3,136,0,0,128,191,189,9,95,95,120,95,57,95,111,117,116,0]}],[4,{"alias":"","id":4,"ioInfo":[[["dynamicOutputs","0"],{"blocking":false,"group":"dynamicOutputs","id":9,"name":"0","queueSize":8,"type":0,"waitForMessage":false}],[["dynamicOutputs","1"],{"blocking":false,"group":"dynamicOutputs","id":8,"name":"1","queueSize":8,"type":0,"waitForMessage":false}],[["","raw"],{"blocking":false,"group":"","id":7,"name":"raw","queueSize":8,"type":0,"waitForMessage":false}],[["","mockIsp"],{"blocking":true,"group":"","id":6,"name":"mockIsp","queueSize":8,"type":3,"waitForMessage":false}],[["","inputControl"],{"blocking":true,"group":"","id":5,"name":"inputControl","queueSize":3,"type":3,"waitForMessage":false}]],"logLevel":1,"name":"Camera","parentId":-1,"properties":[185,17,185,33,0,3,0,136,0,0,0,0,128,176,128,158,185,3,129,255,255,130,224,74,183,97,129,255,255,185,5,129,144,147,129,183,97,129,255,255,0,11,185,5,0,0,129,111,114,129,105,101,130,110,116,97,116,105,111,110,0,0,0,0,0,0,185,3,0,0,0,185,3,0,0,0,0,0,0,132,161,6,0,0,0,0,0,128,163,0,186,0,2,255,189,0,255,255,255,255,255,136,0,0,240,65,0,3,3,4,4,4,186,2,185,5,185,1,184,0,186,2,129,0,5,129,32,3,185,1,190,8,0,190,185,5,185,1,184,0,186,2,129,160,1,129,160,1,185,1,190,8,2,190]}],[14,{"alias":"","id":14,"ioInfo":[[["","out"],{"blocking":false,"group":"","id":27,"name":"out","queueSize":8,"type":0,"waitForMessage":false}]],"logLevel":1,"name":"XLinkIn","parentId":-1,"properties":[185,3,189,13,95,95,120,95,49,48,95,95,105,110,112,117,116,130,0,6,26,1,8]}],[1,{"alias":"tofBase","id":1,"ioInfo":[[["","intensity"],{"blocking":false,"group":"","id":3,"name":"intensity","queueSize":8,"type":0,"waitForMessage":false}],[["","amplitude"],{"blocking":false,"group":"","id":2,"name":"amplitude","queueSize":8,"type":0,"waitForMessage":false}],[["","phase"],{"blocking":false,"group":"","id":4,"name":"phase","queueSize":8,"type":0,"waitForMessage":false}],[["","depth"],{"blocking":false,"group":"","id":1,"name":"depth","queueSize":8,"type":0,"waitForMessage":false}],[["","inputConfig"],{"blocking":true,"group":"","id":0,"name":"inputConfig","queueSize":3,"type":3,"waitForMessage":false}]],"logLevel":1,"name":"ToF","parentId":0,"properties":[185,9,185,11,0,1,0,1,190,190,190,190,190,4,75,4,190,188,0,0,189,0,255,136,0,0,0,0,3]}]]}
[19443010B165705A00] [10.4.8.125] [1757558952.060] [host] [debug] Asset map dump: {"map":{"/node/7/__blob":{"alignment":64,"offset":0,"size":8689834}}}
[19443010B165705A00] [10.4.8.125] [10.920] [system] [warning] PRINT:LeonCss: Calling IPC set assets
[19443010B165705A00] [10.4.8.125] [11.251] [system] [info] Memory Usage - DDR: 8.41 / 333.26 MiB, CMX: 2.04 / 2.50 MiB, LeonOS Heap: 28.84 / 81.17 MiB, LeonRT Heap: 2.90 / 39.23 MiB / NOC ddr: 1 MB/s
[19443010B165705A00] [10.4.8.125] [11.251] [system] [info] Temperatures - Average: 38.41C, CSS: 40.51C, MSS 37.24C, UPA: 37.48C, DSS: 38.41C
[19443010B165705A00] [10.4.8.125] [11.251] [system] [info] Cpu Usage - LeonOS 37.34%, LeonRT: 1.92%
[2025-09-11 02:49:12.502] [depthai] [info] Environment variable DEPTHAI_ENABLE_ANALYTICS_COLLECTION is not set. Using default value: ''
[2025-09-11 02:49:12.502] [depthai] [info] Logging disabled
[19443010B165705A00] [10.4.8.125] [11.302] [Camera(4)] [debug] Sensor type COLOR selected automatically for camera on socket CAM_C. Officially supported sensor types: [COLOR]
[19443010B165705A00] [10.4.8.125] [11.302] [Camera(4)] [debug] Full FoV configs:
[19443010B165705A00] [10.4.8.125] [11.302] [Camera(4)] [debug]   1280x800 @ 129.6fps
[19443010B165705A00] [10.4.8.125] [11.302] [Camera(4)] [debug] Output requests: Request 0: size: 1280x800, fps: 30, type: 8, resizeMode: CROP; Request 1: size: 416x416, fps: 30, type: 8, resizeMode: LETTERBOX
[19443010B165705A00] [10.4.8.125] [11.302] [Camera(4)] [debug] Best camera config selected: 1280x800 @ 30fps
[19443010B165705A00] [10.4.8.125] [11.302] [ImageAlign(10)] [info] Init node ImageAlign
[19443010B165705A00] [10.4.8.125] [11.302] [DetectionParser(8)] [info] Init node DetectionParser
[19443010B165705A00] [10.4.8.125] [11.304] [ToF(1)] [debug] Calculated number of shaves: 2 for ToF fps: 30 and empiric runtime: 48ms

jakaskerl

MichaelKarlesky
It's referring to the camera. Looking forward to the MRE. Might be a simple fix, but I'd need to check myself.

Thanks,
Jaka

MichaelKarlesky

To create an MRE I used an AI tool to convert the early version of our ROS2 node into a single-file Python application. The Python file is at bottom. I performed a small amount of cleanup on the repackaged code, but I haven't been able to fully exercise this MRE because of the camera device memory failure.

An example of running the script and its logging output is below.

> ./human_detector_app.py 10.4.8.125
2025-09-15 20:25:21,423 [INFO]: Starting Human Detector Application...
2025-09-15 20:25:21,423 [INFO]: Human Detector created
2025-09-15 20:25:21,423 [INFO]: Configuring human detector...
2025-09-15 20:25:21,423 [INFO]: Setting up for OAK camera at 10.4.8.125...
2025-09-15 20:25:29,135 [INFO]: Camera found and set up successfully
2025-09-15 20:25:29,136 [INFO]: Creating DepthAI pipeline...
2025-09-15 20:25:34,412 [INFO]: Pipeline created successfully
2025-09-15 20:25:34,412 [INFO]: Human detector configured successfully
2025-09-15 20:25:34,412 [INFO]: Activating human detector...
2025-09-15 20:25:34,412 [INFO]: Starting DepthAI pipeline...
2025-09-15 20:25:34,947 [ERROR]: Failed to activate human detector: ImageManip out of memory; tried to allocate '225472'B out of '142336'B available
2025-09-15 20:25:34,947 [ERROR]: Fatal error in human detector application: ImageManip out of memory; tried to allocate '225472'B out of '142336'B available
2025-09-15 20:25:34,947 [INFO]: Deactivating human detector...
2025-09-15 20:25:34,947 [INFO]: Human detector deactivated successfully
2025-09-15 20:25:34,948 [INFO]: Cleaning up human detector...
2025-09-15 20:25:34,970 [INFO]: Human detector cleaned up successfully
2025-09-15 20:25:34,970 [INFO]: Human Detector Application stopped

#!/usr/bin/env python3

from typing import Optional
import threading
import time
import logging
import argparse
import sys
from datetime import timedelta

import numpy as np

# Import DepthAI components
import depthai as dai

class HumanDetector:

    DEFAULT_FPS = 30

    DEFAULT_CONFIDENCE_THRESHOLD = 0.8

    # Default detection thresholds (in mm)
    DEFAULT_FAULT_THRESHOLD_MM =  500
    DEFAULT_PAUSE_THRESHOLD_MM = 1000
    
    def __init__(self, 
                 logger,
                 camera_ip: str, fps: int = DEFAULT_FPS, 
                 confidence_threshold: float = DEFAULT_CONFIDENCE_THRESHOLD,
                 fault_threshold_mm: float = DEFAULT_FAULT_THRESHOLD_MM,
                 pause_threshold_mm: float = DEFAULT_PAUSE_THRESHOLD_MM,
                 verbose_logging: bool = False):
        
        # Setup logging
        self.logger = logger
        
        # Store configuration
        self._camera_ip = camera_ip
        self._fps = fps
        self._confidence_threshold = confidence_threshold
        self._fault_threshold_mm = fault_threshold_mm
        self._pause_threshold_mm = pause_threshold_mm
        self._verbose_logging = verbose_logging
        
        # DepthAI components
        self._device: Optional[dai.Device] = None
        self._pipeline: Optional[dai.Pipeline] = None
        self._detection_queue: Optional[dai.DataOutputQueue] = None
        self._sync_queue: Optional[dai.DataOutputQueue] = None
        self._detection_network = None
        
        # Detection state
        self._detection_thread: Optional[threading.Thread] = None
        self._detection_running = False
        self._detection_lock = threading.Lock()
        
        self.logger.info("Human Detector created")

    def configure(self):
        """Configure the human detector"""
        self.logger.info("Configuring human detector...")
        
        try:
            self.logger.info(f"Setting up for OAK camera at {self._camera_ip}...")
            self._device = self._create_depthai_device(self._camera_ip)
            self.logger.info("Camera found and set up successfully")

            self.logger.info("Creating DepthAI pipeline...")
            self._pipeline = self._create_depthai_pipeline(self._device)
            self.logger.info("Pipeline created successfully")
                        
        except Exception as ex:
            self.logger.error(f"Failed to configure human detector: {type(ex).__name__}: {ex}")
            raise

        self.logger.info("Human detector configured successfully")

    def cleanup(self):
        """Cleanup the human detector"""
        self.logger.info("Cleaning up human detector...")
        
        try:
            self._cleanup_depthai_resources()
            self.logger.info("Human detector cleaned up successfully")
            
        except Exception as e:
            self.logger.error(f"Failed to cleanup human detector: {e}")
            raise

    def activate(self):
        """Activate the human detector"""
        self.logger.info("Activating human detector...")
        
        try:
            # Connect to DepthAI device and start pipeline
            self._start_detection()
            
            self.logger.info("Human detector activated successfully")
            
        except Exception as e:
            self.logger.error(f"Failed to activate human detector: {e}")
            raise

    def deactivate(self):
        """Deactivate the human detector"""
        self.logger.info("Deactivating human detector...")
        
        try:
            self._stop_detection()
            self.logger.info("Human detector deactivated successfully")
            
        except Exception as e:
            self.logger.error(f"Failed to deactivate human detector: {e}")
            raise

    def _create_depthai_device(self, camera_ip: str) -> dai.Device:
        info = dai.DeviceInfo(camera_ip)
        return dai.Device(info)

    def _create_depthai_pipeline(self, device: dai.Device) -> dai.Pipeline:
        """Create and configure the DepthAI pipeline"""
        pipeline = dai.Pipeline(device)

        # Set up ToF sensor
        tof = pipeline.create(dai.node.ToF).build()
        
        # Set up RGB Camera
        camera_node = pipeline.create(dai.node.Camera).build(
            boardSocket=dai.CameraBoardSocket.CAM_C, # RGB_SOCKET
            sensorFps=self._fps
        )
        camera_full = camera_node.requestFullResolutionOutput(type=dai.ImgFrame.Type.BGR888p)
        camera_nn = camera_node.requestOutput(
            size=(416, 416),
            resizeMode=dai.ImgResizeMode.LETTERBOX,
            type=dai.ImgFrame.Type.BGR888p
        )

        # Set up image processing for ToF alignment
        tof_resize = pipeline.create(dai.node.ImageManip)
        tof_resize.initialConfig.setOutputSize(416, 416, dai.ImageManipConfig.ResizeMode.LETTERBOX)

        # Set up detection network
        self._detection_network = pipeline.create(dai.node.DetectionNetwork)
        self._detection_network.setFromModelZoo(
            dai.NNModelDescription(model="yolov6-nano", platform="RVC2"),
            useCached=True
        )

        # Create sync and align nodes
        sync = pipeline.create(dai.node.Sync)
        sync.setSyncThreshold(timedelta(seconds=0.5 / self._fps))
        align = pipeline.create(dai.node.ImageAlign)

        # Linking
        camera_nn.link(self._detection_network.input)
        tof.depth.link(align.input)
        camera_full.link(align.inputAlignTo)
        align.outputAligned.link(tof_resize.inputImage)
        
        self._detection_network.passthrough.link(sync.inputs["rgb"])
        tof_resize.out.link(sync.inputs["depth_aligned"])
        sync.inputs["rgb"].setBlocking(False)

        # Create output queues
        self._detection_queue = self._detection_network.out.createOutputQueue()
        self._sync_queue = sync.out.createOutputQueue()

        return pipeline

    def _start_detection(self):
        """Start the DepthAI detection process"""
        with self._detection_lock:
            if self._detection_running:
                return
                
            # Start pipeline
            self.logger.info("Starting DepthAI pipeline...")
            self._pipeline.start()
            
            # Start detection thread
            self._detection_running = True
            self._detection_thread = threading.Thread(target=self._detection_loop, daemon=True)
            self._detection_thread.start()
            
            self.logger.info("DepthAI detection started")

    def _stop_detection(self):
        """Stop the DepthAI detection process"""
        with self._detection_lock:
            if not self._detection_running:
                return
                
            self._detection_running = False
            
            if self._detection_thread and self._detection_thread.is_alive():
                self._detection_thread.join(timeout=2.0)
                
            self.logger.info("DepthAI detection stopped")

    def _cleanup_depthai_resources(self):
        """Clean up DepthAI resources"""
        self._stop_detection()

        if self._device:
            self._device.close()
            self._device = None

        if self._pipeline:
            self._pipeline.stop()
            self._pipeline = None

        self._detection_queue = None
        self._sync_queue = None
        self._detection_network = None

    def _detection_loop(self) -> None:
        """Main detection processing loop running in separate thread"""
        self.logger.info("Human detection loop starting...")
        
        try:
            label_map = self._detection_network.getClasses()
        except Exception as e:
            self.logger.error(f"Fatal error starting detection loop: {e}")
            return

        while self._detection_running:
            try:
                # Get detections and synchronized depth data
                input_detections = self._detection_queue.get()
                message_group = self._sync_queue.get()
                
                if input_detections is None:
                    continue

                detections = input_detections.detections
                
                # Get the aligned depth frame
                frame_depth = message_group["depth_aligned"]

                if frame_depth is None:
                    continue

                depth_array = frame_depth.getFrame()
                frame_shape = depth_array.shape
                
                # Process detections
                alert_level = None
                
                # Process each detection
                for detection in detections:
                    # Only process person detections
                    if label_map[detection.label] != 'person':
                        continue

                    if self._verbose_logging:
                        self.logger.info(f"Person detected with confidence: {detection.confidence:.3f}")

                    # Only process person detections with sufficient confidence
                    if detection.confidence < self._confidence_threshold:
                        continue

                    try:
                        # Convert normalized bbox to pixel coordinates
                        bbox = self._frame_norm(
                            frame_shape, 
                            (
                                detection.xmin, detection.ymin, 
                                detection.xmax, detection.ymax
                            )
                        )
                        
                        # Calculate median depth in the bounding box ROI
                        median_depth_mm = self._median_depth_roi(depth_array, bbox)                                    

                        # Only proceed if meaningful depth data
                        if median_depth_mm is None:
                            if self._verbose_logging:
                                self.logger.warning("Person detected but no valid depth data in ROI")
                            continue

                        # Convert to meters for readability
                        distance_m = median_depth_mm / 1000.0
                                                                    
                        # Determine alert level and log accordingly
                        if median_depth_mm < self._fault_threshold_mm and alert_level != "CRITICAL":
                            alert_level = "CRITICAL"
                            self.logger.error(f"Person very close - {distance_m:.2f}m")
                        elif median_depth_mm < self._pause_threshold_mm and alert_level != "WARNING":
                            alert_level = "WARNING"
                            self.logger.warning(f"Person approaching - {distance_m:.2f}m")
                        elif alert_level != "SAFE":
                            alert_level = "SAFE"
                            self.logger.info(f"Person at safe distance - {distance_m:.2f}m")

                        if self._verbose_logging:
                            self.logger.info(f"Person detected at distance {distance_m:.2f}m with confidence: {detection.confidence:.3f}")
                                                                    
                    except Exception as e:
                        self.logger.error(f"Error processing person detection but trying again: {e}")
                                
            except Exception as e:
                self.logger.error(f"Error in detection loop: {e}")

        self.logger.info("Detection loop ended")

    def _median_depth_roi(self, depth_map: np.ndarray, bbox: tuple, ignore_zero: bool = True) -> Optional[float]:
        """
        Compute the median depth within a rectangular ROI of a uint16 depth map.
        
        Parameters
        ----------
        depth_map : np.ndarray
            2D array of shape (H, W), dtype=np.uint16.
        bbox : tuple
            Bounding box as (x1, y1, x2, y2).
        ignore_zero : bool, default=True
            If True, zero-valued pixels (often "no data") will be excluded.
            
        Returns
        -------
        float or None
            Median depth value in mm, or None if the ROI is empty or no valid pixels.
        """
        h, w = depth_map.shape[:2]
        
        x1, y1, x2, y2 = bbox
        
        # Clip to image bounds
        x1 = max(0, min(x1, w))
        y1 = max(0, min(y1, h))
        x2 = max(0, min(x2, w))
        y2 = max(0, min(y2, h))
        
        # Extract ROI
        roi = depth_map[y1:y2, x1:x2]
        if roi.size == 0:
            return None
        
        # Optionally filter out zero values
        if ignore_zero:
            roi = roi[roi > 0]
            if roi.size == 0:
                return None
        
        # Compute and return median
        return float(np.median(roi))

    def _frame_norm(self, frame_shape, bbox):
        """Convert normalized bbox coordinates to pixel coordinates"""
        h, w = frame_shape[:2]
        x1 = int(bbox[0] * w)
        y1 = int(bbox[1] * h) 
        x2 = int(bbox[2] * w)
        y2 = int(bbox[3] * h)
        return (x1, y1, x2, y2)

def setup_logging(verbose: bool = False):
    """Setup logging configuration"""
    log_level = logging.DEBUG if verbose else logging.INFO
    logging.basicConfig(
        level=log_level,
        format='%(asctime)s [%(levelname)s]: %(message)s',
        handlers=[
            logging.StreamHandler(sys.stdout)
        ]
    )


def parse_arguments():
    """Parse command line arguments"""
    parser = argparse.ArgumentParser(description='Human Detection Application')
    
    parser.add_argument('camera_ip', 
                       help='IP address of the OAK camera')
    
    parser.add_argument('--fps', 
                       type=int, 
                       default=HumanDetector.DEFAULT_FPS,
                       help=f'Frame rate for the detection pipeline (default: {HumanDetector.DEFAULT_FPS})')
    
    parser.add_argument('--confidence-threshold', 
                       type=float, 
                       default=HumanDetector.DEFAULT_CONFIDENCE_THRESHOLD,
                       help=f'Person detection confidence threshold 0.0-1.0 (default: {HumanDetector.DEFAULT_CONFIDENCE_THRESHOLD})')
    
    parser.add_argument('--fault-threshold-mm', 
                       type=float, 
                       default=HumanDetector.DEFAULT_FAULT_THRESHOLD_MM,
                       help=f'Distance at which a person\'s approach is considered a fault condition (default: {HumanDetector.DEFAULT_FAULT_THRESHOLD_MM}mm)')
    
    parser.add_argument('--pause-threshold-mm', 
                       type=float, 
                       default=HumanDetector.DEFAULT_PAUSE_THRESHOLD_MM,
                       help=f'Distance at which a person\'s approach is considered a pause condition (default: {HumanDetector.DEFAULT_PAUSE_THRESHOLD_MM}mm)')
    
    parser.add_argument('--verbose', 
                       action='store_true',
                       help='Enable verbose logging')
    
    return parser.parse_args()


def main():
    """Main entry point for the human detector application"""
    args = parse_arguments()
    
    # Setup logging
    setup_logging(args.verbose)
    logger = logging.getLogger('HumanDetector')
    
    logger.info("Starting Human Detector Application...")
    
    detector = None
    
    try:
        # Create detector instance
        detector = HumanDetector(
            logger,
            camera_ip=args.camera_ip,
            fps=args.fps,
            confidence_threshold=args.confidence_threshold,
            fault_threshold_mm=args.fault_threshold_mm,
            pause_threshold_mm=args.pause_threshold_mm,
            verbose_logging=args.verbose
        )
        
        # Execute lifecycle stages in order
        detector.configure()
        detector.activate()
        
        logger.info("Human detector is running. Press Ctrl+C to stop...")
        
        # Keep the application running
        try:
            while True:
                time.sleep(1.0)
        except KeyboardInterrupt:
            logger.info("Received shutdown signal...")
            
    except Exception as e:
        logger.error(f"Fatal error in human detector application: {e}")
        sys.exit(1)
        
    finally:
        if detector:
            try:
                detector.deactivate()
                detector.cleanup()
            except Exception as e:
                logger.error(f"Error during cleanup: {e}")
        
        logger.info("Human Detector Application stopped")


if __name__ == "__main__":
    main()

klabs2025

I'm working on this project as well and am able to recreate the ImageManip out of memory; tried to allocate '225472'B out of '142336'B available error as well, but only when running with 3.0.0. This error did not show up for me when using 3.0.0rc3 or 3.0.0rc4.

From the changelogs, it looks like there were some recent modifications to ImageManip that might be causing this

jakaskerl

klabs2025
Ok, thanks for reporting. I can repro this so we'll be able to see what the issue is.

Thanks,
Jaka

MichaelKarlesky

Curious report from klabs2025. The ImageManip out-of-memory error I have been seeing is with 3.0.0rc4 whereas Jeff did not see the error with 3.0.0rc4.

The communication dropouts and device crash reports first reported (beginning of this thread) were all happening with 3.0.0rc4 as well.

For reasons unknown, the ImageManip out-of-memory error started consistently happening for me without any other code changes. That is, the code that was logging human detection events with occasional communication dropouts and device crash reports no longer starts up at all because of the out-of-memory error.

So glad to hear that you've been able to reproduce the reported issue, jakaskerl! I hope that means we're on our way to resolving these stability problems.