Hello

My problem is the following:

I'm using the oak-d pro w to record camera data segmentwise. what i need are 5 minute video segments for e time period of about 10 hours, To do so i wrote this code:

def record_segment(self) -> str:
	with OakCamera() as camera:
		camera.device
		color = camera.create_camera(
			"color", resolution="1080P", fps=20, encode="H265"
		)
		stereo = camera.create_stereo(resolution="800p", fps=20, encode="H264")
		stereo.config_wls(wls_level="medium")
		outputs = [color.out.encoded, stereo.out.disparity]
		record = camera.record(
			outputs,
			str(get_output_path() / self.subfolder),
			record_type=RecordType.VIDEO,
		) 

		# create pipline for camera sensors and imu sensor
		imu = camera.pipeline.create(dai.node.IMU)
		xlinkOut = camera.pipeline.create(dai.node.XLinkOut)
		xlinkOut.setStreamName("imu")
		imu.enableIMUSensor(
			[dai.IMUSensor.ACCELEROMETER_RAW, dai.IMUSensor.GYROSCOPE_RAW], 100
		)
		imu.setBatchReportThreshold(1)
		imu.setMaxBatchReports(10)
		imu.out.link(xlinkOut.input)

		camera.start()

		imu_out = camera.device.getOutputQueue(
			name="imu", maxSize=50, blocking=False
		)

		start_time = time.monotonic()

		# running until out of time
		with open("imu.csv", "a", newline="") as file:
			writer = csv.writer(file)
			while camera.running():
				# get imu data during the camera is running
				self.write_imu_data(imu_out.get().packets, writer)
				if not self.record_condition(time.monotonic() - start_time):
					camera.device.close()
				camera.poll()

This function is called in a loop and records one segment. This all seems to work for some time but unfortunatly it breaks after one or two hours.

The error i get is X_LINK_DEVICE_NOT_FOUND. Even if i try to look up usb devices in linux (lsusb) i cant find the device. But if i reconnect it (plug in / plug out), it is found again.

In journalctl i see the following:

Jul 17 15:27:31 sensorbox python[844]: [2024-07-17 15:27:31] INFO [root.__exit__:408] Closing OAK camera
Jul 17 15:27:31 sensorbox python[844]: [2024-07-17 15:27:31] INFO [root.close:103] Video Recorder saved stream(s) to folder: /home/admin/media/usb/camera/41-194430105126A12E00
Jul 17 15:27:31 sensorbox python[844]: encoded 0 frames
Jul 17 15:27:31 sensorbox python[844]: 2024-07-17 15:27:31,619 - recorder.py:49 - INFO - Stop writing data segment
Jul 17 15:27:31 sensorbox python[844]: 2024-07-17 15:27:31,619 - recorder.py:41 - INFO - Start writing data segment
Jul 17 15:27:33 sensorbox python[844]: [2024-07-17 15:27:33] INFO [root._run:34] Exiting store frame thread
Jul 17 15:27:33 sensorbox kernel: usb 1-1: USB disconnect, device number 44
Jul 17 15:27:42 sensorbox kernel: usb 1-1: new high-speed USB device number 45 using xhci-hcd
Jul 17 15:27:42 sensorbox kernel: usb 1-1: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
Jul 17 15:27:42 sensorbox kernel: usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jul 17 15:27:42 sensorbox kernel: usb 1-1: Product: Movidius MyriadX
Jul 17 15:27:42 sensorbox kernel: usb 1-1: Manufacturer: Movidius Ltd.
Jul 17 15:27:42 sensorbox kernel: usb 1-1: SerialNumber: 03e72485
Jul 17 15:27:42 sensorbox mtp-probe[4556]: checking bus 1, device 45: "/sys/devices/platform/axi/1000120000.pcie/1f00200000.usb/xhci-hcd.0/usb1/1-1"
Jul 17 15:27:42 sensorbox mtp-probe[4556]: bus: 1, device: 45 was not an MTP device
Jul 17 15:27:42 sensorbox mtp-probe[4557]: checking bus 1, device 45: "/sys/devices/platform/axi/1000120000.pcie/1f00200000.usb/xhci-hcd.0/usb1/1-1"
Jul 17 15:27:42 sensorbox mtp-probe[4557]: bus: 1, device: 45 was not an MTP device
Jul 17 15:27:48 sensorbox python[844]: 2024-07-17 15:27:48,084 - recorder.py:51 - ERROR - An error occured while recording
Jul 17 15:27:48 sensorbox python[844]: Traceback (most recent call last):
Jul 17 15:27:48 sensorbox python[844]:   File "/home/admin/src/app/recorder.py", line 43, in record
Jul 17 15:27:48 sensorbox python[844]:     name = self.record_segment()
Jul 17 15:27:48 sensorbox python[844]:   File "/home/admin/src/app/camera/camera_recorder.py", line 28, in record_segment
Jul 17 15:27:48 sensorbox python[844]:     with OakCamera() as camera:
Jul 17 15:27:48 sensorbox python[844]:   File "/home/admin/environment/lib/python3.9/site-packages/depthai_sdk/oak_camera.py", line 95, in __init__
Jul 17 15:27:48 sensorbox python[844]:     self._init_device(config, device)
Jul 17 15:27:48 sensorbox python[844]:   File "/home/admin/environment/lib/python3.9/site-packages/depthai_sdk/oak_camera.py", line 378, in _init_device
Jul 17 15:27:48 sensorbox python[844]:     self._oak.device = dai.Device(
Jul 17 15:27:48 sensorbox python[844]: RuntimeError: Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND

Does anyone know what could be the problem that the device is not recognized after somne time? Thank you in advance!

    Hi jgfhnw
    Could you run the script with env variable DEPTHAI_DEBUG=1? Also a dmesg log at the time the device is disconnected. Are you experiencing high cpu usage during writing?

    Thanks,
    Jaka

      jakaskerl

      Thank you for your reply., Of course, i will do that. Unfortunately i'm not able to access the device till end of next week. But i will update this thread as soon as i can.

      5 days later

      Ok. So i could make the tests. The camera disconnected after 30 min recording time at 9:26:42

      CPU Usage (uptime):

      09:25:56 up 26 min, 0 user, load average: 0.86, 0.91, 0.76
      09:26:06 up 26 min, 0 user, load average: 0.88, 0.91, 0.76
      09:26:16 up 26 min, 0 user, load average: 0.74, 0.88, 0.75
      09:26:26 up 26 min, 0 user, load average: 0.63, 0.85, 0.74
      09:26:36 up 27 min, 0 user, load average: 0.53, 0.82, 0.73
      09:26:46 up 27 min, 0 user, load average: 0.60, 0.83, 0.73
      09:26:56 up 27 min, 0 user, load average: 0.50, 0.80, 0.73
      09:27:06 up 27 min, 0 user, load average: 0.66, 0.83, 0.73

      dmesg: (error occured about 1617 after the start)

      [   94.059690] pl011-axi 1f00030000.serial: DMA channel RX dma2chan1
      [   95.520686] usb 1-1: USB disconnect, device number 2
      [   96.371581] usb 2-1: new SuperSpeed USB device number 2 using xhci-hcd
      [   96.392524] usb 2-1: New USB device found, idVendor=03e7, idProduct=f63b, bcdDevice= 1.00
      [   96.392529] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
      [   96.392531] usb 2-1: Product: Luxonis Device
      [   96.392534] usb 2-1: Manufacturer: Intel Corporation
      [   96.392536] usb 2-1: SerialNumber: 194430105126A12E00
      [  396.990871] usb 2-1: USB disconnect, device number 2
      [  397.447500] usb 1-1: new high-speed USB device number 4 using xhci-hcd
      [  397.595827] usb 1-1: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
      [  397.595830] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
      [  397.595832] usb 1-1: Product: Movidius MyriadX
      [  397.595833] usb 1-1: Manufacturer: Movidius Ltd.
      [  397.595835] usb 1-1: SerialNumber: 03e72485
      [  398.515465] retire_capture_urb: 1 callbacks suppressed
      [  399.485173] usb 1-1: USB disconnect, device number 4
      [  400.327572] usb 2-1: new SuperSpeed USB device number 3 using xhci-hcd
      [  400.348470] usb 2-1: New USB device found, idVendor=03e7, idProduct=f63b, bcdDevice= 1.00
      [  400.348473] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
      [  400.348475] usb 2-1: Product: Luxonis Device
      [  400.348477] usb 2-1: Manufacturer: Intel Corporation
      [  400.348478] usb 2-1: SerialNumber: 194430105126A12E00
      [  700.943484] usb 2-1: USB disconnect, device number 3
      [  701.399501] usb 1-1: new high-speed USB device number 5 using xhci-hcd
      [  701.547827] usb 1-1: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
      [  701.547831] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
      [  701.547833] usb 1-1: Product: Movidius MyriadX
      [  701.547834] usb 1-1: Manufacturer: Movidius Ltd.
      [  701.547836] usb 1-1: SerialNumber: 03e72485
      [  702.467604] retire_capture_urb: 1 callbacks suppressed
      [  703.438231] usb 1-1: USB disconnect, device number 5
      [  704.279576] usb 2-1: new SuperSpeed USB device number 4 using xhci-hcd
      [  704.300512] usb 2-1: New USB device found, idVendor=03e7, idProduct=f63b, bcdDevice= 1.00
      [  704.300516] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
      [  704.300519] usb 2-1: Product: Luxonis Device
      [  704.300522] usb 2-1: Manufacturer: Intel Corporation
      [  704.300524] usb 2-1: SerialNumber: 194430105126A12E00
      [ 1004.890138] usb 2-1: USB disconnect, device number 4
      [ 1005.343500] usb 1-1: new high-speed USB device number 6 using xhci-hcd
      [ 1005.491816] usb 1-1: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
      [ 1005.491822] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
      [ 1005.491825] usb 1-1: Product: Movidius MyriadX
      [ 1005.491828] usb 1-1: Manufacturer: Movidius Ltd.
      [ 1005.491830] usb 1-1: SerialNumber: 03e72485
      [ 1006.427796] retire_capture_urb: 1 callbacks suppressed
      [ 1007.389139] usb 1-1: USB disconnect, device number 6
      [ 1008.239568] usb 2-1: new SuperSpeed USB device number 5 using xhci-hcd
      [ 1008.260482] usb 2-1: New USB device found, idVendor=03e7, idProduct=f63b, bcdDevice= 1.00
      [ 1008.260485] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
      [ 1008.260487] usb 2-1: Product: Luxonis Device
      [ 1008.260489] usb 2-1: Manufacturer: Intel Corporation
      [ 1008.260490] usb 2-1: SerialNumber: 194430105126A12E00
      [ 1308.854992] usb 2-1: USB disconnect, device number 5
      [ 1309.307507] usb 1-1: new high-speed USB device number 7 using xhci-hcd
      [ 1309.455819] usb 1-1: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
      [ 1309.455824] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
      [ 1309.455826] usb 1-1: Product: Movidius MyriadX
      [ 1309.455828] usb 1-1: Manufacturer: Movidius Ltd.
      [ 1309.455829] usb 1-1: SerialNumber: 03e72485
      [ 1310.288043] retire_capture_urb: 1 callbacks suppressed
      [ 1311.251026] usb 1-1: USB disconnect, device number 7
      [ 1312.099573] usb 2-1: new SuperSpeed USB device number 6 using xhci-hcd
      [ 1312.120470] usb 2-1: New USB device found, idVendor=03e7, idProduct=f63b, bcdDevice= 1.00
      [ 1312.120474] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
      [ 1312.120476] usb 2-1: Product: Luxonis Device
      [ 1312.120478] usb 2-1: Manufacturer: Intel Corporation
      [ 1312.120479] usb 2-1: SerialNumber: 194430105126A12E00
      [ 1612.708029] usb 2-1: USB disconnect, device number 6
      [ 1613.163497] usb 1-1: new high-speed USB device number 8 using xhci-hcd
      [ 1613.311822] usb 1-1: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
      [ 1613.311827] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
      [ 1613.311829] usb 1-1: Product: Movidius MyriadX
      [ 1613.311831] usb 1-1: Manufacturer: Movidius Ltd.
      [ 1613.311832] usb 1-1: SerialNumber: 03e72485
      [ 1614.155204] retire_capture_urb: 1 callbacks suppressed
      [ 1615.100637] usb 1-1: USB disconnect, device number 8
      [ 1624.487498] usb 1-1: new high-speed USB device number 9 using xhci-hcd
      [ 1624.635828] usb 1-1: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
      [ 1624.635837] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
      [ 1624.635840] usb 1-1: Product: Movidius MyriadX
      [ 1624.635842] usb 1-1: Manufacturer: Movidius Ltd.
      [ 1624.635845] usb 1-1: SerialNumber: 03e72485
      [ 1630.721987] retire_capture_urb: 1 callbacks suppressed
      [ 1631.661142] usb 1-1: USB disconnect, device number 9
      [ 1641.047498] usb 1-1: new high-speed USB device number 10 using xhci-hcd
      [ 1641.195826] usb 1-1: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
      [ 1641.195831] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
      [ 1641.195834] usb 1-1: Product: Movidius MyriadX
      [ 1641.195836] usb 1-1: Manufacturer: Movidius Ltd.
      [ 1641.195839] usb 1-1: SerialNumber: 03e72485
      [ 1648.116046] usb 1-1: USB disconnect, device number 10
      [ 1657.503500] usb 1-1: new high-speed USB device number 11 using xhci-hcd
      [ 1657.651827] usb 1-1: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
      [ 1657.651833] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
      [ 1657.651836] usb 1-1: Product: Movidius MyriadX
      [ 1657.651839] usb 1-1: Manufacturer: Movidius Ltd.
      [ 1657.651841] usb 1-1: SerialNumber: 03e72485
      [ 1664.579007] usb 1-1: USB disconnect, device number 11
      [ 1673.963499] usb 1-1: new high-speed USB device number 12 using xhci-hcd
      [ 1674.111826] usb 1-1: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
      [ 1674.111831] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
      [ 1674.111834] usb 1-1: Product: Movidius MyriadX
      [ 1674.111836] usb 1-1: Manufacturer: Movidius Ltd.
      [ 1674.111838] usb 1-1: SerialNumber: 03e72485
      [ 1680.103284] retire_capture_urb: 1 callbacks suppressed
      [ 1681.043646] usb 1-1: USB disconnect, device number 12
      [ 1690.431498] usb 1-1: new high-speed USB device number 13 using xhci-hcd
      [ 1690.579820] usb 1-1: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
      [ 1690.579826] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
      [ 1690.579829] usb 1-1: Product: Movidius MyriadX
      [ 1690.579831] usb 1-1: Manufacturer: Movidius Ltd.
      [ 1690.579833] usb 1-1: SerialNumber: 03e72485
      [ 1696.562054] retire_capture_urb: 1 callbacks suppressed

      Error 1:

      2024-07-20 09:26:42,699 - connectionpool.py:870 - WARNING - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NameResolutionError(": Failed to resolve 'sentry.luxonis.com' ([Errno -3] Temporary failure in name resolution)")': /api/3/envelope/

      Error 2:

      2024-07-20 09:26:25,620 - recorder.py:44 - INFO - CameraRecorder > Start writing data segment
      2024-07-20 09:26:42,085 - recorder.py:52 - ERROR - CameraRecorder > An error occured while recording
      Traceback (most recent call last):
      File "/home/admin/src/app/recorder.py", line 46, in record
      name = self.record_segment()
      File "/home/admin/src/app/camera/camera_recorder.py", line 29, in record_segment
      with OakCamera() as camera:
      File "/home/admin/environment/lib/python3.9/site-packages/depthai_sdk/oak_camera.py", line 95, in __init__
      self._init_device(config, device)
      File "/home/admin/environment/lib/python3.9/site-packages/depthai_sdk/oak_camera.py", line 378, in _init_device
      self._oak.device = dai.Device(
      RuntimeError: Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND

      So cpu usage doesn't seem too high at point of disconnection.

      What i forgot to mention is that the setup was build two times. The other setup works fine without connection failures.

      Thank you in advance!

      I use depthai-sdk version 1.12.1 and this is the bootloader version:

      Found device with name: 1.6
      Version: 0.0.26
      USB Bootloader - supports only Flash memory
      Memory 'Memory.FLASH' size: 67108864, info: JEDEC ID: C2 25 3A

      @jgfhnw latest bootloader is 0.0.28, and latest sdk is 1.15. Please update both first.

      The bootloader and depthai-sdk are updated. But i can't run the script because it fails with the new version of depthai-sdk:

      Exception has occurred: AttributeError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
      module 'depthai.node' has no attribute 'ImageAlign'
        File "C:\_Data\Enviroments\39\rsrg_data_acquisition\Lib\site-packages\depthai_sdk\components\tof_component.py", line 16, in ToFComponent
          _align: Optional[dai.node.ImageAlign] = None
        File "C:\_Data\Enviroments\39\rsrg_data_acquisition\Lib\site-packages\depthai_sdk\components\tof_component.py", line 14, in <module>
          class ToFComponent(Component):
        File "C:\_Data\Enviroments\39\rsrg_data_acquisition\Lib\site-packages\depthai_sdk\oak_camera.py", line 33, in <module>
          from depthai_sdk.components.tof_component import ToFComponent
        File "C:\_Data\Enviroments\39\rsrg_data_acquisition\Lib\site-packages\depthai_sdk\__init__.py", line 5, in <module>
          from depthai_sdk.oak_camera import OakCamera
      AttributeError: module 'depthai.node' has no attribute 'ImageAlign'

      Ok, i removed the type hint that caused the error and it runs now. I give an update tomorrow. You should consider fixing that in the library.

      Are you using the latest depthai version (2.27)?

      No it didn't update the depthai version when i updated the depthai-sdk. Theres no minimum constraint in the requirements.txt of depthai-sdk for depthai. I updated it now manually and this problem seems solved.

      But if i now try to start the camera i instantly get this error:

      $$
      Jul 26 11:32:02 sensorbox python[844]: File "/usr/lib/python3.9/threading.py", line 950, in bootstrap_inner
      Jul 26 11:32:02 sensorbox python[844]: self.run()
      Jul 26 11:32:02 sensorbox python[844]: File "/home/admin/environment/lib/python3.9/site-packages/sentry_sdk/integrations/threading.py", line 69, in run
      Jul 26 11:32:02 sensorbox python[844]: reraise(*
      capture_exception())
      Jul 26 11:32:02 sensorbox python[844]: File "/home/admin/environment/lib/python3.9/site-packages/sentry_sdk/compat.py", line 60, in reraise
      Jul 26 11:32:02 sensorbox python[844]: raise value
      Jul 26 11:32:02 sensorbox python[844]: File "/home/admin/environment/lib/python3.9/site-packages/sentry_sdk/integrations/threading.py", line 67, in run
      Jul 26 11:32:02 sensorbox python[844]: return old_run_func(self, a, **kw)
      Jul 26 11:32:02 sensorbox python[844]: File "/usr/lib/python3.9/threading.py", line 888, in run
      Jul 26 11:32:02 sensorbox python[844]: self.
      target(self.args, **self.kwargs)
      Jul 26 11:32:02 sensorbox python[844]: File "/home/admin/environment/lib/python3.9/site-packages/depthai_sdk/record.py", line 27, in run
      Jul 26 11:32:02 sensorbox python[844]: recorder.write(name, frames[name])
      Jul 26 11:32:02 sensorbox python[844]: File "/home/admin/environment/lib/python3.9/site-packages/depthai_sdk/recorders/video_recorder.py", line 97, in write
      Jul 26 11:32:02 sensorbox python[844]: self.
      writers[name].write(frame)
      Jul 26 11:32:02 sensorbox python[844]: KeyError: '7_disparity;5_out'
      $$

      • erik replied to this.
        14 days later

        fyi. I reworked the code and used depthai instead of depthai-sdk. Directly using depthai allows me to record segementwise without having to open and close the camera everytime a new segment starts as in the code above. It seems that doing that is causing the error.