jakaskerl

Thanks for your assistance

mikegardner
Can you give more info on OS you are using?

I'm using Ubuntu 22.04.5 LTS

mikegardner I used the device manager using the link on the docs page to try to update the firmware using the Danger Zone 'Update Bootloader' and 'Flash Factory Bootloader' options but the 'Flashed bootloader version' always stays at 0.0.21

This would suggest you are using an older version of depthai or that the flashing failed (in which case you would get an error).

I suspected I was using an older version of depthai, so I deleted the depthai, depthai-python, and depthai-core folders and cloned them again from the master branch on the current repo, ran submodule update --init --recursive on each of those folders plus run install_requirements git but still got the same results.

mikegardner I'm up to date with the depthai repo but no idea where the bootloader file is, it's not in the utilities directory with the device_manager.py module

But it should be. luxonis/depthai-pythontree/main/utilities

Looking at the repo at the link you suggested, it matches my local folder contents exactly. As i say, I can see the device_manager.py module but nothing that looks like the actual bootloader file that is installed.

mikegardner If I ping the camera I get on average 10-15ms response with no missed beats

But this would indicate a network issue. What does your setup look like?

I have a Netgear Prosafe 108P 8-port switch with 4 POE ports. The only POE port that is populated is the one for the camera. The switch is connected to my Wifi router and I use a a Thinkpad T440s running Unbuntu 22.04.5 LTS to attach to the camera.

The laptop is dedicated as a Depthai dev system

Thanks,
Jaka

    mikegardner
    Here's a sample of the ping responses, IMHO they don't suggest a network issue...

    PING 192.168.1.133 (192.168.1.133) 56(84) bytes of data.
    64 bytes from 192.168.1.133: icmp_seq=1 ttl=64 time=241 ms
    64 bytes from 192.168.1.133: icmp_seq=2 ttl=64 time=4.16 ms
    64 bytes from 192.168.1.133: icmp_seq=3 ttl=64 time=3.64 ms
    64 bytes from 192.168.1.133: icmp_seq=4 ttl=64 time=8.02 ms
    64 bytes from 192.168.1.133: icmp_seq=5 ttl=64 time=3.86 ms
    64 bytes from 192.168.1.133: icmp_seq=6 ttl=64 time=4.24 ms
    64 bytes from 192.168.1.133: icmp_seq=7 ttl=64 time=3.78 ms
    64 bytes from 192.168.1.133: icmp_seq=8 ttl=64 time=3.30 ms
    64 bytes from 192.168.1.133: icmp_seq=9 ttl=64 time=14.9 ms
    64 bytes from 192.168.1.133: icmp_seq=10 ttl=64 time=3.86 ms
    64 bytes from 192.168.1.133: icmp_seq=11 ttl=64 time=4.27 ms
    64 bytes from 192.168.1.133: icmp_seq=12 ttl=64 time=3.36 ms
    64 bytes from 192.168.1.133: icmp_seq=13 ttl=64 time=3.16 ms
    64 bytes from 192.168.1.133: icmp_seq=14 ttl=64 time=3.92 ms
    64 bytes from 192.168.1.133: icmp_seq=15 ttl=64 time=2.95 ms
    64 bytes from 192.168.1.133: icmp_seq=16 ttl=64 time=3.47 ms
    64 bytes from 192.168.1.133: icmp_seq=17 ttl=64 time=2.60 ms
    64 bytes from 192.168.1.133: icmp_seq=18 ttl=64 time=7.16 ms
    64 bytes from 192.168.1.133: icmp_seq=19 ttl=64 time=2.65 ms
    64 bytes from 192.168.1.133: icmp_seq=20 ttl=64 time=2.73 ms
    64 bytes from 192.168.1.133: icmp_seq=21 ttl=64 time=4.29 ms
    64 bytes from 192.168.1.133: icmp_seq=22 ttl=64 time=3.84 ms
    64 bytes from 192.168.1.133: icmp_seq=23 ttl=64 time=2.77 ms
    64 bytes from 192.168.1.133: icmp_seq=24 ttl=64 time=3.12 ms
    64 bytes from 192.168.1.133: icmp_seq=25 ttl=64 time=2.62 ms
    64 bytes from 192.168.1.133: icmp_seq=26 ttl=64 time=2.43 ms
    64 bytes from 192.168.1.133: icmp_seq=27 ttl=64 time=4.75 ms

    OK - I've deleted all of the folders and recloned them and re-installed depthai

    This has enabled me to use the device manager to update the firmware to 0.0.28 but it still behaves the same way. I've tried the depthai_viewer using the USB C port connected to my laptop and everything works fine.

    I'll try again with a different POE switch or injector.

      mikegardner
      Good. Seems like the network is fine.
      What depthai-viewer version are you using? What do the logs look like?

      Should be something similar to: -- version is 0.2.4

      Backend started successfully.
      2024-09-23 10:53:54.570 python[3104:9491847] +[IMKClient subclass]: chose IMKClient_Legacy
      2024-09-23 10:53:54.570 python[3104:9491847] +[IMKInputSession subclass]: chose IMKInputSession_Legacy
      [2024-09-23T08:53:56Z INFO  ewebsock::native_tungstenite] WebSocket handshake has been successfully completed
      [2024-09-23T08:53:56Z INFO  re_viewer::depthai::ws] Websocket opened
      Selecting device:  18443010B1595B1200
      [2024-09-23 10:54:01.075] [depthai] [warning] [18443010B1595B1200] [10.12.118.85] Flashed bootloader version 0.0.26, less than 0.0.28 is susceptible to bootup/restart failure. Upgrading is advised, flashing main/factory (not user) bootloader. Available: 0.0.28
      Connecting to viewer at 127.0.0.1:9876
      Oak cam:  <depthai_sdk.oak_camera.OakCamera object at 0x16d6cab90>
      Default stereo pair:  None
      Pipeline config:  auto=True cameras=[CameraConfiguration(fps=30, resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'>, kind=<CameraSensorType.COLOR: 0>, board_socket=<CameraBoardSocket.CAM_A: 0>, stream_enabled=True, name='Color', tof_align=None)] stereo=None ai_model=AiModelConfiguration(display_name='Yolo V6', path='yolov6nr3_coco_640x352', camera=<CameraBoardSocket.CAM_A: 0>) imu=ImuConfiguration(report_rate=100, batch_report_threshold=5)
      Updating pipeline...
      Creating auto pipeline config
      Connected camera features:  [{socket: CAM_A, sensorName: IMX378, width: 4056, height: 3040, orientation: AUTO, supportedTypes: [COLOR], hasAutofocus: 1, hasAutofocusIC: 1, name: color}]
      Usb speed:  UsbSpeed.UNKNOWN
      Connected to a PoE device, camera streams will be JPEG encoded...
      Creating camera:  fps=30 resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'> kind=<CameraSensorType.COLOR: 0> board_socket=<CameraBoardSocket.CAM_A: 0> stream_enabled=True name='color' tof_align=None
      Connected cam doesn't have IMU, skipping IMU creation...
      Starting pipeline
      ----------------------------------------
      [0s] System information
      ----------------------------------------
      Ddr used / total - 96.90 / 333.28 MiB
      Cmx used / total - 2.49 / 2.50 MiB
      LeonCss heap used / total - 41.00 / 82.20 MiB
      LeonMss heap used / total - 4.36 / 40.35 MiB
      Chip temperature - average: 28.43, css: 30.80, mss: 26.90, upa: 27.88, dss: 28.13
      Cpu usage - Leon CSS: 5.70 %, Leon MSS: 0.05 %
      Got message to send:  <depthai_viewer._backend.messages.WarningMessage object at 0x17ec72110>
      Sending message:  <depthai_viewer._backend.messages.WarningMessage object at 0x17ec72110>
      [2024-09-23T08:54:13Z WARN  re_viewer::depthai::depthai] Device is connected via PoE. This may cause performance issues.
      ----------------------------------------
      [10s] System information
      ----------------------------------------
      Ddr used / total - 109.08 / 333.28 MiB
      Cmx used / total - 2.49 / 2.50 MiB
      LeonCss heap used / total - 41.89 / 82.20 MiB
      LeonMss heap used / total - 5.21 / 40.35 MiB
      Chip temperature - average: 32.37, css: 33.93, mss: 31.53, upa: 32.25, dss: 31.77
      Cpu usage - Leon CSS: 91.63 %, Leon MSS: 11.28 %

      Thanks,
      Jaka

      Thanks Jaka

      depthai_viewer Version is

      0.2.4 [rustc 1.74.1 (a28077b28 2023-12-04), LLVM 17.0.4] x86_64-unknown-linux-gnu bf0ca8b, built 2024-06-21T10:37:38Z

      With the USB C connection I get the following report when launching
      MESA-INTEL: warning: Haswell Vulkan support is incomplete
      Backend started successfully.
      [2024-09-23T10:24:32Z INFO ewebsock::native_tungstenite] WebSocket handshake has been successfully completed
      [2024-09-23T10:24:32Z INFO re_viewer::depthai::ws] Websocket opened
      Selecting device: 18443010B1E752F500
      Connecting to viewer at 127.0.0.1:9876
      Oak cam: <depthai_sdk.oak_camera.OakCamera object at 0x792dc8101120>
      Default stereo pair: None
      Pipeline config: auto=True cameras=[CameraConfiguration(fps=30, resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'>, kind=<CameraSensorType.COLOR: 0>, board_socket=<CameraBoardSocket.CAM_A: 0>, stream_enabled=True, name='Color', tof_align=None)] stereo=None ai_model=AiModelConfiguration(display_name='Yolo V6', path='yolov6nr3_coco_640x352', camera=<CameraBoardSocket.CAM_A: 0>) imu=ImuConfiguration(report_rate=100, batch_report_threshold=5)
      Updating pipeline...
      Creating auto pipeline config
      Connected camera features: [{socket: CAM_A, sensorName: IMX378, width: 4056, height: 3040, orientation: AUTO, supportedTypes: [COLOR], hasAutofocus: 1, hasAutofocusIC: 1, name: color}]
      Usb speed: UsbSpeed.SUPER
      Creating camera: fps=30 resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'> kind=<CameraSensorType.COLOR: 0> board_socket=<CameraBoardSocket.CAM_A: 0> stream_enabled=True name='color' tof_align=None
      Connected cam doesn't have IMU, skipping IMU creation...

      Starting pipeline

      [0s] System information

      Ddr used / total - 96.89 / 333.28 MiB
      Cmx used / total - 2.49 / 2.50 MiB
      LeonCss heap used / total - 23.21 / 82.20 MiB
      LeonMss heap used / total - 4.26 / 40.35 MiB
      Chip temperature - average: 30.86, css: 33.21, mss: 29.59, upa: 30.08, dss: 30.56

      Cpu usage - Leon CSS: 4.49 %, Leon MSS: 0.05 %

      [10s] System information

      Ddr used / total - 96.89 / 333.28 MiB
      Cmx used / total - 2.49 / 2.50 MiB
      LeonCss heap used / total - 24.10 / 82.20 MiB
      LeonMss heap used / total - 4.92 / 40.35 MiB
      Chip temperature - average: 35.17, css: 37.48, mss: 34.16, upa: 35.11, dss: 33.93
      Cpu usage - Leon CSS: 42.71 %, Leon MSS: 26.02 %

      I'll post the logs with the POE connection shortly.

      Here are the logs when the device is connected via the POE switch.

      MESA-INTEL: warning: Haswell Vulkan support is incomplete
      Backend started successfully.
      [2024-09-23T10:38:49Z INFO ewebsock::native_tungstenite] WebSocket handshake has been successfully completed
      [2024-09-23T10:38:49Z INFO re_viewer::depthai::ws] Websocket opened
      Selecting device: 18443010B1E752F500
      Connecting to viewer at 127.0.0.1:9876
      Oak cam: <depthai_sdk.oak_camera.OakCamera object at 0x7e5dbf06d150>
      Default stereo pair: None
      Pipeline config: auto=True cameras=[CameraConfiguration(fps=30, resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'>, kind=<CameraSensorType.COLOR: 0>, board_socket=<CameraBoardSocket.CAM_A: 0>, stream_enabled=True, name='Color', tof_align=None)] stereo=None ai_model=AiModelConfiguration(display_name='Yolo V6', path='yolov6nr3_coco_640x352', camera=<CameraBoardSocket.CAM_A: 0>) imu=ImuConfiguration(report_rate=100, batch_report_threshold=5)
      Updating pipeline...
      Creating auto pipeline config
      Connected camera features: [{socket: CAM_A, sensorName: IMX378, width: 4056, height: 3040, orientation: AUTO, supportedTypes: [COLOR], hasAutofocus: 1, hasAutofocusIC: 1, name: color}]
      Usb speed: UsbSpeed.UNKNOWN
      Connected to a PoE device, camera streams will be JPEG encoded...
      Creating camera: fps=30 resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'> kind=<CameraSensorType.COLOR: 0> board_socket=<CameraBoardSocket.CAM_A: 0> stream_enabled=True name='color' tof_align=None
      Connected cam doesn't have IMU, skipping IMU creation...
      Starting pipeline
      Got message to send: <depthai_viewer.backend.messages.WarningMessage object at 0x7e5dbfaf7be0>

      Sending message: <depthai_viewer._backend.messages.WarningMessage object at 0x7e5dbfaf7be0>

      [0s] System information


      Ddr used / total - 96.90 / 333.28 MiB
      Cmx used / total - 2.49 / 2.50 MiB
      LeonCss heap used / total - 41.00 / 82.20 MiB
      LeonMss heap used / total - 4.36 / 40.35 MiB
      Chip temperature - average: 31.64, css: 33.69, mss: 30.08, upa: 31.29, dss: 31.53
      Cpu usage - Leon CSS: 16.14 %, Leon MSS: 0.04 %

      [2024-09-23T10:39:55Z WARN re_viewer::depthai::depthai] Device is connected via PoE. This may cause performance issues.

      [19s] System information


      Ddr used / total - 109.08 / 333.28 MiB
      Cmx used / total - 2.49 / 2.50 MiB
      LeonCss heap used / total - 41.89 / 82.20 MiB
      LeonMss heap used / total - 5.21 / 40.35 MiB
      Chip temperature - average: 32.73, css: 34.40, mss: 31.77, upa: 32.97, dss: 31.77

      Cpu usage - Leon CSS: 26.49 %, Leon MSS: 0.42 %

      [27s] System information


      Ddr used / total - 109.08 / 333.28 MiB
      Cmx used / total - 2.49 / 2.50 MiB
      LeonCss heap used / total - 41.89 / 82.20 MiB
      LeonMss heap used / total - 5.21 / 40.35 MiB
      Chip temperature - average: 33.50, css: 34.88, mss: 32.97, upa: 34.16, dss: 32.01
      Cpu usage - Leon CSS: 27.33 %, Leon MSS: 0.42 %
      C[2024-09-23T10:40:41Z WARN re_sdk_comms::server] Closing connection to client: early eof

        Hi Jaka
        I've just seen your reply to syg regarding bandwidth.

        Looking at the link https://docs.luxonis.com/software/depthai/optimizing/#Bandwidth

        What Resolution FPS FPS set PoE Latency [ms] USB Latency [ms] Bandwidth
        Color (isp) 1080P 25 25 51 33 Std: 0.8 622 Mbps

        unless I'm misunderstanding the guidelines, my set-up at 30fps colour 1080P is likely to require in excess of 622 Mbps! That's almost certainly the problem, I'm using a Netgear Prosafe FS108P which is 10/100Mbps each port.
        Thanks

        OK
        with a direct PC to camera ethernet connection, I get
        Downlink 891.7 mbps
        Uplink 228.9 mbps
        Press any key to continue...

        However, with a connection via the switch, I get
        Downlink 356.1 mbps
        Traceback (most recent call last):
        File "/home/mike/Luxonis/depthai-python/utilities/oak_bandwidth_test.py", line 91, in <module>
        qin.send(buffer)
        RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Couldn't write data to stream: 'xin' (X_LINK_ERROR)'

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
        File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
        return _run_code(code, main_globals, None,
        File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
        exec(code, run_globals)
        File "/home/mike/Luxonis/depthai-python/utilities/oak_bandwidth_test.py", line 66, in <module>
        with dai.Device(pipeline, maxUsbSpeed=dai.UsbSpeed.SUPER_PLUS) as device:
        RuntimeError: Device already closed or disconnected: Input/output error
        Stack trace (most recent call last):
        #20 Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in
        #19 Object "python3", at 0x64627243d604, in _start
        #18 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7c47d1429e3f, in __libc_start_main
        #17 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7c47d1429d8f, in
        #16 Object "python3", at 0x64627243d70c, in Py_BytesMain
        #15 Object "python3", at 0x646272467182, in Py_RunMain
        #14 Object "python3", at 0x646272475cd7, in Py_FinalizeEx
        #13 Object "python3", at 0x6462724791d5, in
        #12 Object "python3", at 0x64627233d05e, in
        #11 Object "python3", at 0x6462724796c3, in
        #10 Object "python3", at 0x64627234b63b, in
        #9 Object "python3", at 0x64627234b840, in
        #8 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47ceea44e3, in
        #7 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47ceecc356, in
        #6 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf12f178, in dai:😃evice::Device()
        #5 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf12ed63, in dai:😃evice::Device()
        #4 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf13a7e4, in dai:😃eviceBase::close()
        #3 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf132f04, in dai:😃evice::closeImpl()
        #2 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf14f904, in dai:😃eviceBase::closeImpl()
        #1 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf13ffaa, in dai:😃eviceBase::hasCrashDump()
        #0 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf17331d, in nanorpc::core::client<nanorpc::packer::nlohmann_msgpack>::result nanorpc::core::client<nanorpc::packer::nlohmann_msgpack>::call<>(unsigned long)
        Segmentation fault (Address not mapped to object [0x10])
        Segmentation fault (core dumped)

          mikegardner
          Ok, then everything should work when directly connecting right?

          Seems like the switch allows for only one way communication.. Is your network set to full-duplex? It's likely the switch that is the problem..

          Thanks
          Jaka

          Hi Jaka,
          Yes, I get a reasonable performance if I connect with a laptop directly to the USB C connector, or the directly to the ethernet connector. However in many applications, this is not a realistic use case, because there will be switches and routers between the client and the OAK 1 POE.
          For example, if I get rid of the switch and replace it with a Ubiquiti 802.3AF PoE Injector that I purchased from your web store, connected directly into a Wifi router and then connect via Wifi to the laptop, using the Depthai Viewer I only get around 3 frames per second at 1080P, albeit with no dropout.
          In this configuration, I get the following results from the bandwidth_test...
          Downlink 48.3 mbps
          Uplink 47.9 mbps

          So in summary, (all at 1080P):

          • Laptop connection via Wifi then the Netgear Prosafe FS 108P switch (10/100Mbps) = 1 frame displayed then drop-out

          • Laptop connection to Wifi, then Ubiquiti POE injector from Wifi router to camera = 3 fps appx, no dropout

          • Laptop connection via ethernet direct to the Netgear switch with the OAK also on the switch = 3 fps appx with occasional drop-out

          • Laptop connection directly via Ethernet port to Ethernet port powered via USB C = 10 fps appx with occasional drop-out

          • Laptop with direct USB C to USB C connection, seems like 30fps, no drop-out

          My conclusion is that the performance is directly related to the connectivity, no surprises there I suppose.
          However for many use cases there will rarely be direct Eth to Eth or USB to USB connectivity, and for my applicationI was hoping that I could use a 4g router connected to the camera and then a broadband connection from the laptop.

          Given the raw network ping statistics are quite reasonable, I suspect the depthai_viewer is not well suited to these particular configuration above, and for a 4G use case I suppose I need to develop an app that compresses any video prior to sending it in an effort to improve the frame rate. In that case a OAK-D CM4 is probably a better candidate.
          Mike

            mikegardner For example, if I get rid of the switch and replace it with a Ubiquiti 802.3AF PoE Injector that I purchased from your web store, connected directly into a Wifi router and then connect via Wifi to the laptop, using the Depthai Viewer I only get around 3 frames per second at 1080P, albeit with no dropout.

            WIFI bottleneck is expected. Unless WiFi 5 or better the bandwidth is very limited.

            The bandwidth for normal operations should be 1gbps. Otherwise connectivity issues are expected. The switch is too slow to handle data streaming.
            https://docs.luxonis.com/hardware/platform/deploy/poe-deployment-guide/#PoE%20deployment%20guide-Runtime-Debugging

            If you intend to use wireless transfer, the sending full streams is going to be a problem.

            Thanks,
            Jaka

            Thanks Jaka, that's the conclusion I came to. The OAK 1 POE is suited well to direct POE Ethernet connections with a high bandwidth switch to a server or client. It doesn't perform well in a more conventional network topology, like an IP Camera does.
            I think the better solution for my application is the OAK-D CM4, I can route the messages back via 4g with no bandwidth implications, and store the stream locally. When a message indicates an object of interest has been detected, I can stream back that section in an mp4 clip or similar.
            Thanks for your input and help to understand the characteristics of the OAK1 POE product.
            Mike