OAK 1 POE Severe Dropout

jakaskerl

Thanks for your assistance

mikegardner
Can you give more info on OS you are using?

I'm using Ubuntu 22.04.5 LTS

mikegardner I used the device manager using the link on the docs page to try to update the firmware using the Danger Zone 'Update Bootloader' and 'Flash Factory Bootloader' options but the 'Flashed bootloader version' always stays at 0.0.21

This would suggest you are using an older version of depthai or that the flashing failed (in which case you would get an error).

I suspected I was using an older version of depthai, so I deleted the depthai, depthai-python, and depthai-core folders and cloned them again from the master branch on the current repo, ran submodule update --init --recursive on each of those folders plus run install_requirements git but still got the same results.

mikegardner I'm up to date with the depthai repo but no idea where the bootloader file is, it's not in the utilities directory with the device_manager.py module

But it should be. luxonis/depthai-pythontree/main/utilities

Looking at the repo at the link you suggested, it matches my local folder contents exactly. As i say, I can see the device_manager.py module but nothing that looks like the actual bootloader file that is installed.

mikegardner If I ping the camera I get on average 10-15ms response with no missed beats

But this would indicate a network issue. What does your setup look like?

I have a Netgear Prosafe 108P 8-port switch with 4 POE ports. The only POE port that is populated is the one for the camera. The switch is connected to my Wifi router and I use a a Thinkpad T440s running Unbuntu 22.04.5 LTS to attach to the camera.

The laptop is dedicated as a Depthai dev system

Thanks,
Jaka

mikegardner
Here's a sample of the ping responses, IMHO they don't suggest a network issue...

PING 192.168.1.133 (192.168.1.133) 56(84) bytes of data.
64 bytes from 192.168.1.133: icmp_seq=1 ttl=64 time=241 ms
64 bytes from 192.168.1.133: icmp_seq=2 ttl=64 time=4.16 ms
64 bytes from 192.168.1.133: icmp_seq=3 ttl=64 time=3.64 ms
64 bytes from 192.168.1.133: icmp_seq=4 ttl=64 time=8.02 ms
64 bytes from 192.168.1.133: icmp_seq=5 ttl=64 time=3.86 ms
64 bytes from 192.168.1.133: icmp_seq=6 ttl=64 time=4.24 ms
64 bytes from 192.168.1.133: icmp_seq=7 ttl=64 time=3.78 ms
64 bytes from 192.168.1.133: icmp_seq=8 ttl=64 time=3.30 ms
64 bytes from 192.168.1.133: icmp_seq=9 ttl=64 time=14.9 ms
64 bytes from 192.168.1.133: icmp_seq=10 ttl=64 time=3.86 ms
64 bytes from 192.168.1.133: icmp_seq=11 ttl=64 time=4.27 ms
64 bytes from 192.168.1.133: icmp_seq=12 ttl=64 time=3.36 ms
64 bytes from 192.168.1.133: icmp_seq=13 ttl=64 time=3.16 ms
64 bytes from 192.168.1.133: icmp_seq=14 ttl=64 time=3.92 ms
64 bytes from 192.168.1.133: icmp_seq=15 ttl=64 time=2.95 ms
64 bytes from 192.168.1.133: icmp_seq=16 ttl=64 time=3.47 ms
64 bytes from 192.168.1.133: icmp_seq=17 ttl=64 time=2.60 ms
64 bytes from 192.168.1.133: icmp_seq=18 ttl=64 time=7.16 ms
64 bytes from 192.168.1.133: icmp_seq=19 ttl=64 time=2.65 ms
64 bytes from 192.168.1.133: icmp_seq=20 ttl=64 time=2.73 ms
64 bytes from 192.168.1.133: icmp_seq=21 ttl=64 time=4.29 ms
64 bytes from 192.168.1.133: icmp_seq=22 ttl=64 time=3.84 ms
64 bytes from 192.168.1.133: icmp_seq=23 ttl=64 time=2.77 ms
64 bytes from 192.168.1.133: icmp_seq=24 ttl=64 time=3.12 ms
64 bytes from 192.168.1.133: icmp_seq=25 ttl=64 time=2.62 ms
64 bytes from 192.168.1.133: icmp_seq=26 ttl=64 time=2.43 ms
64 bytes from 192.168.1.133: icmp_seq=27 ttl=64 time=4.75 ms

OK - I've deleted all of the folders and recloned them and re-installed depthai

This has enabled me to use the device manager to update the firmware to 0.0.28 but it still behaves the same way. I've tried the depthai_viewer using the USB C port connected to my laptop and everything works fine.

I'll try again with a different POE switch or injector.

Thanks Jaka

depthai_viewer Version is

0.2.4 [rustc 1.74.1 (a28077b28 2023-12-04), LLVM 17.0.4] x86_64-unknown-linux-gnu bf0ca8b, built 2024-06-21T10:37:38Z

With the USB C connection I get the following report when launching
MESA-INTEL: warning: Haswell Vulkan support is incomplete
Backend started successfully.
[2024-09-23T10:24:32Z INFO ewebsock::native_tungstenite] WebSocket handshake has been successfully completed
[2024-09-23T10:24:32Z INFO re_viewer::depthai::ws] Websocket opened
Selecting device: 18443010B1E752F500
Connecting to viewer at 127.0.0.1:9876
Oak cam: <depthai_sdk.oak_camera.OakCamera object at 0x792dc8101120>
Default stereo pair: None
Pipeline config: auto=True cameras=[CameraConfiguration(fps=30, resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'>, kind=<CameraSensorType.COLOR: 0>, board_socket=<CameraBoardSocket.CAM_A: 0>, stream_enabled=True, name='Color', tof_align=None)] stereo=None ai_model=AiModelConfiguration(display_name='Yolo V6', path='yolov6nr3_coco_640x352', camera=<CameraBoardSocket.CAM_A: 0>) imu=ImuConfiguration(report_rate=100, batch_report_threshold=5)
Updating pipeline...
Creating auto pipeline config
Connected camera features: [{socket: CAM_A, sensorName: IMX378, width: 4056, height: 3040, orientation: AUTO, supportedTypes: [COLOR], hasAutofocus: 1, hasAutofocusIC: 1, name: color}]
Usb speed: UsbSpeed.SUPER
Creating camera: fps=30 resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'> kind=<CameraSensorType.COLOR: 0> board_socket=<CameraBoardSocket.CAM_A: 0> stream_enabled=True name='color' tof_align=None
Connected cam doesn't have IMU, skipping IMU creation...

Starting pipeline

[0s] System information

Ddr used / total - 96.89 / 333.28 MiB
Cmx used / total - 2.49 / 2.50 MiB
LeonCss heap used / total - 23.21 / 82.20 MiB
LeonMss heap used / total - 4.26 / 40.35 MiB
Chip temperature - average: 30.86, css: 33.21, mss: 29.59, upa: 30.08, dss: 30.56

Cpu usage - Leon CSS: 4.49 %, Leon MSS: 0.05 %

[10s] System information

Ddr used / total - 96.89 / 333.28 MiB
Cmx used / total - 2.49 / 2.50 MiB
LeonCss heap used / total - 24.10 / 82.20 MiB
LeonMss heap used / total - 4.92 / 40.35 MiB
Chip temperature - average: 35.17, css: 37.48, mss: 34.16, upa: 35.11, dss: 33.93
Cpu usage - Leon CSS: 42.71 %, Leon MSS: 26.02 %

I'll post the logs with the POE connection shortly.

Here are the logs when the device is connected via the POE switch.

MESA-INTEL: warning: Haswell Vulkan support is incomplete
Backend started successfully.
[2024-09-23T10:38:49Z INFO ewebsock::native_tungstenite] WebSocket handshake has been successfully completed
[2024-09-23T10:38:49Z INFO re_viewer::depthai::ws] Websocket opened
Selecting device: 18443010B1E752F500
Connecting to viewer at 127.0.0.1:9876
Oak cam: <depthai_sdk.oak_camera.OakCamera object at 0x7e5dbf06d150>
Default stereo pair: None
Pipeline config: auto=True cameras=[CameraConfiguration(fps=30, resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'>, kind=<CameraSensorType.COLOR: 0>, board_socket=<CameraBoardSocket.CAM_A: 0>, stream_enabled=True, name='Color', tof_align=None)] stereo=None ai_model=AiModelConfiguration(display_name='Yolo V6', path='yolov6nr3_coco_640x352', camera=<CameraBoardSocket.CAM_A: 0>) imu=ImuConfiguration(report_rate=100, batch_report_threshold=5)
Updating pipeline...
Creating auto pipeline config
Connected camera features: [{socket: CAM_A, sensorName: IMX378, width: 4056, height: 3040, orientation: AUTO, supportedTypes: [COLOR], hasAutofocus: 1, hasAutofocusIC: 1, name: color}]
Usb speed: UsbSpeed.UNKNOWN
Connected to a PoE device, camera streams will be JPEG encoded...
Creating camera: fps=30 resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'> kind=<CameraSensorType.COLOR: 0> board_socket=<CameraBoardSocket.CAM_A: 0> stream_enabled=True name='color' tof_align=None
Connected cam doesn't have IMU, skipping IMU creation...
Starting pipeline
Got message to send: <depthai_viewer.backend.messages.WarningMessage object at 0x7e5dbfaf7be0>

Sending message: <depthai_viewer._backend.messages.WarningMessage object at 0x7e5dbfaf7be0>

[0s] System information

Ddr used / total - 96.90 / 333.28 MiB
Cmx used / total - 2.49 / 2.50 MiB
LeonCss heap used / total - 41.00 / 82.20 MiB
LeonMss heap used / total - 4.36 / 40.35 MiB
Chip temperature - average: 31.64, css: 33.69, mss: 30.08, upa: 31.29, dss: 31.53
Cpu usage - Leon CSS: 16.14 %, Leon MSS: 0.04 %

[2024-09-23T10:39:55Z WARN re_viewer::depthai::depthai] Device is connected via PoE. This may cause performance issues.

[19s] System information

Ddr used / total - 109.08 / 333.28 MiB
Cmx used / total - 2.49 / 2.50 MiB
LeonCss heap used / total - 41.89 / 82.20 MiB
LeonMss heap used / total - 5.21 / 40.35 MiB
Chip temperature - average: 32.73, css: 34.40, mss: 31.77, upa: 32.97, dss: 31.77

Cpu usage - Leon CSS: 26.49 %, Leon MSS: 0.42 %

[27s] System information

Ddr used / total - 109.08 / 333.28 MiB
Cmx used / total - 2.49 / 2.50 MiB
LeonCss heap used / total - 41.89 / 82.20 MiB
LeonMss heap used / total - 5.21 / 40.35 MiB
Chip temperature - average: 33.50, css: 34.88, mss: 32.97, upa: 34.16, dss: 32.01
Cpu usage - Leon CSS: 27.33 %, Leon MSS: 0.42 %
^C[2024-09-23T10:40:41Z WARN re_sdk_comms::server] Closing connection to client: early eof

Hi Jaka
I've just seen your reply to syg regarding bandwidth.

Looking at the link https://docs.luxonis.com/software/depthai/optimizing/#Bandwidth

What Resolution FPS FPS set PoE Latency [ms] USB Latency [ms] Bandwidth
Color (isp) 1080P 25 25 51 33 Std: 0.8 622 Mbps

unless I'm misunderstanding the guidelines, my set-up at 30fps colour 1080P is likely to require in excess of 622 Mbps! That's almost certainly the problem, I'm using a Netgear Prosafe FS108P which is 10/100Mbps each port.
Thanks

OK
with a direct PC to camera ethernet connection, I get
Downlink 891.7 mbps
Uplink 228.9 mbps
Press any key to continue...

However, with a connection via the switch, I get
Downlink 356.1 mbps
Traceback (most recent call last):
File "/home/mike/Luxonis/depthai-python/utilities/oak_bandwidth_test.py", line 91, in <module>
qin.send(buffer)
RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Couldn't write data to stream: 'xin' (X_LINK_ERROR)'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/mike/Luxonis/depthai-python/utilities/oak_bandwidth_test.py", line 66, in <module>
with dai.Device(pipeline, maxUsbSpeed=dai.UsbSpeed.SUPER_PLUS) as device:
RuntimeError: Device already closed or disconnected: Input/output error
Stack trace (most recent call last):
#20 Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in
#19 Object "python3", at 0x64627243d604, in _start
#18 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7c47d1429e3f, in __libc_start_main
#17 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7c47d1429d8f, in
#16 Object "python3", at 0x64627243d70c, in Py_BytesMain
#15 Object "python3", at 0x646272467182, in Py_RunMain
#14 Object "python3", at 0x646272475cd7, in Py_FinalizeEx
#13 Object "python3", at 0x6462724791d5, in
#12 Object "python3", at 0x64627233d05e, in
#11 Object "python3", at 0x6462724796c3, in
#10 Object "python3", at 0x64627234b63b, in
#9 Object "python3", at 0x64627234b840, in
#8 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47ceea44e3, in
#7 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47ceecc356, in
#6 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf12f178, in dai:evice::_Device()
#5 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf12ed63, in dai:evice::_Device()
#4 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf13a7e4, in dai:eviceBase::close()
#3 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf132f04, in dai:evice::closeImpl()
#2 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf14f904, in dai:eviceBase::closeImpl()
#1 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf13ffaa, in dai:eviceBase::hasCrashDump()
#0 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf17331d, in nanorpc::core::client<nanorpc::packer::nlohmann_msgpack>::result nanorpc::core::client<nanorpc::packer::nlohmann_msgpack>::call<>(unsigned long)
Segmentation fault (Address not mapped to object [0x10])
Segmentation fault (core dumped)

Hi Jaka,
Yes, I get a reasonable performance if I connect with a laptop directly to the USB C connector, or the directly to the ethernet connector. However in many applications, this is not a realistic use case, because there will be switches and routers between the client and the OAK 1 POE.
For example, if I get rid of the switch and replace it with a Ubiquiti 802.3AF PoE Injector that I purchased from your web store, connected directly into a Wifi router and then connect via Wifi to the laptop, using the Depthai Viewer I only get around 3 frames per second at 1080P, albeit with no dropout.
In this configuration, I get the following results from the bandwidth_test...
Downlink 48.3 mbps
Uplink 47.9 mbps

So in summary, (all at 1080P):

• Laptop connection via Wifi then the Netgear Prosafe FS 108P switch (10/100Mbps) = 1 frame displayed then drop-out

• Laptop connection to Wifi, then Ubiquiti POE injector from Wifi router to camera = 3 fps appx, no dropout

• Laptop connection via ethernet direct to the Netgear switch with the OAK also on the switch = 3 fps appx with occasional drop-out

• Laptop connection directly via Ethernet port to Ethernet port powered via USB C = 10 fps appx with occasional drop-out

• Laptop with direct USB C to USB C connection, seems like 30fps, no drop-out

My conclusion is that the performance is directly related to the connectivity, no surprises there I suppose.
However for many use cases there will rarely be direct Eth to Eth or USB to USB connectivity, and for my applicationI was hoping that I could use a 4g router connected to the camera and then a broadband connection from the laptop.

Given the raw network ping statistics are quite reasonable, I suspect the depthai_viewer is not well suited to these particular configuration above, and for a 4G use case I suppose I need to develop an app that compresses any video prior to sending it in an effort to improve the frame rate. In that case a OAK-D CM4 is probably a better candidate.
Mike

Thanks Jaka, that's the conclusion I came to. The OAK 1 POE is suited well to direct POE Ethernet connections with a high bandwidth switch to a server or client. It doesn't perform well in a more conventional network topology, like an IP Camera does.
I think the better solution for my application is the OAK-D CM4, I can route the messages back via 4g with no bandwidth implications, and store the stream locally. When a message indicates an object of interest has been detected, I can stream back that section in an mp4 clip or similar.
Thanks for your input and help to understand the characteristics of the OAK1 POE product.
Mike