OAK 1 POE Severe Dropout

mikegardner

I've just purchased an OAK-1 POE device and followed the setup as advised in the docs, then run the DepthAI Viewer.

On launch, the terminal window reports "Device is connected via POE, This may cause performance issues."

The GUI takes about a minute to find the device, shows a single frame, then drops out. At that point, the terminal window reports "ping was missed, closing the device connection" followed by "Flashed bootloader version 0.0.21, less than 0.0.28 is susceptible to boot/restart failure. Upgrading is advised, flashing main/factory (not user) bootloader. Available 0.0.28"

I used the device manager using the link on the docs page to try to update the firmware using the Danger Zone 'Update Bootloader' and 'Flash Factory Bootloader' options but the 'Flashed bootloader version' always stays at 0.0.21

I'm up to date with the depthai repo but no idea where the bootloader file is, it's not in the utilities directory with the device_manager.py module

If I ping the camera I get on average 10-15ms response with no missed beats

Any advice appreciated

jakaskerl

mikegardner
Can you give more info on OS you are using?

mikegardner I used the device manager using the link on the docs page to try to update the firmware using the Danger Zone 'Update Bootloader' and 'Flash Factory Bootloader' options but the 'Flashed bootloader version' always stays at 0.0.21

This would suggest you are using an older version of depthai or that the flashing failed (in which case you would get an error).

mikegardner I'm up to date with the depthai repo but no idea where the bootloader file is, it's not in the utilities directory with the device_manager.py module

But it should be. luxonis/depthai-pythontree/main/utilities

mikegardner If I ping the camera I get on average 10-15ms response with no missed beats

But this would indicate a network issue. What does your setup look like?

Thanks,
Jaka

mikegardner

jakaskerl

Thanks for your assistance

mikegardner
Can you give more info on OS you are using?

I'm using Ubuntu 22.04.5 LTS

mikegardner I used the device manager using the link on the docs page to try to update the firmware using the Danger Zone 'Update Bootloader' and 'Flash Factory Bootloader' options but the 'Flashed bootloader version' always stays at 0.0.21

This would suggest you are using an older version of depthai or that the flashing failed (in which case you would get an error).

I suspected I was using an older version of depthai, so I deleted the depthai, depthai-python, and depthai-core folders and cloned them again from the master branch on the current repo, ran submodule update --init --recursive on each of those folders plus run install_requirements git but still got the same results.

mikegardner I'm up to date with the depthai repo but no idea where the bootloader file is, it's not in the utilities directory with the device_manager.py module

But it should be. luxonis/depthai-pythontree/main/utilities

Looking at the repo at the link you suggested, it matches my local folder contents exactly. As i say, I can see the device_manager.py module but nothing that looks like the actual bootloader file that is installed.

mikegardner If I ping the camera I get on average 10-15ms response with no missed beats

But this would indicate a network issue. What does your setup look like?

I have a Netgear Prosafe 108P 8-port switch with 4 POE ports. The only POE port that is populated is the one for the camera. The switch is connected to my Wifi router and I use a a Thinkpad T440s running Unbuntu 22.04.5 LTS to attach to the camera.

The laptop is dedicated as a Depthai dev system

Thanks,
Jaka

mikegardner

mikegardner
Here's a sample of the ping responses, IMHO they don't suggest a network issue...

PING 192.168.1.133 (192.168.1.133) 56(84) bytes of data.
64 bytes from 192.168.1.133: icmp_seq=1 ttl=64 time=241 ms
64 bytes from 192.168.1.133: icmp_seq=2 ttl=64 time=4.16 ms
64 bytes from 192.168.1.133: icmp_seq=3 ttl=64 time=3.64 ms
64 bytes from 192.168.1.133: icmp_seq=4 ttl=64 time=8.02 ms
64 bytes from 192.168.1.133: icmp_seq=5 ttl=64 time=3.86 ms
64 bytes from 192.168.1.133: icmp_seq=6 ttl=64 time=4.24 ms
64 bytes from 192.168.1.133: icmp_seq=7 ttl=64 time=3.78 ms
64 bytes from 192.168.1.133: icmp_seq=8 ttl=64 time=3.30 ms
64 bytes from 192.168.1.133: icmp_seq=9 ttl=64 time=14.9 ms
64 bytes from 192.168.1.133: icmp_seq=10 ttl=64 time=3.86 ms
64 bytes from 192.168.1.133: icmp_seq=11 ttl=64 time=4.27 ms
64 bytes from 192.168.1.133: icmp_seq=12 ttl=64 time=3.36 ms
64 bytes from 192.168.1.133: icmp_seq=13 ttl=64 time=3.16 ms
64 bytes from 192.168.1.133: icmp_seq=14 ttl=64 time=3.92 ms
64 bytes from 192.168.1.133: icmp_seq=15 ttl=64 time=2.95 ms
64 bytes from 192.168.1.133: icmp_seq=16 ttl=64 time=3.47 ms
64 bytes from 192.168.1.133: icmp_seq=17 ttl=64 time=2.60 ms
64 bytes from 192.168.1.133: icmp_seq=18 ttl=64 time=7.16 ms
64 bytes from 192.168.1.133: icmp_seq=19 ttl=64 time=2.65 ms
64 bytes from 192.168.1.133: icmp_seq=20 ttl=64 time=2.73 ms
64 bytes from 192.168.1.133: icmp_seq=21 ttl=64 time=4.29 ms
64 bytes from 192.168.1.133: icmp_seq=22 ttl=64 time=3.84 ms
64 bytes from 192.168.1.133: icmp_seq=23 ttl=64 time=2.77 ms
64 bytes from 192.168.1.133: icmp_seq=24 ttl=64 time=3.12 ms
64 bytes from 192.168.1.133: icmp_seq=25 ttl=64 time=2.62 ms
64 bytes from 192.168.1.133: icmp_seq=26 ttl=64 time=2.43 ms
64 bytes from 192.168.1.133: icmp_seq=27 ttl=64 time=4.75 ms

mikegardner

OK - I've deleted all of the folders and recloned them and re-installed depthai

This has enabled me to use the device manager to update the firmware to 0.0.28 but it still behaves the same way. I've tried the depthai_viewer using the USB C port connected to my laptop and everything works fine.

I'll try again with a different POE switch or injector.

jakaskerl

mikegardner
Good. Seems like the network is fine.
What depthai-viewer version are you using? What do the logs look like?

Should be something similar to: -- version is 0.2.4

Backend started successfully.
2024-09-23 10:53:54.570 python[3104:9491847] +[IMKClient subclass]: chose IMKClient_Legacy
2024-09-23 10:53:54.570 python[3104:9491847] +[IMKInputSession subclass]: chose IMKInputSession_Legacy
[2024-09-23T08:53:56Z INFO  ewebsock::native_tungstenite] WebSocket handshake has been successfully completed
[2024-09-23T08:53:56Z INFO  re_viewer::depthai::ws] Websocket opened
Selecting device:  18443010B1595B1200
[2024-09-23 10:54:01.075] [depthai] [warning] [18443010B1595B1200] [10.12.118.85] Flashed bootloader version 0.0.26, less than 0.0.28 is susceptible to bootup/restart failure. Upgrading is advised, flashing main/factory (not user) bootloader. Available: 0.0.28
Connecting to viewer at 127.0.0.1:9876
Oak cam:  <depthai_sdk.oak_camera.OakCamera object at 0x16d6cab90>
Default stereo pair:  None
Pipeline config:  auto=True cameras=[CameraConfiguration(fps=30, resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'>, kind=<CameraSensorType.COLOR: 0>, board_socket=<CameraBoardSocket.CAM_A: 0>, stream_enabled=True, name='Color', tof_align=None)] stereo=None ai_model=AiModelConfiguration(display_name='Yolo V6', path='yolov6nr3_coco_640x352', camera=<CameraBoardSocket.CAM_A: 0>) imu=ImuConfiguration(report_rate=100, batch_report_threshold=5)
Updating pipeline...
Creating auto pipeline config
Connected camera features:  [{socket: CAM_A, sensorName: IMX378, width: 4056, height: 3040, orientation: AUTO, supportedTypes: [COLOR], hasAutofocus: 1, hasAutofocusIC: 1, name: color}]
Usb speed:  UsbSpeed.UNKNOWN
Connected to a PoE device, camera streams will be JPEG encoded...
Creating camera:  fps=30 resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'> kind=<CameraSensorType.COLOR: 0> board_socket=<CameraBoardSocket.CAM_A: 0> stream_enabled=True name='color' tof_align=None
Connected cam doesn't have IMU, skipping IMU creation...
Starting pipeline
----------------------------------------
[0s] System information
----------------------------------------
Ddr used / total - 96.90 / 333.28 MiB
Cmx used / total - 2.49 / 2.50 MiB
LeonCss heap used / total - 41.00 / 82.20 MiB
LeonMss heap used / total - 4.36 / 40.35 MiB
Chip temperature - average: 28.43, css: 30.80, mss: 26.90, upa: 27.88, dss: 28.13
Cpu usage - Leon CSS: 5.70 %, Leon MSS: 0.05 %
Got message to send:  <depthai_viewer._backend.messages.WarningMessage object at 0x17ec72110>
Sending message:  <depthai_viewer._backend.messages.WarningMessage object at 0x17ec72110>
[2024-09-23T08:54:13Z WARN  re_viewer::depthai::depthai] Device is connected via PoE. This may cause performance issues.
----------------------------------------
[10s] System information
----------------------------------------
Ddr used / total - 109.08 / 333.28 MiB
Cmx used / total - 2.49 / 2.50 MiB
LeonCss heap used / total - 41.89 / 82.20 MiB
LeonMss heap used / total - 5.21 / 40.35 MiB
Chip temperature - average: 32.37, css: 33.93, mss: 31.53, upa: 32.25, dss: 31.77
Cpu usage - Leon CSS: 91.63 %, Leon MSS: 11.28 %

Thanks,
Jaka

mikegardner

Thanks Jaka

depthai_viewer Version is

0.2.4 [rustc 1.74.1 (a28077b28 2023-12-04), LLVM 17.0.4] x86_64-unknown-linux-gnu bf0ca8b, built 2024-06-21T10:37:38Z

With the USB C connection I get the following report when launching
MESA-INTEL: warning: Haswell Vulkan support is incomplete
Backend started successfully.
[2024-09-23T10:24:32Z INFO ewebsock::native_tungstenite] WebSocket handshake has been successfully completed
[2024-09-23T10:24:32Z INFO re_viewer::depthai::ws] Websocket opened
Selecting device: 18443010B1E752F500
Connecting to viewer at 127.0.0.1:9876
Oak cam: <depthai_sdk.oak_camera.OakCamera object at 0x792dc8101120>
Default stereo pair: None
Pipeline config: auto=True cameras=[CameraConfiguration(fps=30, resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'>, kind=<CameraSensorType.COLOR: 0>, board_socket=<CameraBoardSocket.CAM_A: 0>, stream_enabled=True, name='Color', tof_align=None)] stereo=None ai_model=AiModelConfiguration(display_name='Yolo V6', path='yolov6nr3_coco_640x352', camera=<CameraBoardSocket.CAM_A: 0>) imu=ImuConfiguration(report_rate=100, batch_report_threshold=5)
Updating pipeline...
Creating auto pipeline config
Connected camera features: [{socket: CAM_A, sensorName: IMX378, width: 4056, height: 3040, orientation: AUTO, supportedTypes: [COLOR], hasAutofocus: 1, hasAutofocusIC: 1, name: color}]
Usb speed: UsbSpeed.SUPER
Creating camera: fps=30 resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'> kind=<CameraSensorType.COLOR: 0> board_socket=<CameraBoardSocket.CAM_A: 0> stream_enabled=True name='color' tof_align=None
Connected cam doesn't have IMU, skipping IMU creation...

Starting pipeline

[0s] System information

Ddr used / total - 96.89 / 333.28 MiB
Cmx used / total - 2.49 / 2.50 MiB
LeonCss heap used / total - 23.21 / 82.20 MiB
LeonMss heap used / total - 4.26 / 40.35 MiB
Chip temperature - average: 30.86, css: 33.21, mss: 29.59, upa: 30.08, dss: 30.56

Cpu usage - Leon CSS: 4.49 %, Leon MSS: 0.05 %

[10s] System information

Ddr used / total - 96.89 / 333.28 MiB
Cmx used / total - 2.49 / 2.50 MiB
LeonCss heap used / total - 24.10 / 82.20 MiB
LeonMss heap used / total - 4.92 / 40.35 MiB
Chip temperature - average: 35.17, css: 37.48, mss: 34.16, upa: 35.11, dss: 33.93
Cpu usage - Leon CSS: 42.71 %, Leon MSS: 26.02 %

I'll post the logs with the POE connection shortly.

mikegardner

Here are the logs when the device is connected via the POE switch.

MESA-INTEL: warning: Haswell Vulkan support is incomplete
Backend started successfully.
[2024-09-23T10:38:49Z INFO ewebsock::native_tungstenite] WebSocket handshake has been successfully completed
[2024-09-23T10:38:49Z INFO re_viewer::depthai::ws] Websocket opened
Selecting device: 18443010B1E752F500
Connecting to viewer at 127.0.0.1:9876
Oak cam: <depthai_sdk.oak_camera.OakCamera object at 0x7e5dbf06d150>
Default stereo pair: None
Pipeline config: auto=True cameras=[CameraConfiguration(fps=30, resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'>, kind=<CameraSensorType.COLOR: 0>, board_socket=<CameraBoardSocket.CAM_A: 0>, stream_enabled=True, name='Color', tof_align=None)] stereo=None ai_model=AiModelConfiguration(display_name='Yolo V6', path='yolov6nr3_coco_640x352', camera=<CameraBoardSocket.CAM_A: 0>) imu=ImuConfiguration(report_rate=100, batch_report_threshold=5)
Updating pipeline...
Creating auto pipeline config
Connected camera features: [{socket: CAM_A, sensorName: IMX378, width: 4056, height: 3040, orientation: AUTO, supportedTypes: [COLOR], hasAutofocus: 1, hasAutofocusIC: 1, name: color}]
Usb speed: UsbSpeed.UNKNOWN
Connected to a PoE device, camera streams will be JPEG encoded...
Creating camera: fps=30 resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'> kind=<CameraSensorType.COLOR: 0> board_socket=<CameraBoardSocket.CAM_A: 0> stream_enabled=True name='color' tof_align=None
Connected cam doesn't have IMU, skipping IMU creation...
Starting pipeline
Got message to send: <depthai_viewer.backend.messages.WarningMessage object at 0x7e5dbfaf7be0>

Sending message: <depthai_viewer._backend.messages.WarningMessage object at 0x7e5dbfaf7be0>

[0s] System information

Ddr used / total - 96.90 / 333.28 MiB
Cmx used / total - 2.49 / 2.50 MiB
LeonCss heap used / total - 41.00 / 82.20 MiB
LeonMss heap used / total - 4.36 / 40.35 MiB
Chip temperature - average: 31.64, css: 33.69, mss: 30.08, upa: 31.29, dss: 31.53
Cpu usage - Leon CSS: 16.14 %, Leon MSS: 0.04 %

[2024-09-23T10:39:55Z WARN re_viewer::depthai::depthai] Device is connected via PoE. This may cause performance issues.

[19s] System information

Cpu usage - Leon CSS: 26.49 %, Leon MSS: 0.42 %

[27s] System information

Ddr used / total - 109.08 / 333.28 MiB
Cmx used / total - 2.49 / 2.50 MiB
LeonCss heap used / total - 41.89 / 82.20 MiB
LeonMss heap used / total - 5.21 / 40.35 MiB
Chip temperature - average: 33.50, css: 34.88, mss: 32.97, upa: 34.16, dss: 32.01
Cpu usage - Leon CSS: 27.33 %, Leon MSS: 0.42 %
^C[2024-09-23T10:40:41Z WARN re_sdk_comms::server] Closing connection to client: early eof

jakaskerl

mikegardner
Can you do a quick bandwidth test? luxonis/depthai-experimentsblob/master/random-scripts/oak_bandwidth_test.py

I manually bottlenecked my mac to 100mbps and while the script still worked, it was much slower. The logs looked the same as on your end.

Thanks,
Jaka

mikegardner

Hi Jaka
I've just seen your reply to syg regarding bandwidth.

Looking at the link https://docs.luxonis.com/software/depthai/optimizing/#Bandwidth

What Resolution FPS FPS set PoE Latency [ms] USB Latency [ms] Bandwidth
Color (isp) 1080P 25 25 51 33 Std: 0.8 622 Mbps

unless I'm misunderstanding the guidelines, my set-up at 30fps colour 1080P is likely to require in excess of 622 Mbps! That's almost certainly the problem, I'm using a Netgear Prosafe FS108P which is 10/100Mbps each port.
Thanks

mikegardner

OK
with a direct PC to camera ethernet connection, I get
Downlink 891.7 mbps
Uplink 228.9 mbps
Press any key to continue...

However, with a connection via the switch, I get
Downlink 356.1 mbps
Traceback (most recent call last):
File "/home/mike/Luxonis/depthai-python/utilities/oak_bandwidth_test.py", line 91, in <module>
qin.send(buffer)
RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Couldn't write data to stream: 'xin' (X_LINK_ERROR)'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/mike/Luxonis/depthai-python/utilities/oak_bandwidth_test.py", line 66, in <module>
with dai.Device(pipeline, maxUsbSpeed=dai.UsbSpeed.SUPER_PLUS) as device:
RuntimeError: Device already closed or disconnected: Input/output error
Stack trace (most recent call last):
#20 Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in
#19 Object "python3", at 0x64627243d604, in _start
#18 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7c47d1429e3f, in __libc_start_main
#17 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7c47d1429d8f, in
#16 Object "python3", at 0x64627243d70c, in Py_BytesMain
#15 Object "python3", at 0x646272467182, in Py_RunMain
#14 Object "python3", at 0x646272475cd7, in Py_FinalizeEx
#13 Object "python3", at 0x6462724791d5, in
#12 Object "python3", at 0x64627233d05e, in
#11 Object "python3", at 0x6462724796c3, in
#10 Object "python3", at 0x64627234b63b, in
#9 Object "python3", at 0x64627234b840, in
#8 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47ceea44e3, in
#7 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47ceecc356, in
#6 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf12f178, in dai:😃evice::_Device()
#5 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf12ed63, in dai:😃evice::_Device()
#4 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf13a7e4, in dai:😃eviceBase::close()
#3 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf132f04, in dai:😃evice::closeImpl()
#2 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf14f904, in dai:😃eviceBase::closeImpl()
#1 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf13ffaa, in dai:😃eviceBase::hasCrashDump()
#0 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf17331d, in nanorpc::core::client<nanorpc::packer::nlohmann_msgpack>::result nanorpc::core::client<nanorpc::packer::nlohmann_msgpack>::call<>(unsigned long)
Segmentation fault (Address not mapped to object [0x10])
Segmentation fault (core dumped)

jakaskerl

mikegardner
Ok, then everything should work when directly connecting right?

Seems like the switch allows for only one way communication.. Is your network set to full-duplex? It's likely the switch that is the problem..

Thanks
Jaka

mikegardner

Hi Jaka,
Yes, I get a reasonable performance if I connect with a laptop directly to the USB C connector, or the directly to the ethernet connector. However in many applications, this is not a realistic use case, because there will be switches and routers between the client and the OAK 1 POE.
For example, if I get rid of the switch and replace it with a Ubiquiti 802.3AF PoE Injector that I purchased from your web store, connected directly into a Wifi router and then connect via Wifi to the laptop, using the Depthai Viewer I only get around 3 frames per second at 1080P, albeit with no dropout.
In this configuration, I get the following results from the bandwidth_test...
Downlink 48.3 mbps
Uplink 47.9 mbps

So in summary, (all at 1080P):

Laptop connection via Wifi then the Netgear Prosafe FS 108P switch (10/100Mbps) = 1 frame displayed then drop-out
Laptop connection to Wifi, then Ubiquiti POE injector from Wifi router to camera = 3 fps appx, no dropout
Laptop connection via ethernet direct to the Netgear switch with the OAK also on the switch = 3 fps appx with occasional drop-out
Laptop connection directly via Ethernet port to Ethernet port powered via USB C = 10 fps appx with occasional drop-out
Laptop with direct USB C to USB C connection, seems like 30fps, no drop-out

My conclusion is that the performance is directly related to the connectivity, no surprises there I suppose.
However for many use cases there will rarely be direct Eth to Eth or USB to USB connectivity, and for my applicationI was hoping that I could use a 4g router connected to the camera and then a broadband connection from the laptop.

Given the raw network ping statistics are quite reasonable, I suspect the depthai_viewer is not well suited to these particular configuration above, and for a 4G use case I suppose I need to develop an app that compresses any video prior to sending it in an effort to improve the frame rate. In that case a OAK-D CM4 is probably a better candidate.
Mike

jakaskerl

mikegardner For example, if I get rid of the switch and replace it with a Ubiquiti 802.3AF PoE Injector that I purchased from your web store, connected directly into a Wifi router and then connect via Wifi to the laptop, using the Depthai Viewer I only get around 3 frames per second at 1080P, albeit with no dropout.

WIFI bottleneck is expected. Unless WiFi 5 or better the bandwidth is very limited.

The bandwidth for normal operations should be 1gbps. Otherwise connectivity issues are expected. The switch is too slow to handle data streaming.
https://docs.luxonis.com/hardware/platform/deploy/poe-deployment-guide/#PoE%20deployment%20guide-Runtime-Debugging

If you intend to use wireless transfer, the sending full streams is going to be a problem.

Thanks,
Jaka

mikegardner

Thanks Jaka, that's the conclusion I came to. The OAK 1 POE is suited well to direct POE Ethernet connections with a high bandwidth switch to a server or client. It doesn't perform well in a more conventional network topology, like an IP Camera does.
I think the better solution for my application is the OAK-D CM4, I can route the messages back via 4g with no bandwidth implications, and store the stream locally. When a message indicates an object of interest has been detected, I can stream back that section in an mp4 clip or similar.
Thanks for your input and help to understand the characteristics of the OAK1 POE product.
Mike