OAK 1 POE Severe Dropout

mikegardner
Here's a sample of the ping responses, IMHO they don't suggest a network issue...

PING 192.168.1.133 (192.168.1.133) 56(84) bytes of data.
64 bytes from 192.168.1.133: icmp_seq=1 ttl=64 time=241 ms
64 bytes from 192.168.1.133: icmp_seq=2 ttl=64 time=4.16 ms
64 bytes from 192.168.1.133: icmp_seq=3 ttl=64 time=3.64 ms
64 bytes from 192.168.1.133: icmp_seq=4 ttl=64 time=8.02 ms
64 bytes from 192.168.1.133: icmp_seq=5 ttl=64 time=3.86 ms
64 bytes from 192.168.1.133: icmp_seq=6 ttl=64 time=4.24 ms
64 bytes from 192.168.1.133: icmp_seq=7 ttl=64 time=3.78 ms
64 bytes from 192.168.1.133: icmp_seq=8 ttl=64 time=3.30 ms
64 bytes from 192.168.1.133: icmp_seq=9 ttl=64 time=14.9 ms
64 bytes from 192.168.1.133: icmp_seq=10 ttl=64 time=3.86 ms
64 bytes from 192.168.1.133: icmp_seq=11 ttl=64 time=4.27 ms
64 bytes from 192.168.1.133: icmp_seq=12 ttl=64 time=3.36 ms
64 bytes from 192.168.1.133: icmp_seq=13 ttl=64 time=3.16 ms
64 bytes from 192.168.1.133: icmp_seq=14 ttl=64 time=3.92 ms
64 bytes from 192.168.1.133: icmp_seq=15 ttl=64 time=2.95 ms
64 bytes from 192.168.1.133: icmp_seq=16 ttl=64 time=3.47 ms
64 bytes from 192.168.1.133: icmp_seq=17 ttl=64 time=2.60 ms
64 bytes from 192.168.1.133: icmp_seq=18 ttl=64 time=7.16 ms
64 bytes from 192.168.1.133: icmp_seq=19 ttl=64 time=2.65 ms
64 bytes from 192.168.1.133: icmp_seq=20 ttl=64 time=2.73 ms
64 bytes from 192.168.1.133: icmp_seq=21 ttl=64 time=4.29 ms
64 bytes from 192.168.1.133: icmp_seq=22 ttl=64 time=3.84 ms
64 bytes from 192.168.1.133: icmp_seq=23 ttl=64 time=2.77 ms
64 bytes from 192.168.1.133: icmp_seq=24 ttl=64 time=3.12 ms
64 bytes from 192.168.1.133: icmp_seq=25 ttl=64 time=2.62 ms
64 bytes from 192.168.1.133: icmp_seq=26 ttl=64 time=2.43 ms
64 bytes from 192.168.1.133: icmp_seq=27 ttl=64 time=4.75 ms

OK - I've deleted all of the folders and recloned them and re-installed depthai

This has enabled me to use the device manager to update the firmware to 0.0.28 but it still behaves the same way. I've tried the depthai_viewer using the USB C port connected to my laptop and everything works fine.

I'll try again with a different POE switch or injector.

Thanks Jaka

depthai_viewer Version is

0.2.4 [rustc 1.74.1 (a28077b28 2023-12-04), LLVM 17.0.4] x86_64-unknown-linux-gnu bf0ca8b, built 2024-06-21T10:37:38Z

With the USB C connection I get the following report when launching
MESA-INTEL: warning: Haswell Vulkan support is incomplete
Backend started successfully.
[2024-09-23T10:24:32Z INFO ewebsock::native_tungstenite] WebSocket handshake has been successfully completed
[2024-09-23T10:24:32Z INFO re_viewer::depthai::ws] Websocket opened
Selecting device: 18443010B1E752F500
Connecting to viewer at 127.0.0.1:9876
Oak cam: <depthai_sdk.oak_camera.OakCamera object at 0x792dc8101120>
Default stereo pair: None
Pipeline config: auto=True cameras=[CameraConfiguration(fps=30, resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'>, kind=<CameraSensorType.COLOR: 0>, board_socket=<CameraBoardSocket.CAM_A: 0>, stream_enabled=True, name='Color', tof_align=None)] stereo=None ai_model=AiModelConfiguration(display_name='Yolo V6', path='yolov6nr3_coco_640x352', camera=<CameraBoardSocket.CAM_A: 0>) imu=ImuConfiguration(report_rate=100, batch_report_threshold=5)
Updating pipeline...
Creating auto pipeline config
Connected camera features: [{socket: CAM_A, sensorName: IMX378, width: 4056, height: 3040, orientation: AUTO, supportedTypes: [COLOR], hasAutofocus: 1, hasAutofocusIC: 1, name: color}]
Usb speed: UsbSpeed.SUPER
Creating camera: fps=30 resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'> kind=<CameraSensorType.COLOR: 0> board_socket=<CameraBoardSocket.CAM_A: 0> stream_enabled=True name='color' tof_align=None
Connected cam doesn't have IMU, skipping IMU creation...

Starting pipeline

[0s] System information

Ddr used / total - 96.89 / 333.28 MiB
Cmx used / total - 2.49 / 2.50 MiB
LeonCss heap used / total - 23.21 / 82.20 MiB
LeonMss heap used / total - 4.26 / 40.35 MiB
Chip temperature - average: 30.86, css: 33.21, mss: 29.59, upa: 30.08, dss: 30.56

Cpu usage - Leon CSS: 4.49 %, Leon MSS: 0.05 %

[10s] System information

Ddr used / total - 96.89 / 333.28 MiB
Cmx used / total - 2.49 / 2.50 MiB
LeonCss heap used / total - 24.10 / 82.20 MiB
LeonMss heap used / total - 4.92 / 40.35 MiB
Chip temperature - average: 35.17, css: 37.48, mss: 34.16, upa: 35.11, dss: 33.93
Cpu usage - Leon CSS: 42.71 %, Leon MSS: 26.02 %

I'll post the logs with the POE connection shortly.

Here are the logs when the device is connected via the POE switch.

MESA-INTEL: warning: Haswell Vulkan support is incomplete
Backend started successfully.
[2024-09-23T10:38:49Z INFO ewebsock::native_tungstenite] WebSocket handshake has been successfully completed
[2024-09-23T10:38:49Z INFO re_viewer::depthai::ws] Websocket opened
Selecting device: 18443010B1E752F500
Connecting to viewer at 127.0.0.1:9876
Oak cam: <depthai_sdk.oak_camera.OakCamera object at 0x7e5dbf06d150>
Default stereo pair: None
Pipeline config: auto=True cameras=[CameraConfiguration(fps=30, resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'>, kind=<CameraSensorType.COLOR: 0>, board_socket=<CameraBoardSocket.CAM_A: 0>, stream_enabled=True, name='Color', tof_align=None)] stereo=None ai_model=AiModelConfiguration(display_name='Yolo V6', path='yolov6nr3_coco_640x352', camera=<CameraBoardSocket.CAM_A: 0>) imu=ImuConfiguration(report_rate=100, batch_report_threshold=5)
Updating pipeline...
Creating auto pipeline config
Connected camera features: [{socket: CAM_A, sensorName: IMX378, width: 4056, height: 3040, orientation: AUTO, supportedTypes: [COLOR], hasAutofocus: 1, hasAutofocusIC: 1, name: color}]
Usb speed: UsbSpeed.UNKNOWN
Connected to a PoE device, camera streams will be JPEG encoded...
Creating camera: fps=30 resolution=<CameraSensorResolution.THE_1080_P: 'THE_1080_P'> kind=<CameraSensorType.COLOR: 0> board_socket=<CameraBoardSocket.CAM_A: 0> stream_enabled=True name='color' tof_align=None
Connected cam doesn't have IMU, skipping IMU creation...
Starting pipeline
Got message to send: <depthai_viewer.backend.messages.WarningMessage object at 0x7e5dbfaf7be0>

Sending message: <depthai_viewer._backend.messages.WarningMessage object at 0x7e5dbfaf7be0>

[0s] System information

Ddr used / total - 96.90 / 333.28 MiB
Cmx used / total - 2.49 / 2.50 MiB
LeonCss heap used / total - 41.00 / 82.20 MiB
LeonMss heap used / total - 4.36 / 40.35 MiB
Chip temperature - average: 31.64, css: 33.69, mss: 30.08, upa: 31.29, dss: 31.53
Cpu usage - Leon CSS: 16.14 %, Leon MSS: 0.04 %

[2024-09-23T10:39:55Z WARN re_viewer::depthai::depthai] Device is connected via PoE. This may cause performance issues.

[19s] System information

Ddr used / total - 109.08 / 333.28 MiB
Cmx used / total - 2.49 / 2.50 MiB
LeonCss heap used / total - 41.89 / 82.20 MiB
LeonMss heap used / total - 5.21 / 40.35 MiB
Chip temperature - average: 32.73, css: 34.40, mss: 31.77, upa: 32.97, dss: 31.77

Cpu usage - Leon CSS: 26.49 %, Leon MSS: 0.42 %

[27s] System information

Ddr used / total - 109.08 / 333.28 MiB
Cmx used / total - 2.49 / 2.50 MiB
LeonCss heap used / total - 41.89 / 82.20 MiB
LeonMss heap used / total - 5.21 / 40.35 MiB
Chip temperature - average: 33.50, css: 34.88, mss: 32.97, upa: 34.16, dss: 32.01
Cpu usage - Leon CSS: 27.33 %, Leon MSS: 0.42 %
^C[2024-09-23T10:40:41Z WARN re_sdk_comms::server] Closing connection to client: early eof

Hi Jaka
I've just seen your reply to syg regarding bandwidth.

Looking at the link https://docs.luxonis.com/software/depthai/optimizing/#Bandwidth

What Resolution FPS FPS set PoE Latency [ms] USB Latency [ms] Bandwidth
Color (isp) 1080P 25 25 51 33 Std: 0.8 622 Mbps

unless I'm misunderstanding the guidelines, my set-up at 30fps colour 1080P is likely to require in excess of 622 Mbps! That's almost certainly the problem, I'm using a Netgear Prosafe FS108P which is 10/100Mbps each port.
Thanks

OK
with a direct PC to camera ethernet connection, I get
Downlink 891.7 mbps
Uplink 228.9 mbps
Press any key to continue...

However, with a connection via the switch, I get
Downlink 356.1 mbps
Traceback (most recent call last):
File "/home/mike/Luxonis/depthai-python/utilities/oak_bandwidth_test.py", line 91, in <module>
qin.send(buffer)
RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Couldn't write data to stream: 'xin' (X_LINK_ERROR)'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/mike/Luxonis/depthai-python/utilities/oak_bandwidth_test.py", line 66, in <module>
with dai.Device(pipeline, maxUsbSpeed=dai.UsbSpeed.SUPER_PLUS) as device:
RuntimeError: Device already closed or disconnected: Input/output error
Stack trace (most recent call last):
#20 Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in
#19 Object "python3", at 0x64627243d604, in _start
#18 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7c47d1429e3f, in __libc_start_main
#17 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7c47d1429d8f, in
#16 Object "python3", at 0x64627243d70c, in Py_BytesMain
#15 Object "python3", at 0x646272467182, in Py_RunMain
#14 Object "python3", at 0x646272475cd7, in Py_FinalizeEx
#13 Object "python3", at 0x6462724791d5, in
#12 Object "python3", at 0x64627233d05e, in
#11 Object "python3", at 0x6462724796c3, in
#10 Object "python3", at 0x64627234b63b, in
#9 Object "python3", at 0x64627234b840, in
#8 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47ceea44e3, in
#7 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47ceecc356, in
#6 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf12f178, in dai:evice::_Device()
#5 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf12ed63, in dai:evice::_Device()
#4 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf13a7e4, in dai:eviceBase::close()
#3 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf132f04, in dai:evice::closeImpl()
#2 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf14f904, in dai:eviceBase::closeImpl()
#1 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf13ffaa, in dai:eviceBase::hasCrashDump()
#0 Object "/home/mike/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7c47cf17331d, in nanorpc::core::client<nanorpc::packer::nlohmann_msgpack>::result nanorpc::core::client<nanorpc::packer::nlohmann_msgpack>::call<>(unsigned long)
Segmentation fault (Address not mapped to object [0x10])
Segmentation fault (core dumped)

Hi Jaka,
Yes, I get a reasonable performance if I connect with a laptop directly to the USB C connector, or the directly to the ethernet connector. However in many applications, this is not a realistic use case, because there will be switches and routers between the client and the OAK 1 POE.
For example, if I get rid of the switch and replace it with a Ubiquiti 802.3AF PoE Injector that I purchased from your web store, connected directly into a Wifi router and then connect via Wifi to the laptop, using the Depthai Viewer I only get around 3 frames per second at 1080P, albeit with no dropout.
In this configuration, I get the following results from the bandwidth_test...
Downlink 48.3 mbps
Uplink 47.9 mbps

So in summary, (all at 1080P):

• Laptop connection via Wifi then the Netgear Prosafe FS 108P switch (10/100Mbps) = 1 frame displayed then drop-out

• Laptop connection to Wifi, then Ubiquiti POE injector from Wifi router to camera = 3 fps appx, no dropout

• Laptop connection via ethernet direct to the Netgear switch with the OAK also on the switch = 3 fps appx with occasional drop-out

• Laptop connection directly via Ethernet port to Ethernet port powered via USB C = 10 fps appx with occasional drop-out

• Laptop with direct USB C to USB C connection, seems like 30fps, no drop-out

My conclusion is that the performance is directly related to the connectivity, no surprises there I suppose.
However for many use cases there will rarely be direct Eth to Eth or USB to USB connectivity, and for my applicationI was hoping that I could use a 4g router connected to the camera and then a broadband connection from the laptop.

Given the raw network ping statistics are quite reasonable, I suspect the depthai_viewer is not well suited to these particular configuration above, and for a 4G use case I suppose I need to develop an app that compresses any video prior to sending it in an effort to improve the frame rate. In that case a OAK-D CM4 is probably a better candidate.
Mike

Thanks Jaka, that's the conclusion I came to. The OAK 1 POE is suited well to direct POE Ethernet connections with a high bandwidth switch to a server or client. It doesn't perform well in a more conventional network topology, like an IP Camera does.
I think the better solution for my application is the OAK-D CM4, I can route the messages back via 4g with no bandwidth implications, and store the stream locally. When a message indicates an object of interest has been detected, I can stream back that section in an mp4 clip or similar.
Thanks for your input and help to understand the characteristics of the OAK1 POE product.
Mike