We are working with a few OAK 4 S (Wide) cameras installed at a client's facility, running a single, custom OAK App. We have been experiencing issues where the devices shutdown, and have not been able to determine a cause. The client's IT staff have not found any issues with their infrastructure, and we have not found any issues with the software or camera. We have found some logs like this in dmesg: PMIC PON log: Fundamental Reset: PON_PBL_STATUS=XVDD, DVDD PMIC PON log: FAULT_REASON2=RESTART_PON Some online searches suggest that might indicate a restart due to low voltage, but we have not found a reliable source for that, including in Luxonis documentation or these forums. **Question:** Do you have any suggestions for collecting device logs from the OAK 4 S, and *which* logs would be best to look at? **Logs/Commands** From our own investigation, we have found the following logs/commands, none of which have exposed a root cause: Files/logs: * `/proc/loadavg` - 1/5/15-min load averages * `/proc/uptime` - system uptime * `/proc/meminfo` - specifically the MemAvailable: line * `/sys/class/thermal/thermal_zone*/type` and `/sys/class/thermal/thermal_zone*/temp` - one pair per populated thermal zone * `/sys/class/net/eth0/carrier` - link state * `/sys/class/net/eth0/carrier_changes` - link flap counter * `/sys/class/net/eth0/statistics/rx_errors` and `/sys/class/net/eth0/statistics/tx_errors` * `/overlay/persistent-logs/kern.log` (+ .1, .2.gz) - rsyslog's continuous kernel-message mirror * `/overlay/persistent-logs/syslog` (+ .1, .2.gz) - rsyslog's full syslog mirror, which also captures oak-agent's/agentconfd's own stdout Commands: * `dmesg` - wiped on reboot * `journalctl --list-boots` - reboot history (though on both devices this ended up showing only the current boot, since `journald` here is volatile-storage and every restart's time-jump artifact triggers a rotation) * `journalctl -u oak-agent --since ... --until ...` - the oak-agent daemon's log (startup mode line, mDNS announcements, Hub-token warnings) * `journalctl -u agentconfd --since ... --until ...` - the agentconfd daemon's log (NotSetup/SetupRequest busy-loop signature) * `journalctl -k --since ... --until ...` - kernel journal (OOM, watchdog, thermal, carrier-loss greps) * `systemctl status oak-agent agentconfd --no-pager` - current daemon health * `uptime, date -u` - current boot age / clock sanity * `df -h` - disk usage, specifically /overlay's free space (the real, writable pool behind /etc, /data, /home, etc.) **Potential Solution** One thing we're looking into doing is having another device on the network periodically pull/poll the logs of all the OAK devices and/or run some of the above commands. That would help us retain logs that are wiped whenever an OAK reboots (e.g. dmesg), and get "snapshots" of each OAKs current state. However, we need to be pulling the correct logs and running the correct commands for it to actually help. **Device Details** All devices are OAK 4 S (Wide)'s. They're on a mix of OS versions: 1.33.0, 1.35.0, & 1.37.0 (now that I see that, I'll get them all up to the same version). Thank you.

Hey @"caavang"#p34483 - your external polling approach is a good next step. The PMIC PON entries confirm that a reset was recorded, but they do not identify low input voltage as the root cause by themselves. For OAK4, please first confirm the PoE source: OAK4 needs PoE+ (802.3at, 30 W); standard PoE (802.3af, 15 W) is not sufficient. Please check the switch/injector port power and event logs, cable, and port negotiation. A stable 1-Gbps link does not confirm stable power delivery. [OAK4 power guidance](https://docs.luxonis.com/hardware/platform/deploy/oak4-deployment-guide/oak4-getting-started) For each device, I recommend collecting: - `/overlay/persistent-logs/kern.log*`, `syslog*`, and `logcat.log*` - the app output: `oakctl app logs --no-follow` - after a reboot, `cat /sys/kernel/debug/ipc_logging/pmic_pon/log`, if that path exists - a timestamped snapshot of uptime, free `/overlay` space, thermal zones, and Ethernet error/link-change counters Please search the kernel logs for `watchdog`, `thermal`, `OOM`, `panic`, `SFR`, `remoteproc`, `qcom_q6v5`, `PMIC PON`, and `reset`. Also, please update all devices to the current latest stable Luxonis OS rather than only making their versions equal. OAK4 is specified for -20 °C to +50 °C ambient operation, so please include ambient temperature and enclosure conditions with the log bundle. [Operating-temperature guidance](https://docs.luxonis.com/hardware/platform/environmental-specifications/operating-temperature) Thanks, Oskar

Thank you for the responses, @"Oskar-Luxonis"#3466 This is more than enough info to get us started.

@"caavang"#p34525 - happy to help.

Tips for troubleshooting OAK 4 S shutdowns

caavang

We are working with a few OAK 4 S (Wide) cameras installed at a client's facility, running a single, custom OAK App. We have been experiencing issues where the devices shutdown, and have not been able to determine a cause. The client's IT staff have not found any issues with their infrastructure, and we have not found any issues with the software or camera.

We have found some logs like this in dmesg:
PMIC PON log: Fundamental Reset: PON_PBL_STATUS=XVDD, DVDD
PMIC PON log: FAULT_REASON2=RESTART_PON

Some online searches suggest that might indicate a restart due to low voltage, but we have not found a reliable source for that, including in Luxonis documentation or these forums.

Question: Do you have any suggestions for collecting device logs from the OAK 4 S, and which logs would be best to look at?

Logs/Commands

From our own investigation, we have found the following logs/commands, none of which have exposed a root cause:

Files/logs:

/proc/loadavg - 1/5/15-min load averages
/proc/uptime - system uptime
/proc/meminfo - specifically the MemAvailable: line
/sys/class/thermal/thermal_zone*/type and /sys/class/thermal/thermal_zone*/temp - one pair per populated thermal zone
/sys/class/net/eth0/carrier - link state
/sys/class/net/eth0/carrier_changes - link flap counter
/sys/class/net/eth0/statistics/rx_errors and /sys/class/net/eth0/statistics/tx_errors
/overlay/persistent-logs/kern.log (+ .1, .2.gz) - rsyslog's continuous kernel-message mirror
/overlay/persistent-logs/syslog (+ .1, .2.gz) - rsyslog's full syslog mirror, which also captures oak-agent's/agentconfd's own stdout

Commands:

dmesg - wiped on reboot
journalctl --list-boots - reboot history (though on both devices this ended up showing only the current boot, since journald here is volatile-storage and every restart's time-jump artifact triggers a rotation)
journalctl -u oak-agent --since ... --until ... - the oak-agent daemon's log (startup mode line, mDNS announcements, Hub-token warnings)
journalctl -u agentconfd --since ... --until ... - the agentconfd daemon's log (NotSetup/SetupRequest busy-loop signature)
journalctl -k --since ... --until ... - kernel journal (OOM, watchdog, thermal, carrier-loss greps)
systemctl status oak-agent agentconfd --no-pager - current daemon health
uptime, date -u - current boot age / clock sanity
df -h - disk usage, specifically /overlay's free space (the real, writable pool behind /etc, /data, /home, etc.)

Potential Solution

One thing we're looking into doing is having another device on the network periodically pull/poll the logs of all the OAK devices and/or run some of the above commands. That would help us retain logs that are wiped whenever an OAK reboots (e.g. dmesg), and get "snapshots" of each OAKs current state.

However, we need to be pulling the correct logs and running the correct commands for it to actually help.

Device Details

All devices are OAK 4 S (Wide)'s. They're on a mix of OS versions: 1.33.0, 1.35.0, & 1.37.0 (now that I see that, I'll get them all up to the same version).

Thank you.

Oskar-Luxonis

Hey caavang - your external polling approach is a good next step. The PMIC PON entries confirm that a reset was recorded, but they do not identify low input voltage as the root cause by themselves.

For OAK4, please first confirm the PoE source: OAK4 needs PoE+ (802.3at, 30 W); standard PoE (802.3af, 15 W) is not sufficient. Please check the switch/injector port power and event logs, cable, and port negotiation. A stable 1-Gbps link does not confirm stable power delivery. OAK4 power guidance

For each device, I recommend collecting:

/overlay/persistent-logs/kern.log*, syslog*, and logcat.log*
the app output: oakctl app logs <app-id> --no-follow
after a reboot, cat /sys/kernel/debug/ipc_logging/pmic_pon/log, if that path exists
a timestamped snapshot of uptime, free /overlay space, thermal zones, and Ethernet error/link-change counters

Please search the kernel logs for watchdog, thermal, OOM, panic, SFR, remoteproc, qcom_q6v5, PMIC PON, and reset.

Also, please update all devices to the current latest stable Luxonis OS rather than only making their versions equal. OAK4 is specified for -20 °C to +50 °C ambient operation, so please include ambient temperature and enclosure conditions with the log bundle. Operating-temperature guidance

Thanks,
Oskar

caavang

Hello @Oskar-Luxonis, thank you for your response.

One new question, then I'll respond to specific parts of your message.

Question: We recently had an incident where a network switch reported OAKs were attempting to draw too much power, so it would disable power on their ports. Is there a file, log, or command that can be used to get the amount of power the OAK is attempting to draw, from the OAK's perspective? Could we use the summed values of /sys/class/hwmon/hwmon0/power1_input & /sys/class/hwmon/hwmon1/power1_input?

Responses:

The PMIC PON entries confirm that a reset was recorded, but they do not identify low input voltage as the root cause by themselves.

Noted.

For OAK4, please first confirm the PoE source: […]

This was an issue we addressed in the past. The switch in-use was only providing 15W. That was remedied, but we were still seeing some unexpected shutdowns after that was fixed.

[Y]your external polling approach is a good next step.
[…]
For each device, I recommend collecting: […]
Please search the kernel logs for […]

Thank you. We'll be sure to focus on those logs and search terms.

Also, please update all devices to the current latest stable Luxonis OS rather than only making their versions equal.

Noted. All devices have been updated to Luxonis OS 1.40.0.

OAK4 is specified for -20 °C to +50 °C ambient operation, so please include ambient temperature and enclosure conditions with the log bundle.

Noted. These OAKs are deployed outdoors, in Florida, USA. In the summer (right now), it is consistently ~33 C, but they are exposed to sunlight and it can be quite humid, so their temperature is likely higher than 33 C.

Thank you again.

Oskar-Luxonis

Hey caavang - thanks for the additional information. It is good that all devices are now on Luxonis OS 1.40.0.

For the switch power-limit event: OAK4-S can have power peaks of about 25 W, with a maximum average consumption of about 10–15 W. It therefore requires a PoE+ (802.3at, 30 W) switch or injector for full-power operation.

The switch port log is the best source to identify a PoE power-limit or over-current event. Please check the port’s negotiated PoE class, per-port power limit, total switch power budget, and the timestamp of any power disable event. A switch can disable a port faster than periodic device polling can record the event.

Thanks,
Oskar

caavang

Thank you for the responses, @Oskar-Luxonis This is more than enough info to get us started.

Oskar-Luxonis

caavang - happy to help.