I have found an issue after rebooting 3 OAK4-S cameras (R9, OS RVC4 1.27.1, oak-agent 0.18.3) that caused the /etc/resolv.conf file to be permanently empty within my app's container.
From within the app container, /etc/resolv.conf was a 0-byte file on all 3 devices, so every DNS lookup failed with [Errno -3] Temporary failure in name resolution while the device otherwise looked completely healthy. It's caused by a race condition during boot, combined with the fact that NetworkManager uses an atomic rename to write resolv.conf on the host side.
This is the order of things that had happened:
- Devices reboot.
oak-agent started the app container within ~60s. The container's runc config bind-mounts the host's /etc/resolv.conf into the container (read-only). On the host that path is a symlink to /var/run/resolv.conf, which didn't exist yet because NetworkManager hadn't obtained DHCP, so what actually got mounted was an empty placeholder file.- NetworkManager correctly writes the host-side
resolv.conf at ~80s after boot, but it writes via atomic rename, which replaces the inode. The container's bind mount stays pinned to the old, now-deleted, empty inode forever. Inside the container, /proc/self/mountinfo shows the mount source as /resolv.conf//deleted. The container never sees DNS again until it is recreated.
Note that this isn't just a boot-time occurrence: bind-mounting the file means any resolv.conf rewrite leaves running containers pinned to the old file.
I checked a device on OS 1.37.0 / agent 0.25.0: the bundle config still bind-mounts /etc/resolv.conf the same way, but newer OS versions use systemd-resolved, whose stub-resolv.conf is written early in boot with static content, so the race is much narrower so is unlikely to trigger, but the changes to host resolv.conf still won't be propagated to the app.
A possible solution would be for oak-agent to keep resolv.conf in sync. As far as I know, Docker and Podman write their own resolv.conf within app containers and watch the host-side file for changes to avoid this exact issue.
