[walnascar] systemd-networkd-persistent-storage fails during boot, using read-only-rootfs and overlayfs-etc image features

"Martin Siegumfeldt" <[email protected]> Fri, 04 Jul 2025 03:27:31 -0700
Newsgroups org.yoctoproject.lists.poky
Message-ID <[email protected]>
Hi all,

We have been using read-only-rootfs and overlayfs-etc image features for the last few Yocto baselines most recently on styhead without any issues. Moving on to Walnascar we intermittently observe systemd-networkd-persistent-storage service to fail:

systemd-networkd-persistent-storage.service - Enable Persistent Storage in systemd-networkd
Loaded: loaded (]8;;file://nanomind-z7020-zed/usr/lib/systemd/system/systemd-networkd-persistent-storage.service\/usr/lib/systemd/system/systemd-networkd-persistent-storage.service]8;;\; static)
Active: failed (Result: exit-code) since Thu 2025-05-29 16:30:37 UTC; 1 month 1 day ago
Invocation: 9c37effe85404b6c81ecc9195defb5ff
Docs: ]8;;man:networkctl(1)\man:networkctl(1)]8;;\
Process: 162 ExecStart=networkctl persistent-storage yes (code=exited, status=238/STATE_DIRECTORY)
Main PID: 162 (code=exited, status=238/STATE_DIRECTORY)
CPU: 33ms
May 29 16:30:37 nanomind-z7020-zed systemd[1]: Starting Enable Persistent Storage in systemd-networkd...
May 29 16:30:37 nanomind-z7020-zed (tworkctl)[162]: systemd-networkd-persistent-storage.service: Failed to set up special execution directory in /var/lib: Read-only file system
May 29 16:30:37 nanomind-z7020-zed (tworkctl)[162]: systemd-networkd-persistent-storage.service: Failed at step STATE_DIRECTORY spawning networkctl: Read-only file system
May 29 16:30:37 nanomind-z7020-zed systemd[1]: systemd-networkd-persistent-storage.service: Main process exited, code=exited, status=238/STATE_DIRECTORY
May 29 16:30:37 nanomind-z7020-zed systemd[1]: systemd-networkd-persistent-storage.service: Failed with result 'exit-code'.
May 29 16:30:37 nanomind-z7020-zed systemd[1]: Failed to start Enable Persistent Storage in systemd-networkd.

Looking into the most/direct relevant systemd files from styhead version (v256.5) to walnascar version (v257.6), no obvious related changes appear. It occurs only sporadicly (for 1/5'ish boots) and I thus suspect some sort of race condition to occur. It is observed on an Avnet Zedboard connected to a corporate ethernet network configured for DHCP. overlayfs-etc is configured using:
OVERLAYFS_ETC_USE_ORIG_INIT_NAME = "0"
OVERLAYFS_ETC_CREATE_MOUNT_DIRS = "0"

Anyone else observed this, or has clues for further debug?

Thanks,
Martin