Re: Architectural Review & Bug Report: Remote-Unlock ZFS via Reroot (ABI/VFS Inconsistency in QEMU/IDE)

On Sat, Dec 27, 2025 at 8:21 AM Alex <[email protected]> wrote:

> Dear FreeBSD Developers,
>
> I am developing a set of scripts for automated, production-grade FreeBSD
> installations (targeting 15.0-RELEASE) that prioritize full disk
> encryption (GELI) with remote SSH unlocking capabilities.
>
> We have encountered a persistent VFS/ACL inconsistency after a 'reboot -r'
> operation specifically in virtualized environments. We would like to
> present our architecture and seek your advice on its long-term viability.
>
> 1. Architecture: The "Remote-Unlock Reroot" Concept
> --------------------------------------------------
> To achieve a headless, encrypted-at-rest system that is reachable via
> network before the root is mounted, we implemented the following:
>
>   * Deployment: A specialized installer (install_freebsd.sh) creates a
>     Hybrid Boot (BIOS+UEFI) layout.
>   * Boot Partitions: A 512MB FAT32 ESP and a 1GB UFS Bootpool (mirrored
>     on multi-disk setups) are used as unencrypted staging areas.
>   * Boot Sequence:
>     1. loader.efi / gptzfsboot pulls the Kernel and a custom minimalist
>        Initramfs (MD image) from the unencrypted storage into memory.
>     2. The system boots the ramdisk (MD0). A script initializes DHCP
>        and starts a transient SSHD (on a custom port).
>     3. User unlocks via GELI, and the ZFS pool (zroot) is imported.
>     4. We execute 'reboot -r' to switch to the final ZFS dataset.
>

Seems totally sensible. It would be how I'd recommend it, with the possible
exception of
not bothering with BIOS unless you had legacy or weird VM environment
requirements.


> 2. Maintenance: Automated Lifecycle Management
> ----------------------------------------------
> A key part of our project is the 'update-initramfs' utility we generate
> on the target system. It handles the "Sync-Back" logic:
>
>   * Change Detection: Upon boot or manual trigger, it compares the
>     resident kernel version with /boot/kernel/kernel on ZFS.
>   * Automated Rebuild: If a mismatch is detected (e.g., after
>     freebsd-update), it rebuilds the MD-based Initramfs and re-injects
>     current SSH host keys and configuration.
>   * Multi-Target Sync: It automatically mounts all ESP and Bootpool
>     partitions and syncs the new Kernel and Initramfs image to ensure
>     the next hardware boot uses the updated stack.
>

Also very cool. Maintaining and updating the encrypted images is beyond my
paygrade usually, but this seems sensible. But with one caveat: since the
certs
are unencrypted, they are vulnerable to disclosure. That strikes me as
potentially
an issue, depending on what other measures you take.


> 3. The Bug: 'ls -l' EINVAL (QEMU vs. Bare Metal)
> ------------------------------------------------
> We observed a specific error post-update (e.g., from RELEASE to p1):
>
> Virtualized Environment (QEMU/KVM with IDE emulation):
>   * 'ls -l' on ZFS datasets fails for every entry with "Invalid argument".
>   * Path reporting shows double slashes: "ls: //COPYRIGHT: Invalid
> argument".
>   * truss: __acl_get_link(..., ACL_TYPE_ACCESS, ...) returns ERR#22
> 'EINVAL'.
>   * getfacl retrieves NFSv4 ACLs correctly on the same files.
>
> Hardware Environment (SATA/NVMe):
>   * Using the EXACT same software stack and architecture, the error
>     does NOT occur. 'ls -l' works perfectly.
>

Why IDE emulation in qemu? Maybe you could share the relevant bits of the
qemu
command line?


> 4. Architectural Questions & Request for Advice
> -----------------------------------------------
> 1. Why the EINVAL in QEMU? Does the 'reroot' operation interact
>    unfavorably with the 'ada' (IDE) driver stack or path normalization
>    in virtualized environments when an ABI mismatch (RELEASE kernel vs.
>    p1 userland) is present?
>

It shouldn't. All reroot does is unmount everything and then re-do a root
mount,
etc.  It's dependence on the kernel is minimal, as are most things it would
run.
release vs p1 should never matter.


> 2. Kernel Storage Strategy: We currently sync the kernel to both FAT32
>    and UFS partitions. Given the loss of Unix metadata on FAT32, is moving
>    strictly to a UFS-based Bootpool (pointing loader.efi there) the
>    strongly recommended path for FreeBSD?
>

Either will work. If you're booting encrypted ZFS datasets, then using UFS
would mean you could have a smaller FAT32, but an extra partition over a
FAT32-only solution that you pointed the BIOS boot loader at.


> 3. Reroot best practices: Given our 'reboot -r' approach, is there a
>    cleaner way to handle the transition to a new kernel version without
>    a full hardware power cycle, or is our "Update & Sync-Back"
>    logic the standard way to handle this?
>

You need at least some kind of reboot. While there's work ongoing to support
kexec, that's not yet in the FreeBSD kernel. You need some way to get the
unencrypted kernel updated and booted. And today, the only way to do that,
really, is to copy out the new kernel to an unencrypted store and then
reboot
using that kernel.


> We believe this "Remote-Unlock via Reroot" pattern is highly useful for
> automated cloud and remote server deployments. We would greatly
> appreciate any feedback on how to make this architecture more robust.
>

That's a very cool feature. There's not enough details for me to comment at
length about it though.


> Best regards,
>
> Alex Vai
>