[PATCH v2 00/10] Introduce a new backend based on Nix and systemd
Daniel Gomez <[email protected]> Fri, 12 Jun 2026 12:55:12 +0200
| Newsgroups | dev.linux.lists.kdevops |
|---|---|
| Message-ID | <[email protected]> |
One of the headaches of using kdevops's libvirt + libguestfs backend
comes directly from distro changes that make kdevops non-reproducible
and a pain to fix. To alleviate the symptoms, and hopefully fix them
once and for all, introduce a new backend based on Nix and systemd.
Two standalone, independent projects were created: nixos-flake, hosted
at github.com/linux-kdevops/nixos-flake, and qemu-system-units (qsu),
hosted at github.com/linux-kdevops/qemu-system-units.
nixos-flake leverages Nix to provide truly reproducible environments,
images and closures through its declarative language and its flake
library of modules, packages and overlays. So far it provides two
backend modules for guests: libvirt and imageless. The project
also provides controller (for host setup, build, etc) and baremetal
environments but they are not still wired up in kdevops.
nixos-flake's libvirt backend module produces reproducible images that
libvirtd can consume, and which work as a replacement for libguestfs.
This work was introduced in commit 3089c3fe ("nixos: add NixOS support
as third bringup option with libvirt integration") and is now fully
integrated into nixos-flake as the libvirt backend module.
nixos-flake's imageless backend module can instead produce a
reproducible Nix system closure, a /nix/store path that a local guest
can consume and boot from without creating an image, enabling an even
faster development workflow.
This backend also produces a systemd initramfs from which the NixOS
guest is initialized through the Nix-generated fstab, mounting root
(tmpfs), /nix/store through the virtiofs store tag and /lib/modules
through the virtiofs modules tag. For this boot to work, the
kernel image requires the boot-critical virtio drivers built in
(CONFIG_VIRTIO_FS=y, CONFIG_VIRTIO_PCI=y and CONFIG_TMPFS=y). Every
other driver, even those used later during system boot, can be a kernel
module loaded from /lib/modules after the initramfs switch-root (pivot).
In order to boot this Nix system closure, we require QEMU and virtiofsd.
qemu-system-units (qsu) is used to generate the QEMU command line via
templates and to manage the VM lifecycle via systemd units. qsu removes
these extra layers of complexity and the opinionated QEMU management
of libvirt, offloading the work to templates and systemd. With that
complexity removed, users can now stay closer to the QEMU process and
benefit from easier management and no host privileges (once the user
is added to the kvm and systemd-journal groups), while still scaling up
through systemd-machined registration and a fully featured template. It
supports direct kernel boot for faster boots without tradeoffs (module
loading is supported), virtiofsd with automated process-management
dependency, 9p, vsock (a high-performance communication channel between
the hypervisor and guest virtual machines built on Linux AF_VSOCK), PCIe
passthrough (not yet wired up in kdevops), GDB, cloud-init (not needed
for imageless), UEFI/SeaBIOS, and process resource control.
With these two projects merged as git subtrees, kdevops integrates
them as the NIXOSFL (scripts/nixos-flake/modules/backends/libvirt),
NIXOSFI (scripts/nixos-flake/modules/backends/imageless) and QSU
(scripts/qemu-system-units) components, using Ansible just as a vehicle
to automate and run the commands that generate and deploy the QEMU
systemd service units and the commands that generate the Nix closures.
This means Ansible's idempotency pillar is no longer needed when Nix and
qsu are chosen as the backend. The series also adds a direct boot build
path to the bootlinux role, producing a controller-built kernel that qsu
boots the imageless guest from directly.
Another feature Nix enables is per-package source overrides: any tracked
package (fio, xfstests, xfsprogs, libbpf, ...) can be rebuilt from a
local checkout or a git URL without rebuilding the rest of the closure,
so a local change to a tool can be tested against an otherwise pinned
base.
Finally, the series gates off the test workflows (fstests, blktests,
...) and the monitors, limiting this backend to the bringup stage for
now. A Nix image or closure is built declaratively from the flake, so
once it boots there is nothing for kdevops's per-guest Ansible setup to
configure; even the libvirt qcow2 with its 9p share has no guest-side
step to run, and driving these paths against a Nix guest would only
fail. Wiring those workflows needs a clean controller/guest setup
separation that is not in place yet, so the series gates them rather
than attempt and fail.
Signed-off-by: Daniel Gomez <[email protected]>
---
Changes in v2:
- Rename QSU NVMe knobs
- Add QSU share support
- Add /lib/modules/<ver>/source symlink for libbpf-tools in bootlinux
direct-boot mode
- Update nixos-flake and qemu-system-units to latest
- Update nixos/ -> nixos/generated/ for nixos configuration dirs
- Link to v1: https://patch.msgid.link/[email protected]
---
Daniel Gomez (12):
Squashed 'scripts/nixos-flake/' content from commit b599d320
Merge commit 'a2ce139b0772a59dfbaa8ce50e261ae0dac86682' as 'scripts/nixos-flake'
Squashed 'scripts/qemu-system-units/' content from commit ecafb788
Merge commit '683e90afe566deb78606f2582ac1195675462973' as 'scripts/qemu-system-units'
bootlinux: add direct-boot build path
nixosfl: add the nixos-flake libvirt bringup backend
nixos: drop the legacy libvirt-NixOS backend
nixosfi: add the imageless NixOS bringup backend
nixos_flake: per-package source overrides
qsu: optional virtiofs share into the guest
nixosfi: add nixosfi-test target for operator-edited per-VM flakes
docs: add manual nixos-flake and qsu driving guide
.gitignore | 5 +
MAINTAINERS | 51 +
defconfigs/nixos | 27 -
defconfigs/nixosfi-nvme-4kn-atomics | 24 +
docs/kdevops-nixos.md | 404 ---
docs/nixos-flake-backends.md | 46 +
docs/nixos-flake-manual.md | 327 +++
docs/nixosfi.md | 51 +
docs/nixosfl.md | 46 +
docs/qsu.md | 37 +
kconfigs/Kconfig.ansible_provisioning | 31 +
kconfigs/Kconfig.bringup | 52 +-
kconfigs/Kconfig.bringup.goals | 3 +
kconfigs/Kconfig.kdevops | 16 +
kconfigs/Kconfig.nixos | 130 -
kconfigs/Kconfig.nixos_flake | 367 +++
kconfigs/Kconfig.qsu | 587 ++++
kconfigs/monitors/Kconfig | 6 +
kconfigs/workflows/Kconfig | 40 +-
nixos/flake.nix | 32 -
playbooks/bootlinux.yml | 25 +
playbooks/nixos.yml | 540 ----
playbooks/nixosfi.yml | 43 +
playbooks/nixosfl.yml | 36 +
.../roles/bootlinux/tasks/build/direct-boot.yml | 207 ++
playbooks/roles/bootlinux/tasks/config.yml | 8 +
.../roles/bootlinux/templates/config-imageless | 2957 ++++++++++++++++++++
playbooks/roles/devconfig/tasks/main.yml | 4 +-
playbooks/roles/gen_hosts/templates/fstests.j2 | 10 +-
.../roles/gen_hosts/templates/workflows/linux.j2 | 6 +-
.../roles/gen_hosts/templates/workflows/mix.j2 | 8 +-
playbooks/roles/gen_nodes/tasks/main.yml | 28 +-
playbooks/roles/gen_nodes/templates/nixos_nodes.j2 | 14 -
.../roles/nixos_flake/tasks/runtime-deps/setup.yml | 50 +
.../nixos_flake/tasks/runtime-deps/verify.yml | 29 +
playbooks/roles/nixosfi/tasks/generate-configs.yml | 71 +
.../roles/nixosfi/tasks/imageless-build-per-vm.yml | 40 +
playbooks/roles/nixosfi/tasks/imageless-build.yml | 67 +
playbooks/roles/nixosfi/tasks/main.yml | 87 +
playbooks/roles/nixosfi/tasks/rebuild-boot.yml | 27 +
playbooks/roles/nixosfi/tasks/rebuild-test.yml | 44 +
.../nixosfi/tasks/refresh-source-overrides.yml | 41 +
playbooks/roles/nixosfi/tasks/test.yml | 44 +
playbooks/roles/nixosfi/templates/default.nix.j2 | 110 +
playbooks/roles/nixosfi/templates/flake.nix.j2 | 77 +
playbooks/roles/nixosfl/README.md | 12 +
playbooks/roles/nixosfl/tasks/bringup.yml | 27 +
playbooks/roles/nixosfl/tasks/console.yml | 28 +
playbooks/roles/nixosfl/tasks/destroy.yml | 80 +
playbooks/roles/nixosfl/tasks/generate-configs.yml | 90 +
playbooks/roles/nixosfl/tasks/libvirt-build.yml | 45 +
playbooks/roles/nixosfl/tasks/libvirt-network.yml | 28 +
.../roles/nixosfl/tasks/libvirt-provision.yml | 60 +
playbooks/roles/nixosfl/tasks/main.yml | 34 +
playbooks/roles/nixosfl/tasks/ssh-access.yml | 70 +
playbooks/roles/nixosfl/templates/default.nix.j2 | 116 +
playbooks/roles/nixosfl/templates/flake.nix.j2 | 84 +
.../nixosfl/templates}/vm-libvirt.xml.j2 | 5 -
playbooks/roles/qsu/tasks/bringup.yml | 105 +
playbooks/roles/qsu/tasks/console.yml | 23 +
playbooks/roles/qsu/tasks/destroy.yml | 134 +
playbooks/roles/qsu/tasks/install-deps.yml | 31 +
playbooks/roles/qsu/tasks/render-per-vm.yml | 288 ++
playbooks/roles/qsu/tasks/render-units.yml | 55 +
playbooks/roles/qsu/tasks/restart-vms.yml | 55 +
playbooks/roles/qsu/tasks/start-vms.yml | 46 +
playbooks/roles/qsu/tasks/stop-vms.yml | 25 +
playbooks/roles/qsu/templates/debug-vars.yaml.j2 | 175 ++
playbooks/roles/qsu/vars/main.yml | 27 +
playbooks/roles/update_etc_hosts/tasks/main.yml | 4 +-
playbooks/templates/nixos/configuration.nix.j2 | 138 -
playbooks/templates/nixos/flake.nix.j2 | 38 -
.../templates/nixos/hardware-configuration.nix.j2 | 42 -
playbooks/templates/nixos/run-vm-wrapper.sh.j2 | 159 --
playbooks/templates/nixos/vms.nix.j2 | 45 -
playbooks/templates/nixos/workflow-deps.nix.j2 | 127 -
playbooks/update_ssh_config_nixos.yml | 57 -
scripts/bringup.Makefile | 12 +
scripts/nixos-flake/.editorconfig | 12 +
scripts/nixos-flake/.gitignore | 2 +
scripts/nixos-flake/CLAUDE.md | 244 ++
scripts/nixos-flake/COPYING | 22 +
scripts/nixos-flake/LICENSE | 5 +
.../LICENSES/preferred/copyleft-next-0.3.1 | 239 ++
scripts/nixos-flake/README.md | 179 ++
scripts/nixos-flake/docs/design-decisions.md | 393 +++
scripts/nixos-flake/docs/usage.md | 802 ++++++
scripts/nixos-flake/docs/verifying.md | 72 +
scripts/nixos-flake/flake.lock | 27 +
scripts/nixos-flake/flake.nix | 239 ++
scripts/nixos-flake/lib/toolchain.nix | 50 +
.../modules/backends/imageless/default.nix | 216 ++
.../modules/backends/imageless/user.nix | 24 +
.../modules/backends/libvirt/default.nix | 77 +
.../nixos-flake/modules/backends/libvirt/user.nix | 28 +
scripts/nixos-flake/modules/mounts/shares.nix | 49 +
scripts/nixos-flake/modules/mounts/storage.nix | 130 +
.../nixos-flake/modules/profiles/build-tools.nix | 24 +
.../nixos-flake/modules/profiles/controller.nix | 49 +
scripts/nixos-flake/modules/profiles/devel.nix | 125 +
.../nixos-flake/modules/profiles/monitoring.nix | 464 +++
.../nixos-flake/modules/testSuites/blktests.nix | 30 +
scripts/nixos-flake/modules/testSuites/fstests.nix | 370 +++
scripts/nixos-flake/modules/testSuites/gitr.nix | 22 +
scripts/nixos-flake/modules/testSuites/ltp.nix | 34 +
scripts/nixos-flake/modules/testSuites/mmtests.nix | 24 +
scripts/nixos-flake/modules/testSuites/pynfs.nix | 22 +
.../nixos-flake/modules/testSuites/selftests.nix | 26 +
.../nixos-flake/modules/testSuites/sysbench.nix | 27 +
scripts/nixos-flake/modules/user-options.nix | 38 +
scripts/nixos-flake/modules/user.nix | 38 +
scripts/nixos-flake/overlays/default.nix | 17 +
scripts/nixos-flake/overlays/fio.nix | 36 +
scripts/nixos-flake/overlays/spdk.nix | 17 +
scripts/nixos-flake/overlays/xfstests.nix | 45 +
scripts/nixos-flake/pkgs/cpupower.nix | 65 +
scripts/nixos-flake/pkgs/damo.nix | 63 +
scripts/nixos-flake/pkgs/default.nix | 21 +
scripts/nixos-flake/pkgs/libbpf-tools.nix | 186 ++
scripts/nixos-flake/pkgs/nfstest.nix | 88 +
scripts/nixos-flake/pkgs/pynfs.nix | 96 +
scripts/nixos-flake/pkgs/xnvme.nix | 57 +
.../nixos-flake/templates/imageless/default.nix | 41 +
scripts/nixos-flake/templates/imageless/flake.nix | 70 +
scripts/nixos-flake/templates/libvirt/default.nix | 41 +
scripts/nixos-flake/templates/libvirt/flake.nix | 78 +
scripts/nixos.Makefile | 93 -
scripts/nixosfi.Makefile | 135 +
...xos_ssh_key_name.py => nixosfi_ssh_key_name.py} | 0
scripts/nixosfl.Makefile | 104 +
scripts/nixosfl_ssh_key_name.py | 55 +
scripts/provision.Makefile | 28 +-
scripts/qemu-system-units/.gitignore | 15 +
scripts/qemu-system-units/CLAUDE.md | 257 ++
scripts/qemu-system-units/COPYING | 22 +
scripts/qemu-system-units/LICENSE | 5 +
.../LICENSES/preferred/copyleft-next-0.3.1 | 239 ++
scripts/qemu-system-units/README.md | 86 +
scripts/qemu-system-units/docs/design-decisions.md | 715 +++++
scripts/qemu-system-units/docs/requirements.md | 56 +
scripts/qemu-system-units/docs/transient-units.md | 244 ++
scripts/qemu-system-units/docs/usage.md | 461 +++
scripts/qemu-system-units/docs/vars.md | 541 ++++
scripts/qemu-system-units/docs/verifying.md | 60 +
scripts/qemu-system-units/files/network-config | 18 +
scripts/qemu-system-units/files/qmp-powerdown | 2 +
scripts/qemu-system-units/files/vfio-pci.conf | 1 +
scripts/qemu-system-units/templates/meta-data.j2 | 3 +
scripts/qemu-system-units/templates/nvme.env.j2 | 143 +
.../templates/qemu-system-override.conf.j2 | 88 +
.../templates/[email protected] | 71 +
.../templates/transient-run.sh.j2 | 72 +
scripts/qemu-system-units/templates/user-data.j2 | 44 +
.../templates/[email protected] | 21 +
.../qemu-system-units/templates/vfio-udev.rules.j2 | 20 +
.../templates/virtiofsd-override.conf.j2 | 43 +
.../qemu-system-units/templates/virtiofsd.env.j2 | 11 +
.../templates/[email protected] | 47 +
.../templates/[email protected] | 24 +
scripts/qemu-system-units/templates/vm.env.j2 | 156 ++
scripts/qemu-system-units/vars/example.yaml | 112 +
scripts/status_nixos.sh | 57 -
scripts/update_ssh_config_nixos.py | 133 -
scripts/update_ssh_config_nixosfi.py | 216 ++
scripts/update_ssh_config_nixosfl.py | 216 ++
workflows/linux/Kconfig | 70 +
workflows/linux/Makefile | 12 +-
167 files changed, 16732 insertions(+), 2108 deletions(-)
---
base-commit: a81b18438512916a3cc01a8215442db855bb2bbb
change-id: 20260526-new-backend-nix-systemd-4acc8f6fda8a
Best regards,
--
Daniel Gomez <[email protected]>