[PATCH v2 04/10] nixosfl: add the nixos-flake libvirt bringup backend
Daniel Gomez <[email protected]> Fri, 12 Jun 2026 12:55:16 +0200
| Newsgroups | dev.linux.lists.kdevops |
|---|---|
| Message-ID | <[email protected]> |
From: Daniel Gomez <[email protected]> Add NIXOSFL, a "Node bring up method" that provisions guests as libvirt domains whose qcow2 disk image is built declaratively from the vendored nixos-flake library rather than from a distro image. Each node renders a per-node flake importing nixos-flake.nixosModules.backends.libvirt; kdevops builds the image, stages it into the libvirt storage pool, defines and starts the domain, and waits for SSH. The nixos-flake library is shared. kconfigs/Kconfig.nixos_flake holds the profile, mount, and testSuite module selection, and the nix install runtime-deps live in the nixos_flake role; both are consumed by this backend and reused by the imageless backend that follows. The nixosfl role is otherwise self-contained and reuses the existing libvirt infrastructure the same way GUESTFS does. Generated-by: Claude AI Signed-off-by: Daniel Gomez <[email protected]> --- MAINTAINERS | 16 ++ docs/nixosfl.md | 46 +++++ kconfigs/Kconfig.ansible_provisioning | 31 +++ kconfigs/Kconfig.bringup | 28 ++- kconfigs/Kconfig.bringup.goals | 3 + kconfigs/Kconfig.kdevops | 1 + kconfigs/Kconfig.nixos_flake | 167 ++++++++++++++++ kconfigs/monitors/Kconfig | 9 + kconfigs/workflows/Kconfig | 39 ++-- playbooks/nixosfl.yml | 36 ++++ .../roles/gen_hosts/templates/workflows/linux.j2 | 6 +- playbooks/roles/gen_nodes/tasks/main.yml | 5 +- .../roles/nixos_flake/tasks/runtime-deps/setup.yml | 50 +++++ .../nixos_flake/tasks/runtime-deps/verify.yml | 29 +++ playbooks/roles/nixosfl/README.md | 12 ++ playbooks/roles/nixosfl/tasks/bringup.yml | 27 +++ playbooks/roles/nixosfl/tasks/console.yml | 28 +++ playbooks/roles/nixosfl/tasks/destroy.yml | 80 ++++++++ playbooks/roles/nixosfl/tasks/generate-configs.yml | 90 +++++++++ playbooks/roles/nixosfl/tasks/libvirt-build.yml | 45 +++++ playbooks/roles/nixosfl/tasks/libvirt-network.yml | 28 +++ .../roles/nixosfl/tasks/libvirt-provision.yml | 60 ++++++ playbooks/roles/nixosfl/tasks/main.yml | 34 ++++ playbooks/roles/nixosfl/tasks/ssh-access.yml | 70 +++++++ playbooks/roles/nixosfl/templates/default.nix.j2 | 85 ++++++++ playbooks/roles/nixosfl/templates/flake.nix.j2 | 48 +++++ .../roles/nixosfl/templates/vm-libvirt.xml.j2 | 91 +++++++++ scripts/bringup.Makefile | 12 ++ scripts/nixosfl.Makefile | 104 ++++++++++ scripts/nixosfl_ssh_key_name.py | 55 ++++++ scripts/provision.Makefile | 24 +++ scripts/update_ssh_config_nixosfl.py | 216 +++++++++++++++++++++ 32 files changed, 1556 insertions(+), 19 deletions(-) diff --git a/MAINTAINERS b/MAINTAINERS index 5b9a1f23..f71da5de 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -143,6 +143,22 @@ T: git https://github.com/linux-kdevops/kdevops.git F: workflows/nfstest/ F: playbooks/roles/nfstest/ +NIXOS FLAKE +M: Daniel Gomez <[email protected]> +L: [email protected] +S: Maintained +T: git https://github.com/linux-kdevops/kdevops.git +F: kconfigs/Kconfig.nixos_flake +F: docs/nixosfl.md +F: playbooks/nixosfl.yml +F: playbooks/roles/nixosfl/ +F: playbooks/roles/nixos_flake/ +F: scripts/nixosfl.Makefile +F: scripts/nixosfl_ssh_key_name.py +F: scripts/update_ssh_config_nixosfl.py +F: scripts/nixos-flake/ +N: nixos[-_]flake + PYNFS WORKFLOW M: Chuck Lever <[email protected]> L: [email protected] diff --git a/docs/nixosfl.md b/docs/nixosfl.md new file mode 100644 index 00000000..02fee2fd --- /dev/null +++ b/docs/nixosfl.md @@ -0,0 +1,46 @@ +# NIXOSFL — nixos-flake libvirt backend + +`NIXOSFL` brings guests up as libvirt domains whose root disk is a qcow2 +image built declaratively from the nixos-flake library, instead of from a +distro image. It is the Nix counterpart of `GUESTFS`: the same libvirtd +runtime, a reproducible flake-built image. See +[nixos-flake-backends.md](nixos-flake-backends.md) for how it relates to +the imageless `NIXOSFI` backend. + +## Enable + +``` +make menuconfig # Node bring up method -> NixOS declarative configuration with libvirt +``` + +or in a defconfig, `CONFIG_NIXOSFL=y`. The libvirt-specific knobs are +`NIXOSFL_CONFIG_DIR` (where per-guest flakes render) and +`NIXOSFL_STORAGE_DIR` (where the built qcow2 is staged into the libvirt +pool). Guest contents — profiles, mounts, testSuites — come from the +shared `NIXOS_FLAKE_*` options. + +## Bring up + +``` +make +make bringup # build each guest's qcow2, define + start the domain, wait for SSH +ssh <guest> +``` + +`make bringup` runs `playbooks/nixosfl.yml`: it renders a per-guest +`flake.nix`/`default.nix` under `NIXOSFL_CONFIG_DIR`, runs `nix build` +for the `image` output, stages the qcow2 into the libvirt pool, defines +and starts the domain, and registers it in `~/.ssh/config`. + +## Inspect and tear down + +``` +virsh list --all +virsh console <guest> +make destroy +``` + +## Prerequisites + +A working libvirt/KVM stack (as for `GUESTFS`) plus Nix on the controller. +Install Nix once with `make nixos-flake-runtime-deps-setup`. diff --git a/kconfigs/Kconfig.ansible_provisioning b/kconfigs/Kconfig.ansible_provisioning index 79d49d1d..68b980b7 100644 --- a/kconfigs/Kconfig.ansible_provisioning +++ b/kconfigs/Kconfig.ansible_provisioning @@ -128,6 +128,37 @@ config KDEVOPS_PYTHON_OLD_INTERPRETER KDEVOPSPYTHONOLDINTERPRETER value on your hosts.in file on old systems so that this gets properly placed. +config KDEVOPS_BASELINE_PYTHON_INTERPRETER + string "Python interpreter for the [baseline] host group" + output yaml + default "/run/current-system/sw/bin/python3" if NIXOSFL + default KDEVOPS_PYTHON_INTERPRETER + help + Python interpreter emitted under [baseline:vars] in the generated + Ansible inventory. When the bringup method provisions NixOS + guests their system python lives at + /run/current-system/sw/bin/python3; otherwise this follows + KDEVOPS_PYTHON_INTERPRETER so the group tracks the host default + for other backends. + +config KDEVOPS_DEV_PYTHON_INTERPRETER + string "Python interpreter for the [dev] host group" + output yaml + default "/run/current-system/sw/bin/python3" if NIXOSFL + default KDEVOPS_PYTHON_INTERPRETER + help + Python interpreter emitted under [dev:vars] in the generated + Ansible inventory. See KDEVOPS_BASELINE_PYTHON_INTERPRETER. + +config KDEVOPS_SERVICE_PYTHON_INTERPRETER + string "Python interpreter for the [service] host group" + output yaml + default "/run/current-system/sw/bin/python3" if NIXOSFL + default KDEVOPS_PYTHON_INTERPRETER + help + Python interpreter emitted under [service:vars] in the generated + Ansible inventory. See KDEVOPS_BASELINE_PYTHON_INTERPRETER. + config DEVCONFIG_ENABLE_SYSTEMD_JOURNAL_REMOTE bool "Set up remote systemd-journal-remote" default n diff --git a/kconfigs/Kconfig.bringup b/kconfigs/Kconfig.bringup index 7e6b2d9f..6da8994e 100644 --- a/kconfigs/Kconfig.bringup +++ b/kconfigs/Kconfig.bringup @@ -6,6 +6,14 @@ config KDEVOPS_ENABLE_GUESTFS bool output yaml +config KDEVOPS_ENABLE_NIXOS_FLAKE + bool + output yaml + +config KDEVOPS_ENABLE_NIXOSFL + bool + output yaml + config KDEVOPS_ENABLE_NIXOS bool output yaml @@ -55,6 +63,23 @@ config TERRAFORM If you are not using a cloud environment just disable this. +config NIXOSFL + bool "NixOS declarative configuration with libvirt" + select KDEVOPS_ENABLE_NIXOSFL + select KDEVOPS_ENABLE_NIXOS_FLAKE + select HAVE_CUSTOM_DISTRO_HOST_PREFIX + select HAVE_DISTRO_PREFERS_CUSTOM_HOST_PREFIX + select EXTRA_STORAGE_SUPPORTS_512 + select EXTRA_STORAGE_SUPPORTS_1K + select EXTRA_STORAGE_SUPPORTS_2K + select EXTRA_STORAGE_SUPPORTS_4K + select EXTRA_STORAGE_SUPPORTS_LARGEIO + help + Use the nixos-flake library to provision VMs with libvirt. + Per-node Nix flakes import nixos-flake.nixosModules.backends.libvirt + to produce a qcow2 disk image; kdevops stages it into the libvirt + storage pool, defines and starts the domain, and waits for SSH. + config NIXOS bool "NixOS declarative configuration with libvirt" select KDEVOPS_ENABLE_NIXOS @@ -86,11 +111,12 @@ endchoice config LIBVIRT bool - depends on GUESTFS || NIXOS + depends on GUESTFS || NIXOS || NIXOSFL default y source "kconfigs/Kconfig.guestfs" source "kconfigs/Kconfig.nixos" +source "kconfigs/Kconfig.nixos_flake" source "terraform/Kconfig" source "kconfigs/Kconfig.declared_hosts" if LIBVIRT diff --git a/kconfigs/Kconfig.bringup.goals b/kconfigs/Kconfig.bringup.goals index 9bebc3d1..1f4741c4 100644 --- a/kconfigs/Kconfig.bringup.goals +++ b/kconfigs/Kconfig.bringup.goals @@ -13,6 +13,7 @@ config HAVE_DISTRO_PREFERS_INSTALL_KDEVTOOLS config KDEVOPS_TRY_REFRESH_REPOS bool "Try to refresh repositories on bringup?" + depends on !NIXOSFL default n if !HAVE_DISTRO_PREFERS_REFRESH default y if HAVE_DISTRO_PREFERS_REFRESH help @@ -30,6 +31,7 @@ config KDEVOPS_TRY_REFRESH_REPOS config RHEL_REGISTER bool "Register the guests with Red Hat?" + depends on !NIXOSFL default n help Official RHEL installations require a subscription before you can download @@ -52,6 +54,7 @@ config RHEL_ACTIVATION_KEY config KDEVOPS_CUSTOM_YUM_REPOFILE string "Custom yum repo file" + depends on !NIXOSFL default "" help Add a custom yum repo file to /etc/yum.repos.d before updating? Leave blank to diff --git a/kconfigs/Kconfig.kdevops b/kconfigs/Kconfig.kdevops index fee93f68..7b784a12 100644 --- a/kconfigs/Kconfig.kdevops +++ b/kconfigs/Kconfig.kdevops @@ -28,6 +28,7 @@ config CUSTOM_DISTRO_HOST_PREFIX depends on HAVE_CUSTOM_DISTRO_HOST_PREFIX default GUESTFS_FEDORA_BOX_SHORT if GUESTFS_FEDORA default GUESTFS_DEBIAN_BOX_SHORT if GUESTFS_DEBIAN + default "nixos" if NIXOSFL config CUSTOM_SSH_KEXALGORITHMS string diff --git a/kconfigs/Kconfig.nixos_flake b/kconfigs/Kconfig.nixos_flake new file mode 100644 index 00000000..8589a144 --- /dev/null +++ b/kconfigs/Kconfig.nixos_flake @@ -0,0 +1,167 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# nixos_flake -- shared infrastructure feature for the Nix flake at +# scripts/nixos-flake/. Consumed by the NIXOSFL bringup +# choice today and by the imageless backend (QSU) later. + +if KDEVOPS_ENABLE_NIXOS_FLAKE + +menu "NixOS Flake" + +config NIXOS_FLAKE_PATH + string + output yaml + default "$(TOPDIR_PATH)/scripts/nixos-flake" + help + Absolute path to the vendored nixos-flake on the controller. + Per-VM flakes import this path via `nixos-flake.url = "path:..."` + so a `git subtree pull` against scripts/nixos-flake picks up + automatically. + +config NIXOS_CONFIG_DIR + string + output yaml + default "$(TOPDIR_PATH)/nixos" + help + Directory where gen_nodes writes the kdevops_nodes.yaml + inventory file for the nixos_flake backends. Reuses the same + on-disk path as the legacy nixos workflow so a single nodes + template (nixos_nodes.j2) serves both. + +if NIXOSFL + +config NIXOSFL_CONFIG_DIR + string + output yaml + default "$(TOPDIR_PATH)/nixos/generated" + help + Controller-side directory where the nixos_flake role writes + one subdirectory per guest holding the rendered flake.nix and + default.nix. Defaults to nixos/generated/ at the kdevops + topdir, which is gitignored. + +config NIXOSFL_STORAGE_DIR + string + output yaml + default "{{ kdevops_storage_pool_path }}/nixos" + help + Libvirt storage-pool subdirectory where built qcow2 images and + run-<vm>-wrapper.sh scripts are staged. Resolved at task time + from the libvirt storage pool path so it tracks the libvirt + configuration. + +endif # NIXOSFL + +menu "Profiles" + +config NIXOS_FLAKE_PROFILE_DEVEL + bool "Development profile (kernel testing tools)" + output yaml + default y + help + Imports nixos-flake.nixosModules.profiles.devel into every + guest. Adds kernel testing utilities expected by kdevops + workflows. + +config NIXOS_FLAKE_PROFILE_BUILD_TOOLS + bool "Build tools profile (autotools + headers)" + output yaml + default n + help + Imports nixos-flake.nixosModules.profiles.build-tools. Pulls in + the Autotools build environment when guest-side compilation is + expected. + +config NIXOS_FLAKE_PROFILE_MONITORING + bool "Monitoring profile (workflow-bracketed monitor units)" + output yaml + default n + help + Imports nixos-flake.nixosModules.profiles.monitoring. Brings in + sysstat / cpu-governor / blkalgn / biolatency systemd template + units instantiated per workload run-id. + + Independent of the kdevops ENABLE_MONITORING knob -- the two + monitor stacks cover different ground (kdevops monitoring + focuses on developmental kernel-research stats like folio + migration debugfs; this profile is system-wide perf monitoring + via sadc + libbpf-tools). Enabling this just bakes the systemd + template units into the guest image; nothing in kdevops drives + them today. + +endmenu # Profiles + +menu "Mounts" + +config NIXOS_FLAKE_MOUNT_SHARES + bool "Enable virtiofs shares (mounts/shares.nix)" + output yaml + default n + help + Imports nixos-flake.nixosModules.mounts.shares. Mounts named + virtiofs shares inside the guest. qsu pairs this with the + nixos-flake backends.imageless closure for /nix/store + + /lib/modules + data-* shares. + +config NIXOS_FLAKE_MOUNT_STORAGE + bool "Enable block-device storage mounts (mounts/storage.nix)" + output yaml + default n + help + Imports nixos-flake.nixosModules.mounts.storage. Mounts named + block devices inside the guest, optionally running mkfs first. + +endmenu # Mounts + +menu "Test suites" + +config NIXOS_FLAKE_TESTSUITE_FSTESTS + bool "fstests" + output yaml + default n + help + Imports nixos-flake.nixosModules.testSuites.fstests. Pulls in + the xfstests toolchain + binaries on the guest. The kdevops + fstests workflow's role glue (running tests, harvesting results) + is a separate follow-up. + +config NIXOS_FLAKE_TESTSUITE_BLKTESTS + bool "blktests" + output yaml + default n + +config NIXOS_FLAKE_TESTSUITE_LTP + bool "ltp" + output yaml + default n + +config NIXOS_FLAKE_TESTSUITE_MMTESTS + bool "mmtests" + output yaml + default n + +config NIXOS_FLAKE_TESTSUITE_PYNFS + bool "pynfs" + output yaml + default n + +config NIXOS_FLAKE_TESTSUITE_GITR + bool "gitr" + output yaml + default n + +config NIXOS_FLAKE_TESTSUITE_SELFTESTS + bool "selftests" + output yaml + default n + +config NIXOS_FLAKE_TESTSUITE_SYSBENCH + bool "sysbench" + output yaml + default n + +endmenu # Test suites + +endmenu # NixOS Flake + +endif # KDEVOPS_ENABLE_NIXOS_FLAKE diff --git a/kconfigs/monitors/Kconfig b/kconfigs/monitors/Kconfig index bd4dc81f..14811dab 100644 --- a/kconfigs/monitors/Kconfig +++ b/kconfigs/monitors/Kconfig @@ -4,6 +4,7 @@ config ENABLE_MONITORING bool "Enable monitoring services during workflow execution" output yaml default n + depends on !NIXOSFL help Enable monitoring services to collect statistics during workflow execution. This allows collection of various system metrics while @@ -18,6 +19,14 @@ config ENABLE_MONITORING Currently supported workflows: - fstests + Not available under NIXOSFL: the kdevops monitoring + role's install-deps step uses apt/dnf without a NixOS branch, and + every consumer workflow (fstests, mmtests, sysbench, ai, minio) is + gated out under that backend anyway. The nixos-flake guest-side + monitoring profile (NIXOS_FLAKE_PROFILE_MONITORING) is the + Nix-native equivalent; the kdevops side needs an orchestrator that + drives those systemd template units, which is unscoped work. + if ENABLE_MONITORING config MONITOR_DEVELOPMENTAL_STATS diff --git a/kconfigs/workflows/Kconfig b/kconfigs/workflows/Kconfig index fc4ec439..0b0c47ec 100644 --- a/kconfigs/workflows/Kconfig +++ b/kconfigs/workflows/Kconfig @@ -21,6 +21,7 @@ endmenu choice prompt "Target Linux kernel to use" + default WORKFLOW_LINUX_DISTRO if NIXOSFL default WORKFLOW_LINUX_DISTRO if !QEMU_ENABLE_EXTRA_DRIVE_LARGEIO default WORKFLOW_LINUX_CUSTOM if QEMU_ENABLE_EXTRA_DRIVE_LARGEIO @@ -32,15 +33,24 @@ config WORKFLOW_LINUX_DISTRO config WORKFLOW_LINUX_CUSTOM bool "Upstream Linux or custom kernel" + depends on !NIXOSFL select BOOTLINUX help If you are targeting a workflow to run on a vanilla upstream linux, linux-stable, linux-next, or a custom kernel you want to build and install enable this. + Not available under NIXOSFL. The flake-built qcow2 + bakes its own kernel and modules; swapping in a custom kernel + needs a Nix overlay/package that rebuilds the image, plus a + shutdown-and-switch boot cycle. That workflow is not wired up + yet -- for now the guest runs whatever kernel the NixOS module + pin in scripts/nixos-flake/ produces. + config WORKFLOW_LINUX_PACKAGED bool "Install pre-built kernel package" output yaml + depends on !KDEVOPS_ENABLE_NIXOS_FLAKE help Choose this option to install a pre-built kernel package on each target node. @@ -50,6 +60,9 @@ config WORKFLOW_LINUX_PACKAGED option that lets kdevops pull the packages from other locations. + Not available under nixos_flake guests: the rootfs is built + from a Nix expression with no apt/dnf to consume a .deb/.rpm. + endchoice if WORKFLOW_LINUX_CUSTOM @@ -126,7 +139,7 @@ choice config KDEVOPS_WORKFLOW_DEDICATE_FSTESTS bool "fstests" select KDEVOPS_WORKFLOW_ENABLE_FSTESTS - depends on !KDEVOPS_USE_DECLARED_HOSTS + depends on !KDEVOPS_USE_DECLARED_HOSTS && !KDEVOPS_ENABLE_NIXOS_FLAKE help This will dedicate your configuration only to fstests. @@ -142,14 +155,14 @@ config KDEVOPS_WORKFLOW_DEDICATE_FSTESTS config KDEVOPS_WORKFLOW_DEDICATE_BLKTESTS bool "blktests" - depends on !KDEVOPS_USE_DECLARED_HOSTS + depends on !KDEVOPS_USE_DECLARED_HOSTS && !KDEVOPS_ENABLE_NIXOS_FLAKE select KDEVOPS_WORKFLOW_ENABLE_BLKTESTS help This will dedicate your configuration only to blktests. config KDEVOPS_WORKFLOW_DEDICATE_CXL bool "cxl" - depends on !KDEVOPS_USE_DECLARED_HOSTS + depends on !KDEVOPS_USE_DECLARED_HOSTS && !KDEVOPS_ENABLE_NIXOS_FLAKE select KDEVOPS_WORKFLOW_ENABLE_CXL help This will dedicate your configuration only to cxl work. @@ -164,7 +177,7 @@ config KDEVOPS_WORKFLOW_DEDICATE_CXL config KDEVOPS_WORKFLOW_DEDICATE_PYNFS bool "pynfs" - depends on !KDEVOPS_USE_DECLARED_HOSTS + depends on !KDEVOPS_USE_DECLARED_HOSTS && !KDEVOPS_ENABLE_NIXOS_FLAKE select KDEVOPS_WORKFLOW_ENABLE_PYNFS help This will dedicate your configuration only to running pynfs @@ -172,7 +185,7 @@ config KDEVOPS_WORKFLOW_DEDICATE_PYNFS config KDEVOPS_WORKFLOW_DEDICATE_SELFTESTS bool "Linux kernel selftests" - depends on !KDEVOPS_USE_DECLARED_HOSTS + depends on !KDEVOPS_USE_DECLARED_HOSTS && !KDEVOPS_ENABLE_NIXOS_FLAKE select KDEVOPS_WORKFLOW_ENABLE_SELFTESTS help This will dedicate your configuration only to Linux kernel @@ -181,7 +194,7 @@ config KDEVOPS_WORKFLOW_DEDICATE_SELFTESTS config KDEVOPS_WORKFLOW_DEDICATE_GITR bool "gitr" - depends on !KDEVOPS_USE_DECLARED_HOSTS + depends on !KDEVOPS_USE_DECLARED_HOSTS && !KDEVOPS_ENABLE_NIXOS_FLAKE select KDEVOPS_WORKFLOW_ENABLE_GITR help This will dedicate your configuration to running only the @@ -189,7 +202,7 @@ config KDEVOPS_WORKFLOW_DEDICATE_GITR config KDEVOPS_WORKFLOW_DEDICATE_LTP bool "ltp" - depends on !KDEVOPS_USE_DECLARED_HOSTS + depends on !KDEVOPS_USE_DECLARED_HOSTS && !KDEVOPS_ENABLE_NIXOS_FLAKE select KDEVOPS_WORKFLOW_ENABLE_LTP help This will dedicate your configuration to running only the @@ -197,7 +210,7 @@ config KDEVOPS_WORKFLOW_DEDICATE_LTP config KDEVOPS_WORKFLOW_DEDICATE_NFSTEST bool "nfstest" - depends on !KDEVOPS_USE_DECLARED_HOSTS + depends on !KDEVOPS_USE_DECLARED_HOSTS && !KDEVOPS_ENABLE_NIXOS_FLAKE select KDEVOPS_WORKFLOW_ENABLE_NFSTEST help This will dedicate your configuration to running only the @@ -205,7 +218,7 @@ config KDEVOPS_WORKFLOW_DEDICATE_NFSTEST config KDEVOPS_WORKFLOW_DEDICATE_SYSBENCH bool "sysbench" - depends on !KDEVOPS_USE_DECLARED_HOSTS + depends on !KDEVOPS_USE_DECLARED_HOSTS && !KDEVOPS_ENABLE_NIXOS_FLAKE select KDEVOPS_WORKFLOW_ENABLE_SYSBENCH help This will dedicate your configuration to running only the @@ -213,7 +226,7 @@ config KDEVOPS_WORKFLOW_DEDICATE_SYSBENCH config KDEVOPS_WORKFLOW_DEDICATE_MMTESTS bool "mmtests" - depends on !KDEVOPS_USE_DECLARED_HOSTS + depends on !KDEVOPS_USE_DECLARED_HOSTS && !KDEVOPS_ENABLE_NIXOS_FLAKE select KDEVOPS_WORKFLOW_ENABLE_MMTESTS help This will dedicate your configuration to running only the @@ -221,7 +234,7 @@ config KDEVOPS_WORKFLOW_DEDICATE_MMTESTS config KDEVOPS_WORKFLOW_DEDICATE_FIO_TESTS bool "fio-tests" - depends on !KDEVOPS_USE_DECLARED_HOSTS + depends on !KDEVOPS_USE_DECLARED_HOSTS && !KDEVOPS_ENABLE_NIXOS_FLAKE select KDEVOPS_WORKFLOW_ENABLE_FIO_TESTS help This will dedicate your configuration to running only the @@ -229,7 +242,7 @@ config KDEVOPS_WORKFLOW_DEDICATE_FIO_TESTS config KDEVOPS_WORKFLOW_DEDICATE_AI bool "ai" - depends on !KDEVOPS_USE_DECLARED_HOSTS + depends on !KDEVOPS_USE_DECLARED_HOSTS && !KDEVOPS_ENABLE_NIXOS_FLAKE select KDEVOPS_WORKFLOW_ENABLE_AI help This will dedicate your configuration to running only the @@ -237,6 +250,7 @@ config KDEVOPS_WORKFLOW_DEDICATE_AI config KDEVOPS_WORKFLOW_DEDICATE_VLLM bool "vllm" + depends on !KDEVOPS_ENABLE_NIXOS_FLAKE select KDEVOPS_WORKFLOW_ENABLE_VLLM help This will dedicate your configuration to running only the @@ -245,6 +259,7 @@ config KDEVOPS_WORKFLOW_DEDICATE_VLLM config KDEVOPS_WORKFLOW_DEDICATE_MINIO bool "minio" + depends on !KDEVOPS_ENABLE_NIXOS_FLAKE select KDEVOPS_WORKFLOW_ENABLE_MINIO help This will dedicate your configuration to running only the diff --git a/playbooks/nixosfl.yml b/playbooks/nixosfl.yml new file mode 100644 index 00000000..e6848205 --- /dev/null +++ b/playbooks/nixosfl.yml @@ -0,0 +1,36 @@ +--- +# SPDX-License-Identifier: copyleft-next-0.3.1 +- name: Verify the nixos_flake controller-side setup is complete + hosts: localhost + connection: local + gather_facts: true + tasks: + - name: Import optional extra_args file + ansible.builtin.include_vars: "{{ item }}" + failed_when: false + with_first_found: + - files: + - "../extra_vars.yml" + - "../extra_vars.yaml" + - "../extra_vars.json" + skip: true + tags: [always] + + - name: Run the non-sudo nixos_flake runtime_deps verify path + ansible.builtin.include_role: + name: nixos_flake + tasks_from: runtime-deps/verify.yml + + - name: Run the runtime_deps sudo opt-in (only with --tags nixos_flake_runtime_deps_setup) + ansible.builtin.include_role: + name: nixos_flake + tasks_from: runtime-deps/setup.yml + apply: + tags: [nixos_flake_runtime_deps_setup] + tags: [never, nixos_flake_runtime_deps_setup] + +- name: Run the nixos_flake role on all guests + hosts: all:!localhost + gather_facts: false + roles: + - role: nixosfl diff --git a/playbooks/roles/gen_hosts/templates/workflows/linux.j2 b/playbooks/roles/gen_hosts/templates/workflows/linux.j2 index cdbfe78d..a912e761 100644 --- a/playbooks/roles/gen_hosts/templates/workflows/linux.j2 +++ b/playbooks/roles/gen_hosts/templates/workflows/linux.j2 @@ -13,12 +13,12 @@ ansible_python_interpreter = "{{ kdevops_python_interpreter }}" [baseline] {{ node_name }} [baseline:vars] -ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +ansible_python_interpreter = "{{ kdevops_baseline_python_interpreter }}" [dev] [dev:vars] -ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +ansible_python_interpreter = "{{ kdevops_dev_python_interpreter }}" [service] [service:vars] -ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +ansible_python_interpreter = "{{ kdevops_service_python_interpreter }}" diff --git a/playbooks/roles/gen_nodes/tasks/main.yml b/playbooks/roles/gen_nodes/tasks/main.yml index bdead840..e8c3672c 100644 --- a/playbooks/roles/gen_nodes/tasks/main.yml +++ b/playbooks/roles/gen_nodes/tasks/main.yml @@ -21,12 +21,11 @@ register: my_group changed_when: false -- name: Create guestfs directory +- name: Create the parent directory the nodes template writes into ansible.builtin.file: - path: "{{ guestfs_path }}" + path: "{{ topdir_path }}/{{ kdevops_nodes | dirname }}" state: directory mode: "0755" - when: kdevops_enable_guestfs|bool - name: Create nixos directory ansible.builtin.file: diff --git a/playbooks/roles/nixos_flake/tasks/runtime-deps/setup.yml b/playbooks/roles/nixos_flake/tasks/runtime-deps/setup.yml new file mode 100644 index 00000000..01077ea9 --- /dev/null +++ b/playbooks/roles/nixos_flake/tasks/runtime-deps/setup.yml @@ -0,0 +1,50 @@ +--- +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Controller-side sudo work to install the Nix package manager via +# the official multi-user installer from nixos.org. Invoked via +# `make nixos-flake-runtime-deps-setup` +# (--tags nixos_flake_runtime_deps_setup). No-op when the controller +# is already NixOS (nix ships with the base system). + +- name: Gather facts for the NixOS check + ansible.builtin.setup: + gather_subset: ["min"] + +- name: Skip when controller is already NixOS + ansible.builtin.debug: + msg: "Controller is NixOS; nix is part of the base system, nothing to install." + when: ansible_facts['distribution']|lower == 'nixos' + +- name: Create a private directory for the Nix installer + ansible.builtin.tempfile: + state: directory + suffix: nix-install + register: nixos_flake_nix_installer_tmp + when: ansible_facts['distribution']|lower != 'nixos' + +# Set nixos_flake_nix_installer_checksum (for example +# "sha256:<hex>") to verify the download on untrusted networks; the +# upstream URL is unversioned, so no checksum is pinned by default. +- name: Download the official Nix installer + ansible.builtin.get_url: + url: "{{ nixos_flake_nix_installer_url | default('https://nixos.org/nix/install') }}" + dest: "{{ nixos_flake_nix_installer_tmp.path }}/install-nix.sh" + mode: "0700" + checksum: "{{ nixos_flake_nix_installer_checksum | default(omit) }}" + when: ansible_facts['distribution']|lower != 'nixos' + +- name: Run the Nix installer (multi-user, non-interactive) + become: true + become_method: ansible.builtin.sudo + ansible.builtin.command: + cmd: sh {{ nixos_flake_nix_installer_tmp.path }}/install-nix.sh --daemon --yes + creates: /nix + when: ansible_facts['distribution']|lower != 'nixos' + tags: do_install + +- name: Remove the Nix installer directory + ansible.builtin.file: + path: "{{ nixos_flake_nix_installer_tmp.path }}" + state: absent + when: ansible_facts['distribution']|lower != 'nixos' diff --git a/playbooks/roles/nixos_flake/tasks/runtime-deps/verify.yml b/playbooks/roles/nixos_flake/tasks/runtime-deps/verify.yml new file mode 100644 index 00000000..8e5f0996 --- /dev/null +++ b/playbooks/roles/nixos_flake/tasks/runtime-deps/verify.yml @@ -0,0 +1,29 @@ +--- +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Non-sudo precheck for the nixos_flake controller-side dependency: +# the Nix package manager. Fails with an actionable diagnostic when +# nix is missing from PATH. + +- name: Check whether the nix binary is on PATH + ansible.builtin.shell: + cmd: set -o pipefail && command -v nix + executable: /bin/bash + changed_when: false + failed_when: false + register: nixos_flake_runtime_deps_nix + +- name: Fail with actionable diagnostic when nix is missing + ansible.builtin.fail: + msg: | + The nix package manager is not installed on the controller: + the `nix` binary is not on PATH. nixos_flake cannot build NixOS + closures or disk images without it. + + Two ways to fix this: + 1. Run the opt-in sudo target: + make nixos-flake-runtime-deps-setup + 2. Install nix manually following + nixos-flake/runtime-deps/README.md, then re-run the + original make target. + when: nixos_flake_runtime_deps_nix.rc != 0 diff --git a/playbooks/roles/nixosfl/README.md b/playbooks/roles/nixosfl/README.md new file mode 100644 index 00000000..23fc4dbf --- /dev/null +++ b/playbooks/roles/nixosfl/README.md @@ -0,0 +1,12 @@ +# playbooks/roles/nixos_flake -- nixos_flake role + +Thin role wrapping the vendored nixos-flake Nix library at +`scripts/nixos-flake/`. Per-VM glue templates the role renders +reference the flake via: + +```nix +inputs.nixos-flake.url = "path:{{ nixos_flake_path }}"; +``` + +so updating the subtree via `git subtree pull --squash` immediately +reaches every generated per-VM flake. diff --git a/playbooks/roles/nixosfl/tasks/bringup.yml b/playbooks/roles/nixosfl/tasks/bringup.yml new file mode 100644 index 00000000..9a8cf97c --- /dev/null +++ b/playbooks/roles/nixosfl/tasks/bringup.yml @@ -0,0 +1,27 @@ +--- +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Provision the per-node libvirt guest from a freshly built +# nixos_flake image. Six phases run in order: generate the per-node +# Nix flake, build the image and stage it into the storage pool, +# ensure the default network is active, define and start the libvirt +# domain, wait for DHCP and SSH and update the controller's ssh +# config, and print a closing access banner. + +- name: Generate per-node NixOS configurations + ansible.builtin.import_tasks: generate-configs.yml + +- name: Build per-node disk images and stage into the storage pool + ansible.builtin.import_tasks: libvirt-build.yml + +- name: Ensure the libvirt default network is active + ansible.builtin.import_tasks: libvirt-network.yml + +- name: Define and start the per-node libvirt domain + ansible.builtin.import_tasks: libvirt-provision.yml + +- name: Wait for SSH and update the controller's ~/.ssh/config + ansible.builtin.import_tasks: ssh-access.yml + +- name: Print SSH and virsh access guidance + ansible.builtin.import_tasks: console.yml diff --git a/playbooks/roles/nixosfl/tasks/console.yml b/playbooks/roles/nixosfl/tasks/console.yml new file mode 100644 index 00000000..a6143bd7 --- /dev/null +++ b/playbooks/roles/nixosfl/tasks/console.yml @@ -0,0 +1,28 @@ +--- +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Print SSH and virsh access guidance for the provisioned guests. +# Imported at the tail of bringup.yml so a successful bringup ends +# with the access banner, and reachable on its own via the +# nixosfl_console tag for re-display. + +- name: Display VM access information + delegate_to: localhost + run_once: true + ansible.builtin.debug: + msg: | + NixOS VMs are running and accessible via libvirt. + + SSH Access: + {% for vm in groups['all'] | reject('equalto', 'localhost') | list %} + - {{ vm }}: ssh {{ vm }} + {% endfor %} + + VM Management: + {% for vm in groups['all'] | reject('equalto', 'localhost') | list %} + - {{ vm }}: virsh {start|shutdown|destroy} {{ vm }} + {% endfor %} + + VM Status: + - Check status: virsh list --all + - Get IP: virsh domifaddr <vm_name> diff --git a/playbooks/roles/nixosfl/tasks/destroy.yml b/playbooks/roles/nixosfl/tasks/destroy.yml new file mode 100644 index 00000000..cb403aa3 --- /dev/null +++ b/playbooks/roles/nixosfl/tasks/destroy.yml @@ -0,0 +1,80 @@ +--- +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Tear everything down: libvirt domains, SSH config entries, disk +# images, per-node generated configurations, and the cached NixOS +# disk images in the Nix store. + +- name: Localhost destroy + delegate_to: localhost + run_once: true + tags: do_clean + block: + - name: Shut down VMs via libvirt + ansible.builtin.command: virsh destroy "{{ item }}" + loop: "{{ groups['all'] | reject('equalto', 'localhost') | list }}" + register: nixosfl_virsh_destroy + failed_when: false + changed_when: nixosfl_virsh_destroy.rc == 0 + environment: + LIBVIRT_DEFAULT_URI: "{{ libvirt_uri | default('qemu:///system') }}" + + - name: Remove VM definitions from libvirt + ansible.builtin.command: virsh undefine "{{ item }}" + loop: "{{ groups['all'] | reject('equalto', 'localhost') | list }}" + register: nixosfl_virsh_undefine + failed_when: false + changed_when: nixosfl_virsh_undefine.rc == 0 + environment: + LIBVIRT_DEFAULT_URI: "{{ libvirt_uri | default('qemu:///system') }}" + + - name: Remove SSH config entries for NixOS VMs + ansible.builtin.command: + argv: + - python3 + - "{{ playbook_dir }}/../scripts/update_ssh_config_nixosfl.py" + - remove + - "{{ item }}" + - "" + - "" + - "" + - "{{ nixosfl_ssh_config_file | default(lookup('env', 'HOME') ~ '/.ssh/config') }}" + - "" + - NixOS VM + loop: "{{ groups['all'] | reject('equalto', 'localhost') | list }}" + when: nixosfl_update_ssh_config | default(true) | bool + register: nixosfl_ssh_config_remove + failed_when: false + changed_when: nixosfl_ssh_config_remove.rc == 0 + + - name: Remove VM disk images + ansible.builtin.file: + path: "{{ nixosfl_storage_dir }}/{{ item }}.qcow2" + state: absent + loop: "{{ groups['all'] | reject('equalto', 'localhost') | list }}" + + - name: Clean up per-node generated NixOS configurations + ansible.builtin.file: + path: "{{ nixosfl_config_dir }}/{{ item }}" + state: absent + loop: "{{ groups['all'] | reject('equalto', 'localhost') | list }}" + + - name: Garbage collect cached NixOS disk images from Nix store + ansible.builtin.shell: | + set -o pipefail + if [ -f /nix/var/nix/profiles/default/etc/profile.d/nix.sh ]; then + . /nix/var/nix/profiles/default/etc/profile.d/nix.sh + fi + + NIX_COLLECT_GARBAGE=$(command -v nix-collect-garbage 2>/dev/null \ + || find /nix -name "nix-collect-garbage" -type f 2>/dev/null | head -1) + + if [ -n "$NIX_COLLECT_GARBAGE" ]; then + echo "Running Nix garbage collection to remove cached disk images..." + sudo "$NIX_COLLECT_GARBAGE" -d 2>&1 | grep -E "(deleting|freed|store paths)" || true + else + echo "Warning: nix-collect-garbage not found, cached images may remain" + fi + register: nixosfl_gc_result + failed_when: false + changed_when: "'freed' in nixosfl_gc_result.stdout" diff --git a/playbooks/roles/nixosfl/tasks/generate-configs.yml b/playbooks/roles/nixosfl/tasks/generate-configs.yml new file mode 100644 index 00000000..7331a88e --- /dev/null +++ b/playbooks/roles/nixosfl/tasks/generate-configs.yml @@ -0,0 +1,90 @@ +--- +# SPDX-License-Identifier: copyleft-next-0.3.1 + +- name: Localhost generate-configs + delegate_to: localhost + run_once: true + block: + - name: Create top-level NixOS directories + ansible.builtin.file: + path: "{{ item }}" + state: directory + mode: "0755" + loop: + - "{{ nixosfl_config_dir }}" + - "{{ nixosfl_storage_dir }}" + tags: do_install + + - name: Create per-node configuration directories + ansible.builtin.file: + path: "{{ nixosfl_config_dir }}/{{ item }}" + state: directory + mode: "0755" + loop: "{{ groups['all'] | reject('equalto', 'localhost') | list }}" + tags: do_install + + - name: Determine SSH key path based on directory + ansible.builtin.command: python3 {{ playbook_dir }}/../scripts/nixosfl_ssh_key_name.py --path + register: nixosfl_ssh_key_path_result + changed_when: false + + - name: Set SSH key path + ansible.builtin.set_fact: + nixosfl_ssh_key_path: "{{ nixosfl_ssh_key_path_result.stdout | trim }}" + + - name: Generate SSH key for NixOS VMs if not exists + community.crypto.openssh_keypair: + path: "{{ nixosfl_ssh_key_path }}" + type: rsa + size: 2048 + comment: "kdevops@nixos" + force: false + tags: do_install + + - name: Read SSH public key + ansible.builtin.slurp: + src: "{{ nixosfl_ssh_key_path }}.pub" + register: nixosfl_ssh_public_key + + - name: Set SSH key in fact + ansible.builtin.set_fact: + nixosfl_ssh_authorized_key: "{{ nixosfl_ssh_public_key['content'] | b64decode | trim }}" + + - name: Detect local Nix cache mirror URL if enabled + ansible.builtin.shell: | + bash {{ playbook_dir }}/../scripts/check_nix_mirror.sh NIX_CACHE_MIRROR_URL + register: nixosfl_detected_mirror_url + when: + - nixosfl_use_local_mirror | default(false) | bool + - nixosfl_mirror_url is not defined or nixosfl_mirror_url == "" + changed_when: false + + - name: Set detected mirror URL + ansible.builtin.set_fact: + nixosfl_mirror_url: "{{ nixosfl_detected_mirror_url.stdout | trim }}" + when: + - nixosfl_detected_mirror_url is defined + - nixosfl_detected_mirror_url.stdout is defined + - nixosfl_detected_mirror_url.stdout | trim != "" + + - name: Debug SSH key path + ansible.builtin.debug: + msg: "Using SSH key: {{ nixosfl_ssh_key_path | default('NOT SET') }}" + + - name: Render per-node flake.nix + ansible.builtin.template: + src: flake.nix.j2 + dest: "{{ nixosfl_config_dir }}/{{ item }}/flake.nix" + mode: "0644" + loop: "{{ groups['all'] | reject('equalto', 'localhost') | list }}" + tags: do_install + + - name: Render per-node default.nix + ansible.builtin.template: + src: default.nix.j2 + dest: "{{ nixosfl_config_dir }}/{{ item }}/default.nix" + mode: "0644" + loop: "{{ groups['all'] | reject('equalto', 'localhost') | list }}" + vars: + vm_name: "{{ item }}" + tags: do_install diff --git a/playbooks/roles/nixosfl/tasks/libvirt-build.yml b/playbooks/roles/nixosfl/tasks/libvirt-build.yml new file mode 100644 index 00000000..e65cc81a --- /dev/null +++ b/playbooks/roles/nixosfl/tasks/libvirt-build.yml @@ -0,0 +1,45 @@ +--- +# SPDX-License-Identifier: copyleft-next-0.3.1 + +- name: Localhost libvirt build + delegate_to: localhost + run_once: true + block: + - name: Build per-node NixOS disk image via the flake + ansible.builtin.command: + argv: + - nix + - build + - "path:{{ nixosfl_config_dir }}/{{ item }}#image" + - --out-link + - "{{ nixosfl_config_dir }}/{{ item }}/result" + creates: "{{ nixosfl_config_dir }}/{{ item }}/result" + environment: + PATH: "/nix/var/nix/profiles/default/bin:{{ nixosfl_path_fallback }}" + NIX_CONFIG: "{{ nixosfl_nix_config }}" + vars: + nixosfl_path_fallback: >- + {{ lookup('env', 'PATH') + | default('/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', true) }} + nixosfl_use_mirror: >- + {{ nixosfl_use_local_mirror | default(false) | bool + and (nixosfl_mirror_url | default('') | length > 0) }} + nixosfl_nix_config: >- + {{ ('substituters = ' ~ nixosfl_mirror_url ~ ' https://cache.nixos.org') + if nixosfl_use_mirror else omit }} + loop: "{{ groups['all'] | reject('equalto', 'localhost') | list }}" + loop_control: + label: "{{ item }}" + tags: do_compile + + - name: Stage per-node NixOS disk image into libvirt storage + ansible.builtin.copy: + src: "{{ nixosfl_config_dir }}/{{ item }}/result/nixos.qcow2" + dest: "{{ nixosfl_storage_dir }}/{{ item }}.qcow2" + mode: "0644" + remote_src: true + force: true + loop: "{{ groups['all'] | reject('equalto', 'localhost') | list }}" + loop_control: + label: "{{ item }}" + tags: do_install diff --git a/playbooks/roles/nixosfl/tasks/libvirt-network.yml b/playbooks/roles/nixosfl/tasks/libvirt-network.yml new file mode 100644 index 00000000..cb5bade2 --- /dev/null +++ b/playbooks/roles/nixosfl/tasks/libvirt-network.yml @@ -0,0 +1,28 @@ +--- +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Ensure the libvirt default network is active before trying to +# start any NixOS VM. Only used by the libvirt backend path. + +- name: Localhost libvirt network + delegate_to: localhost + run_once: true + block: + - name: Check if default network exists and is active + ansible.builtin.command: virsh net-info default + register: nixosfl_default_network_info + failed_when: false + changed_when: false + environment: + LIBVIRT_DEFAULT_URI: "{{ libvirt_uri | default('qemu:///system') }}" + + - name: Start default network if not active + ansible.builtin.command: virsh net-start default + when: nixosfl_default_network_info.rc != 0 or 'Active:' not in nixosfl_default_network_info.stdout or 'yes' not in + nixosfl_default_network_info.stdout.split('Active:')[1].split('\n')[0] + register: nixosfl_net_start + failed_when: false + changed_when: nixosfl_net_start.rc == 0 + environment: + LIBVIRT_DEFAULT_URI: "{{ libvirt_uri | default('qemu:///system') }}" + tags: do_deploy diff --git a/playbooks/roles/nixosfl/tasks/libvirt-provision.yml b/playbooks/roles/nixosfl/tasks/libvirt-provision.yml new file mode 100644 index 00000000..0c86badf --- /dev/null +++ b/playbooks/roles/nixosfl/tasks/libvirt-provision.yml @@ -0,0 +1,60 @@ +--- +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Define and start the per-node libvirt domain. Runs against each +# guest host in the baseline/dev inventory groups, but every task +# delegates to localhost because libvirt lives on the control node. +# Only used by the libvirt backend path. + +- name: Check if VM already exists + ansible.builtin.command: virsh domstate "{{ inventory_hostname }}" + register: nixosfl_vm_status + failed_when: false + changed_when: false + delegate_to: localhost + environment: + LIBVIRT_DEFAULT_URI: "{{ libvirt_uri | default('qemu:///system') }}" + +- name: Provision VM with libvirt + when: nixosfl_vm_status.rc != 0 or 'shut off' in nixosfl_vm_status.stdout + delegate_to: localhost + environment: + LIBVIRT_DEFAULT_URI: "{{ libvirt_uri | default('qemu:///system') }}" + block: + - name: Generate libvirt XML for VM + ansible.builtin.template: + src: vm-libvirt.xml.j2 + dest: "{{ nixosfl_storage_dir }}/{{ inventory_hostname }}.xml" + mode: "0644" + vars: + vm_name: "{{ inventory_hostname }}" + vm_memory: "{{ nixosfl_vm_memory_mb | default(4096) }}" + vm_vcpus: "{{ nixosfl_vm_vcpus | default(4) }}" + vm_disk: "{{ nixosfl_storage_dir }}/{{ inventory_hostname }}.qcow2" + tags: do_install + + - name: Define VM in libvirt + ansible.builtin.command: virsh define "{{ nixosfl_storage_dir }}/{{ inventory_hostname }}.xml" + register: nixosfl_virsh_define + failed_when: false + changed_when: nixosfl_virsh_define.rc == 0 + tags: do_deploy + + - name: Start VM + ansible.builtin.command: virsh start "{{ inventory_hostname }}" + register: nixosfl_virsh_start + failed_when: false + changed_when: nixosfl_virsh_start.rc == 0 + tags: do_deploy + +- name: Ensure VM is running + ansible.builtin.command: virsh start "{{ inventory_hostname }}" + register: nixosfl_start_result + failed_when: + - nixosfl_start_result.rc != 0 + - "'already active' not in nixosfl_start_result.stderr" + changed_when: nixosfl_start_result.rc == 0 + delegate_to: localhost + environment: + LIBVIRT_DEFAULT_URI: "{{ libvirt_uri | default('qemu:///system') }}" + tags: do_deploy diff --git a/playbooks/roles/nixosfl/tasks/main.yml b/playbooks/roles/nixosfl/tasks/main.yml new file mode 100644 index 00000000..59a95f40 --- /dev/null +++ b/playbooks/roles/nixosfl/tasks/main.yml @@ -0,0 +1,34 @@ +--- +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Per-host entry point for the nixos_flake role. The runtime_deps +# task group runs from playbooks/nixosfl.yml's first play +# (hosts: localhost); this entry point covers backend-specific phases +# that need the guest inventory available. Each phase is tagged with +# `never` plus its per-phase tag and is reached only via the +# dedicated Make targets: +# +# make nixosfl-bringup +# make nixosfl-destroy +# make nixosfl-console + +- name: Run nixos_flake libvirt bringup (only with --tags nixosfl_bringup) + ansible.builtin.include_tasks: + file: bringup.yml + apply: + tags: [nixosfl_bringup] + tags: [never, nixosfl_bringup] + +- name: Run nixos_flake libvirt destroy (only with --tags nixosfl_destroy) + ansible.builtin.include_tasks: + file: destroy.yml + apply: + tags: [nixosfl_destroy] + tags: [never, nixosfl_destroy] + +- name: Run nixos_flake libvirt console (only with --tags nixosfl_console) + ansible.builtin.include_tasks: + file: console.yml + apply: + tags: [nixosfl_console] + tags: [never, nixosfl_console] diff --git a/playbooks/roles/nixosfl/tasks/ssh-access.yml b/playbooks/roles/nixosfl/tasks/ssh-access.yml new file mode 100644 index 00000000..860ffa50 --- /dev/null +++ b/playbooks/roles/nixosfl/tasks/ssh-access.yml @@ -0,0 +1,70 @@ +--- +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Wait for each guest to acquire a DHCP lease, confirm that sshd is +# reachable, and update the user's ~/.ssh/config so kdevops targets +# the guests by inventory name. Common to both backend paths; qsu +# will populate nixosfl_vm_ips differently but the +# downstream wait_for and ssh-config update are reusable. + +- name: Localhost ssh_access + delegate_to: localhost + run_once: true + block: + - name: Wait for VMs to get IP addresses from DHCP + ansible.builtin.shell: | + set -o pipefail + for _ in $(seq 1 90); do + IP=$(virsh domifaddr {{ item }} --source lease 2>/dev/null \ + | awk '/192\.168\.122\./ {print $4}' \ + | cut -d'/' -f1) + if [ -n "$IP" ]; then + echo "$IP" + exit 0 + fi + sleep 3 + done + exit 1 + loop: "{{ groups['all'] | reject('equalto', 'localhost') | list }}" + register: nixosfl_vm_ips + retries: 2 + delay: 10 + changed_when: false + environment: + LIBVIRT_DEFAULT_URI: "{{ libvirt_uri | default('qemu:///system') }}" + + - name: Set VM IP facts + ansible.builtin.set_fact: + nixosfl_vm_ips: >- + {{ dict(groups['all'] + | reject('equalto', 'localhost') | list + | zip(nixosfl_vm_ips.results | map(attribute='stdout'))) }} + + - name: Determine SSH key path for SSH config update + ansible.builtin.command: python3 {{ playbook_dir }}/../scripts/nixosfl_ssh_key_name.py --path + register: nixosfl_ssh_key_path_for_config + changed_when: false + + - name: Wait for SSH to be available on VMs + ansible.builtin.wait_for: + host: "{{ nixosfl_vm_ips[item] }}" + port: 22 + delay: 10 + timeout: 300 + loop: "{{ groups['all'] | reject('equalto', 'localhost') | list }}" + + - name: Update SSH config for NixOS VMs + ansible.builtin.command: | + python3 {{ playbook_dir }}/../scripts/update_ssh_config_nixosfl.py update \ + {{ item }} \ + {{ nixosfl_vm_ips[item] }} \ + {{ ansible_cfg_ssh_port }} \ + kdevops \ + {{ nixosfl_ssh_config_file | default(lookup('env', 'HOME') ~ '/.ssh/config') }} \ + {{ nixosfl_ssh_key_path_for_config.stdout | trim }} \ + 'NixOS VM' + loop: "{{ groups['all'] | reject('equalto', 'localhost') | list }}" + register: nixosfl_ssh_config_update + changed_when: nixosfl_ssh_config_update.rc == 0 + when: nixosfl_update_ssh_config | default(true) | bool + tags: do_deploy diff --git a/playbooks/roles/nixosfl/templates/default.nix.j2 b/playbooks/roles/nixosfl/templates/default.nix.j2 new file mode 100644 index 00000000..c09c4576 --- /dev/null +++ b/playbooks/roles/nixosfl/templates/default.nix.j2 @@ -0,0 +1,85 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Per-node overrides for {{ vm_name }}. Generated by kdevops. +# +# The flake that imports this module (copied from +# scripts/nixos-flake/templates/libvirt/flake.nix) +# passes `nixos-flake` and the flake inputs through specialArgs, so +# backend, profile, mount, and testSuite modules are imported +# directly and mirror substituters attach to nixos-flake's overlay +# through plain nixpkgs config. +{ config, lib, pkgs, nixos-flake, inputs, ... }: { + imports = [ + nixos-flake.nixosModules.backends.libvirt + nixos-flake.nixosModules.user +{% if nixos_flake_profile_devel is defined and nixos_flake_profile_devel %} + nixos-flake.nixosModules.profiles.devel +{% endif %} +{% if nixos_flake_profile_build_tools is defined and nixos_flake_profile_build_tools %} + nixos-flake.nixosModules.profiles.build-tools +{% endif %} +{% if nixos_flake_profile_monitoring is defined and nixos_flake_profile_monitoring %} + nixos-flake.nixosModules.profiles.monitoring +{% endif %} +{% if nixos_flake_mount_shares is defined and nixos_flake_mount_shares %} + nixos-flake.nixosModules.mounts.shares +{% endif %} +{% if nixos_flake_mount_storage is defined and nixos_flake_mount_storage %} + nixos-flake.nixosModules.mounts.storage +{% endif %} +{% if nixos_flake_testsuite_fstests is defined and nixos_flake_testsuite_fstests %} + nixos-flake.nixosModules.testSuites.fstests +{% endif %} +{% if nixos_flake_testsuite_blktests is defined and nixos_flake_testsuite_blktests %} + nixos-flake.nixosModules.testSuites.blktests +{% endif %} +{% if nixos_flake_testsuite_ltp is defined and nixos_flake_testsuite_ltp %} + nixos-flake.nixosModules.testSuites.ltp +{% endif %} +{% if nixos_flake_testsuite_mmtests is defined and nixos_flake_testsuite_mmtests %} + nixos-flake.nixosModules.testSuites.mmtests +{% endif %} +{% if nixos_flake_testsuite_pynfs is defined and nixos_flake_testsuite_pynfs %} + nixos-flake.nixosModules.testSuites.pynfs +{% endif %} +{% if nixos_flake_testsuite_gitr is defined and nixos_flake_testsuite_gitr %} + nixos-flake.nixosModules.testSuites.gitr +{% endif %} +{% if nixos_flake_testsuite_selftests is defined and nixos_flake_testsuite_selftests %} + nixos-flake.nixosModules.testSuites.selftests +{% endif %} +{% if nixos_flake_testsuite_sysbench is defined and nixos_flake_testsuite_sysbench %} + nixos-flake.nixosModules.testSuites.sysbench +{% endif %} + ]; + + networking.hostName = "{{ vm_name }}"; + + nixos-flake.user.name = "kdevops"; + +{% if nixosfl_ssh_authorized_key is defined %} + users.users.root.openssh.authorizedKeys.keys = [ "{{ nixosfl_ssh_authorized_key }}" ]; + users.users.kdevops.openssh.authorizedKeys.keys = [ "{{ nixosfl_ssh_authorized_key }}" ]; +{% endif %} + +{% if nixosfl_use_local_mirror is defined and nixosfl_use_local_mirror and nixosfl_mirror_url is defined and nixosfl_mirror_url != "" %} + nix.settings = { + substituters = [ + "{{ nixosfl_mirror_url }}" + "https://cache.nixos.org" + ]; + extra-substituters = [ "{{ nixosfl_mirror_url }}" ]; + }; +{% endif %} + +{% if bootlinux_9p is defined and bootlinux_9p %} + # 9P mount for shared kernel source + boot.kernelModules = [ "9p" "9pnet_virtio" ]; + boot.initrd.kernelModules = [ "9p" "9pnet_virtio" ]; + fileSystems."/mnt/linux" = { + device = "linux_source"; + fsType = "9p"; + options = [ "trans=virtio" "version=9p2000.L" "cache=loose" ]; + }; +{% endif %} +} diff --git a/playbooks/roles/nixosfl/templates/flake.nix.j2 b/playbooks/roles/nixosfl/templates/flake.nix.j2 new file mode 100644 index 00000000..9e1be527 --- /dev/null +++ b/playbooks/roles/nixosfl/templates/flake.nix.j2 @@ -0,0 +1,48 @@ +{# + SPDX-License-Identifier: copyleft-next-0.3.1 + + Per-VM libvirt flake rendered by kdevops. Mirrors upstream's starter + at scripts/nixos-flake/templates/libvirt/flake.nix and resolves the + nixos-flake input at the vendored subtree. Builds a qcow2 disk image + (packages.<system>.image) that libvirtd runs. +#} +{ + inputs = { + nixos-flake.url = "path:{{ topdir_path }}/scripts/nixos-flake"; + nixpkgs.follows = "nixos-flake/nixpkgs"; + }; + + outputs = + { + self, + nixpkgs, + nixos-flake, + ... + }@inputs: + let + system = "x86_64-linux"; + pkgs = import nixpkgs { inherit system; }; + in + { + nixosConfigurations.vm = nixpkgs.lib.nixosSystem { + inherit system; + specialArgs = { inherit inputs nixos-flake; }; + modules = [ + nixos-flake.nixosModules.backends.libvirt + nixos-flake.nixosModules.user + { nixpkgs.overlays = [ nixos-flake.overlays.default ]; } + ./default.nix + ]; + }; + + packages.${system}.image = import "${nixpkgs}/nixos/lib/make-disk-image.nix" { + inherit pkgs; + inherit (pkgs) lib; + config = self.nixosConfigurations.vm.config; + diskSize = 20480; + format = "qcow2"; + partitionTableType = "legacy"; + installBootLoader = true; + }; + }; +} diff --git a/playbooks/roles/nixosfl/templates/vm-libvirt.xml.j2 b/playbooks/roles/nixosfl/templates/vm-libvirt.xml.j2 new file mode 100644 index 00000000..2ace5784 --- /dev/null +++ b/playbooks/roles/nixosfl/templates/vm-libvirt.xml.j2 @@ -0,0 +1,91 @@ +<domain type='kvm'> + <name>{{ vm_name }}</name> + <memory unit='MiB'>{{ vm_memory }}</memory> + <vcpu placement='static'>{{ vm_vcpus }}</vcpu> + + <os> + <type arch='x86_64' machine='q35'>hvm</type> + <boot dev='hd'/> + </os> + + <features> + <acpi/> + <apic/> + <vmport state='off'/> + </features> + + <cpu mode='host-passthrough'> + <topology sockets='1' cores='{{ vm_vcpus }}' threads='1'/> + </cpu> + + <clock offset='utc'> + <timer name='rtc' tickpolicy='catchup'/> + <timer name='pit' tickpolicy='delay'/> + <timer name='hpet' present='no'/> + </clock> + + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + + <pm> + <suspend-to-mem enabled='no'/> + <suspend-to-disk enabled='no'/> + </pm> + + <devices> + <emulator>/usr/bin/qemu-system-x86_64</emulator> + + <disk type='file' device='disk'> + <driver name='qemu' type='qcow2' cache='none' io='native'/> + <source file='{{ vm_disk }}'/> + <target dev='vda' bus='virtio'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> + </disk> + +{% if bootlinux_9p is defined and bootlinux_9p %} + <!-- 9P filesystem for kernel source sharing --> + <filesystem type='mount' accessmode='passthrough'> + <source dir='{{ topdir_path }}/linux'/> + <target dir='linux_source'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> + </filesystem> +{% endif %} + + <interface type='network'> + <source network='default'/> + <model type='virtio'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> + </interface> + + <serial type='pty'> + <target type='isa-serial' port='0'> + <model name='isa-serial'/> + </target> + </serial> + + <console type='pty'> + <target type='serial' port='0'/> + </console> + + <input type='tablet' bus='usb'> + <address type='usb' bus='0' port='1'/> + </input> + + <input type='mouse' bus='ps2'/> + <input type='keyboard' bus='ps2'/> + + <graphics type='vnc' port='-1' autoport='yes'> + <listen type='address' address='127.0.0.1'/> + </graphics> + + <memballoon model='virtio'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> + </memballoon> + + <rng model='virtio'> + <backend model='random'>/dev/urandom</backend> + <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> + </rng> + </devices> +</domain> diff --git a/scripts/bringup.Makefile b/scripts/bringup.Makefile index 319c69a4..536997a3 100644 --- a/scripts/bringup.Makefile +++ b/scripts/bringup.Makefile @@ -22,11 +22,23 @@ status: $(KDEVOPS_STATUS_DEPS) destroy: $(KDEVOPS_DESTROY_DEPS) +console: $(KDEVOPS_CONSOLE_DEPS) +PHONY += console + +rebuild-boot: $(KDEVOPS_REBUILD_BOOT_DEPS) +PHONY += rebuild-boot + +rebuild-test: $(KDEVOPS_REBUILD_TEST_DEPS) +PHONY += rebuild-test + bringup-help-menu: @echo "Bringup targets:" @echo "bringup - Brings up target hosts" @echo "status - Reports the status of target hosts" @echo "destroy - Destroy all target hosts" + @echo "console - Attach to a guest serial console (where the backend supports it)" + @echo "rebuild-boot - Rebuild guest closure and restart (imageless / qsu only)" + @echo "rebuild-test - nixos-rebuild test on running guests (imageless / qsu only)" @echo "cleancache - Remove all cached images" @echo "" diff --git a/scripts/nixosfl.Makefile b/scripts/nixosfl.Makefile new file mode 100644 index 00000000..c477107f --- /dev/null +++ b/scripts/nixosfl.Makefile @@ -0,0 +1,104 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Bringup glue for CONFIG_NIXOSFL. +# +# User-facing entry points are the kdevops top-level targets: +# `make bringup`, `make destroy`, `make console`. They dispatch +# through the KDEVOPS_PROVISION_*_METHOD variables below to the +# per-phase tag-gated targets in playbooks/nixosfl.yml. The +# nixosfl-* targets in this file are dispatch internals; +# users do not call them directly. +# +# This backend does not set REBUILD_BOOT_METHOD or REBUILD_TEST_METHOD: +# the libvirt qcow2 bakes its kernel at image-build time. `make +# rebuild-boot` and `make rebuild-test` are imageless-only (qsu). + +KDEVOPS_NODES_TEMPLATE := $(KDEVOPS_NODES_ROLE_TEMPLATE_DIR)/guestfs_nodes.j2 +KDEVOPS_NODES := guestfs/kdevops_nodes.yaml + +export KDEVOPS_PROVISIONED_SSH := $(KDEVOPS_PROVISIONED_SSH_DEFAULT_GUARD) + +NIXOSFL_BRINGUP_DEPS := + +ifeq (y,$(CONFIG_BOOTLINUX_9P)) +NIXOSFL_BRINGUP_DEPS += 9p_linux_clone +endif + +KDEVOPS_PROVISION_METHOD := bringup_nixosfl +KDEVOPS_PROVISION_DESTROY_METHOD := destroy_nixosfl +KDEVOPS_PROVISION_STATUS_METHOD := nixosfl-status +KDEVOPS_PROVISION_CONSOLE_METHOD := nixosfl-console + +$(KDEVOPS_PROVISIONED_SSH): $(KDEVOPS_HOSTS_PREFIX) + $(Q)touch $(KDEVOPS_PROVISIONED_SSH) + +bringup_nixosfl: $(NIXOSFL_BRINGUP_DEPS) + $(Q)$(MAKE) nixosfl-bringup +PHONY += bringup_nixosfl + +destroy_nixosfl: + $(Q)$(MAKE) nixosfl-destroy +PHONY += destroy_nixosfl + +PHONY += nixos-flake-runtime-deps-setup +nixos-flake-runtime-deps-setup: $(KDEVOPS_EXTRA_VARS) + $(Q)ansible-playbook \ + $(KDEVOPS_PLAYBOOKS_DIR)/nixosfl.yml \ + --tags nixos_flake_runtime_deps_setup \ + --extra-vars=@./extra_vars.yaml + +PHONY += nixosfl-bringup +nixosfl-bringup: $(KDEVOPS_EXTRA_VARS) + $(Q)ansible-playbook \ + $(KDEVOPS_PLAYBOOKS_DIR)/nixosfl.yml \ + --tags nixosfl_bringup \ + --extra-vars=@./extra_vars.yaml + +PHONY += nixosfl-destroy +nixosfl-destroy: $(KDEVOPS_EXTRA_VARS) + $(Q)ansible-playbook \ + $(KDEVOPS_PLAYBOOKS_DIR)/nixosfl.yml \ + --tags nixosfl_destroy \ + --extra-vars=@./extra_vars.yaml + +PHONY += nixosfl-console +# Attach to a running libvirt domain's serial console via virsh. +# Defaults to the single inventory host; pass VM=<name> when more +# than one guest is declared. +nixosfl-console: + $(Q)set -e; \ + if [ -n "$$VM" ]; then \ + vm="$$VM"; \ + else \ + hosts=$$(ansible all --list-hosts 2>/dev/null \ + | tail -n +2 | awk '$$1 != "localhost" {print $$1}'); \ + count=$$(printf '%s\n' "$$hosts" | sed '/^$$/d' | wc -l); \ + case $$count in \ + 0) echo "no inventory hosts found"; exit 1 ;; \ + 1) vm="$$hosts" ;; \ + *) echo "multiple guests in inventory, pick one with VM=<name>:"; \ + printf ' %s\n' $$hosts; exit 1 ;; \ + esac; \ + fi; \ + uri=$${LIBVIRT_DEFAULT_URI:-qemu:///system}; \ + command -v virsh >/dev/null || { echo "virsh is required"; exit 1; }; \ + exec virsh -c "$$uri" console "$$vm" + +PHONY += nixosfl-status +# Report virsh domstate for every guest the active inventory +# declares. +nixosfl-status: + $(Q)set -e; \ + hosts=$$(ansible all --list-hosts 2>/dev/null \ + | tail -n +2 | awk '$$1 != "localhost" {print $$1}'); \ + if [ -z "$$hosts" ]; then \ + echo "no inventory hosts found (run 'make' first?)"; \ + exit 0; \ + fi; \ + uri=$${LIBVIRT_DEFAULT_URI:-qemu:///system}; \ + command -v virsh >/dev/null || { echo "virsh is required"; exit 1; }; \ + printf '%-20s %s\n' VM STATE; \ + for vm in $$hosts; do \ + st=$$(virsh -c "$$uri" domstate "$$vm" 2>/dev/null || echo absent); \ + printf '%-20s %s\n' "$$vm" "$$st"; \ + done diff --git a/scripts/nixosfl_ssh_key_name.py b/scripts/nixosfl_ssh_key_name.py new file mode 100755 index 00000000..8dffb993 --- /dev/null +++ b/scripts/nixosfl_ssh_key_name.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: copyleft-next-0.3.1 + +"""Generate SSH key name for NixOS VMs based on directory location.""" + +import os +import sys +import hashlib + + +def get_ssh_key_name(): + """Generate SSH key name based on kdevops project directory.""" + # Find the kdevops root directory + # Start from the script's location + script_dir = os.path.dirname(os.path.abspath(__file__)) + + # The script is in kdevops/scripts/, so go up one level + kdevops_root = os.path.dirname(script_dir) + + # Use the kdevops root directory for consistent key naming + # This ensures the same key is used regardless of where the script is called from + cwd = kdevops_root + + # Get the last two directory components for the key name + path_parts = cwd.split("/") + if len(path_parts) >= 2: + # Use last two directories + key_suffix = "-".join(path_parts[-2:]) + else: + # Use just the last directory + key_suffix = path_parts[-1] if path_parts else "kdevops" + + # Create a short hash to ensure uniqueness + path_hash = hashlib.sha256(cwd.encode()).hexdigest()[:8] + + # Construct the key name + key_name = f"kdevops-nixos-{key_suffix}-{path_hash}" + + return key_name + + +def main(): + """Main function.""" + if len(sys.argv) > 1 and sys.argv[1] == "--path": + # Return full path to key + key_name = get_ssh_key_name() + key_path = os.path.expanduser(f"~/.ssh/{key_name}") + print(key_path) + else: + # Return just the key name + print(get_ssh_key_name()) + + +if __name__ == "__main__": + main() diff --git a/scripts/provision.Makefile b/scripts/provision.Makefile index 378cbde4..9cfeca87 100644 --- a/scripts/provision.Makefile +++ b/scripts/provision.Makefile @@ -12,6 +12,22 @@ KDEVOPS_PROVISION_STATUS_METHOD := # the systems will be removed after this KDEVOPS_PROVISION_DESTROY_METHOD := +# Provisioning methods that expose a guest serial console set this to a +# target that attaches to one. Optional: a backend that has no console +# concept leaves it empty and `make console` is a no-op. +KDEVOPS_PROVISION_CONSOLE_METHOD := + +# Provisioning methods that boot an externally-built kernel can rebuild the +# guest closure (and restart VMs) under a single target. Imageless / qsu +# sets this; libvirt-flake leaves it empty because the kernel is baked into +# the qcow2 at image-build time. +KDEVOPS_PROVISION_REBUILD_BOOT_METHOD := + +# Same shape but for `nixos-rebuild test`-style switches that do not +# require a reboot. Imageless / qsu sets this; libvirt-flake leaves it +# empty. +KDEVOPS_PROVISION_REBUILD_TEST_METHOD := + # The default guard for ssh provisioning. Provisioning methods can set the # KDEVOPS_PROVISIONED_SSH to this if they are OK with the default guard. KDEVOPS_PROVISIONED_SSH_DEFAULT_GUARD := .provisioned_once_ssh @@ -64,6 +80,10 @@ ifeq (y,$(CONFIG_NIXOS)) include scripts/nixos.Makefile endif +ifeq (y,$(CONFIG_NIXOSFL)) +include scripts/nixosfl.Makefile +endif + KDEVOPS_MRPROPER += $(KDEVOPS_PROVISIONED_SSH) KDEVOPS_MRPROPER += $(KDEVOPS_PROVISIONED_DEVCONFIG) @@ -93,3 +113,7 @@ KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_PROVISIONED_SSH) KDEVOPS_STATUS_DEPS += $(KDEVOPS_PROVISION_STATUS_METHOD) KDEVOPS_DESTROY_DEPS += $(KDEVOPS_PROVISION_DESTROY_METHOD) + +KDEVOPS_CONSOLE_DEPS := $(KDEVOPS_PROVISION_CONSOLE_METHOD) +KDEVOPS_REBUILD_BOOT_DEPS := $(KDEVOPS_PROVISION_REBUILD_BOOT_METHOD) +KDEVOPS_REBUILD_TEST_DEPS := $(KDEVOPS_PROVISION_REBUILD_TEST_METHOD) diff --git a/scripts/update_ssh_config_nixosfl.py b/scripts/update_ssh_config_nixosfl.py new file mode 100755 index 00000000..a1a05b5b --- /dev/null +++ b/scripts/update_ssh_config_nixosfl.py @@ -0,0 +1,216 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: copyleft-next-0.3.1 + +"""Update SSH config for NixOS guests provisioned by kdevops. + +Manages SSH configuration entries for kdevops-deployed NixOS guests +provisioned through the nixos_flake libvirt backend, or driven by the +qsu systemd-machined VMs against a nixos-flake imageless closure. + +Usage: + update_ssh_config_nixosfl.py update <hostname> <host> <port> <user> + <ssh_config> <privkey> <tag> + [--vsock-cid <N>] [--default-transport vsock|tcp] + update_ssh_config_nixosfl.py remove <hostname> '' '' '' <ssh_config> + '' <tag> + +When --vsock-cid is given, emit three Host stanzas: + <hostname>-vsock : ProxyCommand systemd-ssh-proxy vsock/<cid> 22 + <hostname>-tcp : TCP via <host>:<port> + <hostname> : alias of whichever --default-transport picks + (defaults to vsock when --vsock-cid is set). + +Without --vsock-cid, one <hostname> stanza is emitted over TCP +(backward-compatible with the nixos_flake libvirt caller). +""" + +import argparse +import os +import re +import sys + +SYSTEMD_SSH_PROXY = "/usr/lib/systemd/systemd-ssh-proxy" + + +def render_vsock_stanza(host_tokens, user, key, cid): + return ( + f"Host {host_tokens}\n" + f" User {user}\n" + f" IdentityFile {key}\n" + f" ProxyCommand {SYSTEMD_SSH_PROXY} vsock/{cid} 22\n" + f" ProxyUseFdpass yes\n" + f" CheckHostIP no\n" + f" StrictHostKeyChecking no\n" + f" UserKnownHostsFile /dev/null\n" + f" LogLevel ERROR\n" + ) + + +def render_tcp_stanza(host_tokens, host_ip, port, user, key): + return ( + f"Host {host_tokens}\n" + f" HostName {host_ip}\n" + f" Port {port}\n" + f" User {user}\n" + f" IdentityFile {key}\n" + f" StrictHostKeyChecking no\n" + f" UserKnownHostsFile /dev/null\n" + f" LogLevel ERROR\n" + ) + + +def build_entries(hostname, host_ip, port, user, key, vsock_cid, default_transport): + """Return a list of (alias, block_body) pairs to write. + + Each block starts with its own `# kdevops-managed: {tag} - {alias}` + marker so remove-on-update can match and drop the old block via + regex. + """ + if vsock_cid is None: + # Legacy single-stanza form used by the nixos_flake libvirt path. + return [ + (hostname, render_tcp_stanza(hostname, host_ip, port, user, key)), + ] + + # Three-stanza form: default alias groups with either -vsock or -tcp. + if default_transport == "tcp": + default_alias = f"{hostname} {hostname}-tcp" + bare_stanza = render_tcp_stanza(default_alias, host_ip, port, user, key) + return [ + (hostname, bare_stanza), + ( + f"{hostname}-vsock", + render_vsock_stanza(f"{hostname}-vsock", user, key, vsock_cid), + ), + ] + + # default_transport == "vsock" + default_alias = f"{hostname} {hostname}-vsock" + bare_stanza = render_vsock_stanza(default_alias, user, key, vsock_cid) + return [ + (hostname, bare_stanza), + ( + f"{hostname}-tcp", + render_tcp_stanza(f"{hostname}-tcp", host_ip, port, user, key), + ), + ] + + +def managed_block_regex(tag, hostname): + """Match every `# kdevops-managed: {tag} - <alias>` block whose alias + is either the base hostname or starts with it followed by a dash. + Catches qsu, qsu-vsock, qsu-tcp in one sweep without disturbing + unrelated entries that share the tag prefix. + """ + return re.compile( + rf"^# kdevops-managed: {re.escape(tag)} - " + rf"{re.escape(hostname)}(?:-[\w-]+)?\n" + r"Host [^\n]+\n" + r"(?:[ \t]+[^\n]+\n)*", + re.MULTILINE, + ) + + +def update_ssh_config( + action, + hostname, + host_ip, + port, + username, + ssh_config_path, + ssh_key_path, + tag, + vsock_cid, + default_transport, +): + ssh_config_path = os.path.expanduser(ssh_config_path) + os.makedirs(os.path.dirname(ssh_config_path), exist_ok=True) + + config_content = "" + if os.path.exists(ssh_config_path): + with open(ssh_config_path, "r") as f: + config_content = f.read() + + entry_pattern = managed_block_regex(tag, hostname) + + # Always drop prior managed blocks for this base hostname. + config_content = entry_pattern.sub("", config_content) + + if action == "remove": + print(f"Removed SSH config entries for {hostname}") + elif action == "update": + blocks = [] + for alias, body in build_entries( + hostname, + host_ip, + port, + username, + ssh_key_path, + vsock_cid, + default_transport, + ): + blocks.append(f"# kdevops-managed: {tag} - {alias}\n{body}") + new_entries = "\n".join(blocks) + config_content = config_content.rstrip() + "\n\n" + new_entries + "\n" + if vsock_cid is None: + print(f"Updated SSH config entry for {hostname} (port {port})") + else: + print( + f"Updated SSH config entries for {hostname}, " + f"{hostname}-vsock, {hostname}-tcp " + f"(default={default_transport}, vsock CID={vsock_cid}, " + f"TCP port={port})" + ) + + with open(ssh_config_path, "w") as f: + f.write(config_content) + + +def main(): + parser = argparse.ArgumentParser( + description="Manage SSH config entries for kdevops-deployed NixOS guests.", + ) + parser.add_argument("action", choices=["update", "remove"]) + parser.add_argument("hostname") + parser.add_argument("host_ip", nargs="?", default="localhost") + parser.add_argument("port", nargs="?", default="22") + parser.add_argument("username", nargs="?", default="kdevops") + parser.add_argument("ssh_config_path") + parser.add_argument("ssh_key_path", nargs="?", default="") + parser.add_argument("tag", nargs="?", default="NixOS VM") + parser.add_argument( + "--vsock-cid", + type=int, + default=None, + help="VSOCK CID of the VM. When set, emit qsu-style three-stanza " + "layout (host, host-vsock, host-tcp).", + ) + parser.add_argument( + "--default-transport", + choices=["vsock", "tcp"], + default="vsock", + help="Which transport the bare <hostname> alias groups with. Only " + "meaningful when --vsock-cid is set.", + ) + args = parser.parse_args() + + try: + update_ssh_config( + args.action, + args.hostname, + args.host_ip or "localhost", + args.port or "22", + args.username or "kdevops", + args.ssh_config_path, + args.ssh_key_path, + args.tag, + args.vsock_cid, + args.default_transport, + ) + except Exception as e: + print(f"Error: {e}") + sys.exit(1) + + +if __name__ == "__main__": + main() -- 2.54.0