[PATCH 1/8] scripts: import nixos-qemu at scripts/nixos-qemu
Daniel Gomez <[email protected]> Thu, 23 Apr 2026 00:48:39 +0200
| Newsgroups | dev.linux.lists.kdevops |
|---|---|
| Message-ID | <20260423-kdevops-series-b-nixos-qemu-v1-1-209154ae54f8@samsung.com> |
From: Daniel Gomez <[email protected]> Import the nixos-qemu NixOS flake from https://github.com/linux-kdevops/nixos-qemu at tag/commit 11b6f9d156ff50c0b73b01933d5589c46da470eb into scripts/nixos-qemu/. nixos-qemu exposes two backend modules for kdevops guests: a libvirt backend (qcow2, grub, DHCP from libvirtd, drop-in for the existing kdevops NIXOS bringup) and an imageless backend (tmpfs root, systemd initramfs, external kernel, /nix/store and /lib/modules mounted over virtiofs from the host) consumed by the follow-up qemu-system-units series. The git-subtree-dir and git-subtree-split trailers below let future syncs with upstream use git subtree pull/push. Generated-by: Claude AI Signed-off-by: Daniel Gomez <[email protected]> git-subtree-dir: scripts/nixos-qemu git-subtree-split: 11b6f9d156ff50c0b73b01933d5589c46da470eb --- scripts/nixos-qemu/.gitignore | 2 + scripts/nixos-qemu/CLAUDE.md | 175 +++++ scripts/nixos-qemu/COPYING | 22 + scripts/nixos-qemu/LICENSE | 5 + .../LICENSES/preferred/copyleft-next-0.3.1 | 239 +++++++ scripts/nixos-qemu/README.md | 116 ++++ scripts/nixos-qemu/docs/design-decisions.md | 351 ++++++++++ scripts/nixos-qemu/docs/usage.md | 722 +++++++++++++++++++++ scripts/nixos-qemu/flake.lock | 27 + scripts/nixos-qemu/flake.nix | 86 +++ scripts/nixos-qemu/modules/build-tools.nix | 23 + scripts/nixos-qemu/modules/devel.nix | 118 ++++ scripts/nixos-qemu/modules/imageless.nix | 114 ++++ scripts/nixos-qemu/modules/libvirt.nix | 58 ++ scripts/nixos-qemu/modules/shares.nix | 41 ++ scripts/nixos-qemu/modules/storage.nix | 106 +++ scripts/nixos-qemu/modules/user.nix | 42 ++ scripts/nixos-qemu/modules/workflows/blktests.nix | 29 + scripts/nixos-qemu/modules/workflows/fstests.nix | 41 ++ scripts/nixos-qemu/modules/workflows/gitr.nix | 21 + scripts/nixos-qemu/modules/workflows/ltp.nix | 33 + scripts/nixos-qemu/modules/workflows/mmtests.nix | 23 + scripts/nixos-qemu/modules/workflows/pynfs.nix | 22 + scripts/nixos-qemu/modules/workflows/selftests.nix | 25 + scripts/nixos-qemu/modules/workflows/sysbench.nix | 26 + scripts/nixos-qemu/overlays/default.nix | 14 + scripts/nixos-qemu/overlays/fio.nix | 36 + scripts/nixos-qemu/overlays/xfstests.nix | 23 + scripts/nixos-qemu/pkgs/cpupower.nix | 64 ++ scripts/nixos-qemu/pkgs/damo.nix | 58 ++ scripts/nixos-qemu/pkgs/default.nix | 21 + scripts/nixos-qemu/pkgs/libbpf-tools.nix | 117 ++++ scripts/nixos-qemu/pkgs/nfstest.nix | 67 ++ scripts/nixos-qemu/pkgs/pynfs.nix | 94 +++ scripts/nixos-qemu/pkgs/xnvme.nix | 49 ++ scripts/nixos-qemu/templates/imageless/default.nix | 33 + scripts/nixos-qemu/templates/imageless/flake.nix | 51 ++ scripts/nixos-qemu/templates/libvirt/default.nix | 33 + scripts/nixos-qemu/templates/libvirt/flake.nix | 58 ++ 39 files changed, 3185 insertions(+) diff --git a/scripts/nixos-qemu/.gitignore b/scripts/nixos-qemu/.gitignore new file mode 100644 index 00000000..34554eca --- /dev/null +++ b/scripts/nixos-qemu/.gitignore @@ -0,0 +1,2 @@ +result +configurations/ diff --git a/scripts/nixos-qemu/CLAUDE.md b/scripts/nixos-qemu/CLAUDE.md new file mode 100644 index 00000000..1a6a5ec5 --- /dev/null +++ b/scripts/nixos-qemu/CLAUDE.md @@ -0,0 +1,175 @@ +# CLAUDE.md + +## Project Overview + +Library flake of NixOS modules, overlays, and templates for +building QEMU VMs. Two backend modules: `imageless` (tmpfs root, +systemd initramfs, external kernel, `/nix/store` and +`/lib/modules` via virtiofs) and `libvirt` (qcow2 disk root, grub, +NixOS-built kernel). Four opt-in modules compose on top: `user`, +`shares`, `storage`, `devel`. + +**License**: copyleft-next-0.3.1 + +## Project Structure + +``` +nixos-qemu/ +├── flake.nix Flake (nixosModules, overlays, templates, packages, checks) +├── flake.lock Pinned nixpkgs revision +├── docs/ +│ ├── usage.md Configurations, overlays, packages, updating +│ └── design-decisions.md Hardcoded choices and upstream references +├── modules/ +│ ├── imageless.nix Imageless boot (tmpfs root, virtiofs, networkd), standalone +│ ├── libvirt.nix Disk-image boot (grub on vda, DHCP, nix+flakes), standalone +│ ├── user.nix Opt-in unprivileged account with configurable name (options.nixos-qemu.user) +│ ├── shares.nix Opt-in virtiofs shares (options.nixos-qemu.shares) +│ ├── storage.nix Opt-in block-device mounts with optional mkfs (options.nixos-qemu.storage) +│ └── devel.nix Development profile (kernel testing tools) +├── pkgs/ +│ ├── default.nix Custom packages via callPackage +│ ├── cpupower.nix Standalone cpupower from kernel source tree +│ ├── damo.nix DAMON user-space tool +│ ├── libbpf-tools.nix CO-RE BPF tracing tools from BCC +│ ├── nfstest.nix NFS test suite +│ ├── pynfs.nix Python NFSv4 conformance tests +│ └── xnvme.nix Cross-platform NVMe library and tools +├── overlays/ +│ ├── default.nix Composes per-package overlays + pkgs/ +│ ├── fio.nix fio with liburing + test suite + examples +│ └── xfstests.nix Bump xfstests to 2026.03.20 +├── templates/ +│ ├── imageless/flake.nix Imageless starter (default) +│ └── libvirt/flake.nix Libvirt disk-image starter +├── LICENSES/ +│ └── preferred/copyleft-next-0.3.1 +├── COPYING License overview and dual-licensing guidance +├── LICENSE Copyright and license reference +└── README.md +``` + +## Critical Rules + +### Never fabricate facts + +Every NixOS option must be verified against the NixOS options search +(search.nixos.org/options) or the nixpkgs source. + +### Long-form command options + +NEVER use short flags when a long-form alternative exists. +`--template` not `-t`, `--recursive` not `-r`, `--parents` not +`-p`. Exception: tools without long-form options (`ssh -p`). + +## Rules + +### Imageless: external kernel, NixOS-built initramfs + +`boot.kernel.enable = false` tells NixOS not to build a kernel. +`boot.initrd.systemd.enable = true` tells NixOS to build a systemd +initramfs that mounts root (tmpfs), /nix/store, and /lib/modules +via virtiofs before switch-root into the system closure. + +### Imageless: tmpfs root + +`fileSystems."/" = lib.mkImageMediaOverride { fsType = "tmpfs"; }` +declares root as tmpfs. `mkImageMediaOverride` has higher priority +than default NixOS filesystem declarations. Changes are lost on +shutdown. + +### Imageless: minimal profile + +`imports = [ (modulesPath + "/profiles/minimal.nix") ]` reduces the +closure size by excluding unnecessary packages. Libvirt does not +import this profile; disk-image deployments expect a full system. + +### Libvirt: grub on /dev/vda, ext4 root + +`boot.loader.grub.device = "/dev/vda"` and +`fileSystems."/" = { device = "/dev/vda1"; fsType = "ext4"; }` +assume the standard virt-builder qcow2 layout. Consumers with +different disk bus or partition schemes override these. + +### Password auth + +`users.users.root.initialPassword = "root"` with +`users.mutableUsers = false` sets the root password on every boot +(tmpfs root resets it). The password is a serial-console +break-glass; SSH itself is key-only (each backend module sets +`PasswordAuthentication = false` inline to keep modules +standalone). + +### Overlays + +Each file in `overlays/` overrides one nixpkgs package using +`overrideAttrs`. The `overlays/default.nix` composes all per-package +overlays and merges custom packages from `pkgs/` into a single +overlay exported as `overlays.default`. + +### Custom packages (pkgs/) + +Packages not available in nixpkgs are defined in `pkgs/` using the +`callPackage` pattern from nix.dev. Each package is a file declaring +a function whose arguments are its dependencies. The overlay imports +`pkgs/default.nix` and merges them into the nixpkgs set. + +### Templates + +`templates/imageless/` (default) and `templates/libvirt/` are +copied by `nix flake init --template`. Each imports its matching +backend plus `user`, `devel`, and `overlays.default`. Users add +their own packages and NixOS options below. + +## Build + +```shell +nix flake check # both backends +nix build .#checks.x86_64-linux.imageless # single backend +readlink --canonicalize result +``` + +The `result` symlink points to the system closure. The NixOS init +is at `<closure>/init`. The init= path changes on every rebuild. + +## Git Commit Guidelines + +### One commit per change + +Atomic commits. Spell fixes go in separate commits from code changes. + +### Commit message format + +``` +subsystem: brief description in imperative mood + +Plain English explanation of the change. NEVER use bullet points +or itemized lists in commit messages. + +Generated-by: Claude AI +Signed-off-by: Your Name <[email protected]> +``` + +### Use Signed-off-by and Generated-by tags + +Generated-by MUST be immediately followed by Signed-off-by with NO +empty lines between them. No Co-Authored-By trailer. + +### No shopping cart lists + +NEVER use bullet points or itemized lists in commit messages. Use +plain English paragraphs. + +### Subsystem prefix + +Use `configuration:` for NixOS config changes, `flake:` for flake +changes, `overlays:` for package overlays, `pkgs:` for custom +packages, `templates:` for template changes, `docs:` for +documentation in docs/, `README:` for README changes. Use +`nixos-qemu:` for cross-cutting changes. + +## Related work + +- [run-kernel](https://github.com/metaspace/run-kernel). Rust init + NixOS boot via virtiofs. The direct inspiration for this project's boot model. +- [nixos-shell](https://github.com/Mic92/nixos-shell). Nix-based lightweight QEMU VMs with host mounts. +- [kernel-development-flake](https://github.com/jordanisaacs/kernel-development-flake). Nix flake for Linux kernel development with QEMU. diff --git a/scripts/nixos-qemu/COPYING b/scripts/nixos-qemu/COPYING new file mode 100644 index 00000000..f5479109 --- /dev/null +++ b/scripts/nixos-qemu/COPYING @@ -0,0 +1,22 @@ +This project is provided under: + + SPDX-License-Identifier: copyleft-next-0.3.1 + LICENSES/preferred/copyleft-next-0.3.1 + +In addition, other licenses may also apply. We embrace the same SPDX +practice as used in the Linux kernel, for those details refer to under +Linux: + + Documentation/process/license-rules.rst + +for more details. + +Note: Linux accepts copyleft-next licensed code, however, even though +copyleft-next is GPL v2 compatible we use a dual license tag on Linux +to err on the side of caution. And so, if you ever do wish to use code +from this project on Linux be sure to use this tag instead: + +// SPDX-License-Identifier: GPL-2.0-or-later OR copyleft-next-0.3.1 + +This project equally accepts GPL-2.0 code only as copyleft-next is GPL +v2 compatible. diff --git a/scripts/nixos-qemu/LICENSE b/scripts/nixos-qemu/LICENSE new file mode 100644 index 00000000..edc992ff --- /dev/null +++ b/scripts/nixos-qemu/LICENSE @@ -0,0 +1,5 @@ +Copyright (c) 2026 Daniel Gomez <[email protected]> + +nixos-qemu is licensed under copyleft-next-0.3.1 + +Refer to LICENSES/preferred/copyleft-next-0.3.1 for license details. diff --git a/scripts/nixos-qemu/LICENSES/preferred/copyleft-next-0.3.1 b/scripts/nixos-qemu/LICENSES/preferred/copyleft-next-0.3.1 new file mode 100644 index 00000000..431ea9e9 --- /dev/null +++ b/scripts/nixos-qemu/LICENSES/preferred/copyleft-next-0.3.1 @@ -0,0 +1,239 @@ +Valid-License-Identifier: copyleft-next-0.3.1 +SPDX-URL: https://spdx.org/licenses/copyleft-next-0.3.1 +Usage-Guide: + This license can be used in code, it has been found to be GPLv2 compatible + by attorneys at Redhat and SUSE, however to err on the side of caution, + if used on Linux it's best to only use it together with a GPL2 compatible + license using "OR". You do not have to do this for this project, kdevops, + as this project is licensed under the copyleft-next-0.3.1 license. + To use the copyleft-next-0.3.1 license on Linux put the following SPDX + tag/value pair into a comment according to the placement guidelines in the + licensing rules documentation: + SPDX-License-Identifier: GPL-2.0 OR copyleft-next-0.3.1 + SPDX-License-Identifier: GPL-2.0-only OR copyleft-next 0.3.1 + SPDX-License-Identifier: GPL-2.0+ OR copyleft-next-0.3.1 + SPDX-License-Identifier: GPL-2.0-or-later OR copyleft-next-0.3.1 +License-Text: + +======================================================================= + + copyleft-next 0.3.1 ("this License") + Release date: 2016-04-29 + +1. License Grants; No Trademark License + + Subject to the terms of this License, I grant You: + + a) A non-exclusive, worldwide, perpetual, royalty-free, irrevocable + copyright license, to reproduce, Distribute, prepare derivative works + of, publicly perform and publicly display My Work. + + b) A non-exclusive, worldwide, perpetual, royalty-free, irrevocable + patent license under Licensed Patents to make, have made, use, sell, + offer for sale, and import Covered Works. + + This License does not grant any rights in My name, trademarks, service + marks, or logos. + +2. Distribution: General Conditions + + You may Distribute Covered Works, provided that You (i) inform + recipients how they can obtain a copy of this License; (ii) satisfy the + applicable conditions of sections 3 through 6; and (iii) preserve all + Legal Notices contained in My Work (to the extent they remain + pertinent). "Legal Notices" means copyright notices, license notices, + license texts, and author attributions, but does not include logos, + other graphical images, trademarks or trademark legends. + +3. Conditions for Distributing Derived Works; Outbound GPL Compatibility + + If You Distribute a Derived Work, You must license the entire Derived + Work as a whole under this License, with prominent notice of such + licensing. This condition may not be avoided through such means as + separate Distribution of portions of the Derived Work. + + If the Derived Work includes material licensed under the GPL, You may + instead license the Derived Work under the GPL. + +4. Condition Against Further Restrictions; Inbound License Compatibility + + When Distributing a Covered Work, You may not impose further + restrictions on the exercise of rights in the Covered Work granted under + this License. This condition is not excused merely because such + restrictions result from Your compliance with conditions or obligations + extrinsic to this License (such as a court order or an agreement with a + third party). + + However, You may Distribute a Covered Work incorporating material + governed by a license that is both OSI-Approved and FSF-Free as of the + release date of this License, provided that compliance with such + other license would not conflict with any conditions stated in other + sections of this License. + +5. Conditions for Distributing Object Code + + You may Distribute an Object Code form of a Covered Work, provided that + you accompany the Object Code with a URL through which the Corresponding + Source is made available, at no charge, by some standard or customary + means of providing network access to source code. + + If you Distribute the Object Code in a physical product or tangible + storage medium ("Product"), the Corresponding Source must be available + through such URL for two years from the date of Your most recent + Distribution of the Object Code in the Product. However, if the Product + itself contains or is accompanied by the Corresponding Source (made + available in a customarily accessible manner), You need not also comply + with the first paragraph of this section. + + Each direct and indirect recipient of the Covered Work from You is an + intended third-party beneficiary of this License solely as to this + section 5, with the right to enforce its terms. + +6. Symmetrical Licensing Condition for Upstream Contributions + + If You Distribute a work to Me specifically for inclusion in or + modification of a Covered Work (a "Patch"), and no explicit licensing + terms apply to the Patch, You license the Patch under this License, to + the extent of Your copyright in the Patch. This condition does not + negate the other conditions of this License, if applicable to the Patch. + +7. Nullification of Copyleft/Proprietary Dual Licensing + + If I offer to license, for a fee, a Covered Work under terms other than + a license that is OSI-Approved or FSF-Free as of the release date of this + License or a numbered version of copyleft-next released by the + Copyleft-Next Project, then the license I grant You under section 1 is no + longer subject to the conditions in sections 3 through 5. + +8. Copyleft Sunset + + The conditions in sections 3 through 5 no longer apply once fifteen + years have elapsed from the date of My first Distribution of My Work + under this License. + +9. Pass-Through + + When You Distribute a Covered Work, the recipient automatically receives + a license to My Work from Me, subject to the terms of this License. + +10. Termination + + Your license grants under section 1 are automatically terminated if You + + a) fail to comply with the conditions of this License, unless You cure + such noncompliance within thirty days after becoming aware of it, or + + b) initiate a patent infringement litigation claim (excluding + declaratory judgment actions, counterclaims, and cross-claims) + alleging that any part of My Work directly or indirectly infringes + any patent. + + Termination of Your license grants extends to all copies of Covered + Works You subsequently obtain. Termination does not terminate the + rights of those who have received copies or rights from You subject to + this License. + + To the extent permission to make copies of a Covered Work is necessary + merely for running it, such permission is not terminable. + +11. Later License Versions + + The Copyleft-Next Project may release new versions of copyleft-next, + designated by a distinguishing version number ("Later Versions"). + Unless I explicitly remove the option of Distributing Covered Works + under Later Versions, You may Distribute Covered Works under any Later + Version. + +** 12. No Warranty ** +** ** +** My Work is provided "as-is", without warranty. You bear the risk ** +** of using it. To the extent permitted by applicable law, each ** +** Distributor of My Work excludes the implied warranties of title, ** +** merchantability, fitness for a particular purpose and ** +** non-infringement. ** + +** 13. Limitation of Liability ** +** ** +** To the extent permitted by applicable law, in no event will any ** +** Distributor of My Work be liable to You for any damages ** +** whatsoever, whether direct, indirect, special, incidental, or ** +** consequential damages, whether arising under contract, tort ** +** (including negligence), or otherwise, even where the Distributor ** +** knew or should have known about the possibility of such damages. ** + +14. Severability + + The invalidity or unenforceability of any provision of this License + does not affect the validity or enforceability of the remainder of + this License. Such provision is to be reformed to the minimum extent + necessary to make it valid and enforceable. + +15. Definitions + + "Copyleft-Next Project" means the project that maintains the source + code repository at <https://github.com/copyleft-next/copyleft-next.git/> + as of the release date of this License. + + "Corresponding Source" of a Covered Work in Object Code form means (i) + the Source Code form of the Covered Work; (ii) all scripts, + instructions and similar information that are reasonably necessary for + a skilled developer to generate such Object Code from the Source Code + provided under (i); and (iii) a list clearly identifying all Separate + Works (other than those provided in compliance with (ii)) that were + specifically used in building and (if applicable) installing the + Covered Work (for example, a specified proprietary compiler including + its version number). Corresponding Source must be machine-readable. + + "Covered Work" means My Work or a Derived Work. + + "Derived Work" means a work of authorship that copies from, modifies, + adapts, is based on, is a derivative work of, transforms, translates or + contains all or part of My Work, such that copyright permission is + required. The following are not Derived Works: (i) Mere Aggregation; + (ii) a mere reproduction of My Work; and (iii) if My Work fails to + explicitly state an expectation otherwise, a work that merely makes + reference to My Work. + + "Distribute" means to distribute, transfer or make a copy available to + someone else, such that copyright permission is required. + + "Distributor" means Me and anyone else who Distributes a Covered Work. + + "FSF-Free" means classified as 'free' by the Free Software Foundation. + + "GPL" means a version of the GNU General Public License or the GNU + Affero General Public License. + + "I"/"Me"/"My" refers to the individual or legal entity that places My + Work under this License. "You"/"Your" refers to the individual or legal + entity exercising rights in My Work under this License. A legal entity + includes each entity that controls, is controlled by, or is under + common control with such legal entity. "Control" means (a) the power to + direct the actions of such legal entity, whether by contract or + otherwise, or (b) ownership of more than fifty percent of the + outstanding shares or beneficial ownership of such legal entity. + + "Licensed Patents" means all patent claims licensable royalty-free by + Me, now or in the future, that are necessarily infringed by making, + using, or selling My Work, and excludes claims that would be infringed + only as a consequence of further modification of My Work. + + "Mere Aggregation" means an aggregation of a Covered Work with a + Separate Work. + + "My Work" means the particular work of authorship I license to You + under this License. + + "Object Code" means any form of a work that is not Source Code. + + "OSI-Approved" means approved as 'Open Source' by the Open Source + Initiative. + + "Separate Work" means a work that is separate from and independent of a + particular Covered Work and is not by its nature an extension or + enhancement of the Covered Work, and/or a runtime library, standard + library or similar component that is used to generate an Object Code + form of a Covered Work. + + "Source Code" means the preferred form of a work for making + modifications to it. diff --git a/scripts/nixos-qemu/README.md b/scripts/nixos-qemu/README.md new file mode 100644 index 00000000..9335f1b6 --- /dev/null +++ b/scripts/nixos-qemu/README.md @@ -0,0 +1,116 @@ +# nixos-qemu + +NixOS modules, overlays, and templates for building QEMU VMs. The +flake exposes two backend modules that differ in how the guest +boots: + +- **imageless**: tmpfs root, systemd initramfs, external kernel, + `/nix/store` and `/lib/modules` mounted via virtiofs from the + host. No disk image, nothing persists across boots. +- **libvirt**: qcow2 root disk exposed as `/dev/vda`, grub + bootloader, NixOS-built kernel, DHCP from libvirt's default + network. + +Both compose with four opt-in modules (`user`, `shares`, `storage`, +`devel`) so a configuration can be minimal or carry the full kernel +testing toolchain. + +**License**: copyleft-next-0.3.1 + +## Prerequisites + +The [Nix package manager](https://nixos.org/download/) with flake +support enabled: + +```shell +mkdir --parents ~/.config/nix +echo 'experimental-features = nix-command flakes' >> ~/.config/nix/nix.conf +``` + +## Quick start + +Validate that both backends evaluate and build: + +```shell +nix flake check +``` + +Build a single backend closure: + +```shell +nix build .#checks.x86_64-linux.imageless # or .libvirt +readlink --canonicalize result +``` + +The `result` symlink points to the system closure. For imageless, +`result/boot.json` contains the `init` and `initrd` paths needed to +configure QEMU: + +```shell +cat result/boot.json +``` + +To create a downstream configuration, use a template: + +```shell +nix flake init --template "github:linux-kdevops/nixos-qemu#imageless" +nix flake init --template "github:linux-kdevops/nixos-qemu#libvirt" +``` + +See [docs/usage.md](docs/usage.md) for customizing packages and +NixOS options. + +## How it boots + +### Imageless + +This backend builds two artifacts: a NixOS system closure and a +systemd initramfs. Booting requires an external kernel and QEMU +with virtiofsd sharing the host's `/nix/store` and `/lib/modules` +into the guest. + +The external kernel must have the boot-critical virtio drivers +built-in (`CONFIG_VIRTIO_FS=y`, `CONFIG_VIRTIO_PCI=y`, +`CONFIG_TMPFS=y`). All other drivers can be kernel modules loaded +from `/lib/modules` after switch-root. + +QEMU needs two virtiofsd instances sharing host directories into +the guest with these tags: + +- `store`: the host's `/nix/store` (read-only) +- `modules`: the kernel build's `/lib/modules` directory + +The kernel command line: + +``` +root=tmpfs console=ttyS0,115200 console=hvc0 init=/nix/store/<hash>/init +``` + +systemd in the initramfs reads the NixOS-generated fstab, mounts +root (tmpfs), `/nix/store` (virtiofs tag `store`), and +`/lib/modules` (virtiofs tag `modules`), then switch-roots into +the system closure. The `init=` and `initrd` paths change on +every rebuild and are available in `result/boot.json`. + +### Libvirt + +This backend builds a full NixOS system closure including its own +kernel. The consumer supplies a qcow2 disk image containing the +closure; libvirt presents it to the guest as `/dev/vda`. Grub on +the MBR loads the kernel, initramfs mounts the ext4 root from +`/dev/vda1`, and systemd activation proceeds normally. Networking +comes up via DHCP from libvirt's default network (typically +`192.168.122.0/24`). + +## Documentation + +| Document | Content | +|---|---| +| [docs/usage.md](docs/usage.md) | Configurations, modules, overlays, packages, home overlay, block-device filesystems | +| [docs/design-decisions.md](docs/design-decisions.md) | Imageless and libvirt boot model, upstream references | + +## Related work + +- [run-kernel](https://github.com/metaspace/run-kernel). Rust init + NixOS boot via virtiofs. The direct inspiration for the imageless boot model. +- [nixos-shell](https://github.com/Mic92/nixos-shell). Nix-based lightweight QEMU VMs with host mounts. +- [kernel-development-flake](https://github.com/jordanisaacs/kernel-development-flake). Nix flake for Linux kernel development with QEMU. diff --git a/scripts/nixos-qemu/docs/design-decisions.md b/scripts/nixos-qemu/docs/design-decisions.md new file mode 100644 index 00000000..595c776e --- /dev/null +++ b/scripts/nixos-qemu/docs/design-decisions.md @@ -0,0 +1,351 @@ +# Design decisions + +This project exposes two NixOS backend modules for QEMU VMs: +`imageless` (tmpfs root, systemd initramfs, external kernel, +`/nix/store` and `/lib/modules` via virtiofs) and `libvirt` (qcow2 +root disk, grub, NixOS-built kernel, libvirt DHCP). Each section +below explains the upstream mechanisms the backend relies on and +the design choices the module makes, deferring to NixOS defaults +where practical and using `lib.mkDefault` so downstream consumers +can override. + +# Imageless + +Boots from a NixOS system closure and systemd initramfs running +against the host's `/nix/store` shared read-only into the guest. +Root is tmpfs and does not persist across boots. + +## systemd initramfs + +NixOS builds a systemd-based initramfs where systemd runs as +PID 1. systemd's +[fstab-generator](https://github.com/systemd/systemd/blob/main/src/fstab-generator/fstab-generator.c) +reads the NixOS-generated initrd fstab and creates mount units +for `/nix/store` automatically. All upstream code, no custom init +binary. + +The NixOS module that implements `boot.initrd.systemd` is +[`nixos/modules/system/boot/systemd/initrd.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/boot/systemd/initrd.nix). +The initrd fstab generation and `SYSTEMD_SYSROOT_FSTAB` wiring is +in +[`nixos/modules/tasks/filesystems.nix`](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/tasks/filesystems.nix). + +The configuration enables the systemd initramfs and declares +virtiofs filesystems: + +```nix +boot.initrd.systemd.enable = true; +boot.initrd.supportedFilesystems = [ "virtiofs" ]; + +fileSystems."/nix/store" = { + device = "store"; + fsType = "virtiofs"; +}; +``` + +Kernel command line (standard parameters only): +``` +root=tmpfs console=ttyS0,115200 console=hvc0 init=/nix/store/<hash>/init +``` + +## How the systemd initramfs works + +The boot sequence uses standard systemd and NixOS mechanisms. Every +step is upstream code. + +### NixOS initrd fstab generation + +NixOS generates an initrd-specific fstab from `fileSystems` +declarations. The function `fsNeededForBoot` in +`nixos/lib/utils.nix` determines which filesystems go into the +initrd fstab. It returns true for any filesystem where either +`neededForBoot = true` or the mount point is in `pathsNeededForBoot`: + +```nix +pathsNeededForBoot = [ + "/" + "/nix" + "/nix/store" + "/var" + "/var/log" + "/var/lib" + "/var/lib/nixos" + "/etc" + "/usr" +]; +``` + +`/nix/store` is explicitly in this list. Any `fileSystems."/nix/store"` +declaration is automatically included in the initrd fstab without +requiring `neededForBoot = true`. + +The initrd fstab is written to a file and passed to systemd via the +`SYSTEMD_SYSROOT_FSTAB` environment variable. NixOS wires this up in +`nixos/modules/tasks/filesystems.nix` through +`boot.initrd.systemd.managerEnvironment` and the `initrd-parse-etc` +service environment. + +### systemd fstab-generator in the initrd + +systemd-fstab-generator runs in the initrd and reads the initrd +fstab (from `SYSTEMD_SYSROOT_FSTAB`). For each entry, it generates +a systemd mount unit. The mount points are prefixed with `/sysroot` +because the generator runs in the initrd context +(`src/fstab-generator/fstab-generator.c`, `prefix_sysroot` logic). + +For our configuration, the generator creates: +- `sysroot.mount`: tmpfs on `/sysroot` (from `root=tmpfs`) +- `sysroot-nix-store.mount`: virtiofs `store` on `/sysroot/nix/store` + +### root=tmpfs handling + +systemd-fstab-generator explicitly supports `root=tmpfs` as a +shortcut for a writable tmpfs root (see the `arg_root_what == "tmpfs"` +branch in `src/fstab-generator/fstab-generator.c`): + +```c +} else if (streq(arg_root_what, "tmpfs")) { + /* If root=tmpfs is specified, then take this as shortcut + for a writable tmpfs mount as root */ + what = strdup("rootfs"); + fstype = arg_root_fstype ?: "tmpfs"; +``` + +This creates a tmpfs mount at `/sysroot` with mode 0755. + +### switch-root sequence + +After all initrd mounts complete (`initrd-fs.target`), systemd +performs switch-root to `/sysroot`: + +1. systemd reaches `initrd.target` (all initrd services done) +2. `initrd-cleanup.service` runs +3. `initrd-switch-root.target` activates +4. `initrd-switch-root.service` calls `systemctl switch-root /sysroot` + +After switch-root, `/sysroot` becomes `/`. The virtiofs mount that +was at `/sysroot/nix/store` is now at `/nix/store`. The NixOS +stage-2 init at `/nix/store/<hash>/init` becomes accessible at its +expected path. + +### Kernel module matching + +When NixOS builds the initramfs, it can include kernel modules +from the NixOS kernel package. These modules must match the +running kernel version exactly. With an external custom kernel +(`boot.kernel.enable = false`), the versions will not match and +module loading in the initramfs will fail. + +The solution is to exclude all kernel modules from the initramfs +and provide them via virtiofs instead: + +```nix +boot.initrd.availableKernelModules = lib.mkForce []; +boot.initrd.kernelModules = lib.mkForce []; +``` + +This requires the external kernel to have the boot-critical +drivers built-in: `CONFIG_VIRTIO_FS=y` (mount /nix/store and +/lib/modules in the initramfs), `CONFIG_VIRTIO_PCI=y` (PCI +transport), and `CONFIG_TMPFS=y` (root filesystem). All other +drivers can be kernel modules (`=m`), loaded from `/lib/modules` +after switch-root. The `/lib/modules` directory is mounted via +virtiofs from the external kernel build's module install path. + +## Root filesystem: tmpfs + +```nix +fileSystems."/" = lib.mkImageMediaOverride { + fsType = "tmpfs"; + options = [ "mode=0755" ]; +}; +``` + +Root is tmpfs. Everything written to `/` is lost on shutdown. This +is the standard NixOS approach for ephemeral systems. The operating +system state comes from `/nix/store` (read-only, shared from host) +and `/etc` (generated by NixOS activation from the store). + +`lib.mkImageMediaOverride` sets the NixOS option priority to 60, +overriding the default root filesystem declaration from NixOS +modules. Without this, NixOS expects a persistent root device. +See: `lib/modules.nix` in nixpkgs (`mkImageMediaOverride`). + +`systemd.services.systemd-remount-fs.enable = false` is set because +there is nothing to remount. The root is already writable tmpfs. + +## /nix/store: virtiofs read-only + +```nix +fileSystems."/nix/store" = { + device = "store"; + fsType = "virtiofs"; +}; +``` + +The Nix store is immutable by design. Packages are content-addressed +and never modified in place. Read-only virtiofs mounting enforces +this at the mount level. The `device` field is the virtiofs tag +name that must match the tag configured in the virtiofsd instance +sharing the host's `/nix/store` into the guest. + +## External kernel + +```nix +boot.kernel.enable = false; +``` + +NixOS does not build a kernel. The kernel is built separately using +Kconfig fragments and installed to a destdir. This allows rapid +kernel development iteration without rebuilding the NixOS closure. + +The kernel command line is passed to QEMU via the `-append` flag +(or equivalent configuration), not by NixOS. The imageless module +does not set `boot.kernelParams` because no NixOS bootloader is +active to enforce it. The QEMU configuration must include +`root=tmpfs` and `init=<closure>/init` explicitly. + +## Minimal profile + +```nix +imports = [ (modulesPath + "/profiles/minimal.nix") ]; +``` + +The minimal profile disables documentation, fonts, and other +non-essential modules. This reduces the system closure size from +~500MB to ~200MB. The closure contains only systemd, SSH, network +configuration, and coreutils. + +## Password authentication + +```nix +users.mutableUsers = false; +users.users.root.initialPassword = "root"; +services.openssh.settings.PasswordAuthentication = lib.mkDefault false; +``` + +SSH itself is key-only: the imageless module sets +`PasswordAuthentication = lib.mkDefault false`, and consumers +inject authorized keys per node. Root keeps a known +`initialPassword` for serial-console break-glass only. Because +root is tmpfs, `/etc/shadow` is generated fresh on every boot from +the NixOS configuration; `mutableUsers = false` ensures the +password is always reset to the configured value. The weak +password is not reachable over SSH and the console socket is only +accessible from the host. + +## systemd-networkd + +```nix +networking.useNetworkd = true; +systemd.network.networks."80-ethernet" = { + matchConfig.Name = "en*"; + networkConfig.DHCP = "yes"; +}; +``` + +systemd-networkd is the standard network manager for systemd-based +systems. NetworkManager is heavier and designed for desktop use. +The network configuration matches all ethernet interfaces (virtio +NIC appears as `enp0s2` in QEMU with q35 machine type) and enables +DHCP. + +# Libvirt + +Boots from a qcow2 disk image that libvirt presents to the guest as +`/dev/vda`. The kernel and initramfs are part of the NixOS closure +installed on the disk. Grub on the MBR loads them; systemd +activation proceeds normally. + +## Grub on /dev/vda + +```nix +boot.loader.grub = { + enable = true; + device = "/dev/vda"; +}; +boot.loader.timeout = lib.mkDefault 1; +``` + +Libvirt's default virtio-blk configuration exposes the qcow2 disk +to the guest as `/dev/vda`. Grub installs into its MBR. The short +one-second timeout skips the interactive menu for test boots. +Consumers with a different disk bus (SCSI, SATA) set +`boot.loader.grub.device` to the matching path; consumers who +prefer the interactive menu override `boot.loader.timeout`. + +## ext4 on /dev/vda1 + +```nix +fileSystems."/" = { + device = "/dev/vda1"; + fsType = "ext4"; +}; +``` + +Assumes the standard virt-builder qcow2 layout (MBR partition +table, first partition formatted ext4). Consumers with a different +partition scheme override `fileSystems."/".device` and `fsType`. + +## Kernel from nixpkgs + +```nix +boot.kernelPackages = lib.mkDefault pkgs.linuxPackages_latest; +``` + +The libvirt backend does not disable `boot.kernel.enable`; NixOS +builds a kernel as part of the closure. `linuxPackages_latest` +tracks the most recent nixpkgs kernel. Consumers who want to pin +to a specific branch or use a custom kernel override +`boot.kernelPackages`. + +## Scripted DHCP + +```nix +networking.useDHCP = lib.mkDefault true; +``` + +Libvirt's default network assigns DHCP leases from +`192.168.122.0/24`. `networking.useDHCP` enables the NixOS +scripted-networking path, which brings up every interface with +DHCP. Consumers who prefer systemd-networkd or need static +addresses override `useDHCP` and declare their own configuration. +(Imageless uses `useNetworkd = true` instead because its minimal +profile excludes scripted-networking; libvirt carries the full +system, so the default scripted path works.) + +## Nix with flakes and weekly garbage collection + +```nix +nix.settings.experimental-features = [ "nix-command" "flakes" ]; +nix.gc = { + automatic = lib.mkDefault true; + dates = lib.mkDefault "weekly"; + options = lib.mkDefault "--delete-older-than 7d"; +}; +``` + +Flakes are required because workflow-driven consumers may +re-evaluate nix expressions inside the guest. Weekly garbage +collection with a seven-day retention keeps the qcow2 from growing +unbounded across long workflow iterations. + +## No nested libvirt + +```nix +virtualisation.libvirtd.enable = lib.mkDefault false; +``` + +Nested virtualization is off by default. Workflows that need +virt-within-virt (KVM nested, for instance) opt in from a per-node +override. + +## SSH posture matches imageless + +The libvirt module inherits the same key-only sshd posture +described under Imageless. It does not carry its own +`initialPassword`: disk persistence means any break-glass setup +survives, and workflow consumers compose the user module (which +does set an initial root password) on top when they need it. A +standalone libvirt consumer with SSH keys injected does not get a +weak password baked in. diff --git a/scripts/nixos-qemu/docs/usage.md b/scripts/nixos-qemu/docs/usage.md new file mode 100644 index 00000000..b5d36f74 --- /dev/null +++ b/scripts/nixos-qemu/docs/usage.md @@ -0,0 +1,722 @@ +# Usage + +nixos-qemu exports NixOS modules, overlays, and a template. +Downstream projects consume these as a flake input to build their +own NixOS system closures. This document covers building, creating +configurations, and customizing packages. + +## Validating the library + +`nix flake check` evaluates and builds both backends so you can +verify the modules still produce a valid system closure: + +```shell +nix flake check +``` + +To build a single backend directly: + +```shell +nix build .#checks.x86_64-linux.imageless +nix build .#checks.x86_64-linux.libvirt +readlink --canonicalize result +``` + +The `result` symlink points to the system closure in `/nix/store`. +The NixOS init is at `<closure>/init`. The `init=` path contains a +Nix store hash that changes on every rebuild. + +For kernel development, create a downstream configuration from a +template; see the next section. + +List all files installed by a package (equivalent to +`dpkg --listfiles` on Debian): + +```shell +find $(dirname $(readlink --canonicalize $(which fio)))/.. -type f | sort +``` + +`readlink --canonicalize` resolves the profile symlink to the +actual store path. Replace `fio` with any binary name. + +## Creating a configuration + +A configuration is a standalone flake project: its own directory +with a `flake.nix` that references nixos-qemu as a flake input +and imports its modules and overlays. Each configuration has its own +`flake.lock` and `result` symlink. Multiple configurations can coexist +independently. + +The `configurations/` directory is gitignored for this purpose. +Create configurations there to keep them out of the tracked tree: + +```shell +mkdir --parents configurations/my-vm && cd configurations/my-vm +``` + +### Using the template + +The flake exports a +[template](https://nix.dev/manual/nix/stable/command-ref/new-cli/nix3-flake-init) +that scaffolds a configuration with the development profile, +overlays, and commented-out examples for building packages from +local source checkouts: + +```shell +nix flake init --template "path:$PWD/../.." +``` + +`nix flake init` copies `templates/imageless/flake.nix` into the +current directory. The copied file is independent of the template: +future changes to the upstream template do not update your copy. +The link between your configuration and nixos-qemu is the +`inputs.nixos-qemu.url` flake input, not the template itself. + +Edit `flake.nix` and set `nixos-qemu.url` to the absolute path of +your nixos-qemu checkout. The `path:` scheme does not expand `~`, +use the full path or `$HOME`: + +```shell +$EDITOR flake.nix +``` + +Nix flakes only evaluate files +[tracked by git](https://nix.dev/manual/nix/stable/command-ref/new-cli/nix3-flake#flake-references). +The `git add` is required before the first build. Subsequent edits +to `flake.nix` are picked up from the working tree without a new +commit: + +```shell +git init && git add flake.nix +nix build .#nixosConfigurations.vm.config.system.build.toplevel +readlink --canonicalize result +``` + +### Writing a configuration from scratch + +The template is optional. Any flake that declares nixos-qemu as an +input and imports its modules works as a configuration: + +```nix +{ + inputs = { + nixos-qemu.url = "path:/home/user/src/nixos-qemu"; + nixpkgs.follows = "nixos-qemu/nixpkgs"; + }; + + outputs = { self, nixpkgs, nixos-qemu, ... }: { + nixosConfigurations.vm = nixpkgs.lib.nixosSystem { + system = "x86_64-linux"; + modules = [ + nixos-qemu.nixosModules.imageless + nixos-qemu.nixosModules.devel + { + nixpkgs.overlays = [ nixos-qemu.overlays.default ]; + environment.systemPackages = with nixpkgs.legacyPackages.x86_64-linux; [ + gdb + ]; + } + ]; + }; + }; +} +``` + +`nixpkgs.follows = "nixos-qemu/nixpkgs"` ensures both flakes use +the same nixpkgs revision. Without it, the configuration would +pull a second copy of nixpkgs. + +### What the modules provide + +`nixosModules.imageless` (`modules/imageless.nix`) declares the +imageless base NixOS system: + +- Root as tmpfs (`fileSystems."/" = lib.mkImageMediaOverride { fsType = "tmpfs"; }`) +- Key-only SSH; root carries a known initial password (`root`) + for serial-console break-glass only +- systemd-networkd with DHCP on `en*` interfaces +- No bootloader, no kernel (external, `boot.kernel.enable = false`) +- Firewall disabled, unnecessary services disabled + +`nixosModules.libvirt` (`modules/libvirt.nix`) is the disk-image +counterpart: grub on `/dev/vda`, ext4 root, DHCP through +scripted networking, nix with flakes, and weekly garbage +collection. Each backend module is standalone (no cross-import); +both set `system.stateVersion` and the sshd posture inline. + +`nixosModules.user` (`modules/user.nix`) adds an opt-in +unprivileged account in `wheel`, `libvirt`, and `kvm` with +passwordless sudo. The account name is parametric through +`options.nixos-qemu.user.name` (default `user`); downstream +consumers override it to their project name. The module also +sets the serial-console break-glass root password. Workflow-driven +nodes import it; hand-development nodes do not. + +`nixosModules.shares` (`modules/shares.nix`) turns a set of +virtiofs shares into `fileSystems` entries. Each share is keyed by +its mount point and declares the virtiofs `tag` announced by the +host and optional mount `options`. The module does nothing else: +overlays, XDG env vars, and directory creation stay the consumer's +responsibility. See "Host configuration via virtiofs" for usage. + +`nixosModules.storage` (`modules/storage.nix`) turns a set of +block devices into `fileSystems` entries. Each entry is keyed by +mount point with fields `device`, `fsType`, `options`, `mkfsArgs`, +and `autoFormat`. Set `mkfsArgs` for a pre-mount `mkfs` oneshot +with custom arguments, or `autoFormat = true` for NixOS's default +first-mount formatting (the two are mutually exclusive). See +"Block device filesystems" for usage. + +`nixosModules.devel` (`modules/devel.nix`) adds kernel testing and +storage tools, grouped by purpose: filesystem and block layer tooling +(xfstests, xfsprogs, btrfs-progs, e2fsprogs, lvm2, parted), NVMe and +SCSI (nvme-cli, libnvme, sg3_utils, xnvme), I/O generation (fio with +liburing, stress-ng), NFS (nfs-utils, nfstest, pynfs), BPF and tracing +(bpftrace, bcc, libbpf-tools, blktrace, trace-cmd), monitoring and +profiling (perf, cpupower, damo, htop, iotop, numactl, pagemon, +powertop, sysstat), selftest and blktests/fstests runtime dependencies +(acl, attr, keyutils, libcap, libseccomp, mdadm, quota, rpcbind), and +general developer comfort (editors, shells, git, jq, strace, kmod). +The module also pulls in `programs.bash.completion.enable` and the +Python data-analysis stack (matplotlib, numpy, pandas). See +`modules/devel.nix` for the authoritative list. + +## Updating nixpkgs + +`flake.lock` pins the exact nixpkgs revision used by every build. +Refresh it to pick up upstream package and module changes: + +```shell +nix flake update +git add flake.lock +nix flake check +``` + +`flake.lock` must be tracked by git for the updated revision to +be picked up on the next build. + +Downstream configurations that use `nixpkgs.follows` inherit +the nixpkgs pin from nixos-qemu. After updating nixos-qemu, run +`nix flake update` in each configuration to pick up the new pin. + +## Overlays + +Nix +[overlays](https://nixos.org/manual/nixpkgs/stable/#chap-overlays) +modify or extend the nixpkgs package set. An overlay is a function +`final: prev: { ... }` where `prev` is the package set from all +previous overlays and `final` is the fully evaluated set including +the current overlay. Overlays are applied in order, so each layer +can see and build on the changes from previous layers. + +[`overrideAttrs`](https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs) +modifies a derivation by merging new attributes into the existing +ones. Attributes not specified in the override are retained from +the original. This means overrides compose: each `overrideAttrs` +call wraps the previous derivation, replacing only the attributes +it specifies. + +### What nixos-qemu provides + +The `overlays/` directory customizes nixpkgs packages. Each file +overrides one package. The default overlay (`overlays.default`) +composes all per-package overlays and merges custom packages from +`pkgs/`. + +Currently included: + +- **fio**: enables liburing, installs t/io_uring exerciser, + NVMe test scripts, and example job files +- **xfstests**: bumps nixpkgs 2023.05.14 (broken with modern + GCC) to 2026.03.20 + +### Overriding a package from a configuration + +A configuration lists overlays in order. nixos-qemu's overlay +goes first, then any user overlays that build on top of it. + +For example, xfstests goes through three layers: + +``` +nixpkgs base → xfstests 2023.05.14 (broken) +nixos-qemu.overlays.default → xfstests 2026.03.20 (bumped) +user overlay → xfstests from local checkout +``` + +The user overlay's `prev.xfstests` is the xfstests from +nixos-qemu's overlay (the bumped version). `overrideAttrs` +replaces only `src`, keeping everything else (build inputs, +install phase, patches) from the previous layer: + +```nix +nixpkgs.overlays = [ + nixos-qemu.overlays.default + (final: prev: { + xfstests = prev.xfstests.overrideAttrs { + src = inputs.xfstests-src; + }; + }) +]; +``` + +The `xfstests-src` input is a local source checkout declared as +a flake input with `flake = false` (see the template for +commented-out examples). + +### Pinning a specific upstream commit + +To pin a package to a specific commit without a local checkout, +use a fetcher in the overlay. The fetcher depends on where the +canonical upstream repository is hosted. + +For projects hosted on kernel.org, use `fetchgit`: + +```nix +# xfstests: git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git +(final: prev: { + xfstests = prev.xfstests.overrideAttrs { + src = final.fetchgit { + url = "git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git"; + rev = "<commit-hash>"; + hash = ""; + }; + }; +}) + +# fio: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/fio +(final: prev: { + fio = prev.fio.overrideAttrs { + src = final.fetchgit { + url = "https://git.kernel.org/pub/scm/linux/kernel/git/axboe/fio.git"; + rev = "<commit-hash>"; + hash = ""; + }; + patches = []; + }; +}) +``` + +For projects hosted on GitHub, use `fetchFromGitHub`: + +```nix +# damo: https://github.com/damonitor/damo +(final: prev: { + damo = prev.damo.overrideAttrs { + src = final.fetchFromGitHub { + owner = "damonitor"; + repo = "damo"; + rev = "<commit-hash>"; + hash = ""; + }; + }; +}) + +# xnvme: https://github.com/xnvme/xnvme +(final: prev: { + xnvme = prev.xnvme.overrideAttrs { + src = final.fetchFromGitHub { + owner = "xnvme"; + repo = "xnvme"; + rev = "<commit-hash>"; + hash = ""; + }; + }; +}) +``` + +Nix will fail with a hash mismatch on the first build and print +the correct `hash` value to use. Clear `patches = [];` when the +upstream source already includes fixes that nixos-qemu's overlay +backports. + +The user never needs to modify nixos-qemu's overlays. All +customization happens in the configuration's own `flake.nix`. + +## Building from local source + +To build a package from a local source checkout (for development +or testing), declare the source directory as a flake input with +`flake = false` and reference it in an overlay. This lets you +rebuild the NixOS closure with your modified source without +publishing it upstream. + +In the `inputs` block of your configuration's `flake.nix`: + +```nix +kmod-src = { + url = "path:/home/user/src/kmod"; + flake = false; +}; +``` + +`flake = false` tells Nix to import the path as plain source +rather than expecting a `flake.nix`. Then in the `nixpkgs.overlays` +list, add an overlay that replaces the package source: + +```nix +(final: prev: { + kmod = prev.kmod.overrideAttrs { + src = inputs.kmod-src; + }; +}) +``` + +`prev.kmod` is kmod from nixpkgs (or from a previous overlay). +`overrideAttrs` replaces only `src`, keeping the build system, +dependencies, and install phase from the original derivation. +Changes to the local source directory are picked up on every +`nix build` without updating the flake lock. + +This pattern works for any nixpkgs package. The template includes +commented-out examples for fio and kmod. Clear `patches = [];` +when the local source already includes fixes that the nixos-qemu +overlay backports. + +The following packages are supported by the devel module and can +be overridden this way: + +| Package | Input name | Overlay | +|---|---|---| +| cpupower | `kernel-src` | `prev.cpupower.overrideAttrs { src = inputs.kernel-src; }` | +| damo | `damo-src` | `prev.damo.overrideAttrs { src = inputs.damo-src; }` | +| fio | `fio-src` | `prev.fio.overrideAttrs { src = inputs.fio-src; patches = []; }` | +| kmod | `kmod-src` | `prev.kmod.overrideAttrs { src = inputs.kmod-src; }` | +| nfstest | `nfstest-src` | `prev.nfstest.overrideAttrs { src = inputs.nfstest-src; }` | +| pynfs | `pynfs-src` | `prev.pynfs.overrideAttrs { src = inputs.pynfs-src; }` | +| xfstests | `xfstests-src` | `prev.xfstests.overrideAttrs { src = inputs.xfstests-src; }` | +| xnvme | `xnvme-src` | `prev.xnvme.overrideAttrs { src = inputs.xnvme-src; }` | + +`cpupower` uses the kernel source tree (it builds from +`tools/power/cpupower/`). Point `kernel-src` to your kernel +checkout to match the running kernel version. + +## Custom packages + +The `pkgs/` directory holds packages not available in nixpkgs. +Each package is a file declaring a function, composed via +[callPackage](https://nix.dev/tutorials/callpackage). The overlay +imports `pkgs/default.nix` and merges them into the nixpkgs set. + +Currently included: + +- **damo**: DAMON user-space tool for data access monitoring +- **libbpf-tools**: standalone CO-RE BPF tracing tools (74 + binaries with `-libbpf` suffix, matching Debian convention) +- **nfstest**: NFS test suite (17 test scripts) +- **pynfs**: Python NFSv4 conformance test suite +- **xnvme**: cross-platform NVMe user space library and tools + +## Host configuration via virtiofs + +XDG-compliant tools (helix, neovim, git, etc.) read configuration +from `XDG_CONFIG_HOME` (per-user, defaults to `~/.config/`) and +fall back to `XDG_CONFIG_DIRS` (system-wide, colon-separated +list). On a tmpfs root `~/.config/` is always empty, so the +fallback is the only source of configuration. + +To share the host's `~/.config` into the guest, add a virtiofsd +instance with tag `xdg` sharing the host's `~/.config` +directory, then mount it in the configuration and set +`XDG_CONFIG_HOME` to point to it: + +```nix +fileSystems."/etc/xdg-host" = { + device = "xdg"; + fsType = "virtiofs"; +}; + +environment.variables.XDG_CONFIG_HOME = "/etc/xdg-host"; +environment.variables.XDG_CONFIG_DIRS = lib.mkForce "/etc/xdg-host:/etc/xdg"; +``` + +Or declare it through the shares module: + +```nix +imports = [ nixos-qemu.nixosModules.shares ]; + +nixos-qemu.shares."/etc/xdg-host" = { tag = "xdg"; }; + +environment.variables.XDG_CONFIG_HOME = "/etc/xdg-host"; +environment.variables.XDG_CONFIG_DIRS = lib.mkForce "/etc/xdg-host:/etc/xdg"; +``` + +The module replaces only the `fileSystems` entry. The XDG env vars +stay in the consumer's module because they are a policy choice tied +to one specific share's role, not a generic virtiofs concern. + +`XDG_CONFIG_HOME` is the primary config directory that all +XDG-compliant tools check first. Some tools (like helix) only +read from `XDG_CONFIG_HOME` and do not fall back to +`XDG_CONFIG_DIRS`, so setting both ensures all tools find the +host configs. On a tmpfs root `~/.config` is always empty, so +redirecting `XDG_CONFIG_HOME` loses nothing. + +The mount point is `/etc/xdg-host` rather than `/etc/xdg` because +NixOS generates its own files in `/etc/xdg`. The `XDG_CONFIG_DIRS` +override uses `lib.mkForce` because NixOS already sets this +variable in `shells-environment.nix`. Changes on the host are live +in the guest without rebuilding. + +This works for any user in the guest (root, test accounts) because +both variables are system-wide. The guest finds host configs at +`/etc/xdg-host/helix/`, `/etc/xdg-host/nvim/`, +`/etc/xdg-host/git/`, and so on. + +### Sharing a curated subset + +The host's `~/.config` may contain application state, browser +profiles, or credentials that should not be exposed to the guest. +To share only specific tool configs, create a dedicated directory +on the host and symlink the configs you want: + +```shell +mkdir --parents ~/.config/vm +ln --symbolic ~/.config/helix ~/.config/vm/helix +ln --symbolic ~/.config/nvim ~/.config/vm/nvim +ln --symbolic ~/.config/git ~/.config/vm/git +``` + +Share `~/.config/vm` with virtiofsd tag `xdg` instead of the +full `~/.config`. The NixOS configuration is the same — only the +virtiofsd source directory changes. + +### Dotfiles repo as a flake input + +A dotfiles repository can be declared as a flake input with +`flake = false` and its files referenced directly in the NixOS +configuration. This bakes the configs into the closure, so +changes require a rebuild: + +```nix +inputs.dotfiles = { + url = "path:/home/user/src/dotfiles"; + flake = false; +}; +``` + +Then in the module block: + +```nix +environment.etc."xdg-host/helix/config.toml".source = + "${inputs.dotfiles}/.config/helix/config.toml"; +environment.etc."xdg-host/nvim".source = + "${inputs.dotfiles}/.config/nvim"; +environment.variables.XDG_CONFIG_DIRS = lib.mkForce "/etc/xdg-host:/etc/xdg"; +``` + +This is declarative and version-controlled. The dotfiles input +is pinned in `flake.lock` like any other dependency. Run +`nix flake update` to pick up changes from the dotfiles repo. + +### Home-manager + +[Home-manager](https://github.com/nix-community/home-manager) is +a Nix tool for managing user configuration declaratively. Instead +of maintaining dotfiles as plain text, tool configs are expressed +as Nix options that home-manager evaluates into the correct files. + +Add it as a flake input in the configuration: + +```nix +inputs.home-manager = { + url = "github:nix-community/home-manager"; + inputs.nixpkgs.follows = "nixos-qemu/nixpkgs"; +}; +``` + +Then import the home-manager NixOS module and declare user +configs: + +```nix +imports = [ inputs.home-manager.nixosModules.home-manager ]; + +home-manager.users.root = { + programs.helix = { + enable = true; + settings = { + theme = "onedark"; + editor.line-number = "relative"; + }; + }; + programs.git = { + enable = true; + userName = "Your Name"; + userEmail = "[email protected]"; + }; +}; +``` + +Home-manager generates the dotfiles from these declarations and +places them in the user's home directory. This is the fully +declarative NixOS-native approach but adds a dependency and +requires expressing configs in Nix rather than using existing +dotfiles directly. + +## Host home directory with ephemeral overlay + +The host's home directory can be shared into the guest via +virtiofs for access to scripts, source trees, and tools. Mounting +it read-only protects the host, and an overlayfs layer on top +provides ephemeral writable storage for programs that write to +the home directory (shell history, `.ssh/known_hosts`, editor +state). Writes go to the tmpfs upper layer and are lost on +shutdown. + +```nix +# Host home via virtiofs (read-only base). +fileSystems."/mnt/home" = { + device = "home"; + fsType = "virtiofs"; + options = [ "ro" ]; +}; + +# Overlay: host home (read-only) + tmpfs (writable, ephemeral). +fileSystems."/root" = { + device = "overlay"; + fsType = "overlay"; + options = [ + "lowerdir=/mnt/home" + "upperdir=/.root-overlay/upper" + "workdir=/.root-overlay/work" + ]; + depends = [ "/mnt/home" ]; +}; + +# Create overlay work directories before the mount. +systemd.services."prepare-root-overlay" = { + description = "Create overlay work directories for /root"; + wantedBy = [ "local-fs.target" ]; + before = [ "root.mount" ]; + unitConfig.DefaultDependencies = false; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + ExecStart = "${pkgs.coreutils}/bin/mkdir --parents /.root-overlay/upper /.root-overlay/work"; + }; +}; +``` + +The read-only base mount can also be declared through the shares +module, leaving the overlay and prepare-root-overlay service in the +consumer's module: + +```nix +imports = [ nixos-qemu.nixosModules.shares ]; + +nixos-qemu.shares."/mnt/home" = { tag = "home"; options = [ "ro" ]; }; + +# Overlay and prepare-root-overlay service as above. +``` + +The virtiofsd instance on the host shares the home directory with +tag `home`. The upper and work directories live on the root tmpfs +(`/`), so they are created fresh on every boot by the +`prepare-root-overlay` service. + +This requires `CONFIG_OVERLAY_FS=y` or `CONFIG_OVERLAY_FS=m` in +the guest kernel. If built as a module, it loads from +`/lib/modules` after switch-root. + +## Block device filesystems + +NixOS can format and mount block devices (NVMe, virtio-blk) +declared in the configuration. + +### Default formatting + +`autoFormat = true` runs `mkfs` with default options if the +device has no filesystem: + +```nix +fileSystems."/mnt/nvme0" = { + device = "/dev/nvme0n1"; + fsType = "xfs"; + autoFormat = true; +}; +``` + +### Custom block and sector size + +For non-default formatting options (block size, sector size, +inode size), use a systemd service that runs before the mount +unit: + +```nix +systemd.services."format-nvme0" = { + description = "Format /dev/nvme0n1 with XFS (16k block/sector)"; + wantedBy = [ "local-fs.target" ]; + before = [ "mnt-nvme0.mount" ]; + requires = [ "dev-nvme0n1.device" ]; + after = [ "dev-nvme0n1.device" ]; + unitConfig.DefaultDependencies = false; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + ExecStart = "${pkgs.xfsprogs}/bin/mkfs.xfs -b size=16k -s size=16k /dev/nvme0n1"; + }; +}; + +fileSystems."/mnt/nvme0" = { + device = "/dev/nvme0n1"; + fsType = "xfs"; +}; +``` + +NixOS generates the mount unit name from the path (`/mnt/nvme0` +becomes `mnt-nvme0.mount`). The ordering chain is: device appears, +format service runs mkfs, mount unit mounts the filesystem. + +### Using the storage module + +The storage module consolidates multiple extra drives into one +attrset keyed by mount point. Devices with `mkfsArgs` go through a +pre-mount format oneshot guarded by `blkid --probe`; devices with +`autoFormat = true` fall through to NixOS's own first-mount +format machinery; devices with neither are assumed pre-formatted: + +```nix +imports = [ nixos-qemu.nixosModules.storage ]; + +nixos-qemu.storage = { + "/mnt/nvme0" = { + device = "/dev/nvme0n1"; + fsType = "xfs"; + mkfsArgs = [ "-b" "size=16k" "-s" "size=16k" ]; + }; + "/mnt/nvme1" = { + device = "/dev/nvme1n1"; + fsType = "xfs"; + autoFormat = true; + }; + "/mnt/data" = { + device = "/dev/vdb"; + fsType = "ext4"; + }; +}; +``` + +`mkfsArgs` and `autoFormat` are mutually exclusive; an assertion +rejects configurations that set both. + +## Multiple configurations + +Each configuration in `configurations/` is fully independent. +Configurations do not share state, lock files, or build results. +Common patterns: + +``` +configurations/ +├── devel/ Development VM with all testing tools +│ ├── flake.nix +│ └── flake.lock +├── storage/ VM for storage subsystem testing only +│ ├── flake.nix +│ └── flake.lock +└── minimal/ Base VM without the development profile + ├── flake.nix + └── flake.lock +``` + +Each configuration can import different modules, apply different +overlays, and add different packages. All of them share the same +nixos-qemu base through the `inputs.nixos-qemu.url` flake input. diff --git a/scripts/nixos-qemu/flake.lock b/scripts/nixos-qemu/flake.lock new file mode 100644 index 00000000..6d86e86b --- /dev/null +++ b/scripts/nixos-qemu/flake.lock @@ -0,0 +1,27 @@ +{ + "nodes": { + "nixpkgs": { + "locked": { + "lastModified": 1776067740, + "narHash": "sha256-B35lpsqnSZwn1Lmz06BpwF7atPgFmUgw1l8KAV3zpVQ=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "7e495b747b51f95ae15e74377c5ce1fe69c1765f", + "type": "github" + }, + "original": { + "owner": "NixOS", + "ref": "nixos-25.11", + "repo": "nixpkgs", + "type": "github" + } + }, + "root": { + "inputs": { + "nixpkgs": "nixpkgs" + } + } + }, + "root": "root", + "version": 7 +} diff --git a/scripts/nixos-qemu/flake.nix b/scripts/nixos-qemu/flake.nix new file mode 100644 index 00000000..b7e3e03c --- /dev/null +++ b/scripts/nixos-qemu/flake.nix @@ -0,0 +1,86 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Library flake: NixOS modules (imageless, libvirt, user, devel), +# overlays, templates, and custom packages for QEMU VMs. +# +# Validate both backends: +# nix flake check +# +# Build a single backend closure: +# nix build .#checks.x86_64-linux.imageless +# nix build .#checks.x86_64-linux.libvirt +# +# Build an individual custom package: +# nix build .#cpupower +# +# Create a downstream configuration: +# nix flake init --template "github:linux-kdevops/nixos-qemu" +# nix flake init --template "github:linux-kdevops/nixos-qemu#libvirt" +{ + description = "NixOS modules and overlays for QEMU VMs"; + + inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.11"; + + outputs = { self, nixpkgs }: + let + systems = [ "x86_64-linux" "aarch64-linux" ]; + forAllSystems = f: nixpkgs.lib.genAttrs systems (system: + f (import nixpkgs { + inherit system; + overlays = [ self.overlays.default ]; + })); + in { + nixosModules = { + build-tools = ./modules/build-tools.nix; + devel = ./modules/devel.nix; + imageless = ./modules/imageless.nix; + libvirt = ./modules/libvirt.nix; + shares = ./modules/shares.nix; + storage = ./modules/storage.nix; + user = ./modules/user.nix; + workflows.blktests = ./modules/workflows/blktests.nix; + workflows.fstests = ./modules/workflows/fstests.nix; + workflows.gitr = ./modules/workflows/gitr.nix; + workflows.ltp = ./modules/workflows/ltp.nix; + workflows.mmtests = ./modules/workflows/mmtests.nix; + workflows.pynfs = ./modules/workflows/pynfs.nix; + workflows.selftests = ./modules/workflows/selftests.nix; + workflows.sysbench = ./modules/workflows/sysbench.nix; + }; + + overlays.default = import ./overlays; + + templates = { + default = self.templates.imageless; + imageless = { + path = ./templates/imageless; + description = "Imageless NixOS VM (tmpfs root, virtiofs /nix/store, external kernel)"; + }; + libvirt = { + path = ./templates/libvirt; + description = "Libvirt-managed disk-image NixOS VM"; + }; + }; + + # Expose the custom packages as direct flake outputs so they can be + # built without going through a NixOS configuration. + packages = forAllSystems (pkgs: { + inherit (pkgs) cpupower damo libbpf-tools nfstest pynfs xnvme; + }); + + # Per-backend system closures exercised by nix flake check. + checks = nixpkgs.lib.genAttrs systems (system: + let + buildBackend = module: (nixpkgs.lib.nixosSystem { + inherit system; + modules = [ + module + { nixpkgs.overlays = [ self.overlays.default ]; } + ]; + }).config.system.build.toplevel; + in { + imageless = buildBackend self.nixosModules.imageless; + libvirt = buildBackend self.nixosModules.libvirt; + }); + }; +} diff --git a/scripts/nixos-qemu/modules/build-tools.nix b/scripts/nixos-qemu/modules/build-tools.nix new file mode 100644 index 00000000..25805c26 --- /dev/null +++ b/scripts/nixos-qemu/modules/build-tools.nix @@ -0,0 +1,23 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Common build toolchain: GNU Autotools plus a handful of headers +# and libraries that kernel-style compile jobs and several in-tree +# test frameworks routinely expect on a target host. +# +# Kept deliberately narrow. This module does not pick a C compiler +# (nixpkgs' stdenv already provides one) and does not pull in +# language-specific build systems; consumers that need more compose +# the additional packages themselves. +{ pkgs, ... }: { + environment.systemPackages = with pkgs; [ + autoconf + automake + libtool + pkg-config + flex + bison + bc + openssl + elfutils + ]; +} diff --git a/scripts/nixos-qemu/modules/devel.nix b/scripts/nixos-qemu/modules/devel.nix new file mode 100644 index 00000000..e5d96f9f --- /dev/null +++ b/scripts/nixos-qemu/modules/devel.nix @@ -0,0 +1,118 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Development profile: kernel testing and storage tools. +# +# Import this module on top of the base configuration to get a +# full kernel development environment with storage, NVMe, NFS, +# BPF, and filesystem testing tools. +# +# Usage in a flake: +# modules = [ +# nixos-qemu.nixosModules.imageless # or .libvirt +# nixos-qemu.nixosModules.devel +# { nixpkgs.overlays = [ nixos-qemu.overlays.default ]; } +# ]; +{ pkgs, ... }: { + # Wire up completion scripts for bash so tab completion works for + # the tools installed below. Just having bash-completion on the path + # does not enable it; the NixOS option sources the completion + # dispatcher from /etc/bashrc. + programs.bash.completion.enable = true; + + environment.systemPackages = with pkgs; [ + # Storage and filesystem tools + btrfs-progs + e2fsprogs + f2fs-tools + lvm2 + libndctl + parted + xfsdump + xfsprogs + + # NVMe and SCSI + libnvme + nvme-cli + sg3_utils + xnvme + + # I/O performance + fio + libaio + liburing + stress-ng + + # NFS + nfstest + nfs-utils + pynfs + + # Network + iperf + + # BPF and tracing + bcc + blktrace + bpftrace + libbpf-tools + trace-cmd + + # Monitoring + cpupower + btop + damo + dmidecode + gnuplot + htop + iotop + lsof + numactl + pagemon + perf + powertop + sysstat + + # Test suites and runtime dependencies + xfstests + + # fstests/blktests/selftests runtime dependencies + acl + attr + keyutils + libcap + libseccomp + mdadm + quota + rpcbind + + # General development + bc + ethtool + file + gawk + git + helix + iproute2 + jq + kmod + neovim + pciutils + screen + strace + tmux + usbutils + util-linux + vim + zellij + + # Python: interpreter plus data analysis libraries. The bare + # python3 interpreter is kept next to the libraries so downstream + # automation (Ansible, ad-hoc scripting) can find + # /run/current-system/sw/bin/python3 without having to pull in a + # workflow module that happens to install it transitively. + python3 + python3Packages.matplotlib + python3Packages.numpy + python3Packages.pandas + ]; +} diff --git a/scripts/nixos-qemu/modules/imageless.nix b/scripts/nixos-qemu/modules/imageless.nix new file mode 100644 index 00000000..57a3d023 --- /dev/null +++ b/scripts/nixos-qemu/modules/imageless.nix @@ -0,0 +1,114 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Minimal NixOS configuration for imageless boot via virtiofs. +# +# Root is tmpfs. /nix/store and /lib/modules are mounted via virtiofs +# from the host. The kernel is built externally with all boot-critical +# drivers built-in (CONFIG_VIRTIO_FS=y, CONFIG_TMPFS=y). NixOS builds +# a systemd initramfs that mounts root (tmpfs), /nix/store, and +# /lib/modules before switch-root. +# +# Key-only SSH. Root password is serial-console break-glass only. +{ pkgs, lib, modulesPath, ... }: { + imports = [ (modulesPath + "/profiles/minimal.nix") ]; + + system.stateVersion = "25.11"; + + services.openssh = { + enable = true; + settings = { + PermitRootLogin = lib.mkDefault "yes"; + PubkeyAuthentication = true; + PasswordAuthentication = lib.mkDefault false; + }; + }; + + # Root as tmpfs: ephemeral, no disk image. systemd in the initramfs + # creates this from root=tmpfs on the kernel command line. + fileSystems."/" = lib.mkImageMediaOverride { + fsType = "tmpfs"; + options = [ "mode=0755" ]; + }; + + # /nix/store via virtiofs (read-only, shared from host). + # Automatically included in initrd fstab (pathsNeededForBoot). + fileSystems."/nix/store" = { + device = "store"; + fsType = "virtiofs"; + }; + + # Kernel modules via virtiofs (from the external kernel build). + # neededForBoot puts it in the initrd fstab since /lib/modules + # is not in pathsNeededForBoot. + fileSystems."/lib/modules" = { + device = "modules"; + fsType = "virtiofs"; + neededForBoot = true; + }; + + # Disable systemd-remount-fs. Nothing to remount on tmpfs. + systemd.services.systemd-remount-fs.enable = lib.mkForce false; + + # No bootloader. Kernel is external, but NixOS builds the initramfs. + boot.loader.grub.enable = false; + boot.kernel.enable = false; + + # Workaround: boot.kernel.enable = false does not define + # system.build.kernel, but boot.initrd.systemd accesses + # kernel.config.isYes and kernel.config.isSet to decide whether + # to include kernel modules in the initramfs. + # + # The initrd module uses: isSet "MODULES" -> isYes "MODULES" + # (Nix implication). To exclude modules, isSet must return true + # and isYes must return false (true -> false = false). + # https://github.com/NixOS/nixpkgs/issues/467069 + system.build.kernel.config = { + isYes = _: false; + isSet = _: true; + }; + + # systemd initramfs: systemd runs as PID 1, reads the initrd fstab, + # mounts root + /nix/store + /lib/modules, then switch-roots. + boot.initrd.systemd.enable = true; + boot.initrd.supportedFilesystems = [ "virtiofs" ]; + + # Passwordless emergency shell in the initramfs. Without this, + # sulogin blocks on "root account is locked" when boot fails. + boot.initrd.systemd.emergencyAccess = true; + + # No kernel modules in the initramfs. The external kernel has all + # boot-critical drivers built-in (CONFIG_VIRTIO_FS=y, CONFIG_TMPFS=y, + # CONFIG_VIRTIO_PCI=y). Runtime modules come from /lib/modules via + # virtiofs after switch-root. + boot.initrd.availableKernelModules = lib.mkForce []; + boot.initrd.kernelModules = lib.mkForce []; + + # Serial getty on hvc0 for interactive login via the console socket. + # hvc0 is a virtio console, handled by systemd's serial-getty@ template + # (the plain getty@ template is VT-only, gated on /dev/tty0). + # Requires CONFIG_VIRTIO_CONSOLE=y (or =m) in the guest kernel. + systemd.services."serial-getty@hvc0" = { + enable = true; + wantedBy = [ "getty.target" ]; + }; + + # Serial-console break-glass. Tmpfs root resets it every boot. + users.mutableUsers = false; + users.users.root.initialPassword = "root"; + + # Networking: systemd-networkd with DHCP on all ethernet interfaces. + networking.useNetworkd = true; + networking.hostName = lib.mkDefault "nixos"; + networking.firewall.enable = false; + systemd.network.networks."80-ethernet" = { + matchConfig.Name = "en*"; + networkConfig.DHCP = "yes"; + }; + + # Disable unnecessary services. + systemd.oomd.enable = false; + nix.enable = false; + services.lvm.enable = false; + + environment.systemPackages = with pkgs; [ coreutils ]; +} diff --git a/scripts/nixos-qemu/modules/libvirt.nix b/scripts/nixos-qemu/modules/libvirt.nix new file mode 100644 index 00000000..d3631b86 --- /dev/null +++ b/scripts/nixos-qemu/modules/libvirt.nix @@ -0,0 +1,58 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Disk-image NixOS boot for libvirt-managed VMs. Expects a qcow2 +# virtio-blk disk at /dev/vda. +# +# Ported from kdevops commit 3089c3fec57 by Luis Chamberlain +# (playbooks/templates/nixos/configuration.nix.j2). Per-node content +# (hostname, keys, 9p, cache) stays in kdevops as additional modules. +{ pkgs, lib, modulesPath, ... }: { + + imports = [ + # qemu-guest wires up virtio drivers, the QEMU guest agent, and + # the small set of host-facing services that every NixOS VM + # under QEMU needs. Brings the hardware setup into the module + # so downstream consumers do not have to hand-write a matching + # hardware-configuration.nix. + (modulesPath + "/profiles/qemu-guest.nix") + ]; + + system.stateVersion = "25.11"; + + # Key-only SSH. Password below is serial-console break-glass. + services.openssh = { + enable = true; + settings = { + PermitRootLogin = lib.mkDefault "yes"; + PubkeyAuthentication = true; + PasswordAuthentication = lib.mkDefault false; + }; + }; + + boot.loader.grub = { + enable = true; + device = "/dev/vda"; + }; + boot.loader.timeout = lib.mkDefault 1; + boot.kernelPackages = lib.mkDefault pkgs.linuxPackages_latest; + + fileSystems."/" = { + device = "/dev/vda1"; + fsType = "ext4"; + }; + + # libvirt's default network hands out 192.168.122.0/24 via DHCP. + networking.useDHCP = lib.mkDefault true; + + virtualisation.libvirtd.enable = lib.mkDefault false; + + time.timeZone = lib.mkDefault "UTC"; + i18n.defaultLocale = lib.mkDefault "en_US.UTF-8"; + + nix.settings.experimental-features = [ "nix-command" "flakes" ]; + nix.gc = { + automatic = lib.mkDefault true; + dates = lib.mkDefault "weekly"; + options = lib.mkDefault "--delete-older-than 7d"; + }; +} diff --git a/scripts/nixos-qemu/modules/shares.nix b/scripts/nixos-qemu/modules/shares.nix new file mode 100644 index 00000000..25efc85a --- /dev/null +++ b/scripts/nixos-qemu/modules/shares.nix @@ -0,0 +1,41 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Opt-in virtiofs shares. Each share is a host-to-guest mount +# announced by the host's virtiofsd under a tag; the guest mounts +# it at a path of the consumer's choice. +# +# Shares are attrsOf submodule keyed by mount point, mirroring the +# fileSystems.<path> convention. The module emits one fileSystems +# entry per share and nothing else: no overlays, no env vars, no +# directory creation. Consumers compose those themselves when they +# need them. +{ config, lib, ... }: +let + cfg = config.nixos-qemu.shares; +in { + options.nixos-qemu.shares = lib.mkOption { + default = {}; + description = "Virtiofs shares to mount in the guest, keyed by mount point."; + type = lib.types.attrsOf (lib.types.submodule ({ name, ... }: { + options = { + tag = lib.mkOption { + type = lib.types.str; + description = "Virtiofs tag announced by the host's virtiofsd."; + }; + options = lib.mkOption { + type = lib.types.listOf lib.types.str; + default = []; + example = [ "ro" ]; + description = "Additional mount options passed to mount(8)."; + }; + }; + })); + }; + + config.fileSystems = lib.mapAttrs (mountPoint: share: { + device = share.tag; + fsType = "virtiofs"; + } // lib.optionalAttrs (share.options != []) { + inherit (share) options; + }) cfg; +} diff --git a/scripts/nixos-qemu/modules/storage.nix b/scripts/nixos-qemu/modules/storage.nix new file mode 100644 index 00000000..f402e38d --- /dev/null +++ b/scripts/nixos-qemu/modules/storage.nix @@ -0,0 +1,106 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Opt-in block-device mounts with optional pre-mount mkfs. Each +# entry is keyed by its mount point and declares the source device, +# filesystem type, mount options, and either a list of mkfs +# arguments or an autoFormat flag for default first-mount +# formatting. The two format modes are mutually exclusive; an +# assertion rejects configurations that set both. +# +# When mkfsArgs is non-empty, a oneshot service runs +# mkfs --type <fsType> with those arguments before the mount unit, +# guarded by blkid so a device that already carries a filesystem is +# left untouched. When autoFormat is true, the fileSystems entry +# passes through to NixOS's own first-mount format machinery. +{ config, lib, pkgs, utils, ... }: +let + cfg = config.nixos-qemu.storage; + + mkFormatUnit = mountPoint: drive: { + name = "format-${utils.escapeSystemdPath drive.device}"; + value = { + description = "Format ${drive.device} before ${mountPoint} mounts"; + wantedBy = [ "${utils.escapeSystemdPath mountPoint}.mount" ]; + before = [ "${utils.escapeSystemdPath mountPoint}.mount" ]; + requires = [ "${utils.escapeSystemdPath drive.device}.device" ]; + after = [ "${utils.escapeSystemdPath drive.device}.device" ]; + unitConfig.DefaultDependencies = false; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + ExecCondition = pkgs.writeShellScript "blkid-empty-${utils.escapeSystemdPath drive.device}" '' + ! ${pkgs.util-linux}/bin/blkid --probe ${lib.escapeShellArg drive.device} + ''; + ExecStart = lib.escapeShellArgs ( + [ "${pkgs.util-linux}/bin/mkfs" "--type" drive.fsType ] ++ drive.mkfsArgs ++ [ drive.device ] + ); + }; + }; + }; +in { + options.nixos-qemu.storage = lib.mkOption { + default = {}; + description = "Extra block-device mounts, keyed by mount point."; + type = lib.types.attrsOf (lib.types.submodule { + options = { + device = lib.mkOption { + type = lib.types.str; + example = "/dev/nvme0n1"; + description = "Source block device."; + }; + fsType = lib.mkOption { + type = lib.types.str; + example = "xfs"; + description = "Filesystem type passed to mount(8) and mkfs(8)."; + }; + options = lib.mkOption { + type = lib.types.listOf lib.types.str; + default = []; + description = "Additional mount options passed to mount(8)."; + }; + mkfsArgs = lib.mkOption { + type = lib.types.listOf lib.types.str; + default = []; + example = [ "-b" "size=16k" "-s" "size=16k" ]; + description = '' + Arguments passed to <literal>mkfs --type <fsType></literal> + before the mount unit runs. The format service skips + devices that already carry a filesystem. Mutually + exclusive with <option>autoFormat</option>. + ''; + }; + autoFormat = lib.mkOption { + type = lib.types.bool; + default = false; + description = '' + Pass through to + <option>fileSystems.<path>.autoFormat</option> so + NixOS formats the device with default mkfs arguments on + first mount if it has no filesystem. Mutually exclusive + with <option>mkfsArgs</option>. + ''; + }; + }; + }); + }; + + config = { + assertions = lib.mapAttrsToList (mountPoint: drive: { + assertion = !(drive.autoFormat && drive.mkfsArgs != []); + message = '' + nixos-qemu.storage."${mountPoint}": autoFormat and mkfsArgs + are mutually exclusive. Pick one. + ''; + }) cfg; + + fileSystems = lib.mapAttrs (mountPoint: drive: { + inherit (drive) device fsType autoFormat; + } // lib.optionalAttrs (drive.options != []) { + inherit (drive) options; + }) cfg; + + systemd.services = lib.listToAttrs ( + lib.mapAttrsToList mkFormatUnit (lib.filterAttrs (_: d: d.mkfsArgs != []) cfg) + ); + }; +} diff --git a/scripts/nixos-qemu/modules/user.nix b/scripts/nixos-qemu/modules/user.nix new file mode 100644 index 00000000..643bd745 --- /dev/null +++ b/scripts/nixos-qemu/modules/user.nix @@ -0,0 +1,42 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Opt-in workflow account with passwordless sudo. Name is +# configurable via options.nixos-qemu.user.name (default "user"); +# downstream consumers like kdevops override it. +# +# Also sets a break-glass root password ("root") for serial-console +# recovery when SSH is unreachable. Safe because the backend modules +# disable password authentication at the sshd layer. +{ config, lib, ... }: +let + cfg = config.nixos-qemu.user; +in { + options.nixos-qemu.user = { + name = lib.mkOption { + type = lib.types.str; + default = "user"; + description = "Unprivileged workflow account name."; + }; + + extraGroups = lib.mkOption { + type = lib.types.listOf lib.types.str; + default = [ "wheel" "libvirt" "kvm" ]; + description = "Groups the workflow account belongs to."; + }; + }; + + config = { + users.users.${cfg.name} = { + isNormalUser = true; + extraGroups = cfg.extraGroups; + openssh.authorizedKeys.keys = []; + }; + + # Unattended tests would deadlock on a sudo prompt. + security.sudo.wheelNeedsPassword = false; + + # Serial-console break-glass. See header for rationale. + users.mutableUsers = false; + users.users.root.initialPassword = "root"; + }; +} diff --git a/scripts/nixos-qemu/modules/workflows/blktests.nix b/scripts/nixos-qemu/modules/workflows/blktests.nix new file mode 100644 index 00000000..25c016a6 --- /dev/null +++ b/scripts/nixos-qemu/modules/workflows/blktests.nix @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# blktests workflow: block layer regression tests. +# +# Upstream: https://github.com/osandov/blktests +# +# Provides the block-device userland tools and SCSI target +# framework that blktests scripts invoke, plus the I/O generator +# and metrics collectors a few test groups rely on. +{ pkgs, ... }: { + environment.systemPackages = with pkgs; [ + # Block-device userland tools + nvme-cli + sg3_utils + multipath-tools + dmraid + lvm2 + mdadm + + # SCSI target framework used by iSCSI and FC test groups + # (nixpkgs ships the fork as targetcli-fb; the upstream name + # 'targetcli' has been retired from nixpkgs.) + targetcli-fb + + # I/O generator and stats used across several groups + fio + sysstat + ]; +} diff --git a/scripts/nixos-qemu/modules/workflows/fstests.nix b/scripts/nixos-qemu/modules/workflows/fstests.nix new file mode 100644 index 00000000..d1e2022f --- /dev/null +++ b/scripts/nixos-qemu/modules/workflows/fstests.nix @@ -0,0 +1,41 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# fstests (xfstests-dev) workflow: filesystem regression tests. +# +# Upstream: https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git +# +# Provides the userland filesystem tools, load generators, and NFS +# and SMB clients that the local and network fstests variants +# invoke on the target host. Services are enabled with mkDefault so +# consumers can turn them off for test subsets that do not exercise +# them. +{ pkgs, lib, ... }: { + environment.systemPackages = with pkgs; [ + # Filesystem userland tools for the filesystems fstests covers + xfsprogs + xfsdump + btrfs-progs + e2fsprogs + f2fs-tools + + # Load generators and stressors fstests scripts invoke. + # (dbench used to be in nixpkgs; upstream removed it in late + # 2025 as unmaintained for 14 years and broken, so fstests + # subsets that need dbench currently fall back to the consumer's + # own checkout.) + fio + stress-ng + + # POSIX attribute, ACL, and quota userland + attr + acl + quota + + # NFS and SMB clients for the network filesystem fstests variants + nfs-utils + cifs-utils + ]; + + services.nfs.server.enable = lib.mkDefault true; + services.rpcbind.enable = lib.mkDefault true; +} diff --git a/scripts/nixos-qemu/modules/workflows/gitr.nix b/scripts/nixos-qemu/modules/workflows/gitr.nix new file mode 100644 index 00000000..1320c5b3 --- /dev/null +++ b/scripts/nixos-qemu/modules/workflows/gitr.nix @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# gitr workflow: git regression tests, the in-tree test suite +# shipped with git.git under t/. +# +# Upstream: https://github.com/git/git +# +# Git's own test suite shells out to Subversion and Mercurial for +# its foreign-SCM suites and exercises perl integration heavily; +# the suite expects gitFull rather than the minimal git package +# so perl bindings and gitweb support are available. +{ pkgs, ... }: { + environment.systemPackages = with pkgs; [ + gitFull + + subversion + mercurial + + perl + ]; +} diff --git a/scripts/nixos-qemu/modules/workflows/ltp.nix b/scripts/nixos-qemu/modules/workflows/ltp.nix new file mode 100644 index 00000000..aa025f24 --- /dev/null +++ b/scripts/nixos-qemu/modules/workflows/ltp.nix @@ -0,0 +1,33 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# LTP workflow: Linux Test Project test suites. +# +# Upstream: https://github.com/linux-test-project/ltp +# +# LTP is cloned and built from source by the consumer's own +# tooling (kdevops clones the upstream tree). This module supplies +# only the build toolchain LTP's configure-and-make cycle expects +# and the runtime libraries the syscall, IPC, security, and +# filesystem suites link against. LTP itself is not in nixpkgs. +{ pkgs, ... }: { + environment.systemPackages = with pkgs; [ + # Build toolchain LTP's configure and make cycle invokes. + autoconf + automake + m4 + libtool + pkg-config + flex + bison + + # Runtime libraries LTP suites link against. libacl is provided + # by the acl package, libnuma by numactl, libssl by openssl. + acl + libcap + libaio + numactl + libsepol + libselinux + openssl + ]; +} diff --git a/scripts/nixos-qemu/modules/workflows/mmtests.nix b/scripts/nixos-qemu/modules/workflows/mmtests.nix new file mode 100644 index 00000000..aed0ffb4 --- /dev/null +++ b/scripts/nixos-qemu/modules/workflows/mmtests.nix @@ -0,0 +1,23 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# mmtests workflow: memory management test and benchmark harness. +# +# Upstream: https://github.com/gormanm/mmtests +# +# mmtests is perl-based and consumes its own result sets through +# gnuplot. CPU frequency control, DMI inventory, and the standard +# system metric collectors are invoked by several benchmark +# harnesses in the suite. +{ pkgs, ... }: { + environment.systemPackages = with pkgs; [ + perl + gnuplot + + cpupower + dmidecode + + sysstat + iotop + powertop + ]; +} diff --git a/scripts/nixos-qemu/modules/workflows/pynfs.nix b/scripts/nixos-qemu/modules/workflows/pynfs.nix new file mode 100644 index 00000000..241a03f4 --- /dev/null +++ b/scripts/nixos-qemu/modules/workflows/pynfs.nix @@ -0,0 +1,22 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# pynfs workflow: Python-based NFSv4 protocol conformance tests. +# +# Upstream: https://git.linux-nfs.org/?p=bfields/pynfs.git +# +# pynfs is cloned and driven by the consumer's own tooling +# (kdevops checks out the upstream tree). This module supplies +# only the Python runtime and ply parser-generator the harness +# imports, plus the NFS userland the scripts shell out to for +# server-side fixtures. A pre-built pynfs binary is available as +# a custom package in this flake (pkgs/pynfs.nix) but is not +# installed by default to avoid duplicating the consumer's own +# checkout. +{ pkgs, ... }: { + environment.systemPackages = with pkgs; [ + python3 + python3Packages.ply + + nfs-utils + ]; +} diff --git a/scripts/nixos-qemu/modules/workflows/selftests.nix b/scripts/nixos-qemu/modules/workflows/selftests.nix new file mode 100644 index 00000000..a0321f03 --- /dev/null +++ b/scripts/nixos-qemu/modules/workflows/selftests.nix @@ -0,0 +1,25 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# selftests workflow: Linux kernel in-tree self-tests. +# +# Upstream: tools/testing/selftests in the Linux kernel tree. +# +# Installs the userland tools selftests invoke: perf and related +# tracing, CPU and memory placement, capability and seccomp +# userland, traffic-control helpers, and ethernet link controls. +# tc(8) is provided by iproute2 on NixOS, so no separate package +# is required for the kselftest net suites. +{ pkgs, ... }: { + environment.systemPackages = with pkgs; [ + perf-tools + + numactl + + libcap + libseccomp + keyutils + + iproute2 + ethtool + ]; +} diff --git a/scripts/nixos-qemu/modules/workflows/sysbench.nix b/scripts/nixos-qemu/modules/workflows/sysbench.nix new file mode 100644 index 00000000..57576154 --- /dev/null +++ b/scripts/nixos-qemu/modules/workflows/sysbench.nix @@ -0,0 +1,26 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# sysbench workflow: scriptable benchmarking tool, commonly used +# for MySQL and PostgreSQL workload measurement. +# +# Upstream: https://github.com/akopytov/sysbench +# +# Enables MariaDB (NixOS' mysql service uses it) and PostgreSQL +# with lib.mkDefault so consumers can disable one or both when +# a run only targets a specific backend. The MySQL service +# package is also set via mkDefault so downstreams that prefer +# the upstream Oracle MySQL can substitute without mkForce. +{ pkgs, lib, ... }: { + environment.systemPackages = with pkgs; [ + sysbench + mariadb + postgresql + ]; + + services.mysql = { + enable = lib.mkDefault true; + package = lib.mkDefault pkgs.mariadb; + }; + + services.postgresql.enable = lib.mkDefault true; +} diff --git a/scripts/nixos-qemu/overlays/default.nix b/scripts/nixos-qemu/overlays/default.nix new file mode 100644 index 00000000..63b6dc7c --- /dev/null +++ b/scripts/nixos-qemu/overlays/default.nix @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Package overlays and custom packages. Each file in this directory +# modifies one nixpkgs package. Custom packages not in nixpkgs are +# defined in pkgs/ using callPackage and merged here. +# +# Usage in a NixOS module: +# { nixpkgs.overlays = [ (import ./overlays) ]; } +# +# Reference: https://nixos.org/manual/nixpkgs/stable/#chap-overlays +final: prev: + (import ./fio.nix final prev) + // (import ./xfstests.nix final prev) + // (import ../pkgs final) diff --git a/scripts/nixos-qemu/overlays/fio.nix b/scripts/nixos-qemu/overlays/fio.nix new file mode 100644 index 00000000..967e266e --- /dev/null +++ b/scripts/nixos-qemu/overlays/fio.nix @@ -0,0 +1,36 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Override fio to enable io_uring support via liburing and install +# the storage stack test suite, io_uring exerciser, and example +# job files that nixpkgs omits. +# +# nixpkgs fio (pkgs/by-name/fi/fio/package.nix) builds with libaio +# and libnbd but does not enable liburing or install t/io_uring, +# the Python NVMe test scripts, or the example .fio files. +final: prev: { + fio = prev.fio.overrideAttrs (old: { + buildInputs = (old.buildInputs or []) ++ [ final.liburing ]; + + postInstall = (old.postInstall or "") + '' + # t/io_uring: standalone io_uring and NVMe passthrough exerciser. + # Not a fio self-test. Uses raw io_uring syscalls to benchmark + # the kernel block layer. Fail the build if it is missing, since + # liburing is now a declared buildInput. + test -x t/io_uring + install --mode=755 t/io_uring $out/bin/fio-io-uring + + # Storage stack test suite. Python scripts that validate kernel + # NVMe features (FDP, DIF/DIX, streams, multi-range TRIM, ZBD) + # using fio as the I/O generator. + install --mode=755 --directory $out/share/fio/t + install --mode=755 --directory $out/share/fio/t/zbd + install --mode=644 t/*.py $out/share/fio/t/ + install --mode=755 t/one-core-peak.sh $out/share/fio/t/ + install --mode=755 t/zbd/* $out/share/fio/t/zbd/ + + # Example job files. + install --mode=755 --directory $out/share/doc/fio/examples + install --mode=644 examples/*.fio $out/share/doc/fio/examples/ + ''; + }); +} diff --git a/scripts/nixos-qemu/overlays/xfstests.nix b/scripts/nixos-qemu/overlays/xfstests.nix new file mode 100644 index 00000000..7c878480 --- /dev/null +++ b/scripts/nixos-qemu/overlays/xfstests.nix @@ -0,0 +1,23 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Override xfstests to a current version. The nixpkgs version +# (2023.05.14) fails to build with modern GCC due to +# -Werror=implicit-function-declaration in ltp/fsx.c. +# +# The nixpkgs recipe handles all the packaging quirks (path +# patching, wrapper script, libtool workaround). We only need +# to bump the version and hash. +final: prev: +let + version = "2026.03.20"; +in { + xfstests = prev.xfstests.overrideAttrs (old: { + inherit version; + src = prev.fetchzip { + url = "https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git/snapshot/xfstests-dev-v${version}.tar.gz"; + hash = "sha256-f82TsOEikGo9uQyM0hQmgy6A6R/2U4Ot/eINq34kohM="; + }; + # Newer xfstests needs pkg-config for autoconf macros. + nativeBuildInputs = (old.nativeBuildInputs or []) ++ [ prev.pkg-config ]; + }); +} diff --git a/scripts/nixos-qemu/pkgs/cpupower.nix b/scripts/nixos-qemu/pkgs/cpupower.nix new file mode 100644 index 00000000..c9c59792 --- /dev/null +++ b/scripts/nixos-qemu/pkgs/cpupower.nix @@ -0,0 +1,64 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Standalone cpupower built from a kernel source tree. +# +# nixpkgs cpupower is tied to linuxPackages and requires a +# NixOS-built kernel. This package builds from any kernel source, +# defaulting to linux_latest when no override is provided. +{ + lib, + stdenv, + buildPackages, + linux_latest, + pciutils, + gettext, + which, +}: + +stdenv.mkDerivation { + pname = "cpupower"; + inherit (linux_latest) version src; + + nativeBuildInputs = [ + gettext + which + ]; + buildInputs = [ pciutils ]; + + postPatch = '' + cd tools/power/cpupower + substituteInPlace Makefile \ + --replace-fail /bin/true ${buildPackages.coreutils}/bin/true \ + --replace-fail /usr/bin/install ${buildPackages.coreutils}/bin/install + ''; + + makeFlags = [ + "CROSS=${stdenv.cc.targetPrefix}" + "CC=${stdenv.cc.targetPrefix}cc" + "LD=${stdenv.cc.targetPrefix}cc" + ]; + + installFlags = lib.mapAttrsToList (n: v: "${n}dir=${placeholder "out"}/${v}") { + bin = "bin"; + sbin = "sbin"; + man = "share/man"; + include = "include"; + lib = "lib"; + libexec = "libexec"; + locale = "share/locale"; + doc = "share/doc/cpupower"; + conf = "etc"; + bash_completion_ = "share/bash-completion/completions"; + unit = "lib/systemd/system"; + }; + + enableParallelBuilding = true; + + meta = { + description = "Tool to examine and tune power saving features"; + homepage = "https://www.kernel.org/"; + license = lib.licenses.gpl2Only; + mainProgram = "cpupower"; + platforms = lib.platforms.linux; + }; +} diff --git a/scripts/nixos-qemu/pkgs/damo.nix b/scripts/nixos-qemu/pkgs/damo.nix new file mode 100644 index 00000000..24af25ba --- /dev/null +++ b/scripts/nixos-qemu/pkgs/damo.nix @@ -0,0 +1,58 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# DAMO: DAMON (Data Access MONitor) user-space tool. +# +# damo's upstream source layout is flat (src/*.py, no __init__.py). +# The packaging/ directory provides setup.py with console_scripts +# and package_dir={"": "src"}, but expects a src/damo/ package +# directory. The postPatch phase performs the same reorganization +# that Debian and packaging/build.sh do: +# +# 1. Copy packaging/pyproject.toml and packaging/setup.py to root +# 2. Create src/damo/ and move all .py files into it +# 3. Create src/damo/__init__.py +# +# Source: https://github.com/damonitor/damo +# Debian: https://packages.debian.org/trixie/damo +# Fedora: https://src.fedoraproject.org/rpms/python-damo +{ lib, python3Packages, fetchFromGitHub }: + +python3Packages.buildPythonApplication rec { + pname = "damo"; + version = "3.2.0"; + pyproject = true; + + src = fetchFromGitHub { + owner = "damonitor"; + repo = "damo"; + rev = "v${version}"; + hash = "sha256-DflgnU7/tepLwnDOOVahqsPXMSu0GJ/OWYnVd5qSCQA="; + }; + + # Reorganize flat src/*.py into a proper Python package at + # src/damo/ so setuptools.find_packages(where="src") finds it. + # This matches Debian's debian/rules execute_before_dh_auto_configure + # and upstream's packaging/build.sh. + postPatch = '' + cp packaging/pyproject.toml pyproject.toml + cp packaging/setup.py setup.py + mkdir --parents src/damo + cp src/*.py src/damo/ + touch src/damo/__init__.py + ''; + + build-system = [ python3Packages.setuptools ]; + + # Pure Python, no runtime dependencies beyond the standard library. + + # Tests require a running kernel with CONFIG_DAMON enabled. + doCheck = false; + + meta = { + description = "DAMON user-space tool for data access monitoring"; + homepage = "https://github.com/damonitor/damo"; + license = lib.licenses.gpl2Only; + platforms = lib.platforms.linux; + mainProgram = "damo"; + }; +} diff --git a/scripts/nixos-qemu/pkgs/default.nix b/scripts/nixos-qemu/pkgs/default.nix new file mode 100644 index 00000000..38c3d91a --- /dev/null +++ b/scripts/nixos-qemu/pkgs/default.nix @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Custom packages not available in nixpkgs. +# +# Each package is a file declaring a function whose arguments are its +# dependencies. Use callPackage to compose them: +# +# xnvme = pkgs.callPackage ./xnvme.nix { }; +# +# The overlay at overlays/default.nix imports this file and merges +# the packages into the nixpkgs set. +# +# Reference: https://nix.dev/tutorials/callpackage +pkgs: { + cpupower = pkgs.callPackage ./cpupower.nix { }; + damo = pkgs.callPackage ./damo.nix { }; + libbpf-tools = pkgs.callPackage ./libbpf-tools.nix { }; + nfstest = pkgs.callPackage ./nfstest.nix { }; + pynfs = pkgs.callPackage ./pynfs.nix { }; + xnvme = pkgs.callPackage ./xnvme.nix { }; +} diff --git a/scripts/nixos-qemu/pkgs/libbpf-tools.nix b/scripts/nixos-qemu/pkgs/libbpf-tools.nix new file mode 100644 index 00000000..ab8fa73d --- /dev/null +++ b/scripts/nixos-qemu/pkgs/libbpf-tools.nix @@ -0,0 +1,117 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# libbpf-tools: standalone CO-RE BPF tracing tools from the BCC project. +# +# These are lightweight alternatives to the Python-based BCC tools. +# Each tool is a statically-linked binary (against vendored libbpf) +# that embeds its BPF program via a skeleton header. No BCC runtime, +# no Python, no LLVM needed at runtime. +# +# The build requires the full BCC source tree with submodules because +# libbpf-tools/ references ../src/cc/libbpf/src (vendored libbpf) and +# ./bpftool/src (vendored bpftool) via relative paths. +# +# Source: https://github.com/iovisor/bcc/tree/master/libbpf-tools +# Debian: https://packages.debian.org/trixie/libbpf-tools +{ + lib, + stdenv, + fetchFromGitHub, + clang, + llvmPackages, + elfutils, + zlib, + openssl, + pkg-config, + gnumake, +}: + +stdenv.mkDerivation rec { + pname = "libbpf-tools"; + version = "0.36.1"; + + src = fetchFromGitHub { + owner = "iovisor"; + repo = "bcc"; + tag = "v${version}"; + hash = "sha256-yfIyV+NKaSYMNjqFpxPwCgetPacC8OrBBGxRG3P2z0o="; + fetchSubmodules = true; + }; + + sourceRoot = "${src.name}/libbpf-tools"; + + nativeBuildInputs = [ + # clang compiles the BPF programs with -target bpf. The BPF target + # is freestanding, so any clang wrapper works; the plain wrapper + # keeps the build closure small. The host-side loader uses + # stdenv.cc via $(CC), not this clang. + llvmPackages.clang + llvmPackages.llvm # llvm-strip + pkg-config + gnumake + ]; + + buildInputs = [ + elfutils # libelf (dynamic) + zlib # libz (dynamic) + openssl # for bpftool build + ]; + + # Nix's hardening flags (-fzero-call-used-regs, -fstack-protector) + # are not valid for the BPF target. Disable them since the BPF + # programs are compiled with clang --target=bpf. + hardeningDisable = [ "zerocallusedregs" "stackprotector" ]; + + makeFlags = [ + "prefix=${placeholder "out"}" + "USE_BLAZESYM=0" + ]; + + enableParallelBuilding = true; + + installPhase = '' + runHook preInstall + + make install prefix=$out + + # Add -libbpf suffix to avoid collisions with BCC's Python + # wrappers that share the same names (biolatency, execsnoop, + # etc.). Follows the Debian libbpf-tools package convention + # (e.g., /usr/sbin/biolatency-libbpf). + for bin in $out/bin/*; do + name=$(basename "$bin") + # Skip if already suffixed (symlink aliases like btrfsdist). + if [[ "$name" != *-libbpf ]]; then + mv "$bin" "''${bin}-libbpf" + fi + done + + # Recreate filesystem aliases with the suffix. + for link in $out/bin/*-libbpf; do + if [ -L "$link" ]; then + target=$(readlink "$link") + ln --symbolic --force "$(basename "$target")-libbpf" "$link" + fi + done + + # Install man pages from the BCC man/man8/ directory for tools + # that have matching names. + mkdir --parents $out/share/man/man8 + for bin in $out/bin/*-libbpf; do + tool=$(basename "$bin" -libbpf) + manpage="../man/man8/''${tool}.8" + if [ -f "$manpage" ]; then + install --mode=644 "$manpage" $out/share/man/man8/ + fi + done + + runHook postInstall + ''; + + meta = { + description = "Standalone libbpf-based BPF tracing tools from BCC"; + homepage = "https://github.com/iovisor/bcc/tree/master/libbpf-tools"; + license = with lib.licenses; [ lgpl21Only bsd2 ]; + platforms = lib.platforms.linux; + }; +} diff --git a/scripts/nixos-qemu/pkgs/nfstest.nix b/scripts/nixos-qemu/pkgs/nfstest.nix new file mode 100644 index 00000000..64ace544 --- /dev/null +++ b/scripts/nixos-qemu/pkgs/nfstest.nix @@ -0,0 +1,67 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# NFStest: NFS test suite covering space reservation, direct I/O, +# interoperability, file locking, POSIX compliance, sparse files, +# and server-side copy. +# +# The upstream setup.py tries to generate manpages during build by +# running installed scripts, which fails before install. Patch it +# out so the build completes without running uninstalled scripts. +# +# Source: git://linux-nfs.org/projects/mora/nfstest +{ lib, python3Packages, fetchurl, nfs-utils, tcpdump }: + +python3Packages.buildPythonApplication rec { + pname = "nfstest"; + version = "3.2"; + pyproject = false; + + src = fetchurl { + url = "http://www.linux-nfs.org/~mora/nfstest/releases/NFStest-${version}.tar.gz"; + hash = "sha256-TUxcWttygx9Xhyde8jxUFP9cV84SDK+xzVPeB2ZXRcE="; + }; + + postPatch = '' + # Disable manpage generation during build. The setup.py build + # command runs installed scripts with --version/--help which + # fails before install. + substituteInPlace setup.py \ + --replace-fail 'create_manpage.run()' 'pass' + + # Replace distutils with setuptools (distutils removed in 3.12). + substituteInPlace setup.py \ + --replace-fail 'from distutils.core import setup' \ + 'from setuptools import setup' \ + --replace-fail 'from distutils.command.build import build' \ + 'from setuptools.command.build import build' + ''; + + nativeBuildInputs = [ python3Packages.pip ]; + build-system = [ python3Packages.setuptools ]; + + buildPhase = '' + runHook preBuild + runHook postBuild + ''; + + installPhase = '' + runHook preInstall + pip install --prefix=$out --no-deps --no-build-isolation . + patchShebangs $out/bin + runHook postInstall + ''; + + # Pure Python, no compiled extensions. + doCheck = false; + + # Runtime: nfs-utils for mount.nfs, tcpdump for packet capture. + propagatedBuildInputs = [ nfs-utils tcpdump ]; + + meta = { + description = "NFS test suite for Linux"; + homepage = "http://www.linux-nfs.org/~mora/nfstest/"; + license = lib.licenses.gpl2Only; + platforms = lib.platforms.linux; + mainProgram = "nfstest_posix"; + }; +} diff --git a/scripts/nixos-qemu/pkgs/pynfs.nix b/scripts/nixos-qemu/pkgs/pynfs.nix new file mode 100644 index 00000000..4e816be9 --- /dev/null +++ b/scripts/nixos-qemu/pkgs/pynfs.nix @@ -0,0 +1,94 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# pynfs: Python NFSv4 conformance test suite covering NFSv4.0, +# NFSv4.1, and pNFS block layouts with Kerberos bindings. +# +# The top-level setup.py delegates to 4 sub-packages (xdr, rpc, +# nfs4.1, nfs4.0). pip cannot handle this; we use setup.py directly. +# The build includes SWIG-generated Kerberos bindings (compiled C). +# +# No formal releases exist. We pin to a commit. +# +# Source: https://github.com/ffilz/pynfs +{ + lib, + python3Packages, + fetchFromGitHub, + swig, + krb5, +}: + +python3Packages.buildPythonApplication rec { + pname = "pynfs"; + version = "unstable-2025-03-05"; + pyproject = false; + + src = fetchFromGitHub { + owner = "ffilz"; + repo = "pynfs"; + rev = "d3a1610815117cb6bdf6567e575baedb0d88095e"; + hash = "sha256-grigxIDAQG4hnwA/YmsHNP3aKX61uw76I/ClgT0+u8Q="; + }; + + nativeBuildInputs = [ + swig + krb5.dev + python3Packages.setuptools + ]; + + buildInputs = [ krb5 ]; + + dependencies = with python3Packages; [ + gssapi + ply + ]; + + # Upstream uses distutils which was removed in Python 3.12. + # Patch to use setuptools instead. + postPatch = '' + substituteInPlace setup.py \ + --replace-fail 'from distutils.core import setup' \ + 'from setuptools import setup' + for f in xdr/setup.py rpc/setup.py nfs4.1/setup.py nfs4.0/setup.py; do + if [ -f "$f" ]; then + substituteInPlace "$f" \ + --replace-fail 'from distutils.core import setup' \ + 'from setuptools import setup' + fi + done + ''; + + # The delegating setup.py handles all 4 sub-packages. + buildPhase = '' + runHook preBuild + python setup.py build + runHook postBuild + ''; + + installPhase = '' + runHook preInstall + python setup.py install --prefix=$out --optimize=1 + + # The delegating setup.py installs libraries but not scripts. + # Install the test runners from the sub-packages manually. + mkdir --parents $out/bin + install --mode=755 nfs4.0/testserver.py $out/bin/nfs4-testserver + install --mode=755 nfs4.0/showresults.py $out/bin/nfs4-showresults + if [ -f nfs4.1/testserver.py ]; then + install --mode=755 nfs4.1/testserver.py $out/bin/nfs41-testserver + fi + + # Patch shebangs. + patchShebangs $out/bin + runHook postInstall + ''; + + doCheck = false; + + meta = { + description = "Python NFSv4 conformance test suite"; + homepage = "https://github.com/ffilz/pynfs"; + license = lib.licenses.gpl2Only; + platforms = lib.platforms.linux; + }; +} diff --git a/scripts/nixos-qemu/pkgs/xnvme.nix b/scripts/nixos-qemu/pkgs/xnvme.nix new file mode 100644 index 00000000..560c3ea0 --- /dev/null +++ b/scripts/nixos-qemu/pkgs/xnvme.nix @@ -0,0 +1,49 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# xNVMe: cross-platform NVMe user space library and tools. +# +# Built with libaio and liburing backends. SPDK, libvfn, and ISA-L +# are disabled (not needed for kernel development workflows). +# +# Source: https://github.com/xnvme/xnvme +{ + lib, + stdenv, + fetchurl, + meson, + ninja, + pkg-config, + libaio, + liburing, +}: + +stdenv.mkDerivation rec { + pname = "xnvme"; + version = "0.7.5"; + + src = fetchurl { + url = "https://github.com/xnvme/xnvme/releases/download/v${version}/xnvme-${version}.tar.gz"; + hash = "sha256-2VGjEV9oaVb1xrg5s5D/tQu0Q/dxH+NfV6B/TQZnJ38="; + }; + + nativeBuildInputs = [ meson ninja pkg-config ]; + + buildInputs = [ libaio liburing ]; + + mesonFlags = [ + "-Dwith-spdk=disabled" + "-Dwith-libvfn=disabled" + "-Dwith-isal=disabled" + "-Dexamples=false" + "-Dtests=false" + "-Dbuild_subprojects=false" + ]; + + meta = { + description = "Cross-platform NVMe user space library and tools"; + homepage = "https://xnvme.io"; + license = lib.licenses.bsd3; + mainProgram = "xnvme"; + platforms = lib.platforms.linux; + }; +} diff --git a/scripts/nixos-qemu/templates/imageless/default.nix b/scripts/nixos-qemu/templates/imageless/default.nix new file mode 100644 index 00000000..499a2abf --- /dev/null +++ b/scripts/nixos-qemu/templates/imageless/default.nix @@ -0,0 +1,33 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Per-VM overrides for the imageless template. +# +# The flake passes nixos-qemu and the flake inputs via specialArgs +# so this module can pull in workflow modules (LTP, fstests, and so +# on) and reference local source checkouts without having to edit +# flake.nix again. +{ config, lib, pkgs, nixos-qemu, inputs, ... }: { + imports = [ + # Development tools (editors, tracing, kernel dev comfort). Drop + # this import for a minimal VM that only runs a specific workflow. + nixos-qemu.nixosModules.devel + + # Pick the workflow modules that match what you intend to run, + # for example: + # nixos-qemu.nixosModules.build-tools + # nixos-qemu.nixosModules.workflows.fstests + # nixos-qemu.nixosModules.workflows.blktests + ]; + + networking.hostName = "vm"; + + # Build a package from a local source checkout. The matching + # fio-src input must be declared in flake.nix. + # nixpkgs.overlays = [ + # (final: prev: { fio = prev.fio.overrideAttrs { src = inputs.fio-src; patches = []; }; }) + # ]; + + # SSH keys for the root and workflow user accounts. + # users.users.root.openssh.authorizedKeys.keys = [ "ssh-ed25519 ..." ]; + # users.users.${config.nixos-qemu.user.name}.openssh.authorizedKeys.keys = [ "ssh-ed25519 ..." ]; +} diff --git a/scripts/nixos-qemu/templates/imageless/flake.nix b/scripts/nixos-qemu/templates/imageless/flake.nix new file mode 100644 index 00000000..6ffcf2aa --- /dev/null +++ b/scripts/nixos-qemu/templates/imageless/flake.nix @@ -0,0 +1,51 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Imageless NixOS VM starter. +# +# Create with: +# nix flake init --template "github:linux-kdevops/nixos-qemu" +# nix build .#nixosConfigurations.vm.config.system.build.toplevel +# +# Per-VM configuration lives in ./default.nix. This flake composes +# the base modules, applies the overlay, and passes nixos-qemu and +# the flake inputs through specialArgs so default.nix can import +# additional modules (for example nixos-qemu.nixosModules.workflows.*) +# and reference local source inputs without re-declaring them. +{ + inputs = { + # Local checkout preferred: downstream consumers (kdevops and + # others) pin to a specific revision via a subtree or vendored + # copy, and should not track upstream HEAD. For upstream, use: + # nixos-qemu.url = "github:linux-kdevops/nixos-qemu"; + nixos-qemu.url = "path:/path/to/nixos-qemu"; + nixpkgs.follows = "nixos-qemu/nixpkgs"; + + # Local source checkouts (uncomment to use): + # fio-src = { url = "path:/home/user/src/fio"; flake = false; }; + # kmod-src = { url = "path:/home/user/src/kmod"; flake = false; }; + }; + + outputs = { self, nixpkgs, nixos-qemu, ... }@inputs: + let + system = "x86_64-linux"; + in { + nixosConfigurations.vm = nixpkgs.lib.nixosSystem { + inherit system; + specialArgs = { inherit inputs nixos-qemu; }; + modules = [ + nixos-qemu.nixosModules.imageless + nixos-qemu.nixosModules.user + { nixpkgs.overlays = [ nixos-qemu.overlays.default ]; } + ./default.nix + ]; + }; + + # System closure for the imageless VM. The output directory + # exposes `kernel` and `initrd` symlinks consumers hand to + # QEMU via -kernel and -initrd; the closure itself is what + # virtiofsd serves to the guest as /nix/store. + # Run: nix build .#toplevel (or .#packages.<system>.toplevel) + packages.${system}.toplevel = + self.nixosConfigurations.vm.config.system.build.toplevel; + }; +} diff --git a/scripts/nixos-qemu/templates/libvirt/default.nix b/scripts/nixos-qemu/templates/libvirt/default.nix new file mode 100644 index 00000000..98186d28 --- /dev/null +++ b/scripts/nixos-qemu/templates/libvirt/default.nix @@ -0,0 +1,33 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Per-VM overrides for the libvirt template. +# +# The flake passes nixos-qemu and the flake inputs via specialArgs +# so this module can pull in workflow modules (LTP, fstests, and so +# on) and reference local source checkouts without having to edit +# flake.nix again. +{ config, lib, pkgs, nixos-qemu, inputs, ... }: { + imports = [ + # Development tools (editors, tracing, kernel dev comfort). Drop + # this import for a minimal VM that only runs a specific workflow. + nixos-qemu.nixosModules.devel + + # Pick the workflow modules that match what you intend to run, + # for example: + # nixos-qemu.nixosModules.build-tools + # nixos-qemu.nixosModules.workflows.fstests + # nixos-qemu.nixosModules.workflows.blktests + ]; + + networking.hostName = "vm"; + + # Build a package from a local source checkout. The matching + # fio-src input must be declared in flake.nix. + # nixpkgs.overlays = [ + # (final: prev: { fio = prev.fio.overrideAttrs { src = inputs.fio-src; patches = []; }; }) + # ]; + + # SSH keys for the root and workflow user accounts. + # users.users.root.openssh.authorizedKeys.keys = [ "ssh-ed25519 ..." ]; + # users.users.${config.nixos-qemu.user.name}.openssh.authorizedKeys.keys = [ "ssh-ed25519 ..." ]; +} diff --git a/scripts/nixos-qemu/templates/libvirt/flake.nix b/scripts/nixos-qemu/templates/libvirt/flake.nix new file mode 100644 index 00000000..69bb0c3a --- /dev/null +++ b/scripts/nixos-qemu/templates/libvirt/flake.nix @@ -0,0 +1,58 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Libvirt disk-image NixOS VM starter. +# +# Create with: +# nix flake init --template "github:linux-kdevops/nixos-qemu#libvirt" +# nix build .#nixosConfigurations.vm.config.system.build.toplevel +# +# Per-VM configuration lives in ./default.nix. This flake composes +# the base modules, applies the overlay, and passes nixos-qemu and +# the flake inputs through specialArgs so default.nix can import +# additional modules (for example nixos-qemu.nixosModules.workflows.*) +# and reference local source inputs without re-declaring them. +{ + inputs = { + # Local checkout preferred: downstream consumers (kdevops and + # others) pin to a specific revision via a subtree or vendored + # copy, and should not track upstream HEAD. For upstream, use: + # nixos-qemu.url = "github:linux-kdevops/nixos-qemu"; + nixos-qemu.url = "path:/path/to/nixos-qemu"; + nixpkgs.follows = "nixos-qemu/nixpkgs"; + + # Local source checkouts (uncomment to use): + # fio-src = { url = "path:/home/user/src/fio"; flake = false; }; + # kmod-src = { url = "path:/home/user/src/kmod"; flake = false; }; + }; + + outputs = { self, nixpkgs, nixos-qemu, ... }@inputs: + let + system = "x86_64-linux"; + pkgs = import nixpkgs { inherit system; }; + in { + nixosConfigurations.vm = nixpkgs.lib.nixosSystem { + inherit system; + specialArgs = { inherit inputs nixos-qemu; }; + modules = [ + nixos-qemu.nixosModules.libvirt + nixos-qemu.nixosModules.user + { nixpkgs.overlays = [ nixos-qemu.overlays.default ]; } + ./default.nix + ]; + }; + + # qcow2 disk image built from the nixosConfiguration above. + # Run: nix build .#image (or .#packages.<system>.image) + # The output is the path to the qcow2 file; libvirt consumes it + # as <disk type='file' device='disk'>/<source file='...'/>. + packages.${system}.image = import "${nixpkgs}/nixos/lib/make-disk-image.nix" { + inherit pkgs; + inherit (pkgs) lib; + config = self.nixosConfigurations.vm.config; + diskSize = 20480; + format = "qcow2"; + partitionTableType = "legacy"; + installBootLoader = true; + }; + }; +} -- 2.53.0