[PATCH v2 02/10] Squashed 'scripts/qemu-system-units/' content from commit ecafb788
Daniel Gomez <[email protected]> Fri, 12 Jun 2026 12:55:14 +0200
| Newsgroups | dev.linux.lists.kdevops |
|---|---|
| Message-ID | <[email protected]> |
From: Daniel Gomez <[email protected]> git-subtree-dir: scripts/qemu-system-units git-subtree-split: ecafb788d2446916f7051e4a9e018b31a8f960e5 --- .gitignore | 15 + CLAUDE.md | 257 ++++++++++++ COPYING | 22 + LICENSE | 5 + LICENSES/preferred/copyleft-next-0.3.1 | 239 +++++++++++ README.md | 86 ++++ docs/design-decisions.md | 715 +++++++++++++++++++++++++++++++++ docs/requirements.md | 56 +++ docs/transient-units.md | 244 +++++++++++ docs/usage.md | 461 +++++++++++++++++++++ docs/vars.md | 541 +++++++++++++++++++++++++ docs/verifying.md | 60 +++ files/network-config | 18 + files/qmp-powerdown | 2 + files/vfio-pci.conf | 1 + templates/meta-data.j2 | 3 + templates/nvme.env.j2 | 143 +++++++ templates/qemu-system-override.conf.j2 | 88 ++++ templates/[email protected] | 71 ++++ templates/transient-run.sh.j2 | 72 ++++ templates/user-data.j2 | 44 ++ templates/[email protected] | 21 + templates/vfio-udev.rules.j2 | 20 + templates/virtiofsd-override.conf.j2 | 43 ++ templates/virtiofsd.env.j2 | 11 + templates/[email protected] | 47 +++ templates/[email protected] | 24 ++ templates/vm.env.j2 | 156 +++++++ vars/example.yaml | 112 ++++++ 29 files changed, 3577 insertions(+) diff --git a/.gitignore b/.gitignore new file mode 100644 index 00000000..c0e26dfc --- /dev/null +++ b/.gitignore @@ -0,0 +1,15 @@ +# User vars files (copy vars/example.yaml to start) +vars/* +!vars/example.yaml + +# Rendered output +*.env + +# Build artifacts and binary images +images/ +*.iso + +# Editor swap files +*.swp +*.swo +*~ diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..d26e4294 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,257 @@ +# CLAUDE.md + +## Project Overview + +qemu-system-units (qsu) — systemd unit templates for running QEMU +virtual machines as first-class systemd services. Jinja2 templates +rendered by `minijinja-cli` into systemd units and environment files. +No daemon, no wrapper. + +**License**: copyleft-next-0.3.1 + +## Project Structure + +``` +qemu-system-units/ +├── templates/ Jinja2 templates (persistent + transient) +├── vars/ YAML variable files (per-VM configuration) +│ └── example.yaml Starting point (copy to vars/<vm_name>.yaml) +├── files/ Static files deployed as-is +├── docs/ Reference documentation +│ ├── vars.md Variable reference +│ ├── usage.md Rendering, deployment, operations +│ ├── design-decisions.md Hardcoded choices and rationale +│ ├── requirements.md Distro-specific packages +│ └── transient-units.md systemd-run patterns +├── LICENSES/ +└── README.md Landing page and quick start +``` + +## Critical Rules + +### Never fabricate facts + +Every command-line flag, sysfs path, systemd directive, and expected +output must be verified against the actual source before being written +into any project file. `man <tool>` or `<tool> --help` before writing +any command. If a tool only has short flags (`lspci`, `ss`, `ip`), use +short flags. + +### Never cheat during verification + +When verifying that templates render and deploy correctly, start from +a clean state. Never pre-deploy files, reuse leftover artifacts, or +skip steps. A verification that depends on state from a previous test +is a lie. + +## Rules + +### Variable names match upstream + +YAML variable names must use the exact terminology from the upstream +tool they configure. Do not invent prefixes or synonyms. + +`ram` maps to QEMU `-m`. `cpu` maps to `-cpu`. `accel` maps to +`-accel`. `image.file` maps to `-drive file=`. `image.format` maps +to `-drive format=`. The test is: can you read the variable name and +immediately know which upstream flag it maps to? + +### Flat variables for cross-cutting, nested for single-consumer + +Variables used by multiple templates stay flat (e.g. `ram`, `shares`, +`vsock_cid`). Variables consumed by a single template block can be +nested when they form a cohesive group (e.g. `kernel.image`, +`kernel.append`, `kernel.initrd` all map to one `KERNEL_ARGS=` block). + +### Backward compatibility + +Every new template variable must be gated with `| default([])` (lists), +`| default('value')` (scalars with defaults), or `is defined` +(optional features). Templates must render identically when new +variables are absent. + +### Non-transitional virtio devices + +All virtio devices use `*-pci-non-transitional` suffix on PCI machines +and `*-device` suffix on microvm (MMIO). Transitional devices require +`CONFIG_VIRTIO_PCI_LEGACY` which custom kernels often disable. The +template auto-detects microvm from `machine_type` and switches device +suffixes. `vhost-user-fs-pci` has no non-transitional variant +(modern-only device). + +### Zero-tool naming + +All rendered output references the consumer (the +`[email protected]` unit), never the generator project. Config directories are named after the +service prefix (`qemu-system`, `virtiofsd`). Only the README and +CLAUDE.md mention the project name. + +### Long-form command options + +NEVER use short flags when a long-form alternative exists in README, +templates, docs, and any commands shown to the user. `--follow` not +`-f`, `--parents` not `-p`, `--append --groups` not `-aG`. +Exception: tools without long-form options (`ssh -p`, `lspci -nv`). + +### Documentation references + +Reference only QEMU and systemd manuals, not wrapper projects. + +`See: man qemu-system` for QEMU flags. `See: man systemd.kill` for +systemd directives. `See: /usr/libexec/virtiofsd --help` for +virtiofsd flags. Use `<qemu_binary>` not `qemu-system-x86_64` when +the flag is architecture-independent. + +### Maps-to references + +"Maps to" in docs/vars.md references the upstream flag or directive, +not template filenames or env variable names. `ram` maps to `-m`, +not "QEMU_ARGS in vm.env." + +### Technical identifiers in docs + +Backtick-format systemd directives (`KillMode=`, `ExecStop=`), signal +constants (`SIGTERM`, `SIGKILL`), function calls (`sd_notify()`), +kernel configs (`CONFIG_VIRTIO_PCI`), and paths (`/dev/kvm`). Plain +text for tool names in natural language (virtiofsd, QEMU, systemd). +Systemd directives include trailing `=` matching man page convention. + +### Shell examples + +No shell variables (`$VM`, `${VM}`) in per-component documentation. +Use literal values that match vars file content (`test`, `dev`). +The deploy-all section at the bottom of docs/usage.md uses variables +for automation convenience. + +Prose instructions (editing files, logging out) go in markdown +paragraphs, not shell comments. + +### vars file convention + +`vars/example.yaml` is the template. Users copy it to +`vars/<vm_name>.yaml` (e.g. `vars/test.yaml` for `vm_name: test`). +The filename matches `vm_name` inside the file. User vars files are +gitignored. + +## Git Commit Guidelines + +### One commit per change + +Atomic commits. Spell fixes go in separate commits from code changes. + +### Commit message format + +``` +subsystem: brief description in imperative mood + +Plain English explanation of the change. NEVER use bullet points +or itemized lists in commit messages. + +Generated-by: Claude AI +Signed-off-by: Your Name <[email protected]> +``` + +### Use Signed-off-by and Generated-by tags + +Generated-by MUST be immediately followed by Signed-off-by with NO +empty lines between them. No Co-Authored-By trailer. + +### No shopping cart lists + +NEVER use bullet points or itemized lists in commit messages. Use +plain English paragraphs. + +### Subsystem prefix + +Use the template or doc name as prefix: `vm.env:`, `virtiofsd:`, +`docs:`, `vars:`, `README:`. Use `qemu-system-units:` for +cross-cutting changes. + +## Key Technical Patterns + +### `EnvironmentFile=` path + +`%E/systemd/qemu-system/%i.env` uses the `%E` specifier for +scope-agnostic paths. User mode: `~/.config/systemd/qemu-system/`. +System mode: `/etc/systemd/qemu-system/`. + +### ExecStart binary path must be literal + +systemd requires the first argument of `ExecStart=` to be a literal +path, not a variable. The binary path is rendered by Jinja2 at +template time. See: `load-fragment.c` in systemd source. + +### $MAINPID in ExecStartPost + +Available for `Type=simple` services. systemd sets the main PID +before entering the start-post phase. No shell wrapper needed for +the `busctl RegisterMachine` call. + +### Journal: --user-unit= not --user -u + +`journalctl --user -u` constrains to user journal files. But user +service output goes through `[email protected]` to the SYSTEM +journal. `journalctl --user-unit=` matches `_SYSTEMD_USER_UNIT` +across all journal files. + +### Cloud-init only runs once + +cloud-init marks first-boot completion in `/var/lib/cloud/`. A new +seed ISO does NOT trigger cloud-init to rerun on an existing image. +To re-provision, either use a fresh disk image or delete +`/var/lib/cloud/` inside the guest. + +### Three-layer template pattern + +Templates follow three layers: (1) static unit structure that never +changes, (2) conditional blocks gated on vars (`{% if X is defined %}`), +and (3) loops over lists (`{% for dev in list | default([]) %}`). +Layers 2 and 3 emit nothing when the variable is absent. New +features add new conditional/loop blocks without modifying the +static layer. + +## Rendering + +All rendering uses `minijinja-cli --trim-blocks`. One vars file drives +all templates. + +```shell +minijinja-cli --trim-blocks \ + --output <deploy-path> \ + templates/<template>.j2 \ + vars/<vm_name>.yaml +``` + +Before committing, run the verification steps in +[docs/verifying.md](docs/verifying.md): render every template, +`systemd-analyze verify` the rendered units, review the +`systemd-analyze security` posture, and check the commit message. + +## Source Code References + +### QEMU + +`qemu-options.hx` (all command-line options), +`hw/nvme/ctrl.c` (NVMe device, serial requirement at line 8600), +`hw/i386/microvm.c` (microvm machine type, pcie=on property), +`system/runstate.c` (SIGTERM handling, force_shutdown at line 786), +`hw/virtio/vhost-vsock.c` (VSOCK CID range validation at line 134). + +### systemd + +`man systemd.service` (Type=, ExecStart=, ExecStop=), +`man systemd.exec` (WorkingDirectory=, LimitMEMLOCK=, %U/%G/%h/%E specifiers), +`man systemd.resource-control` (CPUQuota=, MemoryMax=, Slice=), +`man systemd.kill` (KillMode=, KillSignal=, TimeoutSec=), +`man systemd.special` (machine.slice, machines.target). + +systemd does NOT expand specifiers in `EnvironmentFile=` values. `%h` +in `Environment=` is expanded. `%h` inside an `EnvironmentFile=` is +literal. + +### virtiofsd + +`src/sandbox.rs` (namespace sandbox, setresuid at line 497, +default UID mapping at line 331), +`--sandbox=namespace --uid-map :0:%U:1: --gid-map :0:%G:1:` is the +canonical unprivileged configuration. diff --git a/COPYING b/COPYING new file mode 100644 index 00000000..f5479109 --- /dev/null +++ b/COPYING @@ -0,0 +1,22 @@ +This project is provided under: + + SPDX-License-Identifier: copyleft-next-0.3.1 + LICENSES/preferred/copyleft-next-0.3.1 + +In addition, other licenses may also apply. We embrace the same SPDX +practice as used in the Linux kernel, for those details refer to under +Linux: + + Documentation/process/license-rules.rst + +for more details. + +Note: Linux accepts copyleft-next licensed code, however, even though +copyleft-next is GPL v2 compatible we use a dual license tag on Linux +to err on the side of caution. And so, if you ever do wish to use code +from this project on Linux be sure to use this tag instead: + +// SPDX-License-Identifier: GPL-2.0-or-later OR copyleft-next-0.3.1 + +This project equally accepts GPL-2.0 code only as copyleft-next is GPL +v2 compatible. diff --git a/LICENSE b/LICENSE new file mode 100644 index 00000000..28dadc58 --- /dev/null +++ b/LICENSE @@ -0,0 +1,5 @@ +Copyright (c) 2026 Daniel Gomez <[email protected]> + +qemu-system-units is licensed under copyleft-next-0.3.1 + +Refer to LICENSES/preferred/copyleft-next-0.3.1 for license details. diff --git a/LICENSES/preferred/copyleft-next-0.3.1 b/LICENSES/preferred/copyleft-next-0.3.1 new file mode 100644 index 00000000..5e04da3d --- /dev/null +++ b/LICENSES/preferred/copyleft-next-0.3.1 @@ -0,0 +1,239 @@ +Valid-License-Identifier: copyleft-next-0.3.1 +SPDX-URL: https://spdx.org/licenses/copyleft-next-0.3.1 +Usage-Guide: + This license can be used in code, it has been found to be GPLv2 compatible + by attorneys at Redhat and SUSE, however to err on the side of caution, + if used on Linux it's best to only use it together with a GPL2 compatible + license using "OR". You do not have to do this for this project, + as it is licensed under the copyleft-next-0.3.1 license. + To use the copyleft-next-0.3.1 license on Linux put the following SPDX + tag/value pair into a comment according to the placement guidelines in the + licensing rules documentation: + SPDX-License-Identifier: GPL-2.0 OR copyleft-next-0.3.1 + SPDX-License-Identifier: GPL-2.0-only OR copyleft-next 0.3.1 + SPDX-License-Identifier: GPL-2.0+ OR copyleft-next-0.3.1 + SPDX-License-Identifier: GPL-2.0-or-later OR copyleft-next-0.3.1 +License-Text: + +======================================================================= + + copyleft-next 0.3.1 ("this License") + Release date: 2016-04-29 + +1. License Grants; No Trademark License + + Subject to the terms of this License, I grant You: + + a) A non-exclusive, worldwide, perpetual, royalty-free, irrevocable + copyright license, to reproduce, Distribute, prepare derivative works + of, publicly perform and publicly display My Work. + + b) A non-exclusive, worldwide, perpetual, royalty-free, irrevocable + patent license under Licensed Patents to make, have made, use, sell, + offer for sale, and import Covered Works. + + This License does not grant any rights in My name, trademarks, service + marks, or logos. + +2. Distribution: General Conditions + + You may Distribute Covered Works, provided that You (i) inform + recipients how they can obtain a copy of this License; (ii) satisfy the + applicable conditions of sections 3 through 6; and (iii) preserve all + Legal Notices contained in My Work (to the extent they remain + pertinent). "Legal Notices" means copyright notices, license notices, + license texts, and author attributions, but does not include logos, + other graphical images, trademarks or trademark legends. + +3. Conditions for Distributing Derived Works; Outbound GPL Compatibility + + If You Distribute a Derived Work, You must license the entire Derived + Work as a whole under this License, with prominent notice of such + licensing. This condition may not be avoided through such means as + separate Distribution of portions of the Derived Work. + + If the Derived Work includes material licensed under the GPL, You may + instead license the Derived Work under the GPL. + +4. Condition Against Further Restrictions; Inbound License Compatibility + + When Distributing a Covered Work, You may not impose further + restrictions on the exercise of rights in the Covered Work granted under + this License. This condition is not excused merely because such + restrictions result from Your compliance with conditions or obligations + extrinsic to this License (such as a court order or an agreement with a + third party). + + However, You may Distribute a Covered Work incorporating material + governed by a license that is both OSI-Approved and FSF-Free as of the + release date of this License, provided that compliance with such + other license would not conflict with any conditions stated in other + sections of this License. + +5. Conditions for Distributing Object Code + + You may Distribute an Object Code form of a Covered Work, provided that + you accompany the Object Code with a URL through which the Corresponding + Source is made available, at no charge, by some standard or customary + means of providing network access to source code. + + If you Distribute the Object Code in a physical product or tangible + storage medium ("Product"), the Corresponding Source must be available + through such URL for two years from the date of Your most recent + Distribution of the Object Code in the Product. However, if the Product + itself contains or is accompanied by the Corresponding Source (made + available in a customarily accessible manner), You need not also comply + with the first paragraph of this section. + + Each direct and indirect recipient of the Covered Work from You is an + intended third-party beneficiary of this License solely as to this + section 5, with the right to enforce its terms. + +6. Symmetrical Licensing Condition for Upstream Contributions + + If You Distribute a work to Me specifically for inclusion in or + modification of a Covered Work (a "Patch"), and no explicit licensing + terms apply to the Patch, You license the Patch under this License, to + the extent of Your copyright in the Patch. This condition does not + negate the other conditions of this License, if applicable to the Patch. + +7. Nullification of Copyleft/Proprietary Dual Licensing + + If I offer to license, for a fee, a Covered Work under terms other than + a license that is OSI-Approved or FSF-Free as of the release date of this + License or a numbered version of copyleft-next released by the + Copyleft-Next Project, then the license I grant You under section 1 is no + longer subject to the conditions in sections 3 through 5. + +8. Copyleft Sunset + + The conditions in sections 3 through 5 no longer apply once fifteen + years have elapsed from the date of My first Distribution of My Work + under this License. + +9. Pass-Through + + When You Distribute a Covered Work, the recipient automatically receives + a license to My Work from Me, subject to the terms of this License. + +10. Termination + + Your license grants under section 1 are automatically terminated if You + + a) fail to comply with the conditions of this License, unless You cure + such noncompliance within thirty days after becoming aware of it, or + + b) initiate a patent infringement litigation claim (excluding + declaratory judgment actions, counterclaims, and cross-claims) + alleging that any part of My Work directly or indirectly infringes + any patent. + + Termination of Your license grants extends to all copies of Covered + Works You subsequently obtain. Termination does not terminate the + rights of those who have received copies or rights from You subject to + this License. + + To the extent permission to make copies of a Covered Work is necessary + merely for running it, such permission is not terminable. + +11. Later License Versions + + The Copyleft-Next Project may release new versions of copyleft-next, + designated by a distinguishing version number ("Later Versions"). + Unless I explicitly remove the option of Distributing Covered Works + under Later Versions, You may Distribute Covered Works under any Later + Version. + +** 12. No Warranty ** +** ** +** My Work is provided "as-is", without warranty. You bear the risk ** +** of using it. To the extent permitted by applicable law, each ** +** Distributor of My Work excludes the implied warranties of title, ** +** merchantability, fitness for a particular purpose and ** +** non-infringement. ** + +** 13. Limitation of Liability ** +** ** +** To the extent permitted by applicable law, in no event will any ** +** Distributor of My Work be liable to You for any damages ** +** whatsoever, whether direct, indirect, special, incidental, or ** +** consequential damages, whether arising under contract, tort ** +** (including negligence), or otherwise, even where the Distributor ** +** knew or should have known about the possibility of such damages. ** + +14. Severability + + The invalidity or unenforceability of any provision of this License + does not affect the validity or enforceability of the remainder of + this License. Such provision is to be reformed to the minimum extent + necessary to make it valid and enforceable. + +15. Definitions + + "Copyleft-Next Project" means the project that maintains the source + code repository at <https://github.com/copyleft-next/copyleft-next.git/> + as of the release date of this License. + + "Corresponding Source" of a Covered Work in Object Code form means (i) + the Source Code form of the Covered Work; (ii) all scripts, + instructions and similar information that are reasonably necessary for + a skilled developer to generate such Object Code from the Source Code + provided under (i); and (iii) a list clearly identifying all Separate + Works (other than those provided in compliance with (ii)) that were + specifically used in building and (if applicable) installing the + Covered Work (for example, a specified proprietary compiler including + its version number). Corresponding Source must be machine-readable. + + "Covered Work" means My Work or a Derived Work. + + "Derived Work" means a work of authorship that copies from, modifies, + adapts, is based on, is a derivative work of, transforms, translates or + contains all or part of My Work, such that copyright permission is + required. The following are not Derived Works: (i) Mere Aggregation; + (ii) a mere reproduction of My Work; and (iii) if My Work fails to + explicitly state an expectation otherwise, a work that merely makes + reference to My Work. + + "Distribute" means to distribute, transfer or make a copy available to + someone else, such that copyright permission is required. + + "Distributor" means Me and anyone else who Distributes a Covered Work. + + "FSF-Free" means classified as 'free' by the Free Software Foundation. + + "GPL" means a version of the GNU General Public License or the GNU + Affero General Public License. + + "I"/"Me"/"My" refers to the individual or legal entity that places My + Work under this License. "You"/"Your" refers to the individual or legal + entity exercising rights in My Work under this License. A legal entity + includes each entity that controls, is controlled by, or is under + common control with such legal entity. "Control" means (a) the power to + direct the actions of such legal entity, whether by contract or + otherwise, or (b) ownership of more than fifty percent of the + outstanding shares or beneficial ownership of such legal entity. + + "Licensed Patents" means all patent claims licensable royalty-free by + Me, now or in the future, that are necessarily infringed by making, + using, or selling My Work, and excludes claims that would be infringed + only as a consequence of further modification of My Work. + + "Mere Aggregation" means an aggregation of a Covered Work with a + Separate Work. + + "My Work" means the particular work of authorship I license to You + under this License. + + "Object Code" means any form of a work that is not Source Code. + + "OSI-Approved" means approved as 'Open Source' by the Open Source + Initiative. + + "Separate Work" means a work that is separate from and independent of a + particular Covered Work and is not by its nature an extension or + enhancement of the Covered Work, and/or a runtime library, standard + library or similar component that is used to generate an Object Code + form of a Covered Work. + + "Source Code" means the preferred form of a work for making + modifications to it. diff --git a/README.md b/README.md new file mode 100644 index 00000000..3d70f309 --- /dev/null +++ b/README.md @@ -0,0 +1,86 @@ +# qemu-system-units + +qemu-system-units (qsu) — systemd unit templates for running QEMU +virtual machines as first-class systemd services. + +**License**: copyleft-next-0.3.1 + +## Features + +- **Machine types**: q35, microvm (auto MMIO), ARM virt +- **Boot modes**: cloud image, mkosi, direct kernel, imageless (virtiofs rootfs) +- **File sharing**: virtiofs (socket-activated), 9P fallback +- **Networking**: user-mode with port forwarding, VSOCK +- **Devices**: NVMe emulation (ZNS, multipath, FDP), PCIe passthrough (VFIO) +- **IOMMU**: Intel VT-d, AMD-Vi, virtio-iommu, ARM SMMUv3 +- **Debugging**: GDB via unix socket, paused start +- **Lifecycle**: machined registration, QMP graceful shutdown, resource control + +## Quick start + +```shell +sudo apt install qemu-system-x86 qemu-utils socat \ + systemd-container virtiofsd cloud-image-utils cargo +cargo install minijinja-cli +sudo usermod --append --groups kvm,systemd-journal $(whoami) +``` + +Log out and back in for group changes to take effect. +See [docs/requirements.md](docs/requirements.md) for Fedora, openSUSE, and NixOS. + +```shell +cp vars/example.yaml vars/test.yaml + +mkdir --mode=0755 --parents ~/.config/systemd/user ~/.config/systemd/qemu-system ~/.config/systemd/virtiofsd +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] vars/test.yaml +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] vars/test.yaml +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] vars/test.yaml +mkdir --mode=0755 --parents ~/.config/systemd/user/[email protected] +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected]/override.conf \ + templates/qemu-system-override.conf.j2 vars/test.yaml +mkdir --mode=0755 --parents ~/.config/systemd/user/[email protected] +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected]/override.conf \ + templates/virtiofsd-override.conf.j2 vars/test.yaml +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/qemu-system/test.env \ + templates/vm.env.j2 vars/test.yaml +cp files/qmp-powerdown ~/.config/systemd/qemu-system/qmp-powerdown + +systemctl --user daemon-reload +systemctl --user start qemu-system@test +machinectl --user list +``` + +## Documentation + +| Document | Content | +|---|---| +| [docs/requirements.md](docs/requirements.md) | Packages for Debian, Fedora, openSUSE, NixOS | +| [docs/usage.md](docs/usage.md) | Rendering, deployment, machinectl, console, logs, stop, multiple VMs | +| [docs/vars.md](docs/vars.md) | Variable reference for all template fields | +| [docs/design-decisions.md](docs/design-decisions.md) | Hardcoded choices and upstream references | +| [docs/transient-units.md](docs/transient-units.md) | `systemd-run` and transient unit patterns | +| [docs/verifying.md](docs/verifying.md) | Pre-commit checklist: render, `systemd-analyze verify`, security review | +| [vars/example.yaml](vars/example.yaml) | Starting point for VM configuration | + +## Related work + +Community systemd service files for QEMU VMs: + +- [rafaelmartins/kvm-systemd](https://github.com/rafaelmartins/kvm-systemd). `[email protected]`, `Type=forking`, per-VM conf in `/etc/kvm/` +- [eaon/qemu-kvm-systemd-service](https://codeberg.org/eaon/qemu-kvm-systemd-service). `[email protected]`, per-VM conf in `/etc/qemu/vms/` +- [dehesselle/virtctl](https://github.com/dehesselle/virtctl). `[email protected]`, `hypervisor.target` pattern +- [0xef53/kvmrun](https://github.com/0xef53/kvmrun). Go, systemd-native, per-VM chroot, gRPC API + +QEMU upstream ships helper units (guest-agent, pr-helper, vmsr-helper) in +[contrib/systemd/](https://gitlab.com/qemu-project/qemu/-/tree/master/contrib/systemd) +but no VM lifecycle service. virtiofsd ships no systemd units. Neither +Debian nor Fedora package a `[email protected]` template. diff --git a/docs/design-decisions.md b/docs/design-decisions.md new file mode 100644 index 00000000..627d33ce --- /dev/null +++ b/docs/design-decisions.md @@ -0,0 +1,715 @@ +# Design decisions + +Managing QEMU VMs has two parts: generating the QEMU command line +(machine type, devices, disks, networking, kernel boot, passthrough) +and VM lifecycle (start, stop, restart, dependencies, logging, +resource control). systemd handles the lifecycle. Templates handle +the command line. No new daemon, no new wrapper. + +Rendering and deploying the templates is left to the consumer. +The raw workflow is `minijinja-cli` + `systemctl`. Automation +layers are out of scope for this project. + +The templates aim to be as unopinionated as possible. When a value +is hardcoded, it is either required by the underlying tool (QEMU, +virtiofsd, systemd), a workaround with documented reasoning, or a +sensible default that the user can override. This document lists +every such choice, the upstream reference that justifies it, and +how to change it when possible. + +Source-code references with line numbers (`src/core/...`, +`src/machine/...`, `src/libsystemd/...`) are verified against +systemd `v260.1` (see `meson.version` in the systemd tree). Each +cite was introduced by the commit that added the surrounding +prose; this paragraph just records the version pin. Newer +systemd releases may shift line numbers; the symbol names in +each citation stay stable across recent releases, so grep +upstream by symbol if the cited line no longer matches. + +## Variable naming + +Variable names match upstream flag or parameter names when a 1:1 +mapping exists. When a variable controls a higher-level concept, +a descriptive name is used. + +**`image`** — QEMU's flag is `-drive`. QEMU's own docs describe the +value as "which disk image to use with this drive." The sub-properties +(`image.file`, `image.format`, `image.cache`) match `-drive` property +names exactly. The name `image` describes what the user configures (a +disk image), while the properties map 1:1 to QEMU's `-drive` +properties. + +**`cpus`** — QEMU's flag is `-smp`. The parameter inside `-smp` is +called `cpus=`. The variable matches the parameter name. + +**`vsock_cid`** — QEMU's device property is `guest-cid`. The variable +adds the technology context (`vsock`) because `guest_cid` alone is +ambiguous. Maps to `-device vhost-vsock-pci,guest-cid=`. + +**`ssh_port`** — No upstream equivalent. Maps to +`-nic hostfwd=tcp:127.0.0.1:<port>-:22`. The concept (SSH port +forwarding) spans multiple flag components. + +**`pci_passthrough`** — No upstream equivalent. Maps to +`-device vfio-pci,host=<addr>`. Combines the bus type (PCI) with the +operation (passthrough). + +**`autostart`** — No upstream equivalent. Inverted boolean: `false` +maps to `-S` (QEMU starts paused). QEMU has no "autostart" flag. + +**`firmware`** — QEMU has no `-firmware` flag. Firmware selection is +implicit: pflash0 populated → UEFI, pflash0 absent → BIOS. QEMU's +own firmware specification (`docs/interop/firmware.json`) calls the +top-level concept `Firmware` and the internal function is +`pc_system_firmware_init()`. The sub-properties `code` and `vars` +match the OVMF file naming convention (`OVMF_CODE_4M.fd`, +`OVMF_VARS_4M.fd`). QEMU's spec uses the more verbose `executable` +and `nvram-template` but the OVMF file names are what users encounter +directly. + +**`cloud_init.users[].password`** — Cloud-init's key is +`plain_text_passwd`. The variable uses `password` for simplicity. The +template maps it to `plain_text_passwd`. + +**`nvme.drives[].pmr`** — A nested dict rather than flat keys on the +controller. QEMU's PMR is two constructs: a `-object +memory-backend-file` and a `pmrdev=` link from `-device nvme` to that +object's id. The other controller knobs (`cmb_size_mb`, `mdts`, +`atomic.*`) are flat because each is a single `nvme,` device property, +matching the flat-for-cross-cutting rule. PMR is single-consumer (one +controller owns one backend object) and its sub-keys belong to a +*separate* object, not the `nvme,` device line, so the cohesive nested +group is the honest model. The leaf names (`size`, `mem-path`, `share`, +`pmem`) are the exact `memory-backend-file` property names, so the +mapping stays one-to-one. The object id is generated by the template +(`nvme-pmr-<index>`), not user-supplied, because it only exists to wire +the `-object` to the `pmrdev=` link. + +## User-configurable (vars file) + +Fully controlled by the user. See [vars.md](vars.md) for reference. + +`cpu`, `accel`, `ram`, `cpus`, `machine_type`, `iommu`, +`firmware` (code, vars, vars_format), `gdb`, `autostart`, +`image` (file, format, cache, aio, discard, detect-zeroes), +`drives`, `ssh_port`, `vsock_cid`, `ssh_private_key`, +`kernel` (image, append, initrd), +`shares` (tag, mount, dir, translate_uid, translate_gid), +`share_transport`, `virtiofsd_binary`, +`cloud_init` (seed, locale, ssh_pubkey, users), +`pci_passthrough`, `nvme` (drives, subsystems), +`service` (any `[Service]` directive) + +## systemd service properties + +**`Documentation=`** — Set to +`man:qemu-system(1) man:systemd.service(5) man:systemd.kill(5)`. +References the upstream man pages relevant to the unit: QEMU's own +manual for the binary the service launches, systemd.service(5) for +the unit type and lifecycle directives, systemd.kill(5) for the +deliberate `KillSignal=SIGCONT` + `KillMode=mixed` choice the unit +makes. Surfaces in `systemctl status` and `systemctl help` so an +operator inspecting the unit reaches the canonical references +without leaving the CLI. Project-specific docs are not referenced +from the unit; only upstream man pages whose stability is +guaranteed by their respective maintainers. See: +`man systemd.unit`. + +**`KillMode=`** — Set to `mixed`. Sends `KillSignal=` to the main +process. After the main process exits or `TimeoutStopSec=` elapses, +remaining processes in the cgroup receive `SIGKILL`. For VMs, the +main process is the QEMU process. After `ExecStop=` runs QMP +graceful shutdown, if the QEMU process exits cleanly, any leaked +child processes are killed immediately. The alternative +`control-group` would send `KillSignal=` to all processes +simultaneously, which is unnecessary when only the QEMU process +needs the signal. See: `man systemd.kill`. + +**`TimeoutStopSec=`** — Set to `2min`. Grace period for `ExecStop=` +before systemd sends `SIGKILL`. The systemd default +(`DefaultTimeoutStopSec=`) is 90s. VMs need longer because ACPI +powerdown triggers a full guest OS shutdown sequence (flushing +buffers, stopping services, unmounting filesystems). Override: + +```yaml +service: + TimeoutStopSec: 5min +``` + +See: `man systemd.service`. + +**`Slice=`** — Set to `machine.slice`. All virtual machines and +containers registered with systemd-machined are placed in +machine.slice. Canonical cgroup placement for VMs. +See: `man systemd.special`. + +**`PartOf=`**, **`Before=`**, **`WantedBy=`** — Set to +`machines.target`. Standard target for starting all containers and +virtual machines. `PartOf=` ensures VMs stop when machines.target +stops. `WantedBy=` enables auto-start. See: `man systemd.special`. + +**`Type=`** — Set to `simple`. QEMU does not implement +`sd_notify()`. `notify` would be correct but requires the service to +call `sd_notify(READY=1)` after initialization. A patch adding +`sd_notify()` to QEMU was submitted (qemu-devel, 2025-12-17) but has +not been merged. When QEMU gains `sd_notify()` support, this should +change to `notify`. See: `man systemd.service`. + +**`Restart=`** — Deliberately unset (systemd default `no`). A +restart policy is a deployment choice the templates do not impose: +`Restart=always` would revive a VM an operator powered off on +purpose, and auto-restart on failure is only sometimes wanted. An +operator who wants auto-restart on a kernel-panic-style exit sets +`service: { Restart: on-failure }`; the per-VM drop-in pins +virtiofsd with `Requires=` (see "virtiofsd dependency" below), +which carries `UNIT_ATOM_PULL_IN_START` +(`src/core/unit-dependency-atom.c:17`) and so pulls a fresh +virtiofsd in when the service restarts. See: `man systemd.service`. + +**`SyslogIdentifier=`** — Set to `qemu-system@%i`. Default journal +output (`journalctl --output=short`) prefixes each line with the +syslog identifier, which falls back to the executed process name +when this directive is unset (`man systemd.exec`). The systemd +reference templates `[email protected]` and +`[email protected]` rely on that default because their +binary names match their service names and are informative as a +prefix. The QEMU template does not have that property: every +`qemu-system@<vm>.service` instance runs the same +`qemu-system-x86_64` binary, plus helper processes (`varlinkctl`, +`socat`, `ssh`) for `ExecStartPost=` and `ExecStop=`. Without +override, the default prefix is a mix of `qemu-system-x86_64[PID]`, +`varlinkctl[PID]`, `socat[PID]`, and parens-wrapped variants for +pre-exec failures, none of which carry the VM identity. Setting +`SyslogIdentifier=qemu-system@%i` pulls every per-service journal +record under one prefix that names both the service template and +the instance, e.g. `qemu-system@xarray[PID]`. Manager messages +(`systemd[PID]`) are emitted by systemd itself and remain +unaffected. The architecture suffix lost from the prefix is still +recoverable from the `_EXE` field (`journalctl --output=verbose`), +from `systemctl status` showing the full `ExecStart=` line, and +from QEMU's own self-identification in error messages +(`qemu-system-x86_64: -drive file=...`). See: `man systemd.exec`. + +## Security hardening + +`[email protected]` restricts device access with +`DevicePolicy=closed` plus an explicit `DeviceAllow=` list. This +mirrors `[email protected]`, systemd's own VM unit, whose +*entire* hardening block is exactly a closed device policy with an +allow-list — no `ProtectSystem=`, `RestrictAddressFamilies=`, +`SystemCallFilter=`, or the rest of the `systemd.exec` sandboxing +set. systemd ships its VM unit that way deliberately, and the same +reasoning applies here: a QEMU VM legitimately needs broad +filesystem, syscall, and network access, so the device boundary is +the hardening that fits without fighting the workload. + +The allow-list covers what the QEMU command line opens. `/dev/kvm` +backs `-accel kvm`; `/dev/vhost-vsock` backs the +`vhost-vsock-pci` device. `virtio-rng` reads `/dev/urandom`, which +`DevicePolicy=closed` already permits as part of its default set +(`/dev/null`, `/dev/zero`, `/dev/full`, `/dev/random`, +`/dev/urandom`, `/dev/tty`, the pseudo-TTY nodes). The qemu-system +templates use `-nic user` (userspace SLIRP) and emulated NVMe +backed by image files, neither of which opens a device node, so +nothing else is needed for the common case. + +### PCIe passthrough clashes with the closed policy + +A passthrough VM is the one case where the closed policy is not +self-contained. `-device vfio-pci` opens `/dev/vfio/vfio` (the +container, a fixed path the per-VM drop-in allow-lists) and one +`/dev/vfio/<iommu-group>` node per assigned device — and the IOMMU +group numbers are only known at runtime, so the template cannot +render `DeviceAllow=` lines for them. The operator closes the gap +either by allow-listing the specific group nodes +(`service: { DeviceAllow: "/dev/vfio/42 rw" }`) or by opting that +VM out of the closed policy (`service: { DevicePolicy: auto }`). +This is documented rather than worked around because the group +numbers are genuinely not render-time knowable. + +### Why not the `systemd.exec` sandboxing set + +Each `systemd.exec` directive omitted here was omitted because it +conflicts with a QEMU feature, not by oversight. `PrivateDevices=yes` +would hide `/dev/kvm`. `NoNewPrivileges=yes` breaks +`qemu-bridge-helper`, which is SETUID — relevant if a deployment +switches off `-nic user`. `MemoryDenyWriteExecute=yes` breaks the +TCG JIT, so it is safe only under pure KVM. `SystemCallFilter=` +can block KVM ioctls. `ProtectSystem=strict` needs +`ReadWritePaths=` enumerated for the disk image, state, and +runtime directories; `ProtectHome=` conflicts with a virtiofsd +home share. An operator whose deployment is narrow enough to adopt +any of these adds it through the `service` vars key, which accepts +any `[Service]` directive. See: `man systemd.resource-control`, +`man systemd.exec`, `man systemd-analyze` (the `security` verb), +and `[email protected]`. + +## machined registration + +Each VM registers itself with machined at start-up via +`ExecStartPost=` running +`varlinkctl call <socket> io.systemd.Machine.Register <json>`. +Socket path: `/run/user/%U/systemd/machine/io.systemd.Machine` +(user scope) or `/run/systemd/machine/io.systemd.Machine` +(system scope). Required JSON fields: `name`, `class=vm`, +`service=qemu-system`, `leader=${MAINPID}`. Optional: +`vSockCid` when the vars file sets `vsock_cid`; `id` when the +vars file sets `uuid`. + +### `id` correlates QEMU's UUID with machined's record + +When `uuid` is set in the vars file, the same value is passed to +QEMU's `-uuid` flag (rendered into vm.env's `QEMU_ARGS`) and to +the Register call's `id` field +(`sd_json_dispatch_id128` in `src/machine/machine-varlink.c:133`). +The host sees it as `Id=` in `machinectl show <vm_name>`; the +guest sees it as `dmidecode -s system-uuid` and +`/sys/class/dmi/id/product_uuid`. A single user-supplied UUID +threads through both sides, so tooling that correlates by +machine UUID resolves to the same VM regardless of which side +queries. No auto-derivation: omit `uuid` and QEMU passes its +default (all-zeros) while the Register call omits `id`. + +Registering `vSockCid` is what makes `ssh machine/<vm>` route +over AF_VSOCK. The shipped +`/etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf` hands the +`machine/*` pattern to `systemd-ssh-proxy`, which looks the +machine up over Varlink and reads the registered CID. Without +it, `systemd-ssh-proxy` exits with "Machine has no AF_VSOCK +CID assigned" (`src/ssh-generator/ssh-proxy.c`). + +### Why two `ExecStartPost=` lines across two templates + +The shared `[email protected]` renders once for all +instances and has no per-VM variables at that render time, so +its Jinja cannot branch on `vsock_cid`. It emits one +`ExecStartPost=` that registers without CID - correct for any +VM, always safe. + +The per-VM `qemu-system-override.conf.j2` renders with +`vsock_cid` in scope. When set, it emits `ExecStartPost=` +(empty) followed by `ExecStartPost=-varlinkctl ... vSockCid:N`. +Systemd treats the empty directive as a reset of the +accumulated list, so the drop-in's call replaces the shared +template's call rather than running alongside. Net effect: +every VM makes one registration call, with CID if configured, +without otherwise. VMs that never set `vsock_cid` hit only the +shared template's call and behave exactly as before. + +### Why Varlink and not `busctl` + +The legacy DBus `RegisterMachine` method +(`src/machine/machined-dbus.c`) has a fixed `sayssus` +signature with no `vSockCid` field. `RegisterMachineEx` +accepts `VSockCID` but requires the leader identified by +`LeaderPIDFD` or `LeaderPID+LeaderPIDFDID`, neither of which +a shell script can synthesise. Varlink's +`io.systemd.Machine.Register` leader dispatcher accepts a +bare integer PID and machined acquires the pidfd daemon-side +(`json_dispatch_pidref` in +`src/libsystemd/sd-json/json-util.c`). + +### Quoting and expansion + +`${MAINPID}` uses the brace form because systemd's exec parser +expands `${VAR}` (but not `$VAR`) inside escape-quoted +arguments, letting the JSON body ride as one argv entry +without a `/bin/sh -c` wrapper. The drop-in inlines +`{{ vsock_cid }}` at Jinja render time rather than pulling +`${VSOCK_CID}` from the environment because the value is +already known per-VM. + +The `-` prefix on `ExecStartPost=` keeps the VM running if +machined is unreachable. Registration is informational. + +### `machinectl list` columns OS, VERSION, ADDRESSES are unsupported for VMs + +Both `machine_get_os_release` and `machine_get_addresses` in +machined return `-EOPNOTSUPP` for `MACHINE_VM` +(`src/machine/machined-core.c:418` and `:323`). They handle +`MACHINE_HOST` directly and `MACHINE_CONTAINER` via +`namespace_fork` into the container's mnt+pid+root or net +namespace; neither path applies to a VM whose kernel and netstack +live behind a hypervisor boundary. The varlink protocol exposes +no per-machine update method either - `Register / List / +Unregister / Terminate / Kill / Open / OpenRootDirectory / +MapFrom / MapTo / BindMount / CopyFrom / CopyTo` +(`src/machine/machined-varlink.c:789-800`) - so a registered VM +has no way to push os-release or current addresses back to +machined post-Register. The `ifIndices` field in the Register +dispatch table (`machine-varlink.c:141`) is captured into the +`Machine` struct but `machine_get_addresses` for `MACHINE_VM` +short-circuits with `EOPNOTSUPP` before any per-class logic, so +it is currently inert. + +Operator-side configuration cannot fill those columns. Closing +the gap requires an upstream patch adding push-style varlink +methods plus a guest-side notifier service over `AF_VSOCK`. + +## QEMU workarounds + +**`KillSignal=`** — Set to `SIGCONT`. QEMU on `SIGTERM` calls +`qemu_system_killed()` which sets `force_shutdown=true` and exits +immediately without ACPI powerdown. `SIGCONT` is a no-op signal that +keeps the QEMU process alive, giving `ExecStop=` time to send QMP +system_powerdown for proper ACPI shutdown. After `TimeoutStopSec=`, +systemd sends `SIGKILL` (via `KillMode=mixed`). Not +user-configurable. See: `man systemd.kill`. + +**`ExecStop=`** — Sends QMP system_powerdown via socat for graceful +guest shutdown. QEMU does not translate `SIGTERM` to ACPI powerdown. +When `ssh_private_key` is defined, `ExecStop=` first attempts SSH +shutdown (guest-initiated poweroff) with QMP as fallback. Not +user-configurable (the mechanism is fixed; the SSH path is enabled by +setting `ssh_private_key` and `vsock_cid`). + +## QEMU firmware + +**`-drive if=pflash`** — OVMF firmware via pflash drives. QEMU's +firmware specification (`docs/interop/firmware.json`) defines the +`split` mode: the executable (CODE) is read-only and shared, the +NVRAM template (VARS) is cloned per-VM and configured read-write. +pflash0 holds the executable, pflash1 holds the per-VM NVRAM file. + +QEMU selects firmware mode based on pflash0 presence: if pflash0 is +populated, QEMU enters pflash mode (UEFI); if absent, ROM mode +(SeaBIOS). There is no explicit firmware flag; the presence of pflash +drives is the entire mechanism. See: `hw/i386/pc_sysfw.c` +`pc_system_firmware_init()`. + +pflash is created by `pc_system_flash_create()` which is part of the +PC machine initialization path. microvm does not call this function; +it calls `x86_bios_rom_init()` directly, even with `pcie=on`. The +`pcie=on` property on microvm only enables the GPEX PCIe host bridge +for PCI devices; it does not change the firmware initialization path. +ARM virt machines create pflash independently (256 KB sectors vs +x86's 4 KB). + +The NVRAM file stores UEFI boot order, Secure Boot key databases +(PK, KEK, db, dbx), and guest-written variables. Each VM needs its +own writable NVRAM file for independent variable state. QEMU enforces +this with file-level locking. A second VM attempting to open the same +writable NVRAM file will fail to start. Create a per-VM copy from the +template: + +```shell +cp /usr/share/OVMF/OVMF_VARS_4M.fd images/test-ovmf-vars.fd +``` + +The firmware spec also supports qcow2 format. An alternative to +copying is a qcow2 overlay with the template as backing file: + +```shell +qemu-img create --format qcow2 \ + --backing /usr/share/OVMF/OVMF_VARS_4M.fd \ + --backing-format raw \ + images/test-ovmf-vars.qcow2 +``` + +This saves disk space (only changed variables are stored) and keeps +the system template untouched. Use `format: qcow2` in the vars file +when using a qcow2 NVRAM file. + +When the `firmware` section is absent from vars, no pflash drives are +rendered and QEMU defaults to SeaBIOS. Backward compatible. +User-configurable: set `firmware.code` and `firmware.vars`. + +## QEMU device choices + +**`-device virtio-*-pci-non-transitional`** — All virtio devices use +non-transitional variants. These only require `CONFIG_VIRTIO_PCI` in +the guest kernel. Transitional devices additionally require +`CONFIG_VIRTIO_PCI_LEGACY`, which custom kernels often disable. On +microvm, devices use the `-device virtio-*-device` suffix +(virtio-mmio transport). vhost-user-fs-pci has no non-transitional +variant (modern-only device). Not user-configurable (determined by +`machine_type`). See: `<qemu_binary> -device help`. + +**`-device virtio-rng-*`** — Always present. Provides entropy to the +guest. Without it, `/dev/random` may block and boot can stall waiting +for entropy. Not user-configurable. + +**`-nographic`**, **`-serial mon:stdio`** — Headless operation with +serial console on stdio. Standard interface for kernel development +VMs. The serial output goes to the systemd journal via stdout +capture. VGA, SPICE, and VNC are not supported. Not +user-configurable. + +**`-nic user`** — User-mode networking (SLIRP). No host privileges +required. Port forwarding controlled via `ssh_port`. Alternative +host-guest transport via VSOCK (`vsock_cid`). TAP and bridge +networking are not supported (require root or network namespace +setup). The NIC model is determined by `machine_type`. +See: `man qemu-system`. + +**`-object memory-backend-memfd,share=on`** — Required by virtiofs. +virtiofsd accesses guest memory via shared memory mapping. Without +`share=on`, virtiofsd cannot map guest memory. Only emitted when +virtiofs shares are defined. Not user-configurable. + +**`-object memory-backend-file` (NVMe PMR)** — Emitted once per +`nvme.drives` controller that sets `pmr`, before the `-device nvme` +loop so the `pmrdev=` link resolves (QEMU requires the `-object` to +precede the `-device` that names it). `share` defaults to `on`: a PMR +backed by a file is normally wanted for persistence, and `share=off` +maps the backing file `MAP_PRIVATE` so guest writes never reach disk. +`pmem=on` is emitted only when `share` is on: with `share=off` QEMU maps +`MAP_PRIVATE` and never requests `MAP_SYNC` (`util/mmap-alloc.c`), so +`pmem` is silently inert and QEMU issues no warning — the template +suppresses the no-op rather than emit a misleading flag. +`mem-path` defaults to a bare `nvme-pmr-<index>.img` filename, which +QEMU resolves against its CWD — the unit `WorkingDirectory=` coupled to +`StateDirectory=` (see vars.md "Exec directories") — so the region is a +per-VM state file that survives restart, the same convention NVMe +qcow2 images use. QEMU creates the file when absent and reopens it +otherwise (`file_ram_open` in `system/physmem.c`, `O_RDWR|O_CREAT`), so +no separate create step is needed. QEMU's NVMe code enforces a +power-of-2 size of at least 16 bytes (`hw/nvme/ctrl.c`, `is_power_of_2` +and the `< 16` check in `nvme_init_pmr`), and the memory backend +separately rejects a size below one host page (`file_ram_alloc` in +`system/physmem.c`), so the binding minimum in practice is one page +(4096 bytes on x86_64); the template passes `size` through and lets +QEMU reject bad values rather than duplicating the checks. PMR uses +BAR 4/5 and is +rejected with SR-IOV or an MSI-X exclusive BAR; CMB (BAR 2) and PMR +coexist. See: `<qemu_binary> -object memory-backend-file,help`, +`<qemu_binary> -device nvme,help`. + +**`LimitMEMLOCK=`** — Auto-computed as `<ram+256>M` when +`pci_passthrough` is defined. VFIO DMA mapping requires locked +memory. The 256M overhead covers QEMU allocations beyond guest RAM. +Override: + +```yaml +service: + LimitMEMLOCK: 8G +``` + +See: `man systemd.exec`. + +## virtiofsd + +**`--sandbox=`** — Set to `namespace` with `--uid-map :0:%U:1:` and +`--gid-map :0:%G:1:`. Provides PID, mount, network, and user +namespace isolation as an unprivileged user. The `--uid-map` maps +root inside the namespace to the service user outside (`%U`/`%G` are +systemd specifiers expanded at runtime). Without `--uid-map`, +virtiofsd warns "Couldn't set the process uid as root" because the +default 1-to-1 UID mapping does not include UID 0 (virtiofsd calls +`setresuid(0)` after creating the namespace). Override via per-share +env file: set `VIRTIOFSD_SANDBOX_ARGS=--sandbox=none`. See: +`/usr/libexec/virtiofsd --help`. + +**`--xattr`** — Extended attribute support. Required for correct +POSIX semantics (security labels, capabilities, ACLs). Not +user-configurable. See: `/usr/libexec/virtiofsd --help`. + +**`--no-announce-submounts`** — QEMU does not support submounts. +Prevents virtiofsd from announcing submount boundaries that the VMM +cannot handle. Not user-configurable. See: +`/usr/libexec/virtiofsd --help`. + +**`--fd=3`** — Socket activation. systemd passes the listening socket +as FD 3 per the `sd_listen_fds` protocol. Not user-configurable. +See: `man sd_listen_fds`. + +**`StopWhenUnneeded=yes`** — Set on `[email protected]`. Makes the +service auto-stop when no `qemu-system@<vm>.service` pins it. The +per-VM drop-in (`templates/qemu-system-override.conf.j2`) emits +`Requires=virtiofsd@%i-<tag>.service` for every share, which creates an +implicit reverse `RequiredBy=` dependency on the virtiofsd unit. +`RequiredBy` carries the `UNIT_ATOM_PINS_STOP_WHEN_UNNEEDED` atom +(`src/core/unit-dependency-atom.c:45`); the moment all pinning +dependencies go inactive, `unit_is_unneeded()` +(`src/core/unit.c:2179-2207`) returns true and systemd queues the +service for stop. The listening socket is not pinned by +`qemu-system@<vm>.service`'s `Requires=`, so it stays active across +stop/start cycles and re-activates virtiofsd on the next connection +from the QEMU process. The stop side is +symmetric with the start side: socket activation handles start, +`StopWhenUnneeded=` handles stop, with no explicit lifecycle plumbing +between QEMU and virtiofsd in either direction. Without this directive, +virtiofsd survives QEMU and keeps stale share-directory bindings alive, +requiring an explicit `systemctl stop` of each per-share service to +force re-binding on the next QEMU connection. See: `man systemd.unit`. + +**`Before=qemu-system@<vm>.service`** — Set on every per-instance +`virtiofsd@<vm>-<share>.service.d/override.conf` rendered from +`templates/virtiofsd-override.conf.j2`. Inverts the stop ordering +of the `Requires=virtiofsd@%i-<tag>.service` cascade emitted by the +per-VM `qemu-system-override.conf`. Without it, the stop +transaction `systemctl --user stop qemu-system@<vm>` enqueues both +`qemu-system@<vm>.service`'s stop and virtiofsd's +`RequiredBy=`-cascaded stop and runs them in parallel: +`qemu-system@<vm>.service`'s `ExecStop=ssh root@vsock/<cid> systemctl +poweroff` triggers a guest shutdown, but virtiofsd has already +torn down its vhost-user socket. The guest kernel logs +`virtio-fs: response too short (0)` on every outstanding fs +request, the unmount step in the guest's shutdown sequence hangs +in D-state on `/nix/store`, `/lib/modules`, and the data shares, +ACPI powerdown is never delivered, and `qemu-system@<vm>.service` +hits `TimeoutStopSec=2min`, after which systemd sends `SIGKILL` to +the QEMU process. Adding `Before=qemu-system@<vm>.service` on the +virtiofsd side reverses to `After=virtiofsd@<vm>-<share>.service` in +the stop direction (`man systemd.unit`: *"the inverse of the +start-up order is applied"*). The cascade still queues virtiofsd's stop in the +same transaction (`UNIT_ATOM_PROPAGATE_STOP`, +`src/core/unit-dependency-atom.c:43`), so failure-time +propagation is preserved; the ordering only changes when the +queued stop runs. Socket activation is unaffected because +`Before=` orders but does not pull starts; [email protected] +still starts only when the QEMU process connects to its socket. +See: `man systemd.unit`, `src/core/unit-dependency-atom.c:43` +(`UNIT_REQUIRED_BY` carries `UNIT_ATOM_PROPAGATE_STOP`). + +## virtiofsd dependency: `Requires=`, not `BindsTo=` + +The per-VM `qemu-system-override.conf` emits +`Requires=virtiofsd@%i-<tag>.service` for every share, paired with +`Requires=virtiofsd@%i-<tag>.socket` + `After=virtiofsd@%i-<tag> +.socket` for ordering. `BindsTo=` would be the textbook choice for +"consumer cannot run without producer" and is what +`systemd-networkd-persistent-storage.service` uses against +`systemd-networkd.service`, but `BindsTo=` bundles two atoms +(`unit-dependency-atom.c:31-35`) that we want individually but +not together: forward `UNIT_ATOM_CANNOT_BE_ACTIVE_WITHOUT` (the +death-link checked by `unit_is_bound_by_inactive()` at +`src/core/unit.c:2239`) and reverse `UNIT_ATOM_RETROACTIVE_STOP_ON_STOP` +(line 57-58, the crash-kill propagation). systemd does not ship a +primitive that emits the second without the first. + +The death-link is fatal in our setup because virtiofsd is a +vhost-user slave and self-exits when the QEMU process disconnects. +Any stop phase of `qemu-system@<vm>.service`, including the stop +half of a `JOB_RESTART`, drops virtiofsd to `inactive`. After the +stop phase, `JOB_RESTART` morphs in place to `JOB_START` +(`src/core/job.c:1027`) without re-running transaction construction +(`src/core/transaction.c:1055`), so the `UNIT_ATOM_PULL_IN_START` +walk that would queue a fresh `JOB_START` on virtiofsd never runs. +When `qemu-system@<vm>.service` reaches `UNIT_ACTIVE`, the +death-link evaluator finds virtiofsd in `(inactive, no job)` state +and queues `JOB_STOP` on it; the VM is killed two minutes later via +`TimeoutStopSec=2min`, mid-workload. +This made `systemctl restart qemu-system@<vm>` unusable with +`BindsTo=`. + +`Requires=` drops `CANNOT_BE_ACTIVE_WITHOUT` and keeps everything +else: pinning via `RequiredBy.PINS_STOP_WHEN_UNNEEDED` +(`unit-dependency-atom.c:45`), explicit-stop propagation via +`RequiredBy.PROPAGATE_STOP` (line 43), and clean stop ordering via +the existing `[email protected]` plus the per-instance +`Before=qemu-system@<vm>.service` drop-in +(`templates/virtiofsd-override.conf.j2`). `systemctl --user +restart qemu-system@<vm>` works natively: after morph to +`JOB_START`, `qemu-system@<vm>.service` starts and the QEMU process +connects to [email protected], which socket-activates a fresh [email protected]. Equivalent shipped +shape: `[email protected]` uses +`Requires=systemd-journald.socket` for the same restart-clean +dependency relationship. + +### Tradeoff + +`Requires=` does not propagate when the dep stops *unexpectedly* +(only on explicit stop). If `virtiofsd@<vm>-<tag>.service` crashes +mid-run -- non-zero exit, panic, OOM-kill -- QEMU is no longer +auto-stopped by systemd. The guest sees `virtio-fs: response too +short` errors, hangs on the affected mount, and the operator +notices via test failure or a stuck VM in `machinectl list`. +Manual recovery: `systemctl --user stop qemu-system@<vm>` then +`systemctl --user start qemu-system@<vm>`. virtiofsd crashes are +rare; we have observed none in production. + +### Reverting to crash-kill + +To re-acquire `RETROACTIVE_STOP_ON_STOP` automatic propagation +on virtiofsd failure, change the per-share line in +`templates/qemu-system-override.conf.j2` back to +`BindsTo=virtiofsd@%i-{{ share.tag }}.service`. That re-introduces +the `JOB_RESTART` death-link race documented above, so operators +must then revert to stop + `daemon-reload` + start for unit +re-rendering and never use `systemctl restart` directly. systemd +does not offer a primitive that combines both properties. + +## Cloud-init + +### Network configuration gap + +Debian generic and genericcloud images ship with an empty +`/etc/netplan/` directory. They depend on cloud-init's fallback +network detection to generate netplan YAML at first boot. The +fallback scans all physical NICs and generates DHCP config for the +first one. This works when cloud-init boots with the image's own +distro kernel and initramfs. + +With direct kernel boot and a custom initramfs, the fallback is +unreliable. The symptom: `/etc/netplan/` stays empty, +`systemd-networkd-wait-online.service` hangs, the guest has no +network connectivity. + +The fix: provide a `network-config` file in the seed ISO. The +NoCloud datasource processes `network-config` before the fallback, +so it works regardless of boot mode. `files/network-config` is a +static file (no template rendering needed). Pass it to +`cloud-localds`: + +```shell +cloud-localds --network-config=files/network-config \ + images/seed.iso /tmp/user-data /tmp/meta-data +``` + +The network config matches both predictable interface names (`en*`) +and classic names (`eth*`). Custom kernels without full PCI sysfs +support or with `net.ifnames=0` on the command line leave interfaces +as `eth0`. Both patterns are needed. + +Images that do NOT need this fix: +nocloud (ships `/etc/netplan/90-default.yaml` baked in), +mkosi (ships `/etc/systemd/network/80-dhcp.network` baked in), +imageless (debootstrap includes network config). + +See: https://cloudinit.readthedocs.io/en/latest/reference/network-config-format-v2.html + +### user-data + +Infrastructure requirements for the VM to be usable with the +service templates. Apply only when `cloud_init` is defined. + +**`disable_root`**, **`PermitRootLogin`**, +**`PasswordAuthentication`** — Set to `false`, `yes`, `yes`. Root SSH +access required for non-interactive VM management (`ExecStop=` SSH +shutdown, automated provisioning). Not user-configurable. + +**`locale`** — The `locale:` directive uses cloud-init's built-in +locale module, which handles per-distro differences automatically. +Debian: writes `/etc/locale.gen` and runs `locale-gen`. Fedora: +writes `/etc/locale.conf`. No distro-specific `write_files` or +`packages` needed. + +**`growpart`** and **`resizefs`** — Cloud-init's built-in `growpart` +and `resizefs` modules run automatically (enabled in +`cloud_init_modules` by default on both Debian and Fedora). They +detect the root partition from the mount table and resize it to fill +the disk. No `runcmd` needed. Verified against upstream Fedora Cloud +image `/etc/cloud/cloud.cfg` which lists both modules. + +**`runcmd`** — Runs `touch /etc/cloud/cloud-init.disabled` only. +Disables cloud-init after first run to prevent re-provisioning on +reboot. Not user-configurable. See: `man cloud-init`. + +**`ssh_pwauth`** — Set to `true`. Password authentication fallback +for console access when no SSH key is configured. Not +user-configurable. + +## 9P + +**`security_model=`** — Set to `none`. Passes through file +permissions without UID/GID mapping. Correct for sharing host +directories where the user already owns the files. Other models +(mapped-xattr, passthrough) require root or change on-disk xattrs. +Not user-configurable. See: `man qemu-system`. + +**`multidevs=`** — Set to `remap`. Remaps device/inode numbers for +filesystems spanning multiple host devices. Prevents inode collisions +when sharing directories that contain mount points. Not +user-configurable. See: `man qemu-system`. diff --git a/docs/requirements.md b/docs/requirements.md new file mode 100644 index 00000000..74710093 --- /dev/null +++ b/docs/requirements.md @@ -0,0 +1,56 @@ +# Requirements + +## Debian + +```shell +sudo apt install qemu-system-x86 qemu-utils socat \ + systemd-container virtiofsd cloud-image-utils ovmf cargo +cargo install minijinja-cli +``` + +## Fedora + +```shell +sudo dnf install qemu-system-x86-core qemu-img socat \ + systemd-container virtiofsd cloud-utils-cloud-localds \ + edk2-ovmf cargo +cargo install minijinja-cli +``` + +## openSUSE + +```shell +sudo zypper install qemu-x86 qemu-img socat \ + systemd-container virtiofsd qemu-ovmf-x86_64 cargo +cargo install minijinja-cli +``` + +`cloud-localds` is not packaged on openSUSE. Use `genisoimage` +directly to create seed ISOs for cloud-init boot. + +## NixOS + +```nix +environment.systemPackages = with pkgs; [ + qemu socat virtiofsd minijinja cloud-utils OVMF +]; +``` + +## Groups + +```shell +sudo usermod --append --groups kvm,systemd-journal $(whoami) +``` + +`kvm` is required for `-accel kvm` (`/dev/kvm` access). +`systemd-journal` allows reading the journal without sudo. +Log out and back in for group changes to take effect. + +## Verify + +```shell +qemu-system-x86_64 --version +minijinja-cli --version +id --name --groups | grep kvm +ls -l --all /dev/kvm +``` diff --git a/docs/transient-units.md b/docs/transient-units.md new file mode 100644 index 00000000..f24c9353 --- /dev/null +++ b/docs/transient-units.md @@ -0,0 +1,244 @@ +# Transient and persistent units for QEMU VMs + +## Two patterns in systemd + +systemd provides two mechanisms for managing processes. Both vmspawn +and nspawn use both. + +### Persistent template service (on disk) + +A unit file lives on disk. systemd forks the process, manages its +lifecycle, resolves dependencies, captures logs. + +``` +units/user/[email protected]: +ExecStart=systemd-vmspawn --quiet --keep-unit --register=yes --network-tap --machine=%i +``` + +``` +units/[email protected]: +ExecStart=systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth -U --settings=override --machine=%i +``` + +Source: `systemd/units/user/[email protected]`, +`systemd/units/[email protected]`. + +### Transient scope (programmatic, in memory) + +The process starts first, then gets placed into a transient scope +via the `StartTransientUnit` D-Bus call. No file on disk. The scope +exists only while the process runs. + +```c +/* vmspawn-scope.c:57 */ +/* Creates a transient scope unit which tracks the lifetime of the current process */ +r = bus_message_new_method_call(bus, &m, bus_systemd_mgr, "StartTransientUnit"); +``` + +Source: `systemd/src/vmspawn/vmspawn-scope.c:57-67`, +`systemd/src/nspawn/nspawn.c:5634`. + +### The --keep-unit bridge + +When vmspawn or nspawn runs inside its service template, `--keep-unit` +tells it: "you are already in a service unit, do not create a scope on +top of it." When run directly from the terminal (without the service), +they create a transient scope. + +This is how both patterns coexist. The service template is the managed +path. The transient scope is the ad-hoc path. + +## systemd-run + +`systemd-run` is the CLI for creating transient units. It calls +`StartTransientUnit` on the D-Bus manager interface. + +Source: `systemd/man/systemd-run.xml`, "Run programs in transient +scope units, service units, or path-, socket-, or timer-triggered +service units." + +### Using our infrastructure with systemd-run + +A transient unit can depend on persistent units via `--property=`. +The deployed virtiofsd sockets, vfio-bind services, and +`EnvironmentFile=` paths are all reusable. + +Verified empirically. Transient unit referencing persistent socket: + +```shell +$ systemd-run --user --collect --no-block \ + --unit=transient-test \ + [email protected] \ + [email protected] \ + --property=Slice=machine.slice \ + --property=LimitMEMLOCK=2304M \ + sleep 30 + +$ systemctl --user show transient-test.service \ + | grep -E '^(Requires|After|Slice|LimitMEMLOCK)=' +Requires=machine.slice [email protected] basic.target +After=machine.slice [email protected] -.mount basic.target +Slice=machine.slice +LimitMEMLOCK=2415919104 +``` + +EnvironmentFile loading also works. `$QEMU_ARGS` from the rendered +env file is available inside the transient service: + +```shell +$ systemd-run --user --pty --collect \ + --property=EnvironmentFile=%E/systemd/qemu-system/debian.env \ + sh -c 'echo "QEMU_ARGS=$QEMU_ARGS" | head -c 200' +QEMU_ARGS= -machine type=q35 -accel kvm -cpu host -m 2048 ... +``` + +### Full interactive VM with transient-run template + +The `transient-run.sh.j2` template renders a self-contained script +that combines `--pty` for interactive console, `--property=` for +dependencies and `EnvironmentFile=`, and `sh -c` for shell word +splitting of `$QEMU_ARGS`: + +```shell +minijinja-cli --trim-blocks \ + --output /tmp/run-test.sh \ + templates/transient-run.sh.j2 vars/test.yaml +bash /tmp/run-test.sh +``` + +The script stops the persistent service if running (port conflicts), +sets up `QEMU_EXTRA_ARGS` with resolved paths, and calls +`systemd-run --user --pty` with all virtiofsd dependencies. + +Disconnect with `^]` pressed three times within 1 second. + +Source: `systemd/src/shared/ptyfwd.c:211-230`, `look_for_escape()` +checks for `0x1D` (Ctrl-]) three times within `ESCAPE_USEC` +(1 second). + +### Pitfalls discovered empirically + +**Do not pipe through bash.** `minijinja-cli ... | bash` steals +stdin from the terminal. `systemd-run --pty` needs the terminal's +stdin for interactive console forwarding. Always render to a file +first, then run with `bash /tmp/run-test.sh`. + +**systemd specifiers do not expand in transient Environment=.** +`%t` and `%i` expand in persistent unit files but are passed as +literal strings in `--property=Environment=` for transient units. +The template resolves paths at render time using `$XDG_RUNTIME_DIR` +(shell-expanded before `systemd-run` is called) and passes the +result via `--property=PassEnvironment=QEMU_EXTRA_ARGS`. + +Verified empirically: + +```shell +$ systemd-run --user --collect --no-block --unit=specifier-test \ + --property='Environment=TEST=%t/foo' \ + sh -c 'echo $TEST' +$ journalctl --user-unit=specifier-test --output=cat +%t/foo +``` + +**KERNEL_ARGS requires eval.** The rendered env file contains +`-append "console=ttyS0,115200 init=..."` with embedded quotes. +systemd's `EnvironmentFile=` parser preserves quotes as literal +characters. Without `eval`, the shell passes `"console=...` and +`init=..."` as separate arguments to QEMU. The template uses +`eval exec $QEMU_BINARY ...` to process the quotes correctly. + +### Why sh -c is required + +systemd-run passes command arguments as separate argv entries. +`$QEMU_ARGS` in the `EnvironmentFile=` is a single string with spaces. +Without `sh -c`, systemd would treat `$QEMU_ARGS` as one token. With +`sh -c`, the shell performs word splitting. + +In persistent services, `ExecStart=` handles this natively. systemd's +own command parser expands `$QEMU_ARGS` with word splitting. This is +specific to unit file parsing (`load-fragment.c`), not to the D-Bus +`StartTransientUnit` interface. + +## --pty and persistent services + +`--pty` is a `systemd-run` feature. It opens a PTY master +(`openpt_allocate` in `run.c:2476`), passes the slave FD as +`StandardInputFileDescriptor` / `StandardOutputFileDescriptor` to the +transient unit (`run.c:1501-1506`), and runs `PTYForward` to forward +between the local terminal and the service. + +Source: `systemd/src/run/run.c:1419-1506`, `systemd/src/shared/ptyfwd.c`. + +Persistent services have no controlling terminal by design. There is +no mechanism to retroactively attach a PTY to a running service. +`machinectl login` and `machinectl shell` work for containers (nspawn) +but not for VMs, since the guest OS is fully isolated. + +Source: `systemd/man/machinectl.xml`, login is "only supported for +containers running systemd as init system." + +### Interactive console for persistent VMs (virtconsole) + +The override template adds a virtio console on a unix socket to +every persistent VM. ttyS0 stays on stdio (captured by the journal). +hvc0 is a virtconsole on a socket for interactive access. + +QEMU flags (generated by the override template): + +``` +-device virtio-serial-pci +-chardev socket,id=console0,path=%t/qemu-system/%i/console.sock,server=on,wait=off +-device virtconsole,chardev=console0 +``` + +Connect: + +```shell +socat -,raw,echo=0,escape=0x1d \ + UNIX-CONNECT:$XDG_RUNTIME_DIR/qemu-system/test/console.sock +``` + +Disconnect with `Ctrl-]` (single press, `escape=0x1d`). The VM +keeps running. Reconnect at any time. + +Guest requirements: +- `console=hvc0` on the kernel command line (in addition to + `console=ttyS0`) +- A getty on hvc0 (e.g., `systemd.services."getty@hvc0"` in NixOS) +- `CONFIG_VIRTIO_CONSOLE=y` or `=m` in the guest kernel + +Source: QEMU `qemu-options.hx`, libvirt `virsh console` uses the +same virtio-serial + virtconsole pattern. + +Limitations: +- hvc0 only works after the guest kernel loads. BIOS output and + early boot messages are on ttyS0 (journal only). +- The initramfs emergency shell is on ttyS0, not hvc0. Use the + transient-run template (approach 3) for initramfs debugging. +- QEMU monitor (`Ctrl-a c`) is on ttyS0, not the virtconsole. + +## When to use each pattern + +| Use case | Mechanism | +|---|---| +| Long-lived, managed VMs | Persistent template (`systemctl start qemu-system@test`) | +| Console access to running VMs | `socat` to virtconsole socket (hvc0) | +| Interactive debugging (initramfs) | `transient-run.sh.j2` with `systemd-run --pty` (ttyS0) | +| CI / ephemeral VMs | `systemd-run --user --collect` (no `--pty`) | +| Quick test, no env file | `systemd-run --user --pty /usr/bin/qemu-system-x86_64 -m 2048 ...` | + +Both patterns place VMs in `machine.slice`, both can register with +machined, both reuse the deployed virtiofsd sockets and vfio-bind +services. The infrastructure is shared. + +## Proxmox comparison + +Proxmox uses transient scopes exclusively. The `qm start` command +forks, calls `enter_systemd_scope()` via D-Bus +`StartTransientUnit()`, places the QEMU process in `$VMID.scope` +under `qemu.slice` (not `machine.slice`). No persistent service +files. System scope only (root). + +Source: `git.proxmox.com/pve-common.git`, `src/PVE/Systemd.pm`, +`enter_systemd_scope()`. `git.proxmox.com/qemu-server.git` — +`PVE/QemuServer.pm`. diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 00000000..a03c09fb --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,461 @@ +# Usage + +All templates render with `minijinja-cli --trim-blocks` and one +vars file. Output goes to stdout by default; use `--output <path>` +to write directly to the deploy location. + +`vars/example.yaml` is the starting point and ships with the repo. +Copy it to create per-VM configurations. The filename should match +`vm_name` inside the file (e.g. `vars/test.yaml` for `vm_name: test`, +`vars/dev.yaml` for `vm_name: dev`). User vars files are gitignored. + +The examples below use `vars/test.yaml` (a copy of `vars/example.yaml` +which has `vm_name: test`). User scope deploys to `~/.config/systemd/`. + +## qemu-system + +Service template, per-instance drop-in, `EnvironmentFile=`, and QMP +powerdown commands. + +```shell +# Service template (deploy once) +mkdir --mode=0755 --parents ~/.config/systemd/user +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] \ + vars/test.yaml + +# Per-instance drop-in (one per VM) +mkdir --mode=0755 --parents ~/.config/systemd/user/[email protected] +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected]/override.conf \ + templates/qemu-system-override.conf.j2 \ + vars/test.yaml + +# QEMU environment file (one per VM) +mkdir --mode=0755 --parents ~/.config/systemd/qemu-system +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/qemu-system/test.env \ + templates/vm.env.j2 \ + vars/test.yaml + +# QMP powerdown commands (static file, deploy once) +cp files/qmp-powerdown ~/.config/systemd/qemu-system/qmp-powerdown +``` + +## virtiofsd + +Socket unit, socket-activated service, and per-share `EnvironmentFile=` +for host-guest directory sharing. + +```shell +# Socket and service templates (deploy once) +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] \ + vars/test.yaml + +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] \ + vars/test.yaml + +# Per-share environment (one per non-home share) +mkdir --mode=0755 --parents ~/.config/systemd/virtiofsd +minijinja-cli --trim-blocks \ + --define share_tag=modules \ + --output ~/.config/systemd/virtiofsd/test-modules.env \ + templates/virtiofsd.env.j2 \ + vars/test.yaml + +# Per-instance stop-ordering drop-in (one per (vm, share) tuple) +mkdir --mode=0755 --parents ~/.config/systemd/user/[email protected] +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected]/override.conf \ + templates/virtiofsd-override.conf.j2 \ + vars/test.yaml +``` + +The per-instance drop-in adds `Before=qemu-system@<vm>.service` so +virtiofsd outlives `qemu-system@<vm>.service`'s `ExecStop=` graceful +shutdown. Without it the `Requires=virtiofsd@%i-<tag>.service` +cascade in the `qemu-system-override.conf` drop-in tears virtiofsd +down concurrently with `ExecStop=`, the guest hangs on missing +virtiofs responses, and `qemu-system@<vm>.service` hits +`TimeoutStopSec=` -> `SIGKILL`. +See: design-decisions.md "virtiofsd" section. + +## vfio + +`Type=oneshot` service for device binding and udev rules for unprivileged +access. One-time sudo required for module config, udev rules, and +rule reload. After setup, all VFIO operations run in user mode. + +```shell +# Bind service (deploy once) +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] \ + vars/test.yaml + +# One-time root setup +sudo cp files/vfio-pci.conf /etc/modules-load.d/vfio-pci.conf +sudo modprobe vfio-pci + +minijinja-cli --trim-blocks \ + templates/vfio-udev.rules.j2 \ + vars/test.yaml \ + | sudo tee /etc/udev/rules.d/10-vfio-kvm.rules + +sudo udevadm control --reload-rules +sudo udevadm trigger --subsystem-match=pci +``` + +## cloud-init + +user-data, meta-data, and network-config rendered into a seed ISO for +first-boot provisioning. Only needed for cloud image boot. + +The network-config is required when using direct kernel boot with +generic cloud images (cloud-init's fallback network detection is +unreliable without the distro initramfs). It is optional for cloud +image boot with the distro kernel. See +[design-decisions.md](design-decisions.md) for details. + +```shell +minijinja-cli --trim-blocks \ + --output /tmp/user-data \ + templates/user-data.j2 \ + vars/test.yaml + +minijinja-cli --trim-blocks \ + --output /tmp/meta-data \ + templates/meta-data.j2 \ + vars/test.yaml + +cloud-localds --network-config=files/network-config \ + images/seed.iso /tmp/user-data /tmp/meta-data +``` + +## Start + +```shell +systemctl --user daemon-reload +systemctl --user start qemu-system@test +systemctl --user status qemu-system@test +``` + +## machinectl + +VMs registered with systemd-machined appear alongside containers +in `machinectl`. Registration happens automatically via an +`ExecStartPost=` `varlinkctl` call to `io.systemd.Machine.Register` +in the service template. User scope requires `--user` (systemd +v259+). + +```shell +# List all registered machines +machinectl --user list + +# Machine details (PID, cgroup, service) +machinectl --user status test + +# Machine properties (parseable output) +machinectl --user show test + +# Emergency kill (not graceful, kills the QEMU process immediately) +machinectl --user terminate test +``` + +`machinectl login`, `shell`, `poweroff`, `bind`, and `copy-to/from` +are container-only. See: `man machinectl`, `man systemd-machined`. + +## Console + +Every persistent VM has a virtio console on a unix socket for +interactive access. The serial console (ttyS0) continues to capture +output in the systemd journal. + +```shell +socat -,raw,echo=0,escape=0x1d \ + UNIX-CONNECT:$XDG_RUNTIME_DIR/qemu-system/test/console.sock +``` + +Press Enter to get the login prompt. Disconnect with `Ctrl-]`. +The VM keeps running. Reconnect at any time. + +The guest needs `console=hvc0` on the kernel command line and a +getty on hvc0. The guest kernel needs `CONFIG_VIRTIO_CONSOLE=y` +or `=m`. + +For initramfs debugging (before hvc0 is available), use the +transient-run template: + +```shell +minijinja-cli --trim-blocks \ + --output /tmp/run-test.sh \ + templates/transient-run.sh.j2 vars/test.yaml +bash /tmp/run-test.sh +``` + +See [transient-units.md](transient-units.md) for details and +limitations. + +## Logs + +```shell +# VM service log +journalctl [email protected] + +# virtiofsd log (per share) +journalctl [email protected] + +# Follow live output +journalctl [email protected] --follow + +# All VM-related units since last boot +journalctl --user-unit='qemu-system@*' --user-unit='virtiofsd@*' --boot + +# Follow every VM with a compact line prefix +journalctl --user-unit='qemu-system@*.service' --no-hostname --follow + +# Follow every VM with the full unit path (debugging dependencies) +journalctl --user-unit='qemu-system@*.service' --output=with-unit --follow +``` + +The default `short` output prefixes each line with the syslog +identifier. The `[email protected]` template sets +`SyslogIdentifier=qemu-system@%i`, so the prefix carries both the +service template and the instance name +(`qemu-system@<vm>[PID]:`) without any extra journalctl flag. +Pair with `--no-hostname` for the most compact form. Manager +messages (`systemd[PID]:`) are emitted by systemd itself and keep +their default prefix. See `design-decisions.md` for why this +template overrides `SyslogIdentifier=` while +`[email protected]` and `[email protected]` do not. + +`--output=with-unit` adds the user manager unit and the full +`.service` suffix (`[email protected]/qemu-system@<vm>.service[PID]:`), +useful when debugging dependency or scope issues but verbose for +day-to-day tailing. `--output=cat` strips the prefix entirely for +piping. `--output=json` emits one JSON record per line for machine +parsing. See: `man journalctl`. + +## Stop + +Graceful shutdown via `ExecStop=`: QMP `system_powerdown` triggers +ACPI shutdown inside the guest. When `ssh_private_key` and +`vsock_cid` are configured, SSH shutdown is attempted first with +QMP as fallback. If the guest does not shut down within +`TimeoutSec=` (default 2min), systemd sends `SIGKILL`. + +```shell +# Graceful stop (ExecStop QMP powerdown, then SIGKILL after timeout) +systemctl --user stop qemu-system@test + +# Force kill (immediate SIGKILL, no graceful shutdown) +systemctl --user kill qemu-system@test --signal=SIGKILL + +# Stop all VMs (PartOf=machines.target) +systemctl --user stop machines.target + +# Reset failed state after a crash or forced kill +systemctl --user reset-failed qemu-system@test +``` + +## Multiple VMs + +The `[email protected]` template supports multiple instances. +Each VM gets its own vars file, env file, qemu-system drop-in, and +one virtiofsd drop-in per share for stop ordering. The service +templates and the virtiofsd socket template are shared. + +`vm_name`, `ssh_port`, and `vsock_cid` must be unique per VM. + +```shell +cp vars/example.yaml vars/dev.yaml +``` + +Edit `vars/dev.yaml`. E.g. set `vm_name: dev`, `ssh_port: 10023`, +`vsock_cid: 101`. + +```shell +# Render test VM +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/qemu-system/test.env \ + templates/vm.env.j2 vars/test.yaml +mkdir --mode=0755 --parents ~/.config/systemd/user/[email protected] +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected]/override.conf \ + templates/qemu-system-override.conf.j2 vars/test.yaml + +# Render dev VM +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/qemu-system/dev.env \ + templates/vm.env.j2 vars/dev.yaml +mkdir --mode=0755 --parents ~/.config/systemd/user/[email protected] +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected]/override.conf \ + templates/qemu-system-override.conf.j2 vars/dev.yaml + +systemctl --user daemon-reload +systemctl --user start qemu-system@test qemu-system@dev +machinectl --user list +``` + +A/B kernel testing: two vars files with different `kernel.image` +paths. Both VMs share the same service template. Only the env files +differ. + +## Cleanup + +### Remove a single VM + +Stop the VM and remove its per-instance files. Shared templates +and virtiofsd units stay for other VMs. + +```shell +systemctl --user stop qemu-system@test +rm ~/.config/systemd/qemu-system/test.env +rm --recursive --force ~/.config/systemd/user/[email protected] +rm --recursive --force ~/.config/systemd/user/virtiofsd@test-*.service.d +rm --recursive --force ~/.config/systemd/virtiofsd/test-*.env +systemctl --user daemon-reload +``` + +### Remove everything + +Stop all VMs, remove all deployed units and configuration. + +`virtiofsd@<vm>-<tag>.service` instances auto-stop when no +`qemu-system@<vm>.service` pins them (via `StopWhenUnneeded=yes` on +`[email protected]`, pinned by the drop-in's `Requires=`). The +listening sockets are not pinned by `qemu-system@<vm>.service`, so +they keep socket-activating new virtiofsd processes until stopped +explicitly: + +```shell +systemctl --user stop machines.target +systemctl --user stop 'virtiofsd@*.socket' +rm --recursive --force \ + ~/.config/systemd/user/[email protected] \ + ~/.config/systemd/user/qemu-system@*.service.d \ + ~/.config/systemd/user/[email protected] \ + ~/.config/systemd/user/[email protected] \ + ~/.config/systemd/user/virtiofsd@*.service.d \ + ~/.config/systemd/user/[email protected] \ + ~/.config/systemd/qemu-system \ + ~/.config/systemd/virtiofsd +systemctl --user daemon-reload +``` + +### Upgrading templates + +After re-deploying `[email protected]` with changed sandbox +settings, restart running virtiofsd sockets. `daemon-reload` +reloads unit definitions but does not restart running sockets. +Stale sockets may pass FDs in a state incompatible with the new +service configuration. + +```shell +systemctl --user daemon-reload +systemctl --user restart 'virtiofsd@*.socket' +``` + +### Updating a VM's unit definition + +When `vm.env` or the per-VM drop-in +(`qemu-system@<vm>.service.d/override.conf`) changes -- new +kernel path, NVMe knobs, share list, anything -- re-render, +`daemon-reload`, then `restart`: + +```shell +# re-render override.conf and vm.env at this point +systemctl --user daemon-reload +systemctl --user restart qemu-system@test +``` + +Native `systemctl restart` works because the +`qemu-system-override.conf` drop-in uses +`Requires=virtiofsd@%i-<tag>.service`, not `BindsTo=`. The +former does not emit the `UNIT_ATOM_CANNOT_BE_ACTIVE_WITHOUT` +death-link, so `JOB_RESTART`'s in-place morph to `JOB_START` +(after the stop phase has dropped virtiofsd to `inactive` via +vhost-user disconnect) is not aborted on the +`qemu-system@<vm>.service` side. The service starts, the QEMU +process connects to `[email protected]`, and socket activation +spawns a fresh virtiofsd. The tradeoff is loss of automatic +crash-kill propagation when virtiofsd fails unexpectedly; see +`docs/design-decisions.md` (`virtiofsd dependency`). + +If you switch the dep back to `BindsTo=` to re-acquire crash-kill, +this workflow no longer holds: use stop + `daemon-reload` + start +instead of `restart`. `BindsTo=` + `restart` races against the +death-link and the QEMU process is `SIGKILL`'d after +`TimeoutStopSec=` two minutes into start. + +## Deploy all + +Set `VARS` to your vars file. `VM` is derived from `vm_name` +inside it. + +```shell +VARS=vars/test.yaml +VM=$(minijinja-cli --template '{{ vm_name }}' '' $VARS) + +# Shared templates (deploy once) +mkdir --mode=0755 --parents ~/.config/systemd/user ~/.config/systemd/qemu-system ~/.config/systemd/virtiofsd +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] $VARS +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] $VARS +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] $VARS +cp files/qmp-powerdown ~/.config/systemd/qemu-system/qmp-powerdown + +# Per-VM files +mkdir --mode=0755 --parents ~/.config/systemd/user/qemu-system@${VM}.service.d +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/qemu-system/${VM}.env \ + templates/vm.env.j2 $VARS +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/qemu-system@${VM}.service.d/override.conf \ + templates/qemu-system-override.conf.j2 $VARS + +# Per-share virtiofsd env (only for shares with dir: set) +# Repeat for each non-home share tag: +# minijinja-cli --trim-blocks \ +# --define share_tag=modules \ +# --output ~/.config/systemd/virtiofsd/${VM}-modules.env \ +# templates/virtiofsd.env.j2 $VARS + +systemctl --user daemon-reload +systemctl --user start qemu-system@${VM} +``` + +For multiple VMs, run the per-VM block for each vars file: + +```shell +for VARS in vars/test.yaml vars/dev.yaml; do + VM=$(minijinja-cli --template '{{ vm_name }}' '' $VARS) + minijinja-cli --trim-blocks \ + --output ~/.config/systemd/qemu-system/${VM}.env \ + templates/vm.env.j2 $VARS + mkdir --mode=0755 --parents ~/.config/systemd/user/qemu-system@${VM}.service.d + minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/qemu-system@${VM}.service.d/override.conf \ + templates/qemu-system-override.conf.j2 $VARS + # Per-share virtiofsd env (only for shares with dir: set) + # minijinja-cli --trim-blocks --define share_tag=modules \ + # --output ~/.config/systemd/virtiofsd/${VM}-modules.env \ + # templates/virtiofsd.env.j2 $VARS +done +systemctl --user daemon-reload +for VARS in vars/test.yaml vars/dev.yaml; do + VM=$(minijinja-cli --template '{{ vm_name }}' '' $VARS) + systemctl --user start qemu-system@${VM} +done +``` diff --git a/docs/vars.md b/docs/vars.md new file mode 100644 index 00000000..585fe214 --- /dev/null +++ b/docs/vars.md @@ -0,0 +1,541 @@ +# Variable reference + +One vars file per VM. All templates render from the same file. + +Variable names match upstream terminology when a 1:1 mapping exists +(e.g. `cpu` → `-cpu`, `accel` → `-accel`). When a variable maps to +a parameter inside a flag, the parameter name is used (e.g. `cpus` +→ `-smp cpus=N`). When a variable controls a higher-level concept +that spans multiple flags, a descriptive name is used (e.g. +`ssh_port`, `pci_passthrough`). See [design-decisions.md](design-decisions.md) for +naming rationale. + +Relative paths resolve against `WorkingDirectory=`. The template sets +`WorkingDirectory=%S/qemu-system/%i`, which resolves to +`$XDG_STATE_HOME/qemu-system/<vm>` for user services and +`/var/lib/qemu-system/<vm>` for system services. That directory is +auto-created and owned by the service via `StateDirectory=qemu-system/%i`, +so per-VM state files like NVMe qcow2 images can be referenced with +bare filenames in vars (for example `image: disk.qcow2`) and land in a +predictable, per-unit location. See [exec directories](#exec-directories) +below for when to pick `StateDirectory=` vs the five sibling directives, +and for override semantics. + +## VM identity + +**`vm_name`** — VM instance name. Used as the systemd template +instance specifier (`qemu-system@<vm_name>`), the machined +registration name, and the cloud-init `instance-id` / +`local-hostname`. Required. + +**`service_scope`** — `user` or `system`. Determines deploy paths +and unit behavior. User scope deploys to `~/.config/systemd/user/`, +needs no sudo. System scope deploys to `/etc/systemd/system/`, +requires root. Default: `user`. + +**`uuid`** — RFC 4122 UUID string for the VM. Optional. When set, +maps to `-uuid` (QEMU SMBIOS / DMI identity, surfaces inside the +guest as `dmidecode -s system-uuid` and `/sys/class/dmi/id/product_uuid`) +and to the `id` field of the `io.systemd.Machine.Register` varlink +call (surfaces on the host as `Id=` in `machinectl show <vm_name>`). +Pairing the two correlates the same VM across QEMU's identification +table and machined's record. When omitted, QEMU passes through its +default (all-zeros UUID) and the Register call omits `id` entirely. +See: `man qemu-system`, `src/machine/machine-varlink.c` Register +dispatch table. + +## QEMU + +**`qemu_binary`** — Absolute path to the QEMU system emulator +binary. Required. The name encodes the guest architecture +(`qemu-system-x86_64` for x86_64 guests, `qemu-system-aarch64` for +ARM64). Works on both Intel and AMD hosts; the name is the guest +architecture, not the host CPU vendor. Distros ship symlinks (`kvm`, +`qemu-system-amd64`, `qemu-kvm`) that all point to the same binary. +Use the upstream name. Maps to the `ExecStart=` binary path. +Example: `/usr/bin/qemu-system-x86_64`. + +**`cpu`** — CPU model. Maps to `-cpu`. Default: `host` (passthrough +host CPU features to guest). Common values: `host`, `max` (all +emulatable features, useful with TCG). List available models with +`<qemu_binary> -cpu help`. See: `man qemu-system`. + +**`accel`** — Accelerator. Maps to `-accel`. Default: `kvm`. Common +values: `kvm` (hardware virtualization), `tcg` (software emulation), +`hvf` (macOS Hypervisor.framework). When `iommu` is `intel-iommu` or +`amd-iommu`, the template auto-appends `kernel-irqchip=split` to +`-accel` (required for interrupt remapping). See: `man qemu-system`. + +**`ram`** — Guest memory in megabytes. Integer. Maps to `-m`. Also +used for `-object memory-backend-memfd,size=<ram>M` (virtiofs shared +memory) and `LimitMEMLOCK=<ram+256>M` (VFIO DMA mapping overhead, +see systemd.exec(5)). QEMU default: 128. Suffixes M, G accepted by +QEMU but the template passes the raw integer to `-m` which defaults +to megabytes. See: `man qemu-system`. + +**`cpus`** — Virtual CPU count. Integer. Maps to `-smp`. Default: 1. +See: `man qemu-system`. + +**`machine_type`** — QEMU machine type. Maps to `-machine type=`. +Common values: `q35` (x86_64, modern PCIe), `pc` (x86_64, legacy +ISA/PCI), `virt` (aarch64), `microvm` (x86_64, minimal). `q35` is +required for x86 IOMMUs and recommended for modern x86 guests. + +Machine properties can be appended after the type name: +`microvm,pcie=on,rtc=on`. QEMU parses these as machine options. + +microvm uses virtio-mmio transport instead of PCI. The template +auto-detects microvm from the machine_type value and switches to +`*-device` suffix (MMIO) for all virtio devices. NVMe, VFIO, and +IOMMU are PCI-only and require `pcie=on` on microvm. Without +`pcie=on`, these features are silently omitted. + +List available types with `<qemu_binary> -machine help`. Required. + +**`iommu`** — IOMMU device type. Maps to `-device <iommu>`. + +`intel-iommu`: Intel VT-d. Requires `-machine q35`. Auto-sets +`kernel-irqchip=split` on `-accel` (required for intremap) and +`caching-mode=on` so the guest's IOMMU page-table updates are +trapped into QEMU's DMA path. That is needed both for VFIO +passthrough and for a guest-side userspace driver (VFIO/SPDK/DPDK) +that maps an emulated device for DMA; without it the guest IOMMU +only does interrupt remapping and DPDK fails with "failed to select +IOMMU type". + +`amd-iommu`: AMD-Vi. Requires `-machine q35`. Auto-sets +`kernel-irqchip=split` and `dma-remap=on` (same reason as +intel-iommu's `caching-mode=on`). + +`virtio-iommu-pci`: virtio IOMMU. Works with `-machine q35` +(x86_64) and `-machine virt` (aarch64). No kernel-irqchip +requirement. + +`arm-smmuv3`: ARM SMMUv3. Requires `-machine virt` (aarch64 only). +No kernel-irqchip requirement. + +See: `man qemu-system`. + +**`firmware`** — Dict. UEFI firmware via pflash drives. Omit for +BIOS boot (SeaBIOS default on x86). Maps to two +`-drive if=pflash` entries. Supported on q35, i440fx, and ARM virt. +microvm does not support pflash. See: +[design-decisions.md](design-decisions.md) (QEMU firmware). + +**`firmware.code`** — Path to the shared firmware executable. +Read-only at runtime. Maps to +`-drive if=pflash,format=raw,readonly=on,file=`. x86_64: +`/usr/share/OVMF/OVMF_CODE_4M.fd` (`ovmf`). aarch64: +`/usr/share/AAVMF/AAVMF_CODE.fd` (`qemu-efi-aarch64`). +See: `man qemu-system`. + +**`firmware.vars`** — Path to the per-VM NVRAM file. Writable; each +VM needs its own file for independent UEFI variable state. Raw copy +or qcow2 overlay of `/usr/share/OVMF/OVMF_VARS_4M.fd`. Maps to +`-drive if=pflash,format=<vars_format>,file=`. See: +[design-decisions.md](design-decisions.md) (QEMU firmware). + +**`firmware.vars_format`** — NVRAM file format. Default: `raw`. Set +to `qcow2` when using a qcow2 overlay. Maps to `-drive format=`. +See: `man qemu-system`. + +**`gdb`** — Boolean. Maps to `-gdb unix:<socket>`. The socket is +placed in the systemd `RuntimeDirectory=`. Connect with: +`gdb vmlinux -ex "target remote <socket>"`. + +**`autostart`** — Boolean. Default: `true`. When `false`, maps to +`-S` (QEMU starts paused, waiting for GDB continue). + +## Root disk + +**`image`** — Dict. Root disk image configuration. Omit for imageless +boot (virtiofs rootfs). Maps to `-drive`. + +**`image.file`** — Path to the disk image. Relative paths resolve +against `WorkingDirectory=`. Required when `image` is defined. Maps +to `-drive file=`. See: `man qemu-system`, `man qemu-img`. + +**`image.format`** — Disk image container format. Default: `raw`. +Common values: `qcow2`, `raw`. QEMU can auto-detect but explicit is +safer. Maps to `-drive format=`. See: `man qemu-system`. + +**`image.cache`** — Block cache mode. Maps to `-drive cache=`. +Values: `writeback` (default), `none` (direct I/O, data integrity), +`writethrough`, `directsync`, `unsafe`. See: `man qemu-system`. + +**`image.aio`** — Asynchronous I/O mode. Maps to `-drive aio=`. +Values: `threads` (default), `native` (Linux AIO, requires +`cache=none` or `cache=directsync`), `io_uring`. +See: `man qemu-system`. + +**`image.discard`** — Discard (TRIM/UNMAP) support. Maps to +`-drive discard=`. Values: `ignore` (default), `unmap`. +See: `man qemu-system`. + +**`image.detect-zeroes`** — Zero write optimization. Maps to +`-drive detect-zeroes=`. Values: `off` (default), `on`, `unmap` +(convert zero writes to discard). See: `man qemu-system`. + +## Extra drives + +**`drives`** — List of additional virtio-blk drives. Each entry maps +to a `-drive` with the same backend properties as the root disk: +`file`, `format`, `cache`, `aio`, `discard`, `detect-zeroes`. + +## Networking + +**`ssh_port`** — Host TCP port forwarded to guest port 22. Integer. +Maps to `-nic hostfwd=tcp:127.0.0.1:<port>-:22`. Bound to localhost +only. Must be unique per VM. Omit to disable TCP port forwarding +(use VSOCK instead). + +**`vsock_cid`** — VSOCK Context ID for direct host-guest +communication. Integer, range 3-4294967295 (0=hypervisor, 1=local, +2=host). Must be unique per VM. Maps to +`-device vhost-vsock-pci,guest-cid=<cid>`. Also used by the graceful +shutdown `ExecStop=` which targets `root@vsock/<cid>`. +See: `man qemu-system`. + +**`ssh_private_key`** — Path to the SSH private key for +non-interactive VM access. Used by the graceful shutdown +`ExecStop=ssh -i`. Requires `vsock_cid`. + +## Direct kernel boot + +**`kernel`** — Dict. Presence enables direct kernel boot, bypassing +the disk image bootloader. + +**`kernel.image`** — Path to the kernel image (bzImage/vmlinuz). Maps +to `-kernel`. Required when `kernel` is defined. Example: +`/home/user/kernel/destdir/boot/vmlinuz-6.x.y`. +See: `man qemu-system`. + +**`kernel.append`** — Kernel command line. Maps to `-append`. +Example: `root=/dev/vda1 console=ttyS0,115200 rw`. For imageless +boot: `root=rootfs rootfstype=virtiofs`. + +**`kernel.initrd`** — Path to the initramfs image. Maps to +`-initrd`. Required for imageless boot (the init binary mounts root +before switch_root). Example: `images/initramfs.img`. + +## File sharing + +**`shares`** — List of host-guest directory shares. Each share +produces virtiofsd socket and service dependencies, cloud-init mount +entries, and either virtiofs chardev/device pairs or 9P fsdev/device +pairs depending on `share_transport`. + +Per-share properties: + +`tag`: mount tag visible inside the guest. Required. Used as part of +the virtiofsd instance name (`virtiofsd@<vm>-<tag>`). + +`mount`: guest mount point. Required. Rendered into cloud-init +`mounts:` entries. + +`dir`: host directory to share. Optional. When omitted, virtiofsd +serves the user's home directory (systemd `%h` specifier). When set, +overrides `VIRTIOFSD_SHARED_DIR` via a per-share `EnvironmentFile=`. + +`translate_uid`: virtiofsd UID mapping for imageless boot. Maps to +virtiofsd `--translate-uid`. Format: +`map:<guest-base>:<host-base>:<count>`. Example: `map:0:1000:1` maps +host UID 1000 to guest UID 0 (root). See: +`/usr/libexec/virtiofsd --help`. + +`translate_gid`: same as `translate_uid` but for GIDs. Maps to +virtiofsd `--translate-gid`. Defaults to `translate_uid` value when +omitted. + +**`virtiofsd_binary`** — Path to the virtiofsd binary. Default: +`/usr/libexec/virtiofsd`. Maps to the virtiofsd `ExecStart=` binary +path. + +**`share_transport`** — `virtiofs` (default) or `9p`. virtiofs uses +socket-activated virtiofsd daemons per share. 9P is built into QEMU +(`-fsdev` + `-device virtio-9p-pci`), no daemon, no socket +activation. When using 9P, `dir` is required for each share (no +implicit home default). See: kernel +`Documentation/filesystems/9p.rst`. + +## Cloud-init + +**`cloud_init`** — Dict. Presence enables cloud-init provisioning. +Omit the entire section for mkosi or imageless boot. When defined, +the seed ISO is attached as a `-drive` and cloud-init +user-data/meta-data are rendered. + +**`cloud_init.seed`** — Path to the cloud-init seed ISO. Maps to +`-drive file=<seed>,format=raw`. Generated by `cloud-localds` from +rendered user-data and meta-data. Always raw format (ISO9660 inside +raw container). See: `man cloud-localds`. + +**`cloud_init.locale`** — VM locale. Default: `en_US.UTF-8`. Maps to +cloud-init `locale:` and `write_files:` for `/etc/locale.gen`. + +**`cloud_init.ssh_pubkey`** — SSH public key. Applied to all users +via cloud-init `ssh_authorized_keys:`. Required for non-interactive +SSH access (password auth needs `/dev/tty`). + +**`cloud_init.users`** — List of user accounts to create. Each entry +maps to a cloud-init `users:` entry. Root is not special; include it +in the list with a password to enable root login. The template always +emits `disable_root: false` and a `PermitRootLogin yes` sshd drop-in +regardless of the user list. + +Per-user properties: + +`name`: username. Required. Maps to cloud-init `name`. + +`password`: plain text password. Default: same as `name`. Maps to +cloud-init `plain_text_passwd`. + +`groups`: comma-separated group list. Example: `sudo`. Maps to +cloud-init `groups`. + +`sudo`: sudo rule. Example: `ALL=(ALL) NOPASSWD:ALL`. Maps to +cloud-init `sudo`. + +`shell`: login shell path. Example: `/bin/bash`. Maps to cloud-init +`shell`. + +See: https://cloudinit.readthedocs.io/en/latest/reference/modules.html + +## PCIe passthrough + +**`pci_passthrough`** — List of host PCI devices to pass through via +VFIO. Maps to `-device vfio-pci,host=<addr>`. Also generates systemd +`Requires=vfio-bind@<addr>.service` dependencies and +`LimitMEMLOCK=<ram+256>M` (see systemd.exec(5)). + +Requires one-time root setup: deploy rendered udev rules and +`vfio-pci` module config. After setup, all operations run in user +mode. + +Per-device properties: + +`addr`: PCI address in domain:bus:device.function format. Required. +Example: `0000:2d:00.0`. Identify with `lspci -nv`. All devices in +the same IOMMU group must be passed through together. + +`opts`: additional `-device vfio-pci` properties. Example: +`rombar=0`. + +See: kernel `Documentation/driver-api/vfio.rst`. + +## NVMe + +**`nvme`** — Dict containing `drives` and `subsystems` lists. Maps +to `-device nvme`, `-device nvme-ns`, `-device nvme-subsys`, and +their associated `-drive` entries. + +Guest device paths: `/dev/nvme0n1`, `/dev/nvme1n1`, ... + +Stable names: +`/dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_<serial>_1`. + +`serial` is required by QEMU (errors with "serial property not set" +if omitted). The template defaults to `nvme0`, `nvme1`, etc. + +**`nvme.drives`** — List of NVMe controllers. Simple form (no +`namespaces` key) creates one controller with one implicit namespace. +Explicit `namespaces` key gives full per-namespace control. + +Drive backend properties (simple and explicit forms): `file`, +`format` (default: qcow2), `aio`, `cache`, `discard`, +`detect-zeroes`. + +Controller properties (maps to `-device nvme`): `serial`, +`max_ioqpairs`, `msix_qsize`, `mdts`, `vsl`, `cmb_size_mb`, +`legacy-cmb`, `pmr` (see below), `ioeventfd`, `ocp`, `use-intel-id`, +`dbcs`, `aerl`, `aer_max_queued`, `mqes`, `ctratt.mem`, `atomic.dn`, +`atomic.awun`, `atomic.awupf`, `zoned.zasl`, `zoned.auto_transition`, +`opts`. + +**`nvme.drives[].pmr`** — Dict. Adds a Persistent Memory Region to that +controller. Maps to a `-device nvme,pmrdev=<id>` link plus a +`-object memory-backend-file` that the link names. The PMR occupies PCI +BAR 4/5 (`hw/nvme/ctrl.c`). CMB and PMR can coexist on one controller. +PMR is rejected by QEMU under SR-IOV and when MSI-X claims the +exclusive BAR. `nvme.drives` controllers only (both the simple and +the explicit-namespace form), not subsystem controllers. Omit the +key for a controller with no PMR (backward compatible). + +`size`: backing region size in bytes. Required. Must be a power of 2; +QEMU's NVMe code requires at least 16 bytes and the memory backend +additionally rejects a size below one host page, so the practical +minimum is one page (4096 bytes on x86_64). Maps to +`-object memory-backend-file,size=`. + +`mem-path`: backing file path. Default: `nvme-pmr-<index>.img`, a bare +filename that resolves against the unit `WorkingDirectory=` +(`StateDirectory=`), so the region persists across restarts in the +per-VM state directory. QEMU creates the file on first boot and reopens +it afterwards. Maps to `-object memory-backend-file,mem-path=`. + +`share`: whether guest writes are written through to the backing file. +Default: `true` (`share=on`, persistent). Set `false` for a volatile +region (`share=off`). Maps to `-object memory-backend-file,share=`. + +`pmem`: real persistent-memory flush semantics for the backing file. +Default: off. Only takes effect with `share` on — with `share` off QEMU +maps the region `MAP_PRIVATE`, never requests `MAP_SYNC`, and silently +ignores `pmem` (no error, no warning), so the template emits `pmem=on` +only when `share` is also on. Maps to +`-object memory-backend-file,pmem=`. See: +`<qemu_binary> -object memory-backend-file,help`. + +BlockConf (simple form only, inherited by implicit namespace): +`logical_block_size`, `physical_block_size`, `min_io_size`, +`opt_io_size`, `discard_granularity`, `write-cache`, `share-rw`. + +Namespace properties (maps to `-device nvme-ns`): `nsid`, `uuid`, +`eui64`, `shared`, `detached`, `logical_block_size`, +`physical_block_size`, `min_io_size`, `opt_io_size`, +`discard_granularity`, `write-cache`, `share-rw`, `zoned`, +`zoned.zone_size`, `zoned.zone_capacity`, `zoned.max_active`, +`zoned.max_open`, `zoned.cross_read`, `zoned.descr_ext_size`, +`zoned.numzrwa`, `zoned.zrwas`, `zoned.zrwafg`, `ms`, `mset`, `pi`, +`pil`, `pif`, `mssrl`, `mcl`, `msrc`, `fdp.ruhs`, `atomic.nawun`, +`atomic.nawupf`, `atomic.nabsn`, `atomic.nabspf`, `atomic.nabo`, +`opts`. + +See: `<qemu_binary> -device nvme,help`, +`<qemu_binary> -device nvme-ns,help`, man qemu-system. + +**`nvme.subsystems`** — List of NVMe subsystems for multipath (shared +namespaces across controllers). Maps to `-device nvme-subsys`. + +Subsystem properties: `nqn`, `fdp`, `fdp.nrg`, `fdp.nruh`, +`fdp.runs`. + +Each subsystem contains `controllers` (list with per-controller +`serial`, `max_ioqpairs`, `msix_qsize`, etc.) and `namespaces` (list +with per-namespace properties, `shared: true` for multipath). + +See: `<qemu_binary> -device nvme-subsys,help`. + +## Service overrides + +**`service`** — Dict of systemd `[Service]` section directives. Any +systemd.exec(5), systemd.resource-control(5), or systemd.kill(5) +directive. Use `systemctl --user set-property` for runtime changes +without re-rendering. + +User scope (delegated: cpu, memory, pids): `CPUQuota=`, +`CPUWeight=`, `CPUQuotaPeriodSec=`, `MemoryMax=`, `MemoryHigh=`, +`MemoryMin=`, `MemorySwapMax=`, `TasksMax=`, `WorkingDirectory=`, +`LimitMEMLOCK=`, `LimitNOFILE=`, `Nice=`, `OOMScoreAdjust=`, +`TimeoutSec=`. + +System scope only (not delegated to user services): `IOWeight=`, +`IODeviceWeight=`, `IOReadBandwidthMax=`, `IOWriteBandwidthMax=`, +`AllowedCPUs=`, `AllowedMemoryNodes=`, `DeviceAllow=`, +`DevicePolicy=`. + +See: `man systemd.resource-control`, `man systemd.exec`, +`man systemd.kill`. + +## Exec directories + +systemd provides six directives that each address a specific lifecycle +for per-unit file storage. The template hardcodes four of them so every +VM gets a consistent filesystem layout out of the box. Pick the right +one when you need to extend the layout; don't reuse `WorkingDirectory=` +as a catch-all storage knob. + +| Directive | Purpose | User root | System root | Lifetime | Auto-created | Cleaned on stop | +|---|---|---|---|---|---|---| +| `ConfigurationDirectory=` | read-only unit configuration (env files, helper scripts) | `$XDG_CONFIG_HOME` (`~/.config`) | `/etc` | forever | yes | no | +| `RuntimeDirectory=` | ephemeral runtime state (sockets, PID files, IPC) | `$XDG_RUNTIME_DIR` (`/run/user/<uid>`) | `/run` | unit runtime only | yes | **yes** (unless `RuntimeDirectoryPreserve=`) | +| `StateDirectory=` | persistent per-unit state that must survive restart | `$XDG_STATE_HOME` (`~/.local/state`) | `/var/lib` | forever | yes | no | +| `CacheDirectory=` | regeneratable data; safe to delete | `$XDG_CACHE_HOME` (`~/.cache`) | `/var/cache` | until purge | yes | no | +| `LogsDirectory=` | log files | `$XDG_STATE_HOME/log` | `/var/log` | forever | yes | no | +| `WorkingDirectory=` | CWD for the process | n/a (scalar path) | n/a (scalar path) | n/a | **no** — does not create | n/a | + +The template sets four of these: + +``` +ConfigurationDirectory=systemd/qemu-system +RuntimeDirectory=qemu-system/%i +StateDirectory=qemu-system/%i +WorkingDirectory=%S/qemu-system/%i +``` + +`%S` resolves to the same root that `StateDirectory=` uses (systemd +`unit-printf.c:117-121` + `manager.c:717,725`). `WorkingDirectory=` on +its own does not create or own a directory — coupling it with +`StateDirectory=` is what gives you a predictable per-VM CWD that +auto-exists. + +### Mapping file kinds to directives + +**QEMU per-VM state (NVMe qcow2 images, seed ISOs, cloud-init data)** — +`StateDirectory=`. These must survive a service stop and restart; they +encode the guest's own filesystem state. Reference them by bare filename +in vars (`file: nvme0.qcow2`, `file: seed.iso`) and the relative path +resolves against `WorkingDirectory=%S/qemu-system/%i`. + +**QEMU runtime sockets (qmp.sock, console.sock, gdb.sock)** — +`RuntimeDirectory=`. These are unix sockets the supervisor opens for +the VM's lifetime and that should disappear when the service stops. +The template already wires them with `%t/qemu-system/%i/...` which is +the `RuntimeDirectory` root. + +**Unit configuration files (per-VM env file, QMP helper script)** — +`ConfigurationDirectory=`. The per-VM env file lives at +`%E/systemd/qemu-system/%i.env` and must be present before `ExecStart=` +runs. Consumers that render these — from a flake, a script, or by +hand — place them under `$XDG_CONFIG_HOME/systemd/qemu-system/` for +user services. + +**Disk image caches (intermediate images, overlay images that can be +re-exported)** — `CacheDirectory=`. If the workflow can recreate the +image from a source tree or build artefact, cache-class storage is +correct: a machine-wide cache purge won't lose unique state. + +**VM console and test logs that outlive the service** — +`LogsDirectory=`. Typically systemd already journals service stdout; +use `LogsDirectory=` only for artefacts you want persisted alongside +other `/var/log/` entries (e.g. timestamped test output captured by a +harness running inside the VM). + +**Just changing CWD without any lifecycle management** — +`WorkingDirectory=` on its own. Rarely correct — if you care about the +directory existing, chown, or survival, you want one of the five above +pointing at it first. + +### Overriding via the `service` dict + +`WorkingDirectory=` is a scalar directive: setting +`service: { WorkingDirectory: /some/other/path }` in a vars file +produces an override drop-in that cleanly replaces the default. For +relative paths in vars to keep working you need the override target +to exist and be writable by the service user. + +`StateDirectory=`, `RuntimeDirectory=`, `CacheDirectory=`, +`LogsDirectory=`, `ConfigurationDirectory=` are list-type directives +(`systemd.exec(5)` "whitespace-separated list of directory names"). +A drop-in that sets a new value **appends** to the template default. +To replace, first reset with an empty assignment. Via the `service` +dict you cannot express the two-line reset+set with a single yaml +key; drop to writing a drop-in by hand for that case: + +``` +[Service] +StateDirectory= +StateDirectory=myconvention/%i +``` + +(Reset semantics are in systemd `load-fragment.c:4475-4478`: +`isempty(rvalue)` triggers `exec_directory_done(ed)`.) + +### Specifiers used here + +- `%i` — instance name, the part after `@` in the unit + (`qemu-system@<vm>.service` → `<vm>`). +- `%E` — configuration directory root (`$XDG_CONFIG_HOME` or `/etc`). +- `%S` — state directory root (`$XDG_STATE_HOME` or `/var/lib`). +- `%t` — runtime directory root (`$XDG_RUNTIME_DIR` or `/run`). + +`systemd.unit(5)` SPECIFIERS table has the full list. diff --git a/docs/verifying.md b/docs/verifying.md new file mode 100644 index 00000000..f8cb1a3a --- /dev/null +++ b/docs/verifying.md @@ -0,0 +1,60 @@ +# Verifying changes + +Run these steps before every commit. They are manual for now; whether +and how to wire them as git hooks is a later decision. Each step states +why it exists and what it does. + +## 1. Render the templates — `minijinja-cli --trim-blocks` + +Why: a template that fails to render, or renders a malformed unit, is a +bug. Rendering is the only way to see what systemd actually consumes. + +What: render every template you changed against a vars file (see the +rendering invocation in CLAUDE.md and the README quick start). A +non-zero exit or a Jinja error means the template is broken; +`virtiofsd.env.j2` and the per-VM override templates also need their +`--define` arguments. + +## 2. Verify the unit files — `systemd-analyze verify` + +Why: catches unknown directives and sections (which systemd silently +ignores at runtime), unresolvable `Documentation=` man pages, +`ExecStart=` binaries that are not found, and socket-to-service name +mismatches. + +What: run `systemd-analyze verify` on the rendered `.service` and +`.socket` units. For the `@`-template units it substitutes a test +instance. A clean run prints nothing. + +```shell +systemd-analyze verify <rendered>/[email protected] +systemd-analyze verify <rendered>/[email protected] <rendered>/[email protected] +``` + +## 3. Review the security posture — `systemd-analyze security` + +Why: the service units ship unhardened by design (see +design-decisions.md "Security hardening"), so they score UNSAFE. This +step is to confirm a change has not *regressed* the posture or removed +a directive an operator's override depends on — not to chase a perfect +score. + +```shell +systemd-analyze security <rendered>/[email protected] +``` + +## 4. Syntax-check the shell template + +Why: `transient-run.sh.j2` renders an executable script; a Jinja change +can produce a syntactically broken shell script that renders fine but +fails at `bash`. + +What: `bash -n <rendered>/transient-run.sh` after rendering it. + +## 5. Review the commit message + +Why: the project enforces strict commit conventions (see CLAUDE.md). + +What: subject in imperative mood; body in plain-English paragraphs, +never bullet lists; `Generated-by:` immediately followed by +`Signed-off-by:` with no blank line between them. diff --git a/files/network-config b/files/network-config new file mode 100644 index 00000000..bdb19dc2 --- /dev/null +++ b/files/network-config @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Cloud-init network config (NoCloud datasource, version 2). +# Matches both predictable names (enp0s2) and classic names (eth0). +# Custom kernels without full PCI sysfs support or with net.ifnames=0 +# leave interfaces as eth0. Both patterns are needed. +# See: https://cloudinit.readthedocs.io/en/latest/reference/network-config-format-v2.html +network: + version: 2 + ethernets: + all-en: + match: + name: en* + dhcp4: true + all-eth: + match: + name: eth* + dhcp4: true diff --git a/files/qmp-powerdown b/files/qmp-powerdown new file mode 100644 index 00000000..9569d474 --- /dev/null +++ b/files/qmp-powerdown @@ -0,0 +1,2 @@ +{"execute":"qmp_capabilities"} +{"execute":"system_powerdown"} diff --git a/files/vfio-pci.conf b/files/vfio-pci.conf new file mode 100644 index 00000000..7ce42142 --- /dev/null +++ b/files/vfio-pci.conf @@ -0,0 +1 @@ +vfio-pci diff --git a/templates/meta-data.j2 b/templates/meta-data.j2 new file mode 100644 index 00000000..a427836f --- /dev/null +++ b/templates/meta-data.j2 @@ -0,0 +1,3 @@ +{# SPDX-License-Identifier: copyleft-next-0.3.1 #} +instance-id: {{ vm_name }} +local-hostname: {{ vm_name }} diff --git a/templates/nvme.env.j2 b/templates/nvme.env.j2 new file mode 100644 index 00000000..0bc21663 --- /dev/null +++ b/templates/nvme.env.j2 @@ -0,0 +1,143 @@ +{#- SPDX-License-Identifier: copyleft-next-0.3.1 -#} +{#- NVMe device macros, included by vm.env.j2. -#} +{#- Each macro produces one QEMU argument (no trailing \ or whitespace). -#} +{#- See: <qemu_binary> -device nvme,help -#} +{#- See: <qemu_binary> -device nvme-ns,help -#} +{#- See: <qemu_binary> -device nvme-subsys,help -#} +{#- -drive backend arguments for an NVMe image file. -#} +{% macro nvme_drive(d) -%} +file={{ d.file }},if=none,format={{ d.format | default('qcow2') }} +{%- if d.aio is defined %},aio={{ d.aio }}{% endif -%} +{%- if d.cache is defined %},cache={{ d.cache }}{% endif -%} +{%- if d.discard is defined %},discard={{ d.discard }}{% endif -%} +{%- if d['detect-zeroes'] is defined %},detect-zeroes={{ d['detect-zeroes'] }}{% endif -%} +{% endmacro -%} + +{#- -device nvme controller arguments. -#} +{#- In simple mode (drive= attached): also gets BlockConf properties. -#} +{#- In explicit mode (namespaces): no drive=, gets id= instead. -#} +{% macro nvme_ctrl(d, idx) -%} +nvme,serial={{ d.serial | default('nvme' ~ idx) }} +{%- if d.namespaces is not defined %},drive=nvme-drive-{{ idx }}{% endif -%} +{%- if d.namespaces is defined %},id=nvme{{ idx }}{% endif -%} +{%- if d.max_ioqpairs is defined %},max_ioqpairs={{ d.max_ioqpairs }}{% endif -%} +{%- if d.msix_qsize is defined %},msix_qsize={{ d.msix_qsize }}{% endif -%} +{%- if d.mdts is defined %},mdts={{ d.mdts }}{% endif -%} +{%- if d.vsl is defined %},vsl={{ d.vsl }}{% endif -%} +{%- if d.cmb_size_mb is defined %},cmb_size_mb={{ d.cmb_size_mb }}{% endif -%} +{%- if d['legacy-cmb'] | default(false) %},legacy-cmb=on{% endif -%} +{%- if d.pmr is defined %},pmrdev=nvme-pmr-{{ idx }}{% endif -%} +{%- if d.ioeventfd | default(false) %},ioeventfd=on{% endif -%} +{%- if d.dbcs is defined %},dbcs={{ d.dbcs }}{% endif -%} +{%- if d.ocp | default(false) %},ocp=on{% endif -%} +{%- if d['use-intel-id'] | default(false) %},use-intel-id=on{% endif -%} +{%- if d.aerl is defined %},aerl={{ d.aerl }}{% endif -%} +{%- if d.aer_max_queued is defined %},aer_max_queued={{ d.aer_max_queued }}{% endif -%} +{%- if d.mqes is defined %},mqes={{ d.mqes }}{% endif -%} +{%- if d['zoned.zasl'] is defined %},zoned.zasl={{ d['zoned.zasl'] }}{% endif -%} +{%- if d['zoned.auto_transition'] is defined %},zoned.auto_transition={{ d['zoned.auto_transition'] }}{% endif -%} +{%- if d['ctratt.mem'] | default(false) %},ctratt.mem=on{% endif -%} +{%- if d['atomic.dn'] | default(false) %},atomic.dn=on{% endif -%} +{%- if d['atomic.awun'] is defined %},atomic.awun={{ d['atomic.awun'] }}{% endif -%} +{%- if d['atomic.awupf'] is defined %},atomic.awupf={{ d['atomic.awupf'] }}{% endif -%} +{#- BlockConf (simple form only, implicit namespace inherits these) -#} +{%- if d.namespaces is not defined -%} +{%- if d.logical_block_size is defined %},logical_block_size={{ d.logical_block_size }}{% endif -%} +{%- if d.physical_block_size is defined %},physical_block_size={{ d.physical_block_size }}{% endif -%} +{%- if d.min_io_size is defined %},min_io_size={{ d.min_io_size }}{% endif -%} +{%- if d.opt_io_size is defined %},opt_io_size={{ d.opt_io_size }}{% endif -%} +{%- if d.discard_granularity is defined %},discard_granularity={{ d.discard_granularity }}{% endif -%} +{%- if d['write-cache'] is defined %},write-cache={{ d['write-cache'] }}{% endif -%} +{%- if d['share-rw'] | default(false) %},share-rw=on{% endif -%} +{%- endif -%} +{%- if d.opts is defined %},{{ d.opts }}{% endif -%} +{% endmacro -%} + +{#- -object memory-backend-file backing an NVMe controller PMR (BAR 4/5). -#} +{#- Linked from -device nvme,pmrdev=<id>. size must be a power of 2 and at -#} +{#- least one host page (the memory backend rejects sub-page sizes; NVMe -#} +{#- also requires >= 16 bytes). share=on persists writes through to the -#} +{#- backing file; pmem=on is emitted only with share=on because with -#} +{#- share=off QEMU maps MAP_PRIVATE, never requests MAP_SYNC, and silently -#} +{#- ignores pmem. mem-path defaults to <id>.img, resolved against the unit -#} +{#- WorkingDirectory (StateDirectory); QEMU creates it on first boot. -#} +{#- See: <qemu_binary> -device nvme,help; backends/hostmem-file.c -#} +{% macro nvme_pmr_object(p, id) -%} +memory-backend-file,id={{ id }},mem-path={{ p['mem-path'] | default(id ~ '.img') }},size={{ p.size }} +{%- if p.share | default(true) %},share=on{% else %},share=off{% endif -%} +{%- if p.pmem | default(false) and p.share | default(true) %},pmem=on{% endif -%} +{% endmacro -%} + +{#- -device nvme-ns namespace arguments. -#} +{#- Full NVMe namespace properties: ZNS, metadata, PI, atomics, etc. -#} +{% macro nvme_ns(ns, ctrl_id, ns_idx) -%} +nvme-ns,drive={{ ctrl_id }}-ns{{ ns_idx }},bus={{ ctrl_id }} +{%- if ns.logical_block_size is defined %},logical_block_size={{ ns.logical_block_size }}{% endif -%} +{%- if ns.physical_block_size is defined %},physical_block_size={{ ns.physical_block_size }}{% endif -%} +{%- if ns.min_io_size is defined %},min_io_size={{ ns.min_io_size }}{% endif -%} +{%- if ns.opt_io_size is defined %},opt_io_size={{ ns.opt_io_size }}{% endif -%} +{%- if ns.discard_granularity is defined %},discard_granularity={{ ns.discard_granularity }}{% endif -%} +{%- if ns['write-cache'] is defined %},write-cache={{ ns['write-cache'] }}{% endif -%} +{%- if ns['share-rw'] | default(false) %},share-rw=on{% endif -%} +{%- if ns.nsid is defined %},nsid={{ ns.nsid }}{% endif -%} +{%- if ns.uuid is defined %},uuid={{ ns.uuid }}{% endif -%} +{%- if ns.eui64 is defined %},eui64={{ ns.eui64 }}{% endif -%} +{%- if ns.shared | default(false) %},shared=on{% endif -%} +{%- if ns.detached | default(false) %},detached=on{% endif -%} +{%- if ns.zoned | default(false) %},zoned=on +{%- if ns['zoned.zone_size'] is defined %},zoned.zone_size={{ ns['zoned.zone_size'] }}{% endif -%} +{%- if ns['zoned.zone_capacity'] is defined %},zoned.zone_capacity={{ ns['zoned.zone_capacity'] }}{% endif -%} +{%- if ns['zoned.max_active'] is defined %},zoned.max_active={{ ns['zoned.max_active'] }}{% endif -%} +{%- if ns['zoned.max_open'] is defined %},zoned.max_open={{ ns['zoned.max_open'] }}{% endif -%} +{%- if ns['zoned.cross_read'] | default(false) %},zoned.cross_read=on{% endif -%} +{%- if ns['zoned.descr_ext_size'] is defined %},zoned.descr_ext_size={{ ns['zoned.descr_ext_size'] }}{% endif -%} +{%- if ns['zoned.numzrwa'] is defined %},zoned.numzrwa={{ ns['zoned.numzrwa'] }}{% endif -%} +{%- if ns['zoned.zrwas'] is defined %},zoned.zrwas={{ ns['zoned.zrwas'] }}{% endif -%} +{%- if ns['zoned.zrwafg'] is defined %},zoned.zrwafg={{ ns['zoned.zrwafg'] }}{% endif -%} +{% endif -%} +{%- if ns.ms is defined %},ms={{ ns.ms }}{% endif -%} +{%- if ns.mset is defined %},mset={{ ns.mset }}{% endif -%} +{%- if ns.pi is defined %},pi={{ ns.pi }}{% endif -%} +{%- if ns.pil is defined %},pil={{ ns.pil }}{% endif -%} +{%- if ns.pif is defined %},pif={{ ns.pif }}{% endif -%} +{%- if ns.mssrl is defined %},mssrl={{ ns.mssrl }}{% endif -%} +{%- if ns.mcl is defined %},mcl={{ ns.mcl }}{% endif -%} +{%- if ns.msrc is defined %},msrc={{ ns.msrc }}{% endif -%} +{%- if ns['fdp.ruhs'] is defined %},fdp.ruhs={{ ns['fdp.ruhs'] }}{% endif -%} +{%- if ns['atomic.nawun'] is defined %},atomic.nawun={{ ns['atomic.nawun'] }}{% endif -%} +{%- if ns['atomic.nawupf'] is defined %},atomic.nawupf={{ ns['atomic.nawupf'] }}{% endif -%} +{%- if ns['atomic.nabsn'] is defined %},atomic.nabsn={{ ns['atomic.nabsn'] }}{% endif -%} +{%- if ns['atomic.nabspf'] is defined %},atomic.nabspf={{ ns['atomic.nabspf'] }}{% endif -%} +{%- if ns['atomic.nabo'] is defined %},atomic.nabo={{ ns['atomic.nabo'] }}{% endif -%} +{%- if ns.opts is defined %},{{ ns.opts }}{% endif -%} +{% endmacro -%} + +{#- -device nvme-subsys subsystem arguments. -#} +{% macro nvme_subsys(s, idx) -%} +nvme-subsys,id=nvme-subsys-{{ idx }} +{%- if s.nqn is defined %},nqn={{ s.nqn }}{% endif -%} +{%- if s.fdp | default(false) %},fdp=on +{%- if s['fdp.nrg'] is defined %},fdp.nrg={{ s['fdp.nrg'] }}{% endif -%} +{%- if s['fdp.nruh'] is defined %},fdp.nruh={{ s['fdp.nruh'] }}{% endif -%} +{%- if s['fdp.runs'] is defined %},fdp.runs={{ s['fdp.runs'] }}{% endif -%} +{% endif -%} +{% endmacro -%} + +{#- -device nvme controller in subsystem mode (with subsys= link). -#} +{% macro nvme_subsys_ctrl(ctrl, subsys_id, idx) -%} +nvme,id={{ subsys_id }}-ctrl{{ idx }},serial={{ ctrl.serial | default(subsys_id ~ '-ctrl' ~ idx) }},subsys={{ subsys_id }} +{%- if ctrl.max_ioqpairs is defined %},max_ioqpairs={{ ctrl.max_ioqpairs }}{% endif -%} +{%- if ctrl.msix_qsize is defined %},msix_qsize={{ ctrl.msix_qsize }}{% endif -%} +{%- if ctrl.mdts is defined %},mdts={{ ctrl.mdts }}{% endif -%} +{%- if ctrl.vsl is defined %},vsl={{ ctrl.vsl }}{% endif -%} +{%- if ctrl.cmb_size_mb is defined %},cmb_size_mb={{ ctrl.cmb_size_mb }}{% endif -%} +{%- if ctrl['legacy-cmb'] | default(false) %},legacy-cmb=on{% endif -%} +{%- if ctrl.ioeventfd | default(false) %},ioeventfd=on{% endif -%} +{%- if ctrl.ocp | default(false) %},ocp=on{% endif -%} +{%- if ctrl['zoned.zasl'] is defined %},zoned.zasl={{ ctrl['zoned.zasl'] }}{% endif -%} +{%- if ctrl['zoned.auto_transition'] is defined %},zoned.auto_transition={{ ctrl['zoned.auto_transition'] }}{% endif -%} +{%- if ctrl['atomic.dn'] | default(false) %},atomic.dn=on{% endif -%} +{%- if ctrl['atomic.awun'] is defined %},atomic.awun={{ ctrl['atomic.awun'] }}{% endif -%} +{%- if ctrl['atomic.awupf'] is defined %},atomic.awupf={{ ctrl['atomic.awupf'] }}{% endif -%} +{%- if ctrl.opts is defined %},{{ ctrl.opts }}{% endif -%} +{% endmacro -%} diff --git a/templates/qemu-system-override.conf.j2 b/templates/qemu-system-override.conf.j2 new file mode 100644 index 00000000..4488d484 --- /dev/null +++ b/templates/qemu-system-override.conf.j2 @@ -0,0 +1,88 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Per-instance drop-in for qemu-system@{{ vm_name }}.service +# Rendered from: templates/qemu-system-override.conf.j2 +# +# Deploy: +# mkdir --parents ~/.config/systemd/user/qemu-system@{{ vm_name }}.service.d +# cp <rendered> ~/.config/systemd/user/qemu-system@{{ vm_name }}.service.d/override.conf + +[Unit] +{% if share_transport | default('virtiofs') != '9p' %} +{% for share in shares | default([]) %} +Requires=virtiofsd@%i-{{ share.tag }}.socket +After=virtiofsd@%i-{{ share.tag }}.socket +Requires=virtiofsd@%i-{{ share.tag }}.service +{% endfor %} +{% endif %} +{% for dev in pci_passthrough | default([]) %} +Requires=vfio-bind@{{ dev.addr }}.service +After=vfio-bind@{{ dev.addr }}.service +{% endfor %} + +[Service] +{% if qemu_binary is defined %} +ExecStart= +ExecStart={{ qemu_binary }} $QEMU_ARGS $QEMU_EXTRA_ARGS \ + $KERNEL_ARGS \ + -qmp unix:%t/qemu-system/%i/qmp.sock,server=on,wait=off +{% endif %} +{# QEMU_EXTRA_ARGS: single-line output, {{- " " }} injects spaces between flags #} +{% set has_virtiofs = shares | default([]) | length > 0 and share_transport | default('virtiofs') != '9p' %} +{% set microvm = "microvm" in machine_type %} +Environment="QEMU_EXTRA_ARGS= +{%- if has_virtiofs -%} +{{- " " }}-object memory-backend-memfd,id=mem,size={{ ram }}M,share=on +{{- " " }}-machine memory-backend=mem +{%- for share in shares -%} +{{- " " }}-chardev socket,id=virtiofs_{{ share.tag }},path=%t/virtiofsd/%i-{{ share.tag }}/virtiofsd.sock +{{- " " }}-device {{ "vhost-user-fs-device" if microvm else "vhost-user-fs-pci" }},queue-size=1024,chardev=virtiofs_{{ share.tag }},tag={{ share.tag }} +{%- endfor -%} +{%- endif -%} +{{- " " }}-device {{ "virtio-serial-device" if microvm else "virtio-serial-pci" }} +{{- " " }}-chardev socket,id=console0,path=%t/qemu-system/%i/console.sock,server=on,wait=off +{{- " " }}-device virtconsole,chardev=console0 +{%- if gdb | default(false) -%} +{{- " " }}-gdb unix:%t/qemu-system/%i/gdb.sock +{%- endif -%} +{%- if not autostart | default(true) -%} +{{- " " }}-S +{%- endif -%} +" +{# Override the base registration with one that carries vSockCid. + See docs/design-decisions.md "machined registration". #} +{% if vsock_cid is defined %} +{% if service_scope | default("user") == "user" %} +{% set varlink_socket = "/run/user/%U/systemd/machine/io.systemd.Machine" %} +{% else %} +{% set varlink_socket = "/run/systemd/machine/io.systemd.Machine" %} +{% endif %} +ExecStartPost= +ExecStartPost=-varlinkctl call \ + {{ varlink_socket }} \ + io.systemd.Machine.Register \ + "{\"name\":\"%i\",\"class\":\"vm\",\"service\":\"qemu-system\",\"leader\":${MAINPID},\"vSockCid\":{{ vsock_cid }}{% if uuid is defined %},\"id\":\"{{ uuid }}\"{% endif %}}" +{% endif %} +{# Empty ExecStop= clears base, then SSH (clean) + QMP (fallback) #} +{% if ssh_private_key is defined %} +ExecStop= +ExecStop=-ssh -o ConnectTimeout=5 \ + -o UserKnownHostsFile=/dev/null \ + -o StrictHostKeyChecking=no \ + -o IdentitiesOnly=yes \ + -i ${SSH_KEY_PATH} \ + root@vsock/${VSOCK_CID} \ + systemctl poweroff +ExecStop=-socat OPEN:%E/systemd/qemu-system/qmp-powerdown!!OPEN:/dev/null,wronly UNIX-CONNECT:%t/qemu-system/%i/qmp.sock +{% endif %} +{% if pci_passthrough is defined %} +LimitMEMLOCK={{ ram + 256 }}M +{# /dev/vfio/vfio is the fixed container path; the per-group + /dev/vfio/<iommu-group> nodes are runtime-numbered, so the + operator allow-lists them via service: or relaxes with + DevicePolicy=auto. See design-decisions.md "Security hardening". #} +DeviceAllow=/dev/vfio/vfio rw +{% endif %} +{% for key, value in service | default({}) | items %} +{{ key }}={{ value }} +{% endfor %} diff --git a/templates/[email protected] b/templates/[email protected] new file mode 100644 index 00000000..c88b8dab --- /dev/null +++ b/templates/[email protected] @@ -0,0 +1,71 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# QEMU VM template service +# Rendered from: templates/[email protected] +# +# Scope: {{ service_scope | default('user') }} +{% if service_scope | default('user') == 'system' %} +# Deploy: sudo cp <rendered> /etc/systemd/system/[email protected] +# Usage: sudo systemctl start qemu-system@<name> +# Logs: journalctl --unit=qemu-system@<name>.service +{% else %} +# Deploy: mkdir --parents ~/.config/systemd/user && cp <rendered> ~/.config/systemd/user/[email protected] +# Usage: systemctl --user start qemu-system@<name> +# Logs: journalctl --user-unit=qemu-system@<name>.service +{% endif %} + +[Unit] +Description=QEMU System Virtual Machine %i +Documentation=man:qemu-system(1) man:systemd.service(5) man:systemd.kill(5) +PartOf=machines.target +Before=machines.target +{% if service_scope | default('user') == 'system' %} +After=network.target +{% endif %} + +[Service] +Type=simple +SyslogIdentifier=qemu-system@%i +ConfigurationDirectory=systemd/qemu-system +RuntimeDirectory=qemu-system/%i +StateDirectory=qemu-system/%i +WorkingDirectory=%S/qemu-system/%i +EnvironmentFile=%E/systemd/qemu-system/%i.env +Environment=QEMU_EXTRA_ARGS= +ExecStart={{ qemu_binary }} $QEMU_ARGS $QEMU_EXTRA_ARGS \ + $KERNEL_ARGS \ + -qmp unix:%t/qemu-system/%i/qmp.sock,server=on,wait=off +{# Register with machined over Varlink. See docs/design-decisions.md + "machined registration" for why Varlink (and not busctl). The + shared template registers without vSockCid because it does not + know per-VM values; the per-VM drop-in in + templates/qemu-system-override.conf.j2 overrides this when + vsock_cid is set in the vars file. #} +{% if service_scope | default("user") == "user" %} +{% set varlink_socket = "/run/user/%U/systemd/machine/io.systemd.Machine" %} +{% else %} +{% set varlink_socket = "/run/systemd/machine/io.systemd.Machine" %} +{% endif %} +ExecStartPost=-varlinkctl call \ + {{ varlink_socket }} \ + io.systemd.Machine.Register \ + "{\"name\":\"%i\",\"class\":\"vm\",\"service\":\"qemu-system\",\"leader\":${MAINPID}}" +ExecStop=-socat OPEN:%E/systemd/qemu-system/qmp-powerdown!!OPEN:/dev/null,wronly UNIX-CONNECT:%t/qemu-system/%i/qmp.sock +KillSignal=SIGCONT +KillMode=mixed +TimeoutStopSec=2min +Slice=machine.slice + +# Device access restricted to what the QEMU command line opens, +# mirroring [email protected]. /dev/kvm backs -accel kvm; +# /dev/vhost-vsock backs the vhost-vsock-pci device; virtio-rng's +# /dev/urandom is permitted by the closed policy's defaults. PCIe +# passthrough also needs /dev/vfio -- the per-VM drop-in adds it. +# See: man systemd.resource-control, design-decisions.md +# "Security hardening". +DevicePolicy=closed +DeviceAllow=/dev/kvm rw +DeviceAllow=/dev/vhost-vsock rw + +[Install] +WantedBy=machines.target diff --git a/templates/transient-run.sh.j2 b/templates/transient-run.sh.j2 new file mode 100644 index 00000000..091fa9b3 --- /dev/null +++ b/templates/transient-run.sh.j2 @@ -0,0 +1,72 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Interactive QEMU session for {{ vm_name }}. +# Rendered from: templates/transient-run.sh.j2 +# +# Render and run: +# minijinja-cli --trim-blocks \ +# --output /tmp/run-{{ vm_name }}.sh \ +# templates/transient-run.sh.j2 vars/{{ vm_name }}.yaml +# bash /tmp/run-{{ vm_name }}.sh +# +# Disconnect: press Ctrl-] three times within 1 second. +# The VM terminates when the session disconnects. +# +# Do not pipe through bash (| bash). The pipe steals stdin from the +# terminal, which systemd-run --pty needs for interactive console. + +set -euo pipefail + +unit=qemu-{{ vm_name }}-interactive + +# Stop the persistent service if running (same ports, same shares). +if systemctl --user is-active --quiet qemu-system@{{ vm_name }}.service 2>/dev/null; then + echo "Stopping persistent service qemu-system@{{ vm_name }}..." >&2 + systemctl --user stop qemu-system@{{ vm_name }}.service +fi + +# Stop a previous interactive session if still running. +if systemctl --user is-active --quiet "$unit".service 2>/dev/null; then + echo "Stopping previous interactive session $unit..." >&2 + systemctl --user stop "$unit".service +fi + +{% set has_virtiofs = shares | default([]) | length > 0 and share_transport | default('virtiofs') != '9p' %} +{#- QEMU_EXTRA_ARGS is built in the shell, not in Environment=, because + systemd does not expand %t or $XDG_RUNTIME_DIR in Environment= + directives of transient units. The shell resolves $XDG_RUNTIME_DIR + at exec time. -#} +{% if has_virtiofs %} +QEMU_EXTRA_ARGS=" +{%- for share in shares | default([]) -%} +{%- if not loop.first %} {% endif %}-chardev socket,id=virtiofs_{{ share.tag }},path=$XDG_RUNTIME_DIR/virtiofsd/{{ vm_name }}-{{ share.tag }}/virtiofsd.sock +{{- " " }}-device {{ "vhost-user-fs-device" if "microvm" in machine_type else "vhost-user-fs-pci" }},queue-size=1024,chardev=virtiofs_{{ share.tag }},tag={{ share.tag }} +{%- endfor -%} +{{- " " }}-object memory-backend-memfd,id=mem,size={{ ram }}M,share=on +{{- " " }}-machine memory-backend=mem" +{% else %} +QEMU_EXTRA_ARGS="" +{% endif %} +export QEMU_EXTRA_ARGS + +exec systemd-run --user --pty --collect \ + --unit="$unit" \ + --property=EnvironmentFile="$HOME"/.config/systemd/qemu-system/{{ vm_name }}.env \ + --property=PassEnvironment=QEMU_EXTRA_ARGS \ +{% if has_virtiofs %} +{% for share in shares | default([]) %} + --property=Requires=virtiofsd@{{ vm_name }}-{{ share.tag }}.socket \ + --property=After=virtiofsd@{{ vm_name }}-{{ share.tag }}.socket \ + --property=BindsTo=virtiofsd@{{ vm_name }}-{{ share.tag }}.service \ +{% endfor %} +{% endif %} +{% if gdb | default(false) %} + --property="Environment=QEMU_GDB_ARGS=-gdb unix:$XDG_RUNTIME_DIR/qemu-system/{{ vm_name }}/gdb.sock" \ +{% endif %} +{% if pci_passthrough is defined %} + --property=LimitMEMLOCK={{ ram + 256 }}M \ +{% endif %} +{% if service is defined and service.WorkingDirectory is defined %} + --property=WorkingDirectory={{ service.WorkingDirectory }} \ +{% endif %} + sh -c 'eval exec $QEMU_BINARY $QEMU_ARGS $QEMU_EXTRA_ARGS ${QEMU_GDB_ARGS:-} $KERNEL_ARGS' diff --git a/templates/user-data.j2 b/templates/user-data.j2 new file mode 100644 index 00000000..e9805c3a --- /dev/null +++ b/templates/user-data.j2 @@ -0,0 +1,44 @@ +{# SPDX-License-Identifier: copyleft-next-0.3.1 #} +#cloud-config +locale: {{ cloud_init.locale | default('en_US.UTF-8') if cloud_init is defined else 'en_US.UTF-8' }} +disable_root: false +ssh_pwauth: true +{% if cloud_init is defined and cloud_init.users is defined and cloud_init.users | length > 0 %} +users: +{% for u in cloud_init.users %} + - name: {{ u.name }} + lock_passwd: false + plain_text_passwd: {{ u.password | default(u.name) }} +{% if u.groups is defined %} + groups: {{ u.groups }} +{% endif %} +{% if u.sudo is defined %} + sudo: {{ u.sudo }} +{% endif %} +{% if u.shell is defined %} + shell: {{ u.shell }} +{% endif %} +{% if cloud_init.ssh_pubkey is defined %} + ssh_authorized_keys: + - {{ cloud_init.ssh_pubkey }} +{% endif %} +{% endfor %} +{% endif %} +{% if shares | default([]) | length > 0 %} +mounts: +{% for share in shares %} +{% if share_transport | default('virtiofs') == '9p' %} + - [ "{{ share.tag }}", "{{ share.mount }}", "9p", "trans=virtio,version=9p2000.L,nofail", "0", "0" ] +{% else %} + - [ "{{ share.tag }}", "{{ share.mount }}", "virtiofs", "defaults,nofail", "0", "0" ] +{% endif %} +{% endfor %} +{% endif %} +write_files: + - path: /etc/ssh/sshd_config.d/99-permit-root-login.conf + permissions: '0600' + content: | + PermitRootLogin yes + PasswordAuthentication yes +runcmd: + - touch /etc/cloud/cloud-init.disabled diff --git a/templates/[email protected] b/templates/[email protected] new file mode 100644 index 00000000..c8e0b3f0 --- /dev/null +++ b/templates/[email protected] @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# VFIO PCI device binding service +# Rendered from: templates/[email protected] +# +# Deploy: mkdir --parents ~/.config/systemd/user && cp <rendered> ~/.config/systemd/user/[email protected] +# Usage: systemctl --user start vfio-bind@0000:2d:00.0 +# Requires: udev rules deployed (templates/vfio-udev.rules.j2), vfio-pci module loaded + +[Unit] +Description=VFIO PCI bind for %I + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=sh -c 'echo vfio-pci > /sys/bus/pci/devices/%I/driver_override' +ExecStart=-sh -c 'echo %I > /sys/bus/pci/devices/%I/driver/unbind' +ExecStart=sh -c 'echo %I > /sys/bus/pci/drivers_probe' +ExecStop=sh -c 'echo > /sys/bus/pci/devices/%I/driver_override' +ExecStop=-sh -c 'echo %I > /sys/bus/pci/devices/%I/driver/unbind' +ExecStop=sh -c 'echo %I > /sys/bus/pci/drivers_probe' diff --git a/templates/vfio-udev.rules.j2 b/templates/vfio-udev.rules.j2 new file mode 100644 index 00000000..fbe980fb --- /dev/null +++ b/templates/vfio-udev.rules.j2 @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# VFIO device node permissions, allow kvm group access +# Rendered from: templates/vfio-udev.rules.j2 +# Deploy: sudo tee /etc/udev/rules.d/10-vfio-kvm.rules +# Reload: sudo udevadm control --reload-rules && sudo udevadm trigger --subsystem-match=pci +SUBSYSTEM=="vfio", OWNER="root", GROUP="kvm", MODE="0660" +{% for dev in pci_passthrough | default([]) %} + +# Per-device sysfs permissions for unprivileged VFIO binding: {{ dev.addr }} +ACTION=="add", SUBSYSTEM=="pci", KERNEL=="{{ dev.addr }}", \ + RUN+="/bin/sh -c '\ + chgrp kvm /sys/bus/pci/devices/{{ dev.addr }}/driver_override; \ + chmod 664 /sys/bus/pci/devices/{{ dev.addr }}/driver_override; \ + chgrp kvm /sys/bus/pci/drivers_probe; \ + chmod 220 /sys/bus/pci/drivers_probe'" +ACTION=="bind", SUBSYSTEM=="pci", KERNEL=="{{ dev.addr }}", \ + RUN+="/bin/sh -c '\ + chgrp kvm /sys/bus/pci/devices/{{ dev.addr }}/driver/unbind; \ + chmod 220 /sys/bus/pci/devices/{{ dev.addr }}/driver/unbind'" +{% endfor %} diff --git a/templates/virtiofsd-override.conf.j2 b/templates/virtiofsd-override.conf.j2 new file mode 100644 index 00000000..6762b546 --- /dev/null +++ b/templates/virtiofsd-override.conf.j2 @@ -0,0 +1,43 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Per-instance drop-in for virtiofsd@<vm>-<share>.service +# Rendered from: templates/virtiofsd-override.conf.j2 +# +# Deploy one drop-in per (vm, share) tuple: +# mkdir --parents ~/.config/systemd/user/virtiofsd@{{ vm_name }}-<share>.service.d +# cp <rendered> ~/.config/systemd/user/virtiofsd@{{ vm_name }}-<share>.service.d/override.conf +# +# Adds `Before=qemu-system@<vm>.service` so virtiofsd outlives +# qemu-system@<vm>.service's `ExecStop=` graceful shutdown. The +# per-VM `qemu-system-override.conf` already emits +# `Requires=virtiofsd@%i-<tag>.service` for every share. Without +# stop ordering, that `Requires=` cascade fires the virtiofsd stop +# concurrently with `ExecStop=ssh root@vsock/<cid> systemctl +# poweroff`. The guest's unmount of `/nix/store`, `/lib/modules`, +# and any virtiofs share starts seeing `virtio-fs: response too +# short` errors as virtiofsd's vhost-user socket disappears +# mid-request. The guest hangs on filesystem IO that will never +# complete, ACPI shutdown is never delivered, and +# qemu-system@<vm>.service reaches `TimeoutStopSec=2min`, after +# which systemd sends `SIGKILL` to the QEMU process. +# +# `Before=qemu-system@<vm>.service` does not pull +# qemu-system@<vm>.service in; it only orders. Stop direction is +# the inverse of start direction (`man systemd.unit`), so +# `Before=qemu-system@<vm>.service` here means virtiofsd stops +# AFTER qemu-system@<vm>.service in the stop transaction. The +# existing `Requires=` cascade still queues virtiofsd's stop in +# the same transaction; the new ordering just runs it after the +# QEMU process has fully exited rather than concurrently with its +# `ExecStop=`. [email protected] then stops via the queued job +# plus `StopWhenUnneeded=yes`, the listening socket stops next, +# and the guest's clean unmount completes inside that window. +# +# Socket activation is unaffected: `Before=` only orders, it does +# not start. The transaction that starts qemu-system@<vm>.service +# does not pull [email protected] into a `JOB_START`; that +# service still starts via socket activation when the QEMU process +# connects to virtiofsd@%i-<tag>.socket. + +[Unit] +Before=qemu-system@{{ vm_name }}.service diff --git a/templates/virtiofsd.env.j2 b/templates/virtiofsd.env.j2 new file mode 100644 index 00000000..4b5e7897 --- /dev/null +++ b/templates/virtiofsd.env.j2 @@ -0,0 +1,11 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# virtiofsd environment for {{ vm_name }}-{{ share_tag }} +# Rendered from: templates/virtiofsd.env.j2 --define share_tag={{ share_tag }} +{% for s in shares | default([]) if s.tag == share_tag %} +{% if s.dir is defined %} +VIRTIOFSD_SHARED_DIR={{ s.dir }} +{% endif %} +{% if s.translate_uid is defined %} +VIRTIOFSD_EXTRA_OPTS=--translate-uid {{ s.translate_uid }} --translate-gid {{ s.translate_gid | default(s.translate_uid) }} +{% endif %} +{% endfor %} diff --git a/templates/[email protected] b/templates/[email protected] new file mode 100644 index 00000000..b718b8ab --- /dev/null +++ b/templates/[email protected] @@ -0,0 +1,47 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# virtiofsd service for QEMU VM instance (socket-activated) +# Rendered from: templates/[email protected] +# +# Instance name encodes VM and share: virtiofsd@<vm>-<tag>.service +# Example: [email protected] +# +# Default shared directory is the user's home (%h specifier). Non-home +# shares get a rendered env file (from templates/virtiofsd.env.j2) that +# overrides VIRTIOFSD_SHARED_DIR. +# +# Default sandbox is namespace with UID/GID mapping (%U/%G specifiers). +# This provides PID, mount, network, and user namespace isolation. +# Shares with translate_uid add --translate-uid via VIRTIOFSD_EXTRA_OPTS +# (both mechanisms coexist: namespace maps virtiofsd's process identity, +# translate-uid maps file ownership to the guest). +# Override VIRTIOFSD_SANDBOX_ARGS in the env file for --sandbox=none. +# +# This service is started by systemd socket activation when the QEMU +# process connects to the virtiofsd socket. Do not start it directly; +# start the .socket unit instead. +# +# StopWhenUnneeded=yes makes the service auto-stop when no +# qemu-system@<vm>.service pins it (via the per-VM drop-in's +# Requires=virtiofsd@%i-<tag>.service). +# Stop side mirrors the start side: socket activation handles start, +# StopWhenUnneeded handles stop. See docs/design-decisions.md. + +[Unit] +Description=virtiofsd for %i +StopWhenUnneeded=yes + +[Service] +Type=simple +ConfigurationDirectory=systemd/virtiofsd +Environment="VIRTIOFSD_SANDBOX_ARGS=--sandbox=namespace --uid-map :0:%U:1: --gid-map :0:%G:1:" +Environment=VIRTIOFSD_SHARED_DIR=%h +Environment=VIRTIOFSD_EXTRA_OPTS= +EnvironmentFile=-%E/systemd/virtiofsd/%i.env +ExecStart={{ virtiofsd_binary | default('/usr/libexec/virtiofsd') }} \ + $VIRTIOFSD_SANDBOX_ARGS \ + --shared-dir=${VIRTIOFSD_SHARED_DIR} \ + --fd=3 \ + --xattr \ + --no-announce-submounts \ + $VIRTIOFSD_EXTRA_OPTS diff --git a/templates/[email protected] b/templates/[email protected] new file mode 100644 index 00000000..960df3b8 --- /dev/null +++ b/templates/[email protected] @@ -0,0 +1,24 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# virtiofsd socket for QEMU VM instance +# Rendered from: templates/[email protected] +# +# Instance name encodes VM and share: virtiofsd@<vm>-<tag>.socket +# Example: [email protected] +# +{% if service_scope | default('user') == 'system' %} +# Deploy: sudo cp <rendered> /etc/systemd/system/[email protected] +{% else %} +# Deploy: cp <rendered> ~/.config/systemd/user/[email protected] +{% endif %} + +[Unit] +Description=virtiofsd socket for %i + +[Socket] +ListenStream=%t/virtiofsd/%i/virtiofsd.sock +SocketMode=0660 +RuntimeDirectory=virtiofsd/%i + +[Install] +WantedBy=sockets.target diff --git a/templates/vm.env.j2 b/templates/vm.env.j2 new file mode 100644 index 00000000..ca44daf2 --- /dev/null +++ b/templates/vm.env.j2 @@ -0,0 +1,156 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +{# Import NVMe macros. `{% from ... import ... %}` works in strict + Jinja2 (ansible.builtin.template) and in minijinja (minijinja-cli); + the prior `{% include %}` relied on a minijinja extension that + exports macros to the caller, which Jinja2 does not honour. #} +{% from "nvme.env.j2" import nvme_drive, nvme_ctrl, nvme_ns, nvme_subsys, nvme_subsys_ctrl, nvme_pmr_object %} +{#- Machine detection: microvm uses virtio-mmio, all others use PCI. + microvm with pcie=on has both transports; we use MMIO for virtio + devices and PCI for NVMe/VFIO/IOMMU. -#} +{% set microvm = "microvm" in machine_type %} +{% set has_pcie = not microvm or "pcie" in machine_type %} +{#- Device names: MMIO uses *-device, PCI uses *-pci-non-transitional. + vhost-user-fs has no non-transitional variant (modern-only). -#} +{% set dev_net = "virtio-net-device" if microvm else "virtio-net-pci-non-transitional" %} +{% set dev_rng = "virtio-rng-device" if microvm else "virtio-rng-pci-non-transitional" %} +{% set dev_vsock = "vhost-vsock-device" if microvm else "vhost-vsock-pci-non-transitional" %} +{% set dev_9p = "virtio-9p-device" if microvm else "virtio-9p-pci-non-transitional" %} +{% set dev_fs = "vhost-user-fs-device" if microvm else "vhost-user-fs-pci" %} +{#- Drive backend: -drive arguments shared by root disk, seed, and extra drives. + microvm: if=none with explicit -device virtio-blk-device after each drive. + PCI: if=virtio (implicit virtio-blk-pci, no separate -device needed). -#} +{% macro drive_backend(d, drive_id) -%} +file={{ d.file }}, +{%- if microvm -%} +if=none,id={{ drive_id }}, +{%- else -%} +if=virtio, +{%- endif -%} +format={{ d.format | default('raw') }} +{%- if d.cache is defined %},cache={{ d.cache }}{% endif -%} +{%- if d.aio is defined %},aio={{ d.aio }}{% endif -%} +{%- if d.discard is defined %},discard={{ d.discard }}{% endif -%} +{%- if d['detect-zeroes'] is defined %},detect-zeroes={{ d['detect-zeroes'] }}{% endif -%} +{% endmacro -%} +# VM environment for {{ vm_name }} +# Rendered from vars file. Source this, then run: $QEMU_BINARY $QEMU_ARGS +QEMU_BINARY={{ qemu_binary }} +QEMU_ARGS="\ + -machine type={{ machine_type }} \ +{% if iommu is defined and iommu in ['intel-iommu', 'amd-iommu'] %} + -accel {{ accel | default('kvm') }},kernel-irqchip=split \ +{% else %} + -accel {{ accel | default('kvm') }} \ +{% endif %} + -cpu {{ cpu | default('host') }} \ + -m {{ ram }} \ + -smp {{ cpus }} \ +{% if uuid is defined %} + -uuid {{ uuid }} \ +{% endif %} +{% if has_pcie and iommu is defined %} +{% if iommu == 'intel-iommu' %} + -device intel-iommu,intremap=on,caching-mode=on \ +{% elif iommu == 'amd-iommu' %} + -device amd-iommu,intremap=on,dma-remap=on \ +{% elif iommu == 'virtio-iommu-pci' %} + -device virtio-iommu-pci \ +{% elif iommu == 'arm-smmuv3' %} + -device arm-smmuv3 \ +{% endif %} +{% endif %} +{% if firmware is defined %} + -drive if=pflash,format=raw,readonly=on,file={{ firmware.code }} \ + -drive if=pflash,format={{ firmware.vars_format | default('raw') }},file={{ firmware.vars }} \ +{% endif %} +{% if image is defined %} + -drive {{ drive_backend(image, 'root') }} \ +{% if microvm %} + -device virtio-blk-device,drive=root \ +{% endif %} +{% endif %} +{% if cloud_init is defined and cloud_init.seed is defined %} + -drive {{ drive_backend({'file': cloud_init.seed, 'format': 'raw'}, 'seed') }} \ +{% if microvm %} + -device virtio-blk-device,drive=seed \ +{% endif %} +{% endif %} +{% for d in drives | default([]) %} + -drive {{ drive_backend(d, 'drive' ~ loop.index0) }} \ +{% if microvm %} + -device virtio-blk-device,drive=drive{{ loop.index0 }} \ +{% endif %} +{% endfor %} +{% if microvm %} + -netdev user,id=net0{% if ssh_port is defined %},hostfwd=tcp:127.0.0.1:{{ ssh_port }}-:22{% endif %} \ + -device {{ dev_net }},netdev=net0 \ +{% else %} + -nic user,model={{ dev_net }}{% if ssh_port is defined %},hostfwd=tcp:127.0.0.1:{{ ssh_port }}-:22{% endif %} \ +{% endif %} +{% if vsock_cid is defined %} + -device {{ dev_vsock }},guest-cid={{ vsock_cid }} \ +{% endif %} +{% if has_pcie %} +{% for dev in pci_passthrough | default([]) %} + -device vfio-pci,host={{ dev.addr }}{% if dev.opts is defined %},{{ dev.opts }}{% endif %} \ +{% endfor %} +{% endif %} +{% if share_transport | default('virtiofs') == '9p' %} +{% for share in shares | default([]) %} + -fsdev local,id=fsdev_{{ share.tag }},path={{ share.dir }},security_model=none,multidevs=remap \ + -device {{ dev_9p }},fsdev=fsdev_{{ share.tag }},mount_tag={{ share.tag }} \ +{% endfor %} +{% endif %} +{% if has_pcie %} +{# NVMe PMR backends: -object must precede the nvme -device that links it via pmrdev= #} +{% for drive in (nvme.drives if nvme is defined) | default([]) %} +{% if drive.pmr is defined %} + -object {{ nvme_pmr_object(drive.pmr, 'nvme-pmr-' ~ loop.index0) }} \ +{% endif %} +{% endfor %} +{# NVMe subsystems (multipath, shared namespaces across controllers) #} +{% for subsys in (nvme.subsystems if nvme is defined) | default([]) %} +{% set subsys_id = 'nvme-subsys-' ~ loop.index0 %} + -device {{ nvme_subsys(subsys, loop.index0) }} \ +{% for ctrl in subsys.controllers | default([]) %} + -device {{ nvme_subsys_ctrl(ctrl, subsys_id, loop.index0) }} \ +{% endfor %} +{% for ns in subsys.namespaces | default([]) %} + -drive {{ nvme_drive(ns) }},id={{ subsys_id }}-ctrl0-ns{{ loop.index0 }} \ + -device {{ nvme_ns(ns, subsys_id ~ '-ctrl0', loop.index0) }} \ +{% endfor %} +{% endfor %} +{# NVMe drives (1 controller per drive, or explicit namespaces) #} +{% for drive in (nvme.drives if nvme is defined) | default([]) %} +{% if drive.namespaces is defined %} +{% set ctrl_id = 'nvme' ~ loop.index0 %} + -device {{ nvme_ctrl(drive, loop.index0) }} \ +{% for ns in drive.namespaces %} + -drive {{ nvme_drive(ns) }},id={{ ctrl_id }}-ns{{ loop.index0 }} \ + -device {{ nvme_ns(ns, ctrl_id, loop.index0) }} \ +{% endfor %} +{% else %} + -drive {{ nvme_drive(drive) }},id=nvme-drive-{{ loop.index0 }} \ + -device {{ nvme_ctrl(drive, loop.index0) }} \ +{% endif %} +{% endfor %} +{% endif %} + -device {{ dev_rng }} \ + -nographic \ + -serial mon:stdio" +{% if vsock_cid is defined %} +VSOCK_CID={{ vsock_cid }} +{% endif %} +{% if ssh_private_key is defined %} +SSH_KEY_PATH={{ ssh_private_key }} +{% endif %} +{% if kernel is defined %} +KERNEL_ARGS=-kernel {{ kernel.image }} +{%- if kernel.append is defined %} + -append "{{ kernel.append }}" +{%- endif %} +{%- if kernel.initrd is defined %} + -initrd {{ kernel.initrd }} +{%- endif %} + +{% endif %} diff --git a/vars/example.yaml b/vars/example.yaml new file mode 100644 index 00000000..941ca0ae --- /dev/null +++ b/vars/example.yaml @@ -0,0 +1,112 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# VM configuration. See docs/vars.md for field reference. + +vm_name: test +service_scope: user +# uuid: 00112233-4455-6677-8899-aabbccddeeff + +# QEMU +qemu_binary: /usr/bin/qemu-system-x86_64 +cpu: host +accel: kvm +ram: 2048 +cpus: 2 +machine_type: q35 +# gdb: true +# autostart: false +# iommu: intel-iommu + +# Firmware (UEFI boot; omit for SeaBIOS default) +# firmware: +# code: /usr/share/OVMF/OVMF_CODE_4M.fd +# vars: images/test-ovmf-vars.fd +# vars_format: raw + +# Root disk +image: + file: images/test.qcow2 + format: qcow2 + # cache: none + # aio: native + # discard: unmap + # detect-zeroes: unmap + +# Extra drives (virtio-blk) +# drives: +# - file: images/data.qcow2 +# format: qcow2 +# discard: unmap +# - file: images/scratch.raw +# format: raw + +# Networking +ssh_port: 10022 +# vsock_cid: 100 +# ssh_private_key: ~/.ssh/id_ed25519 + +# Direct kernel boot +# kernel: +# image: /home/user/kernel/destdir/boot/vmlinuz-6.x.y +# append: root=/dev/vda1 console=ttyS0,115200 rw +# initrd: /home/user/kernel/destdir/boot/initramfs-6.x.y.img + +# File sharing (virtiofs default, 9P fallback) +shares: + - tag: home + mount: /mnt/home +# - tag: modules +# dir: /home/user/kernel/destdir/lib/modules +# mount: /lib/modules +# virtiofsd_binary: /usr/libexec/virtiofsd +# share_transport: 9p + +# Cloud-init (omit entire section for mkosi or imageless boot) +# cloud_init: +# seed: images/seed.iso +# locale: en_US.UTF-8 +# ssh_pubkey: ssh-ed25519 AAAA... user@host +# users: +# - name: root +# password: root +# - name: user +# password: user +# groups: sudo +# sudo: ALL=(ALL) NOPASSWD:ALL + +# PCIe passthrough via VFIO +# pci_passthrough: +# - addr: "0000:2d:00.0" +# - addr: "0000:03:00.0" +# opts: "rombar=0" + +# NVMe emulated drives +# nvme: +# drives: +# - file: images/nvme0.qcow2 +# - file: images/nvme1.qcow2 +# format: qcow2 +# serial: data-drive +# # Persistent Memory Region (BAR 4/5). +# # pmem takes effect only with share: true. +# pmr: +# size: 16777216 +# share: true +# # pmem: true +# subsystems: +# - nqn: subsys0 +# controllers: +# - serial: path-a +# - serial: path-b +# namespaces: +# - file: images/shared.raw +# format: raw +# shared: true + +# [Service] section overrides +# service: +# CPUQuota: 200% +# MemoryMax: 4G +# TasksMax: 512 +# TimeoutSec: 5min +# WorkingDirectory: /home/user/machines -- 2.54.0