[PATCH 01/12] scripts: import qemu-system-units at scripts/qemu-system-units
Daniel Gomez <[email protected]> Thu, 23 Apr 2026 13:30:53 +0200
| Newsgroups | dev.linux.lists.kdevops |
|---|---|
| Message-ID | <20260423-kdevops-series-c-qemu-system-units-v1-1-b7bab3225a36@samsung.com> |
From: Daniel Gomez <[email protected]> Import the qemu-system-units project from https://github.com/linux-kdevops/qemu-system-units at commit 12346c9955e775c16f661323357c85e46a08e388 into scripts/qemu-system-units/. qemu-system-units (qsu) ships Jinja2 templates that render the QEMU command line into a systemd EnvironmentFile plus the matching [email protected], [email protected] and .service, VFIO device binding oneshot, per-VM override drop-ins, and QMP powerdown commands. Consumed in the follow-up patches as kdevops' QEMU_SYSTEM_UNITS bringup backend. The git-subtree-dir and git-subtree-split trailers below let future syncs with upstream use git subtree pull/push. Generated-by: Claude AI Signed-off-by: Daniel Gomez <[email protected]> git-subtree-dir: scripts/qemu-system-units git-subtree-split: 12346c9955e775c16f661323357c85e46a08e388 --- scripts/qemu-system-units/.gitignore | 11 + scripts/qemu-system-units/CLAUDE.md | 251 +++++++++++++ scripts/qemu-system-units/COPYING | 22 ++ scripts/qemu-system-units/LICENSE | 5 + .../LICENSES/preferred/copyleft-next-0.3.1 | 239 ++++++++++++ scripts/qemu-system-units/README.md | 81 +++++ scripts/qemu-system-units/docs/design-decisions.md | 404 +++++++++++++++++++++ scripts/qemu-system-units/docs/requirements.md | 56 +++ scripts/qemu-system-units/docs/transient-units.md | 244 +++++++++++++ scripts/qemu-system-units/docs/usage.md | 378 +++++++++++++++++++ scripts/qemu-system-units/docs/vars.md | 381 +++++++++++++++++++ scripts/qemu-system-units/files/network-config | 18 + scripts/qemu-system-units/files/qmp-powerdown | 2 + scripts/qemu-system-units/files/vfio-pci.conf | 1 + scripts/qemu-system-units/templates/meta-data.j2 | 3 + scripts/qemu-system-units/templates/nvme.env.j2 | 126 +++++++ .../templates/qemu-system-override.conf.j2 | 83 +++++ .../templates/[email protected] | 56 +++ .../templates/transient-run.sh.j2 | 72 ++++ scripts/qemu-system-units/templates/user-data.j2 | 44 +++ .../templates/[email protected] | 21 ++ .../qemu-system-units/templates/vfio-udev.rules.j2 | 20 + .../qemu-system-units/templates/virtiofsd.env.j2 | 9 + .../templates/[email protected] | 40 ++ .../templates/[email protected] | 24 ++ scripts/qemu-system-units/templates/vm.env.j2 | 144 ++++++++ scripts/qemu-system-units/vars/example.yaml | 105 ++++++ 27 files changed, 2840 insertions(+) diff --git a/scripts/qemu-system-units/.gitignore b/scripts/qemu-system-units/.gitignore new file mode 100644 index 00000000..2e636e78 --- /dev/null +++ b/scripts/qemu-system-units/.gitignore @@ -0,0 +1,11 @@ +# User vars files (copy vars/example.yaml to start) +vars/* +!vars/example.yaml + +# Rendered output +*.env + +# Editor swap files +*.swp +*.swo +*~ diff --git a/scripts/qemu-system-units/CLAUDE.md b/scripts/qemu-system-units/CLAUDE.md new file mode 100644 index 00000000..739b99ef --- /dev/null +++ b/scripts/qemu-system-units/CLAUDE.md @@ -0,0 +1,251 @@ +# CLAUDE.md + +## Project Overview + +Systemd unit templates for running QEMU virtual machines as first-class +systemd services. Jinja2 templates rendered by `minijinja-cli` into +systemd units and environment files. No daemon, no wrapper. + +**License**: copyleft-next-0.3.1 + +## Project Structure + +``` +qemu-system-units/ +├── templates/ Jinja2 templates (persistent + transient) +├── vars/ YAML variable files (per-VM configuration) +│ └── example.yaml Starting point (copy to vars/<vm_name>.yaml) +├── files/ Static files deployed as-is +├── docs/ Reference documentation +│ ├── vars.md Variable reference +│ ├── usage.md Rendering, deployment, operations +│ ├── design-decisions.md Hardcoded choices and rationale +│ ├── requirements.md Distro-specific packages +│ └── transient-units.md systemd-run patterns +├── LICENSES/ +└── README.md Landing page and quick start +``` + +## Critical Rules + +### Never fabricate facts + +Every command-line flag, sysfs path, systemd directive, and expected +output must be verified against the actual source before being written +into any project file. `man <tool>` or `<tool> --help` before writing +any command. If a tool only has short flags (`lspci`, `ss`, `ip`), use +short flags. + +### Never cheat during verification + +When verifying that templates render and deploy correctly, start from +a clean state. Never pre-deploy files, reuse leftover artifacts, or +skip steps. A verification that depends on state from a previous test +is a lie. + +## Rules + +### Variable names match upstream + +YAML variable names must use the exact terminology from the upstream +tool they configure. Do not invent prefixes or synonyms. + +`ram` maps to QEMU `-m`. `cpu` maps to `-cpu`. `accel` maps to +`-accel`. `image.file` maps to `-drive file=`. `image.format` maps +to `-drive format=`. The test is: can you read the variable name and +immediately know which upstream flag it maps to? + +### Flat variables for cross-cutting, nested for single-consumer + +Variables used by multiple templates stay flat (e.g. `ram`, `shares`, +`vsock_cid`). Variables consumed by a single template block can be +nested when they form a cohesive group (e.g. `kernel.image`, +`kernel.append`, `kernel.initrd` all map to one `KERNEL_ARGS=` block). + +### Backward compatibility + +Every new template variable must be gated with `| default([])` (lists), +`| default('value')` (scalars with defaults), or `is defined` +(optional features). Templates must render identically when new +variables are absent. + +### Non-transitional virtio devices + +All virtio devices use `*-pci-non-transitional` suffix on PCI machines +and `*-device` suffix on microvm (MMIO). Transitional devices require +`CONFIG_VIRTIO_PCI_LEGACY` which custom kernels often disable. The +template auto-detects microvm from `machine_type` and switches device +suffixes. `vhost-user-fs-pci` has no non-transitional variant +(modern-only device). + +### Zero-tool naming + +All rendered output references the consumer (qemu-system@ service), +never the generator project. Config directories are named after the +service prefix (`qemu-system`, `virtiofsd`). Only the README and +CLAUDE.md mention the project name. + +### Long-form command options + +NEVER use short flags when a long-form alternative exists in README, +templates, docs, and any commands shown to the user. `--follow` not +`-f`, `--parents` not `-p`, `--append --groups` not `-aG`. +Exception: tools without long-form options (`ssh -p`, `lspci -nv`). + +### Documentation references + +Reference only QEMU and systemd manuals, not wrapper projects. + +`See: man qemu-system` for QEMU flags. `See: man systemd.kill` for +systemd directives. `See: /usr/libexec/virtiofsd --help` for +virtiofsd flags. Use `<qemu_binary>` not `qemu-system-x86_64` when +the flag is architecture-independent. + +### Maps-to references + +"Maps to" in docs/vars.md references the upstream flag or directive, +not template filenames or env variable names. `ram` maps to `-m`, +not "QEMU_ARGS in vm.env." + +### Technical identifiers in docs + +Backtick-format systemd directives (`KillMode=`, `ExecStop=`), signal +constants (`SIGTERM`, `SIGKILL`), function calls (`sd_notify()`), +kernel configs (`CONFIG_VIRTIO_PCI`), and paths (`/dev/kvm`). Plain +text for tool names in natural language (virtiofsd, QEMU, systemd). +Systemd directives include trailing `=` matching man page convention. + +### Shell examples + +No shell variables (`$VM`, `${VM}`) in per-component documentation. +Use literal values that match vars file content (`test`, `dev`). +The deploy-all section at the bottom of docs/usage.md uses variables +for automation convenience. + +Prose instructions (editing files, logging out) go in markdown +paragraphs, not shell comments. + +### vars file convention + +`vars/example.yaml` is the template. Users copy it to +`vars/<vm_name>.yaml` (e.g. `vars/test.yaml` for `vm_name: test`). +The filename matches `vm_name` inside the file. User vars files are +gitignored. + +## Git Commit Guidelines + +### One commit per change + +Atomic commits. Spell fixes go in separate commits from code changes. + +### Commit message format + +``` +subsystem: brief description in imperative mood + +Plain English explanation of the change. NEVER use bullet points +or itemized lists in commit messages. + +Generated-by: Claude AI +Signed-off-by: Your Name <[email protected]> +``` + +### Use Signed-off-by and Generated-by tags + +Generated-by MUST be immediately followed by Signed-off-by with NO +empty lines between them. No Co-Authored-By trailer. + +### No shopping cart lists + +NEVER use bullet points or itemized lists in commit messages. Use +plain English paragraphs. + +### Subsystem prefix + +Use the template or doc name as prefix: `vm.env:`, `virtiofsd:`, +`docs:`, `vars:`, `README:`. Use `qemu-system-units:` for +cross-cutting changes. + +## Key Technical Patterns + +### `EnvironmentFile=` path + +`%E/systemd/qemu-system/%i.env` uses the `%E` specifier for +scope-agnostic paths. User mode: `~/.config/systemd/qemu-system/`. +System mode: `/etc/systemd/qemu-system/`. + +### ExecStart binary path must be literal + +systemd requires the first argument of `ExecStart=` to be a literal +path, not a variable. The binary path is rendered by Jinja2 at +template time. See: `load-fragment.c` in systemd source. + +### $MAINPID in ExecStartPost + +Available for `Type=simple` services. systemd sets the main PID +before entering the start-post phase. No shell wrapper needed for +the `busctl RegisterMachine` call. + +### Journal: --user-unit= not --user -u + +`journalctl --user -u` constrains to user journal files. But user +service output goes through `[email protected]` to the SYSTEM +journal. `journalctl --user-unit=` matches `_SYSTEMD_USER_UNIT` +across all journal files. + +### Cloud-init only runs once + +cloud-init marks first-boot completion in `/var/lib/cloud/`. A new +seed ISO does NOT trigger cloud-init to rerun on an existing image. +To re-provision, either use a fresh disk image or delete +`/var/lib/cloud/` inside the guest. + +### Three-layer template pattern + +Templates follow three layers: (1) static unit structure that never +changes, (2) conditional blocks gated on vars (`{% if X is defined %}`), +and (3) loops over lists (`{% for dev in list | default([]) %}`). +Layers 2 and 3 emit nothing when the variable is absent. New +features add new conditional/loop blocks without modifying the +static layer. + +## Rendering + +All rendering uses `minijinja-cli --trim-blocks`. One vars file drives +all templates. + +```shell +minijinja-cli --trim-blocks \ + --output <deploy-path> \ + templates/<template>.j2 \ + vars/<vm_name>.yaml +``` + +## Source Code References + +### QEMU + +`qemu-options.hx` (all command-line options), +`hw/nvme/ctrl.c` (NVMe device, serial requirement at line 8600), +`hw/i386/microvm.c` (microvm machine type, pcie=on property), +`system/runstate.c` (SIGTERM handling, force_shutdown at line 786), +`hw/virtio/vhost-vsock.c` (VSOCK CID range validation at line 134). + +### systemd + +`man systemd.service` (Type=, ExecStart=, ExecStop=), +`man systemd.exec` (WorkingDirectory=, LimitMEMLOCK=, %U/%G/%h/%E specifiers), +`man systemd.resource-control` (CPUQuota=, MemoryMax=, Slice=), +`man systemd.kill` (KillMode=, KillSignal=, TimeoutSec=), +`man systemd.special` (machine.slice, machines.target). + +systemd does NOT expand specifiers in `EnvironmentFile=` values. `%h` +in `Environment=` is expanded. `%h` inside an `EnvironmentFile=` is +literal. + +### virtiofsd + +`src/sandbox.rs` (namespace sandbox, setresuid at line 497, +default UID mapping at line 331), +`--sandbox=namespace --uid-map :0:%U:1: --gid-map :0:%G:1:` is the +canonical unprivileged configuration. diff --git a/scripts/qemu-system-units/COPYING b/scripts/qemu-system-units/COPYING new file mode 100644 index 00000000..f5479109 --- /dev/null +++ b/scripts/qemu-system-units/COPYING @@ -0,0 +1,22 @@ +This project is provided under: + + SPDX-License-Identifier: copyleft-next-0.3.1 + LICENSES/preferred/copyleft-next-0.3.1 + +In addition, other licenses may also apply. We embrace the same SPDX +practice as used in the Linux kernel, for those details refer to under +Linux: + + Documentation/process/license-rules.rst + +for more details. + +Note: Linux accepts copyleft-next licensed code, however, even though +copyleft-next is GPL v2 compatible we use a dual license tag on Linux +to err on the side of caution. And so, if you ever do wish to use code +from this project on Linux be sure to use this tag instead: + +// SPDX-License-Identifier: GPL-2.0-or-later OR copyleft-next-0.3.1 + +This project equally accepts GPL-2.0 code only as copyleft-next is GPL +v2 compatible. diff --git a/scripts/qemu-system-units/LICENSE b/scripts/qemu-system-units/LICENSE new file mode 100644 index 00000000..28dadc58 --- /dev/null +++ b/scripts/qemu-system-units/LICENSE @@ -0,0 +1,5 @@ +Copyright (c) 2026 Daniel Gomez <[email protected]> + +qemu-system-units is licensed under copyleft-next-0.3.1 + +Refer to LICENSES/preferred/copyleft-next-0.3.1 for license details. diff --git a/scripts/qemu-system-units/LICENSES/preferred/copyleft-next-0.3.1 b/scripts/qemu-system-units/LICENSES/preferred/copyleft-next-0.3.1 new file mode 100644 index 00000000..5e04da3d --- /dev/null +++ b/scripts/qemu-system-units/LICENSES/preferred/copyleft-next-0.3.1 @@ -0,0 +1,239 @@ +Valid-License-Identifier: copyleft-next-0.3.1 +SPDX-URL: https://spdx.org/licenses/copyleft-next-0.3.1 +Usage-Guide: + This license can be used in code, it has been found to be GPLv2 compatible + by attorneys at Redhat and SUSE, however to err on the side of caution, + if used on Linux it's best to only use it together with a GPL2 compatible + license using "OR". You do not have to do this for this project, + as it is licensed under the copyleft-next-0.3.1 license. + To use the copyleft-next-0.3.1 license on Linux put the following SPDX + tag/value pair into a comment according to the placement guidelines in the + licensing rules documentation: + SPDX-License-Identifier: GPL-2.0 OR copyleft-next-0.3.1 + SPDX-License-Identifier: GPL-2.0-only OR copyleft-next 0.3.1 + SPDX-License-Identifier: GPL-2.0+ OR copyleft-next-0.3.1 + SPDX-License-Identifier: GPL-2.0-or-later OR copyleft-next-0.3.1 +License-Text: + +======================================================================= + + copyleft-next 0.3.1 ("this License") + Release date: 2016-04-29 + +1. License Grants; No Trademark License + + Subject to the terms of this License, I grant You: + + a) A non-exclusive, worldwide, perpetual, royalty-free, irrevocable + copyright license, to reproduce, Distribute, prepare derivative works + of, publicly perform and publicly display My Work. + + b) A non-exclusive, worldwide, perpetual, royalty-free, irrevocable + patent license under Licensed Patents to make, have made, use, sell, + offer for sale, and import Covered Works. + + This License does not grant any rights in My name, trademarks, service + marks, or logos. + +2. Distribution: General Conditions + + You may Distribute Covered Works, provided that You (i) inform + recipients how they can obtain a copy of this License; (ii) satisfy the + applicable conditions of sections 3 through 6; and (iii) preserve all + Legal Notices contained in My Work (to the extent they remain + pertinent). "Legal Notices" means copyright notices, license notices, + license texts, and author attributions, but does not include logos, + other graphical images, trademarks or trademark legends. + +3. Conditions for Distributing Derived Works; Outbound GPL Compatibility + + If You Distribute a Derived Work, You must license the entire Derived + Work as a whole under this License, with prominent notice of such + licensing. This condition may not be avoided through such means as + separate Distribution of portions of the Derived Work. + + If the Derived Work includes material licensed under the GPL, You may + instead license the Derived Work under the GPL. + +4. Condition Against Further Restrictions; Inbound License Compatibility + + When Distributing a Covered Work, You may not impose further + restrictions on the exercise of rights in the Covered Work granted under + this License. This condition is not excused merely because such + restrictions result from Your compliance with conditions or obligations + extrinsic to this License (such as a court order or an agreement with a + third party). + + However, You may Distribute a Covered Work incorporating material + governed by a license that is both OSI-Approved and FSF-Free as of the + release date of this License, provided that compliance with such + other license would not conflict with any conditions stated in other + sections of this License. + +5. Conditions for Distributing Object Code + + You may Distribute an Object Code form of a Covered Work, provided that + you accompany the Object Code with a URL through which the Corresponding + Source is made available, at no charge, by some standard or customary + means of providing network access to source code. + + If you Distribute the Object Code in a physical product or tangible + storage medium ("Product"), the Corresponding Source must be available + through such URL for two years from the date of Your most recent + Distribution of the Object Code in the Product. However, if the Product + itself contains or is accompanied by the Corresponding Source (made + available in a customarily accessible manner), You need not also comply + with the first paragraph of this section. + + Each direct and indirect recipient of the Covered Work from You is an + intended third-party beneficiary of this License solely as to this + section 5, with the right to enforce its terms. + +6. Symmetrical Licensing Condition for Upstream Contributions + + If You Distribute a work to Me specifically for inclusion in or + modification of a Covered Work (a "Patch"), and no explicit licensing + terms apply to the Patch, You license the Patch under this License, to + the extent of Your copyright in the Patch. This condition does not + negate the other conditions of this License, if applicable to the Patch. + +7. Nullification of Copyleft/Proprietary Dual Licensing + + If I offer to license, for a fee, a Covered Work under terms other than + a license that is OSI-Approved or FSF-Free as of the release date of this + License or a numbered version of copyleft-next released by the + Copyleft-Next Project, then the license I grant You under section 1 is no + longer subject to the conditions in sections 3 through 5. + +8. Copyleft Sunset + + The conditions in sections 3 through 5 no longer apply once fifteen + years have elapsed from the date of My first Distribution of My Work + under this License. + +9. Pass-Through + + When You Distribute a Covered Work, the recipient automatically receives + a license to My Work from Me, subject to the terms of this License. + +10. Termination + + Your license grants under section 1 are automatically terminated if You + + a) fail to comply with the conditions of this License, unless You cure + such noncompliance within thirty days after becoming aware of it, or + + b) initiate a patent infringement litigation claim (excluding + declaratory judgment actions, counterclaims, and cross-claims) + alleging that any part of My Work directly or indirectly infringes + any patent. + + Termination of Your license grants extends to all copies of Covered + Works You subsequently obtain. Termination does not terminate the + rights of those who have received copies or rights from You subject to + this License. + + To the extent permission to make copies of a Covered Work is necessary + merely for running it, such permission is not terminable. + +11. Later License Versions + + The Copyleft-Next Project may release new versions of copyleft-next, + designated by a distinguishing version number ("Later Versions"). + Unless I explicitly remove the option of Distributing Covered Works + under Later Versions, You may Distribute Covered Works under any Later + Version. + +** 12. No Warranty ** +** ** +** My Work is provided "as-is", without warranty. You bear the risk ** +** of using it. To the extent permitted by applicable law, each ** +** Distributor of My Work excludes the implied warranties of title, ** +** merchantability, fitness for a particular purpose and ** +** non-infringement. ** + +** 13. Limitation of Liability ** +** ** +** To the extent permitted by applicable law, in no event will any ** +** Distributor of My Work be liable to You for any damages ** +** whatsoever, whether direct, indirect, special, incidental, or ** +** consequential damages, whether arising under contract, tort ** +** (including negligence), or otherwise, even where the Distributor ** +** knew or should have known about the possibility of such damages. ** + +14. Severability + + The invalidity or unenforceability of any provision of this License + does not affect the validity or enforceability of the remainder of + this License. Such provision is to be reformed to the minimum extent + necessary to make it valid and enforceable. + +15. Definitions + + "Copyleft-Next Project" means the project that maintains the source + code repository at <https://github.com/copyleft-next/copyleft-next.git/> + as of the release date of this License. + + "Corresponding Source" of a Covered Work in Object Code form means (i) + the Source Code form of the Covered Work; (ii) all scripts, + instructions and similar information that are reasonably necessary for + a skilled developer to generate such Object Code from the Source Code + provided under (i); and (iii) a list clearly identifying all Separate + Works (other than those provided in compliance with (ii)) that were + specifically used in building and (if applicable) installing the + Covered Work (for example, a specified proprietary compiler including + its version number). Corresponding Source must be machine-readable. + + "Covered Work" means My Work or a Derived Work. + + "Derived Work" means a work of authorship that copies from, modifies, + adapts, is based on, is a derivative work of, transforms, translates or + contains all or part of My Work, such that copyright permission is + required. The following are not Derived Works: (i) Mere Aggregation; + (ii) a mere reproduction of My Work; and (iii) if My Work fails to + explicitly state an expectation otherwise, a work that merely makes + reference to My Work. + + "Distribute" means to distribute, transfer or make a copy available to + someone else, such that copyright permission is required. + + "Distributor" means Me and anyone else who Distributes a Covered Work. + + "FSF-Free" means classified as 'free' by the Free Software Foundation. + + "GPL" means a version of the GNU General Public License or the GNU + Affero General Public License. + + "I"/"Me"/"My" refers to the individual or legal entity that places My + Work under this License. "You"/"Your" refers to the individual or legal + entity exercising rights in My Work under this License. A legal entity + includes each entity that controls, is controlled by, or is under + common control with such legal entity. "Control" means (a) the power to + direct the actions of such legal entity, whether by contract or + otherwise, or (b) ownership of more than fifty percent of the + outstanding shares or beneficial ownership of such legal entity. + + "Licensed Patents" means all patent claims licensable royalty-free by + Me, now or in the future, that are necessarily infringed by making, + using, or selling My Work, and excludes claims that would be infringed + only as a consequence of further modification of My Work. + + "Mere Aggregation" means an aggregation of a Covered Work with a + Separate Work. + + "My Work" means the particular work of authorship I license to You + under this License. + + "Object Code" means any form of a work that is not Source Code. + + "OSI-Approved" means approved as 'Open Source' by the Open Source + Initiative. + + "Separate Work" means a work that is separate from and independent of a + particular Covered Work and is not by its nature an extension or + enhancement of the Covered Work, and/or a runtime library, standard + library or similar component that is used to generate an Object Code + form of a Covered Work. + + "Source Code" means the preferred form of a work for making + modifications to it. diff --git a/scripts/qemu-system-units/README.md b/scripts/qemu-system-units/README.md new file mode 100644 index 00000000..6f5031b9 --- /dev/null +++ b/scripts/qemu-system-units/README.md @@ -0,0 +1,81 @@ +# qemu-system-units + +Systemd unit templates for running QEMU virtual machines as first-class +systemd services. + +**License**: copyleft-next-0.3.1 + +## Features + +- **Machine types**: q35, microvm (auto MMIO), ARM virt +- **Boot modes**: cloud image, mkosi, direct kernel, imageless (virtiofs rootfs) +- **File sharing**: virtiofs (socket-activated), 9P fallback +- **Networking**: user-mode with port forwarding, VSOCK +- **Devices**: NVMe emulation (ZNS, multipath, FDP), PCIe passthrough (VFIO) +- **IOMMU**: Intel VT-d, AMD-Vi, virtio-iommu, ARM SMMUv3 +- **Debugging**: GDB via unix socket, paused start +- **Lifecycle**: machined registration, QMP graceful shutdown, resource control + +## Quick start + +```shell +sudo apt install qemu-system-x86 qemu-utils socat \ + systemd-container virtiofsd cloud-image-utils cargo +cargo install minijinja-cli +sudo usermod --append --groups kvm $(whoami) +``` + +Log out and back in for group changes to take effect. +See [docs/requirements.md](docs/requirements.md) for Fedora, openSUSE, and NixOS. + +```shell +cp vars/example.yaml vars/test.yaml + +mkdir --mode=0755 --parents ~/.config/systemd/user ~/.config/systemd/qemu-system ~/.config/systemd/virtiofsd +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] vars/test.yaml +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] vars/test.yaml +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] vars/test.yaml +mkdir --mode=0755 --parents ~/.config/systemd/user/[email protected] +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected]/override.conf \ + templates/qemu-system-override.conf.j2 vars/test.yaml +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/qemu-system/test.env \ + templates/vm.env.j2 vars/test.yaml +cp files/qmp-powerdown ~/.config/systemd/qemu-system/qmp-powerdown + +systemctl --user daemon-reload +systemctl --user start qemu-system@test +machinectl --user list +``` + +## Documentation + +| Document | Content | +|---|---| +| [docs/requirements.md](docs/requirements.md) | Packages for Debian, Fedora, openSUSE, NixOS | +| [docs/usage.md](docs/usage.md) | Rendering, deployment, machinectl, console, logs, stop, multiple VMs | +| [docs/vars.md](docs/vars.md) | Variable reference for all template fields | +| [docs/design-decisions.md](docs/design-decisions.md) | Hardcoded choices and upstream references | +| [docs/transient-units.md](docs/transient-units.md) | `systemd-run` and transient unit patterns | +| [vars/example.yaml](vars/example.yaml) | Starting point for VM configuration | + +## Related work + +Community systemd service files for QEMU VMs: + +- [rafaelmartins/kvm-systemd](https://github.com/rafaelmartins/kvm-systemd). `[email protected]`, `Type=forking`, per-VM conf in `/etc/kvm/` +- [eaon/qemu-kvm-systemd-service](https://codeberg.org/eaon/qemu-kvm-systemd-service). `[email protected]`, per-VM conf in `/etc/qemu/vms/` +- [dehesselle/virtctl](https://github.com/dehesselle/virtctl). `[email protected]`, `hypervisor.target` pattern +- [0xef53/kvmrun](https://github.com/0xef53/kvmrun). Go, systemd-native, per-VM chroot, gRPC API + +QEMU upstream ships helper units (guest-agent, pr-helper, vmsr-helper) in +[contrib/systemd/](https://gitlab.com/qemu-project/qemu/-/tree/master/contrib/systemd) +but no VM lifecycle service. virtiofsd ships no systemd units. Neither +Debian nor Fedora package a `[email protected]` template. diff --git a/scripts/qemu-system-units/docs/design-decisions.md b/scripts/qemu-system-units/docs/design-decisions.md new file mode 100644 index 00000000..c5f03dc6 --- /dev/null +++ b/scripts/qemu-system-units/docs/design-decisions.md @@ -0,0 +1,404 @@ +# Design decisions + +Managing QEMU VMs has two parts: generating the QEMU command line +(machine type, devices, disks, networking, kernel boot, passthrough) +and VM lifecycle (start, stop, restart, dependencies, logging, +resource control). systemd handles the lifecycle. Templates handle +the command line. No new daemon, no new wrapper. + +Rendering and deploying the templates is left to the consumer. +The raw workflow is `minijinja-cli` + `systemctl`. Automation +layers are out of scope for this project. + +The templates aim to be as unopinionated as possible. When a value +is hardcoded, it is either required by the underlying tool (QEMU, +virtiofsd, systemd), a workaround with documented reasoning, or a +sensible default that the user can override. This document lists +every such choice, the upstream reference that justifies it, and +how to change it when possible. + +## Variable naming + +Variable names match upstream flag or parameter names when a 1:1 +mapping exists. When a variable controls a higher-level concept, +a descriptive name is used. + +**`image`** — QEMU's flag is `-drive`. QEMU's own docs describe the +value as "which disk image to use with this drive." The sub-properties +(`image.file`, `image.format`, `image.cache`) match `-drive` property +names exactly. The name `image` describes what the user configures (a +disk image), while the properties map 1:1 to QEMU's `-drive` +properties. + +**`cpus`** — QEMU's flag is `-smp`. The parameter inside `-smp` is +called `cpus=`. The variable matches the parameter name. + +**`vsock_cid`** — QEMU's device property is `guest-cid`. The variable +adds the technology context (`vsock`) because `guest_cid` alone is +ambiguous. Maps to `-device vhost-vsock-pci,guest-cid=`. + +**`ssh_port`** — No upstream equivalent. Maps to +`-nic hostfwd=tcp:127.0.0.1:<port>-:22`. The concept (SSH port +forwarding) spans multiple flag components. + +**`pci_passthrough`** — No upstream equivalent. Maps to +`-device vfio-pci,host=<addr>`. Combines the bus type (PCI) with the +operation (passthrough). + +**`autostart`** — No upstream equivalent. Inverted boolean: `false` +maps to `-S` (QEMU starts paused). QEMU has no "autostart" flag. + +**`firmware`** — QEMU has no `-firmware` flag. Firmware selection is +implicit: pflash0 populated → UEFI, pflash0 absent → BIOS. QEMU's +own firmware specification (`docs/interop/firmware.json`) calls the +top-level concept `Firmware` and the internal function is +`pc_system_firmware_init()`. The sub-properties `code` and `vars` +match the OVMF file naming convention (`OVMF_CODE_4M.fd`, +`OVMF_VARS_4M.fd`). QEMU's spec uses the more verbose `executable` +and `nvram-template` but the OVMF file names are what users encounter +directly. + +**`cloud_init.users[].password`** — Cloud-init's key is +`plain_text_passwd`. The variable uses `password` for simplicity. The +template maps it to `plain_text_passwd`. + +## User-configurable (vars file) + +Fully controlled by the user. See [vars.md](vars.md) for reference. + +`cpu`, `accel`, `ram`, `cpus`, `machine_type`, `iommu`, +`firmware` (code, vars, vars_format), `gdb`, `autostart`, +`image` (file, format, cache, aio, discard, detect-zeroes), +`drives`, `ssh_port`, `vsock_cid`, `ssh_private_key`, +`kernel` (image, append, initrd), +`shares` (tag, mount, dir, translate_uid, translate_gid), +`share_transport`, `virtiofsd_binary`, +`cloud_init` (seed, locale, ssh_pubkey, users), +`pci_passthrough`, `nvme` (drives, subsystems), +`service` (any `[Service]` directive) + +## systemd service properties + +**`KillMode=`** — Set to `mixed`. Sends `KillSignal=` to the main +process. After the main process exits or `TimeoutSec=` elapses, +remaining processes in the cgroup receive `SIGKILL`. For VMs, the +main process is QEMU. After `ExecStop=` runs QMP graceful shutdown, +if QEMU exits cleanly, any leaked child processes are killed +immediately. The alternative `control-group` would send `KillSignal=` +to all processes simultaneously, which is unnecessary when only QEMU +needs the signal. See: `man systemd.kill`. + +**`TimeoutSec=`** — Set to `2min`. Grace period for `ExecStop=` +before systemd sends `SIGKILL`. The systemd default +(`DefaultTimeoutStopSec=`) is 90s. VMs need longer because ACPI +powerdown triggers a full guest OS shutdown sequence (flushing +buffers, stopping services, unmounting filesystems). Override: + +```yaml +service: + TimeoutSec: 5min +``` + +See: `man systemd.service`. + +**`Slice=`** — Set to `machine.slice`. All virtual machines and +containers registered with systemd-machined are placed in +machine.slice. Canonical cgroup placement for VMs. +See: `man systemd.special`. + +**`PartOf=`**, **`Before=`**, **`WantedBy=`** — Set to +`machines.target`. Standard target for starting all containers and +virtual machines. `PartOf=` ensures VMs stop when machines.target +stops. `WantedBy=` enables auto-start. See: `man systemd.special`. + +**`Type=`** — Set to `simple`. QEMU does not implement +`sd_notify()`. `notify` would be correct but requires the service to +call `sd_notify(READY=1)` after initialization. A patch adding +`sd_notify()` to QEMU was submitted (qemu-devel, 2025-12-17) but has +not been merged. When QEMU gains `sd_notify()` support, this should +change to `notify`. See: `man systemd.service`. + +## machined registration + +Each VM registers itself with machined at start-up via +`ExecStartPost=` running +`varlinkctl call <socket> io.systemd.Machine.Register <json>`. +Socket path: `/run/user/%U/systemd/machine/io.systemd.Machine` +(user scope) or `/run/systemd/machine/io.systemd.Machine` +(system scope). Required JSON fields: `name`, `class=vm`, +`service=qemu-system`, `leader=${MAINPID}`. Optional: +`vSockCid` when the vars file sets `vsock_cid`. + +Registering `vSockCid` is what makes `ssh machine/<vm>` route +over AF_VSOCK. The shipped +`/etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf` hands the +`machine/*` pattern to `systemd-ssh-proxy`, which looks the +machine up over Varlink and reads the registered CID. Without +it, `systemd-ssh-proxy` exits with "Machine has no AF_VSOCK +CID assigned" (`src/ssh-generator/ssh-proxy.c`). + +### Why two `ExecStartPost=` lines across two templates + +The shared `[email protected]` renders once for all +instances and has no per-VM variables at that render time, so +its Jinja cannot branch on `vsock_cid`. It emits one +`ExecStartPost=` that registers without CID - correct for any +VM, always safe. + +The per-VM `qemu-system-override.conf.j2` renders with +`vsock_cid` in scope. When set, it emits `ExecStartPost=` +(empty) followed by `ExecStartPost=-varlinkctl ... vSockCid:N`. +Systemd treats the empty directive as a reset of the +accumulated list, so the drop-in's call replaces the shared +template's call rather than running alongside. Net effect: +every VM makes one registration call, with CID if configured, +without otherwise. VMs that never set `vsock_cid` hit only the +shared template's call and behave exactly as before. + +### Why Varlink and not `busctl` + +The legacy DBus `RegisterMachine` method +(`src/machine/machined-dbus.c`) has a fixed `sayssus` +signature with no `vSockCid` field. `RegisterMachineEx` +accepts `VSockCID` but requires the leader identified by +`LeaderPIDFD` or `LeaderPID+LeaderPIDFDID`, neither of which +a shell script can synthesise. Varlink's +`io.systemd.Machine.Register` leader dispatcher accepts a +bare integer PID and machined acquires the pidfd daemon-side +(`json_dispatch_pidref` in +`src/libsystemd/sd-json/json-util.c`). + +### Quoting and expansion + +`${MAINPID}` uses the brace form because systemd's exec parser +expands `${VAR}` (but not `$VAR`) inside escape-quoted +arguments, letting the JSON body ride as one argv entry +without a `/bin/sh -c` wrapper. The drop-in inlines +`{{ vsock_cid }}` at Jinja render time rather than pulling +`${VSOCK_CID}` from the environment because the value is +already known per-VM. + +The `-` prefix on `ExecStartPost=` keeps the VM running if +machined is unreachable. Registration is informational. + +## QEMU workarounds + +**`KillSignal=`** — Set to `SIGCONT`. QEMU on `SIGTERM` calls +`qemu_system_killed()` which sets `force_shutdown=true` and exits +immediately without ACPI powerdown. `SIGCONT` is a no-op signal that +keeps QEMU alive, giving `ExecStop=` time to send QMP +system_powerdown for proper ACPI shutdown. After `TimeoutSec=`, +systemd sends `SIGKILL` (via `KillMode=mixed`). Not +user-configurable. See: `man systemd.kill`. + +**`ExecStop=`** — Sends QMP system_powerdown via socat for graceful +guest shutdown. QEMU does not translate `SIGTERM` to ACPI powerdown. +When `ssh_private_key` is defined, `ExecStop=` first attempts SSH +shutdown (guest-initiated poweroff) with QMP as fallback. Not +user-configurable (the mechanism is fixed; the SSH path is enabled by +setting `ssh_private_key` and `vsock_cid`). + +## QEMU firmware + +**`-drive if=pflash`** — OVMF firmware via pflash drives. QEMU's +firmware specification (`docs/interop/firmware.json`) defines the +`split` mode: the executable (CODE) is read-only and shared, the +NVRAM template (VARS) is cloned per-VM and configured read-write. +pflash0 holds the executable, pflash1 holds the per-VM NVRAM file. + +QEMU selects firmware mode based on pflash0 presence: if pflash0 is +populated, QEMU enters pflash mode (UEFI); if absent, ROM mode +(SeaBIOS). There is no explicit firmware flag; the presence of pflash +drives is the entire mechanism. See: `hw/i386/pc_sysfw.c` +`pc_system_firmware_init()`. + +pflash is created by `pc_system_flash_create()` which is part of the +PC machine initialization path. microvm does not call this function; +it calls `x86_bios_rom_init()` directly, even with `pcie=on`. The +`pcie=on` property on microvm only enables the GPEX PCIe host bridge +for PCI devices; it does not change the firmware initialization path. +ARM virt machines create pflash independently (256 KB sectors vs +x86's 4 KB). + +The NVRAM file stores UEFI boot order, Secure Boot key databases +(PK, KEK, db, dbx), and guest-written variables. Each VM needs its +own writable NVRAM file for independent variable state. QEMU enforces +this with file-level locking. A second VM attempting to open the same +writable NVRAM file will fail to start. Create a per-VM copy from the +template: + +```shell +cp /usr/share/OVMF/OVMF_VARS_4M.fd images/test-ovmf-vars.fd +``` + +The firmware spec also supports qcow2 format. An alternative to +copying is a qcow2 overlay with the template as backing file: + +```shell +qemu-img create --format qcow2 \ + --backing /usr/share/OVMF/OVMF_VARS_4M.fd \ + --backing-format raw \ + images/test-ovmf-vars.qcow2 +``` + +This saves disk space (only changed variables are stored) and keeps +the system template untouched. Use `format: qcow2` in the vars file +when using a qcow2 NVRAM file. + +When the `firmware` section is absent from vars, no pflash drives are +rendered and QEMU defaults to SeaBIOS. Backward compatible. +User-configurable: set `firmware.code` and `firmware.vars`. + +## QEMU device choices + +**`-device virtio-*-pci-non-transitional`** — All virtio devices use +non-transitional variants. These only require `CONFIG_VIRTIO_PCI` in +the guest kernel. Transitional devices additionally require +`CONFIG_VIRTIO_PCI_LEGACY`, which custom kernels often disable. On +microvm, devices use the `-device virtio-*-device` suffix +(virtio-mmio transport). vhost-user-fs-pci has no non-transitional +variant (modern-only device). Not user-configurable (determined by +`machine_type`). See: `<qemu_binary> -device help`. + +**`-device virtio-rng-*`** — Always present. Provides entropy to the +guest. Without it, `/dev/random` may block and boot can stall waiting +for entropy. Not user-configurable. + +**`-nographic`**, **`-serial mon:stdio`** — Headless operation with +serial console on stdio. Standard interface for kernel development +VMs. The serial output goes to the systemd journal via stdout +capture. VGA, SPICE, and VNC are not supported. Not +user-configurable. + +**`-nic user`** — User-mode networking (SLIRP). No host privileges +required. Port forwarding controlled via `ssh_port`. Alternative +host-guest transport via VSOCK (`vsock_cid`). TAP and bridge +networking are not supported (require root or network namespace +setup). The NIC model is determined by `machine_type`. +See: `man qemu-system`. + +**`-object memory-backend-memfd,share=on`** — Required by virtiofs. +virtiofsd accesses guest memory via shared memory mapping. Without +`share=on`, virtiofsd cannot map guest memory. Only emitted when +virtiofs shares are defined. Not user-configurable. + +**`LimitMEMLOCK=`** — Auto-computed as `<ram+256>M` when +`pci_passthrough` is defined. VFIO DMA mapping requires locked +memory. The 256M overhead covers QEMU allocations beyond guest RAM. +Override: + +```yaml +service: + LimitMEMLOCK: 8G +``` + +See: `man systemd.exec`. + +## virtiofsd + +**`--sandbox=`** — Set to `namespace` with `--uid-map :0:%U:1:` and +`--gid-map :0:%G:1:`. Provides PID, mount, network, and user +namespace isolation as an unprivileged user. The `--uid-map` maps +root inside the namespace to the service user outside (`%U`/`%G` are +systemd specifiers expanded at runtime). Without `--uid-map`, +virtiofsd warns "Couldn't set the process uid as root" because the +default 1-to-1 UID mapping does not include UID 0 (virtiofsd calls +`setresuid(0)` after creating the namespace). Override via per-share +env file: set `VIRTIOFSD_SANDBOX_ARGS=--sandbox=none`. See: +`/usr/libexec/virtiofsd --help`. + +**`--xattr`** — Extended attribute support. Required for correct +POSIX semantics (security labels, capabilities, ACLs). Not +user-configurable. See: `/usr/libexec/virtiofsd --help`. + +**`--no-announce-submounts`** — QEMU does not support submounts. +Prevents virtiofsd from announcing submount boundaries that the VMM +cannot handle. Not user-configurable. See: +`/usr/libexec/virtiofsd --help`. + +**`--fd=3`** — Socket activation. systemd passes the listening socket +as FD 3 per the `sd_listen_fds` protocol. Not user-configurable. +See: `man sd_listen_fds`. + +## Cloud-init + +### Network configuration gap + +Debian generic and genericcloud images ship with an empty +`/etc/netplan/` directory. They depend on cloud-init's fallback +network detection to generate netplan YAML at first boot. The +fallback scans all physical NICs and generates DHCP config for the +first one. This works when cloud-init boots with the image's own +distro kernel and initramfs. + +With direct kernel boot and a custom initramfs, the fallback is +unreliable. The symptom: `/etc/netplan/` stays empty, +`systemd-networkd-wait-online.service` hangs, the guest has no +network connectivity. + +The fix: provide a `network-config` file in the seed ISO. The +NoCloud datasource processes `network-config` before the fallback, +so it works regardless of boot mode. `files/network-config` is a +static file (no template rendering needed). Pass it to +`cloud-localds`: + +```shell +cloud-localds --network-config=files/network-config \ + images/seed.iso /tmp/user-data /tmp/meta-data +``` + +The network config matches both predictable interface names (`en*`) +and classic names (`eth*`). Custom kernels without full PCI sysfs +support or with `net.ifnames=0` on the command line leave interfaces +as `eth0`. Both patterns are needed. + +Images that do NOT need this fix: +nocloud (ships `/etc/netplan/90-default.yaml` baked in), +mkosi (ships `/etc/systemd/network/80-dhcp.network` baked in), +imageless (debootstrap includes network config). + +See: https://cloudinit.readthedocs.io/en/latest/reference/network-config-format-v2.html + +### user-data + +Infrastructure requirements for the VM to be usable with the +service templates. Apply only when `cloud_init` is defined. + +**`disable_root`**, **`PermitRootLogin`**, +**`PasswordAuthentication`** — Set to `false`, `yes`, `yes`. Root SSH +access required for non-interactive VM management (`ExecStop=` SSH +shutdown, automated provisioning). Not user-configurable. + +**`locale`** — The `locale:` directive uses cloud-init's built-in +locale module, which handles per-distro differences automatically. +Debian: writes `/etc/locale.gen` and runs `locale-gen`. Fedora: +writes `/etc/locale.conf`. No distro-specific `write_files` or +`packages` needed. + +**`growpart`** and **`resizefs`** — Cloud-init's built-in `growpart` +and `resizefs` modules run automatically (enabled in +`cloud_init_modules` by default on both Debian and Fedora). They +detect the root partition from the mount table and resize it to fill +the disk. No `runcmd` needed. Verified against upstream Fedora Cloud +image `/etc/cloud/cloud.cfg` which lists both modules. + +**`runcmd`** — Runs `touch /etc/cloud/cloud-init.disabled` only. +Disables cloud-init after first run to prevent re-provisioning on +reboot. Not user-configurable. See: `man cloud-init`. + +**`ssh_pwauth`** — Set to `true`. Password authentication fallback +for console access when no SSH key is configured. Not +user-configurable. + +## 9P + +**`security_model=`** — Set to `none`. Passes through file +permissions without UID/GID mapping. Correct for sharing host +directories where the user already owns the files. Other models +(mapped-xattr, passthrough) require root or change on-disk xattrs. +Not user-configurable. See: `man qemu-system`. + +**`multidevs=`** — Set to `remap`. Remaps device/inode numbers for +filesystems spanning multiple host devices. Prevents inode collisions +when sharing directories that contain mount points. Not +user-configurable. See: `man qemu-system`. diff --git a/scripts/qemu-system-units/docs/requirements.md b/scripts/qemu-system-units/docs/requirements.md new file mode 100644 index 00000000..74710093 --- /dev/null +++ b/scripts/qemu-system-units/docs/requirements.md @@ -0,0 +1,56 @@ +# Requirements + +## Debian + +```shell +sudo apt install qemu-system-x86 qemu-utils socat \ + systemd-container virtiofsd cloud-image-utils ovmf cargo +cargo install minijinja-cli +``` + +## Fedora + +```shell +sudo dnf install qemu-system-x86-core qemu-img socat \ + systemd-container virtiofsd cloud-utils-cloud-localds \ + edk2-ovmf cargo +cargo install minijinja-cli +``` + +## openSUSE + +```shell +sudo zypper install qemu-x86 qemu-img socat \ + systemd-container virtiofsd qemu-ovmf-x86_64 cargo +cargo install minijinja-cli +``` + +`cloud-localds` is not packaged on openSUSE. Use `genisoimage` +directly to create seed ISOs for cloud-init boot. + +## NixOS + +```nix +environment.systemPackages = with pkgs; [ + qemu socat virtiofsd minijinja cloud-utils OVMF +]; +``` + +## Groups + +```shell +sudo usermod --append --groups kvm,systemd-journal $(whoami) +``` + +`kvm` is required for `-accel kvm` (`/dev/kvm` access). +`systemd-journal` allows reading the journal without sudo. +Log out and back in for group changes to take effect. + +## Verify + +```shell +qemu-system-x86_64 --version +minijinja-cli --version +id --name --groups | grep kvm +ls -l --all /dev/kvm +``` diff --git a/scripts/qemu-system-units/docs/transient-units.md b/scripts/qemu-system-units/docs/transient-units.md new file mode 100644 index 00000000..f24c9353 --- /dev/null +++ b/scripts/qemu-system-units/docs/transient-units.md @@ -0,0 +1,244 @@ +# Transient and persistent units for QEMU VMs + +## Two patterns in systemd + +systemd provides two mechanisms for managing processes. Both vmspawn +and nspawn use both. + +### Persistent template service (on disk) + +A unit file lives on disk. systemd forks the process, manages its +lifecycle, resolves dependencies, captures logs. + +``` +units/user/[email protected]: +ExecStart=systemd-vmspawn --quiet --keep-unit --register=yes --network-tap --machine=%i +``` + +``` +units/[email protected]: +ExecStart=systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth -U --settings=override --machine=%i +``` + +Source: `systemd/units/user/[email protected]`, +`systemd/units/[email protected]`. + +### Transient scope (programmatic, in memory) + +The process starts first, then gets placed into a transient scope +via the `StartTransientUnit` D-Bus call. No file on disk. The scope +exists only while the process runs. + +```c +/* vmspawn-scope.c:57 */ +/* Creates a transient scope unit which tracks the lifetime of the current process */ +r = bus_message_new_method_call(bus, &m, bus_systemd_mgr, "StartTransientUnit"); +``` + +Source: `systemd/src/vmspawn/vmspawn-scope.c:57-67`, +`systemd/src/nspawn/nspawn.c:5634`. + +### The --keep-unit bridge + +When vmspawn or nspawn runs inside its service template, `--keep-unit` +tells it: "you are already in a service unit, do not create a scope on +top of it." When run directly from the terminal (without the service), +they create a transient scope. + +This is how both patterns coexist. The service template is the managed +path. The transient scope is the ad-hoc path. + +## systemd-run + +`systemd-run` is the CLI for creating transient units. It calls +`StartTransientUnit` on the D-Bus manager interface. + +Source: `systemd/man/systemd-run.xml`, "Run programs in transient +scope units, service units, or path-, socket-, or timer-triggered +service units." + +### Using our infrastructure with systemd-run + +A transient unit can depend on persistent units via `--property=`. +The deployed virtiofsd sockets, vfio-bind services, and +`EnvironmentFile=` paths are all reusable. + +Verified empirically. Transient unit referencing persistent socket: + +```shell +$ systemd-run --user --collect --no-block \ + --unit=transient-test \ + [email protected] \ + [email protected] \ + --property=Slice=machine.slice \ + --property=LimitMEMLOCK=2304M \ + sleep 30 + +$ systemctl --user show transient-test.service \ + | grep -E '^(Requires|After|Slice|LimitMEMLOCK)=' +Requires=machine.slice [email protected] basic.target +After=machine.slice [email protected] -.mount basic.target +Slice=machine.slice +LimitMEMLOCK=2415919104 +``` + +EnvironmentFile loading also works. `$QEMU_ARGS` from the rendered +env file is available inside the transient service: + +```shell +$ systemd-run --user --pty --collect \ + --property=EnvironmentFile=%E/systemd/qemu-system/debian.env \ + sh -c 'echo "QEMU_ARGS=$QEMU_ARGS" | head -c 200' +QEMU_ARGS= -machine type=q35 -accel kvm -cpu host -m 2048 ... +``` + +### Full interactive VM with transient-run template + +The `transient-run.sh.j2` template renders a self-contained script +that combines `--pty` for interactive console, `--property=` for +dependencies and `EnvironmentFile=`, and `sh -c` for shell word +splitting of `$QEMU_ARGS`: + +```shell +minijinja-cli --trim-blocks \ + --output /tmp/run-test.sh \ + templates/transient-run.sh.j2 vars/test.yaml +bash /tmp/run-test.sh +``` + +The script stops the persistent service if running (port conflicts), +sets up `QEMU_EXTRA_ARGS` with resolved paths, and calls +`systemd-run --user --pty` with all virtiofsd dependencies. + +Disconnect with `^]` pressed three times within 1 second. + +Source: `systemd/src/shared/ptyfwd.c:211-230`, `look_for_escape()` +checks for `0x1D` (Ctrl-]) three times within `ESCAPE_USEC` +(1 second). + +### Pitfalls discovered empirically + +**Do not pipe through bash.** `minijinja-cli ... | bash` steals +stdin from the terminal. `systemd-run --pty` needs the terminal's +stdin for interactive console forwarding. Always render to a file +first, then run with `bash /tmp/run-test.sh`. + +**systemd specifiers do not expand in transient Environment=.** +`%t` and `%i` expand in persistent unit files but are passed as +literal strings in `--property=Environment=` for transient units. +The template resolves paths at render time using `$XDG_RUNTIME_DIR` +(shell-expanded before `systemd-run` is called) and passes the +result via `--property=PassEnvironment=QEMU_EXTRA_ARGS`. + +Verified empirically: + +```shell +$ systemd-run --user --collect --no-block --unit=specifier-test \ + --property='Environment=TEST=%t/foo' \ + sh -c 'echo $TEST' +$ journalctl --user-unit=specifier-test --output=cat +%t/foo +``` + +**KERNEL_ARGS requires eval.** The rendered env file contains +`-append "console=ttyS0,115200 init=..."` with embedded quotes. +systemd's `EnvironmentFile=` parser preserves quotes as literal +characters. Without `eval`, the shell passes `"console=...` and +`init=..."` as separate arguments to QEMU. The template uses +`eval exec $QEMU_BINARY ...` to process the quotes correctly. + +### Why sh -c is required + +systemd-run passes command arguments as separate argv entries. +`$QEMU_ARGS` in the `EnvironmentFile=` is a single string with spaces. +Without `sh -c`, systemd would treat `$QEMU_ARGS` as one token. With +`sh -c`, the shell performs word splitting. + +In persistent services, `ExecStart=` handles this natively. systemd's +own command parser expands `$QEMU_ARGS` with word splitting. This is +specific to unit file parsing (`load-fragment.c`), not to the D-Bus +`StartTransientUnit` interface. + +## --pty and persistent services + +`--pty` is a `systemd-run` feature. It opens a PTY master +(`openpt_allocate` in `run.c:2476`), passes the slave FD as +`StandardInputFileDescriptor` / `StandardOutputFileDescriptor` to the +transient unit (`run.c:1501-1506`), and runs `PTYForward` to forward +between the local terminal and the service. + +Source: `systemd/src/run/run.c:1419-1506`, `systemd/src/shared/ptyfwd.c`. + +Persistent services have no controlling terminal by design. There is +no mechanism to retroactively attach a PTY to a running service. +`machinectl login` and `machinectl shell` work for containers (nspawn) +but not for VMs, since the guest OS is fully isolated. + +Source: `systemd/man/machinectl.xml`, login is "only supported for +containers running systemd as init system." + +### Interactive console for persistent VMs (virtconsole) + +The override template adds a virtio console on a unix socket to +every persistent VM. ttyS0 stays on stdio (captured by the journal). +hvc0 is a virtconsole on a socket for interactive access. + +QEMU flags (generated by the override template): + +``` +-device virtio-serial-pci +-chardev socket,id=console0,path=%t/qemu-system/%i/console.sock,server=on,wait=off +-device virtconsole,chardev=console0 +``` + +Connect: + +```shell +socat -,raw,echo=0,escape=0x1d \ + UNIX-CONNECT:$XDG_RUNTIME_DIR/qemu-system/test/console.sock +``` + +Disconnect with `Ctrl-]` (single press, `escape=0x1d`). The VM +keeps running. Reconnect at any time. + +Guest requirements: +- `console=hvc0` on the kernel command line (in addition to + `console=ttyS0`) +- A getty on hvc0 (e.g., `systemd.services."getty@hvc0"` in NixOS) +- `CONFIG_VIRTIO_CONSOLE=y` or `=m` in the guest kernel + +Source: QEMU `qemu-options.hx`, libvirt `virsh console` uses the +same virtio-serial + virtconsole pattern. + +Limitations: +- hvc0 only works after the guest kernel loads. BIOS output and + early boot messages are on ttyS0 (journal only). +- The initramfs emergency shell is on ttyS0, not hvc0. Use the + transient-run template (approach 3) for initramfs debugging. +- QEMU monitor (`Ctrl-a c`) is on ttyS0, not the virtconsole. + +## When to use each pattern + +| Use case | Mechanism | +|---|---| +| Long-lived, managed VMs | Persistent template (`systemctl start qemu-system@test`) | +| Console access to running VMs | `socat` to virtconsole socket (hvc0) | +| Interactive debugging (initramfs) | `transient-run.sh.j2` with `systemd-run --pty` (ttyS0) | +| CI / ephemeral VMs | `systemd-run --user --collect` (no `--pty`) | +| Quick test, no env file | `systemd-run --user --pty /usr/bin/qemu-system-x86_64 -m 2048 ...` | + +Both patterns place VMs in `machine.slice`, both can register with +machined, both reuse the deployed virtiofsd sockets and vfio-bind +services. The infrastructure is shared. + +## Proxmox comparison + +Proxmox uses transient scopes exclusively. The `qm start` command +forks, calls `enter_systemd_scope()` via D-Bus +`StartTransientUnit()`, places the QEMU process in `$VMID.scope` +under `qemu.slice` (not `machine.slice`). No persistent service +files. System scope only (root). + +Source: `git.proxmox.com/pve-common.git`, `src/PVE/Systemd.pm`, +`enter_systemd_scope()`. `git.proxmox.com/qemu-server.git` — +`PVE/QemuServer.pm`. diff --git a/scripts/qemu-system-units/docs/usage.md b/scripts/qemu-system-units/docs/usage.md new file mode 100644 index 00000000..177b310b --- /dev/null +++ b/scripts/qemu-system-units/docs/usage.md @@ -0,0 +1,378 @@ +# Usage + +All templates render with `minijinja-cli --trim-blocks` and one +vars file. Output goes to stdout by default; use `--output <path>` +to write directly to the deploy location. + +`vars/example.yaml` is the starting point and ships with the repo. +Copy it to create per-VM configurations. The filename should match +`vm_name` inside the file (e.g. `vars/test.yaml` for `vm_name: test`, +`vars/dev.yaml` for `vm_name: dev`). User vars files are gitignored. + +The examples below use `vars/test.yaml` (a copy of `vars/example.yaml` +which has `vm_name: test`). User scope deploys to `~/.config/systemd/`. + +## qemu-system + +Service template, per-instance drop-in, `EnvironmentFile=`, and QMP +powerdown commands. + +```shell +# Service template (deploy once) +mkdir --mode=0755 --parents ~/.config/systemd/user +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] \ + vars/test.yaml + +# Per-instance drop-in (one per VM) +mkdir --mode=0755 --parents ~/.config/systemd/user/[email protected] +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected]/override.conf \ + templates/qemu-system-override.conf.j2 \ + vars/test.yaml + +# QEMU environment file (one per VM) +mkdir --mode=0755 --parents ~/.config/systemd/qemu-system +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/qemu-system/test.env \ + templates/vm.env.j2 \ + vars/test.yaml + +# QMP powerdown commands (static file, deploy once) +cp files/qmp-powerdown ~/.config/systemd/qemu-system/qmp-powerdown +``` + +## virtiofsd + +Socket unit, socket-activated service, and per-share `EnvironmentFile=` +for host-guest directory sharing. + +```shell +# Socket and service templates (deploy once) +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] \ + vars/test.yaml + +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] \ + vars/test.yaml + +# Per-share environment (one per non-home share) +mkdir --mode=0755 --parents ~/.config/systemd/virtiofsd +minijinja-cli --trim-blocks \ + --define share_tag=modules \ + --output ~/.config/systemd/virtiofsd/test-modules.env \ + templates/virtiofsd.env.j2 \ + vars/test.yaml +``` + +## vfio + +`Type=oneshot` service for device binding and udev rules for unprivileged +access. One-time sudo required for module config, udev rules, and +rule reload. After setup, all VFIO operations run in user mode. + +```shell +# Bind service (deploy once) +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] \ + vars/test.yaml + +# One-time root setup +sudo cp files/vfio-pci.conf /etc/modules-load.d/vfio-pci.conf +sudo modprobe vfio-pci + +minijinja-cli --trim-blocks \ + templates/vfio-udev.rules.j2 \ + vars/test.yaml \ + | sudo tee /etc/udev/rules.d/10-vfio-kvm.rules + +sudo udevadm control --reload-rules +sudo udevadm trigger --subsystem-match=pci +``` + +## cloud-init + +user-data, meta-data, and network-config rendered into a seed ISO for +first-boot provisioning. Only needed for cloud image boot. + +The network-config is required when using direct kernel boot with +generic cloud images (cloud-init's fallback network detection is +unreliable without the distro initramfs). It is optional for cloud +image boot with the distro kernel. See +[design-decisions.md](design-decisions.md) for details. + +```shell +minijinja-cli --trim-blocks \ + --output /tmp/user-data \ + templates/user-data.j2 \ + vars/test.yaml + +minijinja-cli --trim-blocks \ + --output /tmp/meta-data \ + templates/meta-data.j2 \ + vars/test.yaml + +cloud-localds --network-config=files/network-config \ + images/seed.iso /tmp/user-data /tmp/meta-data +``` + +## Start + +```shell +systemctl --user daemon-reload +systemctl --user start qemu-system@test +systemctl --user status qemu-system@test +``` + +## machinectl + +VMs registered with systemd-machined appear alongside containers +in `machinectl`. Registration happens automatically via +`ExecStartPost=busctl RegisterMachine` in the service template. +User scope requires `--user` (systemd v259+). + +```shell +# List all registered machines +machinectl --user list + +# Machine details (PID, cgroup, service) +machinectl --user status test + +# Machine properties (parseable output) +machinectl --user show test + +# Emergency kill (not graceful, kills QEMU immediately) +machinectl --user terminate test +``` + +`machinectl login`, `shell`, `poweroff`, `bind`, and `copy-to/from` +are container-only. See: `man machinectl`, `man systemd-machined`. + +## Console + +Every persistent VM has a virtio console on a unix socket for +interactive access. The serial console (ttyS0) continues to capture +output in the systemd journal. + +```shell +socat -,raw,echo=0,escape=0x1d \ + UNIX-CONNECT:$XDG_RUNTIME_DIR/qemu-system/test/console.sock +``` + +Press Enter to get the login prompt. Disconnect with `Ctrl-]`. +The VM keeps running. Reconnect at any time. + +The guest needs `console=hvc0` on the kernel command line and a +getty on hvc0. The guest kernel needs `CONFIG_VIRTIO_CONSOLE=y` +or `=m`. + +For initramfs debugging (before hvc0 is available), use the +transient-run template: + +```shell +minijinja-cli --trim-blocks \ + --output /tmp/run-test.sh \ + templates/transient-run.sh.j2 vars/test.yaml +bash /tmp/run-test.sh +``` + +See [transient-units.md](transient-units.md) for details and +limitations. + +## Logs + +```shell +# VM service log +journalctl [email protected] + +# virtiofsd log (per share) +journalctl [email protected] + +# Follow live output +journalctl [email protected] --follow + +# All VM-related units since last boot +journalctl --user-unit='qemu-system@*' --user-unit='virtiofsd@*' --boot +``` + +## Stop + +Graceful shutdown via `ExecStop=`: QMP `system_powerdown` triggers +ACPI shutdown inside the guest. When `ssh_private_key` and +`vsock_cid` are configured, SSH shutdown is attempted first with +QMP as fallback. If the guest does not shut down within +`TimeoutSec=` (default 2min), systemd sends `SIGKILL`. + +```shell +# Graceful stop (ExecStop QMP powerdown, then SIGKILL after timeout) +systemctl --user stop qemu-system@test + +# Force kill (immediate SIGKILL, no graceful shutdown) +systemctl --user kill qemu-system@test --signal=SIGKILL + +# Stop all VMs (PartOf=machines.target) +systemctl --user stop machines.target + +# Reset failed state after a crash or forced kill +systemctl --user reset-failed qemu-system@test +``` + +## Multiple VMs + +The `[email protected]` template supports multiple instances. +Each VM gets its own vars file, env file, and drop-in. The service +template and virtiofsd units are shared. + +`vm_name`, `ssh_port`, and `vsock_cid` must be unique per VM. + +```shell +cp vars/example.yaml vars/dev.yaml +``` + +Edit `vars/dev.yaml`. E.g. set `vm_name: dev`, `ssh_port: 10023`, +`vsock_cid: 101`. + +```shell +# Render test VM +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/qemu-system/test.env \ + templates/vm.env.j2 vars/test.yaml +mkdir --mode=0755 --parents ~/.config/systemd/user/[email protected] +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected]/override.conf \ + templates/qemu-system-override.conf.j2 vars/test.yaml + +# Render dev VM +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/qemu-system/dev.env \ + templates/vm.env.j2 vars/dev.yaml +mkdir --mode=0755 --parents ~/.config/systemd/user/[email protected] +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected]/override.conf \ + templates/qemu-system-override.conf.j2 vars/dev.yaml + +systemctl --user daemon-reload +systemctl --user start qemu-system@test qemu-system@dev +machinectl --user list +``` + +A/B kernel testing: two vars files with different `kernel.image` +paths. Both VMs share the same service template. Only the env files +differ. + +## Cleanup + +### Remove a single VM + +Stop the VM and remove its per-instance files. Shared templates +and virtiofsd units stay for other VMs. + +```shell +systemctl --user stop qemu-system@test +rm ~/.config/systemd/qemu-system/test.env +rm --recursive --force ~/.config/systemd/user/[email protected] +rm --recursive --force ~/.config/systemd/virtiofsd/test-*.env +systemctl --user daemon-reload +``` + +### Remove everything + +Stop all VMs, remove all deployed units and configuration. + +```shell +systemctl --user stop machines.target +systemctl --user stop 'virtiofsd@*.socket' +rm --recursive --force \ + ~/.config/systemd/user/[email protected] \ + ~/.config/systemd/user/qemu-system@*.service.d \ + ~/.config/systemd/user/[email protected] \ + ~/.config/systemd/user/[email protected] \ + ~/.config/systemd/user/[email protected] \ + ~/.config/systemd/qemu-system \ + ~/.config/systemd/virtiofsd +systemctl --user daemon-reload +``` + +### Upgrading templates + +After re-deploying `[email protected]` with changed sandbox +settings, restart running virtiofsd sockets. `daemon-reload` +reloads unit definitions but does not restart running sockets. +Stale sockets may pass FDs in a state incompatible with the new +service configuration. + +```shell +systemctl --user daemon-reload +systemctl --user restart 'virtiofsd@*.socket' +``` + +## Deploy all + +Set `VARS` to your vars file. `VM` is derived from `vm_name` +inside it. + +```shell +VARS=vars/test.yaml +VM=$(minijinja-cli --template '{{ vm_name }}' '' $VARS) + +# Shared templates (deploy once) +mkdir --mode=0755 --parents ~/.config/systemd/user ~/.config/systemd/qemu-system ~/.config/systemd/virtiofsd +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] $VARS +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] $VARS +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/[email protected] \ + templates/[email protected] $VARS +cp files/qmp-powerdown ~/.config/systemd/qemu-system/qmp-powerdown + +# Per-VM files +mkdir --mode=0755 --parents ~/.config/systemd/user/qemu-system@${VM}.service.d +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/qemu-system/${VM}.env \ + templates/vm.env.j2 $VARS +minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/qemu-system@${VM}.service.d/override.conf \ + templates/qemu-system-override.conf.j2 $VARS + +# Per-share virtiofsd env (only for shares with dir: set) +# Repeat for each non-home share tag: +# minijinja-cli --trim-blocks \ +# --define share_tag=modules \ +# --output ~/.config/systemd/virtiofsd/${VM}-modules.env \ +# templates/virtiofsd.env.j2 $VARS + +systemctl --user daemon-reload +systemctl --user start qemu-system@${VM} +``` + +For multiple VMs, run the per-VM block for each vars file: + +```shell +for VARS in vars/test.yaml vars/dev.yaml; do + VM=$(minijinja-cli --template '{{ vm_name }}' '' $VARS) + minijinja-cli --trim-blocks \ + --output ~/.config/systemd/qemu-system/${VM}.env \ + templates/vm.env.j2 $VARS + mkdir --mode=0755 --parents ~/.config/systemd/user/qemu-system@${VM}.service.d + minijinja-cli --trim-blocks \ + --output ~/.config/systemd/user/qemu-system@${VM}.service.d/override.conf \ + templates/qemu-system-override.conf.j2 $VARS + # Per-share virtiofsd env (only for shares with dir: set) + # minijinja-cli --trim-blocks --define share_tag=modules \ + # --output ~/.config/systemd/virtiofsd/${VM}-modules.env \ + # templates/virtiofsd.env.j2 $VARS +done +systemctl --user daemon-reload +for VARS in vars/test.yaml vars/dev.yaml; do + VM=$(minijinja-cli --template '{{ vm_name }}' '' $VARS) + systemctl --user start qemu-system@${VM} +done +``` diff --git a/scripts/qemu-system-units/docs/vars.md b/scripts/qemu-system-units/docs/vars.md new file mode 100644 index 00000000..19084df2 --- /dev/null +++ b/scripts/qemu-system-units/docs/vars.md @@ -0,0 +1,381 @@ +# Variable reference + +One vars file per VM. All templates render from the same file. + +Variable names match upstream terminology when a 1:1 mapping exists +(e.g. `cpu` → `-cpu`, `accel` → `-accel`). When a variable maps to +a parameter inside a flag, the parameter name is used (e.g. `cpus` +→ `-smp cpus=N`). When a variable controls a higher-level concept +that spans multiple flags, a descriptive name is used (e.g. +`ssh_port`, `pci_passthrough`). See [design-decisions.md](design-decisions.md) for +naming rationale. + +Relative paths resolve against `WorkingDirectory=`, which defaults to +`$HOME` for user services and `/` for system services +(systemd.exec(5)). + +## VM identity + +**`vm_name`** — VM instance name. Used as the systemd template +instance specifier (`qemu-system@<vm_name>`), the machined +registration name, and the cloud-init `instance-id` / +`local-hostname`. Required. + +**`service_scope`** — `user` or `system`. Determines deploy paths +and unit behavior. User scope deploys to `~/.config/systemd/user/`, +needs no sudo. System scope deploys to `/etc/systemd/system/`, +requires root. Default: `user`. + +## QEMU + +**`qemu_binary`** — Absolute path to the QEMU system emulator +binary. Required. The name encodes the guest architecture +(`qemu-system-x86_64` for x86_64 guests, `qemu-system-aarch64` for +ARM64). Works on both Intel and AMD hosts; the name is the guest +architecture, not the host CPU vendor. Distros ship symlinks (`kvm`, +`qemu-system-amd64`, `qemu-kvm`) that all point to the same binary. +Use the upstream name. Maps to the `ExecStart=` binary path. +Example: `/usr/bin/qemu-system-x86_64`. + +**`cpu`** — CPU model. Maps to `-cpu`. Default: `host` (passthrough +host CPU features to guest). Common values: `host`, `max` (all +emulatable features, useful with TCG). List available models with +`<qemu_binary> -cpu help`. See: `man qemu-system`. + +**`accel`** — Accelerator. Maps to `-accel`. Default: `kvm`. Common +values: `kvm` (hardware virtualization), `tcg` (software emulation), +`hvf` (macOS Hypervisor.framework). When `iommu` is `intel-iommu` or +`amd-iommu`, the template auto-appends `kernel-irqchip=split` to +`-accel` (required for interrupt remapping). See: `man qemu-system`. + +**`ram`** — Guest memory in megabytes. Integer. Maps to `-m`. Also +used for `-object memory-backend-memfd,size=<ram>M` (virtiofs shared +memory) and `LimitMEMLOCK=<ram+256>M` (VFIO DMA mapping overhead, +see systemd.exec(5)). QEMU default: 128. Suffixes M, G accepted by +QEMU but the template passes the raw integer to `-m` which defaults +to megabytes. See: `man qemu-system`. + +**`cpus`** — Virtual CPU count. Integer. Maps to `-smp`. Default: 1. +See: `man qemu-system`. + +**`machine_type`** — QEMU machine type. Maps to `-machine type=`. +Common values: `q35` (x86_64, modern PCIe), `pc` (x86_64, legacy +ISA/PCI), `virt` (aarch64), `microvm` (x86_64, minimal). `q35` is +required for x86 IOMMUs and recommended for modern x86 guests. + +Machine properties can be appended after the type name: +`microvm,pcie=on,rtc=on`. QEMU parses these as machine options. + +microvm uses virtio-mmio transport instead of PCI. The template +auto-detects microvm from the machine_type value and switches to +`*-device` suffix (MMIO) for all virtio devices. NVMe, VFIO, and +IOMMU are PCI-only and require `pcie=on` on microvm. Without +`pcie=on`, these features are silently omitted. + +List available types with `<qemu_binary> -machine help`. Required. + +**`iommu`** — IOMMU device type. Maps to `-device <iommu>`. + +`intel-iommu`: Intel VT-d. Requires `-machine q35`. Auto-sets +`kernel-irqchip=split` on `-accel` (required for intremap). Adds +`caching-mode=on` when `pci_passthrough` is defined. + +`amd-iommu`: AMD-Vi. Requires `-machine q35`. Auto-sets +`kernel-irqchip=split`. Adds `dma-remap=on` when `pci_passthrough` +is defined. + +`virtio-iommu-pci`: virtio IOMMU. Works with `-machine q35` +(x86_64) and `-machine virt` (aarch64). No kernel-irqchip +requirement. + +`arm-smmuv3`: ARM SMMUv3. Requires `-machine virt` (aarch64 only). +No kernel-irqchip requirement. + +See: `man qemu-system`. + +**`firmware`** — Dict. UEFI firmware via pflash drives. Omit for +BIOS boot (SeaBIOS default on x86). Maps to two +`-drive if=pflash` entries. Supported on q35, i440fx, and ARM virt. +microvm does not support pflash. See: +[design-decisions.md](design-decisions.md) (QEMU firmware). + +**`firmware.code`** — Path to the shared firmware executable. +Read-only at runtime. Maps to +`-drive if=pflash,format=raw,readonly=on,file=`. x86_64: +`/usr/share/OVMF/OVMF_CODE_4M.fd` (`ovmf`). aarch64: +`/usr/share/AAVMF/AAVMF_CODE.fd` (`qemu-efi-aarch64`). +See: `man qemu-system`. + +**`firmware.vars`** — Path to the per-VM NVRAM file. Writable; each +VM needs its own file for independent UEFI variable state. Raw copy +or qcow2 overlay of `/usr/share/OVMF/OVMF_VARS_4M.fd`. Maps to +`-drive if=pflash,format=<vars_format>,file=`. See: +[design-decisions.md](design-decisions.md) (QEMU firmware). + +**`firmware.vars_format`** — NVRAM file format. Default: `raw`. Set +to `qcow2` when using a qcow2 overlay. Maps to `-drive format=`. +See: `man qemu-system`. + +**`gdb`** — Boolean. Maps to `-gdb unix:<socket>`. The socket is +placed in the systemd `RuntimeDirectory=`. Connect with: +`gdb vmlinux -ex "target remote <socket>"`. + +**`autostart`** — Boolean. Default: `true`. When `false`, maps to +`-S` (QEMU starts paused, waiting for GDB continue). + +## Root disk + +**`image`** — Dict. Root disk image configuration. Omit for imageless +boot (virtiofs rootfs). Maps to `-drive`. + +**`image.file`** — Path to the disk image. Relative paths resolve +against `WorkingDirectory=`. Required when `image` is defined. Maps +to `-drive file=`. See: `man qemu-system`, `man qemu-img`. + +**`image.format`** — Disk image container format. Default: `raw`. +Common values: `qcow2`, `raw`. QEMU can auto-detect but explicit is +safer. Maps to `-drive format=`. See: `man qemu-system`. + +**`image.cache`** — Block cache mode. Maps to `-drive cache=`. +Values: `writeback` (default), `none` (direct I/O, data integrity), +`writethrough`, `directsync`, `unsafe`. See: `man qemu-system`. + +**`image.aio`** — Asynchronous I/O mode. Maps to `-drive aio=`. +Values: `threads` (default), `native` (Linux AIO, requires +`cache=none` or `cache=directsync`), `io_uring`. +See: `man qemu-system`. + +**`image.discard`** — Discard (TRIM/UNMAP) support. Maps to +`-drive discard=`. Values: `ignore` (default), `unmap`. +See: `man qemu-system`. + +**`image.detect-zeroes`** — Zero write optimization. Maps to +`-drive detect-zeroes=`. Values: `off` (default), `on`, `unmap` +(convert zero writes to discard). See: `man qemu-system`. + +## Extra drives + +**`drives`** — List of additional virtio-blk drives. Each entry maps +to a `-drive` with the same backend properties as the root disk: +`file`, `format`, `cache`, `aio`, `discard`, `detect-zeroes`. + +## Networking + +**`ssh_port`** — Host TCP port forwarded to guest port 22. Integer. +Maps to `-nic hostfwd=tcp:127.0.0.1:<port>-:22`. Bound to localhost +only. Must be unique per VM. Omit to disable TCP port forwarding +(use VSOCK instead). + +**`vsock_cid`** — VSOCK Context ID for direct host-guest +communication. Integer, range 3-4294967295 (0=hypervisor, 1=local, +2=host). Must be unique per VM. Maps to +`-device vhost-vsock-pci,guest-cid=<cid>`. Also used by the graceful +shutdown `ExecStop=` which targets `root@vsock/<cid>`. +See: `man qemu-system`. + +**`ssh_private_key`** — Path to the SSH private key for +non-interactive VM access. Used by the graceful shutdown +`ExecStop=ssh -i`. Requires `vsock_cid`. + +## Direct kernel boot + +**`kernel`** — Dict. Presence enables direct kernel boot, bypassing +the disk image bootloader. + +**`kernel.image`** — Path to the kernel image (bzImage/vmlinuz). Maps +to `-kernel`. Required when `kernel` is defined. Example: +`/home/user/kernel/destdir/boot/vmlinuz-6.x.y`. +See: `man qemu-system`. + +**`kernel.append`** — Kernel command line. Maps to `-append`. +Example: `root=/dev/vda1 console=ttyS0,115200 rw`. For imageless +boot: `root=rootfs rootfstype=virtiofs`. + +**`kernel.initrd`** — Path to the initramfs image. Maps to +`-initrd`. Required for imageless boot (the init binary mounts root +before switch_root). Example: `images/initramfs.img`. + +## File sharing + +**`shares`** — List of host-guest directory shares. Each share +produces virtiofsd socket and service dependencies, cloud-init mount +entries, and either virtiofs chardev/device pairs or 9P fsdev/device +pairs depending on `share_transport`. + +Per-share properties: + +`tag`: mount tag visible inside the guest. Required. Used as part of +the virtiofsd instance name (`virtiofsd@<vm>-<tag>`). + +`mount`: guest mount point. Required. Rendered into cloud-init +`mounts:` entries. + +`dir`: host directory to share. Optional. When omitted, virtiofsd +serves the user's home directory (systemd `%h` specifier). When set, +overrides `VIRTIOFSD_SHARED_DIR` via a per-share `EnvironmentFile=`. + +`translate_uid`: virtiofsd UID mapping for imageless boot. Maps to +virtiofsd `--translate-uid`. Format: +`map:<guest-base>:<host-base>:<count>`. Example: `map:0:1000:1` maps +host UID 1000 to guest UID 0 (root). See: +`/usr/libexec/virtiofsd --help`. + +`translate_gid`: same as `translate_uid` but for GIDs. Maps to +virtiofsd `--translate-gid`. Defaults to `translate_uid` value when +omitted. + +**`virtiofsd_binary`** — Path to the virtiofsd binary. Default: +`/usr/libexec/virtiofsd`. Maps to the virtiofsd `ExecStart=` binary +path. + +**`share_transport`** — `virtiofs` (default) or `9p`. virtiofs uses +socket-activated virtiofsd daemons per share. 9P is built into QEMU +(`-fsdev` + `-device virtio-9p-pci`), no daemon, no socket +activation. When using 9P, `dir` is required for each share (no +implicit home default). See: kernel +`Documentation/filesystems/9p.rst`. + +## Cloud-init + +**`cloud_init`** — Dict. Presence enables cloud-init provisioning. +Omit the entire section for mkosi or imageless boot. When defined, +the seed ISO is attached as a `-drive` and cloud-init +user-data/meta-data are rendered. + +**`cloud_init.seed`** — Path to the cloud-init seed ISO. Maps to +`-drive file=<seed>,format=raw`. Generated by `cloud-localds` from +rendered user-data and meta-data. Always raw format (ISO9660 inside +raw container). See: `man cloud-localds`. + +**`cloud_init.locale`** — VM locale. Default: `en_US.UTF-8`. Maps to +cloud-init `locale:` and `write_files:` for `/etc/locale.gen`. + +**`cloud_init.ssh_pubkey`** — SSH public key. Applied to all users +via cloud-init `ssh_authorized_keys:`. Required for non-interactive +SSH access (password auth needs `/dev/tty`). + +**`cloud_init.users`** — List of user accounts to create. Each entry +maps to a cloud-init `users:` entry. Root is not special; include it +in the list with a password to enable root login. The template always +emits `disable_root: false` and a `PermitRootLogin yes` sshd drop-in +regardless of the user list. + +Per-user properties: + +`name`: username. Required. Maps to cloud-init `name`. + +`password`: plain text password. Default: same as `name`. Maps to +cloud-init `plain_text_passwd`. + +`groups`: comma-separated group list. Example: `sudo`. Maps to +cloud-init `groups`. + +`sudo`: sudo rule. Example: `ALL=(ALL) NOPASSWD:ALL`. Maps to +cloud-init `sudo`. + +`shell`: login shell path. Example: `/bin/bash`. Maps to cloud-init +`shell`. + +See: https://cloudinit.readthedocs.io/en/latest/reference/modules.html + +## PCIe passthrough + +**`pci_passthrough`** — List of host PCI devices to pass through via +VFIO. Maps to `-device vfio-pci,host=<addr>`. Also generates systemd +`Requires=vfio-bind@<addr>.service` dependencies and +`LimitMEMLOCK=<ram+256>M` (see systemd.exec(5)). + +Requires one-time root setup: deploy rendered udev rules and +`vfio-pci` module config. After setup, all operations run in user +mode. + +Per-device properties: + +`addr`: PCI address in domain:bus:device.function format. Required. +Example: `0000:2d:00.0`. Identify with `lspci -nv`. All devices in +the same IOMMU group must be passed through together. + +`opts`: additional `-device vfio-pci` properties. Example: +`rombar=0`. + +See: kernel `Documentation/driver-api/vfio.rst`. + +## NVMe + +**`nvme`** — Dict containing `drives` and `subsystems` lists. Maps +to `-device nvme`, `-device nvme-ns`, `-device nvme-subsys`, and +their associated `-drive` entries. + +Guest device paths: `/dev/nvme0n1`, `/dev/nvme1n1`, ... + +Stable names: +`/dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_<serial>_1`. + +`serial` is required by QEMU (errors with "serial property not set" +if omitted). The template defaults to `nvme0`, `nvme1`, etc. + +**`nvme.drives`** — List of NVMe controllers. Simple form (no +`namespaces` key) creates one controller with one implicit namespace. +Explicit `namespaces` key gives full per-namespace control. + +Drive backend properties (simple and explicit forms): `file`, +`format` (default: qcow2), `aio`, `cache`, `discard`, +`detect-zeroes`. + +Controller properties (maps to `-device nvme`): `serial`, +`max_ioqpairs`, `msix_qsize`, `mdts`, `vsl`, `cmb_size_mb`, +`ioeventfd`, `ocp`, `use-intel-id`, `dbcs`, `aerl`, +`aer_max_queued`, `mqes`, `legacy-cmb`, `ctratt.mem`, `atomic.dn`, +`atomic.awun`, `atomic.awupf`, `zoned.zasl`, `zoned.auto_transition`, +`opts`. + +BlockConf (simple form only, inherited by implicit namespace): +`logical_block_size`, `physical_block_size`, `min_io_size`, +`opt_io_size`, `discard_granularity`, `write-cache`, `share-rw`. + +Namespace properties (maps to `-device nvme-ns`): `nsid`, `uuid`, +`eui64`, `shared`, `detached`, `logical_block_size`, +`physical_block_size`, `min_io_size`, `opt_io_size`, +`discard_granularity`, `write-cache`, `share-rw`, `zoned`, +`zoned.zone_size`, `zoned.zone_capacity`, `zoned.max_active`, +`zoned.max_open`, `zoned.cross_read`, `zoned.descr_ext_size`, +`zoned.numzrwa`, `zoned.zrwas`, `zoned.zrwafg`, `ms`, `mset`, `pi`, +`pil`, `pif`, `mssrl`, `mcl`, `msrc`, `fdp.ruhs`, `atomic.nawun`, +`atomic.nawupf`, `atomic.nabsn`, `atomic.nabspf`, `atomic.nabo`, +`opts`. + +See: `<qemu_binary> -device nvme,help`, +`<qemu_binary> -device nvme-ns,help`, man qemu-system. + +**`nvme.subsystems`** — List of NVMe subsystems for multipath (shared +namespaces across controllers). Maps to `-device nvme-subsys`. + +Subsystem properties: `nqn`, `fdp`, `fdp.nrg`, `fdp.nruh`, +`fdp.runs`. + +Each subsystem contains `controllers` (list with per-controller +`serial`, `max_ioqpairs`, `msix_qsize`, etc.) and `namespaces` (list +with per-namespace properties, `shared: true` for multipath). + +See: `<qemu_binary> -device nvme-subsys,help`. + +## Service overrides + +**`service`** — Dict of systemd `[Service]` section directives. Any +systemd.exec(5), systemd.resource-control(5), or systemd.kill(5) +directive. Use `systemctl --user set-property` for runtime changes +without re-rendering. + +User scope (delegated: cpu, memory, pids): `CPUQuota=`, +`CPUWeight=`, `CPUQuotaPeriodSec=`, `MemoryMax=`, `MemoryHigh=`, +`MemoryMin=`, `MemorySwapMax=`, `TasksMax=`, `WorkingDirectory=`, +`LimitMEMLOCK=`, `LimitNOFILE=`, `Nice=`, `OOMScoreAdjust=`, +`TimeoutSec=`. + +System scope only (not delegated to user services): `IOWeight=`, +`IODeviceWeight=`, `IOReadBandwidthMax=`, `IOWriteBandwidthMax=`, +`AllowedCPUs=`, `AllowedMemoryNodes=`, `DeviceAllow=`, +`DevicePolicy=`. + +See: `man systemd.resource-control`, `man systemd.exec`, +`man systemd.kill`. diff --git a/scripts/qemu-system-units/files/network-config b/scripts/qemu-system-units/files/network-config new file mode 100644 index 00000000..bdb19dc2 --- /dev/null +++ b/scripts/qemu-system-units/files/network-config @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Cloud-init network config (NoCloud datasource, version 2). +# Matches both predictable names (enp0s2) and classic names (eth0). +# Custom kernels without full PCI sysfs support or with net.ifnames=0 +# leave interfaces as eth0. Both patterns are needed. +# See: https://cloudinit.readthedocs.io/en/latest/reference/network-config-format-v2.html +network: + version: 2 + ethernets: + all-en: + match: + name: en* + dhcp4: true + all-eth: + match: + name: eth* + dhcp4: true diff --git a/scripts/qemu-system-units/files/qmp-powerdown b/scripts/qemu-system-units/files/qmp-powerdown new file mode 100644 index 00000000..9569d474 --- /dev/null +++ b/scripts/qemu-system-units/files/qmp-powerdown @@ -0,0 +1,2 @@ +{"execute":"qmp_capabilities"} +{"execute":"system_powerdown"} diff --git a/scripts/qemu-system-units/files/vfio-pci.conf b/scripts/qemu-system-units/files/vfio-pci.conf new file mode 100644 index 00000000..7ce42142 --- /dev/null +++ b/scripts/qemu-system-units/files/vfio-pci.conf @@ -0,0 +1 @@ +vfio-pci diff --git a/scripts/qemu-system-units/templates/meta-data.j2 b/scripts/qemu-system-units/templates/meta-data.j2 new file mode 100644 index 00000000..a427836f --- /dev/null +++ b/scripts/qemu-system-units/templates/meta-data.j2 @@ -0,0 +1,3 @@ +{# SPDX-License-Identifier: copyleft-next-0.3.1 #} +instance-id: {{ vm_name }} +local-hostname: {{ vm_name }} diff --git a/scripts/qemu-system-units/templates/nvme.env.j2 b/scripts/qemu-system-units/templates/nvme.env.j2 new file mode 100644 index 00000000..1a22c5d5 --- /dev/null +++ b/scripts/qemu-system-units/templates/nvme.env.j2 @@ -0,0 +1,126 @@ +{#- SPDX-License-Identifier: copyleft-next-0.3.1 -#} +{#- NVMe device macros, included by vm.env.j2. -#} +{#- Each macro produces one QEMU argument (no trailing \ or whitespace). -#} +{#- See: qemu-system-x86_64 -device nvme,help -#} +{#- See: qemu-system-x86_64 -device nvme-ns,help -#} +{#- See: qemu-system-x86_64 -device nvme-subsys,help -#} +{#- -drive backend arguments for an NVMe image file. -#} +{% macro nvme_drive(d) -%} +file={{ d.file }},if=none,format={{ d.format | default('qcow2') }} +{%- if d.aio is defined %},aio={{ d.aio }}{% endif -%} +{%- if d.cache is defined %},cache={{ d.cache }}{% endif -%} +{%- if d.discard is defined %},discard={{ d.discard }}{% endif -%} +{%- if d['detect-zeroes'] is defined %},detect-zeroes={{ d['detect-zeroes'] }}{% endif -%} +{% endmacro -%} + +{#- -device nvme controller arguments. -#} +{#- In simple mode (drive= attached): also gets BlockConf properties. -#} +{#- In explicit mode (namespaces): no drive=, gets id= instead. -#} +{% macro nvme_ctrl(d, idx) -%} +nvme,serial={{ d.serial | default('nvme' ~ idx) }} +{%- if d.namespaces is not defined %},drive=nvme-drive-{{ idx }}{% endif -%} +{%- if d.namespaces is defined %},id=nvme{{ idx }}{% endif -%} +{%- if d.max_ioqpairs is defined %},max_ioqpairs={{ d.max_ioqpairs }}{% endif -%} +{%- if d.msix_qsize is defined %},msix_qsize={{ d.msix_qsize }}{% endif -%} +{%- if d.mdts is defined %},mdts={{ d.mdts }}{% endif -%} +{%- if d.vsl is defined %},vsl={{ d.vsl }}{% endif -%} +{%- if d.cmb_size_mb is defined %},cmb_size_mb={{ d.cmb_size_mb }}{% endif -%} +{%- if d['legacy-cmb'] | default(false) %},legacy-cmb=on{% endif -%} +{%- if d.ioeventfd | default(false) %},ioeventfd=on{% endif -%} +{%- if d.dbcs is defined %},dbcs={{ d.dbcs }}{% endif -%} +{%- if d.ocp | default(false) %},ocp=on{% endif -%} +{%- if d['use-intel-id'] | default(false) %},use-intel-id=on{% endif -%} +{%- if d.aerl is defined %},aerl={{ d.aerl }}{% endif -%} +{%- if d.aer_max_queued is defined %},aer_max_queued={{ d.aer_max_queued }}{% endif -%} +{%- if d.mqes is defined %},mqes={{ d.mqes }}{% endif -%} +{%- if d['zoned.zasl'] is defined %},zoned.zasl={{ d['zoned.zasl'] }}{% endif -%} +{%- if d['zoned.auto_transition'] is defined %},zoned.auto_transition={{ d['zoned.auto_transition'] }}{% endif -%} +{%- if d['ctratt.mem'] | default(false) %},ctratt.mem=on{% endif -%} +{%- if d['atomic.dn'] | default(false) %},atomic.dn=on{% endif -%} +{%- if d['atomic.awun'] is defined %},atomic.awun={{ d['atomic.awun'] }}{% endif -%} +{%- if d['atomic.awupf'] is defined %},atomic.awupf={{ d['atomic.awupf'] }}{% endif -%} +{#- BlockConf (simple form only, implicit namespace inherits these) -#} +{%- if d.namespaces is not defined -%} +{%- if d.logical_block_size is defined %},logical_block_size={{ d.logical_block_size }}{% endif -%} +{%- if d.physical_block_size is defined %},physical_block_size={{ d.physical_block_size }}{% endif -%} +{%- if d.min_io_size is defined %},min_io_size={{ d.min_io_size }}{% endif -%} +{%- if d.opt_io_size is defined %},opt_io_size={{ d.opt_io_size }}{% endif -%} +{%- if d.discard_granularity is defined %},discard_granularity={{ d.discard_granularity }}{% endif -%} +{%- if d['write-cache'] is defined %},write-cache={{ d['write-cache'] }}{% endif -%} +{%- if d['share-rw'] | default(false) %},share-rw=on{% endif -%} +{%- endif -%} +{%- if d.opts is defined %},{{ d.opts }}{% endif -%} +{% endmacro -%} + +{#- -device nvme-ns namespace arguments. -#} +{#- Full NVMe namespace properties: ZNS, metadata, PI, atomics, etc. -#} +{% macro nvme_ns(ns, ctrl_id, ns_idx) -%} +nvme-ns,drive={{ ctrl_id }}-ns{{ ns_idx }},bus={{ ctrl_id }} +{%- if ns.logical_block_size is defined %},logical_block_size={{ ns.logical_block_size }}{% endif -%} +{%- if ns.physical_block_size is defined %},physical_block_size={{ ns.physical_block_size }}{% endif -%} +{%- if ns.min_io_size is defined %},min_io_size={{ ns.min_io_size }}{% endif -%} +{%- if ns.opt_io_size is defined %},opt_io_size={{ ns.opt_io_size }}{% endif -%} +{%- if ns.discard_granularity is defined %},discard_granularity={{ ns.discard_granularity }}{% endif -%} +{%- if ns['write-cache'] is defined %},write-cache={{ ns['write-cache'] }}{% endif -%} +{%- if ns['share-rw'] | default(false) %},share-rw=on{% endif -%} +{%- if ns.nsid is defined %},nsid={{ ns.nsid }}{% endif -%} +{%- if ns.uuid is defined %},uuid={{ ns.uuid }}{% endif -%} +{%- if ns.eui64 is defined %},eui64={{ ns.eui64 }}{% endif -%} +{%- if ns.shared | default(false) %},shared=on{% endif -%} +{%- if ns.detached | default(false) %},detached=on{% endif -%} +{%- if ns.zoned | default(false) %},zoned=on +{%- if ns['zoned.zone_size'] is defined %},zoned.zone_size={{ ns['zoned.zone_size'] }}{% endif -%} +{%- if ns['zoned.zone_capacity'] is defined %},zoned.zone_capacity={{ ns['zoned.zone_capacity'] }}{% endif -%} +{%- if ns['zoned.max_active'] is defined %},zoned.max_active={{ ns['zoned.max_active'] }}{% endif -%} +{%- if ns['zoned.max_open'] is defined %},zoned.max_open={{ ns['zoned.max_open'] }}{% endif -%} +{%- if ns['zoned.cross_read'] | default(false) %},zoned.cross_read=on{% endif -%} +{%- if ns['zoned.descr_ext_size'] is defined %},zoned.descr_ext_size={{ ns['zoned.descr_ext_size'] }}{% endif -%} +{%- if ns['zoned.numzrwa'] is defined %},zoned.numzrwa={{ ns['zoned.numzrwa'] }}{% endif -%} +{%- if ns['zoned.zrwas'] is defined %},zoned.zrwas={{ ns['zoned.zrwas'] }}{% endif -%} +{%- if ns['zoned.zrwafg'] is defined %},zoned.zrwafg={{ ns['zoned.zrwafg'] }}{% endif -%} +{% endif -%} +{%- if ns.ms is defined %},ms={{ ns.ms }}{% endif -%} +{%- if ns.mset is defined %},mset={{ ns.mset }}{% endif -%} +{%- if ns.pi is defined %},pi={{ ns.pi }}{% endif -%} +{%- if ns.pil is defined %},pil={{ ns.pil }}{% endif -%} +{%- if ns.pif is defined %},pif={{ ns.pif }}{% endif -%} +{%- if ns.mssrl is defined %},mssrl={{ ns.mssrl }}{% endif -%} +{%- if ns.mcl is defined %},mcl={{ ns.mcl }}{% endif -%} +{%- if ns.msrc is defined %},msrc={{ ns.msrc }}{% endif -%} +{%- if ns['fdp.ruhs'] is defined %},fdp.ruhs={{ ns['fdp.ruhs'] }}{% endif -%} +{%- if ns['atomic.nawun'] is defined %},atomic.nawun={{ ns['atomic.nawun'] }}{% endif -%} +{%- if ns['atomic.nawupf'] is defined %},atomic.nawupf={{ ns['atomic.nawupf'] }}{% endif -%} +{%- if ns['atomic.nabsn'] is defined %},atomic.nabsn={{ ns['atomic.nabsn'] }}{% endif -%} +{%- if ns['atomic.nabspf'] is defined %},atomic.nabspf={{ ns['atomic.nabspf'] }}{% endif -%} +{%- if ns['atomic.nabo'] is defined %},atomic.nabo={{ ns['atomic.nabo'] }}{% endif -%} +{%- if ns.opts is defined %},{{ ns.opts }}{% endif -%} +{% endmacro -%} + +{#- -device nvme-subsys subsystem arguments. -#} +{% macro nvme_subsys(s, idx) -%} +nvme-subsys,id=nvme-subsys-{{ idx }} +{%- if s.nqn is defined %},nqn={{ s.nqn }}{% endif -%} +{%- if s.fdp | default(false) %},fdp=on +{%- if s['fdp.nrg'] is defined %},fdp.nrg={{ s['fdp.nrg'] }}{% endif -%} +{%- if s['fdp.nruh'] is defined %},fdp.nruh={{ s['fdp.nruh'] }}{% endif -%} +{%- if s['fdp.runs'] is defined %},fdp.runs={{ s['fdp.runs'] }}{% endif -%} +{% endif -%} +{% endmacro -%} + +{#- -device nvme controller in subsystem mode (with subsys= link). -#} +{% macro nvme_subsys_ctrl(ctrl, subsys_id, idx) -%} +nvme,id={{ subsys_id }}-ctrl{{ idx }},serial={{ ctrl.serial | default(subsys_id ~ '-ctrl' ~ idx) }},subsys={{ subsys_id }} +{%- if ctrl.max_ioqpairs is defined %},max_ioqpairs={{ ctrl.max_ioqpairs }}{% endif -%} +{%- if ctrl.msix_qsize is defined %},msix_qsize={{ ctrl.msix_qsize }}{% endif -%} +{%- if ctrl.mdts is defined %},mdts={{ ctrl.mdts }}{% endif -%} +{%- if ctrl.vsl is defined %},vsl={{ ctrl.vsl }}{% endif -%} +{%- if ctrl.cmb_size_mb is defined %},cmb_size_mb={{ ctrl.cmb_size_mb }}{% endif -%} +{%- if ctrl.ioeventfd | default(false) %},ioeventfd=on{% endif -%} +{%- if ctrl.ocp | default(false) %},ocp=on{% endif -%} +{%- if ctrl['zoned.zasl'] is defined %},zoned.zasl={{ ctrl['zoned.zasl'] }}{% endif -%} +{%- if ctrl['zoned.auto_transition'] is defined %},zoned.auto_transition={{ ctrl['zoned.auto_transition'] }}{% endif -%} +{%- if ctrl['atomic.dn'] | default(false) %},atomic.dn=on{% endif -%} +{%- if ctrl['atomic.awun'] is defined %},atomic.awun={{ ctrl['atomic.awun'] }}{% endif -%} +{%- if ctrl['atomic.awupf'] is defined %},atomic.awupf={{ ctrl['atomic.awupf'] }}{% endif -%} +{%- if ctrl.opts is defined %},{{ ctrl.opts }}{% endif -%} +{% endmacro -%} diff --git a/scripts/qemu-system-units/templates/qemu-system-override.conf.j2 b/scripts/qemu-system-units/templates/qemu-system-override.conf.j2 new file mode 100644 index 00000000..438d2aa6 --- /dev/null +++ b/scripts/qemu-system-units/templates/qemu-system-override.conf.j2 @@ -0,0 +1,83 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Per-instance drop-in for qemu-system@{{ vm_name }}.service +# Rendered from: templates/qemu-system-override.conf.j2 +# +# Deploy: +# mkdir --parents ~/.config/systemd/user/qemu-system@{{ vm_name }}.service.d +# cp <rendered> ~/.config/systemd/user/qemu-system@{{ vm_name }}.service.d/override.conf + +[Unit] +{% if share_transport | default('virtiofs') != '9p' %} +{% for share in shares | default([]) %} +Requires=virtiofsd@%i-{{ share.tag }}.socket +After=virtiofsd@%i-{{ share.tag }}.socket +BindsTo=virtiofsd@%i-{{ share.tag }}.service +{% endfor %} +{% endif %} +{% for dev in pci_passthrough | default([]) %} +Requires=vfio-bind@{{ dev.addr }}.service +After=vfio-bind@{{ dev.addr }}.service +{% endfor %} + +[Service] +{% if qemu_binary is defined %} +ExecStart= +ExecStart={{ qemu_binary }} $QEMU_ARGS $QEMU_EXTRA_ARGS \ + $KERNEL_ARGS \ + -qmp unix:%t/qemu-system/%i/qmp.sock,server=on,wait=off +{% endif %} +{# QEMU_EXTRA_ARGS: single-line output, {{- " " }} injects spaces between flags #} +{% set has_virtiofs = shares | default([]) | length > 0 and share_transport | default('virtiofs') != '9p' %} +{% set microvm = "microvm" in machine_type %} +Environment="QEMU_EXTRA_ARGS= +{%- if has_virtiofs -%} +{{- " " }}-object memory-backend-memfd,id=mem,size={{ ram }}M,share=on +{{- " " }}-machine memory-backend=mem +{%- for share in shares -%} +{{- " " }}-chardev socket,id=virtiofs_{{ share.tag }},path=%t/virtiofsd/%i-{{ share.tag }}/virtiofsd.sock +{{- " " }}-device {{ "vhost-user-fs-device" if microvm else "vhost-user-fs-pci" }},queue-size=1024,chardev=virtiofs_{{ share.tag }},tag={{ share.tag }} +{%- endfor -%} +{%- endif -%} +{{- " " }}-device {{ "virtio-serial-device" if microvm else "virtio-serial-pci" }} +{{- " " }}-chardev socket,id=console0,path=%t/qemu-system/%i/console.sock,server=on,wait=off +{{- " " }}-device virtconsole,chardev=console0 +{%- if gdb | default(false) -%} +{{- " " }}-gdb unix:%t/qemu-system/%i/gdb.sock +{%- endif -%} +{%- if not autostart | default(true) -%} +{{- " " }}-S +{%- endif -%} +" +{# Override the base registration with one that carries vSockCid. + See docs/design-decisions.md "machined registration". #} +{% if vsock_cid is defined %} +{% if service_scope | default("user") == "user" %} +{% set varlink_socket = "/run/user/%U/systemd/machine/io.systemd.Machine" %} +{% else %} +{% set varlink_socket = "/run/systemd/machine/io.systemd.Machine" %} +{% endif %} +ExecStartPost= +ExecStartPost=-varlinkctl call \ + {{ varlink_socket }} \ + io.systemd.Machine.Register \ + "{\"name\":\"%i\",\"class\":\"vm\",\"service\":\"qemu-system\",\"leader\":${MAINPID},\"vSockCid\":{{ vsock_cid }}}" +{% endif %} +{# Empty ExecStop= clears base, then SSH (clean) + QMP (fallback) #} +{% if ssh_private_key is defined %} +ExecStop= +ExecStop=-ssh -o ConnectTimeout=5 \ + -o UserKnownHostsFile=/dev/null \ + -o StrictHostKeyChecking=no \ + -o IdentitiesOnly=yes \ + -i ${SSH_KEY_PATH} \ + root@vsock/${VSOCK_CID} \ + systemctl poweroff +ExecStop=-socat OPEN:%E/systemd/qemu-system/qmp-powerdown!!OPEN:/dev/null,wronly UNIX-CONNECT:%t/qemu-system/%i/qmp.sock +{% endif %} +{% if pci_passthrough is defined %} +LimitMEMLOCK={{ ram + 256 }}M +{% endif %} +{% for key, value in service | default({}) | items %} +{{ key }}={{ value }} +{% endfor %} diff --git a/scripts/qemu-system-units/templates/[email protected] b/scripts/qemu-system-units/templates/[email protected] new file mode 100644 index 00000000..01a21760 --- /dev/null +++ b/scripts/qemu-system-units/templates/[email protected] @@ -0,0 +1,56 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# QEMU VM template service +# Rendered from: templates/[email protected] +# +# Scope: {{ service_scope | default('user') }} +{% if service_scope | default('user') == 'system' %} +# Deploy: sudo cp <rendered> /etc/systemd/system/[email protected] +# Usage: sudo systemctl start qemu-system@<name> +# Logs: journalctl --unit=qemu-system@<name>.service +{% else %} +# Deploy: mkdir --parents ~/.config/systemd/user && cp <rendered> ~/.config/systemd/user/[email protected] +# Usage: systemctl --user start qemu-system@<name> +# Logs: journalctl --user-unit=qemu-system@<name>.service +{% endif %} + +[Unit] +Description=QEMU System Virtual Machine %i +PartOf=machines.target +Before=machines.target +{% if service_scope | default('user') == 'system' %} +After=network.target +{% endif %} + +[Service] +Type=simple +ConfigurationDirectory=systemd/qemu-system +RuntimeDirectory=qemu-system/%i +EnvironmentFile=%E/systemd/qemu-system/%i.env +Environment=QEMU_EXTRA_ARGS= +ExecStart={{ qemu_binary }} $QEMU_ARGS $QEMU_EXTRA_ARGS \ + $KERNEL_ARGS \ + -qmp unix:%t/qemu-system/%i/qmp.sock,server=on,wait=off +{# Register with machined over Varlink. See docs/design-decisions.md + "machined registration" for why Varlink (and not busctl). The + shared template registers without vSockCid because it does not + know per-VM values; the per-VM drop-in in + templates/qemu-system-override.conf.j2 overrides this when + vsock_cid is set in the vars file. #} +{% if service_scope | default("user") == "user" %} +{% set varlink_socket = "/run/user/%U/systemd/machine/io.systemd.Machine" %} +{% else %} +{% set varlink_socket = "/run/systemd/machine/io.systemd.Machine" %} +{% endif %} +ExecStartPost=-varlinkctl call \ + {{ varlink_socket }} \ + io.systemd.Machine.Register \ + "{\"name\":\"%i\",\"class\":\"vm\",\"service\":\"qemu-system\",\"leader\":${MAINPID}}" +ExecStop=-socat OPEN:%E/systemd/qemu-system/qmp-powerdown!!OPEN:/dev/null,wronly UNIX-CONNECT:%t/qemu-system/%i/qmp.sock +KillSignal=SIGCONT +KillMode=mixed +TimeoutSec=2min +Slice=machine.slice + +[Install] +WantedBy=machines.target diff --git a/scripts/qemu-system-units/templates/transient-run.sh.j2 b/scripts/qemu-system-units/templates/transient-run.sh.j2 new file mode 100644 index 00000000..0b660a31 --- /dev/null +++ b/scripts/qemu-system-units/templates/transient-run.sh.j2 @@ -0,0 +1,72 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Interactive QEMU session for {{ vm_name }}. +# Rendered from: templates/transient-run.sh.j2 +# +# Render and run: +# minijinja-cli --trim-blocks \ +# --output /tmp/run-{{ vm_name }}.sh \ +# templates/transient-run.sh.j2 vars/{{ vm_name }}.yaml +# bash /tmp/run-{{ vm_name }}.sh +# +# Disconnect: press Ctrl-] three times within 1 second. +# The VM terminates when the session disconnects. +# +# Do not pipe through bash (| bash). The pipe steals stdin from the +# terminal, which systemd-run --pty needs for interactive console. + +set -euo pipefail + +unit=qemu-{{ vm_name }}-interactive + +# Stop the persistent service if running (same ports, same shares). +if systemctl --user is-active --quiet qemu-system@{{ vm_name }}.service 2>/dev/null; then + echo "Stopping persistent service qemu-system@{{ vm_name }}..." >&2 + systemctl --user stop qemu-system@{{ vm_name }}.service +fi + +# Stop a previous interactive session if still running. +if systemctl --user is-active --quiet "$unit".service 2>/dev/null; then + echo "Stopping previous interactive session $unit..." >&2 + systemctl --user stop "$unit".service +fi + +{% set has_virtiofs = shares | default([]) | length > 0 and share_transport | default('virtiofs') != '9p' %} +{#- QEMU_EXTRA_ARGS is built in the shell, not in Environment=, because + systemd does not expand %t or $XDG_RUNTIME_DIR in Environment= + directives of transient units. The shell resolves $XDG_RUNTIME_DIR + at exec time. -#} +{% if has_virtiofs %} +QEMU_EXTRA_ARGS=" +{%- for share in shares -%} +{%- if not loop.first %} {% endif %}-chardev socket,id=virtiofs_{{ share.tag }},path=$XDG_RUNTIME_DIR/virtiofsd/{{ vm_name }}-{{ share.tag }}/virtiofsd.sock +{{- " " }}-device {{ "vhost-user-fs-device" if "microvm" in machine_type else "vhost-user-fs-pci" }},queue-size=1024,chardev=virtiofs_{{ share.tag }},tag={{ share.tag }} +{%- endfor -%} +{{- " " }}-object memory-backend-memfd,id=mem,size={{ ram }}M,share=on +{{- " " }}-machine memory-backend=mem" +{% else %} +QEMU_EXTRA_ARGS="" +{% endif %} +export QEMU_EXTRA_ARGS + +exec systemd-run --user --pty --collect \ + --unit="$unit" \ + --property=EnvironmentFile="$HOME"/.config/systemd/qemu-system/{{ vm_name }}.env \ + --property=PassEnvironment=QEMU_EXTRA_ARGS \ +{% if has_virtiofs %} +{% for share in shares %} + --property=Requires=virtiofsd@{{ vm_name }}-{{ share.tag }}.socket \ + --property=After=virtiofsd@{{ vm_name }}-{{ share.tag }}.socket \ + --property=BindsTo=virtiofsd@{{ vm_name }}-{{ share.tag }}.service \ +{% endfor %} +{% endif %} +{% if gdb | default(false) %} + --property="Environment=QEMU_GDB_ARGS=-gdb unix:$XDG_RUNTIME_DIR/qemu-system/{{ vm_name }}/gdb.sock" \ +{% endif %} +{% if pci_passthrough is defined %} + --property=LimitMEMLOCK={{ ram + 256 }}M \ +{% endif %} +{% if service is defined and service.WorkingDirectory is defined %} + --property=WorkingDirectory={{ service.WorkingDirectory }} \ +{% endif %} + sh -c 'eval exec $QEMU_BINARY $QEMU_ARGS $QEMU_EXTRA_ARGS ${QEMU_GDB_ARGS:-} $KERNEL_ARGS' diff --git a/scripts/qemu-system-units/templates/user-data.j2 b/scripts/qemu-system-units/templates/user-data.j2 new file mode 100644 index 00000000..e9805c3a --- /dev/null +++ b/scripts/qemu-system-units/templates/user-data.j2 @@ -0,0 +1,44 @@ +{# SPDX-License-Identifier: copyleft-next-0.3.1 #} +#cloud-config +locale: {{ cloud_init.locale | default('en_US.UTF-8') if cloud_init is defined else 'en_US.UTF-8' }} +disable_root: false +ssh_pwauth: true +{% if cloud_init is defined and cloud_init.users is defined and cloud_init.users | length > 0 %} +users: +{% for u in cloud_init.users %} + - name: {{ u.name }} + lock_passwd: false + plain_text_passwd: {{ u.password | default(u.name) }} +{% if u.groups is defined %} + groups: {{ u.groups }} +{% endif %} +{% if u.sudo is defined %} + sudo: {{ u.sudo }} +{% endif %} +{% if u.shell is defined %} + shell: {{ u.shell }} +{% endif %} +{% if cloud_init.ssh_pubkey is defined %} + ssh_authorized_keys: + - {{ cloud_init.ssh_pubkey }} +{% endif %} +{% endfor %} +{% endif %} +{% if shares | default([]) | length > 0 %} +mounts: +{% for share in shares %} +{% if share_transport | default('virtiofs') == '9p' %} + - [ "{{ share.tag }}", "{{ share.mount }}", "9p", "trans=virtio,version=9p2000.L,nofail", "0", "0" ] +{% else %} + - [ "{{ share.tag }}", "{{ share.mount }}", "virtiofs", "defaults,nofail", "0", "0" ] +{% endif %} +{% endfor %} +{% endif %} +write_files: + - path: /etc/ssh/sshd_config.d/99-permit-root-login.conf + permissions: '0600' + content: | + PermitRootLogin yes + PasswordAuthentication yes +runcmd: + - touch /etc/cloud/cloud-init.disabled diff --git a/scripts/qemu-system-units/templates/[email protected] b/scripts/qemu-system-units/templates/[email protected] new file mode 100644 index 00000000..c8e0b3f0 --- /dev/null +++ b/scripts/qemu-system-units/templates/[email protected] @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# VFIO PCI device binding service +# Rendered from: templates/[email protected] +# +# Deploy: mkdir --parents ~/.config/systemd/user && cp <rendered> ~/.config/systemd/user/[email protected] +# Usage: systemctl --user start vfio-bind@0000:2d:00.0 +# Requires: udev rules deployed (templates/vfio-udev.rules.j2), vfio-pci module loaded + +[Unit] +Description=VFIO PCI bind for %I + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=sh -c 'echo vfio-pci > /sys/bus/pci/devices/%I/driver_override' +ExecStart=-sh -c 'echo %I > /sys/bus/pci/devices/%I/driver/unbind' +ExecStart=sh -c 'echo %I > /sys/bus/pci/drivers_probe' +ExecStop=sh -c 'echo > /sys/bus/pci/devices/%I/driver_override' +ExecStop=-sh -c 'echo %I > /sys/bus/pci/devices/%I/driver/unbind' +ExecStop=sh -c 'echo %I > /sys/bus/pci/drivers_probe' diff --git a/scripts/qemu-system-units/templates/vfio-udev.rules.j2 b/scripts/qemu-system-units/templates/vfio-udev.rules.j2 new file mode 100644 index 00000000..fbe980fb --- /dev/null +++ b/scripts/qemu-system-units/templates/vfio-udev.rules.j2 @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# VFIO device node permissions, allow kvm group access +# Rendered from: templates/vfio-udev.rules.j2 +# Deploy: sudo tee /etc/udev/rules.d/10-vfio-kvm.rules +# Reload: sudo udevadm control --reload-rules && sudo udevadm trigger --subsystem-match=pci +SUBSYSTEM=="vfio", OWNER="root", GROUP="kvm", MODE="0660" +{% for dev in pci_passthrough | default([]) %} + +# Per-device sysfs permissions for unprivileged VFIO binding: {{ dev.addr }} +ACTION=="add", SUBSYSTEM=="pci", KERNEL=="{{ dev.addr }}", \ + RUN+="/bin/sh -c '\ + chgrp kvm /sys/bus/pci/devices/{{ dev.addr }}/driver_override; \ + chmod 664 /sys/bus/pci/devices/{{ dev.addr }}/driver_override; \ + chgrp kvm /sys/bus/pci/drivers_probe; \ + chmod 220 /sys/bus/pci/drivers_probe'" +ACTION=="bind", SUBSYSTEM=="pci", KERNEL=="{{ dev.addr }}", \ + RUN+="/bin/sh -c '\ + chgrp kvm /sys/bus/pci/devices/{{ dev.addr }}/driver/unbind; \ + chmod 220 /sys/bus/pci/devices/{{ dev.addr }}/driver/unbind'" +{% endfor %} diff --git a/scripts/qemu-system-units/templates/virtiofsd.env.j2 b/scripts/qemu-system-units/templates/virtiofsd.env.j2 new file mode 100644 index 00000000..7c8a3b25 --- /dev/null +++ b/scripts/qemu-system-units/templates/virtiofsd.env.j2 @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# virtiofsd environment for {{ vm_name }}-{{ share_tag }} +# Rendered from: templates/virtiofsd.env.j2 --define share_tag={{ share_tag }} +{% for s in shares if s.tag == share_tag %} +VIRTIOFSD_SHARED_DIR={{ s.dir }} +{% if s.translate_uid is defined %} +VIRTIOFSD_EXTRA_OPTS=--translate-uid {{ s.translate_uid }} --translate-gid {{ s.translate_gid | default(s.translate_uid) }} +{% endif %} +{% endfor %} diff --git a/scripts/qemu-system-units/templates/[email protected] b/scripts/qemu-system-units/templates/[email protected] new file mode 100644 index 00000000..e32f449b --- /dev/null +++ b/scripts/qemu-system-units/templates/[email protected] @@ -0,0 +1,40 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# virtiofsd service for QEMU VM instance (socket-activated) +# Rendered from: templates/[email protected] +# +# Instance name encodes VM and share: virtiofsd@<vm>-<tag>.service +# Example: [email protected] +# +# Default shared directory is the user's home (%h specifier). Non-home +# shares get a rendered env file (from templates/virtiofsd.env.j2) that +# overrides VIRTIOFSD_SHARED_DIR. +# +# Default sandbox is namespace with UID/GID mapping (%U/%G specifiers). +# This provides PID, mount, network, and user namespace isolation. +# Shares with translate_uid add --translate-uid via VIRTIOFSD_EXTRA_OPTS +# (both mechanisms coexist: namespace maps virtiofsd's process identity, +# translate-uid maps file ownership to the guest). +# Override VIRTIOFSD_SANDBOX_ARGS in the env file for --sandbox=none. +# +# This service is started by systemd socket activation when QEMU connects +# to the virtiofsd socket. Do not start it directly; start the .socket +# unit instead. + +[Unit] +Description=virtiofsd for %i + +[Service] +Type=simple +ConfigurationDirectory=systemd/virtiofsd +Environment="VIRTIOFSD_SANDBOX_ARGS=--sandbox=namespace --uid-map :0:%U:1: --gid-map :0:%G:1:" +Environment=VIRTIOFSD_SHARED_DIR=%h +Environment=VIRTIOFSD_EXTRA_OPTS= +EnvironmentFile=-%E/systemd/virtiofsd/%i.env +ExecStart={{ virtiofsd_binary | default('/usr/libexec/virtiofsd') }} \ + $VIRTIOFSD_SANDBOX_ARGS \ + --shared-dir=${VIRTIOFSD_SHARED_DIR} \ + --fd=3 \ + --xattr \ + --no-announce-submounts \ + $VIRTIOFSD_EXTRA_OPTS diff --git a/scripts/qemu-system-units/templates/[email protected] b/scripts/qemu-system-units/templates/[email protected] new file mode 100644 index 00000000..960df3b8 --- /dev/null +++ b/scripts/qemu-system-units/templates/[email protected] @@ -0,0 +1,24 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# virtiofsd socket for QEMU VM instance +# Rendered from: templates/[email protected] +# +# Instance name encodes VM and share: virtiofsd@<vm>-<tag>.socket +# Example: [email protected] +# +{% if service_scope | default('user') == 'system' %} +# Deploy: sudo cp <rendered> /etc/systemd/system/[email protected] +{% else %} +# Deploy: cp <rendered> ~/.config/systemd/user/[email protected] +{% endif %} + +[Unit] +Description=virtiofsd socket for %i + +[Socket] +ListenStream=%t/virtiofsd/%i/virtiofsd.sock +SocketMode=0660 +RuntimeDirectory=virtiofsd/%i + +[Install] +WantedBy=sockets.target diff --git a/scripts/qemu-system-units/templates/vm.env.j2 b/scripts/qemu-system-units/templates/vm.env.j2 new file mode 100644 index 00000000..54adef45 --- /dev/null +++ b/scripts/qemu-system-units/templates/vm.env.j2 @@ -0,0 +1,144 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +{# Jinja2 include must appear before macros are called #} +{% include "nvme.env.j2" %} +{#- Machine detection: microvm uses virtio-mmio, all others use PCI. + microvm with pcie=on has both transports; we use MMIO for virtio + devices and PCI for NVMe/VFIO/IOMMU. -#} +{% set microvm = "microvm" in machine_type %} +{% set has_pcie = not microvm or "pcie" in machine_type %} +{#- Device names: MMIO uses *-device, PCI uses *-pci-non-transitional. + vhost-user-fs has no non-transitional variant (modern-only). -#} +{% set dev_net = "virtio-net-device" if microvm else "virtio-net-pci-non-transitional" %} +{% set dev_rng = "virtio-rng-device" if microvm else "virtio-rng-pci-non-transitional" %} +{% set dev_vsock = "vhost-vsock-device" if microvm else "vhost-vsock-pci-non-transitional" %} +{% set dev_9p = "virtio-9p-device" if microvm else "virtio-9p-pci-non-transitional" %} +{% set dev_fs = "vhost-user-fs-device" if microvm else "vhost-user-fs-pci" %} +{#- Drive backend: -drive arguments shared by root disk, seed, and extra drives. + microvm: if=none with explicit -device virtio-blk-device after each drive. + PCI: if=virtio (implicit virtio-blk-pci, no separate -device needed). -#} +{% macro drive_backend(d, drive_id) -%} +file={{ d.file }}, +{%- if microvm -%} +if=none,id={{ drive_id }}, +{%- else -%} +if=virtio, +{%- endif -%} +format={{ d.format | default('raw') }} +{%- if d.cache is defined %},cache={{ d.cache }}{% endif -%} +{%- if d.aio is defined %},aio={{ d.aio }}{% endif -%} +{%- if d.discard is defined %},discard={{ d.discard }}{% endif -%} +{%- if d['detect-zeroes'] is defined %},detect-zeroes={{ d['detect-zeroes'] }}{% endif -%} +{% endmacro -%} +# VM environment for {{ vm_name }} +# Rendered from vars file. Source this, then run: $QEMU_BINARY $QEMU_ARGS +QEMU_BINARY={{ qemu_binary }} +QEMU_ARGS="\ + -machine type={{ machine_type }} \ +{% if iommu is defined and iommu in ['intel-iommu', 'amd-iommu'] %} + -accel {{ accel | default('kvm') }},kernel-irqchip=split \ +{% else %} + -accel {{ accel | default('kvm') }} \ +{% endif %} + -cpu {{ cpu | default('host') }} \ + -m {{ ram }} \ + -smp {{ cpus }} \ +{% if has_pcie and iommu is defined %} +{% if iommu == 'intel-iommu' %} + -device intel-iommu,intremap=on{% if pci_passthrough is defined %},caching-mode=on{% endif %} \ +{% elif iommu == 'amd-iommu' %} + -device amd-iommu,intremap=on{% if pci_passthrough is defined %},dma-remap=on{% endif %} \ +{% elif iommu == 'virtio-iommu-pci' %} + -device virtio-iommu-pci \ +{% elif iommu == 'arm-smmuv3' %} + -device arm-smmuv3 \ +{% endif %} +{% endif %} +{% if firmware is defined %} + -drive if=pflash,format=raw,readonly=on,file={{ firmware.code }} \ + -drive if=pflash,format={{ firmware.vars_format | default('raw') }},file={{ firmware.vars }} \ +{% endif %} +{% if image is defined %} + -drive {{ drive_backend(image, 'root') }} \ +{% if microvm %} + -device virtio-blk-device,drive=root \ +{% endif %} +{% endif %} +{% if cloud_init is defined and cloud_init.seed is defined %} + -drive {{ drive_backend({'file': cloud_init.seed, 'format': 'raw'}, 'seed') }} \ +{% if microvm %} + -device virtio-blk-device,drive=seed \ +{% endif %} +{% endif %} +{% for d in drives | default([]) %} + -drive {{ drive_backend(d, 'drive' ~ loop.index0) }} \ +{% if microvm %} + -device virtio-blk-device,drive=drive{{ loop.index0 }} \ +{% endif %} +{% endfor %} +{% if microvm %} + -netdev user,id=net0{% if ssh_port is defined %},hostfwd=tcp:127.0.0.1:{{ ssh_port }}-:22{% endif %} \ + -device {{ dev_net }},netdev=net0 \ +{% else %} + -nic user,model={{ dev_net }}{% if ssh_port is defined %},hostfwd=tcp:127.0.0.1:{{ ssh_port }}-:22{% endif %} \ +{% endif %} +{% if vsock_cid is defined %} + -device {{ dev_vsock }},guest-cid={{ vsock_cid }} \ +{% endif %} +{% if has_pcie %} +{% for dev in pci_passthrough | default([]) %} + -device vfio-pci,host={{ dev.addr }}{% if dev.opts is defined %},{{ dev.opts }}{% endif %} \ +{% endfor %} +{% endif %} +{% if share_transport | default('virtiofs') == '9p' %} +{% for share in shares | default([]) %} + -fsdev local,id=fsdev_{{ share.tag }},path={{ share.dir }},security_model=none,multidevs=remap \ + -device {{ dev_9p }},fsdev=fsdev_{{ share.tag }},mount_tag={{ share.tag }} \ +{% endfor %} +{% endif %} +{% if has_pcie %} +{# NVMe subsystems (multipath, shared namespaces across controllers) #} +{% for subsys in (nvme.subsystems if nvme is defined) | default([]) %} +{% set subsys_id = 'nvme-subsys-' ~ loop.index0 %} + -device {{ nvme_subsys(subsys, loop.index0) }} \ +{% for ctrl in subsys.controllers | default([]) %} + -device {{ nvme_subsys_ctrl(ctrl, subsys_id, loop.index0) }} \ +{% endfor %} +{% for ns in subsys.namespaces | default([]) %} + -drive {{ nvme_drive(ns) }},id={{ subsys_id }}-ctrl0-ns{{ loop.index0 }} \ + -device {{ nvme_ns(ns, subsys_id ~ '-ctrl0', loop.index0) }} \ +{% endfor %} +{% endfor %} +{# NVMe drives (1 controller per drive, or explicit namespaces) #} +{% for drive in (nvme.drives if nvme is defined) | default([]) %} +{% if drive.namespaces is defined %} +{% set ctrl_id = 'nvme' ~ loop.index0 %} + -device {{ nvme_ctrl(drive, loop.index0) }} \ +{% for ns in drive.namespaces %} + -drive {{ nvme_drive(ns) }},id={{ ctrl_id }}-ns{{ loop.index0 }} \ + -device {{ nvme_ns(ns, ctrl_id, loop.index0) }} \ +{% endfor %} +{% else %} + -drive {{ nvme_drive(drive) }},id=nvme-drive-{{ loop.index0 }} \ + -device {{ nvme_ctrl(drive, loop.index0) }} \ +{% endif %} +{% endfor %} +{% endif %} + -device {{ dev_rng }} \ + -nographic \ + -serial mon:stdio" +{% if vsock_cid is defined %} +VSOCK_CID={{ vsock_cid }} +{% endif %} +{% if ssh_private_key is defined %} +SSH_KEY_PATH={{ ssh_private_key }} +{% endif %} +{% if kernel is defined %} +KERNEL_ARGS=-kernel {{ kernel.image }} +{%- if kernel.append is defined %} + -append "{{ kernel.append }}" +{%- endif %} +{%- if kernel.initrd is defined %} + -initrd {{ kernel.initrd }} +{%- endif %} + +{% endif %} \ No newline at end of file diff --git a/scripts/qemu-system-units/vars/example.yaml b/scripts/qemu-system-units/vars/example.yaml new file mode 100644 index 00000000..0469a248 --- /dev/null +++ b/scripts/qemu-system-units/vars/example.yaml @@ -0,0 +1,105 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# VM configuration. See docs/vars.md for field reference. + +vm_name: test +service_scope: user + +# QEMU +qemu_binary: /usr/bin/qemu-system-x86_64 +cpu: host +accel: kvm +ram: 2048 +cpus: 2 +machine_type: q35 +# gdb: true +# autostart: false +# iommu: intel-iommu + +# Firmware (UEFI boot; omit for SeaBIOS default) +# firmware: +# code: /usr/share/OVMF/OVMF_CODE_4M.fd +# vars: images/test-ovmf-vars.fd +# vars_format: raw + +# Root disk +image: + file: images/test.qcow2 + format: qcow2 + # cache: none + # aio: native + # discard: unmap + # detect-zeroes: unmap + +# Extra drives (virtio-blk) +# drives: +# - file: images/data.qcow2 +# format: qcow2 +# discard: unmap +# - file: images/scratch.raw +# format: raw + +# Networking +ssh_port: 10022 +# vsock_cid: 100 +# ssh_private_key: ~/.ssh/id_ed25519 + +# Direct kernel boot +# kernel: +# image: /home/user/kernel/destdir/boot/vmlinuz-6.x.y +# append: root=/dev/vda1 console=ttyS0,115200 rw +# initrd: /home/user/kernel/destdir/boot/initramfs-6.x.y.img + +# File sharing (virtiofs default, 9P fallback) +shares: + - tag: home + mount: /mnt/home +# - tag: modules +# dir: /home/user/kernel/destdir/lib/modules +# mount: /lib/modules +# virtiofsd_binary: /usr/libexec/virtiofsd +# share_transport: 9p + +# Cloud-init (omit entire section for mkosi or imageless boot) +# cloud_init: +# seed: images/seed.iso +# locale: en_US.UTF-8 +# ssh_pubkey: ssh-ed25519 AAAA... user@host +# users: +# - name: root +# password: root +# - name: user +# password: user +# groups: sudo +# sudo: ALL=(ALL) NOPASSWD:ALL + +# PCIe passthrough via VFIO +# pci_passthrough: +# - addr: "0000:2d:00.0" +# - addr: "0000:03:00.0" +# opts: "rombar=0" + +# NVMe emulated drives +# nvme: +# drives: +# - file: images/nvme0.qcow2 +# - file: images/nvme1.qcow2 +# format: qcow2 +# serial: data-drive +# subsystems: +# - nqn: subsys0 +# controllers: +# - serial: path-a +# - serial: path-b +# namespaces: +# - file: images/shared.raw +# format: raw +# shared: true + +# [Service] section overrides +# service: +# CPUQuota: 200% +# MemoryMax: 4G +# TasksMax: 512 +# TimeoutSec: 5min +# WorkingDirectory: /home/user/machines -- 2.53.0