Re: [PATCH v5 2/2] crosvm-image-minimal: add a reference image for crosvm demo

Hi Keerthivasan,

Inline review of 2/2. Same caveat as before — if any of these are
addressed in v1, please ignore.

On Thu, May 28, 2026 at 00:40 +0530, Keerthivasan Raghavan wrote:
> A reference image called `crosvm-image-minimal` is provided
> for testing/validation on supported machines(arm64 and x86-64).
> Instructions are provided to test on QEMU(x86-64 only).
> 
> Testing was done on x86-64 by achieving boot to shell on a crosvm vm
> instance running inside a vm instance on QEMU, i.e nested virtualization
> with machine as qemux86-64.
> Testing was done on arm64 by achieving boot to shell on a crosvm vm
> instance running directly on meta-qcom machine qcs9100-ride-sx.
> 
> Signed-off-by: Keerthivasan Raghavan <[email protected]>
> ---
>  recipes-extended/images/README-crosvm.md      | 68 +++++++++++++++++++
>  .../images/crosvm-image-minimal.bb            | 68 +++++++++++++++++++
>  2 files changed, 136 insertions(+)
>  create mode 100644 recipes-extended/images/README-crosvm.md
>  create mode 100644 recipes-extended/images/crosvm-image-minimal.bb
> 
> diff --git a/recipes-extended/images/README-crosvm.md b/recipes-extended/images/README-crosvm.md
> new file mode 100644
> index 00000000..2c36da7d
> --- /dev/null
> +++ b/recipes-extended/images/README-crosvm.md
> @@ -0,0 +1,68 @@
> +# crosvm-image-minimal: build and test
> +
> +Steps to test/validate crosvm are given below.
> +A reference image `crosvm-image-minimal` is provided
> +for this purpose.
> +
> +## local.conf example
> +
> +```conf
> +# virtualization
> +MACHINE ??= "qemux86-64"
> +DISTRO ??= "nodistro"
> +BBMULTICONFIG ?= ""
> +
> +DISTRO_FEATURES += " kvm virtualization"
> +IMAGE_FEATURES += " empty-root-password allow-empty-password allow-root-login"
> +IMAGE_FSTYPES = "ext4"
> +IMAGE_FSTYPES:remove = "ext4.zst"
> +```
> +
> +## Build
> +
> +```bash
> +bitbake crosvm-image-minimal
> +```
> +
> +## Boot in QEMU
> +
> +When the target `MACHINE` is `qemux86-64`, the host running QEMU must support:
> +- nested virtualization
> +- Intel VT-x or AMD SVM
> +- exposed `/dev/kvm`
> +
> +Run from `tmp/deploy/images/qemux86-64/` (or adjust paths):
> +
> +```bash
> +qemu-system-x86_64 \
> +  -enable-kvm \
> +  -machine q35 \
> +  -cpu host \
> +  -smp 4 \
> +  -m 4096 \
> +  -kernel bzImage \
> +  -drive file=crosvm-image-minimal-qemux86-64.rootfs.ext4,format=raw,if=virtio \
> +  -append "root=/dev/vda rw console=ttyS0 earlyprintk=serial nokaslr" \
> +  -nographic
> +```
> +
> +## Run crosvm inside the guest
> +
> +The image bundles:
> +
> +- `/var/lib/crosvm/images/guest-kernel`
> +- `/var/lib/crosvm/images/guest.ext4`
> +
> +```bash
> +crosvm --log-level "debug,disk=off" run \
> +  --disable-sandbox  /var/lib/crosvm/images/guest-kernel \
> +  --block /var/lib/crosvm/images/guest.ext4,root \
> +  -p "root=/dev/vda rw console=ttyS0 earlyprintk=serial nokaslr"
> +```
> +
> +## Quick sanity checks
> +
> +```bash
> +test -e /dev/kvm && echo "/dev/kvm present"
> +ls -l /var/lib/crosvm/images/
> +```
> diff --git a/recipes-extended/images/crosvm-image-minimal.bb b/recipes-extended/images/crosvm-image-minimal.bb
> new file mode 100644
> index 00000000..de503a99
> --- /dev/null
> +++ b/recipes-extended/images/crosvm-image-minimal.bb
> @@ -0,0 +1,68 @@
> +SUMMARY = "A minimal image containing crosvm to demo its uage."
> +
> +KVM_MODULES = ""
> +
> +KVM_MODULES:x86-64 = " \
> +    kernel-module-kvm \
> +    kernel-module-kvm-intel \
> +    kernel-module-kvm-amd \
> +"
> +
> +IMAGE_INSTALL = "packagegroup-core-boot crosvm"
> +IMAGE_INSTALL:append = " ${KVM_MODULES}"
> +
> +IMAGE_LINGUAS = " "
> +
> +LICENSE = "MIT"
> +
> +inherit core-image
> +
> +# Guest artifact source settings (can be overridden in local.conf).
> +CROSVM_GUEST_IMAGE_RECIPE ?= "core-image-minimal"
> +CROSVM_GUEST_IMAGE_FSTYPE ?= "ext4"
> +CROSVM_GUEST_ROOTFS ?= "${CROSVM_GUEST_IMAGE_RECIPE}-${MACHINE}.rootfs.${CROSVM_GUEST_IMAGE_FSTYPE}"
> +CROSVM_GUEST_KERNEL ?= "${KERNEL_IMAGETYPE}"
> +
> +# Final location inside the host/root image.
> +CROSVM_GUEST_TARGET_DIR ?= "/var/lib/crosvm/images"
> +CROSVM_GUEST_TARGET_ROOTFS_NAME ?= "guest.${CROSVM_GUEST_IMAGE_FSTYPE}"
> +CROSVM_GUEST_TARGET_KERNEL_NAME ?= "guest-kernel"
> +
> +# Ensure guest image and kernel are available before rootfs postprocess runs.
> +# The same kernel is used for both the host and guest.
> +do_rootfs[depends] += "${CROSVM_GUEST_IMAGE_RECIPE}:do_image_complete virtual/kernel:do_deploy"
> +
> +# Copy guest rootfs+kernel into the host rootfs for crosvm demo runtime.
> +bundle_crosvm_guest() {
> +    set -e
> +
> +    local deploy_dir="${DEPLOY_DIR_IMAGE}"
> +    local rootfs_src="${deploy_dir}/${CROSVM_GUEST_ROOTFS}"
> +    local kernel_src="${deploy_dir}/${CROSVM_GUEST_KERNEL}"
> +    local target_dir="${IMAGE_ROOTFS}${CROSVM_GUEST_TARGET_DIR}"
> +
> +    if [ ! -e "$rootfs_src" ]; then
> +        bbfatal "Cannot find guest rootfs. Tried: \
> +                ${deploy_dir}/${CROSVM_GUEST_ROOTFS} and \
> +                ${deploy_dir}/${CROSVM_GUEST_IMAGE_RECIPE}-${MACHINE}.rootfs.${CROSVM_GUEST_IMAGE_FSTYPE}"
> +    fi
> +
> +    if [ ! -e "$kernel_src" ]; then
> +        bbfatal "Cannot find guest kernel. Tried: \
> +                ${deploy_dir}/${CROSVM_GUEST_KERNEL}"
> +    fi
> +
> +    install -d "$target_dir"
> +
> +    install -m 0644 "$(readlink -f "$rootfs_src")" \
> +                    "$target_dir/${CROSVM_GUEST_TARGET_ROOTFS_NAME}"
> +
> +    install -m 0644 "$(readlink -f "$kernel_src")" \
> +                    "$target_dir/${CROSVM_GUEST_TARGET_KERNEL_NAME}"
> +}
> +
> +ROOTFS_POSTPROCESS_COMMAND += "bundle_crosvm_guest;"
> +
> +IMAGE_ROOTFS_SIZE = "0"
> +
> +KERNEL_MODULE_AUTOLOAD:append:x86-64 = " kvm kvm_amd kvm_intel"

A few observations:

1. README is appreciated; I asked for this on v4 and it's here.
   Skimmed it, looks fine. Will check it more carefully once I'm
   actually building.

2. Test coverage gap

> Testing was done on x86-64 by achieving boot to shell on a crosvm vm
> instance running inside a vm instance on QEMU, i.e nested virtualization
> with machine as qemux86-64.
> Testing was done on arm64 by achieving boot to shell on a crosvm vm
> instance running directly on meta-qcom machine qcs9100-ride-sx.

Good progress over v4 — x86_64 nested under qemu lets me actually try
it locally, which is what I needed. Thank you for that.

The remaining gap is qemuarm64. I can't reproduce qcs9100-ride-sx
without the hardware, so the arm64 path is effectively un-testable
from my side right now. Did you try qemuarm64 with nested
virtualization (the way you did qemux86-64), and if so what
happened? Even a "I tried, it didn't boot because X" would tell me
whether we need to ask upstream for qemuarm64 support or whether it
just needs some Yocto config wiring.

3. Image recipe — bundle_crosvm_guest postprocess

> +# Ensure guest image and kernel are available before rootfs postprocess runs.
> +# The same kernel is used for both the host and guest.
> +do_rootfs[depends] += "${CROSVM_GUEST_IMAGE_RECIPE}:do_image_complete virtual/kernel:do_deploy"

> +bundle_crosvm_guest() {
> +    set -e
[...]
> +    install -m 0644 "$(readlink -f "$rootfs_src")" \
> +                    "$target_dir/${CROSVM_GUEST_TARGET_ROOTFS_NAME}"
[...]
> +ROOTFS_POSTPROCESS_COMMAND += "bundle_crosvm_guest;"

This is the same pattern we use elsewhere (container-cross-install
does similar bundle-into-rootfs work). Worth being aware that
DEPLOY_DIR_IMAGE-as-input has the sstate sharp-edge we documented
recently — the recipe that produces ${CROSVM_GUEST_ROOTFS} can hit
sstate while its deployed output is stale from a prior run, and
then this image hits sstate with the stale guest baked in. Same
class of bug we fixed for vcontainer-tarball with explicit
mcdepends on rootfs-image:do_image_complete (commit 6fc1aa83
on master).

Not necessarily blocking — the do_rootfs[depends] line is doing
the right thing for ordering — but worth being aware of for when
this becomes a CI build and stale-guest reports start appearing.

4. KVM modules layout

> +KVM_MODULES = ""
> +
> +KVM_MODULES:x86-64 = " \
> +    kernel-module-kvm \
> +    kernel-module-kvm-intel \
> +    kernel-module-kvm-amd \
> +"
[...]
> +KERNEL_MODULE_AUTOLOAD:append:x86-64 = " kvm kvm_amd kvm_intel"

x86_64 is handled. arm64 is empty (KVM_MODULES = "" with no
:aarch64 override). On arm64, KVM is built into the kernel and
not a loadable module on most boards — so an empty list is right.
But should we add explicit a :aarch64 = "" comment or some
documentation so the reader doesn't wonder if it's a bug?

5. CROSVM_GUEST_IMAGE_RECIPE default

> +CROSVM_GUEST_IMAGE_RECIPE ?= "core-image-minimal"

Sensible default. The PARSE-TIME do_rootfs[depends] += that uses
this variable means changing it in local.conf actually works
correctly (because += happens at parse time, before the dep is
read). Worth a comment in the recipe to that effect so users know
they can override without surprises.

Bruce