NetBSD/amd64 current performance patch

During the past couple of months I've been working on reducing
NetBSD/amd64 virtual machine kernel boot time to the minimum.
Today, on a not-so-modern i5-7600K, using qemu 6.2 with the
microvm[1] machine type on a Linux Mint 21.2 host, NetBSD kernel
can boot in less than 15ms; this is from locore.S (new) PVH
entry point to just before handing over to userland in
init_main.c. Before the performances additions, the same path
took around 350ms.

https://imil.net/NetBSD/tslog-perf.svg
vs
https://imil.net/NetBSD/tslog-noperf.svg

The micro vm uses the following trimmed down kernel configuration:

# start of kernel configuration

machine amd64 x86 xen
include         "conf/std"      # MI standard options
include         "arch/xen/conf/std.xenversion"
options         CPU_IN_CKSUM
options         EXEC_ELF64      # exec ELF binaries
options         EXEC_SCRIPT     # exec #! scripts
options         MTRR
options         MULTIPROCESSOR
options         CHILD_MAX=1024  # 160 is too few
options         OPEN_MAX=1024   # 128 is too few
mainbus0 at root
cpu* at mainbus?
ioapic* at mainbus? apid ?
options         INCLUDE_CONFIG_FILE     # embed config file in kernel binary
maxusers        8               # estimated number of users
options         INSECURE        # disable kernel security levels - X 
needs this
options         RTC_OFFSET=0    # hardware clock is this many mins. west 
of GMT
options         PIPE_SOCKETPAIR # smaller, but slower pipe(2)
options         XENPVHVM
options         XEN
hypervisor*     at mainbus?             # Xen hypervisor
xenbus*         at hypervisor?          # Xen virtual bus
xencons*        at hypervisor?          # Xen virtual console
makeoptions     COPTS="-O2 -fno-omit-frame-pointer"
file-system FFS
file-system KERNFS
options         FFS_NO_SNAPSHOT # No FFS snapshot support
options         WAPBL           # File system journaling support
options         INET            # IP + ICMP + TCP + UDP
options         INET6           # IPV6
config          netbsd  root on ? type ?

# some microvms don't have ACPI
options         MPBIOS          # configure CPUs and APICs using MPBIOS
options         MPTABLE_LINUX_BUG_COMPAT # fix to locate correct ACPI 
location
isa0    at mainbus?
com0    at isa? port 0x3f8 irq 4        # Standard PC serial ports

# new pv bus for MMIO backed devices

pv* at pvbus?
pvclock* at pv?     # pvclock uses KVM capabilities
virtio* at pv?      # virtio attaches to pv with MMIO
ld*     at virtio?  # Virtio disk device
vioif*  at virtio?  # Virtio network device
viornd* at virtio?  # Virtio entropy device
viocon* at virtio?

pseudo-device   bpfilter
pseudo-device   loop
pseudo-device   pty

# new

options         TSLOG    # enable tslog(4) tracing facility
options         BOOTTIME # prints boot time relative to rdtsc

# end of kernel configuration

The "performance branch" is available here: 
https://github.com/NetBSDfr/NetBSD-src/tree/perf and implements the 
following features:

- Generic PVH boot
- A new PV bus for hypervisors without PCI
- MMIO backed devices (no PCI needed!)
- Various performance fixes
- pvclock to use KVM timecounter
- TSLOG framework from FreeBSD to trace performance

You can try it using qemu either on Linux with KVM:

qemu-system-x86_64 -M microvm,x-option-roms=off,rtc=on,acpi=off,pic=off 
-enable-kvm -m 128 -cpu host,+invtsc -kernel ${KERNEL} -append 
"root=ld0a console=com rw -z" -display none -device 
virtio-blk-device,drive=hd0 -drive file=${IMG},format=raw,id=hd0 -global 
virtio-mmio.force-legacy=false -serial stdio

Or NetBSD using NVMM (slower):

qemu-system-x86_64 -M microvm,x-option-roms=off,rtc=off,acpi=off,pic=off 
-accel nvmm -m 128 -cpu host -kernel ${KERNEL} -append "root=ld0a 
console=com rw -z" -serial stdio -display none -device 
virtio-blk-device,drive=hd0 -drive file=${IMG},format=raw,id=hd0 -global 
virtio-mmio.force-legacy=false

after setting up KERNEL and IMG variables accordingly.

This kernel can also boot with a yet to be merged PVH branch of
AWS's Firecracker:

https://github.com/firecracker-microvm/firecracker/tree/feature/pvh

Here's a basic configuration for this one:

$ cat vmconfig.json
{
   "boot-source": {
     "kernel_image_path": "netbsd-perf",
     "boot_args": "console=com root=ld0a -z"
   },
   "drives": [
     {
       "drive_id": "rootfs",
       "path_on_host": "disk.img",
       "is_root_device": false,
       "is_read_only": false
     }
   ],
   "machine-config": {
     "vcpu_count": 1,
     "mem_size_mib": 128
   }
}

then start it this way:

firecracker --no-api --config-file vmconfig.json

Firecracker must be killed to terminate the vm in --no-api mode.

For anyone wanting to try this out without compiling, I've uploaded a
kernel and minimal root disk at https://imil.net/NetBSD/netbsd-perf
and https://imil.net/NetBSD/disk.img

Feedback as always very welcome.

Cheers,

[1]: https://www.qemu.org/docs/master/system/i386/microvm.html

-- 
------------------------------------------------------------------------
Emile `iMil' Heitor <imil@{home.imil.net,NetBSD.org}> | https://imil.net