Bug#1132330: cloud.debian.org: arm64 boot failures following bookworm -> trixie upgrade

Package: cloud.debian.org
Severity: important
X-Debbugs-Cc: [email protected]
User: [email protected]
Usertags: arm64

Azure cloud VMs fail to cold boot following an upgrade from bookworm to trixie.
Azure supports arm64 and amd64 architectures, but this issue only seems to
occur on arm64.  It does not seem to occur on other arm64 cloud environments
(tested with Amazon EC2).

The problem can be worked around by passing the `--force-extra-removable` to
grub-install.  This can be done after the upgrade (but before rebooting), or
it'll happen automatically if the grub2/force_efi_extra_removable variable
to true before the upgrade, e.g.

echo "grub-efi-arm64  grub2/force_efi_extra_removable boolean true" | \
  sudo debconf-set-selections

With this option, grub-install updates the content of /boot/efi/EFI/BOOT,
which otherwise does not get updated.

# state of /boot/efi following bookworm -> trixie upgrade. In this
# configuration, hard reboot of the VM fails to load grub.
noahm@temp-520f:~$ find /boot/efi/ -type f -ls
        5    966 -rwxr-xr-x   1 root     root       987432 Mar 30 15:05 /boot/efi/EFI/debian/shimaa64.efi
        6   2790 -rwxr-xr-x   1 root     root      2856384 Mar 30 15:05 /boot/efi/EFI/debian/grubaa64.efi
        7    868 -rwxr-xr-x   1 root     root       887472 Mar 30 15:05 /boot/efi/EFI/debian/mmaa64.efi
        8     90 -rwxr-xr-x   1 root     root        90752 Mar 30 15:05 /boot/efi/EFI/debian/fbaa64.efi
        9      2 -rwxr-xr-x   1 root     root          110 Mar 30 15:05 /boot/efi/EFI/debian/BOOTAA64.CSV
       10      2 -rwxr-xr-x   1 root     root          126 Mar 30 15:05 /boot/efi/EFI/debian/grub.cfg
       16    968 -rwxr-xr-x   1 root     root       990600 Mar 16 05:02 /boot/efi/EFI/BOOT/BOOTAA64.EFI
       17   4258 -rwxr-xr-x   1 root     root      4359616 Mar 16 05:02 /boot/efi/EFI/BOOT/grubaa64.efi
       18    868 -rwxr-xr-x   1 root     root       887472 Mar 16 05:02 /boot/efi/EFI/BOOT/mmaa64.efi
noahm@temp-520f:~$ find /boot/efi/ -type f | xargs -n1 md5sum
4a5bdc18d7a879274abd54f665687ea7  /boot/efi/EFI/debian/shimaa64.efi
b7d3dd296477665343aad1f933bae866  /boot/efi/EFI/debian/grubaa64.efi
469f860cb9341cb6c93e4fcce90b9360  /boot/efi/EFI/debian/mmaa64.efi
4040b7ac1fa599a04c0656823a80a00d  /boot/efi/EFI/debian/fbaa64.efi
8e5c0af17dbee0bce6736af4ac3b2e5e  /boot/efi/EFI/debian/BOOTAA64.CSV
7612f70c11d62d644039a7ace7bd032e  /boot/efi/EFI/debian/grub.cfg
855150291d3c78128a097838aa04e1d8  /boot/efi/EFI/BOOT/BOOTAA64.EFI
bb1973f4f1d020348077ad5a56ccc690  /boot/efi/EFI/BOOT/grubaa64.efi
a5cce71764941bb3c16d6487a5eb9140  /boot/efi/EFI/BOOT/mmaa64.efi
noahm@temp-520f:~$ 

# updating /boot/efi/EFI/BOOT/
# This configuration reboots successfully.  Note that /boot/efi/EFI/BOOT/*
# get updated.
noahm@temp-520f:~$ sudo grub-install --target=arm64-efi --efi-directory=/boot/efi --no-nvram --force-extra-removable
Installing for arm64-efi platform.
Installation finished. No error reported.
noahm@temp-520f:~$ find /boot/efi/ -type f -ls
        5    966 -rwxr-xr-x   1 root     root       987432 Mar 30 15:08 /boot/efi/EFI/debian/shimaa64.efi
        6   2790 -rwxr-xr-x   1 root     root      2856384 Mar 30 15:08 /boot/efi/EFI/debian/grubaa64.efi
        7    868 -rwxr-xr-x   1 root     root       887472 Mar 30 15:08 /boot/efi/EFI/debian/mmaa64.efi
        8     90 -rwxr-xr-x   1 root     root        90752 Mar 30 15:08 /boot/efi/EFI/debian/fbaa64.efi
        9      2 -rwxr-xr-x   1 root     root          110 Mar 30 15:08 /boot/efi/EFI/debian/BOOTAA64.CSV
       10      2 -rwxr-xr-x   1 root     root          126 Mar 30 15:08 /boot/efi/EFI/debian/grub.cfg
       16    966 -rwxr-xr-x   1 root     root       987432 Mar 30 15:08 /boot/efi/EFI/BOOT/BOOTAA64.EFI
       17   2790 -rwxr-xr-x   1 root     root      2856384 Mar 30 15:08 /boot/efi/EFI/BOOT/grubaa64.efi
       18    868 -rwxr-xr-x   1 root     root       887472 Mar 30 15:08 /boot/efi/EFI/BOOT/mmaa64.efi
noahm@temp-520f:~$ find /boot/efi/ -type f | xargs -n1 md5sum
4a5bdc18d7a879274abd54f665687ea7  /boot/efi/EFI/debian/shimaa64.efi
b7d3dd296477665343aad1f933bae866  /boot/efi/EFI/debian/grubaa64.efi
469f860cb9341cb6c93e4fcce90b9360  /boot/efi/EFI/debian/mmaa64.efi
4040b7ac1fa599a04c0656823a80a00d  /boot/efi/EFI/debian/fbaa64.efi
8e5c0af17dbee0bce6736af4ac3b2e5e  /boot/efi/EFI/debian/BOOTAA64.CSV
7612f70c11d62d644039a7ace7bd032e  /boot/efi/EFI/debian/grub.cfg
4a5bdc18d7a879274abd54f665687ea7  /boot/efi/EFI/BOOT/BOOTAA64.EFI
b7d3dd296477665343aad1f933bae866  /boot/efi/EFI/BOOT/grubaa64.efi
469f860cb9341cb6c93e4fcce90b9360  /boot/efi/EFI/BOOT/mmaa64.efi

The output of efibootmgr is:

BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0003,0001,0000
Boot0000* EFI Network   AcpiEx(VMBus,,)/VenHw(9b17e5a2-0891-42dd-b653-80b5c22809ba,635161f83edfc546913ff2d2f965ed0e973a0d004eb00d003a97b04e000d3a97)/MAC(000000000000,0)/IPv4(0.0.0.00.0.0.0,0,0)
Boot0001* EFI SCSI Device       AcpiEx(VMBus,,)/VenHw(9b17e5a2-0891-42dd-b653-80b5c22809ba,d96361baa104294db60572e2ffb1dc7f1a78b3f8821e1848a1c363d806ec15bb)/SCSI(0,0)
Boot0002* EFI SCSI Device       AcpiEx(VMBus,,)/VenHw(9b17e5a2-0891-42dd-b653-80b5c22809ba,d96361baa104294db60572e2ffb1dc7f1a78b3f8821e1848a1c363d806ec15bb)/SCSI(0,2)
Boot0003* debian        HD(15,GPT,9c46e4b8-e4f3-4b30-a740-a811a8695549,0x800,0x3f800)/File(\EFI\debian\shimaa64.efi)