Bug#1031696: Also affects bookworm

Hi Steve,

On 2023.03.12 14:53, Steve McIntyre wrote:
> It looks like James (with some help from Thomas) has worked out a
> quick way to change things to make things better for you, which is
> good! (Thanks, guys! I'm about to test the change locally.)

Yes, I'm happy to see this development as well. And I have to stress 
out, since this appears to be gist of your query, that mine is far from 
a self centred goal, but something that aims at providing a *choice* to 
people, when it comes to the manner in which they should create their 
boot media, the reasons for which I will elaborate on hereafter.

> *However* you're the *only* person I've seen complaining about this
> problem with "file system transposition".

I will venture that this is because, and this is not criticism per se, 
most distributions currently make that process quite difficult or 
fraught with so many hurdles that people who would want to use FST will 
not yet bother trying it. For one thing, outside of Debian, most 
distributions still seem to be relying on labels to locate the 
installation media, which of course renders the process of using FST not 
as straightforward as it should be, especially if the label used doesn't 
meet the FAT limitations of using 11 uppercase characters (with further 
additional restrictions), as this may result in a need to manually edit 
GRUB/Syslinux configuration files.

Which means that, currently, unless you are using Debian (or Arch, which 
does make an effort to use labels are FAT compatible), you are better of 
not trying to use FST, as you are taking a gamble as to whether your 
media is actually going to work unless you know precisely what you're doing.

And this gamble is precisely what I aim at trying to eliminate moving 
forward, with the help of distro maintainers.

In other words, as a person invested in the means of providing people 
with *choice* on how they should create their media, especially on 
Windows platforms, I am trying to make the current problematic situation 
evolve for the better. Which means that, yes, you will find me at the 
forefront of trying to guide distributions into establishing FST as a 
viable alternative to dd, and, sadly, as a potentially lone voice in 
doing so.

But I have to be 100% clear about this, even if it would be easy to draw 
the conclusion that, as the purveyor of Rufus, a boot media creation 
application which tries not to use 'dd' as default (for the reasons that 
will be explained below) I might have some clear vested interest, my 
goal here, even if it may seem to be counter productive, is to genuinely 
allow people to create their Linux installation media *without* having 
to use third party applications like Rufus, since this is something 
that, as opposed to BIOS, UEFI certainly makes achievable. This is 
because I do consider that people who want to install a new OS should be 
given the choice on how they should proceed with how they do so. But 
currently, because ISOHybrid + 'dd' appears to be largely being seen as 
the ultimate panacea by distro maintainers (and I can certainly see the 
reasons why, because it certainly does make life easier for distro 
maintainers), I do have to point out that it does come with some 
drawbacks, along with my concerns at how little traction there appears 
to be in terms of wanting to establish an alternative to only using dd 
with ISOHybrids. That's because, from where I stand, I see this "dd uber 
alles" stance as ultimately damaging for end users in the long run, even 
if it does appear that I am currently have to be the one person that has 
to try to convince distro maintainers that they may want to apply 
careful consideration before putting all of their eggs in the ISOHybrid 
as 'dd' basket...

> I genuinely have not seen
> issues raised by people doing this, except where (it seems) Rufus is
> working this way.

I will assert that this is because you are looking at this the wrong way 
around. The underlying problem is that, and this always seems to come as 
a shock to people who may be Linux-centric, people trying to create boot 
media through ISOHybrid + dd *are* actually experiencing issues that are 
effectively preventing them from installing Linux systems altogether.

And the annoying thing is that, since these are mostly Windows users, 
very few of these issues actually make their way towards the attention 
of Linux distro maintainers...

So, to starts us, here's a non-exhaustive list of reports I compiled 
some time ago, to demonstrate this very point (and I'm going to stop at 
12 examples, because I will assume that, even if I could add more, you 
will get the point that this is a *real* issue faced by users, and not 
something that should be dismissed as a low impact problem):

* 
https://old.reddit.com/r/linuxquestions/comments/j4nolf/creating_a_bootable_usb_drive_for_linux/
* 
https://old.reddit.com/r/ManjaroLinux/comments/gofq71/problem_with_rufus_310_and_manjaro_2001/
* 
https://old.reddit.com/r/ManjaroLinux/comments/gjdpi4/cannot_create_bootable_usb_usb_size_shrinks_after/
* 
https://old.reddit.com/r/ManjaroLinux/comments/c58bb8/manjaro_boot_media_shows_up_as_miso_efi/
* 
https://old.reddit.com/r/techsupport/comments/499b5c/usb_stick_capacity_shrunk_to_2mb/
* 
https://superuser.com/questions/752874/16-gb-usb-flash-drive-capacity-down-to-938-mb
* 
https://www.diskgenius.com/how-to/how-to-restore-usb-drive-back-to-full-capacity.php
* 
https://www.quora.com/After-making-my-32-GB-pen-drive-Kali-Linux-bootable-its-size-reduced-to-712-KB-Is-it-normal-If-not-then-how-can-I-fix-it
* 
https://askubuntu.com/questions/289971/usbs-storage-capacity-reduced-to-2-mb-from-16-gb
* 
https://askubuntu.com/questions/611325/capacity-of-pen-drive-shown-is-less-than-the-actual
* 
https://askubuntu.com/questions/586118/16gb-pen-drive-showing-2mb-space-after-formatting-on-windows-7
* 
https://forums.tomshardware.com/threads/usb-flash-drive-8gb-is-now-only-1gb.660997/

Note that some of these links may mention Rufus, because I mostly got 
them while looking for issues that people may run into while running 
Rufus. But since it's about 'dd' mode creation from ISOHybrid, and not 
FST, the application being used here is irrelevant, as you will have no 
trouble getting similar reports from people using balenaEtcher or any 
'dd' like application you can use on Windows.

The core of the issue is that, because ISOHybrid media written in DD 
mode on Windows usually results in a multipartioned device, with the 
larger partition usually using a file system that Windows will not mount 
and the smaller partition being the ESP, users are being thrown off by 
the result, and are considering that the media creation process did not 
complete successfully. Thus, they will not even try to use their media 
to install Linux.

And I know that it is very tempting to say that "This is a user problem" 
or that "These people should just plow through and it's their fault for 
not doing so", but that doesn't it make less of an issue.

I could also mention that some people want to make sure that, if they 
use a computer that supports UEFI and Legacy, they do install their OS 
in pure UEFI mode, which is something that can easily be sorted out by 
ensuring that the media is partitioned as GPT, but which an ISOHybrid 
almost never is (and, I should add, if it is, then, because it will not 
have the backup GPT at the expected position when written in dd mode, 
then any GPT based ISOHYbrid media written in dd mode will actually be 
"broken" in terms of UEFI compliance... and some UEFI firmwares might be 
pedantic enough not to boot a media with a broken backup GPT). So, there 
again, we are facing an issue with using ISOHybrid as 'dd', that can 
easily be sorted out by switching to FST.

Then there are some opportunities of doing things, like installing 
Debian ARM64 in full UEFI mode on a Raspberry Pi 4 using a single media 
[1] (and I think this is part of a conversation you and I were already 
involved in the past) that will never ever be achievable if treating 
ISOHybrid only as a glorified dd image. And there again, I know that one 
might be tempted to dismiss the idea of doing single media installation 
on an SBC, when someone *should* just be able to use 2 drives anyway, 
but if that is the case, I have to warn about idea once again of 
*limiting* the end-user's choice (or deciding that all users, and 
especially low priced SBC ones, should be wealthy enough to afford two 
flash media).

I believe that we pretty much all came to FLOSS because we didn't like 
to be limited by software in terms of choice. Which is why, even if it 
appears that, (much to my dismay) I might be a single voice in doing so, 
I feel it is my duty to continue to hammer the message that users should 
be given the choice on how they create their installation media and 
that, precisely because most distros are still placing major hurdles in 
that respect, we must ensure that the hurdles that prevent choice are 
being cleared.

I'll also conclude this section by stating that, from a technical 
perspective, I can't help but feel a bit both amused and slightly 
dismayed that, whereas UEFI was designed from the ground up to do away 
with the necessity, enacted by BIOS, of creating boot media at the 
sector level, to instead allow for the creation of boot media purely at 
the file system level, we've now seen a push from many Linux distro 
maintainers to completely ignore this most welcome UEFI improvement, and 
instead do a complete 180 on it by more or less enforcing the writing of 
boot media at the sector level, through the promotion of ISOHybrid as dd 
only...

> Let's be clear: "transposition" == "copying things
> onto a less capable filesystem".

I'd reframe this, because we're really only dealing with UEFI here, as:

"transposition" == "copying things onto the file system that is mandated 
by UEFI as supported for boot"

The fact that it is indeed less capable is an unfortunate design 
decision by the UEFI committee (which is hard to fathom where they had 
plenty of more capable file systems to pick from at the time, especially 
as FAT did come with licensing issues that Microsoft had to provide an 
exception for). But this is neither something that you, nor I, nor the 
end-user, has any choice with in the first place. In other words, 
transposition is not a choice of a lesser file system. It's simply the 
hand we are being dealt with by UEFI.

> I'm normally really impressed by your
> skills and knowledge, but I don't understand why you seem to be so
> obsessed by this.

I just want future users to be able to create Debian installation media 
that they can use for their purpose, which, unlike what some people here 
may think, may mean providing them with a *choice* to use either dd of 
FST when creating the media, since each mode has its advantages and 
drawbacks (and let me also be clear here, since it may appear that I am 
bashing ISOHybrid as dd, I am in no way implying that one is "better" 
than the other, as I very much see both as having their place). But 
right now, because I would say Linux-centric folks appear to be less 
alert to the current plight of Windows users, I feel like I have no 
choice to raise my voice to make this plight heard, and the need for FST 
support, as an alternative to dd, be understood. And it is very 
disheartening for me to be faced with Linux-centric person after 
Linux-centric person not coming to the logical conclusion that:

1. ISOHybrid as DD has its limits, and some of it are negatively 
impacting the very same users that ISOHybrid as DD was supposed to help 
in the first place.

2. Treating FST on an equal standing as DD will benefit users, and 
therefore Linux distributions that support it, in the long run.

> With some tweaks like this MR, simply copying files onto FAT32 can
> clearly be made to work for UEFI-only media. That's fine. But it still
> breaks other useful features, at least:
> 
>   * BIOS boot

BIOS boot breakage is a given and is out of scope, since I am only 
considering UEFI FST (which I will assert, considering that almost all 
modern systems are UEFI based, should be fair).

Rufus can actually work some magic, to some extent, to make BIOS based 
systems also work with FST, but that's well beyond the scope of what I 
consider we should concern ourselves with (and, in case this is a 
question, I am certainly not asking for anything from Debian that is 
directly related to BIOS FST boot)

>   * image checksums

It seems you aren't aware that the default behaviour of Windows is to 
create a 'System Volume Information' folder on any file system it 
recognises.

Which means that, if you create an ISOHybrid using 'dd' on Windows, and 
it includes a ESP, Windows will create a 'System Volume Information' 
folder, which will break the full image checksum, since the ESP content 
will have been altered the minute you created the media on Windows.

This is a Windows behaviour that is enabled by default and that you 
can't disable without a reboot.

On the other hand, individual file checksums, which is what Ubuntu (and 
others) have been using to validate their media, does work quite well 
with both FST and DD.

Furthermore, the expectation (or at least my expectation) is that we're 
moving towards trusted validation of binaries that are being loaded 
during the boot sequence, which implicitly includes individual file 
checksums, and makes part of the need for whole image checksums 
redundant (and which, again, will work regardless of whether you're 
using a "limited" file system like FAT or a more "capable" one).

> and those are important for me and a lot our users.

They are important for me too. That's why, if you want BIOS boot, I 
would encourage you to use Rufus, as it will take care of creating a 
Debian bootable media that just works, regardless of whether you are 
*choosing* to use FST or DD

> Seriously, just
> using DD or similar gives people a verifiable, known-good copy of our
> installer image that will boot on as many machines as possible and
> work well in the debian-installer environment. That's my primary goal
> here.

Then I hope some of the points I highlighted above, and that should help 
appreciate that ISOHybrid as DD might not be the foolproof panacea that 
quite a few distro maintainers appear to think it is, will hit home.

Regards,

/Pete

[1] https://forums.raspberrypi.com//viewtopic.php?f=50&t=282839