Re: Using loader to switch kernel and root device on Pi2

On Mar 31, 2025, at 00:04, Mark Millard <[email protected]> wrote:

> On Mar 30, 2025, at 22:55, Mark Millard <[email protected]> wrote:
> 
>> [This is just a resend with the subject line fixed.
>> How/when I had turned into just "UORE", I do not
>> know.]
>> 
>> On Mar 30, 2025, at 21:17, bob prohaska <[email protected]> wrote:
>> 
>> Note: My initial wording is based on not doing
>> any hacking to form a work around. Later I have
>> notes based on hacking a work around based on
>> your providing more description of your overall
>> issue.
>> 
>>> On Sun, Mar 30, 2025 at 06:49:48PM -0700, Mark Millard wrote:
>>>> On Mar 30, 2025, at 16:40, bob prohaska <[email protected]> wrote:
>>>> 
>>>>> An old Raspberry Pi2 v1.1 (so, armv7) has quit booting from USB when
>>>>> using the "bootcode.bin" method (that file only on the MSDOS partition,
>>>>> the usual layout on the USB disk). The machine gets stuck in a netboot
>>>>> loop. 
>>>>> 
>>>>> However, it appears possible to boot using the microSD card, then load 
>>>>> the kernel and mount the root filesystem from USB at the loader prompt:
>>>>> 
>>>>> Hit [Enter] to boot immediately, or any other key for command prompt.
>>>>> Booting [/boot/kernel/kernel] in 10 seconds... 
>>>>> 
>>>>> Type '?' for a list of commands, 'help' for more detailed help.
>>>>> OK lsdev
>>>>> disk devices:
>>>> 
>>>> The below disks look like they might be 2
>>>> separate media, each separately being
>>>> bootable.
>>>> 
>>>>> disk0:    1953525168 X 512 blocks
>>>>>  disk0s1: DOS/Windows
>>>> 
>>>> Does this have a bootcode.bin and other
>>>> RPi* firmware related files? U-Boot?
>>>> 
>>>> The above being a EFI msdsofs, likely with
>>>> its own EFI/BOOT/*.efi file.
>>>> 
>>>>>  disk0s2: FreeBSD
>>>>>    disk0s2a: FreeBSD UFS
>>>> 
>>>> The above being a UFS-based / file system,
>>>> likely with its own /boot/ .
>>>> 
>>>>>    disk0s2b: FreeBSD swap
>>>>>    disk0s2d: FreeBSD UFS
>>>> 
>>>> It is unclear if the above also has a /boot/
>>>> as well.
>>>> 
>>>>> disk1:    125042688 X 512 blocks (removable)
>>>>>  disk1s1: DOS/Windows
>>>> 
>>>> Does this also have a bootcode.bin ? If yes,
>>>> the same vintage as above?  What about other
>>>> RPi* firmware related files? U-Boot?
>>>> 
>>>> The above also being another EFI msdsofs,
>>>> likely with its own EFI/BOOT/*.efi file?
>>>> 
>>>>>   disk1s2: FreeBSD
>>>>>    disk1s2a: FreeBSD UFS
>>>> 
>>>> The above being another UFS-based / file
>>>> system, likely with its own /boot/ ?
>>>> 
>>>> 
>>>> It that the case? The below is only for for the
>>>> context: "yes" to there being multiple variants
>>>> of various directories and files used in booting,
>>>> which might not be the actual type of context.
>>>> 
>>> 
>>> Both disks are complete FreeBSD installations.
>>> Disk0 is a USB mechanical hard disk. Disk1 is
>>> a microSD card with a FreeBSD image downloaded 
>>> from snapshots.
>> 
>> Ahh, that USB vs. microsd card context is
>> important to interpreting things for your
>> problem. (Later notes by you make it even
>> more so.)
>> 
>>> Files from that image populated
>>> the usb disk initially, before updating via source.
>> 
>> Ahh. The RPi2 v1.1 has a means of controlling
>> microsd vs. USB being first. But USB here is a
>> bus, not a specific device (unless there is only
>> one USB storage media present on the USB bus).
>> 
>> The way I read the following, bootcode.bin (and
>> possibly timeout) are the only RPi* firmware files
>> that should be on the microsd card (in standard
>> places) in order to then do a RPI* firmware
>> based USB host boot:
>> 
>> QUOTE
>> Format an SD card as FAT32 and copy over the latest bootcode.bin.
>> The SD card must be present in the Raspberry Pi for it to boot.
>> Once bootcode.bin is loaded from the SD card, the Raspberry Pi
>> continues booting using USB host mode.
>> 
>> This is useful for the Raspberry Pi 1, 2, and Zero models, which
>> are based on the BCM2835 and BCM2836 chips, and in situations
>> where a Raspberry Pi 3 fails to boot (the latest bootcode.bin
>> includes additional bugfixes for the Raspberry Pi 3B, compared
>> to the boot code burned into the BCM2837A0).
>> If you have a problem with a mass storage device still not
>> working, even with this bootcode.bin, then add a new file called
>> "timeout" to the SD card. This will extend to six seconds the
>> time for which it waits for the mass storage device to initialise.
>> END QUOTE
>> 
>> If you have more of the RPi* firmware files on the
>> microsd card, it may try to boot from the microsd
>> card instead.
>> 
>> NOTE added later:
>> Later notes are tied to having the FreeBSD
>> world loaded from USB by a FreeBSD kernel not
>> boot-time loaded from USB but from the microsd
>> card. Note that such is not an example of the
>> RPi* firmware booting from USB at all. Until
>> later my notes are generally based on presuming
>> a RPI* firmware based USB boot.
>> 
>>> The numbering seems backwards to me, I'd expect
>>> disk0 to be microSD and usb to be disk1, but no.
>> 
>> I expect that the disk0 vs. disk1 part of the
>> naming is from U-Boot's naming assignments.
>> 
>>>> If yes, then how do you expect the RPi2 to
>>>> pick which EFI msdosfs it uses each time? (That
>>>> is just one type of example "pick" operation.)
>>>> Does it need to be more stable than random for
>>>> which it picks?
>>>> 
>>> 
>>> In practice, the Raspberry Pi boots from the microSD
>>> card, even if a second bootable device is present.
>> 
>> For an RPi firmware based USB boot, things are
>> different than hacking in a non-RPI* firmware
>> based USB boot that does a partial USB based
>> boot in a later stage.
>> 
>> 
>>>> Once it picks, how do you expect the loader to
>>>> pick which UFS-based /boot/ it should use? Does it
>>>> need to be more stable than random?
>>>> 
>>> 
>>> I pick which ufs is used, via the
>>> OK set currdev="disk0s2a"
>>> OK set vfs.root.mountfrom="ufs:/dev/da0s2a"
>> 
>> That is only via a manual intervention, not the
>> automatic boot handling.
>> 
>>>> Note that which /boot/loader.conf file would be
>>>> used depends on which /boot/ is used if there are
>>>> multiple /boot/ directories present.
>>> 
>>> Disk 1 (microSD) is active at the time of booting, 
>>> so its /boot/loader.conf will contain the modifications.
>> 
>> I'll have more realted notes later.
>> 
>>>> As I understand things, whichever  EFT/BOOT/*.efi
>>>> is used, a /boot/ from the same media (not the
>>>> same partition or file system) is used. If there
>>>> is more than one such /boot/ on that media
>>>> (via distinct file systems on the media), then
>>>> the question of which /boot/ is picked is
>>>> repeated.
>>>> 
>>>> Can you make the disk1s1 such that is not a valid
>>>> RPi2* firmware source and not a valid EFI msdosfs?
>>>> An example could be renaming the EFI/BOOT/*.efi
>>>> file to be a name that is not one that would be
>>>> found as a standard .efi file name to be used --so
>>>> that it would be ignored.
>>> 
>>> No, because that is the firmware and loader that the
>>> Pi is able to start. It won't even try to boot usb.
>> 
>> The only 2 files that are needed on the microsd card
>> for an RPi* firmware based USB boot are: bootcode.bin
>> and, possibly, timeout .
>> 
>> It is the bootcode.bin file that enables the RPi*
>> fimrware based USB boot. No other files contribute
>> (other than timeout leading to a longer delay).
>> 
>> However, you later provide notes indicating that
>> you need the USB activity to be in a later stage
>> than the RPi* firmware doing the USB activity.
>> 
>>>>> http: (unknown)
>>>>> net devices:
>>>>> net0:
>>>>> OK unload
>>>>> OK show currdev
>>>>> disk1s2a:
>>>>> OK set currdev="disk0s2a"
>>>>> OK show currdev
>>>>> disk0s2a
>>>>> OK load kernel
>>>>> /boot/kernel/kernel text=0x1b4 text=0x6df178 text=0x1f9cb8 data=0xc37a8 data=0x0+0x254000 0x4+0xb1af0+0x4+0x134365|
>>>>> OK set vfs.root.mountfrom="ufs:/dev/da0s2a"
>>>> 
>>>> You did not first show the value of
>>>> vfs.root.mountfrom from before you
>>>> changed it. Which /boot/ would it
>>>> have found if you had not changed
>>>> the value?
>>> 
>>> By default the Pi boots from the microSD, disk1 in
>>> this context. Presumably that means the value of
>>> vfs.root.mountfrom would have been  /dev/mmcsd0s2a
>>> or some equivalent label pointing to the same device.
>> 
>> A probably hypothesis, given disk1 being the
>> microsd card that has a lot of files that it
>> should not have on it for USB booting: just
>> bootcode.bin and, possibly, timeout relative
>> to RPi* firmware (in places were it such
>> would be found).
>> 
>>>>> OK boot -s
>>>>> Using DTB provided by EFI at 0x39f23000.
>>>>> Kernel entry at 0x33e00200...
>>>>> Kernel args: -s
>>>>> .....
>>>>> 
>>>>> The machine comes up with kernel and root from the USB disk, as desired.
>>>>> 
>>>>> It appears that the 
>>>>> currdev="disk0s2a"
>>>>> and 
>>>>> vfs.root.mountfrom="ufs:/dev/da0s2a"
>>>>> 
>>>>> could be placed in /boot/loader.conf,
>>>>> but where would the unload and load
>>>>> commands go?
>>>> 
>>>> Is there more than one /boot/ and, so, more
>>>> than one /boot/loader.conf that might be
>>>> used from one boot attempt to the next? If
>>>> yes, how do you know which /boot/loader.conf
>>>> to edit?
>>>> 
>>>> Getting which EFI/BOOT/*.efi and /boot/
>>>> picking to be stable and as-desired is
>>>> something to have in place before worrying
>>>> about the content of any specific
>>>> /boot/loader.conf or other such file.
>>>> (Presumes you do not make all the various
>>>> examples [if multiple] be duplicates of
>>>> each other for all the files would end up
>>>> involved.)
>>>> 
>>>>> Without them the machine
>>>>> is stuck with the microSD's kernel.
>>>> 
>>>> You should only have one RPI* firmware
>>>> soruce and only one EFI/BOOT/*.efi (that
>>>> has the right filenaming convention) and
>>>> only one /boot/ in normal usage contexts.
>>>> Having only one /boot/ means having only
>>>> one /boot/kernel/kernel as well. There
>>>> should not be alternatives that might
>>>> instead be found and used.
>>>> 
>>> That harks back to having bootcode.bin alone on
>>> the microSD dos partition, which then finds and
>>> boots the usb disk. That's how the system started
>>> out and worked through several cycles of self-hosting.
>>> After an upgrade bootcode.bin couldn't find the usb
>>> disk and got stuck in a netboot loop from which it
>>> couldn't be extricated. A fresh install behaved the
>>> same way; worked fine through a few builworld/kernel
>>> cycles, then started trying only to netboot.
>> 
>> I think the above context was missing in the earlier
>> information (above and the prior Email).
>> 
>> This a different problem: hacking a workaround
>> so that a later, non-RPi* firmware stage, does
>> a partial USB based boot after some stages of
>> microsd card based boot activity.
>> 
>>> The usual suspects (power supplies, bad cables, bad
>>> usb-sata bridges and faulty microSD cards) were ruled
>>> out by experiment.
>> 
>> Did your testing include testing with the timeout
>> file being present?
>> 
>>> A correspondent on comp.sys.raspberry-pi suggested
>>> that maybe the FreeBSD loader installed on the 
>>> microSD could be coaxed into booting the usb disk,
>>> since booting the microSD works reliably. That seems
>>> to be true, at least for now.
>> 
>> I've used USB hardware that the FreeBSD kernel
>> could handle but the prior U-Boot stage could
>> not. (It was not an RPi* context.) Such things
>> lead to hacks to make things limp along.
>> 
>> In that case I duplicated /boot/ (and a few more
>> files) onto the microsd card in a UFS file
>> system and had the RPi* firmware, U-Boot, and
>> EFI/BOOT/*.efi in the microsd card's msdosfs.
>> (I did not want later modules to be loaded from
>> the microsd card but from the USB media.) One
>> of the duplicated files had the vfs.root.mountfrom
>> to use to find the USB media.
>> 
>> I did not duplicate the rest of the world: that
>> was only on the USB media.
>> 
>> Every update needed to make sure that things
>> were duplicated accurately. Some files had to
>> be handled specially.
>> 
>>> currdev="disk0s2a"
>>> and
>>> vfs.root.mountfrom="ufs:/dev/da0s2a"
>>> both look like easy additions to /boot/loader.conf
>>> on the microSD card.  
>>> 
>>> The question that's gone unanswered is where to put
>>> the "unload" command to clear out the kernel picked
>>> up during loader startup so that it will be replaced
>>> by the new kernel from the usb disk.
>> 
>> What I did had no unload or reload required: I used
>> the duplication of content in to 2 places to avoid
>> such activity being needed. The initial load of
>> the kernel was exactly the right content and compatible
>> with any later kernel module loads done from the USB
>> media.
>> 
>>> It's evident my question is somewhat out of left field.
>>> All I really meant to ask was how to configure an
>>> unload command via loader.conf. The man page does not
>>> mention "unload", must it go elsewhere?
>> 
>> I suggest avoiding the need to unload and reload
>> by setting up to have the initial load have the
>> correct content in the first place. The tradeoff
>> is needing to keep the duplicated information
>> on the microsd card up to date at every update.
>> 
>> Note: Eventually U-Boot updated to handled the
>> USB context that it originally did not and I was
>> able to stop using the hackish technique.
>> 
>> So it has been a while since I've last dealt with
>> this sort of thing.
> 
> 
> I no longer have media organized for the
> kernel-stage USB world part of the boot
> for the Rock64. Nor do I seem to have a
> copy of the old instructions/reminders
> file that I used while updating.
> 
> Some of the below words things as-if the
> Rock64 context was more like the RPi*
> context than it actually was.
> 
> Some old notes indicate:
> 
> Having a UFS file system in a partition/slice
> on the microsd card in order for it to 
> contain:
> 
> ) A recursive copy of /boot/
> 
> ) But I avoided keeping a copy of /boot/entropy on
>  the microsd card as I remember. This could be a
>   security issue.
> 
> ) A copy of /etc/hostid (as that path is accessed
>  very early, much like /boot/loader.conf is
>  accessed very early, before the normal root
>  file system mount).
> 
> I used rsync with a delete option and an exclude
> option to deal with the recursive copies as I
> remember.
> 
> The /boot/loader.conf had a:
> 
> vfs.root.mountfrom="ufs:/dev/gpt/Rock64root"
> 
> One of the considerations was if the tree at
> 
> /usr/lib/debug/boot/
> 
> should also be copied to the microsd card's UFS
> partition in order to have better problem analysis
> context for kernel boot failures.
> 
> I can not claim to know if the above notes
> are complete vs. not.

I found the old e.MMC media that was involved on
the Rock64 before U-Boot started handling the USB3
port for booting. So here is the procedure that
was used to duplicate things to the microsd card.
I'll note that I renamed no linger used directories
with _disabled suffixes when I was preserving the
content at the time.

# more /mnt/boot_disabled/copy_boot_to_microsd.sh
test -f /microsd_ufs/boot/entropy || mount -onoatime /dev/mmcsd0s2a /microsd_ufs/
rsync -axHh --info=progress2 --exclude=/boot/entropy --delete -r /boot /microsd_ufs/
mkdir -p /microsd_ufs/etc
cp -aRx /etc/hostid /microsd_ufs/etc/

Note: /microsd_ufs/ dismounts were handled elsewhere
when mounts had occurred.

It may be that the /microsd_ufs/boot/entropy was already
the more up to date copy and so was not updated from an
older copy that was on the USB media but was unused. So
the --exclude=/boot/entropy might have been for improved
security.

The -aRx for cp is partially redundant. But the form
used reminded me to pay attention to an issue related to
recursive copies so I was, by habit, explicit.

Copying almost all of /boot/ avoiding needing to track
the details of what was needed vs. not and how that
might change over time. The tradeoff was possibly having
lots of files copied that go unused.

===
Mark Millard
marklmi at yahoo.com