Re: Booting rock64 from USB SSD

W dniu 1.04.2022 o 22:52, Mark Millard pisze:
> On 2022-Apr-1, at 06:51, Łukasz Moskała <[email protected]> wrote:
> 
>> Hi everyone,
>>
>> I want to boot my rock64 from SSD (because all sd cards I could find are crap).
>>
>> I followed those instructions to flash u-boot to SPI: https://github.com/ayufan-rock64/linux-build/blob/master/recipes/flash-spi.md
>>
>> I was then able to boot FreeBSD from SSD (dd FreeBSD-13.0-RELEASE-arm64-aarch64-ROCK64.img to SSD, just like it would be to SD card), but ethernet wasn't working. Actually, it could receive packets but no outgoing packets were sent. Not even ARP packets - switch didn't detect any device on that port, but link was UP
>>
>> I assumed it was something to do with u-boot, so I erased u-boot from flash, then DD FreeBSD-13.0-RELEASE-arm64-aarch64-ROCK64.img to SD card, and it worked without problems.
>>
>> Do I understand correctly that this means that FreeBSD on rock64 uses custom u-boot? Can I somehow flash it to SPI?
>>
>> Possible workaround that I can think of would be to put /boot on sd card, install u-boot to sd card, then put / on SSD.
> 
> My memory is that USB2 booting could be more normal, in that the
> U-Boot from ports could access USB2 storage but not USB3 storage,
> at least when I set this up long ago. I wanted USB3 use, thus the
> below details.
> 
> I have U-Boot on the /media/ device below. As I remember, it could not
> deal with the USB3 port, which is the one I wanted to use. The
> FreeBSD kernel was the first stage that could deal with the USB3 port
> back when I set up the Rock64 that I have access to.
> 
> I've not had to change the basic structure since I set it up. There
> could be aspects that could be different these days and I'd not know.
> Think of any comments as potentially being old information.
> 
> Note: mmcsd0 here is actually an eMMC device instead of the microsd
> card slot contents, leaving the microsd card slot free for other uses.
> But I'd used a microsd card this way before using the eMMC device
> (as I remember anyway).
> 
> # gpart show -p
> =>       63  244277185    mmcsd0  MBR  (116G)
>           63      32705            - free -  (16M)
>        32768     102312  mmcsd0s1  fat32lba  [active]  (50M)
>       135080      28760            - free -  (14M)
>       163840  241172480  mmcsd0s2  freebsd  (115G)
>    241336320    2940928            - free -  (1.4G)
> 
> =>        0  241172480   mmcsd0s2  BSD  (115G)
>            0  230686720  mmcsd0s2a  freebsd-ufs  (110G)
>    230686720   10485760             - free -  (5.0G)
> 
> =>        40  1953525088    da0  GPT  (932G)
>            40      532480  da0p1  efi  (260M)
>        532520        2008         - free -  (1.0M)
>        534528     7340032  da0p2  freebsd-swap  (3.5G)
>       7874560     1048576         - free -  (512M)
>       8923136    23068672  da0p3  freebsd-swap  (11G)
>      31991808     2097152         - free -  (1.0G)
>      34088960    33554432  da0p4  freebsd-swap  (16G)
>      67643392  1740636160  da0p5  freebsd-ufs  (830G)
>    1808279552     4194304  da0p6  freebsd-swap  (2.0G)
>    1812473856   141051272         - free -  (67G)
> 
> For reference, from "gpart show -pl" :
> 
>      67643392  1740636160  da0p5  Rock64root  (830G)
> 
> (The USB drive can boot other systems as well, with
> widely varying amounts of RAM. Thus the efi partition
> and the odd set of freebsd-swap partitions.)
> 
> So, /media/ below is mmcsd0s1's fat32lba and /mnt/ is
> mmcsd0s2a's freebsd-ufs. da0 is the USB3 SSD media,
> which I do not give other details of here. (I manually
> mounted these for this note.)
> 
> # ls -Tld /media/*/*/* /mnt/* /mnt/etc/*
> -r-xr-xr-x   1 root  wheel  1243772 Jan 28 12:33:00 2022 /media/EFI/BOOT/bootaa64.efi
> -r-xr-xr-x   1 root  wheel    50618 Jan 28 12:32:28 2022 /media/dtb/rockchip/rk3328-rock64.dtb
> -r--r--r--   1 root  wheel     6170 Feb  1 04:48:34 2020 /mnt/COPYRIGHT
> drwxr-xr-x  23 root  wheel     1536 Jan 28 15:26:41 2022 /mnt/boot
> drwxr-xr-x   2 root  wheel      512 Apr 26 14:39:22 2020 /mnt/etc
> -rw-r--r--   1 root  wheel       37 Dec 31 16:00:18 2009 /mnt/etc/hostid
> drwx------   2 root  wheel    33280 Nov 27 09:46:08 2019 /mnt/lost+found
> 
> # ls -Tld /mnt/boot/dtb/overlays/rk3328-*
> -r--r--r--  1 root  wheel   238 Jan 28 12:32:28 2022 /mnt/boot/dtb/overlays/rk3328-analog-sound.dtbo
> -r--r--r--  1 root  wheel  1281 Jan 28 12:32:28 2022 /mnt/boot/dtb/overlays/rk3328-dwc3.dtbo
> 
> (I make no use of rk3328-analog-sound.dtbo .)
> 
> /mnt/boot/loader.conf has, in part,
> 
> # A msdosfs /MNTPNT/dtb/rockchip/rk3328-rock64.dtb
> # copy of the ufs /boot/dtb/rockchip/rk3328-rock64.dtb
> # uses the intended DTB in u-boot and the kernel --and
> # avoids needing to tell the kernel where to find a
> # copy . . .
> #rk3328_rock64_load="YES"
> #rk3328_rock64_type="dtb"
> #rk3328_rock64_name="/boot/dtb/rockchip/rk3328-rock64.dtb"
> #
> # rk3328 USB3-related:
> fdt_overlays="rk3328-dwc3.dtbo"
> # ucom is not automatically being loaded when umodem is loaded at boot.
> ucom_load="YES"
> umodem_load="YES"
> #
> vfs.root.mountfrom="ufs:/dev/gpt/Rock64root"
> kern.cam.boot_delay=10000
> vfs.mountroot.timeout=10
> vfs.root_mount_always_wait=1
> 
> ===
> Mark Millard
> marklmi at yahoo.com
> 

Hi Mark,

With some idea from what you did, I managed to do this:

=================== BEGIN COMMAND LIST ============================
sudo pkg install u-boot-rock64
truncate -s 512M rock64-sd.img
truncate -s 5G rock64-disk.img
sudo mdconfig -a -t vnode -f 
FreeBSD-13.0-RELEASE-arm64-aarch64-ROCK64.img #returned md0
sudo mdconfig -a -t vnode -f rock64-sd.img #returned md1
sudo mdconfig -a -t vnode -f rock64-disk.img #returned md2
sudo gpart create -s gpt md1
sudo gpart add -t efi -b 32768 -s 50M -i 1 md1
sudo gpart add -t freebsd-ufs -i 2 md1
sudo newfs -O 2 -L bootfs /dev/md1p2
sudo dd if=/dev/md0p1 of=/dev/md1p1 bs=1M
mkdir mp_old mp_new
sudo mount -o ro /dev/md0p2 mp_old
sudo mount -o rw /dev/md1p2 mp_new
sudo rsync -rxav mp_old/boot/ mp_new/boot/
echo 'fdt_overlays="rk3328-dwc3.dtbo"' | sudo tee -a mp_new/boot/loader.conf
echo 'vfs.root.mountfrom="ufs:/dev/ufs/rootfs"' | sudo tee -a 
mp_new/boot/loader.conf
echo 'kern.cam.boot_delay=10000' | sudo tee -a mp_new/boot/loader.conf
echo 'vfs.mountroot.timeout=10' | sudo tee -a mp_new/boot/loader.conf
echo 'vfs.root_mount_always_wait=1' | sudo tee -a mp_new/boot/loader.conf
sudo gpart create -s gpt md2
sudo gpart add -t freebsd-ufs -a 4096 -i 1 md2
sudo newfs -O 2 -t -L rootfs /dev/md2p1
mkdir mp_newroot
sudo mount -o rw /dev/md2p1 mp_newroot
sudo rsync -rxav --exclude=boot mp_old/ mp_newroot/
sudo rm mp_newroot/etc/fstab
sudo mkdir mp_newroot/media/sd
echo '/dev/ufs/rootfs / ufs rw 1 1' | sudo tee -a mp_newroot/etc/fstab
echo '/dev/ufs/bootfs /media/sd ufs rw,noatime 1 2' | sudo tee -a 
mp_newroot/etc/fstab
echo '/dev/msdosfs/EFI /media/sd/boot/efi msdosfs rw,noatime 0 0' | sudo 
tee -a mp_newroot/etc/fstab
echo 'tmpfs /tmp tmpfs rw,mode=1777,size=50m 0 0' | sudo tee -a 
mp_newroot/etc/fstab
cd mp_newroot
sudo ln -sf media/sd/boot boot
cd ..
sudo umount mp_newroot
sudo umount mp_new
sudo umount mp_old
sudo dd if=/usr/local/share/u-boot/u-boot-rock64/idbloader.img 
of=/dev/md1 seek=64 bs=512 conv=sync
sudo dd if=/usr/local/share/u-boot/u-boot-rock64/u-boot.itb of=/dev/md1 
seek=16384 bs=512 conv=sync
sudo mdconfig -d -u 2
sudo mdconfig -d -u 1
sudo mdconfig -d -u 0
=================== END COMMAND LIST ============================
(my apologies if some lines wrap improperly)

then, DD rock64-sd.img to sdcard and rock64-disk.img to disk.
It would be possible to do that directly on disk and sdcard itself, but 
in my case, machine on which I ran these commands doesn't have sd card 
reader.

Anyway, it successfully booted with SSD plugged into USB3 port, despite 
those messages:
Root mount waiting for: usbus0 usbus2 usbus3 usbus4 CAM
usb_alloc_device: set address 2 failed (USB_ERR_IOERROR, ignored)
usbd_setup_device_desc: getting device descriptor at addr 2 failed, 
USB_ERR_IOERROR
after which da0 was detected and boot continued. Network is working, and 
everything seems okay (uptime is 6 minutes as of moment I'm writing 
this). I'm using official pine64 usb-sata adapter, but I suppose it 
doesn't really matter. I don't have any other adapters to check. I 
copied 2.5GiB file to it and it seems stable.

I got a lot of "devmatch: Can't read linker hints file."
That was fixed by kldxref -R /media/sd/boot

root@generic:~ # df -h
Filesystem          Size    Used   Avail Capacity  Mounted on
/dev/ufs/rootfs     108G    4.2G     95G     4%    /
devfs               1.0K    1.0K      0B   100%    /dev
/dev/ufs/bootfs     432M     82M    316M    21%    /media/sd
/dev/msdosfs/EFI     50M    2.7M     47M     5%    /media/sd/boot/efi
tmpfs                50M    4.0K     50M     0%    /tmp
root@generic:~ # mount
/dev/ufs/rootfs on / (ufs, local)
devfs on /dev (devfs)
/dev/ufs/bootfs on /media/sd (ufs, local, noatime)
/dev/msdosfs/EFI on /media/sd/boot/efi (msdosfs, local, noatime)
tmpfs on /tmp (tmpfs, local)
root@generic:~ # gpart show -p
=>     40  1048496    mmcsd0  GPT  (7.4G) [CORRUPT]
        40    32728            - free -  (16M)
     32768   102400  mmcsd0p1  efi  (50M)
    135168   913368  mmcsd0p2  freebsd-ufs  (446M)

=>       40  234441568    da0  GPT  (112G)
          40       4056         - free -  (2.0M)
        4096  234437512  da0p1  freebsd-ufs  (112G)

Thanks for your help, and I hope that whatever I made here will be 
usefull to someone.

-- 
Łukasz Moskała