Re: PR 252541: Early kernel panic on RPi4B (Too many early devmatch mappings)

On 2021-Jan-11, at 18:10, Mark Millard <marklmi at yahoo.com> wrote:

> On 2021-Jan-11, at 16:19, Mark Millard <marklmi at yahoo.com> wrote:
> 
>> 
>> 
>> On 2021-Jan-11, at 14:23, Mark Millard <marklmi at yahoo.com> wrote:
>> 
>> 
>>> On 2021-Jan-11, at 12:47, Gordon Bergling <gbe at freebsd.org> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> I am currently investigating PR 252541 (Too many early devmatch mappings) [1].
>>>> 
>>>> The kernel panic happends on the RPi4B. Has anyone successfully booted a rivision
>>>> on the RPi4B after:
>>>> 
>>>> ---------------------------------------------------------------------------------
>>>> commit e83fdf8bb391579fa422d34663cd8c1f82a00dc0
>>>> Author:     Chuck Tuffli <[email protected]>
>>>> AuthorDate: 2021-01-08 22:36:37 +0000
>>>> Commit:     Chuck Tuffli <[email protected]>
>>>> CommitDate: 2021-01-08 22:41:45 +0000
>>>> 
>>>> fix big-endian platforms after 6733401935f8
>>>> 
>>>> The NVMe byte-swap routines for big-endian platforms used memcpy() to
>>>> move the unaligned 64-bit value into a temp register to byte swap it.
>>>> Instead of introducing a dependency, manually byte-swap the values in
>>>> place.
>>>> ---------------------------------------------------------------------------------
>>>> 
>>>> --Gordon
>>>> 
>>>> [1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252541
>>> 
>>> I do my own builds but I'm running based on 19cca0b9613d
>>> with CommitDate 2021-01-09 16:21:33 -0800 :
>>> 
>>> # ~/fbsd-based-on-what-freebsd-main.sh mm-src
>>> 19cca0b9613d7c3058e41baf0204245119732235
>>> CommitDate: 2021-01-09 16:21:33 -0800
>>> 5d333ee67ac3 19cca0b9613d (HEAD -> mm-src) mm-src snapshot for mm's patched build in git context.
>>> FreeBSD RPi4B 13.0-CURRENT FreeBSD 13.0-CURRENT mm-src-c255807-g5d333ee67ac3 GENERIC-NODBG  arm64 aarch64 1300134 1300134
>>> 
>>> In other words, the history spanned for e83fdf8bb391 and
>>> after is:
>>> 
>>> * 	aio: fix the tests when ZFS is not available	Alan Somers	46 hours	2	-0/+5
>>> * 	linuxkpi: Fix the "error: unknown type name 'u32'" compilation issue when	Neel Chauhan	47 hours	1	-0/+1
>>> * 	netmap: vtnet: stop krings during interface reset	Vincenzo Maffione	48 hours	1	-1/+7
>>> * 	netmap: refactor netmap_reset	Vincenzo Maffione	2 days	1	-45/+20
>>> * 	netmap: iflib: fix asserts in netmap_fl_refill()	Vincenzo Maffione	2 days	1	-1/+2
>>> * 	netmap: iflib: stop krings during interface reset	Vincenzo Maffione	2 days	2	-1/+10
>>> * 	fileargs: add tests	Mariusz Zaborski	2 days	3	-0/+625
>>> * 	tcp: don't use KTLS socket option on listening sockets	Michael Tuexen	4 days	1	-0/+10
>>> * 	arm: revert MAXDSIZ change from 202aea9c82ea	Kyle Evans	2 days	1	-1/+1
>>> * 	kevent(2): Bugfix for wrong EVFILT_TIMER timeouts	Jan Kokemüller	2 days	1	-1/+1
>>> * 	ldd: renumber executable type constants	Ed Maste	2 days	1	-2/+2
>>> * 	diff: honour flags with -q	Ed Maste	2 days	2	-1/+13
>>> * 	sysctl: improve debug.kdb.panic_str description	Warner Losh	2 days	1	-1/+1
>>> * 	last(1): Add EXAMPLES section	Fernando Apesteguía	2 days	1	-4/+22
>>> * 	man(1): Bump .Dd	Fernando Apesteguía	2 days	1	-1/+1
>>> * 	man(1): Add EXAMPLES section	Fernando Apesteguía	2 days	1	-0/+35
>>> * 	mvneta: Acquire the softc lock before clearing the MIB	Mark Johnston	2 days	1	-0/+2
>>> * 	Add fib lookup testing module.	Alexander V. Chernikov	2 days	2	-0/+548
>>> * 	Bring DPDK route lookups to FreeBSD.	Alexander V. Chernikov	2 days	17	-0/+6030
>>> * 	Fix LINT kernel build after 01f2e864f79584c0cd250a8e7cfb501a9985768a.	Hans Petter Selasky	3 days	1	-1/+4
>>> * 	certctl: factor out certname resolution	Kyle Evans	3 days	1	-2/+17
>>> * 	certctl: replace hardcoded uses of /usr/local	Kyle Evans	3 days	1	-2/+3
>>> * 	fix big-endian platforms after 6733401935f8	Chuck Tuffli	3 days	1	-5/+9
>>> 
>>> The RPi4B is a 8 GiByte one, booted directly from a USB3 SSD,
>>> no microsd card involved. I can boot either u-boot style or
>>> UEFI/ACPI style from the same media, just switching config.txt
>>> content.
>>> 
>>> I do not have MMCCAM or the like:
>>> 
>>> # more /usr/fbsd/mm-src/sys/arm64/conf/GENERIC-NODBG 
>>> #
>>> # GENERIC -- Custom configuration for the arm64/aarch64
>>> #
>>> 
>>> include "GENERIC"
>>> 
>>> ident   GENERIC-NODBG
>>> 
>>> makeoptions     DEBUG=-g                # Build kernel with gdb(1) debug symbols
>>> 
>>> options         ALT_BREAK_TO_DEBUGGER
>>> 
>>> options         KDB                     # Enable kernel debugger support
>>> 
>>> # For minimum debugger support (stable branch) use:
>>> #options        KDB_TRACE               # Print a stack trace for a panic
>>> options         DDB                     # Enable the kernel debugger
>>> 
>>> # Extra stuff:
>>> #options        VERBOSE_SYSINIT=0       # Enable verbose sysinit messages
>>> #options        BOOTVERBOSE=1
>>> #options        BOOTHOWTO=RB_VERBOSE
>>> #options        KTR
>>> #options        KTR_MASK=KTR_TRAP
>>> ##options       KTR_CPUMASK=0xF
>>> #options        KTR_VERBOSE
>>> 
>>> # Disable any extra checking for. . .
>>> nooptions       DEADLKRES               # Enable the deadlock resolver
>>> nooptions       INVARIANTS              # Enable calls of extra sanity checking
>>> nooptions       INVARIANT_SUPPORT       # Extra sanity checks of internal structures, required by INVARIANTS
>>> nooptions       WITNESS                 # Enable checks to detect deadlocks and cycles
>>> nooptions       WITNESS_SKIPSPIN        # Don't run witness on spinlocks for speed
>>> nooptions       DIAGNOSTIC
>>> nooptions       MALLOC_DEBUG_MAXZONES   # Separate malloc(9) zones
>>> nooptions       BUF_TRACKING
>>> nooptions       FULL_BUF_TRACKING
>>> 
>> 
>> Looks like the message is from a KASSERT that does nothing
>> unless INVARIANTS/INVARIANT_SUPPORT is enabled.
>> 
>> Unfortunately, arftifacts.ci.freebsd.org has not started
>> getting git-based main builds yet. Normally I'd support
>> an official debug kernel from there and see if I could
>> repeat the problem.
>> 
>> So I've made my own debug kernel build for things as they
>> are in my context and it reproduced the problem:
>> 
>> Hit [Enter] to boot immediately, or any other key for command prompt.
>> Booting [/boot/kernel/kernel]...               
>> Using DTB provided by EFI at 0x7ef0000.
>> EFI framebuffer information:
>> addr, size     0x3e2fe000, 0x7e9000
>> dimensions     1920 x 1080
>> stride         1920
>> masks          0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000
>> ---<<BOOT>>---
>> panic: Too many early devmap mappings
>> cpuid = 0
>> time = 1
>> KDB: stack backtrace:
>> (null)() at 0xffff000000116980
>>        pc = 0xffff000000772af4  lr = 0xffff000000116980
>>        sp = 0xffff0000011f1320  fp = 0xffff0000011f1520
>> 
>> (null)() at 0xffff000000464710
>>        pc = 0xffff000000116980  lr = 0xffff000000464710
>>        sp = 0xffff0000011f1530  fp = 0xffff0000011f1590
>> 
>> (null)() at 0xffff0000004644b4
>>        pc = 0xffff000000464710  lr = 0xffff0000004644b4
>>        sp = 0xffff0000011f15a0  fp = 0xffff0000011f1650
>> 
>> (null)() at 0xffff0000007e9838
>>        pc = 0xffff0000004644b4  lr = 0xffff0000007e9838
>>        sp = 0xffff0000011f1660  fp = 0xffff0000011f1660
>> 
>> (null)() at 0xffff00000076f744
>>        pc = 0xffff0000007e9838  lr = 0xffff00000076f744
>>        sp = 0xffff0000011f1670  fp = 0xffff0000011f1690
>> 
>> (null)() at 0xffff000000782904
>>        pc = 0xffff00000076f744  lr = 0xffff000000782904
>>        sp = 0xffff0000011f16a0  fp = 0xffff0000011f16c0
>> 
>> (null)() at 0xffff0000002896b0
>>        pc = 0xffff000000782904  lr = 0xffff0000002896b0
>>        sp = 0xffff0000011f16d0  fp = 0xffff0000011f1790
>> 
>> (null)() at 0xffff0000007d9bb0
>>        pc = 0xffff0000002896b0  lr = 0xffff0000007d9bb0
>>        sp = 0xffff0000011f17a0  fp = 0xffff0000011f1820
>> 
>> (null)() at 0xffff00000028b814
>>        pc = 0xffff0000007d9bb0  lr = 0xffff00000028b814
>>        sp = 0xffff0000011f1830  fp = 0xffff0000011f1840
>> 
>> (null)() at 0xffff00000039e448
>>        pc = 0xffff00000028b814  lr = 0xffff00000039e448
>>        sp = 0xffff0000011f1850  fp = 0xffff0000011f1870
>> 
>> (null)() at 0xffff0000004af2ac
>>        pc = 0xffff00000039e448  lr = 0xffff0000004af2ac
>>        sp = 0xffff0000011f1880  fp = 0xffff0000011f18b0
>> 
>> (null)() at 0xffff00000077ef90
>>        pc = 0xffff0000004af2ac  lr = 0xffff00000077ef90
>>        sp = 0xffff0000011f18c0  fp = 0xffff0000011f1a00
>> 
>> (null)() at 0xffff00000000089c
>>        pc = 0xffff00000077ef90  lr = 0xffff00000000089c
>>        sp = 0xffff0000011f1a10  fp = 0x0000000000000000
>> 
>> KDB: enter: panic
>> [ thread pid 0 tid 0 ]
>> Stopped at      0xffff0000004aeeb4
>> db> dump
>> Cannot dump: no dump device specified.
>> db> 
> 
> 
> I stuck in some printf's showing figures in hexadecimal:
> 
> . . .
> ---<<BOOT>>---
> pmap_mapdev early_boot: akva_devmap_vaddr: ffff007fff816000 size: 1000
> pmap_mapdev early_boot: va: ffff007fff815000 VM_MAX_KERNEL_ADDRESS: ffff008000000000 L2_SIZE: 200000
> panic: Too many early devmap mappings
> cpuid = 0
> . . .
> 
> For reference:
> 
> #if defined(__aarch64__) || defined(__riscv)
>        if (early_boot) {
> printf("pmap_mapdev early_boot: akva_devmap_vaddr: %jx size: %jx\n",
> (uintmax_t) akva_devmap_vaddr, (uintmax_t) size);
>                akva_devmap_vaddr = trunc_page(akva_devmap_vaddr - size);
>                va = akva_devmap_vaddr;
> printf("pmap_mapdev early_boot: va: %jx VM_MAX_KERNEL_ADDRESS: %jx L2_SIZE: %jx\n",
> (uintmax_t) va, (uintmax_t) VM_MAX_KERNEL_ADDRESS, (uintmax_t) L2_SIZE);
>                KASSERT(va >= VM_MAX_KERNEL_ADDRESS - L2_SIZE,
>                    ("Too many early devmap mappings"));
>        } else
> #endif
> 
> So (hexadecimal):
> 
> VM_MAX_KERNEL_ADDRESS - L2_SIZE == ffff007fffe00000
> 
> and so va < VM_MAX_KERNEL_ADDRESS - L2_SIZE:
> 
> ffff007fff815000 < ffff007fffe00000
> 
> by:
> 
> ffff007fffe00000-ffff007fff815000 == 5eb000
> 
> I've not done anything to track down a relationship to
> e83fdf8bb391 .

The bisect point appears to make no sense, in that the
change was:

diff --git a/sys/dev/nvme/nvme.h b/sys/dev/nvme/nvme.h
index 67d02ba73fd8..b28a8d4348db 100644
--- a/sys/dev/nvme/nvme.h
+++ b/sys/dev/nvme/nvme.h
@@ -2042,16 +2042,20 @@ static inline void
 nvme_device_self_test_swapbytes(struct nvme_device_self_test_page *s __unused)
 {
 #if _BYTE_ORDER != _LITTLE_ENDIAN
-	uint64_t failing_lba;
-	uint32_t r;
+	uint8_t *tmp;
+	uint32_t r, i;
+	uint8_t b;
 
 	for (r = 0; r < 20; r++) {
 		s->result[r].poh = le64toh(s->result[r].poh);
 		s->result[r].nsid = le32toh(s->result[r].nsid);
 		/* Unaligned 64-bit loads fail on some architectures */
-		memcpy(&failing_lba, s->result[r].failing_lba, sizeof(failing_lba));
-		failing_lba = le64toh(failing_lba);
-		memcpy(s->result[r].failing_lba, &failing_lba, sizeof(failing_lba));
+		tmp = s->result[r].failing_lba;
+		for (i = 0; i < 4; i++) {
+			b = tmp[i];
+			tmp[i] = tmp[7-i];
+			tmp[7-i] = b;
+		}
 	}
 #endif
 }


This seems to only matter for the nvme device handling and
it appears to be a no-op for little endian contexts, both
before and after the change. So far as I know nvme is not
involved in the failing context and the code will be
compiled for little endian for the failing context.

The commits just-before and just-after also seem unlikely
candidates.

One of the reasons I prefer to test with artifact.ci
debug kernels is that it avoids things like my use of
-mcpu=cortex-a72 that my normal procedures are set up
for. Also, my context was a amd64->aarch64 cross-build
instead of being an aarch64 native build.

For reference, my aarch64 debug kernel config file,
cortex-A72 src.conf like file, make.conf like file, and
the script that runs the cortex-A72 debug kernel build
look like:

# more sys/arm64/conf/GENERIC-DBG
#
# GENERIC -- Custom configuration for the arm64/aarch64
#

include "GENERIC"

ident   GENERIC-DBG

makeoptions     DEBUG=-g                # Build kernel with gdb(1) debug symbols

options         ALT_BREAK_TO_DEBUGGER

options         KDB                     # Enable kernel debugger support

# For minimum debugger support (stable branch) use:
options         KDB_TRACE               # Print a stack trace for a panic
options         DDB                     # Enable the kernel debugger

# Extra stuff:
#options        VERBOSE_SYSINIT=0       # Enable verbose sysinit messages
#options        BOOTVERBOSE=1
#options        BOOTHOWTO=RB_VERBOSE
#options        KTR
#options        KTR_MASK=KTR_TRAP|KTR_PROC
##options       KTR_CPUMASK=0xF
#options        KTR_VERBOSE

# Enable any extra checking for. . .
options         DEADLKRES               # Enable the deadlock resolver
options         INVARIANTS              # Enable calls of extra sanity checking
options         INVARIANT_SUPPORT       # Extra sanity checks of internal structures, required by INVARIANTS
options         WITNESS                 # Enable checks to detect deadlocks and cycles
options         WITNESS_SKIPSPIN        # Don't run witness on spinlocks for speed
options         DIAGNOSTIC
options         MALLOC_DEBUG_MAXZONES=8 # Separate malloc(9) zones
options         BUF_TRACKING
options         FULL_BUF_TRACKING


# more ~/src.configs/src.conf.cortexA72dbg-clang-bootstrap.amd64-host 
TO_TYPE=aarch64
TOOLS_TO_TYPE=${TO_TYPE}
#
KERNCONF=GENERIC-DBG
TARGET=arm64
.if ${.MAKE.LEVEL} == 0
TARGET_ARCH=${TO_TYPE}
.export TARGET_ARCH
.endif
#
#WITH_CROSS_COMPILER=
WITH_SYSTEM_COMPILER=
WITH_SYSTEM_LINKER=
#
WITH_LIBCPLUSPLUS=
#WITH_LLD_BOOTSTRAP=
WITHOUT_BINUTILS_BOOTSTRAP=
WITH_ELFTOOLCHAIN_BOOTSTRAP=
#Disables avoiding bootstrap: WITHOUT_LLVM_TARGET_ALL=
WITH_LLVM_TARGET_AARCH64=
WITH_LLVM_TARGET_ARM=
WITHOUT_LLVM_TARGET_MIPS=
WITHOUT_LLVM_TARGET_POWERPC=
WITHOUT_LLVM_TARGET_RISCV=
WITHOUT_LLVM_TARGET_X86=
#WITH_CLANG_BOOTSTRAP=
WITH_CLANG=
WITH_CLANG_IS_CC=
WITH_CLANG_FULL=
WITH_CLANG_EXTRAS=
WITH_LLD=
WITH_LLD_IS_LD=
WITHOUT_BINUTILS=
WITH_LLDB=
#
WITH_BOOT=
WITHOUT_LIB32=
#
#
WITHOUT_WERROR=
#WERROR=
#MALLOC_PRODUCTION=
WITHOUT_MALLOC_PRODUCTION=
WITH_ASSERT_DEBUG=
WITH_LLVM_ASSERTIONS=
#
# Avoid stripping but do not control host -g status as well:
DEBUG_FLAGS+=
#
WITH_REPRODUCIBLE_BUILD=
WITH_DEBUG_FILES=
#
XCFLAGS+= -mcpu=cortex-a72
XCXXFLAGS+= -mcpu=cortex-a72
# There is no XCPPFLAGS but XCPP gets XCFLAGS content.
ACFLAGS.arm64cpuid.S+=  -mcpu=cortex-a72+crypto
ACFLAGS.aesv8-armx.S+=  -mcpu=cortex-a72+crypto
ACFLAGS.ghashv8-armx.S+=        -mcpu=cortex-a72+crypto

The ~/src.conf/make.conf is just comments.

# more ~/sys_build_scripts.amd64-host/make-cortexA72-debug-clang-bootstrap-amd64-host.sh 
kldload -n filemon && \
script ~/sys_typescripts/typescript_make_cortexA72_debug_clang_bootstrap-amd64-host-$(date +%Y-%m-%d:%H:%M:%S) \
env __MAKE_CONF="/root/src.configs/make.conf" SRCCONF="/dev/null" SRC_ENV_CONF="/root/src.configs/src.conf.cortexA72dbg-clang-bootstrap.amd64-host" \
WITH_META_MODE=yes \
MAKEOBJDIRPREFIX="/usr/obj/cortexA72dbg_clang/arm64.aarch64" \
make $*


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

_______________________________________________
[email protected] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arm
To unsubscribe, send any mail to "[email protected]"