Re: Page fault early in boot

> Date: Fri, 11 Apr 2025 08:22:59 +0000
> From: Emmanuel Dreyfus <[email protected]>
> 
> I tried to boot NetBSD-current (i386 and amd6), BIOS and UEFI modes) on a 
> Dell server, it always panic on a "fatal page fault in supervisor mode"
> early during the boot. The backtrace says we went there through
> main/fork1/lwp_create/memset
> 
> A screenshot is available there:
> https://dl.espci.fr/ticket/2a34048a0a9749c68cfb0dc380cc2259
> 
> Any idea how to debug that? I expected NetBSD to lack some hardware
> support on this recent machine, not to be unable to reach single user 
> mode.

This looks like the same issue as:

PR kern/57661: Crash when booting on Xeon Silver 4416+ in KVM/Qemu
https://gnats.netbsd.org/57661

Some additional diagnostics from the PR:

> The stack trace is a bit of a red herring, I traced down the memset to line 343 of src/sys/arch/x86/x86/fpu.c, so we actually have:
> lwp_create() -> uvm_lwp_fork() -> cpu_lwp_fork() -> fpu_lwp_fork() -> memset()
> 
> Adding
>   printf("DEBUG: sizeof(pcb2->pcb_savefpu)==%ld\n", sizeof(pcb2->pcb_savefpu));
>   printf("DEBUG: x86_fpu_save_size==%d\n", x86_fpu_save_size);
> 
> before the memset() call prints
> 
>   [   1.8432366] DEBUG: sizeof(pcb2->pcb_savefpu)==576
>   [   1.8432366] DEBUG: x86_fpu_save_size==11008
> 
> Changing the VM's CPU to Sandy Bridge prints
> 
>   [  1.8897648] DEBUG: sizeof(pcb2->pcb_savefpu)==576
>   [  1.8897648] DEBUG: x86_fpu_save_size==832

So, the x86_fpu_save_size is larger than we can handle and we don't
report the problem gracefully either -- my initial attempt to put a
more obvious panic in x86/identcpu.c when we initially determine
x86_fpu_save_size hit the problem that it happens too early and sends
the machine into a reboot loop.

Unfortunately, simply expanding the space for pcb_savefpu is not
trivial:

     80 /*
     81  * IMPORTANT NOTE: this structure, including the variable-sized FPU state at
     82  * the end, must fit within one page.
     83  */
     84 struct pcb {
...
    103 	union savefpu	pcb_savefpu __aligned(64); /* floating point state */
    104 	/* **** DO NOT ADD ANYTHING HERE **** */
    105 };

https://nxr.netbsd.org/xref/src/sys/arch/amd64/include/pcb.h?r=1.32#80

The top comment came from this commit:

> Module Name:    src
> Committed By:   maxv
> Date:           Tue Mar 17 17:18:49 UTC 2020
> 
> Modified Files:
>         src/sys/arch/amd64/include: cpu.h param.h pcb.h types.h
>         src/sys/arch/x86/x86: vm_machdep.c
> 
> Log Message:
> Add a redzone between the pcb and the stack. Sent to port-amd64@.

https://mail-index.netbsd.org/source-changes/2020/03/17/msg115178.html

So, the immediate issue is that the memset in fpu_lwp_fork is hitting
the guard page.  Perhaps resolving that is simply a matter of teaching
cpu_uarea_alloc to advance by round_page(offsetof(struct pcb,
pcb_savefpu) + x86_fpu_save_size) rather than by PAGE_SIZE when
deciding where to set the guard page:

    364 void *
    365 cpu_uarea_alloc(bool system)
    366 {
    367 	vaddr_t base, va;
    368 	paddr_t pa;
    369 
    370 	base = uvm_km_alloc(kernel_map, USPACE + PAGE_SIZE, 0,
    371 	    UVM_KMF_WIRED|UVM_KMF_WAITVA);
    372 
    373 	/* Page[1] = RedZone */
    374 	va = base + PAGE_SIZE;
    375 	if (!pmap_extract(pmap_kernel(), va, &pa)) {
    376 		panic("%s: impossible, Page[1] unmapped", __func__);
    377 	}
    378 	pmap_kremove(va, PAGE_SIZE);
    379 	uvm_pagefree(PHYS_TO_VM_PAGE(pa));

https://nxr.netbsd.org/xref/src/sys/arch/x86/x86/vm_machdep.c?r=1.46#352

But the bottom comment, /* **** DO NOT ADD ANYTHING HERE **** */, is
much older:

> Module Name:    src
> Committed By:   dsl
> Date:           Thu Feb 20 18:19:10 UTC 2014
> 
> Modified Files:
>         src/sys/arch/amd64/amd64: genassym.cf machdep.c
>         src/sys/arch/amd64/include: pcb.h proc.h
>         src/sys/arch/i386/i386: locore.S machdep.c
>         src/sys/arch/i386/include: pcb.h proc.h
>         src/sys/arch/x86/x86: vm_machdep.c
>         src/sys/sys: param.h
> 
> Log Message:
> Move the amd64 and i386 pcb to the bottom of the uarea, and move the
>   kernel stack to the top.
> Change the pcb layouts so that fpu save area is at the end and is
>   64byte aligned ready for xsave (saving the ymm registers).
> Welcome to 6.99.32

https://mail-index.netbsd.org/source-changes/2014/02/20/msg051841.html

It's not obvious to me whether the comment affects anything at hand.
Perhaps it's just admonishing future developers not to add things
_other than fpu state_ because the fpu state is actually variable size
and may extend past the size of a union savefpu object.