Re: milk-v jupiter pool (pmap?) corruption

On Sat, Apr 18, 2026 at 03:00:57PM +0200, Mark Kettenis wrote:
> > Date: Sat, 18 Apr 2026 14:35:49 +0200
> > From: Jeremie Courreges-Anglas <[email protected]>
> > 
> > Hey hey,
> > 
> > First report below was obtained while running cvs co/up over NFS over
> > smte0, disk is nvme.  This is after the pmap_growkernel() fix.
> 
> The first report is *exactly* what the pmap_growkernel() fix fixes.
> So I suspect your kernel didn't have the fix yet.

That doesn't match my memories but let's hope you're right. ;)

> The second one doesn't look familliar, but the "locking against
> myself" panic is the result of an unexpected kernel page fault that
> occurs shortly after allocating some new KVA.

Here's a series of lockups running one or two cvs up processes over
NFS/udp.  Usually the machine is hanging, but in the first case it
hits faults in a loop.

The two only other diffs applied are your com diff, your dwpcie diff
and the diff below which pretty-prints code page faults.  (ok?)


Index: arch/riscv64//riscv64/trap.c
===================================================================
RCS file: /cvs/src/sys/arch/riscv64/riscv64/trap.c,v
diff -u -p -r1.22 trap.c
--- arch/riscv64//riscv64/trap.c	5 Apr 2026 22:13:21 -0000	1.22
+++ arch/riscv64//riscv64/trap.c	20 Apr 2026 17:54:19 -0000
@@ -91,6 +91,11 @@ do_trap_supervisor(struct trapframe *fra
 	case EXCP_LOAD_PAGE_FAULT:
 		kdata_abort(frame);
 		break;
+	case EXCP_INST_PAGE_FAULT:
+		dump_regs(frame);
+		panic("Page fault while loading instruction at 0x%lx",
+		    frame->tf_sepc);
+		break;
 	case EXCP_BREAKPOINT:
 #ifdef DDB
 		db_trapper(frame->tf_sepc,0/*XXX*/, frame, exception);


OpenBSD 7.9-current (GENERIC.MP) #7: Tue Apr 21 09:45:45 CEST 2026
    [email protected]:/sys/arch/riscv64/compile/GENERIC.MP
[...]
OpenBSD/riscv64 (jupiter.leard.wxcvbn.org) (console)

login:

OpenBSD/riscv64 (jupiter.leard.wxcvbn.org) (console)

login: panic: softclock: invalid to_clock: 541414738
Stopped at      panic+0xfc:     addi    a0,zero,256    TID    PID    UID     PR
FLAGS     PFLAGS  CPU   C OM M AN D
 t  [p0]a n=i=c:  k0xerfneflff ffdci0ag7n8co5st9i9c 0a8ss
rttio[n 1"]IS SE=T(=c l-0>cxlff_fflfafgsf,c0 0C02LaSTa_ePEbN4
ItNG[2)"]  f=a=i le0d:x ff                                   D
filfeff c"00/0ubescr/as9rc0/
stys[/3]k er=n= /0xk00e00r0n00_00c0l00o00c01k
nttr.[c4]" , =l=in e0 4x820                  i
0000000000191              00
t[5] == 0x00000000028ac000
t[6] == 0x000000000000000c
s[0] == 0xffffffc078c59910
s[1] == 0xffffffc0792f4f80
s[2] == 0xffffffc00085881c
s[3] == 0xffffffc078d86d20
s[4] == 0xffffffc000828c33
s[5] == 0xffffffc07ea37dd0
s[6] == 0x0000000000000020
s[7] == 0xffffffc07ea37aa8
s[8] == 0x000000000000006f
s[9] == 0x000000000000006f
s[10] == 0xffffffc00081e8da
s[11] == 0x000000000000006e
a[0] == 0xffffffc000a617c0
a[1] == 0x2f29726964637273
a[2] == 0x0000000000308963
a[3] == 0x0000000000000007
a[4] == 0xffffffc0003cd6c8
a[5] == 0x0000000000000007
a[6] == 0x0000000000000000
a[7] == 0x0000000000000000
sepc == 0xffffffc0003125e2
sstatus == 0x0000000200000120
stval == 0x0000006964637293
scause == 0x000000000000000d
panic: Fatal page fault at 0xffffffc0003125e2: 0x6964637293
Stopped at      panic+0xfc:     addi    a0,zero,256    TID    PID    UID     PR
FLAGS     PFLAGS  CPU  COMMAND
t[0] == 0xffffffc078c59498
t[1] == 0xffffffc0002aaeb4
t[2] == 0xffffffc000beca90
t[3] == 0x0000000000000001
t[4] == 0x0000000000000191
t[5] == 0x00000000028ac000
t[6] == 0x000000000000000c
s[0] == 0xffffffc078c594a0
s[1] == 0xffffffc0792f4f80
s[2] == 0xffffffc00085881c
s[3] == 0xffffffc078d86d20
s[4] == 0xffffffc000828c33
s[5] == 0xffffffc07ea37dd0
s[6] == 0x0000000000000020
s[7] == 0xffffffc07ea37aa8
s[8] == 0x000000000000006f
s[9] == 0x000000000000006f
s[10] == 0xffffffc00081e8da
s[11] == 0x000000000000006e
a[0] == 0xffffffc000a617c0
a[1] == 0x2f29726964637273
a[2] == 0x0000000000308963
a[3] == 0x0000000000000007
a[4] == 0xffffffc0003cd6c8
a[5] == 0x0000000000000007
a[6] == 0x0000000000000000
a[7] == 0x0000000000000000
sepc == 0xffffffc0003125e2
sstatus == 0x0000000200000120
stval == 0x0000006964637293
scause == 0x000000000000000d
panic: Fatal page fault at 0xffffffc0003125e2: 0x6964637293
Stopped at      panic+0xfc:     addi    a0,zero,256    TID    PID    UID     PR
FLAGS     PFLAGS  CPU  COMMAND
t[0] == 0xffffffc078c59028
t[1] == 0xffffffc0002aaeb4
t[2] == 0xffffffc000beca90
t[3] == 0x0000000000000001
t[4] == 0x0000000000000191
t[5] == 0x00000000028ac000
t[6] == 0x000000000000000c
s[0] == 0xffffffc078c59030
s[1] == 0xffffffc0792f4f80
s[2] == 0xffffffc00085881c
s[3] == 0xffffffc078d86d20
s[4] == 0xffffffc000828c33
s[5] == 0xffffffc07ea37dd0
s[6] == 0x0000000000000020
s[7] == 0xffffffc07ea37aa8
s[8] == 0x000000000000006f
s[9] == 0x000000000000006f
s[10] == 0xffffffc00081e8da
s[11] == 0x000000000000006e
a[0] == 0xffffffc000a617c0
a[1] == 0x2f29726964637273
a[2] == 0x0000000000308963
a[3] == 0x0000000000000007
a[4] == 0xffffffc0003cd6c8
a[5] == 0x0000000000000007
a[6] == 0x0000000000000000
a[7] == 0x0000000000000000
sepc == 0xffffffc0003125e2
sstatus == 0x0000000200000120
stval == 0x0000006964637293
scause == 0x000000000000000d
panic: Fatal page fault at 0xffffffc0003125e2: 0x6964637293
Stopped at      panic+0xfc:     addi    a0,zero,256    TID    PID    UID     PR
FLAGS     PFLAGS  CPU  COMMAND
t[0] == 0xffffffc078c58bb8
t[1] == 0xffffffc0002aaeb4
t[2] == 0xffffffc000beca90
t[3] == 0x0000000000000001
t[4] == 0x0000000000000191
t[5] == 0x00000000028ac000
t[6] == 0x000000000000000c
s[0] == 0xffffffc078c58bc0
s[1] == 0xffffffc0792f4f80
s[2] == 0xffffffc00085881c
s[3] == 0xffffffc078d86d20
s[4] == 0xffffffc000828c33
s[5] == 0xffffffc07ea37dd0
s[6] == 0x0000000000000020
s[7] == 0xffffffc07ea37aa8
s[8] == 0x000000000000006f
s[9] == 0x000000000000006f
s[10] == 0xffffffc00081e8da
s[11] == 0x000000000000006e
a[0] == 0xffffffc000a617c0
a[1] == 0x2f29726964637273
a[2] == 0x0000000000308963
a[3] == 0x0000000000000007
a[4] == 0xffffffc0003cd6c8
a[5] == 0x0000000000000007
a[6] == 0x0000000000000000
a[7] == 0x0000000000000000
sepc == 0xffffffc0003125e2
sstatus == 0x0000000200000120
stval == 0x0000006964637293
scause == 0x000000000000000d
panic: Fatal page fault at 0xffffffc0003125e2: 0x6964637293
Stopped at      panic+0xfc:     addi    a0,zero,256    TID    PID    UID     PR
FLAGS     PFLAGS  CPU  COMMAND
t[0] == 0xffffffc078c58748
t[1] == 0xffffffc0002aaeb4
t[2] == 0xffffffc000beca90
t[3] == 0x0000000000000001
t[4] == 0x0000000000000191
t[5] == 0x00000000028ac000
t[6] == 0x000000000000000c
s[0] == 0xffffffc078c58750
s[1] == 0xffffffc0792f4f80
s[2] == 0xffffffc00085881c
s[3] == 0xffffffc078d86d20
s[4] == 0xffffffc000828c33
s[5] == 0xffffffc07ea37dd0
s[6] == 0x0000000000000020
s[7] == 0xffffffc07ea37aa8
s[8] == 0x000000000000006f
s[9] == 0x000000000000006f
s[10] == 0xffffffc00081e8da
s[11] == 0x000000000000006e
a[0] == 0xffffffc000a617c0
a[1] == 0x2f29726964637273
a[2] == 0x0000000000308963
a[3] == 0x0000000000000007
a[4] == 0xffffffc0003cd6c8
a[5] == 0x0000000000000007
a[6] == 0x0000000000000000
a[7] == 0x0000000000000000
sepc == 0xffffffc0003125e2
sstatus == 0x0000000200000120
stval == 0x0000006964637293
scause == 0x000000000000000d
panic: Fatal page fault at 0xffffffc0003125e2: 0x6964637293
Stopped at      panic+0xfc:     addi    a0,zero,256    TID    PID    UID     PR
FLAGS     PFLAGS  CPU  COMMAND
t[0] == 0xffffffc078c582d8
t[1] == 0xffffffc0002aaeb4
t[2] == 0xffffffc000beca90
t[3] == 0x0000000000000001
t[4] == 0x0000000000000191
t[5] == 0x00000000028ac000
t[6] == 0x000000000000000c
s[0] == 0xffffffc078c582e0
s[1] == 0xffffffc0792f4f80
s[2] == 0xffffffc00085881c
s[3] == 0xffffffc078d86d20
s[4] == 0xffffffc000828c33
s[5] == 0xffffffc07ea37dd0
s[6] == 0x0000000000000020
s[7] == 0xffffffc07ea37aa8
s[8] == 0x000000000000006f
s[9] == 0x000000000000006f
s[10] == 0xffffffc00081e8da
s[11] == 0x000000000000006e
a[0] == 0xffffffc000a617c0
a[1] == 0x2f29726964637273
a[2] == 0x0000000000308963
a[3] == 0x0000000000000007
a[4] == 0xffffffc0003cd6c8
a[5] == 0x0000000000000007
a[6] == 0x0000000000000000
a[7] == 0x0000000000000000
sepc == 0xffffffc0003125e2
sstatus == 0x0000000200000120
stval == 0x0000006964637293
scause == 0x000000000000000d
panic: Fatal page fault at 0xffffffc0003125e2: 0x6964637293
Stopped at      panic+0xfc:     addi    a0,zero,256    TID    PID    UID     PR
--db_more--t[0] == 0x8000000000000005
<reboot>

[...]

OpenBSD 7.9-current (GENERIC.MP) #7: Tue Apr 21 09:45:45 CEST 2026
    [email protected]:/sys/arch/riscv64/compile/GENERIC.MP

[...]

OpenBSD/riscv64 (jupiter.leard.wxcvbn.org) (console)

login: t[0] == 0xffffffc0009d8d80
t[1] == 0xffffffc00075026c
t[2] == 0xffffffffff000100
t[3] == 0x0000000000000000
t[4] == 0x0000000000000001
t[5] == 0x000000000000000f
t[6] == 0x0000000000000034
s[0] == 0xffffffc078cf1c00
s[1] == 0xffffffc0009d8ba2
s[2] == 0xffffffc0233b7a00
s[3] == 0xffffffc080825b00
s[4] == 0xffffffc0808238c8
s[5] == 0xffffffc000a30778
s[6] == 0x0000000000000000
s[7] == 0x0000000000000001
s[8] == 0x0000000000100000
s[9] == 0x0000000000100010
s[10] == 0xffffffffffefffff
s[11] == 0x0000000000000004
a[0] == 0xffffffc0808238c8
a[1] == 0x202020090a3b2973
a[2] == 0xffffffc0233b7a00
a[3] == 0x0000000000000000
a[4] == 0xffffffc0003cd6c8
a[5] == 0xfffffffffffffffc
a[6] == 0x0000000000000001
a[7] == 0x0000000000000006
sepc == 0xffffffc000759d9e
sstatus == 0x0000000200000120
stval == 0x000000090a3b29ab
scause == 0x000000000000000d
panic: Fatal page fault at 0xffffffc000759d9e: 0x90a3b29ab
Stopped atsys: 0x200

<reboot>
[...]

OpenBSD 7.9-current (GENERIC.MP) #7: Tue Apr 21 09:45:45 CEST 2026
    [email protected]:/sys/arch/riscv64/compile/GENERIC.MP

[...]

OpenBSD/riscv64 (jupiter.leard.wxcvbn.org) (console)

login: t[0] == 0xffffffc000ae6768
t[1] == 0xffffffc000312948
t[2] == 0x6666666666666667
t[3] == 0x0000000000000061
t[4] == 0xffffffc000ae6748
t[5] == 0xd280f06544adaab4
t[6] == 0x1915a10c2da83795
s[0] == 0xffffffc07a28d7b0
s[1] == 0xffffffc0233b7ba0
s[2] == 0x0000000000000000
s[3] == 0xffffffc080b6e8f0
s[4] == 0xffffffc080b6e8f0
s[5] == 0xffffffc000a30778
s[6] == 0x0000000000000001
s[7] == 0xffffffc0009d8bf0
s[8] == 0xffffffc0009d950a
s[9] == 0x00000000000001a4
s[10] == 0x00000000000001a4
s[11] == 0x0000000000000601
a[0] == 0xffffffc0233b7ba0
a[1] == 0x0000000000000009
a[2] == 0x000000000020b000
a[3] == 0x0000000000000000
a[4] == 0xffffffc0003cd6c8
a[5] == 0xfffffffffffffffc
a[6] == 0x0000000000000001
a[7] == 0x00000000001fa400
sepc == 0xffffffc00034f84c
sstatus == 0x0000000200000120
stval == 0x0000000000000008
scause == 0x000000000000000d
panic: Fatal page fault at 0xffffffc00034f84c: 0x000008
Stopped at�sys: 0x200

[...]

OpenBSD 7.9-current (GENERIC.MP) #7: Tue Apr 21 09:45:45 CEST 2026
    [email protected]:/sys/arch/riscv64/compile/GENERIC.MP

[...]

OpenBSD/riscv64 (jupiter.leard.wxcvbn.org) (console)

login: panic: kernel diagnostic assertion "refs > 1" failed: file "/usr/src/sys/kern/k
ern_synch.c", line 920
Stopped at      panic+0xfc:     addi    a0,zero,256    TID    PID    UID     PR
FLAGS     PFLAGS  CPU  COMMAND
*178372  45694   1000    0x100003          0    2K cvs
panic() at panic+0xfc
panic() at panic
refcnt_rele() at refcnt_rele
if_get() at if_get+0x4a
ip_output() at ip_output+0x1b2
udp_output() at udp_output+0x39c
sosend() at sosend+0x2b6
nfs_send() at nfs_send+0x56
nfs_request() at nfs_request+0x244
nfs_lookup() at nfs_lookup+0x2a8
VOP_LOOKUP() at VOP_LOOKUP+0x2e
vfs_lookup() at vfs_lookup+0x2d2
namei() at namei+0x2c2
vn_open() at vn_open+0x96
end trace frame: 0xffffffc080521b80, count: 0
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports.  Insufficient info makes it difficult to find and fix bugs.
ddb{2}> sh panic
*cpu2: kernel diagnostic assertion "refs > 1" failed: file "/usr/src/sys/kern/k
ern_synch.c", line 920
ddb{2}> mach cpuinfo
    0: stopping
    1: stopped
*   2: ddb
    3: stopped
    4: stopped
    5: stopped
    6: stopped
    7: stopped
ddb{2}> ps /o
    TID    PID    UID     PRFLAGS     PFLAGS  CPU  COMMAND
*178372  45694   1000    0x100003          0    2K cvs
ddb{2}> bo re
rebooting...

[...]

OpenBSD 7.9-current (GENERIC.MP) #7: Tue Apr 21 09:45:45 CEST 2026
    [email protected]:/sys/arch/riscv64/compile/GENERIC.MP

[...]

OpenBSD/riscv64 (jupiter.leard.wxcvbn.org) (console)

login: t[0] == 0xffffffc000ae6768
t[1] == 0xffffffc000312e40
t[2] == 0x6666666666666667
t[3] == 0x0000000000000061
t[4] == 0xffffffc000ae6748
t[5] == 0x00000000051eb850
t[6] == 0x00000000028f5c28
s[0] == 0xffffffc07a2bbc60
s[1] == 0xffffffc0233b29c0
s[2] == 0xffffffc000a8e0c0
s[3] == 0xffffffc0233b2f90
s[4] == 0xffffffc0009d93aa
s[5] == 0x00000003a641d034
s[6] == 0x0000000000000000
s[7] == 0xffffffc0009d952c
s[8] == 0xffffffc078ce59b0
s[9] == 0xffffffc0009d93b8
s[10] == 0xffffffc078daa800
s[11] == 0xffffffc0008a63c8
a[0] == 0xff1359e04c1cccd9
a[1] == 0xff1359e04c1cccd9
a[2] == 0x0000000000000007
a[3] == 0x0000000000000003
a[4] == 0x0000000000000000
a[5] == 0x0000000000000003
a[6] == 0x0000000000000000
a[7] == 0x0000000000000000
sepc == 0xffffffc0003ae43a
sstatus == 0x0000000200000120
stval == 0xffffffe04c1ccce1
scause == 0x000000000000000d
panic: Fatal page fault at 0xffffffc0003ae43a: 0xffffffe04c1ccce1
Stopped atsys: 0x200

<reboot>

-- 
jca