Re: kern/60029: panic: cpu0: softints stuck for 16 seconds
"Thomas Klausner via gnats" <[email protected]>
| Newsgroups | gmane.os.netbsd.bugs |
|---|---|
| Message-ID | <[email protected]> |
The following reply was made to PR kern/60029; it has been noted by GNATS. From: Thomas Klausner <[email protected]> To: [email protected] Cc: Taylor R Campbell <[email protected]> Subject: Re: kern/60029: panic: cpu0: softints stuck for 16 seconds Date: Mon, 17 Aug 2026 16:41:30 +0200 Hi! Taylor helped with some debugging advice, scripts and patches. With this dtrace script running when it panics: #!/bin/sh dtrace -x cleanrate=50hz -n ' fbt::softint_dispatch:entry { self->starttime = timestamp } fbt::softint_dispatch:return { @[stringof curthread->l_name] = max(timestamp - self->starttime); self->starttime = 0; } profile:::tick-10sec { printf("########\n"); printa(@) } ' the last output is: 1 58467 :tick-10sec ######## softnet/16 5279 softnet/7 5851 softnet/24 5881 softnet/4 5931 softnet/13 6061 softnet/14 6081 softnet/15 6301 softnet/11 6853 softnet/18 6953 softnet/5 7063 softnet/12 7123 softnet/9 7234 softnet/6 7835 softnet/8 8095 softnet/26 9067 softnet/25 9468 softnet/22 10299 softnet/27 10410 softnet/10 10860 softnet/23 11361 softnet/28 11441 softnet/30 11902 softnet/21 11973 softnet/19 12664 softclk/26 12954 softclk/24 13295 softclk/18 13636 softnet/31 13755 softnet/20 14166 softnet/29 14197 softclk/16 15258 softclk/22 16771 softnet/17 17804 softclk/19 21841 softclk/17 21941 softclk/6 22092 softclk/31 22482 softclk/13 23093 softclk/29 23203 softclk/3 24395 softclk/20 24747 softclk/30 24756 softclk/21 25217 softclk/11 26089 softclk/10 26159 softclk/7 27141 softclk/5 33313 softclk/28 34134 softclk/27 35206 softclk/23 37480 softclk/12 38402 softnet/3 38552 softclk/9 39233 softclk/25 42529 softclk/4 45275 softclk/14 47859 softclk/2 49733 softnet/2 53009 softbio/6 298539 softbio/2 517519 softbio/11 569747 softbio/10 583824 softclk/1 646461 softbio/13 665937 softser/22 696714 softser/26 697867 softser/16 699560 softser/21 700773 softser/0 702666 softser/24 702877 softser/4 705201 softbio/7 707064 softser/17 710030 softser/30 712153 softser/7 717263 softser/29 718355 softser/28 718445 softser/11 718786 softser/3 719307 softser/9 721431 softser/20 724678 softser/5 727382 softser/8 730188 softser/19 732672 softser/10 734997 softser/25 735087 softser/18 740526 softser/27 743503 softser/14 747029 softser/13 751327 softser/31 752259 softser/12 756086 softser/6 765373 softser/1 785642 softser/2 796221 softser/23 803685 softbio/12 943056 softbio/15 995114 softbio/9 1084441 softbio/5 1195899 softbio/3 1325803 softbio/14 1467388 softclk/0 1870883 softbio/4 2178510 softbio/8 2452403 softclk/8 4178885 softbio/1 6734161 softser/15 6966325 softclk/15 6975753 softnet/1 8306846 softnet/0 8647985 softbio/0 8764967253 i.e. softbio is at 8764967253 ns =8.764.967.253 ns ~ 9s. With: kern.dump_on_panic=0 ddb.onpanic=1 ddb.commandonenter='bt;sync' I got a kernel dump that contained the CPU states for all CPUs (the default of 'kern.dump_on_panic = 1' doesn't provide these). using crash to dump the CPU states, I see: CPU 0 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 0 __kernel_end() at 0 vpanic() at vpanic+0x189 panic() at vprintf heartbeat() at heartbeat+0x315 hardclock() at hardclock+0x89 Xresume_lapic_ltimer() at Xresume_lapic_ltimer+0x1e --- interrupt --- mutex_spin_retry() at mutex_spin_retry+0x6f uvm_pageout_done() at uvm_pageout_done+0x3c uvm_aio_aiodone_pages() at uvm_aio_aiodone_pages+0x162 uvm_aio_aiodone() at uvm_aio_aiodone+0xcc dkiodone() at dkiodone+0xa8 biointr() at biointr+0x5c softint_dispatch() at softint_dispatch+0x112 CPU 1 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 1 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 747f2ee0b8bc: CPU 2 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 2 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 742f5a5a8ba2: CPU 3 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 3 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 763fa664c8bc: CPU 4 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 4 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 7152eab5490f: CPU 5 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 5 __kernel_end() at 0 uvm_availmem() at uvm_availmem+0xd uvm_kick_pdaemon() at uvm_kick_pdaemon+0xb uvm_pagealloc_strat() at uvm_pagealloc_strat+0x116 uvmfault_promote() at uvmfault_promote+0x17e uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 6f8e2590228d: CPU 6 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 6 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 76c7056a68bc: CPU 7 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 7 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 7152eab4e8bc: CPU 8 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 8 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 763fa664c8bc: CPU 9 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 9 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 7694206ed308: CPU a crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 10 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 76942020b8bc: CPU b crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 11 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 7694207b928d: CPU c crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 12 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 7b4bd772ae98: CPU d crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 13 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 76c705b18ba2: CPU e crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 14 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 7694207b928d: CPU f crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 15 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 7152eab4e8bc: CPU 10 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 16 __kernel_end() at 0 uvmpd_trylockowner() at uvmpd_trylockowner+0x8e uvmpdpol_selectvictim() at uvmpdpol_selectvictim+0x13e uvm_pageout() at uvm_pageout+0x1c6 CPU 11 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 17 __kernel_end() at 0 mtsleep() at mtsleep+0x71 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 7b4bd726d8bc: CPU 12 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 18 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 7db21b8118bc: CPU 13 crash: _kvm_kvatop(0) crash: _kvm_kvatop(7f7fff2e9d38) crash: kvm_read(0x7f7fff2e9d38, 8): invalid translation (invalid level 4 PDE) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 19 __kernel_end() at 0 CPU 14 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 20 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 7152eb0fc28d: CPU 15 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 21 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 7841037ee90f: CPU 16 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 22 __kernel_end() at 0 uvm_availmem() at uvm_availmem+0xd uvm_kick_pdaemon() at uvm_kick_pdaemon+0xb uvm_pagealloc_strat() at uvm_pagealloc_strat+0x116 uvmfault_promote() at uvmfault_promote+0xd9 uvm_fault_internal() at uvm_fault_internal+0x208f trap() at trap+0x46b --- trap (number 6) --- 76942063dd5e: CPU 17 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 23 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 76942020b8bc: CPU 18 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 24 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 71f7c938628d: CPU 19 crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 25 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0x208f trap() at trap+0x46b --- trap (number 6) --- 7b4bd7819845: CPU 1a crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 26 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 71f7c8dd88bc: CPU 1b crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 27 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 73f23af588bc: CPU 1c crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 28 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 7152eab4e8bc: CPU 1d crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 29 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 71f7c8dd88bc: CPU 1e crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 30 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 7b4bd727390f: CPU 1f crash: _kvm_kvatop(0) Crash version 11.99.7, image version 11.99.7. Kernel compiled without options LOCKDEBUG. System panicked: dump forced via kernel debugger Backtrace from time of crash is available. using CPU 31 __kernel_end() at 0 uvm_wait() at uvm_wait+0x28 uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0xd5a trap() at trap+0x46b --- trap (number 6) --- 71f7c938628d: and here is the lock status: crash> show all locks/t PID LID CPU STRUCT LWP * OWNER BITS LOCK 1245 > 1245 27 ffff860661bab000 10600 1 uvmpd_lock 3850 3850 4 ffff85fd150fa000 ffff8609afddd800 7 ffff85fa43393c40 trace: pid 11048 lid 11048 at 0xffffcea48d5f07b0 sleepq_block() at sleepq_block+0xea mtsleep() at mtsleep+0x11f uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0x208f trap() at trap+0x46b --- trap (number 6) --- copyout() at copyout+0x33 ubc_uiomove() at ubc_uiomove+0x12e tmpfs_read() at tmpfs_read+0xbe VOP_READ() at VOP_READ+0x42 vn_read() at vn_read+0x133 dofileread() at dofileread+0x75 sys_read() at sys_read+0x49 syscall() at syscall+0x9d --- syscall (number 3) --- syscall+0x9d: 13987>13987 2 ffff8609ae8fc000 10600 1 uvmpd_lock 2850 2850 27 ffff8606e9b74400 ffff8609afddd800 7 ffff85fa43393c40 trace: pid 11048 lid 11048 at 0xffffcea48d5f07b0 sleepq_block() at sleepq_block+0xea mtsleep() at mtsleep+0x11f uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0x208f trap() at trap+0x46b --- trap (number 6) --- copyout() at copyout+0x33 ubc_uiomove() at ubc_uiomove+0x12e tmpfs_read() at tmpfs_read+0xbe VOP_READ() at VOP_READ+0x42 vn_read() at vn_read+0x133 dofileread() at dofileread+0x75 sys_read() at sys_read+0x49 syscall() at syscall+0x9d --- syscall (number 3) --- syscall+0x9d: 11242>11242 1 ffff86065f36f800 10600 1 uvmpd_lock 26941>26941 12 ffff85fa60786400 10600 1 uvmpd_lock 21018>21018 30 ffff85fa39db2800 10600 1 uvmpd_lock 7755 7755 22 ffff8609af9ce800 ffff8609afddd800 7 ffff85fa43393c40 trace: pid 11048 lid 11048 at 0xffffcea48d5f07b0 sleepq_block() at sleepq_block+0xea mtsleep() at mtsleep+0x11f uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0x208f trap() at trap+0x46b --- trap (number 6) --- copyout() at copyout+0x33 ubc_uiomove() at ubc_uiomove+0x12e tmpfs_read() at tmpfs_read+0xbe VOP_READ() at VOP_READ+0x42 vn_read() at vn_read+0x133 dofileread() at dofileread+0x75 sys_read() at sys_read+0x49 syscall() at syscall+0x9d --- syscall (number 3) --- syscall+0x9d: 5709 > 5709 25 ffff85f9e2cd7c00 10600 1 uvmpd_lock 17056>17056 21 ffff8606092c8c00 10600 1 uvmpd_lock 21161>21161 23 ffff85fc4c140400 10600 1 uvmpd_lock 24514>24514 11 ffff85fc1d0d2c00 10600 1 uvmpd_lock 13396>13396 14 ffff85fc1d0d2800 10600 1 uvmpd_lock 23306>23306 9 ffff85fc1d0d2000 10600 1 uvmpd_lock 3832 > 3832 10 ffff85fe38511000 10600 1 uvmpd_lock 11184 11184 21 ffff85f95d697c00 ffff8609afddd800 7 ffff85fa43393c40 trace: pid 11048 lid 11048 at 0xffffcea48d5f07b0 sleepq_block() at sleepq_block+0xea mtsleep() at mtsleep+0x11f uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0x208f trap() at trap+0x46b --- trap (number 6) --- copyout() at copyout+0x33 ubc_uiomove() at ubc_uiomove+0x12e tmpfs_read() at tmpfs_read+0xbe VOP_READ() at VOP_READ+0x42 vn_read() at vn_read+0x133 dofileread() at dofileread+0x75 sys_read() at sys_read+0x49 syscall() at syscall+0x9d --- syscall (number 3) --- syscall+0x9d: 27982 27982 21 ffff85fcf3d51400 ffff8609afddd800 7 ffff85fa43393c40 trace: pid 11048 lid 11048 at 0xffffcea48d5f07b0 sleepq_block() at sleepq_block+0xea mtsleep() at mtsleep+0x11f uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0x208f trap() at trap+0x46b --- trap (number 6) --- copyout() at copyout+0x33 ubc_uiomove() at ubc_uiomove+0x12e tmpfs_read() at tmpfs_read+0xbe VOP_READ() at VOP_READ+0x42 vn_read() at vn_read+0x133 dofileread() at dofileread+0x75 sys_read() at sys_read+0x49 syscall() at syscall+0x9d --- syscall (number 3) --- syscall+0x9d: 1639 > 1639 13 ffff86065ca17000 10600 1 uvmpd_lock 4016 > 4016 6 ffff8609cda5e400 10600 1 uvmpd_lock 23230>23230 3 ffff85f832a8bc00 10600 1 uvmpd_lock 13750>13750 28 ffff8600715b5400 10600 1 uvmpd_lock 7511 > 7511 20 ffff860604642800 10600 1 uvmpd_lock 4462 > 4462 8 ffff8605fe69b800 10600 1 uvmpd_lock 20430>20430 15 ffff85faf7222400 10600 1 uvmpd_lock 3237 > 3237 4 ffff8609f5c5e400 10600 1 uvmpd_lock 5690 > 5690 7 ffff860604642c00 10600 1 uvmpd_lock 24357>24357 31 ffff8609b4a12c00 10600 1 uvmpd_lock 6012 > 6012 24 ffff860643ba1c00 10600 1 uvmpd_lock 19606>19606 26 ffff8608b2513400 10600 1 uvmpd_lock 29897 29897 3 ffff8609b3037400 ffff8609afddd800 7 ffff85fa43393c40 trace: pid 11048 lid 11048 at 0xffffcea48d5f07b0 sleepq_block() at sleepq_block+0xea mtsleep() at mtsleep+0x11f uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0x208f trap() at trap+0x46b --- trap (number 6) --- copyout() at copyout+0x33 ubc_uiomove() at ubc_uiomove+0x12e tmpfs_read() at tmpfs_read+0xbe VOP_READ() at VOP_READ+0x42 vn_read() at vn_read+0x133 dofileread() at dofileread+0x75 sys_read() at sys_read+0x49 syscall() at syscall+0x9d --- syscall (number 3) --- syscall+0x9d: 5433 > 5433 29 ffff86005e886800 10600 1 uvmpd_lock 15629 15629 6 ffff8609b197d400 ffff8609afddd800 7 ffff85fa43393c40 trace: pid 11048 lid 11048 at 0xffffcea48d5f07b0 sleepq_block() at sleepq_block+0xea mtsleep() at mtsleep+0x11f uvmfault_promote() at uvmfault_promote+0x4bb uvm_fault_internal() at uvm_fault_internal+0x208f trap() at trap+0x46b --- trap (number 6) --- copyout() at copyout+0x33 ubc_uiomove() at ubc_uiomove+0x12e tmpfs_read() at tmpfs_read+0xbe VOP_READ() at VOP_READ+0x42 vn_read() at vn_read+0x133 dofileread() at dofileread+0x75 sys_read() at sys_read+0x49 syscall() at syscall+0x9d --- syscall (number 3) --- syscall+0x9d: 23359>23359 18 ffff8609ec131800 10600 1 uvmpd_lock 0 > 935 16 ffff8604d7b76c00 ffff86142525f800 4 ffff85f78afa0f40 trace: pid 0 lid 4 at 0xffffcea469e83fe0 uvm_aio_aiodone() at uvm_aio_aiodone+0xcc dkiodone() at dkiodone+0xa8 biointr() at biointr+0x5c softint_dispatch() at softint_dispatch+0x112 0 > 4 0 ffff86142525f800 10600 1 uvmpd_lock As you can see, uvmpd_lock is heavily contested, but we don't understand yet if the contention alone is the reason for the panic. Cheers, Thomas