Out of memory debug
Emmanuel Dreyfus <[email protected]>
| Newsgroups | gmane.os.netbsd.devel.kernel |
|---|---|
| Message-ID | <[email protected]> |
Hello
I need some help to debug this system freeze (NetBSD 10.0/i386)
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
5734 5734 3 0 0 c7b0db40 httpd pnbufpl
19385 19385 3 0 0 c7b0d5c0 sshd pnbufpl
1502 1502 3 0 0 c7b0d300 httpd pnbufpl
938 938 3 0 0 c7b0d040 sshd pnbufpl
35 35 3 0 0 c7b198c0 httpd pnbufpl
4668 4668 3 0 0 c7b19600 sshd pnbufpl
13911 13911 3 0 0 c7b19080 httpd pnbufpl
8228 8228 3 0 0 c7111c80 sshd pnbufpl
26176 26176 3 0 0 c71119c0 sshd pnbufpl
5685 5685 3 0 0 c7111440 sshd pnbufpl
19503 19503 3 0 0 c7b5ec00 sshd pnbufpl
25400 25400 3 0 0 c7b5e100 sshd pnbufpl
3232 3232 3 0 0 c79f7d00 sshd pnbufpl
25136 25136 3 1 1000000 c6e37200 send_nsca kva-4096
2123 2123 3 0 180 c879ec80 sh wait
6187 6187 3 0 180 c879e9c0 ksh pause
19400 19400 3 0 180 c879e700 cron pipe_rd
9271 9271 3 0 0 c71a1740 httpd pnbufpl
4430 4430 3 0 0 c6e39500 httpd pnbufpl
9670 9670 3 0 0 c79f7780 du vmem
2384 2384 3 1 180 c71a1a00 sh wait
7015 7015 3 1 180 c6e37d00 cron pipe_rd
28504 28504 3 1 180 c79f74c0 lastfss nanoslp
1957 1957 3 0 0 c71a11c0 httpd pnbufpl
16223 16223 3 0 0 c79f7a40 httpd pnbufpl
28978 28978 3 0 0 c7b19340 httpd pnbufpl
16981 16981 3 0 0 c7b19b80 httpd pnbufpl
24276 24276 3 0 0 c71a1480 httpd pnbufpl
26540 26540 3 0 0 c879e180 httpd pnbufpl
20819 20819 3 0 0 c7111700 httpd pnbufpl
4173 4173 3 0 0 c7b0d880 httpd pnbufpl
10817 10817 3 0 180 c879e440 lastfss nanoslp
27594 27594 3 1 180 c7b5e940 lastfss nanoslp
978 978 3 0 1000000 c7111180 ksh pnbufpl
829 829 3 0 40 c6e37a40 httpd sigacts
977 977 3 0 40 c71a1cc0 cron pnbufpl
975 975 3 0 0 c79f7200 sshd kva-4096
601 601 3 0 180 c6e39240 syslogd kqueue
1 1 3 0 180 c6e5e640 init wait
0 14299 3 0 200 c7b5e3c0 fss2 fssbs
0 10332 3 1 200 c6f1f400 fss1 fssbs
0 27862 3 0 200 c7b5e680 fss0 fssbs
0 329 3 0 200 c6f1f6c0 physiod physiod
0 196 3 0 240 c6f1fc40 ioflush biowait
0 195 3 0 240 c6e7c100 pooldrain xclocv
0 > 194 7 1 40200 c6f1f980 pgdaemon
0 234 3 1 200 c6f1f140 raidio1 raidiow
0 233 3 0 200 c6e7cc00 raid1 rfnodeq
0 168 3 1 200 c6e7c940 raidio0 raidiow
0 166 3 0 200 c6e7c680 raid0 rfnodeq
0 167 3 1 200 c6e7c3c0 swwreboot swwreboot
0 31 3 1 200 c6e37780 scsibus0 sccomp
0 63 3 0 200 c6e374c0 usb2 usbevt
0 126 3 0 200 c6e5ebc0 usb3 usbevt
0 125 3 0 200 c6e5e900 npfgc0 npfgcw
0 124 3 1 200 c6e5e380 rt_free rt_free
0 123 3 1 200 c6e5e0c0 unpgc unpgc
0 122 3 1 200 c6e54b80 key_timehandler key_timehandler
0 121 3 1 200 c6e548c0 carp_wqinput/1 carp_wqinput
0 120 3 0 200 c6e54600 carp_wqinput/0 carp_wqinput
0 119 3 1 200 c6e54340 icmp_wqinput/1 icmp_wqinput
0 118 3 0 200 c6e54080 icmp_wqinput/0 icmp_wqinput
0 117 3 1 200 c6e3fb40 rt_timer rt_timer
0 116 3 0 200 c6e397c0 vmem_rehash vmem_rehash
0 115 3 0 200 c6e3f880 usb1 usbevt
0 114 3 0 200 c6e3f5c0 usb0 usbevt
0 111 3 0 200 c6e39d40 acpitz1 acpitz1
0 110 3 0 200 c6e39a80 acpitz0 acpitz0
0 101 3 0 200 c6d96cc0 entbutler entropy
0 100 3 0 240 c6d96a00 atabus3 atath
0 99 3 0 240 c6d96740 atabus2 atath
0 98 3 0 240 c6d96480 atabus1 atath
0 97 3 0 240 c6d961c0 atabus0 atath
0 96 3 0 200 c6d07c80 wm1Reset wm1Reset
0 30 3 1 200 c6d079c0 wm1TxRx/1 wm1TxRx
0 29 3 0 200 c6d07700 wm1TxRx/0 wm1TxRx
0 28 3 0 200 c6d07440 wm0Reset wm0Reset
0 27 3 1 200 c6d07180 wm0TxRx/1 wm0TxRx
0 26 3 0 200 c6c5bc40 wm0TxRx/0 wm0TxRx
0 25 3 0 200 c6c5b980 usbtask-dr usbtsk
0 24 3 0 200 c6c5b6c0 usbtask-hc usbtsk
0 23 3 0 200 c6c5b400 apm0 apmev
0 22 2 1 200 c6c5b140 xcall/1
0 21 1 1 200 c6c58c00 softser/1
0 20 1 1 200 c6c58940 softclk/1
0 19 1 1 200 c6c58680 softbio/1
0 18 1 1 200 c6c583c0 softnet/1
0 17 1 1 201 c6c58100 idle/1
0 16 3 0 200 c6957bc0 sysmon smtaskq
0 23 3 0 200 c6c5b400 apm0 apmev
0 22 2 1 200 c6c5b140 xcall/1
0 21 1 1 200 c6c58c00 softser/1
0 20 1 1 200 c6c58940 softclk/1
0 19 1 1 200 c6c58680 softbio/1
0 18 1 1 200 c6c583c0 softnet/1
0 17 1 1 201 c6c58100 idle/1
0 16 3 0 200 c6957bc0 sysmon smtaskq
0 15 3 0 200 c6957900 pmfsuspend pmfsuspend
0 14 3 0 200 c6957640 pmfevent pmfevent
0 13 3 0 200 c6957380 sopendfree sopendfr
0 12 3 0 200 c69570c0 ifwdog ifwdog
0 11 3 1 200 c6949b80 iflnkst iflnkst
0 10 3 0 200 c69498c0 nfssilly nfssilly
0 9 3 0 240 c6949600 vdrain biolock
0 8 3 0 200 c6949340 modunload mod_unld
0 7 3 0 200 c6949080 xcall/0 xcall
0 6 1 0 200 c6947b40 softser/0
0 5 1 0 200 c6947880 softclk/0
0 4 1 0 200 c69475c0 softbio/0
0 3 1 0 200 c6947300 softnet/0
0 > 2 1 0 201 c6947040 idle/0
0 0 3 0 200 c42f9180 swapper uvm
Right, everyone is waiting for memory. show uvmexp says
Current UVM status:
pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12, ncolors=8
470488 VM pages: 214225 active, 104655 inactive, 35 wired, 6916 free
pages 4175 anon, 311864 file, 2876 exec
freemin=512, free-target=682, wired-max=156829
resv-pg=1, resv-kernel=10
bootpages=8731, poolpages=122009
faults=32797486, traps=32383441, intrs=23973183, ctxswitch=55821500
softint=50312117, syscalls=51227003
fault counts:
noram=15, noanon=0, pgwait=0, pgrele=0
ok relocks(total)=138534(138534), anget(retrys)=4696112(0), amapcopy=3836260
neighbor anon/obj pg=3139182/42459207, gets(lock/unlock)=12762198/138534
cases: anon=2997851, anoncow=1698261, obj=10951495, prcopy=1810668, przero=5
845588
daemon and swap counts:
woke=3620, revs=3629, scans=9839194, obscans=9716102, anscans=0
busy=0, freed=9716102, reactivate=52, deactivate=9955744
pageouts=0, pending=0, nswget=0
nswapdev=0, swpgavail=0
swpages=0, swpginuse=0, swpgonly=0, paging=0
There are free pages, and repeated calls show only faults/traps/intrs/etc
changing:
< faults=32797486, traps=32383440, intrs=23972454, ctxswitch=55817448
< softint=50306508, syscalls=51226997
---
> faults=32797486, traps=32383441, intrs=23973183, ctxswitch=55821500
> softint=50312117, syscalls=51227003
Running bt/a c6f1f980 at various times shows pageadaemon is alive. Is
it just that the system is out of memory, and nothing can be reclaimed?
There is no swap configured.
--
Emmanuel Dreyfus
[email protected]