Re: very strange "temporary" filesytem lockups on netbsd-4 (i386)
[email protected] (Christos Zoulas)
| Newsgroups | gmane.os.netbsd.help,gmane.os.netbsd.devel.kernel |
|---|---|
| Message-ID | <[email protected]> |
In article <[email protected]>, Greg A. Woods <[email protected]> wrote: >OK, here's what's been happening since my first report.... > >I built a new kernel with the recent changes on netbsd-4 done to fix >many deadlock problems in NetBSD's nullfs and unionfs virtual >filesystems, hoping without much expectation that this would change the >behaviour of the temporary freezes I was seeing. Indeed there were no >improvements and no change in behaviour either. > >I've worked around the problem for now with the following line added to >the bottom of /etc/rc.local: > > nohup sh -c 'while : ; do dd if=/dev/rld0d of=/dev/null count=1; sleep >1; done' & > >This does an IO on the stuck device once per second, restarting it, and >it seems to keep everything moving along reasonably smoothly. > >I did capture some not-terribly-interesting kernel info from DDB though >and all you basically see is "ld" stuck in getblk state: > >[Mon Feb 19 22:50:28 2007]login: [-- break #1 sent -- `\z' -- Mon Feb >19 22:54:51 2007] >Stopped at netbsd:cpu_Debugger+0x4: popl %ebp >[Mon Feb 19 22:54:51 2007]db{0}> ps >[Mon Feb 19 22:54:53 2007] PID PPID PGRP UID S >FLAGS LWPS COMMAND WAIT >[Mon Feb 19 22:54:53 2007] 3920 18420 10164 1000 2 >0x4002 1 ld getblk >[Mon Feb 19 22:54:53 2007] 18420 16402 10164 1000 2 >0x4002 1 collect2 wait >[Mon Feb 19 22:54:53 2007] 16402 3137 10164 1000 2 >0x4002 1 i386--netbsdelf- wait >[Mon Feb 19 22:54:53 2007] 3137 9416 10164 1000 2 >0x4002 1 sh wait >[Mon Feb 19 22:54:53 2007] 9416 13265 10164 1000 2 >0x4002 1 nbmake wait >[Mon Feb 19 22:54:53 2007] 13265 3034 10164 1000 2 >0x4002 1 sh wait >[Mon Feb 19 22:54:53 2007] 3034 15061 10164 1000 2 >0x4002 1 nbmake wait >[Mon Feb 19 22:54:53 2007] 15061 2593 10164 1000 2 >0x4002 1 sh wait >[Mon Feb 19 22:54:53 2007] 2593 27295 10164 1000 2 >0x4002 1 nbmake wait >[Mon Feb 19 22:54:53 2007] 27295 15819 10164 1000 2 >0x4002 1 sh wait >[Mon Feb 19 22:54:53 2007] 18596 6309 18596 1000 2 >0x4002 1 tail kqread >[Mon Feb 19 22:54:53 2007] 6309 20011 6309 1000 2 >0x4002 1 ksh pause >[Mon Feb 19 22:54:53 2007] 20011 763 763 0 2 >0x4100 1 rlogind poll >[Mon Feb 19 22:54:53 2007] 15819 10280 10164 1000 2 >0x4002 1 nbmake wait >[Mon Feb 19 22:54:53 2007] 10280 28521 10164 1000 2 >0x4002 1 sh wait >[Mon Feb 19 22:54:53 2007] 28521 27102 10164 1000 2 >0x4002 1 nbmake wait >[Mon Feb 19 22:54:53 2007] 27102 2473 10164 1000 2 >0x4002 1 sh wait >[Mon Feb 19 22:54:53 2007] 12119 621 12119 1000 2 >0x4102 1 systat ttyin >[Mon Feb 19 22:54:53 2007] 2473 9639 10164 1000 2 >0x4002 1 nbmake wait >[Mon Feb 19 22:54:53 2007] 9639 10936 10164 1000 2 >0x4002 1 sh wait >[Mon Feb 19 22:54:53 2007] 10936 10384 10164 1000 2 >0x4002 1 nbmake wait >[Mon Feb 19 22:54:53 2007] 10384 10164 10164 1000 2 >0x4002 1 sh wait >[Mon Feb 19 22:54:53 2007] 10164 621 10164 1000 2 >0x4002 1 ksh pause >[Mon Feb 19 22:54:53 2007] 621 620 621 1000 2 >0x4003 1 ksh pause >[Mon Feb 19 22:54:54 2007] 620 763 763 0 2 >0x4100 1 rlogind poll >[Mon Feb 19 22:54:54 2007] 695 1 695 0 2 >0x4002 1 getty ttyin >[Mon Feb 19 22:54:54 2007] 826 1 826 0 2 >0x4002 1 getty ttyin >[Mon Feb 19 22:54:54 2007] 782 1 782 0 2 >0x4002 1 getty ttyin >[Mon Feb 19 22:54:54 2007]--db_more-- >[Mon Feb 19 22:55:01 2007]db{0}> bt >[Mon Feb 19 22:55:08 >2007]cpu_Debugger(c5145200,0,cea02f38,c0334f70,c5145260) at >netbsd:cpu_Debugger+0x4 >[Mon Feb 19 22:55:08 2007]comintr(c5145200,0,cea00010,30,10) at >netbsd:comintr+0x6e0 >[Mon Feb 19 22:55:08 2007]Xintr_ioapic_edge4() at >netbsd:Xintr_ioapic_edge4+0x9c >[Mon Feb 19 22:55:08 2007]--- interrupt --- >[Mon Feb 19 22:55:08 2007]netbsd:cpu_switch+0xab: >[Mon Feb 19 22:55:46 2007]db{0}> machine cpu 1 >[Mon Feb 19 22:55:55 2007]using CPU 1 >[Mon Feb 19 22:55:55 2007]db{0}> bt >[Mon Feb 19 22:55:58 2007]netbsd:cpu_switch+0xab: >[Mon Feb 19 22:55:58 2007]db{0}> machine cpu 2 >[Mon Feb 19 22:56:04 2007]using CPU 2 >[Mon Feb 19 22:56:04 2007]db{0}> bt >[Mon Feb 19 22:56:05 2007]netbsd:cpu_switch+0xab: >[Mon Feb 19 22:56:05 2007]db{0}> machine cpu 3 >[Mon Feb 19 22:56:08 2007]using CPU 3 >[Mon Feb 19 22:56:08 2007]db{0}> bt >[Mon Feb 19 22:56:09 2007]netbsd:cpu_switch+0xab: >[Mon Feb 19 22:56:09 2007]db{0}> machine cpu 0 >[Mon Feb 19 22:56:14 2007]using CPU 0 >[Mon Feb 19 22:56:14 2007]db{0}> bt >[Mon Feb 19 22:56:17 >2007]cpu_Debugger(c5145200,0,cea02f38,c0334f70,c5145260) at >netbsd:cpu_Debugger+0x4 >[Mon Feb 19 22:56:17 2007]comintr(c5145200,0,cea00010,30,10) at >netbsd:comintr+0x6e0 >[Mon Feb 19 22:56:17 2007]Xintr_ioapic_edge4() at >netbsd:Xintr_ioapic_edge4+0x9c >[Mon Feb 19 22:56:17 2007]--- interrupt --- >[Mon Feb 19 22:56:17 2007]netbsd:cpu_switch+0xab: >[Mon Feb 19 22:56:17 2007]db{0}> cont >[Mon Feb 19 22:57:03 2007] >[Mon Feb 19 22:57:08 2007] >[Mon Feb 19 22:57:08 2007]NetBSD/i386 (once) (constty) >[Mon Feb 19 22:57:08 2007] >[Mon Feb 19 22:57:08 2007]login: > > >It looks more and more that the big culprit is the extreme age and >bit-rot in the aac(4) driver is at fault. There have been many >bug-fixes, improvements, and updates to the original FreeBSD sources >since it was ported to NetBSD, including this interesting one with a >possible fix and explanation for lost interrupts which would explain why >the I/O requests pending at the time of the hang never complete until >another I/O is started. > >Unfortunately this fix was done after the driver's interrupt handling >was threaded: > >---------------------------- >revision 1.46 >date: 2003/01/11 01:59:21; author: scottl; state: Exp; lines: +50 -39 >Major bugfixes for large memory and fast systems. > >aac.c: > Re-arrange the interrupt handler to optimize the common case of > the adapter interrupting us because one or more commands are complete, > and do a read across the pci bus to ensure that all posted status > writes are flushed. This should close a race that could cause command > completion interrupts to be lost. > Follow the spec a bit closer when filling out command structures. > Enable the Fast Response feature to eliminate the need for the card > to DMA successfull command completions back into host memory. > Tell the controller how much physical memory we have. Without this > there was a chance that our DMA regions would collide with the > memory window used by the cache on the controller. The result would > be massive data corruption. This seemed to mainly affect systems with > >2GB of memory. > Fix a few whitespace problems. > >aac_debug.c: > Add an extra diagnostic when printing out commands. > >aac_disk.c: > Add extra sanity checks. > >aacreg.h: > Prepare for making this 64-bit clean by reducing the use of enumeration > types in structures. > >Many thanks to Justin Gibbs for helping track these down. >---------------------------- > > >Re-porting the driver from FreeBSD seems quite daunting, given the >major divergences growing between FreeBSD and NetBSD, particularly in >storage device drivers. > >However much/all of the good work put into the FreeBSD driver has been >picked up by OpenBSD in their re-write of the aac(4) driver. It would >seem on first glance to be much easier to port the new OpenBSD driver >back over to NetBSD. Unfortunately I don't have time to do this all on >my own at the moment, though I would be able to help test, at least on >the netbsd-4 branch, as I have several HP NetRAID cards as well as this >Dell PE2650 with the embedded PERC controller. > >At minimum it sure would be nice if someone could figure some way to fix >the race condition that seems to be causing interrupts to be lost. > >Should I open a PR with the info in this and the last e-mail? > Yes, please file a PR. We *really* need to fix the aac driver... christos