RE: kernel core dump hangs with the SMP kernel on dual cpu board (FreeBSD 6.2)
"Yong Rao" <[email protected]> Fri, 6 Jul 2007 22:01:08 -0700
| Newsgroups | gmane.os.freebsd.devel.smp |
|---|---|
| Message-ID | <9E2742C54E161041A53F36F9A8DC31BE0AE53B@EXCH-CLUSTER-04.force10networks.com> |
Hello, I am attaching more debugs. After core dump hangs, I am able to get into ddb. But the trace for the thread where core dump is running does not look right. I was able to isolate the dumper function - ad_dump where it seems to be hanging. Hopefully these debugs will give more clues. Please advise on what more to look for. I am new to SMP and still coming up to speed. The following debugs are taken after core dump hangs: a) show pcpu b) bt 100054 -- 100054 is the thread ID, which does the core dump c) alltrace (in attached file "alltrace.txt") d) step and next command in debug a) --- db> show pcpu cpuid = 1 curthread = 0xc9450d80: pid 16 "swi6: Giant taskq" curpcb = 0xe7fe9d90 fpcurthread = none idlethread = 0xc944f600: pid 10 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: db> show pcpu 0 cpuid = 0 curthread = 0xc96eb000: pid 894 "disablemeta" curpcb = 0xe9b4fd90 fpcurthread = none idlethread = 0xc944f780: pid 11 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: db> b) --- db> bt 100054 Tracing pid 894 tid 100054 td 0xc96eb000 sched_switch(3225858983,10,0,3382180240,3377804288,...) at sched_switch+331 _end(53,128,606517178,0,0,...) at 3920951848 _end(3377796632,3379230208,3379230476,3377796608,3379229696,...) at 3377796608 _end(3228796064,3228796064,3228796064,3228796064,3228796064,...) at 3379230208 null_method(0,0,0,0,0,...) at null_method db> c) --- Please see attached file "alltrace.txt" for details. d) trace --- db> trace Tracing pid 943 tid 100080 td 0xc998d000 kdb_enter(3228283314) at kdb_enter+43 panic(3390017776,3237235458,3956644984,21554,3389785424,...) at panic+295 mtpioctl(3389997312,3237235458,3389811456,3,3382235136,...) at mtpioctl+193 devfs_ioctl_f(3379694976,3237235458,3389811456,3376546688,3382235136) at devfs_5 ioctl(3382235136,3956645124) at ioctl+874 syscall(59,59,59,3216893976,134516360,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (54, FreeBSD ELF32, ioctl), eip = 672367343, esp = 3216893900, ebp - db> s [thread pid 943 tid 100080 ] Stopped at kdb_enter+44: leave db> s [thread pid 943 tid 100080 ] Stopped at kdb_enter+45: ret db> s [thread pid 943 tid 100080 ] Stopped at panic+295: addl $4,%esp db> s [thread pid 943 tid 100080 ] Stopped at panic+298: pushl $561 db> s [thread pid 943 tid 100080 ] Stopped at panic+303: pushl $3228283350 db> s [thread pid 943 tid 100080 ] Stopped at panic+308: pushl $0 db> s [thread pid 943 tid 100080 ] Stopped at panic+310: pushl $3228776128 db> s [thread pid 943 tid 100080 ] Stopped at panic+315: call _mtx_lock_spin_flags db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags: pushl %ebp db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+1: movl %esp,%ebp db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+3: pushl %edi db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+4: pushl %esi db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+5: pushl %ebx db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+6: movl 8(%ebp),%esi db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+9: movl 20(%ebp),%edi db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+12: pushl %edi db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+13: pushl 16(%ebp) db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+16: movl 12(%ebp),%eax db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+19: orl $9,%eax db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+22: pushl %eax db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+23: pushl %esi db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+24: call witness_checkorder db> n After 22 instructions (0 loads, 0 stores), [thread pid 943 tid 100080 ] Stopped at witness_checkorder+1259: ret db> n KDB: stack backtrace: hangs here ... Thanks, Yong Rao Force10 Networks Inc. 408 571 3617 -----Original Message----- From: Yong Rao Sent: Wednesday, June 20, 2007 4:54 PM To: [email protected] Cc: Jagjit Choudhary Subject: kernel core dump hangs with the SMP kernel (FreeBSD 6.2) Hello, I am able to isolate the problem to kernel configurations which have SMP enabled when used with 2 cpus. With ONE cpu the core dump works ok. I built the kernel with GENERIC, and deliberately crash the kernel (for testing purpose). The core dump works fine. Only added the "options SMP" and crashed the kernel, then prior to any pages being dumped out, it hangs there. Has someone successfully core dumped on a system using SMP kernel with multiple CPUs? I tried on two different boxes (different motherboards, CPUs and hard disks). Both got failed. I tried to enable the DDB, but don't know what to look for when it goes into ddb. Appreciate any pointers. a) The CPU information is CPU: Dual Core AMD Opteron(tm) Processor 280 (2405.47-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x20f12 Stepping = 2 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE ,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,S SE,SSE2,HTT> Features2=0x1<SSE3> AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow+,3DNow> AMD Features2=0x3<LAHF,CMP> Cores per package: 2 b) We also tried on another mother board, which has 2 CPUs. The CPU information is below. CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2800.11-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf29 Stepping = 9 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x4400<CNTX-ID,<b14>> real memory = 2147418112 (2047 MB) avail memory = 2096300032 (1999 MB) ACPI APIC Table: <A M I OEMAPIC > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 6 c) The following are the prints when the dump hung. mem dump: start address = 0x4352, len=0x30 Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 01 fault virtual address = 0x4352 fault code = supervisor read, page not present instruction pointer = 0x20:0xc9e9fc92 stack pointer = 0x28:0xebdbdbdc frame pointer = 0x28:0xebdbdbf8 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 74231 (pnicdbg) trap number = 12 panic: page fault cpuid = 1 Uptime: 1d18h27m42s Dumping 4030 MB (2 chunks) chunk 0: 1MB (154 pages) ... ok chunk 1: 4031MB (1031776 pages) (stopped and hung here) Thanks, Yong _______________________________________________ [email protected] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-smp To unsubscribe, send any mail to "[email protected]"
alltrace.txt
(text/plain, 39 KB)
db> alltrace Tracing command disablemeta pid 894 tid 100054 td 0xc96eb000 sched_switch(3225858983,10,0,3382180240,3377804288,...) at sched_switch+331 _end(53,128,606517178,0,0,...) at 3920951848 _end(3377796632,3379230208,3379230476,3377796608,3379229696,...) at 3377796608 _end(3228796064,3228796064,3228796064,3228796064,3228796064,...) at 3379230208 null_method(0,0,0,0,0,...) at null_method Tracing command bash pid 884 tid 100058 td 0xc9991c00 sched_switch(3382254592,0,1) at sched_switch+331 mi_switch(1,0,0,3956742568,3226748917,...) at mi_switch+426 sleepq_switch(3377983504) at sleepq_switch+113 sleepq_wait_sig(3377983504) at sleepq_wait_sig+29 msleep(3377983504,0,345,3228302894,0) at msleep+510 ttysleep(3377983488,3377983504,345,3228302894,0,...) at ttysleep+33 ttread(3377983488,3956743356,0) at ttread+1167 ttyread(3377918464,3956743356,0,3228776192,0,...) at ttyread+47 giant_read(3377918464,3956743356,0) at giant_read+45 devfs_read_f(3379716600,3956743356,3387476352,0,3382254592) at devfs_read_f+98 dofileread(3382254592,0,3379716600,3956743356,4294967295,...) at dofileread+133 kern_readv(3382254592,0,3956743356,3217025183,1,...) at kern_readv+54 read(3382254592,3956743428) at read+69 syscall(59,59,59,0,3217025183,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (3, FreeBSD ELF32, read), eip = 135257039, esp = 3217025148, ebp = - Tracing command bash pid 878 tid 100071 td 0xc96ea000 sched_switch(3379470336,0,1) at sched_switch+331 mi_switch(1,0,0,3920829452,3226748917,...) at mi_switch+426 sleepq_switch(3382776344) at sleepq_switch+113 sleepq_wait_sig(3382776344) at sleepq_wait_sig+29 msleep(3382776344,3382776448,348,3228297560,0) at msleep+510 kern_wait(3379470336,4294967295,3920829572,6,0) at kern_wait+1699 wait4(3379470336,3920829700) at wait4+42 syscall(59,59,59,894,0,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (7, FreeBSD ELF32, wait4), eip = 135042415, esp = 3217025340, ebp =- Tracing command login pid 866 tid 100083 td 0xc9992780 sched_switch(3382257536,0,1) at sched_switch+331 mi_switch(1,0,0,3956829196,3226748917,...) at mi_switch+426 sleepq_switch(3385359504) at sleepq_switch+113 sleepq_wait_sig(3385359504) at sleepq_wait_sig+29 msleep(3385359504,3385359608,348,3228297560,0) at msleep+510 kern_wait(3382257536,884,3956829316,0,0) at kern_wait+1699 wait4(3382257536,3956829444) at wait4+42 syscall(59,59,59,582,1,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (7, FreeBSD ELF32, wait4), eip = 672093903, esp = 3217026396, ebp =- Tracing command getty pid 865 tid 100090 td 0xc9c8c000 sched_switch(3385376768,0,1) at sched_switch+331 mi_switch(1,0,0,3957242260,3226748917,...) at mi_switch+426 sleepq_switch(3377985552) at sleepq_switch+113 sleepq_wait_sig(3377985552) at sleepq_wait_sig+29 msleep(3377985552,0,345,3228302894,0) at msleep+510 ttysleep(3377985536,3377985552,345,3228302894,0,...) at ttysleep+33 ttread(3377985536,3957243068,0) at ttread+1167 ttyread(3378041344,3957243068,0,3957242900,3226488553,...) at ttyread+47 scread(3378041344,3957243068,0,3228776192,0,...) at scread+34 giant_read(3378041344,3957243068,0) at giant_read+45 devfs_read_f(3379723360,3957243068,3376727936,0,3385376768) at devfs_read_f+98 dofileread(3385376768,0,3379723360,3957243068,4294967295,...) at dofileread+133 kern_readv(3385376768,0,3957243068,3217026591,1,...) at kern_readv+54 read(3385376768,3957243140) at read+69 syscall(59,59,59,0,134538624,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (3, FreeBSD ELF32, read), eip = 672428879, esp = 3217026556, ebp = - Tracing command getty pid 864 tid 100068 td 0xc96ea480 sched_switch(3379471488,0,1) at sched_switch+331 mi_switch(1,0,0,3920865684,3226748917,...) at mi_switch+426 sleepq_switch(3377979408) at sleepq_switch+113 sleepq_wait_sig(3377979408) at sleepq_wait_sig+29 msleep(3377979408,0,345,3228302894,0) at msleep+510 ttysleep(3377979392,3377979408,345,3228302894,0,...) at ttysleep+33 ttread(3377979392,3920866492,0) at ttread+1167 ttyread(3378041088,3920866492,0,3920866324,3226488553,...) at ttyread+47 scread(3378041088,3920866492,0,3228776192,0,...) at scread+34 giant_read(3378041088,3920866492,0) at giant_read+45 devfs_read_f(3382342424,3920866492,3376727936,0,3379471488) at devfs_read_f+98 dofileread(3379471488,0,3382342424,3920866492,4294967295,...) at dofileread+133 kern_readv(3379471488,0,3920866492,3217026591,1,...) at kern_readv+54 read(3379471488,3920866564) at read+69 syscall(59,59,59,0,134538624,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (3, FreeBSD ELF32, read), eip = 672428879, esp = 3217026556, ebp = - Tracing command getty pid 863 tid 100042 td 0xc9566480 sched_switch(3377882240,0,1) at sched_switch+331 mi_switch(1,0,0,3920550292,3226748917,...) at mi_switch+426 sleepq_switch(3378064400) at sleepq_switch+113 sleepq_wait_sig(3378064400) at sleepq_wait_sig+29 msleep(3378064400,0,345,3228302894,0) at msleep+510 ttysleep(3378064384,3378064400,345,3228302894,0,...) at ttysleep+33 ttread(3378064384,3920551100,0) at ttread+1167 ttyread(3378040832,3920551100,0,3920550932,3226488553,...) at ttyread+47 scread(3378040832,3920551100,0,3228776192,0,...) at scread+34 giant_read(3378040832,3920551100,0) at giant_read+45 devfs_read_f(3382344728,3920551100,3376727936,0,3377882240) at devfs_read_f+98 dofileread(3377882240,0,3382344728,3920551100,4294967295,...) at dofileread+133 kern_readv(3377882240,0,3920551100,3217026591,1,...) at kern_readv+54 read(3377882240,3920551172) at read+69 syscall(59,59,59,0,134538624,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (3, FreeBSD ELF32, read), eip = 672428879, esp = 3217026556, ebp = - Tracing command getty pid 862 tid 100072 td 0xc9a12900 sched_switch(3382782208,0,1) at sched_switch+331 mi_switch(1,0,0,3957016980,3226748917,...) at mi_switch+426 sleepq_switch(3378063376) at sleepq_switch+113 sleepq_wait_sig(3378063376) at sleepq_wait_sig+29 msleep(3378063376,0,345,3228302894,0) at msleep+510 ttysleep(3378063360,3378063376,345,3228302894,0,...) at ttysleep+33 ttread(3378063360,3957017788,0) at ttread+1167 ttyread(3377924352,3957017788,0,3957017620,3226488553,...) at ttyread+47 scread(3377924352,3957017788,0,3228776192,0,...) at scread+34 giant_read(3377924352,3957017788,0) at giant_read+45 devfs_read_f(3382342640,3957017788,3376727936,0,3382782208) at devfs_read_f+98 dofileread(3382782208,0,3382342640,3957017788,4294967295,...) at dofileread+133 kern_readv(3382782208,0,3957017788,3217026591,1,...) at kern_readv+54 read(3382782208,3957017860) at read+69 syscall(59,59,59,0,134538624,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (3, FreeBSD ELF32, read), eip = 672428879, esp = 3217026556, ebp = - Tracing command getty pid 861 tid 100079 td 0xc9992d80 sched_switch(3382259072,0,1) at sched_switch+331 mi_switch(1,0,0,3956877716,3226748917,...) at mi_switch+426 sleepq_switch(3378059280) at sleepq_switch+113 sleepq_wait_sig(3378059280) at sleepq_wait_sig+29 msleep(3378059280,0,345,3228302894,0) at msleep+510 ttysleep(3378059264,3378059280,345,3228302894,0,...) at ttysleep+33 ttread(3378059264,3956878524,0) at ttread+1167 ttyread(3377924608,3956878524,0,3956878356,3226488553,...) at ttyread+47 scread(3377924608,3956878524,0,3228776192,0,...) at scread+34 giant_read(3377924608,3956878524,0) at giant_read+45 devfs_read_f(3379719696,3956878524,3376727936,0,3382259072) at devfs_read_f+98 dofileread(3382259072,0,3379719696,3956878524,4294967295,...) at dofileread+133 kern_readv(3382259072,0,3956878524,3217026591,1,...) at kern_readv+54 read(3382259072,3956878596) at read+69 syscall(59,59,59,0,134538624,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (3, FreeBSD ELF32, read), eip = 672428879, esp = 3217026556, ebp = - Tracing command getty pid 860 tid 100057 td 0xc9991d80 sched_switch(3382254976,0,1) at sched_switch+331 mi_switch(1,0,0,3956754836,3226748917,...) at mi_switch+426 sleepq_switch(3378058256) at sleepq_switch+113 sleepq_wait_sig(3378058256) at sleepq_wait_sig+29 msleep(3378058256,0,345,3228302894,0) at msleep+510 ttysleep(3378058240,3378058256,345,3228302894,0,...) at ttysleep+33 ttread(3378058240,3956755644,0) at ttread+1167 ttyread(3377924864,3956755644,0,3956755476,3226488553,...) at ttyread+47 scread(3377924864,3956755644,0,3228776192,0,...) at scread+34 giant_read(3377924864,3956755644,0) at giant_read+45 devfs_read_f(3379722928,3956755644,3376727936,0,3382254976) at devfs_read_f+98 dofileread(3382254976,0,3379722928,3956755644,4294967295,...) at dofileread+133 kern_readv(3382254976,0,3956755644,3217026591,1,...) at kern_readv+54 read(3382254976,3956755716) at read+69 syscall(59,59,59,0,134538624,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (3, FreeBSD ELF32, read), eip = 672428879, esp = 3217026556, ebp = - Tracing command getty pid 859 tid 100088 td 0xc9c8c300 sched_switch(3385377536,0,1) at sched_switch+331 mi_switch(1,0,0,3957266836,3226748917,...) at mi_switch+426 sleepq_switch(3378062352) at sleepq_switch+113 sleepq_wait_sig(3378062352) at sleepq_wait_sig+29 msleep(3378062352,0,345,3228302894,0) at msleep+510 ttysleep(3378062336,3378062352,345,3228302894,0,...) at ttysleep+33 ttread(3378062336,3957267644,0) at ttread+1167 ttyread(3377925120,3957267644,0,3957267476,3226488553,...) at ttyread+47 scread(3377925120,3957267644,0,3228776192,0,...) at scread+34 giant_read(3377925120,3957267644,0) at giant_read+45 devfs_read_f(3382342928,3957267644,3376727936,0,3385377536) at devfs_read_f+98 dofileread(3385377536,0,3382342928,3957267644,4294967295,...) at dofileread+133 kern_readv(3385377536,0,3957267644,3217026591,1,...) at kern_readv+54 read(3385377536,3957267716) at read+69 syscall(59,59,59,0,134538624,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (3, FreeBSD ELF32, read), eip = 672428879, esp = 3217026556, ebp = - Tracing command login pid 858 tid 100059 td 0xc9991a80 sched_switch(3382254208,0,1) at sched_switch+331 mi_switch(1,0,0,3956730892,3226748917,...) at mi_switch+426 sleepq_switch(3382249032) at sleepq_switch+113 sleepq_wait_sig(3382249032) at sleepq_wait_sig+29 msleep(3382249032,3382249136,348,3228297560,0) at msleep+510 kern_wait(3382254208,878,3956731012,0,0) at kern_wait+1699 wait4(3382254208,3956731140) at wait4+42 syscall(59,59,59,582,1,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (7, FreeBSD ELF32, wait4), eip = 672093903, esp = 3217026396, ebp =- Tracing command cron pid 815 tid 100080 td 0xc9992c00 sched_switch(3382258688,0,1) at sched_switch+331 mi_switch(1,0,0,3956866108,3226749082,...) at mi_switch+426 sleepq_switch(3228779692) at sleepq_switch+113 sleepq_timedwait_sig(3228779692) at sleepq_timedwait_sig+30 msleep(3228779692,0,348,3228286296,60001,...) at msleep+468 kern_nanosleep(3382258688,3956866252,3956866244) at kern_nanosleep+171 nanosleep(3382258688,3956866308) at nanosleep+79 syscall(59,59,59,134545548,60,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 672337699, esp = 3217026268,- Tracing command sendmail pid 808 tid 100065 td 0xc96ea900 sched_switch(3379472640,0,1) at sched_switch+331 mi_switch(1,0,0,3920903260,3226748917,...) at mi_switch+426 sleepq_switch(3382772836) at sleepq_switch+113 sleepq_wait_sig(3382772836) at sleepq_wait_sig+29 msleep(3382772836,3382772888,360,3228284168,0) at msleep+510 kern_sigsuspend(3379472640,0,0,0,0,...) at kern_sigsuspend+163 sigsuspend(3379472640,3920903428) at sigsuspend+51 syscall(59,59,59,0,3217018880,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 673683383, esp = 3217018860- Tracing command sendmail pid 804 tid 100041 td 0xc9566600 sched_switch(3377882624,0,1) at sched_switch+331 mi_switch(1,0,0,3920562976,3226749082,...) at mi_switch+426 sleepq_switch(3229095940) at sleepq_switch+113 sleepq_timedwait_sig(3229095940,3229095940,5001,3920563008,582,...) at sleepq_t0 cv_timedwait_sig(3229095940,3229095904,5001) at cv_timedwait_sig+290 kern_select(3377882624,4,3217016064,0,0,...) at kern_select+1169 select(3377882624,3920563460) at select+68 syscall(59,59,59,1,1,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (93, FreeBSD ELF32, select), eip = 674198223, esp = 3217015820, ebp- Tracing command sshd pid 798 tid 100082 td 0xc9992900 sched_switch(3382257920,0,1) at sched_switch+331 mi_switch(1,0,0,3956841260,3226748917,...) at mi_switch+426 sleepq_switch(3229095940) at sleepq_switch+113 sleepq_wait_sig(3229095940,582,3228776128,3228296630,771,...) at sleepq_wait_si9 cv_wait_sig(3229095940,3229095904) at cv_wait_sig+281 kern_select(3382257920,5,134723872,0,0,...) at kern_select+1191 select(3382257920,3956841732) at select+68 syscall(59,59,59,1,10,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (93, FreeBSD ELF32, select), eip = 674546383, esp = 3217023340, ebp- Tracing command usbd pid 761 tid 100050 td 0xc96eb600 sched_switch(3379475968,0,1) at sched_switch+331 mi_switch(1,0,0,3921001248,3226749082,...) at mi_switch+426 sleepq_switch(3229095940) at sleepq_switch+113 sleepq_timedwait_sig(3229095940,3229095940,30001,3921001280,582,...) at sleepq_0 cv_timedwait_sig(3229095940,3229095904,30001) at cv_timedwait_sig+290 kern_select(3379475968,6,3217026464,3217026336,0,...) at kern_select+1169 select(3379475968,3921001732) at select+68 syscall(59,59,59,3217026336,30,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (93, FreeBSD ELF32, select), eip = 672379599, esp = 3217026252, ebp- Tracing command smbd pid 735 tid 100078 td 0xc9a12000 sched_switch(3382779904,0,1) at sched_switch+331 mi_switch(1,0,0,3956943964,3226748917,...) at mi_switch+426 sleepq_switch(3385356876) at sleepq_switch+113 sleepq_wait_sig(3385356876) at sleepq_wait_sig+29 msleep(3385356876,3385356928,360,3228284168,0) at msleep+510 kern_sigsuspend(3382779904,1073746048,0,0,0,...) at kern_sigsuspend+163 sigsuspend(3382779904,3956944132) at sigsuspend+51 syscall(59,59,59,137408512,137408512,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 676951991, esp = 3217021452- Tracing command smbd pid 714 tid 100086 td 0xc9992300 sched_switch(3382256384,0,1) at sched_switch+331 mi_switch(1,0,0,3956792108,3226748917,...) at mi_switch+426 sleepq_switch(3229095940) at sleepq_switch+113 sleepq_wait_sig(3229095940,582,3228776128,3228296630,771,...) at sleepq_wait_si9 cv_wait_sig(3229095940,3229095904) at cv_wait_sig+281 kern_select(3382256384,22,3217021808,0,0,...) at kern_select+1191 select(3382256384,3956792580) at select+68 syscall(59,59,59,137408512,0,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (93, FreeBSD ELF32, select), eip = 677466831, esp = 3217021372, ebp- Tracing command nmbd pid 710 tid 100089 td 0xc9c8c180 sched_switch(3385377152,0,1) at sched_switch+331 mi_switch(1,0,0,3957254944,3226749082,...) at mi_switch+426 sleepq_switch(3229095940) at sleepq_switch+113 sleepq_timedwait_sig(3229095940,3229095940,10001,3957254976,582,...) at sleepq_0 cv_timedwait_sig(3229095940,3229095904,10001) at cv_timedwait_sig+290 kern_select(3385377152,13,3217025088,0,0,...) at kern_select+1169 select(3385377152,3957255428) at select+68 syscall(59,59,59,13,3217025088,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (93, FreeBSD ELF32, select), eip = 674271951, esp = 3217024796, ebp- Tracing command syslogd pid 631 tid 100044 td 0xc9566180 sched_switch(3377881472,0,1) at sched_switch+331 mi_switch(1,0,0,3920526124,3226748917,...) at mi_switch+426 sleepq_switch(3229095940) at sleepq_switch+113 sleepq_wait_sig(3229095940,582,3228776128,3228296630,771,...) at sleepq_wait_si9 cv_wait_sig(3229095940,3229095904) at cv_wait_sig+281 kern_select(3377881472,9,134570080,0,0,...) at kern_select+1191 select(3377881472,3920526596) at select+68 syscall(59,59,59,134570084,134570080,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (93, FreeBSD ELF32, select), eip = 672445135, esp = 3217021484, ebp- Tracing command devd pid 565 tid 100060 td 0xc9991900 sched_switch(3382253824,0,1) at sched_switch+331 mi_switch(1,0,0,3956718380,3226748917,...) at mi_switch+426 sleepq_switch(3229095940) at sleepq_switch+113 sleepq_wait_sig(3229095940,582,3228776128,3228296630,771,...) at sleepq_wait_si9 cv_wait_sig(3229095940,3229095904) at cv_wait_sig+281 kern_select(3382253824,5,3217025488,0,0,...) at kern_select+1191 select(3382253824,3956718852) at select+68 syscall(59,59,59,0,8,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (93, FreeBSD ELF32, select), eip = 134698999, esp = 3217025388, ebp- Tracing command adjkerntz pid 143 tid 100064 td 0xc96eaa80 sched_switch(3379473024,0,1) at sched_switch+331 mi_switch(1,0,0,3920915548,3226748917,...) at mi_switch+426 sleepq_switch(3382772300) at sleepq_switch+113 sleepq_wait_sig(3382772300) at sleepq_wait_sig+29 msleep(3382772300,3382772352,360,3228284168,0) at msleep+510 kern_sigsuspend(3379473024,0,0,0,0,...) at kern_sigsuspend+163 sigsuspend(3379473024,3920915716) at sigsuspend+51 syscall(59,59,59,0,25200,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 671856567, esp = 3217026460- Tracing command schedcpu pid 42 tid 100046 td 0xc951bd80 sched_switch(3377577344,0,1) at sched_switch+331 mi_switch(1,0,0,3919867052,3226749014,...) at mi_switch+426 sleepq_switch(3919867124,3228776128,0,3228292512,566,...) at sleepq_switch+113 sleepq_timedwait(3919867124) at sleepq_timedwait+34 msleep(3919867124,0,0,3228262727,1000) at msleep+489 schedcpu_thread(0,3919867192,3228776128,0,3228273492,...) at schedcpu_thread+38 fork_exit(3226695036,0,3919867192) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3919867244, ebp = 0 --- Tracing command softdepflush pid 41 tid 100047 td 0xc951bc00 sched_switch(3377576960,0,1) at sched_switch+331 mi_switch(1,0,0,3919854756,3226749014,...) at mi_switch+426 sleepq_switch(3229151220,3228776128,0,3228292512,566,...) at sleepq_switch+113 sleepq_timedwait(3229151220) at sleepq_timedwait+34 msleep(3229151220,3229151168,68,3228349621,1000) at msleep+489 softdep_flush(0,3919854904,3228776128,0,3228273492,...) at softdep_flush+499 fork_exit(3227615092,0,3919854904) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3919854956, ebp = 0 --- Tracing command vnlru pid 40 tid 100048 td 0xc951ba80 sched_switch(3377576576,0,1) at sched_switch+331 mi_switch(1,0,1,3919842460,3226749014,...) at mi_switch+426 sleepq_switch(3379465360,3228776128,0,3228292512,566,...) at sleepq_switch+113 sleepq_timedwait(3379465360) at sleepq_timedwait+34 msleep(3379465360,3229098336,592,3228312087,1000,3229098464) at msleep+489 vnlru_proc(0,3919842616,3228776128,0,3228273492,...) at vnlru_proc+223 fork_exit(3226951860,0,3919842616) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3919842668, ebp = 0 --- Tracing command syncer pid 39 tid 100049 td 0xc951b900 sched_switch(3377576192,0,1) at sched_switch+331 mi_switch(1,0,1,3919830160,3226748861,...) at mi_switch+426 sleepq_switch(3228779036,3228776128,0,3228292512,529,...) at sleepq_switch+113 sleepq_wait(3228779036,0,3226599612,3229098400,8,...) at sleepq_wait+33 msleep(3228779036,0,104,3228312226,0) at msleep+525 sched_sync(0,3919830328,3228776128,0,3228273492,...) at sched_sync+777 fork_exit(3226957540,0,3919830328) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3919830380, ebp = 0 --- Tracing command bufdaemon pid 38 tid 100027 td 0xc951b180 sched_switch(3377574272,0,1) at sched_switch+331 mi_switch(1,0,1,3919768756,3226749014,...) at mi_switch+426 sleepq_switch(3229097324,3228776128,0,3228292512,566,...) at sleepq_switch+113 sleepq_timedwait(3229097324) at sleepq_timedwait+34 msleep(3229097324,3229097344,68,3228307860,1000) at msleep+489 buf_daemon(0,3919768888,3228776128,0,3228273492,...) at buf_daemon+264 fork_exit(3226903868,0,3919768888) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3919768940, ebp = 0 --- Tracing command pagezero pid 37 tid 100028 td 0xc951b000 sched_switch(3377573888,0,1) at sched_switch+331 mi_switch(1,0,0,3919756452,3226749014,...) at mi_switch+426 sleepq_switch(3229155172,3228776128,0,3228292512,566,...) at sleepq_switch+113 sleepq_timedwait(3229155172) at sleepq_timedwait+34 msleep(3229155172,3229153856,512,3228365548,300000,3229153856,0,3228365522,157)9 vm_pagezero(0,3919756600,3228776128,0,3228273492,...) at vm_pagezero+108 fork_exit(3227806548,0,3919756600) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3919756652, ebp = 0 --- Tracing command idlepoll pid 36 tid 100029 td 0xc9454d80 sched_switch(3376762240,0,1) at sched_switch+331 mi_switch(1,0,0,3892346024,3226749014,...) at mi_switch+426 sleepq_switch(3228776544,3228776128,0,3228292512,566,...) at sleepq_switch+113 sleepq_timedwait(3228776544) at sleepq_timedwait+34 msleep(3228776544,0,0,3228281840,3000) at msleep+489 poll_idle(0,3892346168,3228776128,0,3228273492,...) at poll_idle+226 fork_exit(3226607484,0,3892346168) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892346220, ebp = 0 --- Tracing command vmdaemon pid 35 tid 100030 td 0xc9454c00 sched_switch(3376761856,0,1) at sched_switch+331 mi_switch(1,0,1,3892333716,3226748861,...) at mi_switch+426 sleepq_switch(3229153972,3228776128,0,3228292512,529,...) at sleepq_switch+113 sleepq_wait(3229153972,0,3226599551,3228776192,3228365165,...) at sleepq_wait+33 msleep(3229153972,0,104,3228307860,0) at msleep+525 vm_daemon(0,3892333880,3228776128,0,3228273492,...) at vm_daemon+54 fork_exit(3227802892,0,3892333880) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892333932, ebp = 0 --- Tracing command pagedaemon pid 34 tid 100031 td 0xc9454a80 sched_switch(3376761472,0,1) at sched_switch+331 mi_switch(1,0,0,3892321456,3226749014,...) at mi_switch+426 sleepq_switch(3229153904,3228776128,0,3228292512,566,...) at sleepq_switch+113 sleepq_timedwait(3229153904) at sleepq_timedwait+34 msleep(3229153904,3229153856,68,3228307860,5000) at msleep+489 vm_pageout(0,3892321592,3228776128,0,3228273492,...) at vm_pageout+608 fork_exit(3227802060,0,3892321592) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892321644, ebp = 0 --- Tracing command irq7: ppc0 pid 33 tid 100032 td 0xc9454900 fork_trampoline() at fork_trampoline Tracing command swi0: sio pid 32 tid 100033 td 0xc9454780 sched_switch(3376760704,0,1) at sched_switch+331 mi_switch(1,0) at mi_switch+426 ithread_loop(3377944496,3892297016,3228776128,0,3228273492,...) at ithread_loop2 fork_exit(3226555672,3377944496,3892297016) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892297068, ebp = 0 --- Tracing command fdc0 pid 31 tid 100034 td 0xc9454600 sched_switch(3376760320,0,1) at sched_switch+331 mi_switch(1,0,0,3892284500,3226749014,...) at mi_switch+426 sleepq_switch(3377805884,3228776128,0,3228292512,566,...) at sleepq_switch+113 sleepq_timedwait(3377805884) at sleepq_timedwait+34 msleep(3377805884,3377806064,76,3228262727,1000) at msleep+489 fdc_worker(3377805824,3377806064,0,3228371674,1166) at fdc_worker+458 fdc_thread(3377805824,3892284728,3228776128,0,3228273492,...) at fdc_thread+84 fork_exit(3227861752,3377805824,3892284728) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892284780, ebp = 0 --- Tracing command irq1: atkbd0 pid 30 tid 100035 td 0xc9454480 sched_switch(3376759936,0,1) at sched_switch+331 mi_switch(1,0) at mi_switch+426 ithread_loop(3377944864,3892272440,3228776128,0,3228273492,...) at ithread_loop2 fork_exit(3226555672,3377944864,3892272440) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892272492, ebp = 0 --- Tracing command irq24: bge1 pid 29 tid 100036 td 0xc9566d80 sched_switch(3377884544,0,1) at sched_switch+331 mi_switch(1,0) at mi_switch+426 ithread_loop(3377837568,3920624952,3228776128,0,3228273492,...) at ithread_loop2 fork_exit(3226555672,3377837568,3920624952) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3920625004, ebp = 0 --- Tracing command irq27: bge0 pid 28 tid 100037 td 0xc9566c00 sched_switch(3377884160,0,1) at sched_switch+331 mi_switch(1,0) at mi_switch+426 ithread_loop(3377838352,3920612664,3228776128,0,3228273492,...) at ithread_loop2 fork_exit(3226555672,3377838352,3920612664) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3920612716, ebp = 0 --- Tracing command irq15: ata1 pid 27 tid 100017 td 0xc9450900 fork_trampoline() at fork_trampoline Tracing command irq14: ata0 pid 26 tid 100018 td 0xc9450780 fork_trampoline() at fork_trampoline Tracing command irq16: atapci0 pid 25 tid 100019 td 0xc9450600 sched_switch(3376743936,0,1) at sched_switch+331 mi_switch(1,0) at mi_switch+426 ithread_loop(3377734416,3892161848,3228776128,0,3228273492,...) at ithread_loop2 fork_exit(3226555672,3377734416,3892161848) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892161900, ebp = 0 --- Tracing command usb1 pid 24 tid 100020 td 0xc9450480 sched_switch(3376743552,0,1) at sched_switch+331 mi_switch(1,0,1,3892149420,3226749014,...) at mi_switch+426 sleepq_switch(3377725968,3228776128,0,3228292512,566,...) at sleepq_switch+113 sleepq_timedwait(3377725968) at sleepq_timedwait+34 msleep(3377725968,0,92,3228259087,60000,3377742272) at msleep+489 usb_event_thread(3377742272,3892149560,3228776128,0,3228273492,...) at usb_even2 fork_exit(3226327968,3377742272,3892149560) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892149612, ebp = 0 --- Tracing command usbtask pid 23 tid 100021 td 0xc9450300 sched_switch(3376743168,0,1) at sched_switch+331 mi_switch(1,0,1,3892137140,3226748861,...) at mi_switch+426 sleepq_switch(3228766148,3228776128,0,3228292512,529,...) at sleepq_switch+113 sleepq_wait(3228766148,0,3226599551,3228776192,3228259073,...) at sleepq_wait+33 msleep(3228766148,0,92,3228259094,0,...) at msleep+525 usb_task_thread(0,3892137272,3228776128,0,3228273492,...) at usb_task_thread+71 fork_exit(3226328112,0,3892137272) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892137324, ebp = 0 --- Tracing command usb0 pid 22 tid 100022 td 0xc9450180 sched_switch(3376742784,0,1) at sched_switch+331 mi_switch(1,0,1,3892124844,3226749014,...) at mi_switch+426 sleepq_switch(3377709584,3228776128,0,3228292512,566,...) at sleepq_switch+113 sleepq_timedwait(3377709584) at sleepq_timedwait+34 msleep(3377709584,0,92,3228259087,60000,3377645696) at msleep+489 usb_event_thread(3377645696,3892124984,3228776128,0,3228273492,...) at usb_even2 fork_exit(3226327968,3377645696,3892124984) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892125036, ebp = 0 --- Tracing command irq19: ohci0 ohci1 pid 21 tid 100023 td 0xc951b780 fork_trampoline() at fork_trampoline Tracing command irq9: acpi0 pid 20 tid 100024 td 0xc951b600 fork_trampoline() at fork_trampoline Tracing command swi6: task queue pid 19 tid 100025 td 0xc951b480 sched_switch(3377575040,0,1) at sched_switch+331 mi_switch(1,0) at mi_switch+426 ithread_loop(3376511392,3919793464,3228776128,0,3228273492,...) at ithread_loop2 fork_exit(3226555672,3376511392,3919793464) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3919793516, ebp = 0 --- Tracing command kqueue taskq pid 9 tid 100026 td 0xc951b300 sched_switch(3377574656,0,1) at sched_switch+331 mi_switch(1,0,0,3919781032,3226748861,...) at mi_switch+426 sleepq_switch(3377065088,3228776128,0,3228292512,529,...) at sleepq_switch+113 sleepq_wait(3377065088,0,3228293171,73,8,...) at sleepq_wait+33 msleep(3377065088,3377065116,0,3228262727,0) at msleep+525 taskqueue_thread_loop(3228769632,3919781176,3228776128,0,3228273492,...) at tas0 fork_exit(3226753588,3228769632,3919781176) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3919781228, ebp = 0 --- Tracing command acpi_task_2 pid 8 tid 100008 td 0xc944f300 sched_switch(3376739072,0,1) at sched_switch+331 mi_switch(1,0,0,3892014248,3226748861,...) at mi_switch+426 sleepq_switch(3377065344,3228776128,0,3228292512,529,...) at sleepq_switch+113 sleepq_wait(3377065344,0,3228293171,73,8,...) at sleepq_wait+33 msleep(3377065344,3377065372,0,3228262727,0) at msleep+525 taskqueue_thread_loop(3230234752,3892014392,3228776128,0,3228273492,...) at tas0 fork_exit(3226753588,3230234752,3892014392) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892014444, ebp = 0 --- Tracing command acpi_task_1 pid 7 tid 100009 td 0xc944f180 sched_switch(3376738688,0,1) at sched_switch+331 mi_switch(1,0,0,3892001960,3226748861,...) at mi_switch+426 sleepq_switch(3377065344,3228776128,0,3228292512,529,...) at sleepq_switch+113 sleepq_wait(3377065344,0,3228293171,73,8,...) at sleepq_wait+33 msleep(3377065344,3377065372,0,3228262727,0) at msleep+525 taskqueue_thread_loop(3230234752,3892002104,3228776128,0,3228273492,...) at tas0 fork_exit(3226753588,3230234752,3892002104) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892002156, ebp = 0 --- Tracing command acpi_task_0 pid 6 tid 100010 td 0xc944f000 sched_switch(3376738304,0,1) at sched_switch+331 mi_switch(1,0,0,3891989672,3226748861,...) at mi_switch+426 sleepq_switch(3377065344,3228776128,0,3228292512,529,...) at sleepq_switch+113 sleepq_wait(3377065344,0,3228293171,73,3377600608,...) at sleepq_wait+33 msleep(3377065344,3377065372,0,3228262727,0) at msleep+525 taskqueue_thread_loop(3230234752,3891989816,3228776128,0,3228273492,...) at tas0 fork_exit(3226753588,3230234752,3891989816) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3891989868, ebp = 0 --- Tracing command swi2: cambio pid 18 tid 100011 td 0xc9454300 sched_switch(3376759552,0,1) at sched_switch+331 mi_switch(1,0) at mi_switch+426 ithread_loop(3376511456,3892260152,3228776128,0,3228273492,...) at ithread_loop2 fork_exit(3226555672,3376511456,3892260152) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892260204, ebp = 0 --- Tracing command swi5: + pid 17 tid 100012 td 0xc9454180 fork_trampoline() at fork_trampoline Tracing command thread taskq pid 5 tid 100013 td 0xc9454000 sched_switch(3376758784,0,1) at sched_switch+331 mi_switch(1,0,0,3892235432,3226748861,...) at mi_switch+426 sleepq_switch(3377103104,3228776128,0,3228292512,529,...) at sleepq_switch+113 sleepq_wait(3377103104,0,3228293171,73,8,...) at sleepq_wait+33 msleep(3377103104,3377103132,0,3228262727,0) at msleep+525 taskqueue_thread_loop(3228801864,3892235576,3228776128,0,3228273492,...) at tas0 fork_exit(3226753588,3228801864,3892235576) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892235628, ebp = 0 --- Tracing command swi6: Giant taskq pid 16 tid 100014 td 0xc9450d80 kdb_enter(3228375620) at kdb_enter+43 scgetc(3229266944,2) at scgetc+1348 sckbdevent(3377072384,0,3229266944) at sckbdevent+456 kbdmux_intr(3377072384,0,3377105560,3892223160,3226752756,...) at kbdmux_intr+28 kbdmux_kbd_intr(3377072384,1) at kbdmux_kbd_intr+26 taskqueue_run(3377103360,3892223196,3226555506,0,3376511600,...) at taskqueue_r0 taskqueue_swi_giant_run(0) at taskqueue_swi_giant_run+14 ithread_execute_handlers(3377065984,3377103232) at ithread_execute_handlers+206 ithread_loop(3376511600,3892223288,3228776128,0,3228273492,...) at ithread_loop9 fork_exit(3226555672,3376511600,3892223288) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892223340, ebp = 0 --- Tracing command yarrow pid 15 tid 100015 td 0xc9450c00 sched_switch(3376745472,0,1) at sched_switch+331 mi_switch(1,0,0,3892210848,3226749014,...) at mi_switch+426 sleepq_switch(3228757952,3228776128,0,3228292512,566,...) at sleepq_switch+113 sleepq_timedwait(3228757952) at sleepq_timedwait+34 msleep(3228757952,0,0,3228262727,100) at msleep+489 random_kthread(0,3892211000,3228776128,0,3228273492,...) at random_kthread+369 fork_exit(3226233400,0,3892211000) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892211052, ebp = 0 --- Tracing command g_down pid 4 tid 100016 td 0xc9450a80 sched_switch(3376745088,0,1) at sched_switch+331 mi_switch(1,0,0,3892198556,3226749014,...) at mi_switch+426 sleepq_switch(3228767432,3228776128,0,3228292512,566,...) at sleepq_switch+113 sleepq_timedwait(3228767432) at sleepq_timedwait+34 msleep(3228767432,3228767144,588,3228262727,100) at msleep+489 g_io_schedule_down(3376745088) at g_io_schedule_down+86 g_down_procbody(0,3892198712,3228776128,0,3228273492,...) at g_down_procbody+70 fork_exit(3226426420,0,3892198712) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892198764, ebp = 0 --- Tracing command g_up pid 3 tid 100000 td 0xc9450000 sched_switch(3376742400,0,1) at sched_switch+331 mi_switch(1,0,0,3892112544,3226749014,...) at mi_switch+426 sleepq_switch(3228767428,3228776128,0,3228292512,566,...) at sleepq_switch+113 sleepq_timedwait(3228767428) at sleepq_timedwait+34 msleep(3228767428,3228767208,588,3228262727,100) at msleep+489 g_io_schedule_up(3376742400) at g_io_schedule_up+204 g_up_procbody(0,3892112696,3228776128,0,3228273492,...) at g_up_procbody+70 fork_exit(3226426344,0,3892112696) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892112748, ebp = 0 --- Tracing command g_event pid 2 tid 100001 td 0xc944fd80 sched_switch(3376741760,0,1) at sched_switch+331 mi_switch(1,0,0,3892100260,3226749014,...) at mi_switch+426 sleepq_switch(3228767420,3228776128,0,3228292512,566,...) at sleepq_switch+113 sleepq_timedwait(3228767420) at sleepq_timedwait+34 msleep(3228767420,0,76,3228262727,100) at msleep+489 g_event_procbody(0,3892100408,3228776128,0,3228273492,...) at g_event_procbody+0 fork_exit(3226426496,0,3892100408) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892100460, ebp = 0 --- Tracing command swi3: vm pid 14 tid 100002 td 0xc944fc00 fork_trampoline() at fork_trampoline Tracing command swi4: clock sio pid 13 tid 100003 td 0xc944fa80 sched_switch(3376740992,0,1) at sched_switch+331 mi_switch(1,0) at mi_switch+426 ithread_loop(3376511136,3892075832,3228776128,0,3228273492,...) at ithread_loop2 fork_exit(3226555672,3376511136,3892075832) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892075884, ebp = 0 --- Tracing command swi1: net pid 12 tid 100004 td 0xc944f900 sched_switch(3376740608,0,1) at sched_switch+331 mi_switch(1,0) at mi_switch+426 ithread_loop(3376511152,3892063544,3228776128,0,3228273492,...) at ithread_loop2 fork_exit(3226555672,3376511152,3892063544) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892063596, ebp = 0 --- Tracing command idle: cpu0 pid 11 tid 100005 td 0xc944f780 sched_switch(3376740224,3376743936,6) at sched_switch+331 mi_switch(6,3376743936,3376744280,3376743936,3892050952,...) at mi_switch+426 maybe_preempt(3376743936) at maybe_preempt+196 sched_add(3376743936,4,3376743936,3377734416,3377063168,...) at sched_add+240 setrunqueue(3376743936,4) at setrunqueue+99 intr_event_schedule_thread(3377063168) at intr_event_schedule_thread+163 intr_execute_handlers(3376699000,3892051072,16,3892051140,3227972099,...) at in0 lapic_handle_intr(50) at lapic_handle_intr+46 Xapic_isr1() at Xapic_isr1+51 --- interrupt, eip = 3230175589, esp = 3892051140, ebp = 3892051140 --- acpi_cpu_c1(3228776128,0,3892051172,1,4294967294,...) at acpi_cpu_c1+5 acpi_cpu_idle(3892051196,3226552629,0,3376736888,3892051236,...) at acpi_cpu_id8 cpu_idle(0,3376736888,3892051236,3226552149,0,...) at cpu_idle+40 idle_proc(0,3892051256,3228776128,0,3228273492,...) at idle_proc+77 fork_exit(3226552552,0,3892051256) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892051308, ebp = 0 --- Tracing command idle: cpu1 pid 10 tid 100006 td 0xc944f600 sched_switch(3376739840,0,1) at sched_switch+331 mi_switch(1,0,3228776128,0,3228273740,...) at mi_switch+426 idle_proc(0,3892038968,3228776128,0,3228273492,...) at idle_proc+123 fork_exit(3226552552,0,3892038968) at fork_exit+97 fork_trampoline() at fork_trampoline+8 --- trap 1, eip = 0, esp = 3892039020, ebp = 0 --- Tracing command init pid 1 tid 100007 td 0xc944f480 sched_switch(3376739456,0,1) at sched_switch+331 mi_switch(1,0,0,3892026380,3226748917,...) at mi_switch+426 sleepq_switch(3376754688) at sleepq_switch+113 sleepq_wait_sig(3376754688) at sleepq_wait_sig+29 msleep(3376754688,3376754792,348,3228297560,0) at msleep+510 kern_wait(3376739456,4294967295,3892026500,0,0) at kern_wait+1699 wait4(3376739456,3892026628) at wait4+42 syscall(59,59,59,0,0,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (7, FreeBSD ELF32, wait4), eip = 134557511, esp = 3217026428, ebp =- Tracing command swapper pid 0 tid 0 td 0xc07317e0 sched_switch(3228768224,0,1) at sched_switch+331 mi_switch(1,0) at mi_switch+426 scheduler(0,12708864,12705792,0,3225640485,...) at scheduler+385 mi_startup() at mi_startup+150 begin() at begin+44 db>