Re: kernel BUG with 2.4.27-om20041102-tab on SMP when doing heavy I/O
Peter Cordes <[email protected]>
| Newsgroups | gmane.linux.cluster.openmosix.devel |
|---|---|
| Message-ID | <[email protected]> |
On Tue, Sep 27, 2005 at 03:13:32PM -0300, Peter Cordes wrote: > I have been running 2.4.27-om20041102-tab (peter-OPrYhD/TZC/[email protected]) > (gcc version 3.3.5 (Debian 1:3.3.5-13)) #1 SMP Wed Sep 7 on my dual Opteron > cluster. The master node has 4GB of RAM, and the other 7 nodes have 2GB. > All with gigabit ethernet (tg3). The master has an Intel e100 built in to the > mobo too, which is used to connect to the outside world. It seems that channel bonding didn't change things after all (or at least didn't completely avoid the problem). I had two more BUG messages yesterday, which come from sshd. From my other logs, I see that there were ssh logins at the times of those two BUGs. (One of them was my login, and I didn't notice anything strange, so it didn't actually break sshd!) I happened to have sshd's DEBUG logging turned on, and I don't see anything odd in it, even. Oct 3 19:06:49 nyneve kernel: kernel BUG at page_alloc.c:274! Oct 3 19:06:49 nyneve kernel: invalid operand: 0000 Oct 3 19:06:49 nyneve kernel: CPU: 0 Oct 3 19:06:49 nyneve kernel: EIP: 0010:[rmqueue+534/576] Not tainted Oct 3 19:06:49 nyneve kernel: EFLAGS: 00010202 Oct 3 19:06:49 nyneve kernel: eax: 0200084c ebx: c33643d8 ecx: 00038000 edx: 000a1ca3 Oct 3 19:06:49 nyneve kernel: esi: c03c9758 edi: c03c9794 ebp: c03c9758 esp: ebe15e40 Oct 3 19:06:49 nyneve kernel: ds: 0018 es: 0018 ss: 0018 Oct 3 19:06:49 nyneve kernel: Process sshd (pid: 13039, stackpage=ebe15000) Oct 3 19:06:49 nyneve kernel: Stack: 00038000 c471bfa8 00069ca3 00000282 00000000 c03c9758 c03c9758 c03c9858 Oct 3 19:06:49 nyneve kernel: 00000002 00000001 c01484b5 00000119 f79961d4 00000128 f7793dc0 f7793e84 Oct 3 19:06:49 nyneve kernel: f6b8bb40 c03c9758 c03c9854 00000000 000001d2 c013b2f0 db19c440 00000000 Oct 3 19:06:49 nyneve kernel: Call Trace: [__alloc_pages+101/688] [do_no_page+128/592] [do_wp_page+117/736] [handle_mm_fault+271/304] [do_page_fault+872/1289] [sock_sendmsg+104/160] [sock_write+158/192] [local_pre_usermode_actions+21/192] [do_page_fault+0/1289] [error_code+52/60] Oct 3 19:06:49 nyneve kernel: Code: 0f 0b 12 01 34 e2 37 c0 8b 43 18 a8 80 74 08 0f 0b 14 01 34 >>ebx; c33643d8 <_end+2ee5bec/386e1894> >>esi; c03c9758 <contig_page_data+1b8/3b8> >>edi; c03c9794 <contig_page_data+1f4/3b8> >>ebp; c03c9758 <contig_page_data+1b8/3b8> >>esp; ebe15e40 <_end+2b997654/386e1894> Code; 00000000 Before first symbol 00000000 <_EIP>: Code; 00000000 Before first symbol 0: 0f 0b ud2a Code; 00000002 Before first symbol 2: 12 01 adc (%ecx),%al Code; 00000004 Before first symbol 4: 34 e2 xor $0xe2,%al Code; 00000006 Before first symbol 6: 37 aaa Code; 00000007 Before first symbol 7: c0 8b 43 18 a8 80 74 rorb $0x74,0x80a81843(%ebx) Code; 0000000e Before first symbol e: 08 0f or %cl,(%edi) Code; 00000010 Before first symbol 10: 0b 14 01 or (%ecx,%eax,1),%edx Code; 00000013 Before first symbol 13: 34 00 xor $0x0,%al Oct 3 21:43:51 nyneve kernel: kernel BUG at page_alloc.c:274! Oct 3 21:43:51 nyneve kernel: invalid operand: 0000 Oct 3 21:43:51 nyneve kernel: CPU: 0 Oct 3 21:43:51 nyneve kernel: EIP: 0010:[rmqueue+534/576] Not tainted Oct 3 21:43:51 nyneve kernel: EFLAGS: 00010202 Oct 3 21:43:51 nyneve kernel: eax: 02000850 ebx: c37a5a48 ecx: 00038000 edx: 000b53e5 Oct 3 21:43:51 nyneve kernel: esi: c03c9758 edi: c03c9794 ebp: c03c9758 esp: f4cc1e40 Oct 3 21:43:51 nyneve kernel: ds: 0018 es: 0018 ss: 0018 Oct 3 21:43:51 nyneve kernel: Process sshd (pid: 14942, stackpage=f4cc1000) Oct 3 21:43:51 nyneve kernel: Stack: 00038000 c471bfa8 0007d3e5 00000282 00000000 c03c9758 c03c9758 c03c9858 Oct 3 21:43:51 nyneve kernel: 00000002 00000001 c01484b5 00000119 f79961d4 00000128 f7793dc0 f7793e84 Oct 3 21:43:51 nyneve kernel: f6b8bb40 c03c9758 c03c9854 00000000 000001d2 c013b2f0 f56cc840 00000000 Oct 3 21:43:51 nyneve kernel: Call Trace: [__alloc_pages+101/688] [do_no_page+128/592] [do_wp_page+117/736] [handle_mm_fault+271/304] [do_page_fault+872/1289] [sock_sendmsg+104/160] [sock_write+158/192] [local_pre_usermode_actions+21/192] [do_page_fault+0/1289] [error_code+52/60] Oct 3 21:43:51 nyneve kernel: Code: 0f 0b 12 01 34 e2 37 c0 8b 43 18 a8 80 74 08 0f 0b 14 01 34 >>ebx; c37a5a48 <_end+332725c/386e1894> >>esi; c03c9758 <contig_page_data+1b8/3b8> >>edi; c03c9794 <contig_page_data+1f4/3b8> >>ebp; c03c9758 <contig_page_data+1b8/3b8> >>esp; f4cc1e40 <_end+34843654/386e1894> Code; 00000000 Before first symbol 00000000 <_EIP>: Code; 00000000 Before first symbol 0: 0f 0b ud2a Code; 00000002 Before first symbol 2: 12 01 adc (%ecx),%al Code; 00000004 Before first symbol 4: 34 e2 xor $0xe2,%al Code; 00000006 Before first symbol 6: 37 aaa Code; 00000007 Before first symbol 7: c0 8b 43 18 a8 80 74 rorb $0x74,0x80a81843(%ebx) Code; 0000000e Before first symbol e: 08 0f or %cl,(%edi) Code; 00000010 Before first symbol 10: 0b 14 01 or (%ecx,%eax,1),%edx Code; 00000013 Before first symbol 13: 34 00 xor $0x0,%al My guess is that a some data structure in kernel space is corrupt. If the memtester results are any indication (see my other message in openmosix-general), something is corrupting memory, be it hardware or software. -- #define X(x,y) x##y Peter Cordes ; e-mail: X(peter@cor , des.ca) "The gods confound the man who first found out how to distinguish the hours! Confound him, too, who in this place set up a sundial, to cut and hack my day so wretchedly into small pieces!" -- Plautus, 200 BC ------------------------------------------------------- This SF.Net email is sponsored by: Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl