kernel BUG with 2.4.27-om20041102-tab on SMP when doing heavy I/O

Peter Cordes <[email protected]>
Newsgroups gmane.linux.cluster.openmosix.devel
Message-ID <[email protected]> 
 I have been running  2.4.27-om20041102-tab (peter-OPrYhD/TZC/[email protected])
(gcc version 3.3.5 (Debian 1:3.3.5-13)) #1 SMP Wed Sep 7  on my dual Opteron
cluster.  The master node has 4GB of RAM, and the other 7 nodes have 2GB.
All with gigabit ethernet.  The master has an Intel e100 built in to the
mobo too, which is used to connect to the outside world.

 My kernel config is
CONFIG_MOSIX=y
# CONFIG_MOSIX_TOPOLOGY is not set
CONFIG_MOSIX_SECUREPORTS=y
CONFIG_MOSIX_DISCLOSURE=3
# CONFIG_MOSIX_FS is not set
CONFIG_MOSIX_PIPE_EXCEPTIONS=y
# CONFIG_MOSIX_NO_OOM is not set
CONFIG_MOSIX_EXT_LOCALTIME=y

 My filesystems are JFS on RAID0 and RAID1 partitions of two SATA drives,
connected to the SATA_SIL ports on the Tyan S2882 motherboard.

 While doing a big file mirroring update with rsync (of BLAST dna sequence
databases...), the kernel logged some errors.  I don't think this has
happened consistently during the bi-monthly updates, though.  Anyway:
first one:
Sep 22 00:12:34 nyneve kernel: kernel BUG at page_alloc.c:144!
Sep 22 00:12:34 nyneve kernel: invalid operand: 0000
Sep 22 00:12:34 nyneve kernel: CPU:    1
Sep 22 00:12:34 nyneve kernel: EIP:    0010:[__free_pages_ok+80/768]    Not tainted
Sep 22 00:12:34 nyneve kernel: EFLAGS: 00010282
Sep 22 00:12:34 nyneve kernel: eax: c03c9834   ebx: c2da16a0   ecx: 00000000   edx: ebf04604
Sep 22 00:12:34 nyneve kernel: esi: c2da16a0   edi: 00000000   ebp: 0001ee61   esp: c43c9f08
Sep 22 00:12:34 nyneve kernel: ds: 0018   es: 0018   ss: 0018
Sep 22 00:12:34 nyneve kernel: Process kswapd (pid: 5, stackpage=c43c9000)
Sep 22 00:12:34 nyneve kernel: Stack: 00000282 00000003 c03c9ec4 ebf06540 ebf06540 ebf06540 c2da16a0 c0155369 
Sep 22 00:12:34 nyneve kernel:        ebf06540 00000000 c2da16a0 c03c9758 0001ee61 c014736b c2da16a0 000001d0 
Sep 22 00:12:34 nyneve kernel:        c43c8000 00000c80 000001d0 0000001f 00000020 000001d0 c03c9758 c03c9758 
Sep 22 00:12:34 nyneve kernel: Call Trace:    [try_to_free_buffers+185/336] [shrink_cache+843/1008] [shrink_caches+74/96] [try_to_free_pages_zone+83/224] [kswapd_balance_pgdat+94/160]
Sep 22 00:12:34 nyneve kernel: Code: 0f 0b 90 00 34 e2 37 c0 89 d8 2b 05 d0 63 44 c0 c1 f8 03 69 

>>eax; c03c9834 <contig_page_data+294/3b8>
>>ebx; c2da16a0 <_end+2922eb4/386e1894>
>>edx; ebf04604 <_end+2ba85e18/386e1894>
>>esi; c2da16a0 <_end+2922eb4/386e1894>
>>esp; c43c9f08 <_end+3f4b71c/386e1894>

Code;  00000000 Before first symbol
00000000 <_EIP>:
Code;  00000000 Before first symbol
   0:   0f 0b                     ud2a   
Code;  00000002 Before first symbol
   2:   90                        nop    
Code;  00000003 Before first symbol
   3:   00 34 e2                  add    %dh,(%edx,8)
Code;  00000006 Before first symbol
   6:   37                        aaa    
Code;  00000007 Before first symbol
   7:   c0 89 d8 2b 05 d0 63      rorb   $0x63,0xd0052bd8(%ecx)
Code;  0000000e Before first symbol
   e:   44                        inc    %esp
Code;  0000000f Before first symbol
   f:   c0 c1 f8                  rol    $0xf8,%cl
Code;  00000012 Before first symbol
  12:   03 69 00                  add    0x0(%ecx),%ebp


############### next one from rsync: ###############################
Sep 22 00:12:34 nyneve kernel:  kernel BUG at page_alloc.c:144!
Sep 22 00:12:34 nyneve kernel: invalid operand: 0000
Sep 22 00:12:34 nyneve kernel: CPU:    1
Sep 22 00:12:34 nyneve kernel: EIP:    0010:[__free_pages_ok+80/768]    Not tainted
Sep 22 00:12:34 nyneve kernel: EFLAGS: 00010282
Sep 22 00:12:34 nyneve kernel: eax: c03c9834   ebx: c2da4a40   ecx: 00000000   edx: ebedabc4
Sep 22 00:12:34 nyneve kernel: esi: c2da4a40   edi: 00000000   ebp: 0001ee6f   esp: d324de18
Sep 22 00:12:34 nyneve kernel: ds: 0018   es: 0018   ss: 0018
Sep 22 00:12:34 nyneve kernel: Process rsync (pid: 21475, stackpage=d324d000)
Sep 22 00:12:34 nyneve kernel: Stack: 00000282 00000003 c03c9ec4 ebea56c0 ebea56c0 ebea56c0 c2da4a40 c0155369 
Sep 22 00:12:34 nyneve kernel:        ebea56c0 00000000 c2da4a40 c03c9758 0001ee6f c014736b c2da4a40 000001d2 
Sep 22 00:12:34 nyneve kernel:        d324c000 00000c1c 000001d2 00000001 0000001f 000001d2 c03c9758 c03c9758 
Sep 22 00:12:34 nyneve kernel: Call Trace:    [try_to_free_buffers+185/336] [shrink_cache+843/1008] [shrink_caches+74/96] [try_to_free_pages_zone+83/224] [balance_classzone+77/480]
Sep 22 00:12:34 nyneve kernel: Code: 0f 0b 90 00 34 e2 37 c0 89 d8 2b 05 d0 63 44 c0 c1 f8 03 69 

>>eax; c03c9834 <contig_page_data+294/3b8>
>>ebx; c2da4a40 <_end+2926254/386e1894>
>>edx; ebedabc4 <_end+2ba5c3d8/386e1894>
>>esi; c2da4a40 <_end+2926254/386e1894>
>>esp; d324de18 <_end+12dcf62c/386e1894>

Code;  00000000 Before first symbol
00000000 <_EIP>:
Code;  00000000 Before first symbol
   0:   0f 0b                     ud2a   
Code;  00000002 Before first symbol
   2:   90                        nop    
Code;  00000003 Before first symbol
   3:   00 34 e2                  add    %dh,(%edx,8)
Code;  00000006 Before first symbol
   6:   37                        aaa    
Code;  00000007 Before first symbol
   7:   c0 89 d8 2b 05 d0 63      rorb   $0x63,0xd0052bd8(%ecx)
Code;  0000000e Before first symbol
   e:   44                        inc    %esp
Code;  0000000f Before first symbol
   f:   c0 c1 f8                  rol    $0xf8,%cl
Code;  00000012 Before first symbol
  12:   03 69 00                  add    0x0(%ecx),%ebp



 A few days later I ran my update script manually, and hit problems again.
formatdb (which reads giant flat text files and writes indexed databases
that BLAST can search against) was segfaulting...

This one from rsync again:
Sep 26 15:05:21 nyneve kernel:  kernel BUG at page_alloc.c:144!
Sep 26 15:05:21 nyneve kernel: invalid operand: 0000
Sep 26 15:05:21 nyneve kernel: CPU:    0
Sep 26 15:05:21 nyneve kernel: EIP:    0010:[__free_pages_ok+80/768]    Not tainted
Sep 26 15:05:21 nyneve kernel: EFLAGS: 00010282
Sep 26 15:05:21 nyneve kernel: eax: c03c9834   ebx: c2f77970   ecx: 00000000   edx: ea9f0504
Sep 26 15:05:21 nyneve kernel: esi: c2f77970   edi: 00000000   ebp: 0001f0de   esp: ec4a1e18
Sep 26 15:05:21 nyneve kernel: ds: 0018   es: 0018   ss: 0018
Sep 26 15:05:21 nyneve kernel: Process rsync (pid: 12692, stackpage=ec4a1000)
Sep 26 15:05:21 nyneve kernel: Stack: 00000282 00000003 c03c9ec4 ea9f9cc0 ea9f9cc0 ea9f9cc0 c2f77970 c0155369
Sep 26 15:05:21 nyneve kernel:        ea9f9cc0 00000000 c2f77970 c03c9758 0001f0de c014736b c2f77970 000001d2
Sep 26 15:05:21 nyneve kernel:        ec4a0000 00000c80 000001d2 0000000d 00000020 000001d2 c03c9758 c03c9758
Sep 26 15:05:21 nyneve kernel: Call Trace:    [try_to_free_buffers+185/336] [shrink_cache+843/1008] [shrink_caches+74/96] [try_to_free_pages_zone+83/224] [balance_classzone+77/480]
Sep 26 15:05:21 nyneve kernel: Code: 0f 0b 90 00 34 e2 37 c0 89 d8 2b 05 d0 63 44 c0 c1 f8 03 69

>>eax; c03c9834 <contig_page_data+294/3b8>
>>ebx; c2f77970 <_end+2af9184/386e1894>
>>edx; ea9f0504 <_end+2a571d18/386e1894>
>>esi; c2f77970 <_end+2af9184/386e1894>
>>esp; ec4a1e18 <_end+2c02362c/386e1894>

Code;  00000000 Before first symbol
00000000 <_EIP>:
Code;  00000000 Before first symbol
   0:   0f 0b                     ud2a   
Code;  00000002 Before first symbol
   2:   90                        nop    
Code;  00000003 Before first symbol
   3:   00 34 e2                  add    %dh,(%edx,8)
Code;  00000006 Before first symbol
   6:   37                        aaa    
Code;  00000007 Before first symbol
   7:   c0 89 d8 2b 05 d0 63      rorb   $0x63,0xd0052bd8(%ecx)
Code;  0000000e Before first symbol
   e:   44                        inc    %esp
Code;  0000000f Before first symbol
   f:   c0 c1 f8                  rol    $0xf8,%cl
Code;  00000012 Before first symbol
  12:   03 69 00                  add    0x0(%ecx),%ebp


################ and one from formatdb ###########
Sep 26 15:05:22 nyneve kernel:  kernel BUG at page_alloc.c:144!
Sep 26 15:05:22 nyneve kernel: invalid operand: 0000
Sep 26 15:05:22 nyneve kernel: CPU:    1
Sep 26 15:05:22 nyneve kernel: EIP:    0010:[__free_pages_ok+80/768]    Not tainted
Sep 26 15:05:22 nyneve kernel: EFLAGS: 00010282
Sep 26 15:05:22 nyneve kernel: eax: c03c9834   ebx: c2f80650   ecx: 00000000   edx: ea916dc4
Sep 26 15:05:22 nyneve kernel: esi: c2f80650   edi: 00000000   ebp: 0001f0db   esp: d8f39e18
Sep 26 15:05:22 nyneve kernel: ds: 0018   es: 0018   ss: 0018
Sep 26 15:05:22 nyneve kernel: Process formatdb (pid: 12688, stackpage=d8f39000)
Sep 26 15:05:22 nyneve kernel: Stack: 00000282 00000003 c03c9ec4 ea917640 ea917640 ea917640 c2f80650 c0155369 
Sep 26 15:05:22 nyneve kernel:        ea917640 00000000 c2f80650 c03c9758 0001f0db c014736b c2f80650 000001d2 
Sep 26 15:05:22 nyneve kernel:        d8f38000 00000c1c 000001d2 00000002 0000001f 000001d2 c03c9758 c03c9758 
Sep 26 15:05:22 nyneve kernel: Call Trace:    [try_to_free_buffers+185/336] [shrink_cache+843/1008] [shrink_caches+74/96] [try_to_free_pages_zone+83/224] [balance_classzone+77/480]
Sep 26 15:05:22 nyneve kernel: Code: 0f 0b 90 00 34 e2 37 c0 89 d8 2b 05 d0 63 44 c0 c1 f8 03 69 

>>eax; c03c9834 <contig_page_data+294/3b8>
>>ebx; c2f80650 <_end+2b01e64/386e1894>
>>edx; ea916dc4 <_end+2a4985d8/386e1894>
>>esi; c2f80650 <_end+2b01e64/386e1894>
>>esp; d8f39e18 <_end+18abb62c/386e1894>

Code;  00000000 Before first symbol
00000000 <_EIP>:
Code;  00000000 Before first symbol
   0:   0f 0b                     ud2a   
Code;  00000002 Before first symbol
   2:   90                        nop    
Code;  00000003 Before first symbol
   3:   00 34 e2                  add    %dh,(%edx,8)
Code;  00000006 Before first symbol
   6:   37                        aaa    
Code;  00000007 Before first symbol
   7:   c0 89 d8 2b 05 d0 63      rorb   $0x63,0xd0052bd8(%ecx)
Code;  0000000e Before first symbol
   e:   44                        inc    %esp
Code;  0000000f Before first symbol
   f:   c0 c1 f8                  rol    $0xf8,%cl
Code;  00000012 Before first symbol
  12:   03 69 00                  add    0x0(%ecx),%ebp



 I'm having a really hard time finding a 2.4 kernel that's stable on SMP.
I'm reluctant to go back to 2.4.22 or .21, because I have SATA drives in
this machine.

-- 
#define X(x,y) x##y
Peter Cordes ;  e-mail: X(peter@cor , des.ca)

"The gods confound the man who first found out how to distinguish the hours!
 Confound him, too, who in this place set up a sundial, to cut and hack
 my day so wretchedly into small pieces!" -- Plautus, 200 BC
signature.asc (application/pgp-signature, 351 B) 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iQC1AwUBQzmLzAWkmhLkWuRTAQLhqwT/bBcKzRCjJQAVSsmxb/e6CPtXDRfTb//c
jtVXnxuOXLdsC2DYY660VlnEkq8N1xkRbKoGROgCZcuGoDiyW4uhtmpbcWrdXn/y
rZG13TDUoZQqOMBFOKggW8DbgSnjGQmfr/wnFqF6T7uVRl4gPuNq4JDuVcrUSIkn
n50esESAGtnzaTUPJNF5SkguYviqc5Rkf0QjkCGGwbMjSN4GQZ2chw==
=b9tB
-----END PGP SIGNATURE-----
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.