Re: kernel BUG with 2.4.27-om20041102-tab on SMP when doing heavy I/O

Tomas Vinar <[email protected]>
Newsgroups gmane.linux.cluster.openmosix.devel
Message-ID <[email protected]> 
We use 2.4.24-om2 and it seems to be pretty stable.
We have 2x4-way, 1x2-way, and several single CPU machines.

Tomas


On 9/27/05, Peter Cordes <[email protected]> wrote:
>  I have been running  2.4.27-om20041102-tab (peter-OPrYhD/TZC/[email protected])
> (gcc version 3.3.5 (Debian 1:3.3.5-13)) #1 SMP Wed Sep 7  on my dual Opteron
> cluster.  The master node has 4GB of RAM, and the other 7 nodes have 2GB.
> All with gigabit ethernet.  The master has an Intel e100 built in to the
> mobo too, which is used to connect to the outside world.
>
>  My kernel config is
> CONFIG_MOSIX=y
> # CONFIG_MOSIX_TOPOLOGY is not set
> CONFIG_MOSIX_SECUREPORTS=y
> CONFIG_MOSIX_DISCLOSURE=3
> # CONFIG_MOSIX_FS is not set
> CONFIG_MOSIX_PIPE_EXCEPTIONS=y
> # CONFIG_MOSIX_NO_OOM is not set
> CONFIG_MOSIX_EXT_LOCALTIME=y
>
>  My filesystems are JFS on RAID0 and RAID1 partitions of two SATA drives,
> connected to the SATA_SIL ports on the Tyan S2882 motherboard.
>
>  While doing a big file mirroring update with rsync (of BLAST dna sequence
> databases...), the kernel logged some errors.  I don't think this has
> happened consistently during the bi-monthly updates, though.  Anyway:
> first one:
> Sep 22 00:12:34 nyneve kernel: kernel BUG at page_alloc.c:144!
> Sep 22 00:12:34 nyneve kernel: invalid operand: 0000
> Sep 22 00:12:34 nyneve kernel: CPU:    1
> Sep 22 00:12:34 nyneve kernel: EIP:    0010:[__free_pages_ok+80/768]    Not tainted
> Sep 22 00:12:34 nyneve kernel: EFLAGS: 00010282
> Sep 22 00:12:34 nyneve kernel: eax: c03c9834   ebx: c2da16a0   ecx: 00000000   edx: ebf04604
> Sep 22 00:12:34 nyneve kernel: esi: c2da16a0   edi: 00000000   ebp: 0001ee61   esp: c43c9f08
> Sep 22 00:12:34 nyneve kernel: ds: 0018   es: 0018   ss: 0018
> Sep 22 00:12:34 nyneve kernel: Process kswapd (pid: 5, stackpage=c43c9000)
> Sep 22 00:12:34 nyneve kernel: Stack: 00000282 00000003 c03c9ec4 ebf06540 ebf06540 ebf06540 c2da16a0 c0155369
> Sep 22 00:12:34 nyneve kernel:        ebf06540 00000000 c2da16a0 c03c9758 0001ee61 c014736b c2da16a0 000001d0
> Sep 22 00:12:34 nyneve kernel:        c43c8000 00000c80 000001d0 0000001f 00000020 000001d0 c03c9758 c03c9758
> Sep 22 00:12:34 nyneve kernel: Call Trace:    [try_to_free_buffers+185/336] [shrink_cache+843/1008] [shrink_caches+74/96] [try_to_free_pages_zone+83/224] [kswapd_balance_pgdat+94/160]
> Sep 22 00:12:34 nyneve kernel: Code: 0f 0b 90 00 34 e2 37 c0 89 d8 2b 05 d0 63 44 c0 c1 f8 03 69
>
> >>eax; c03c9834 <contig_page_data+294/3b8>
> >>ebx; c2da16a0 <_end+2922eb4/386e1894>
> >>edx; ebf04604 <_end+2ba85e18/386e1894>
> >>esi; c2da16a0 <_end+2922eb4/386e1894>
> >>esp; c43c9f08 <_end+3f4b71c/386e1894>
>
> Code;  00000000 Before first symbol
> 00000000 <_EIP>:
> Code;  00000000 Before first symbol
>    0:   0f 0b                     ud2a
> Code;  00000002 Before first symbol
>    2:   90                        nop
> Code;  00000003 Before first symbol
>    3:   00 34 e2                  add    %dh,(%edx,8)
> Code;  00000006 Before first symbol
>    6:   37                        aaa
> Code;  00000007 Before first symbol
>    7:   c0 89 d8 2b 05 d0 63      rorb   $0x63,0xd0052bd8(%ecx)
> Code;  0000000e Before first symbol
>    e:   44                        inc    %esp
> Code;  0000000f Before first symbol
>    f:   c0 c1 f8                  rol    $0xf8,%cl
> Code;  00000012 Before first symbol
>   12:   03 69 00                  add    0x0(%ecx),%ebp
>
>
> ############### next one from rsync: ###############################
> Sep 22 00:12:34 nyneve kernel:  kernel BUG at page_alloc.c:144!
> Sep 22 00:12:34 nyneve kernel: invalid operand: 0000
> Sep 22 00:12:34 nyneve kernel: CPU:    1
> Sep 22 00:12:34 nyneve kernel: EIP:    0010:[__free_pages_ok+80/768]    Not tainted
> Sep 22 00:12:34 nyneve kernel: EFLAGS: 00010282
> Sep 22 00:12:34 nyneve kernel: eax: c03c9834   ebx: c2da4a40   ecx: 00000000   edx: ebedabc4
> Sep 22 00:12:34 nyneve kernel: esi: c2da4a40   edi: 00000000   ebp: 0001ee6f   esp: d324de18
> Sep 22 00:12:34 nyneve kernel: ds: 0018   es: 0018   ss: 0018
> Sep 22 00:12:34 nyneve kernel: Process rsync (pid: 21475, stackpage=d324d000)
> Sep 22 00:12:34 nyneve kernel: Stack: 00000282 00000003 c03c9ec4 ebea56c0 ebea56c0 ebea56c0 c2da4a40 c0155369
> Sep 22 00:12:34 nyneve kernel:        ebea56c0 00000000 c2da4a40 c03c9758 0001ee6f c014736b c2da4a40 000001d2
> Sep 22 00:12:34 nyneve kernel:        d324c000 00000c1c 000001d2 00000001 0000001f 000001d2 c03c9758 c03c9758
> Sep 22 00:12:34 nyneve kernel: Call Trace:    [try_to_free_buffers+185/336] [shrink_cache+843/1008] [shrink_caches+74/96] [try_to_free_pages_zone+83/224] [balance_classzone+77/480]
> Sep 22 00:12:34 nyneve kernel: Code: 0f 0b 90 00 34 e2 37 c0 89 d8 2b 05 d0 63 44 c0 c1 f8 03 69
>
> >>eax; c03c9834 <contig_page_data+294/3b8>
> >>ebx; c2da4a40 <_end+2926254/386e1894>
> >>edx; ebedabc4 <_end+2ba5c3d8/386e1894>
> >>esi; c2da4a40 <_end+2926254/386e1894>
> >>esp; d324de18 <_end+12dcf62c/386e1894>
>
> Code;  00000000 Before first symbol
> 00000000 <_EIP>:
> Code;  00000000 Before first symbol
>    0:   0f 0b                     ud2a
> Code;  00000002 Before first symbol
>    2:   90                        nop
> Code;  00000003 Before first symbol
>    3:   00 34 e2                  add    %dh,(%edx,8)
> Code;  00000006 Before first symbol
>    6:   37                        aaa
> Code;  00000007 Before first symbol
>    7:   c0 89 d8 2b 05 d0 63      rorb   $0x63,0xd0052bd8(%ecx)
> Code;  0000000e Before first symbol
>    e:   44                        inc    %esp
> Code;  0000000f Before first symbol
>    f:   c0 c1 f8                  rol    $0xf8,%cl
> Code;  00000012 Before first symbol
>   12:   03 69 00                  add    0x0(%ecx),%ebp
>
>
>
>  A few days later I ran my update script manually, and hit problems again.
> formatdb (which reads giant flat text files and writes indexed databases
> that BLAST can search against) was segfaulting...
>
> This one from rsync again:
> Sep 26 15:05:21 nyneve kernel:  kernel BUG at page_alloc.c:144!
> Sep 26 15:05:21 nyneve kernel: invalid operand: 0000
> Sep 26 15:05:21 nyneve kernel: CPU:    0
> Sep 26 15:05:21 nyneve kernel: EIP:    0010:[__free_pages_ok+80/768]    Not tainted
> Sep 26 15:05:21 nyneve kernel: EFLAGS: 00010282
> Sep 26 15:05:21 nyneve kernel: eax: c03c9834   ebx: c2f77970   ecx: 00000000   edx: ea9f0504
> Sep 26 15:05:21 nyneve kernel: esi: c2f77970   edi: 00000000   ebp: 0001f0de   esp: ec4a1e18
> Sep 26 15:05:21 nyneve kernel: ds: 0018   es: 0018   ss: 0018
> Sep 26 15:05:21 nyneve kernel: Process rsync (pid: 12692, stackpage=ec4a1000)
> Sep 26 15:05:21 nyneve kernel: Stack: 00000282 00000003 c03c9ec4 ea9f9cc0 ea9f9cc0 ea9f9cc0 c2f77970 c0155369
> Sep 26 15:05:21 nyneve kernel:        ea9f9cc0 00000000 c2f77970 c03c9758 0001f0de c014736b c2f77970 000001d2
> Sep 26 15:05:21 nyneve kernel:        ec4a0000 00000c80 000001d2 0000000d 00000020 000001d2 c03c9758 c03c9758
> Sep 26 15:05:21 nyneve kernel: Call Trace:    [try_to_free_buffers+185/336] [shrink_cache+843/1008] [shrink_caches+74/96] [try_to_free_pages_zone+83/224] [balance_classzone+77/480]
> Sep 26 15:05:21 nyneve kernel: Code: 0f 0b 90 00 34 e2 37 c0 89 d8 2b 05 d0 63 44 c0 c1 f8 03 69
>
> >>eax; c03c9834 <contig_page_data+294/3b8>
> >>ebx; c2f77970 <_end+2af9184/386e1894>
> >>edx; ea9f0504 <_end+2a571d18/386e1894>
> >>esi; c2f77970 <_end+2af9184/386e1894>
> >>esp; ec4a1e18 <_end+2c02362c/386e1894>
>
> Code;  00000000 Before first symbol
> 00000000 <_EIP>:
> Code;  00000000 Before first symbol
>    0:   0f 0b                     ud2a
> Code;  00000002 Before first symbol
>    2:   90                        nop
> Code;  00000003 Before first symbol
>    3:   00 34 e2                  add    %dh,(%edx,8)
> Code;  00000006 Before first symbol
>    6:   37                        aaa
> Code;  00000007 Before first symbol
>    7:   c0 89 d8 2b 05 d0 63      rorb   $0x63,0xd0052bd8(%ecx)
> Code;  0000000e Before first symbol
>    e:   44                        inc    %esp
> Code;  0000000f Before first symbol
>    f:   c0 c1 f8                  rol    $0xf8,%cl
> Code;  00000012 Before first symbol
>   12:   03 69 00                  add    0x0(%ecx),%ebp
>
>
> ################ and one from formatdb ###########
> Sep 26 15:05:22 nyneve kernel:  kernel BUG at page_alloc.c:144!
> Sep 26 15:05:22 nyneve kernel: invalid operand: 0000
> Sep 26 15:05:22 nyneve kernel: CPU:    1
> Sep 26 15:05:22 nyneve kernel: EIP:    0010:[__free_pages_ok+80/768]    Not tainted
> Sep 26 15:05:22 nyneve kernel: EFLAGS: 00010282
> Sep 26 15:05:22 nyneve kernel: eax: c03c9834   ebx: c2f80650   ecx: 00000000   edx: ea916dc4
> Sep 26 15:05:22 nyneve kernel: esi: c2f80650   edi: 00000000   ebp: 0001f0db   esp: d8f39e18
> Sep 26 15:05:22 nyneve kernel: ds: 0018   es: 0018   ss: 0018
> Sep 26 15:05:22 nyneve kernel: Process formatdb (pid: 12688, stackpage=d8f39000)
> Sep 26 15:05:22 nyneve kernel: Stack: 00000282 00000003 c03c9ec4 ea917640 ea917640 ea917640 c2f80650 c0155369
> Sep 26 15:05:22 nyneve kernel:        ea917640 00000000 c2f80650 c03c9758 0001f0db c014736b c2f80650 000001d2
> Sep 26 15:05:22 nyneve kernel:        d8f38000 00000c1c 000001d2 00000002 0000001f 000001d2 c03c9758 c03c9758
> Sep 26 15:05:22 nyneve kernel: Call Trace:    [try_to_free_buffers+185/336] [shrink_cache+843/1008] [shrink_caches+74/96] [try_to_free_pages_zone+83/224] [balance_classzone+77/480]
> Sep 26 15:05:22 nyneve kernel: Code: 0f 0b 90 00 34 e2 37 c0 89 d8 2b 05 d0 63 44 c0 c1 f8 03 69
>
> >>eax; c03c9834 <contig_page_data+294/3b8>
> >>ebx; c2f80650 <_end+2b01e64/386e1894>
> >>edx; ea916dc4 <_end+2a4985d8/386e1894>
> >>esi; c2f80650 <_end+2b01e64/386e1894>
> >>esp; d8f39e18 <_end+18abb62c/386e1894>
>
> Code;  00000000 Before first symbol
> 00000000 <_EIP>:
> Code;  00000000 Before first symbol
>    0:   0f 0b                     ud2a
> Code;  00000002 Before first symbol
>    2:   90                        nop
> Code;  00000003 Before first symbol
>    3:   00 34 e2                  add    %dh,(%edx,8)
> Code;  00000006 Before first symbol
>    6:   37                        aaa
> Code;  00000007 Before first symbol
>    7:   c0 89 d8 2b 05 d0 63      rorb   $0x63,0xd0052bd8(%ecx)
> Code;  0000000e Before first symbol
>    e:   44                        inc    %esp
> Code;  0000000f Before first symbol
>    f:   c0 c1 f8                  rol    $0xf8,%cl
> Code;  00000012 Before first symbol
>   12:   03 69 00                  add    0x0(%ecx),%ebp
>
>
>
>  I'm having a really hard time finding a 2.4 kernel that's stable on SMP.
> I'm reluctant to go back to 2.4.22 or .21, because I have SATA drives in
> this machine.
>
> --
> #define X(x,y) x##y
> Peter Cordes ;  e-mail: X(peter@cor , des.ca)
>
> "The gods confound the man who first found out how to distinguish the hours!
>  Confound him, too, who in this place set up a sundial, to cut and hack
>  my day so wretchedly into small pieces!" -- Plautus, 200 BC
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.1 (GNU/Linux)
>
> iQC1AwUBQzmLzAWkmhLkWuRTAQLhqwT/bBcKzRCjJQAVSsmxb/e6CPtXDRfTb//c
> jtVXnxuOXLdsC2DYY660VlnEkq8N1xkRbKoGROgCZcuGoDiyW4uhtmpbcWrdXn/y
> rZG13TDUoZQqOMBFOKggW8DbgSnjGQmfr/wnFqF6T7uVRl4gPuNq4JDuVcrUSIkn
> n50esESAGtnzaTUPJNF5SkguYviqc5Rkf0QjkCGGwbMjSN4GQZ2chw==
> =b9tB
> -----END PGP SIGNATURE-----
>
>
>


--
--------------------------------------------------------------------------
Tomas Vinar, University of Waterloo
E-mail:     [email protected]
Office:      DC2582A, (519)888-4567 ext. 3564
Home phone: (519)725-5440


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.