Re: [myrinet] Re: Re: GM-1.5, Linux-2.4.14, gm_board_info hangs

Below is the result from the ksymoops on the node David is
having problems with.

The read/write speeds from gm_debug appear normal.  There does
not appear to be any back packets from gm_debug on the node
(badcrc).



Nov 13 22:58:32 t0002 kernel: cpu: 0, clocks: 991310, slice: 330436
Nov 13 22:58:32 t0002 kernel: cpu: 1, clocks: 991310, slice: 330436
Nov 13 22:58:32 t0002 kernel:   Receiver lock-up bug exists -- enabling work-around.
Nov 14 20:34:05 t0002 kernel: invalid operand: 0000
Nov 14 20:34:05 t0002 kernel: CPU:    0
Nov 14 20:34:05 t0002 kernel: EIP:    0010:[__free_pages_ok+92/512]    Not tainted
Nov 14 20:34:05 t0002 kernel: EIP:    0010:[<c012e9ec>]    Not tainted
Nov 14 20:34:05 t0002 kernel: EFLAGS: 00010202
Nov 14 20:34:05 t0002 kernel: eax: 00000890   ebx: 00000000   ecx: c1000000   edx: 00000000
Nov 14 20:34:05 t0002 kernel: esi: c1e48100   edi: f9a6c104   ebp: 00000000   esp: e42a3d38
Nov 14 20:34:05 t0002 kernel: ds: 0018   es: 0018   ss: 0018
Nov 14 20:34:05 t0002 kernel: Process hybcst (pid: 4435, stackpage=e42a3000)
Nov 14 20:34:05 t0002 kernel: Stack: f9a69088 f9a6908c f9a69090 f9a69090 00000000 c1e48100 f9a6c104 e42a3d78 
Nov 14 20:34:05 t0002 kernel:        f88b0076 f7ac2000 39204000 00000000 f7ac20f4 f7ac2000 e9a47010 00000000 
Nov 14 20:34:05 t0002 kernel:        e42a3dc0 f88a7bb9 f9a6c0f8 00000002 40bbd000 f7ac2000 00000000 39204000 
Nov 14 20:34:05 t0002 kernel: Call Trace: [<f88b0076>] [<f88a7bb9>] [<f88a7fb0>] [<f88aec19>] [<f88ae5e6>] 
Nov 14 20:34:05 t0002 kernel:    [<f88b1548>] [<f88b14e5>] [fput+78/240] [do_check_pgt_cache+263/304] [filp_close+141/160] [put_files_struct+77/208] 
Nov 14 20:34:05 t0002 kernel:    [<f88b1548>] [<f88b14e5>] [<c01361fe>] [<c01130e7>] [<c013505d>] [<c0119a2d>] 
Nov 14 20:34:05 t0002 kernel:    [<c011a22f>] [<c011f673>] [<c011f72d>] [<c0106dc4>] [<c01145b9>] [<c0106f54>] 
Nov 14 20:34:05 t0002 kernel: Code: 0f 0b 83 66 18 eb b8 00 e0 ff ff 21 e0 f6 40 05 20 0f 85 31 

>>EIP; c012e9ec <delete_from_swap_cache+3c/50>   <=====
Trace; f88b0076 <[gm]gm_instance_load_mcp+422/d2c>
Trace; f88a7bb9 <[rtc]__module_license+a4343/a57ea>
Trace; f88a7fb0 <[rtc]__module_license+a473a/a57ea>
Trace; f88aec19 <[gm]gm_test_and_clear_sram+1b9/2e4>
Trace; f88ae5e6 <[gm]gm_map_board_region+46/4c>
Trace; f88b1548 <[gm]gm_minor_alloc+b0/f4>
Trace; f88b14e5 <[gm]gm_minor_alloc+4d/f4>
Trace; f88b1548 <[gm]gm_minor_alloc+b0/f4>
Trace; f88b14e5 <[gm]gm_minor_alloc+4d/f4>
Trace; c01361fe <bread+6e/70>
Trace; c01130e7 <__verify_write+157/190>
Trace; c013505d <write_some_buffers+5d/f0>
Trace; c0119a2d <exit_notify+4d/2f0>
Trace; c011a22f <sys_wait4+2df/400>
Trace; c011f673 <send_sig_info+33/a0>
Trace; c011f72d <force_sig_info+4d/b0>
Trace; c0106dc4 <do_signal+244/2b0>
Trace; c01145b9 <complete+79/d0>
Trace; c0106f54 <signal_return+14/18>
Code;  c012e9ec <delete_from_swap_cache+3c/50>
00000000 <_EIP>:
Code;  c012e9ec <delete_from_swap_cache+3c/50>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c012e9ee <delete_from_swap_cache+3e/50>
   2:   83 66 18 eb               andl   $0xffffffeb,0x18(%esi)
Code;  c012e9f2 <delete_from_swap_cache+42/50>
   6:   b8 00 e0 ff ff            mov    $0xffffe000,%eax
Code;  c012e9f7 <delete_from_swap_cache+47/50>
   b:   21 e0                     and    %esp,%eax
Code;  c012e9f9 <delete_from_swap_cache+49/50>
   d:   f6 40 05 20               testb  $0x20,0x5(%eax)
Code;  c012e9fd <delete_from_swap_cache+4d/50>
  11:   0f 85 31 00 00 00         jne    48 <_EIP+0x48> c012ea34 <free_page_and_swap_cache+34/40>




On Thu, Nov 15, 2001 at 01:40:55PM -0800, Bob Felderman wrote:
> You might also want to try
> 
> 	dmesg | ksymoops
> 
> to see if the backtrace it gives you provides any useful information.
> 
> 
> 
> On Thu, 15 Nov 2001, Susan Blackford wrote:
> >Hi,
> >
> >This is a kernel "oops"/seg fault.
> >
> >Did you see any strange messages in the kernel log (dmesg) when you
> >GM_INSTALL?
> >
> >What about when you ran the mapper, did you see any strange messages?
> >
> >Did the gm_board_info hang happen on all of the nodes in your cluster,
> >or only a specific node?
> >
> >Would it be possible for us to have remote access to this machine and
> >be able to load/unload the GM module (root) and experiment?
> >
> >This is very strange.
> >
> >Susan
> >
> >
> >On Thursday 15 November 2001 02:48 pm, Ritch, David wrote:
> >> I'm a relative newcomer to myrinet, and I have a question.
> >>
> >> I'm attempting to run GM-1.5 on a cluster of dual-processor Pentium-4 nodes
> >> under RedHat-7.1.  I compiled the 2.4.14 kernel with kgcc, and compiled gm
> >> with CC="kgcc -I/usr/src/linux/include", so as to get the include files
> >> associated with the correct kernel.  I'm also running mpi-1.2.1..7,
> >> compiled with gcc.
> >>
> >> GM appears to work OK at first, but usually when I run a real application
> >> on it, a gm_board_info process is started by root, and goes into a disk
> >> hang (ps reports status of D).  At about the same time, the following
> >> appears in /var/log/messages:
> >>
> >> Nov 13 17:42:35 t0001 kernel: invalid operand: 0000
> >> Nov 13 17:42:35 t0001 kernel: CPU:    0
> >> Nov 13 17:42:35 t0001 kernel: EIP:    0010:[__free_pages_ok+92/512]    Not
> >> tainted
> >> Nov 13 17:42:35 t0001 kernel: EIP:    0010:[<c012ea1c>]    Not tainted
> >> Nov 13 17:42:35 t0001 kernel: EFLAGS: 00010202
> >> Nov 13 17:42:35 t0001 kernel: eax: 00000890   ebx: 00000000   ecx: c1000000
> >> edx: 00000000
> >> Nov 13 17:42:35 t0001 kernel: esi: c1e3e540   edi: f8920c24   ebp: 00000000
> >> esp: d0c91d38
> >> Nov 13 17:42:35 t0001 kernel: ds: 0018   es: 0018   ss: 0018
> >> Nov 13 17:42:35 t0001 kernel: Process systest (pid: 21412,
> >> stackpage=d0c91000)
> >> Nov 13 17:42:35 t0001 kernel: Stack: f891fe08 f891fe0c f891fe10 f891fe10
> >> 00000000 c1e3e540 f8920c24 d0c91d78
> >> Nov 13 17:42:35 t0001 kernel:        f88b4076 f7a82000 38f95000 00000000
> >> f7a820f4 f7a82000 d0ba0000 00000000
> >> Nov 13 17:42:35 t0001 kernel:        d0c91dc0 f88abbb9 f8920c18 00000004
> >> 080d1000 f7a82000 00000000 38f95000
> >>
> >> I'm not sure what is going on, but my myrinet does not appear to be
> >> working. Could someone help to enlighten me on this?
> >>
> >> Thanks,
> >>
> >> dbr
> >> --
> >> David B. Ritch
> >> High Performance Technologies, Inc.
> >> [email protected]
> >
> >----------------------------------------
> >Content-Type: text/html; charset="iso-8859-1"; name="Attachment: 1"
> >Content-Transfer-Encoding: quoted-printable
> >Content-Description:
> >----------------------------------------
> >
> >

-- 
Craig Tierney ([email protected])