[Myricom help #8248] RE: Re: GM-1.5, Linux-2.4.14, gm_board_info hangs

On Fri, Nov 16, 2001 at 02:22:14AM -0500, Ritch, David wrote:
> I did a bit of work to try to clean up and implement all the suggestions
> that were made.  I discovered that I had compiled the kernel with 4GB memory
> support, which I thought I had turned off.  In the current configuration, it
> is turned off.  Of course, I also recompiled and reinstalled gm.

Note that GM should work flawlessly with a kernel with 4GB support (if
there is any problem when enabling this option, please report it),
only the 64GB Linux/x86 option is not supported.  The message "Unusual
memory configuration" is misleading, it only means you have highmem
support, but is harmless.

> [...]
> 
> Here is the ksymoops output on node 1 (called t0001);
> 
> Code: 0f 0b 83 66 18 eb b8 00 e0 ff ff 21 e0 f6 40 05 20 0f 85 31 
> 
> >>EIP; c012de1c <__free_pages_ok+5c/200>   <=====
> Trace; f88b7076 <[gm]gm_arch_unlock_user_buffer_page+196/1a4>
> Trace; f88aebb9 <[gm]gm_dereference_mapping+1ed/430>
> Trace; f88aefb0 <[gm]gm_disable_port_DMAs+1b4/280>
> Trace; f88b5c19 <[gm]gm_port_state_close+91/158>

You just discovered a bug that will affect GM with Linux kernel >=
2.4.14. As the linux VM was rewritten between 2.4.9 and 2.4.10 and
began just now to stabilize, some of the semi-internal Linux APIs GM
use for memory registration have changed with the last release and
broke GM in the situation you were describing.

Can you try the one-line attached patch to GM and tell us if it solves
the problem you are seeing?

Apparently the set of  conditions needed together to trigger this bug
are:
  - use kernel >= 2.4.14
  - probably need to have an heavy use of memory
  - unclean exit of the application (or in general unmapping pages
     before they deregistered).

Fortunately the bug was not insidious, either everything happen
normally, or you will trigger the bug and get an oops pointing
directly to the problem.

(the technical change is that __free_page and page_cache_release are
no longer equivalent starting from 2.4.14, and a quick exam reveal
that freeing anonymous memory should use page_cache_release instead of
__free_page). 


Regards,

Loic PRYLLI
LIP/ENS-Lyon, FRANCE (also sometimes working with Myricom)
email: [email protected] (or [email protected])

Index: drivers/linux/gm/gm_arch.c
===================================================================
RCS file: /ufs/src/repositories/gm/drivers/linux/gm/gm_arch.c,v
retrieving revision 1.277
diff -u -r1.277 gm_arch.c
--- drivers/linux/gm/gm_arch.c	2001/10/18 23:55:13	1.277
+++ drivers/linux/gm/gm_arch.c	2001/11/16 16:41:34
@@ -1539,7 +1539,7 @@
   user_unlock_cnt++;
 #endif
 
-  __free_page (page);
+  page_cache_release (page);
   gm_linux_max_user_locked_pages += 1;
 }