Re: FW: increasing the amount of memory that can be registered

This is a multi-part message in MIME format.
--------------050604070104090300090909
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

Reese Faucette wrote:
>> Word of mouth is that it is possible to specify the amount of memory
>> that is available to be registered as a parameter. Can I increasing the
>> upper limit for the amount of memory gm can register by specifying it as
>> a parameter for the gm module? Could you tell me how to do that? I was
>> unable to find it in the documentation.
> 
> 
> When you insmod the driver, specify "gm_max_locked_mbytes=MMM" when MMM 
> is the maximum number of megabytes of memory you with to allow GM to 
> register.
> 

Let me add a few more precisions:

If you load gm through the standard /etc/init.d/gm script (if don't 
insmod manually), you have the possibility of creating a file 
<gm-installation-dir>/etc/gm/module.options containing options to pass 
to the gm modules, in particular you can put there the parameter:
    gm_max_locked_mbytes=MMM
/etc/init.d/gm will read that file and pass that to insmod.

>> I was seeing in the documents somewhere that when same memory is
>> registered for 2 ports, gm realizes it and avoids duplication. Is it
>> true.

Altough the physical page is properly shared in this case, the counter 
of pages registered is currently decremented twice, one time for each 
port. So you will reach the limit specified in gm_max_locked_mbytes=MMM 
(or the default limit of 50% of physical memory) earlier than planned. 
This is a bug and will be corrected in the next version. In the meantime 
you can either cheat and put a bigger value for gm_max_locked_mbytes=MMM 
(eventually more than the total amount of physical memory). Or you can 
apply the attached patch to the gm source code to correct the accounting 
(patch work for at least gm-2.0.12 => gm-2.0.15 and gm-2.1.3 => 2.1.5, 
and probably others)

Regards,

Loic


--------------050604070104090300090909
Content-Type: text/x-patch;
 name="shared-page-accounting.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="shared-page-accounting.patch"

Index: drivers/linux/gm/gm_arch.c
===================================================================
RCS file: /repository/gm/drivers/linux/gm/gm_arch.c,v
retrieving revision 1.43.2.28
diff -u -r1.43.2.28 gm_arch.c
--- drivers/linux/gm/gm_arch.c	5 Nov 2004 16:03:44 -0000	1.43.2.28
+++ drivers/linux/gm/gm_arch.c	22 Nov 2004 23:29:21 -0000
@@ -1487,6 +1487,7 @@
       int counter = 1;
       gm_arch_mutex_enter (&gm_linux_pfn_sync);
       status = gm_hash_insert (gm_linux_pfn_hash, &pfn, &counter);
+      atomic_dec (&gm_linux_max_user_locked_pages);
       gm_arch_mutex_exit (&gm_linux_pfn_sync);
       if (status != GM_SUCCESS)
        {
@@ -1507,7 +1508,6 @@
 #if GM_DEBUG
   user_lock_cnt++;
 #endif
-  atomic_dec (&gm_linux_max_user_locked_pages);
 
 #if 0 /* GM_ENABLE_FORK_SYSTEM*/
   /* loic: this is a clever hack from bob, which transforms fork into
@@ -1623,6 +1623,7 @@
   else
     {
       counter_ptr[0]--;
+      atomic_inc (&gm_linux_max_user_locked_pages);
       counter_ptr = gm_hash_remove (gm_linux_pfn_hash, &pfn);
       if (!counter_ptr)
        {
@@ -1725,8 +1726,6 @@
       put_page (page);
     }
 
-  atomic_inc (&gm_linux_max_user_locked_pages);
-
   GM_RETURN_NOTHING ();
 }
 

--------------050604070104090300090909
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Myrinet mailing list
[email protected]
http://email.osc.edu/mailman/listinfo/myrinet

--------------050604070104090300090909--