Re: [Myricom help #8219] Fwd: [myrinet] Registered memory in MPICH over GM

Karen Wang wrote:

> > I have the following question.  If you call MPI_Send or MPI_Receive, are
> > the respective buffers copied into DMA-able memory with
> > gm_register_memory or is it assumed that the buffers are already
> > gm_registered?
> 
> GM send/receive functions need to operate memory that is DMAable.
> It is user's responsibility to prepare the buffer to be DMAable
> before calling GM send/receive functions.

I am sure Bill knows that if his GM code works :-)

> In your case, you will need to implement MPI_Send/MPI_Receive using
> GM functions.

No, the implementation is already done in MPICH-GM. I would say this 
is even the main interest of MPICH-GM :-))

Seriously, to reply to the question: it depends upon the size 
of the message.
For the Eager protocol (up to 16 KB), there is always a copy 
on both sides, because it's cheap to copy (to and from a pre-allocated 
registered area) and you need a non-blocking behaviour at the MPI level 
(the send immediately returns, even if the receive is not posted)

For large messages, the Rendez-vous protocole uses gm_directed_send()s 
with a synchronization between sender and receiver. In this case, the 
application buffer on both sides have to be registered (zero-copy), 
but it's quite expensive to register/unregister each time. So we use 
a registration cache in MPICH-GM (file mpid/ch_gm/regcache.c) to keep 
memory registered. It will unregister by large blocks when you use 
all of the amount of DMA-able memory (80% of the RAM with GM-1.5) or 
when you free() a malloc()ed buffer.

For a description of the Eager and Rendez-vous protocols:
   http://www.myri.com/scs/GM_FAQ.html#mpich21

Hope it helps.

Patrick

----------------------------------------------------------
|   Patrick Geoffray, Ph.D.      [email protected] 
|   Myricom, Inc.                http://www.myri.com
|   Cell:  865-389-8852          685 Emory Valley Rd (B)
|   Phone: 865-425-0978          Oak Ridge, TN 37830
----------------------------------------------------------