Re: [Myricom help #8219] Fwd: [myrinet] Registered memory in MPICH over GM

Thanx, Patrick!  It seemed to me that MPICH-GM would <not> register memory
each time a send or receive is issued and that some sort of caching
mechanism would be necessary to avoid this.  This is what you affirmed and
you even told me what source file to check it out in.

In my own stuff, for the time being, I register a big block at startup,
never register it again, and manage this block as needed.  It would be
nice not to grab a big block in the beginning.  In order to know how to
deal with all this, one needs to know just how expensive it is to register
memory.  Is there a short answer to this?  If push comes to shove I can
experiment but life is short...

William K. Allard
Professor of Mathematics
Box 90320
Durham, NC 27708-0320
(919) 660-2861 Fax:(919) 660-2864

On Mon, 12 Nov 2001, Patrick Geoffray wrote:

> Karen Wang wrote:
>
> > > I have the following question.  If you call MPI_Send or MPI_Receive, are
> > > the respective buffers copied into DMA-able memory with
> > > gm_register_memory or is it assumed that the buffers are already
> > > gm_registered?
> >
> > GM send/receive functions need to operate memory that is DMAable.
> > It is user's responsibility to prepare the buffer to be DMAable
> > before calling GM send/receive functions.
>
> I am sure Bill knows that if his GM code works :-)
>
> > In your case, you will need to implement MPI_Send/MPI_Receive using
> > GM functions.
>
> No, the implementation is already done in MPICH-GM. I would say this
> is even the main interest of MPICH-GM :-))
>
> Seriously, to reply to the question: it depends upon the size
> of the message.
> For the Eager protocol (up to 16 KB), there is always a copy
> on both sides, because it's cheap to copy (to and from a pre-allocated
> registered area) and you need a non-blocking behaviour at the MPI level
> (the send immediately returns, even if the receive is not posted)
>
> For large messages, the Rendez-vous protocole uses gm_directed_send()s
> with a synchronization between sender and receiver. In this case, the
> application buffer on both sides have to be registered (zero-copy),
> but it's quite expensive to register/unregister each time. So we use
> a registration cache in MPICH-GM (file mpid/ch_gm/regcache.c) to keep
> memory registered. It will unregister by large blocks when you use
> all of the amount of DMA-able memory (80% of the RAM with GM-1.5) or
> when you free() a malloc()ed buffer.
>
> For a description of the Eager and Rendez-vous protocols:
>    http://www.myri.com/scs/GM_FAQ.html#mpich21
>
> Hope it helps.
>
> Patrick
>
> ----------------------------------------------------------
> |   Patrick Geoffray, Ph.D.      [email protected]
> |   Myricom, Inc.                http://www.myri.com
> |   Cell:  865-389-8852          685 Emory Valley Rd (B)
> |   Phone: 865-425-0978          Oak Ridge, TN 37830
> ----------------------------------------------------------
>