Re: [Myrinet] questions

Hmm... This is better than I thought.  Just to make sure, you seem to 
be saying that <at the time> when GM_SUCCESS is returned to the sender, 
the receiver is filled with the data.  This implies to me that there is 
some sort of ACK sent by the receiver NIC when the all the data is 
received.

In particular if a bunch of nodes do directed sends, wait for GM_SUCCESS 
as above (and below) and then everyone makes a barrier call then, after 
the barrier returns, everyone can be sure that all the data to be received 
is in place. Note that this allows compuation to be done while CPU is busy 
with something else (assuming send tokens, etc. are managed, etc.).

On Tue, 11 Jun 2002, Patrick Geoffray wrote:

> Bill Allard wrote:
> > What does return from the handler passed to gm_directed_send_with_callback 
> > signify?  I would be nice if it signified that that the data sent was in 
> > the receiver space but I suspect it only signifies that that the data has 
> > left the sending NIC.
> 
> If it returns GM_SUCCESS, that means it has been delivered, Directed 
> sends are reliable. So the sender is effectively notified that the data 
> has been completely written, but the receiver is not notified of anything.
> Actually, there is one case where the sender can get a GM_SUCCESS but 
> the data is not written on the remote side: it's when the remote buffer 
> is not DMA-able (not registerted or not allocated with gm_dma_malloc). 
> In this case, there will be a cryptic message on the receiving side 
> about bogus page recv... Don't ask me why it does not generate a send 
> completion failure, it's GM.
> 
> > Also, gm_directed_send_with_callback (or 
> > gm_send_with_callback, for that matter) returns as soon as its arguments 
> > have been written to the NIC, which is almost immediatedly, right?
> 
> Yes, the send operation itself return has soon as the send token has 
> been written to the NIC. However, the callback is called only when the 
> data has been received if reliable sends, or after the data has left the 
> sender NIC if datagrams.
> 
> Patrick
> 
> ----------------------------------------------------------
> |   Patrick Geoffray, Ph.D.      [email protected]
> |   Myricom, Inc.                http://www.myri.com
> |   Cell:  865-389-8852          685 Emory Valley Rd (B)
> |   Phone: 865-425-0978          Oak Ridge, TN 37830
> ----------------------------------------------------------
> 

-- 
William K. Allard
Professor of Mathematics
Duke University 
Box 90320
Durham, NC 27708-0320
(919) 660-2861 Fax:(919) 660-2864