Re: [myrinet] Re: Fast Directed Writes

Thanks for the information. In my previous mail, I tried to simplify
things for easy understanding. Actually we need to set two flags as
follows.


global flag: [____]

   p0        p1       p2        p3       p4         p6     ...
-----------------------------------------------------------
|        |         |        |        |          |         |
|        |         |        |        |          |         |
|        |         |        |        |          |         |
|        |         |        |        |          |         | ...
|        |         |        |        |          |         |
| .......|         |        |        |          |         |
|......SF|       SF|      SF|      SF|        SF|       SF|
-----------------------------------------------------------

The last two words SF indicates Msg_Size and Per_Processor_Flag
resepctively.

Here is hoe the sender works: Assuming each buffer has a fixed size
(size_of_buffer): (We are sending message to p5, hence p5 does not have a
buffer for itself).


sendMessage(void* msg,int msg_size)
{
   starting_offset = size_of_buffer -
                    (msg_size +
                     siezof(per_processor_flag) +
                     sizeof(per_processor_msg_size_indicator));

   append the message with msg_size and followed by a nonzero value

   trasmit the message to our reserved buffer starting from
    starting_offset in the remote buffer. (*)

   Transmit some non-zero value to the global flag(**)
}

In the figure, we are assuming p0 is transmitting the message to p5. So we
needed one DMA to transfer the message, the size and the
per_processor_flag (marked by *) and another one to set the global flag
(marked by **).

Here is the Recipient code: Periodically check the global flag.
If (global_flag){
  reset_flag();
  for all_processors {
    if (per_processor_flag != 0){
       reset_per_processor_flag
       get_the_size
       determine_where_the_message_starts
       process_the_message
  }
}

Since the global flag will be set by all the sender processors, it cannot
be part of all the buffers and hence we need another transmission for that
flag to be set. If there is a way of setting this 1 word long flag with a
register write, this will save us a great hassle.

Thanks again.

Umit.

On Wed, 5 Sep 2001, Greg Lindahl wrote:

> On Wed, Sep 05, 2001 at 02:32:24PM -0400, Umit Rencuzogullari wrote:
>
> > 1. Has the performance of this call has been measured in terms of
> > bandwidth and latency. In my experiments I found it to have lower
> > bandwidth than gm_send*() calls, but it might be just a measurement error.
> > Any statistics would be greatly appreciated.
>
> I found that for small writes, directed send was about 3 usec slower
> on a 133 mhz Lanai. That shouldn't give a significant bandwidth difference.
>
> > 2. Upon sending a message, we would like to set a polling flag on the
> > remote node which is going to indicate availability of a message. This
> > flag has to be independent of the message and has to arrive after the
> > message is written completely. My questions are:
> > 	2.1. Given that writing the flag is using the same priority, is
> > the order the message and the flag are sent preserved? (i.e. regardless
> > the size of the message, am I guaranteed to get the flag after the
> > message, if I happened to send them in that order?).
>
> Yes. GM delivers messages in order. And when it DMAs things, it always
> DMAs the message in order, too.
>
> However, if you don't want to send a separate message as a flag, you
> can put the flag at the end of your message, and a pointer to the flag
> at the start of the message. If the message is fixed size this is
> efficient and easy to do; if it's variable in size it's a bit more
> complicated to make sure that the flag value is going to be different
> from what's already there, if you don't want to zero the entire
> receive buffer.
>
> greg
>
>