Re: [myrinet] mpich-gm hangs for >=128 size messages

| MPICH-GM seems to hang for messages 128 bytes and larger over Myrinet.
| I can only get MPICH-GM to work for 64 byte messages and smaller.  
| If I run MPICH-GM over shared memory the problem does not occur. 
| MPI-LAM works for the *same* program over TCP for large messages 
| and shared memory.
| 
| Any ideas how to fix this problem for MPICH-GM?
| I am only using simple send and receives i.e.:
|       
|       MPI_Recv( msg, count, MPI_CHAR, nodeid, tag, MPI_COMM_WORLD, &st);
|       MPI_Send( msg, count, MPI_CHAR, nodeid, tag, MPI_COMM_WORLD );

I am curious as to whether your program is "safe".  Some MPI programs are
what the standard calls "unsafe", and their completion can depend on the
implementation and the sizes of messages.  An example of an unsafe program
is the following one, which exchanges data:

          Process 0              Process 1
          ---------              ---------

          MPI_Send to 1          MPI_Send to 0
          MPI_Recv from 1        MPI_Recv from 0

If messages are above a certain size, then the messages must be buffered
so that the sends can complete.  In fact, one can see that this program
cannot complete if the messages are very large.  The exact size depends
on the implementation.  For performance reasons, some implementations define
the limit on unsafe programs to be fairly small.  64 bytes is reasonable.

You can convert this program into a safe one in three ways.

1.  Interleave sends and receives:
            
          Process 0              Process 1
          ---------              ---------

          MPI_Send to 1          MPI_Recv from 0
          MPI_Recv from 1        MPI_Send to 0

2.  Use the MPI_Sendrecv function:

          Process 0              Process 1
          ---------              ---------

          MPI_Sendrecv to/from 1   MPI_Sendrecv to/from 0

3.  Use the MPI non-blocking operations (this is the most common solution):

          Process 0              Process 1
          ---------              ---------

          MPI_Isend to 1         MPI_Isend to 0
          MPI_Irecv from 1       MPI_Irecv from 0
          MPI_Waitall            MPI_Waitall 

Each of these solutions allows the receive to cooperate in moving the data
from the user's send buffer to the user's receive buffer, eliminating the
need for system buffering.  My guess is that any of these modifications will
cause your program to work on MPICH-GM.

Regards,
Rusty Lusk