RE: [Myrinet] [Myricom help #13378] hybrid mpich support

Patrick,

My use of "hybrid" was very unfortunate, and I think that may
be leading to confusion.

I think you mean "hybrid" in the blocking/polling sense, where
receives block after polling for a while.  I did report that
hybrid and blocking did not work with shared memory [#12777],
but this is a different issue.

I meant "hybrid" in the sense that shared memory is used between
_some_ processes on the local machine, while others use Myrinet
for communications.

The patch I provided allows the mpi program to use shared memory
for "close" processors (using the same Myrinet card), and Myrinet
for "far" processors (using different cards).  Of course, for this
to work, you also need a way to specify which CPU a process should
run on (beyond the scope of mpirun -- I do it similar to running
a program under gdb, with a wrapper program that reads the MPI
env variables and does the "right thing"), and you need the machine
file set up to spawn the right processors on the right cards.
[Note that with tweaking, you could flip the usage, which might be
desirable if processors on the same node are assigned different cards.]

I'm not claming a hybrid approach is right in all (any?) circumstances,
but it is something to experiment with, especially if you have a NUMA
or multiple-processor-bus machine with a Myrinet interface per node.

Thanks,
Kevin


> -----Original Message-----
> From: Patrick Geoffray [mailto:[email protected]]
> Sent: Monday, October 07, 2002 4:42 PM
> To: Van Maren, Kevin
> Cc: '[email protected]'; '[email protected]'
> Subject: Re: [Myrinet] hybrid mpich support
> 
> 
> Hi Kevin,
> 
> On Mon, 2002-10-07 at 18:21, Van Maren, Kevin wrote:
> > Here is a patch, relative to the 1.2.4..8a release, that
> > adds the ability to restrict shared memory on a SMP to
> > only those processes that use the same Myrinet interface.
> 
> A similar patch is already in the CVS (1.2.4..8b is close to release).
> 
> > I developed this code to do some experimentation on a
> > NUMA-ish machine, but I thought it may be more generally
> > useful.  It is not complete (MPD support isn't runtime-
> > tunable nad is untested), but I would appreciate feedback/
> > comments on the code.  It works for me, but YMMV.
> 
> It is much simpler than that: shared memory should be always disabled
> when the receive mode is not polling. When using blocking or 
> hyrid, the
> ch_gm device may block, and it's forbidden in a multi-device
> configuration with MPICH.
> 
> The code looks like that:
> 
>   /* Set the shared memory support */
>   if (gmpi.gm_receive_mode == gm_receive)
>     {
>       if (gmpi_shmem == NULL) 
>         {
>           gmpi.shmem = 1;
>         }
>       else
>         {
>           if (strcmp (gmpi_shmem, "1") == 0) 
>             {
>               gmpi.shmem = 1;
>             }
>           else
>             {
>               gmpi.shmem = 0;
>             }
>         }
>     }
>   else
>     {
>       gmpi.shmem = 0;
>     }
> 
>  
> > Also included is a fix for shared memory (Myricom #13195)
> > to correct the mmap() size calculation, and some changes
> > to eliminate warnings (unsigned i < 0 comparisons).
> 
> The mmap change is also already committed, but I will take 
> the warnings
> patch :-)
> 
> Thanks Kevin.
> 
> Patrick
> -- 
> ----------------------------------------------------------
> |   Patrick Geoffray, Ph.D.      [email protected] 
> |   Myricom, Inc.                http://www.myri.com
> |   Cell:  865-389-8852          685 Emory Valley Rd (B)
> |   Phone: 626-821-5555          Oak Ridge, TN 37830
> ----------------------------------------------------------
>