RE: [Myrinet] [Myricom help #13378] hybrid mpich support

On Mon, 2002-10-07 at 19:03, Van Maren, Kevin wrote:
> Patrick,
> 
> My use of "hybrid" was very unfortunate, and I think that may
> be leading to confusion.

I see. Indeed, I was referring to hybrid as one of the receive mode.
Your patch makes much more sense in this context. 

> run on (beyond the scope of mpirun -- I do it similar to running
> a program under gdb, with a wrapper program that reads the MPI
> env variables and does the "right thing"), and you need the machine

Are you sure that mpirun cannot help you there. mpirun.ch_gm is quite
open, we can add some specific support if needed.

> I'm not claming a hybrid approach is right in all (any?) circumstances,
> but it is something to experiment with, especially if you have a NUMA
> or multiple-processor-bus machine with a Myrinet interface per node.

The vast majority of our customers use single or dual CPU boxes, rarely
quad, but not NUMA. So we have never investigated this case where the
memory access is not uniform. A such hybrid configuration makes perfect
sense (it would even make much more sense if there would not be a memory
copy for small messages in MPICH-GM to deal with registration and GM):
using the NIC for local communications can be seen as an asynchronous
offloaded memory copy. In a perfect world, a such hybrid behavior would
be extended to different message sizes: shared memory for small
messages, even the cache-coherency overhead would be small compare to
the gain in latency, but Myrinet for large messages to avoid CPU waste
and cache trashing. However, I agree that a spacial partitioning is a
good first step.

I will look closely at your patch and including it before 1.2.4..8b.

Patrick
-- 
----------------------------------------------------------
|   Patrick Geoffray, Ph.D.      [email protected] 
|   Myricom, Inc.                http://www.myri.com
|   Cell:  865-389-8852          685 Emory Valley Rd (B)
|   Phone: 626-821-5555          Oak Ridge, TN 37830
----------------------------------------------------------