Re: [myrinet] collective comms w/ SMP nodes

sorry if you receive multiple copies, I didn't see the last
tries...

On Mon, 7 Aug 2000, Matthew Barone wrote:

> We are considering a cluster configuration with dual Pentium III nodes
> connected by Myrinet.  Has anyone looked at the performance of
> collective communication routines (all-to-all, etc.) using MPICH on
> Myrinet with/without  SMP support?  It seems that collective
> communications would be slow when multiple processors are sharing a PCI
> bus.

Yes, it would be a little bit.

At this time, the collective communications of MPI-GM are the
generic
MPICH collectives coms. That means it's slow and not optimized at
all
(compare to the performance of Myrinet).

There's 3 points :
* 1) The generic code of MPICH is simple and the spanning tree
used can be
improved for Myrinet (very small gap in the LogP model). This is
common to
all of the MPICH-based codes.
* 2) Using two devices (GM-Myrinet and Shared-memory/SMP) is a
problem for
collectives coms as you have virtually two networks with
differents
parameters and you share the PCI bus.
* 3) The right place for collective coms is in the Myrinet board,
not in
the host. Myrinet gives the flexibility to do it.

1) and 2) will be fixed in a short-medium term : i was talking
yesterday
with Thielo Kielmann and it will be a good idea to use MagPie in
MPI-GM.
That will provide a much better collectives coms support than
MPICH
generic code and support the SMP device.
3) needs to be implemented in a medium-long term as we plan to
move to the
Lanai some MPI code, like hand-shaking protocol, etc.

A quick solution would be to write carefully the .gmpi/conf file.
The
spanning tree used in MPICH is hypercube-based :

0->8
 ->4
    ->6
       ->7
    ->5
 ->2
    ->3
 ->1

So, the good mapping is to have the SMP processes on the same
branch of
the tree, without any children, like :
MPI 0 and MPI 1;
MPI 2 and MPI 3;
MPI 4 and MPI 5;
etc.

If you write your conf file in this way :
node1 4
node1 5
node2 4
node2 5
node3 4
node3 5

you would not be too bad :-)

I have no idea about the delay to have MagPie in MPI-GM or a
production quality Lanai-embedded MPI. 

Patrick