Re: Re: [GE users] Myrinet setup (was: RE: [Beowulf])

Hi Rayson,

Rayson Ho wrote:
> The main difference with myrinet is that Myrinet/GM has a limited number of
> "ports" on each card. You want to allocate free ports to your job on each
> host, not the used ones.

FWIW, MPICH-GM does the ports allocation itself, it should be 
transparent from the user (and from the scheduler) point of view. When a 
MPICH-GM process is spawned, it will try to open a GM port, the forst 
available on the first available NIC. When one port is found, it sends 
the port number, among other things, to the mpirun.ch_gm perl script. 
When all processes have reported their local information, mpirun.ch_gm 
distribute the whole knowledge to all of the processes so they can speak 
to each other through GM.

Usually, the batch queue system does not need to know about that except if:

* you want to be sure that you never run out of GM ports: the default GM 
build provides 16 ports, 13 of them are useable for MPICH-GM processes. 
So, the maximum number of MPICH-GM processes running on one machine 
(assuming one NIC per machine, the limit is per NIC actually) at one 
time is limited to 13 with a default build. Unless you have a machine 
with more than 13 processors (in this case you would want to have 
several NICs anyway) or if you happen to spawn a lot more processes than 
processors (very uncommon in HPC), you don't need to care.

* the batch queue is not using mpirun.ch_gm and spawn processes 
directly. That's the case of mpiexec (http://www.osc.edu/~pw/mpiexec/) 
for example. In this case, the spawner should duplicate the protocol 
used by mpirun.ch_gm. This protocol is not explicitely documented, but 
it's trivial and the source is open. It used to changes slighlty between 
releases of MPICH-GM, but it has been stable for a while.

> (The program you need to use is gm_board_info, which will tell you which
> ones are free. But, if you don't have interactive users, you don't really
> care about ports get allocated outside of SGE, and can use a hardcoded
> value.)

Using hardcoded values or using gm_board_info (or gm_yp on GM-2) is a 
kludge. Supporting the mpirun.ch_gm information exchange mechanism is 
much easier/cleaner.

> Lastly, it would be nice if Myricom can do the integration themselves, or
> at least supply the batch system developers with hardware. (I requested
> this many years ago!!!)

We cannot ship free hardware to all batch queue developers for various 
reasons, but we can help with access to Myricom developers and to 
developement clusters for test (all software is already open source).

Patrick
-- 

Patrick Geoffray
Myricom, Inc.
http://www.myri.com