Re: [GE users] RE: Myrinet setup (was: RE: [Beowulf])

Thanks Patrick...

Also forwarding Ron's reply to this list.

Rayson


>The original SGE-Myrinet integration was written by
>Shannon in 2002, and I believe it needs to write to a
>common location to get the host/GM port info.
>
>Newer MPICH-GM versions automatically allocate the
>required number of GM ports on each of the nodes, and
>thus SGE doesn't need to list the port in the machine
>file.
>
>Myrinet (GM-2) with SGE6
>========================
>1) GM2 has 16 ports (vs. 8 with GM-1) available per
>NIC, and is usually more than the number of processors
>on each node. If the setup is like this, then just
>setup a normal tight-MPICH PE should be fine. (unless
>your users DoS the cluster by allocating a large
>number of GM-ports)
>
>Make sure you read Reuti's "Tight Integration of MPICH
>and SGE" HOWTO.
>
>
>2) If the nodes have more CPUs than the number of
>Myrinet ports, and if you want to use serial jobs to
>backfill the cluster, then you need to setup GM-port
>as a resource and also a loadsensor to report the
>number of free GM-ports available. (use gm_board_info
>or gm_yp)
>
>Jobs then request for a GM-port (SGE will multiplie
>that with the number of SLOTS) and the SGE scheduler
>will pick nodes with the needed number of CPUs and
>GM-ports available to run the parallel job.
>
> -Ron
>
>
>--- Reuti <[email protected]> wrote:
>> in principle you can have $TMPDIR on a file server.
>> The 65534 was the 
>> nobody/nogroup as the root_squash to the
>> anonuid/anongid is the default for 
>> exports. But instead of trimming the file server to
>> support it in a proper way, 
>> it's safer (and faster) to have it locally on the
>> nodes as you will have it 
>> now. 
>> 
>> CU - Reuti
>> 
>> Quoting William Burke <[email protected]>:
>> 
>> > Hi Reuti,
>> > 
>> > Quoting Reuti [[email protected]]:
>> > 
>> > <snip>
>> > > First one additional question (before I route
>> you in the wrong
>> > direction):
>> > 
>> > > is it necessary for you to have a shared $TMPDIR
>> for SGE? ...More common 
>> > > and faster is to use the local /tmp on the nodes
>> for this
>> > 
>> > Actually there is no reason that I should have a
>> shared $TMPDIR for SGE,
>> > except that I read some where (I am not sure
>> where)that it was recommended
>> > to share the $TMPDIR. However I just seached the
>> SGE and N1GE docs I found
>> > no evidence supporting that notion and now
>> thinking about it setting
>> > $TMPDIR
>> > to local /tmp would simplfy things. Thxs
>> > 
>> > Regards,
>> > William
>> > 
>> > -----Original Message-----
>> > From: Reuti [mailto:[email protected]] 
>> > Sent: Sunday, March 27, 2005 7:09 AM
>> > To: William Burke
>> > Cc: [email protected];
>> [email protected]; [email protected];
>> > 'John Hearns'
>> > Subject: Myrinet setup (was: RE: [Beowulf])
>> > 
>> > Hi Will,
>> > 
>> > Quoting William Burke <[email protected]>:
>> > 
>> > <snip>
>> > > I have but I do not see my name yet? How long is
>> the verification
>> > process?
>> > 
>> > did you register as an observer - AFAIK you can
>> post to the SGE list
>> > without
>> > 
>> > being registered.
>> > 
>> > <snip>
>> > > The mpi directory's mpich.template doesn't use
>> mpirun.ch_gm so how does
>> > it
>> > > know what version of mpirun to use? If I use the
>> mpi what changes do I
>> > have
>> > > to make?
>> > 
>> > You can have more than one MPI implementation
>> installed in your cluster,
>> > and
>> > it 
>> > may need some planning to set up the correct $PATH
>> for each of the 
>> > implementations you want to use (and the this way
>> located mpirun must fit
>> > to
>> > 
>> > your used version of MPI during compilation of
>> your program). You may use a
>> > 
>> > "which mpirun" to check it in your job script.
>> > 
>> > Also the supplied sge_mpirun will not use any
>> Myrinet version on it's own -
>> > 
>> > it's just a wrapper to the mpirun you set in the
>> PE, so that you don't have
>> > to 
>> > specify the usual options 'mpirun -machinefile
>> $TMPDIR/machines -np $NSLOTS
>> > 
>> > mypgm'.
>> > 
>> > I must admit: seems that the Myrinet stuff was
>> more for 5.3 and not
>> > updated,
>> > as 
>> > in 6.0 you can have more than one line for
>> "start_proc_args" in your PE 
>> > definition - so it just grabs the last \ in the
>> first and only line
>> > beginning 
>> > with "start_proc_args" line as mpirun command -
>> will give the error message
>> > you 
>> > got, that "\" is not existing.
>> > 
>> > As I said: we can use the default MPICH
>> integration also for Myrinet and 
>> > proceed this way.
>> > 
>> > > >> Can you please give more details of your
>> queue and PE setup (qconf
>> > > -sq/sp
>> > 
>> > Thx, I will keep the stuff. First one additional
>> question (before I route
>> > you 
>> > in the wrong direction): is it necessary for you
>> to have a shared $TMPDIR
>> > for 
>> > SGE? This is the one you set in your queue
>> configuration (tmpdir 
>> > /WEMS/grid/tmp) and seems for now to be on a file
>> server.
>> > 
>> > More common and faster is to use the local /tmp on
>> the nodes for this (you
>> > are 
>> > right: SGE want to create there a directory for
>> this job and some file for
>> > its 
>> > own usage - but you are free to use this directory
>> $TMPDIR also in your job
>> > 
>> > script). It will be created for your job, and
>> cleanly deleted after the
>> > job,
>> > so 
>> > you won't have any leftover files.
>> > 
>> > Cheers - Reuti
>> > 
>> 
>> 
>> 
>>
>---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> [email protected]
>> For additional commands, e-mail:
>> [email protected]
>> 
>> 
>
>
>
>
>		
>__________________________________ 
>Do you Yahoo!? 
>Yahoo! Small Business - Try our new resources site!
>http://smallbusiness.yahoo.com/resources/ 
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: [email protected]
>For additional commands, e-mail: [email protected]
>
>
---------------------------------------------------------
Get your FREE E-mail account at http://www.eseenet.com !