Re: Best queue system for myrinet?

Bill Broadley wrote:
> I'm reinstalling a 32 node (64 cpu) myrinet cluster.
> 
> I'm looking for a batch queue system, but want to avoid the problem
> I've had in the past (although this info maybe somewhat dated).
> 
> Basically I want to schedule my cluster as 64 cpu's, but if I
> tell the queue system I have more then 32 cpu it needs to know
> that the first job on a given node gets Myrinet channel 2, and the second 
> job on the same node gets Myrinet channel 4.
> 
> What is working well for people?
> 

We have been doing some batch queue testing with ESP2, becasue we are 
greatly disappointed with the reliability (and ability of) OpenPBS.


We've tested to date the following:

1) OpenPBS w/Maui
2) SGE (free edition)
3) Platform's LSF
4) Maui/ME

The problems of each:

1) Hangs; has tendency to put a job in a wierd state, that sends 
thousands of email messages saying so; only way to clear is stop the 
master, and delete the job file by hand.

2) Scheduler doesn't work very well - wants to run all the small jobs 
first, priorities only apply after jobs are sorted by size.

3) Can be expensive.  BUT, we got it perform just as well as OpenPBS 
w/Maui.  Very configurable; even has a plugin scheduler with Maui 
scheduler support.

4) If you can get to even install and run correctly, please send me 
email - we'd like to know what you did.

We also looked for other schedulers with MPI support - this is all we 
could find.  Again, if you know of a scheduler not on this, please email 
us (with a pointer to the section of documentation that states this), 
and we will add it to the test.