Re: mpich-gm1.2.5..10 and PBSpro

Hi James,

On Tue, 2003-10-14 at 19:38, James Lowey wrote:
> I am having some problems getting mpich-gm and pbspro to work correctly
> together.
> If I submit a sample job using pbspro I get the following output:
> 
> Ignoring message from the MPI Id 0 (./cpi) !

That means that the mpirun.ch_gm.pl script received a initialization
message from the MPI process of rank 0 via socket but it already 
received previously a init message coming from the same process.

> This will go on for 1000's and 1000's of lines then
> you will see some of these messages:

1000's of lines ? That means someone is sending these init message to
the mpirun.ch_gm.pl over and over, as the socket is closed after each
one.

> Received spurious abort message, keep listening...

It received a message after everybody is configured, but it's not an
abort message (which is the only type of message you can receive at that
time).

> Until the job is killed.  The job will run successfully but it appears
> that it outputs every

You have a rogue process that keeps sending the same init message over
and over to the mpirun.ch_gm.pl script. Can you add the flag "-v" to
have more info ? Also, could you run tcpdump during the job to see who
is sending these messages ? No MPICH-GM process should send anything
after the MPI job has started. So if you spawn an MPI code that sleeps
after printing something, and you see all of them reaching the sleep,
and you see someone sending to the mpirun.ch_gm.pl script, this is man.

> message that is there.  We came up with a workaround by modifying the
> mpirun.ch_gm script 
> setting warnings to 0 but when you submit a job via PBSpro the head mom

How did you set the warnings to 0 ?

> node shows a perl process from mpich-gm consuming a tremendous amount of
> processor resources. So the "fix" only masks the problem, it does not
> solve it.

The root cause should be found, this is not normal. BTW, does it work
without PBSpro, ie using mpirun.ch_gm by hand ?

Thanks

Patrick
-- 

Patrick Geoffray
Myricom, Inc.
http://www.myri.com