communication pattern causes corrupted data

Hello,

We've uncovered an anomaly with mpich/gm that occurs when certain
communication patterns occur in a code.  The problem does not exist when
using the p4 device.  So far I've verified the problem exists with the
following configurations:

**** THE CONFIG ****

RedHat Alpha linux 7.1
gm-1.6.3 / mpich-1.2.5..9, mpich-1.2.4..8a, mpich-1.2.1..7
gm-1.5.2.1 / mpich-1.2.5..9, mpich-1.2.4..8a, mpich-1.2.1..7

The problem does not occur with:
RedHat Alpha linux 6.2
gm-1.4 / mpich-1.2.1..5

I'm trying to get the gm-1.4/mpich-1.2.1..5 combo working under RH7.1
but having issues so far.  I'll also be testing if the problem occurs on
x86 platforms.


**** THE PROBLEM ****

To my understanding, with certain communication patterns (ie: buffer
sizes sent in X-node runs), the last 4 bytes of the buffer is often not
received.  When sending an array of integers then, the last integer
becomes corrupted.  Changing the buffer sizes make the problem go away.

I've included a test program that demonstrates this.  Basically, it
reads a description of a communication pattern that sometimes
(but not always) fails and performs that communication
over and over and over.  Most times, the communication works fine, but
occasionally, one integer is not correctly received.  This error is
seen in the test program as a -2 in the receive buffer after
communication is done.  An error statement is printed whenever the error
occurs.  Note the program is set up to run across 16 processors.


I'd certainly appreciate some help at this point.  It took over a month
to reduce the problem to this test program and no one here is familiar
with gm/mpich-gm internals enough to effectively debug any further.

Please feel free to contact me to discuss the problem further.

Thank you,

-Josh
-----------------------------------------------
Josh England
Sandia National Laboratory, Livermore, CA
High Performance Computing and Networking
email: [email protected]
phone: (925) 294-2076