gm_allsize shared memory performance

We are trying to resolve an issue regarding interprocess communication
using MPICH_GM (mpich-1.2..8) on our CSPI PPC (G4) distributed memory (not
SMP) architecture.  We are running multiple processes per node using unique
ports.  We are also testing using Apple PowerMac G4 (Yellow Dog Linux) with
the same results.  We are getting very poor benchmarking results when two
processes (on the same physical node) communicate with each other.  We have
been doing further benchmarking with gm_allsize (gm-1.4 and linux kernel
2.2.17) and have verified that communication between two processes on the
same node get much less performance (100-1000x) than between separate nodes
(typical Myrinet 1280 legacy performance).  Working with Brian Waite (CSPI)
and Tim Scott (Xontech) to try to resolve.  We have experimented as
instructed with the following
--disable-directcopy
--blocking
#define RECV_TOKENS_OF_SIZE 200
#define SEND_BUFFERS 20

Is this a shared memory or SMP issue that is improved with linux kernel
2.4.X, or are we discovering some thrashing issue in GM?  If this gets
resolved in GM, are there any issues regarding MPICH to use these
improvements (direct shared memory copying).  How shall we proceed or
assist in resolution?  Thanks.

Steve Prause
CSP Inc.


Process-to-process communication using GM (not MPI-GM)

gm_allsize -p 2 -d 4 --slave &
gm_allsize -p 4 -d 2 -u -bw -h yourhostname --geometric

HOST only         Port2 <-> Port4       ~ 0.9 MB/sec  (3804 Byte messages)

One Node only   Port2 <-> Port4       ~ 0.08 MB/sec

Node to Node    P2 <-> P2               ~ 80 MB/sec

Node to Node      P2 <-> P4              ~ 20 MB/sec
with 2 processes   P4 <-> P2