SMP performance

Hi,

We have a cluster of pentium III's (four boxes, each dual processor)
running RedHat Linux (kernel 2.4). We have been looking at SMP performance
on one box. We originally expected to find that the performance using a
ping-pong test to be independent of the MPI implementation we used, but
this is not the case.

 i) MPICH using the ch_p4 device and comm=shared delivers a peak data
    transfer rate of 255MB/s on messages of size 32KB. For large messages
    we get an assymptotic transfer rate of 115MB/s.

ii) MPICH running over GM delivers a peak data transfer rate of 977MB/s on
    messages of size 32KB. For large messages we get an assymptotic
    transfer rate of 233MB/s.

Thus there is a factor of about 4 in the peak rates, and of about 2 in
assymptotic rates.

The memory bus has a clock speed of 133MHz, so we should be able to get
about 1064MB/s transfer rate between one processor and memory. This
correlates with the MPICH/GM peak result, suggesting data is being written
from cache on one processor into shared memory readable by the other.
Larger messages cannot fit into L1 cache (L1=16K, and L2=256K), so degrade
performance.

The peak rate for the MPICH/ch_p4, and the assymptotic rate for MPICH/GM,
suggest that they both require 4 memory reads/writes.

The assymptotic MPICH/ch_p4 rate corresponds to 8 reads/writes.

Can anyone explain these results?

Cheers,

Andrew