Re: [myrinet] RE: GPSHMEM and Myrinet (one-sided communication)

Greg Lindahl wrote:
> 
> > It is based on Dr. Jarek Nieplocha's ARMCI one-sided
> > library which is rather robust and supports myrinet.
> 
> Right. So either (1) it doesn't use GM, since it's been around for a while
> and can't possibly be using the not-yet-released GM with "get", or (2) it
> uses 2-sided messaging to implement 1-sided messaging, which can be done but
> is usually slow. I couldn't tell from the one paper I read about it in.
> 
> I have a subset prototype SHMEM library based on GM. Since only "put" is in
> the production version of GM, the subset is pretty obvious: no get, no
> atomic ops. There are some subtle aspects to implementing things correctly.
> 
> -- g


Greg,

ARMCI (and therefore GPSHMEM) does support get and put over GM
but it has some built-in assumptions and limitations that correspond
to what ARMCI was designed for. Its limitations and resource 
consumption include the following:

1. It requires MPICH/GM for the process initialization and startup
   (after MPI_Initialize, one needs to call ARMCI_Init collectively
    to initialize the library)
2. It uses an extra thread per each SMP node and two extra GM ports
   irregardless of how many tasks are running on this node and
   in addition to what MPICH/GM is consuming already. For example,
   with 2 tasks per SMP node, MPICH/GM uses 2 ports and ARMCI takes 2 
   which takes 4 total. With 1 task per node, the total port consumption 
   is 3.
3. It requires all MPI tasks on the SMP node to be consecutively numbered
   (this requirement can be avoided but it is there for simplicity
   reasons in the initial implementation)
4. It has been tested only on Linux x86 and sparc, and Solaris sparc.
   We are looking for access to other platforms (like Linux/alpha)
   with Myrinet to port it.

Other features include:
1. No need to preregister memory by the user.
   GM put (direct send) and likely get (Bob can you confirm it for get?)
   require memory to be registered. This is a significant constraint
   for a scientific apps that need one-sided communication model and
   requires a user-level library such as SHMEM to work around it
   (it is unrealistic to assume that large amounts of memory that
   apps use to store their data structures can be pinned statically
   for the duration of the program execution).
   Therefore, in ARMCI, memory accessed by 1-sided communication is 
   NOT required to be registered by the user in advance. The library 
   tries to do it dynamically and if that does not work (Solaris, 
   or shared memory in Linux) it will use an intermediate pinned 
   buffer and extra memcopy (which of course leads to a lower b/w).
2. All the memory exposed to remote tasks via 1-sided communication 
   must be allocated by a special function ARMCI_Malloc. It is
   similar to shmalloc in the Cray SHMEM or MPI_Alloc_mem in MPI-2. 
   This requirement is to allow ARMCI allocate shared memory on 
   the SMP nodes (put/get within the node map then to memcpy rather 
   than to the GM calls).
3. There is no fortran interface in ARMCI, GPSHMEM mentioned by Ricky 
   can be used as one.

Regarding the performance, it is only partially optimized for now.
Quoting numbers from a recent paper we wrote, for example latency
of get operation in the interrupt mode (i.e., remote task is 
not polling for incoming requests) in Linux/x86, 500MHz PIII, 
32-bit Myrinet cards, is 75 uS, and b/w 73.5MB/s. This compares 
to 101.8MB/s b/w reported by gm_allsize (one way) and 45.8 MB/s 
round trip on the same system. gm_allsize uses preregistered 
buffers while ARMCI numbers include all the overhead associated 
with registering or extra copies etc needed to work for any normal 
(non-registered) memory buffers used by the application. 
The MPICH-GM roundtrip send-receive latency on the same system 
is 30uS, and of course it is a two-sided protocol with both sides 
receiving messages in the polling mode. Under Solaris, b/w 
in ARMCI is worse as the OS/GM does not allow to register memory.

The GM port of ARMCI is fairly new and only recently real apps
like NWChem http://www.emsl.pnl.gov:2080/docs/nwchem
started using it. However, we are serious about this project a
and will be optimizing and supporting this implementation.
The URL for the ARMCI project is
http://www.emsl.pnl.gov:2080/docs/parsoft/armci

Hope it helps to answer at least some of your questions about 
the GPSHMEM implementation.

Jarek