Re: MPICH-GM in kernel space?

Ashish Bijlani wrote:
> Am I correct in assuming that there is in fact an mmap of the myricom
> device's register set and the MPICH MPI implementation you speak of is
> an user space library using these registers ?
> 

Yes that's the case, part of the device is mmap'ed into the user-space 
process memory. There is some embedded memory on the Myricom device, 
as well as a small CPU running some firmware. And to be more accurate, 
rather than registers, it's part of that embedded device memory which 
is mapped in the user-space process, and is used by the user process 
to send requests to the firmware. Most of the registers and device 
memory are not visible to the user-space process for security reasons, 
all initialization is still done by the kernel. The kernel is bypassed 
only for communications requests, but not for setup.

> 
> That sounds good, although I will need more information. What do folks
> needing MPI on top of myricom cards use currrently ? 

A majority use MPICH-GM, altough we have recently released MX and 
MPICH-MX which are intended to progressively replace GM, so some 
people are also using MX. For people with MPI applications, only a 
recompilation should be needed to switch from one to another (or from 
or to any other MPI implementation for that matter).

> Also, I read about
> MX stuff on Myricom's homepage. It seems to be doing exactly what I 
> want. It has API in kernel space, true? 

For your purpose, you can consider MX and GM have the same overall 
design. Both uses kernel bypass. There is an MPI implementation for 
both. The MX kernel library is only intended for kernel programmers 
and in your case you should not use it. You should use the MX 
user-space library and MPICH-MX if you have a MPI application.

> How does application at top of
> MX Kernel-API access/call the required MX API function? If this
> happens through system calls, then doesn't this design hit
> performance?
> 

Same as GM, part of the Myricom device is mapped in user-space. No 
system calls happen for small-sized communications. And in all cases 
the user-data is never copied to kernel buffers for communications, 
but goes directly from user-space to the device.


Loic