Re: MPICH-GM in kernel space?

On 6/21/05, Loic Prylli <[email protected]> wrote:
> Ashish Bijlani wrote:
> > Hi People,
> >
> > Does MPICH-GM implementation exist as a kernel module?
>
> No. Out of curiosity could you detail for what purposes you would need su=
ch a
> kernel module.
Sure. I have an user space simulation needing a high speed distributed
compute engine. Its just a toy simulation that uses MPI and I was
wondering what sort of performance bottlenecks exist in MPICH-GM (if
any) since I've been told that it "bypasses" the kernel.

My viewpoint was that the kernel is the dude in charge and the only
way an user space application could "bypass" the kernel to access a
device (closest analogy I could draw to an user space app using the
myricom card) would be mmap'ing the device's registers into its memory
and proceeding this way.

Am I correct in assuming that there is in fact an mmap of the myricom
device's register set and the MPICH MPI implementation you speak of is
an user space library using these registers ?

> > If no, then is
> > it a good idea to implement it?
>
> Probably not. There would be major problems with such a port, including t=
he
> fact MPICH-GM does not provide the ability to report all kind of errors t=
o the
> MPI user (and aborting in the kernel is not a good idea), the configurati=
on of
> MPI_COMM_WORLD might not easy to adapt to kernel use.
>
> Moreoever if you want to use the MPI API in the kernel, I would suggest y=
ou
> base your work on MX rather than GM (see the announcement on the
> www.myri.com). It would probably be easier to implement minimal MPI=20
> wrappers around the MX API to provide the MPI subset you need rather than=
=20
> trying to port MPICH-MX. The basic MX communications primitives are very=
=20
> close the MPI ones. Probably the best solution would be to directly use t=
he=20
> MX API.

That sounds good, although I will need more information. What do folks
needing MPI on top of myricom cards use currrently ? Also, I read about
MX stuff on Myricom's homepage. It seems to be doing exactly what I=20
want. It has API in kernel space, true? How does application at top of
MX Kernel-API access/call the required MX API function? If this
happens through system calls, then doesn't this design hit
performance?

-Ashish