Re: [Myrinet] (No Subject)

Salut Julien,

TOUCHE Julien wrote:
> so register don't allocate anything, but why use gm_malloc instead 
> of gm_dma_malloc ? or can i use both of us ?

If you want to send/recv from/to a buffer in memory, the pages have to 
be locked and the virtual/physical translation prepared in the kernel 
DMA-able memory). That is done via 2 ways:

* If the memory is no yet allocated, gm_dma_malloc will allocate memory 
and make it DMA-able. This is implementated in many drivers by 
(malloc+register) inside.
* If the memory is already allocated (static arrays, allocated with 
malloc but the user has no freedom to change the malloc in 
gm_dma_malloc, MPI where there is no notion of registration, etc), 
gm_register_memory will make this piece of memory DMA-able.

You can mix this 2 methods in the same code, but it's nonsense to 
gm_dma_malloc a buffer and register it afterwards.

This is not the best document in the world, but all of this is described 
in the GM' API document: http://www.myri.com/scs/GM/doc/gm_toc.html

>>* The value of remote_address is uninitialized.
>
> what do you mean by uninitialized. i could put a NULL at var creation,
> but remote_address is to get the distant memory address so it's the
> receive which must initialized it.

No, remote_address in the gm_directed_send is the virtual address of the 
remote buffer on the receiving side. If it's NULL of something 
undefined, you will try to remote write to the address NULL or something 
undefined, in both cases it won't work :-)
remote_address must contains the address of the remote buffer, that you 
previously received from the other side in a message, or via another way.

> to receive the message from gm_directed_send(), which function must
> i use ? gm_receive as for gm_send ?

Nothing. It's a "one-sided" communication, the sender sends directely to 
a remote address. The remote buffer should be DMA-able (gm_dma_malloc or 
gm_register_memory) and that's all. The receving side doesn't need to 
provide it as a receive buffer and won't be notified that someone as 
"PUT" something in it's memory.


>>* you didn't understand the notion of Sizes in GM. I don't blame you, 
>>it's one of the very bad parts of the API, it took me months to (almost) 
>>understand what it is. It's like a tag with a relationship with the 
>>length of the message, insane. Anyway, think about it like a tag with a 
>>value between 8 and 31. The relationship with the length has been 
>>relaxed a long time ago actually.
>
> not much clear ;)
> for a char[size], which length ?
>             size*sizeof(char)
> for a int[size] <-> size*sizeof(int) or size ?

I know it's not clear. Believe me I told several times the GM guy that 
it's insane :-) Anyway, look at the definition of sizes 
in:http://www.myri.com/scs/GM/doc/gm_4.html#SEC4

How do I use the GM sizes ? I pick a number between 8 and 31, and use it 
like a Tag in MPI: messages in GM match with priority and size, like 
they match with (source, tag, communicator) in MPI.
So you can use the same GM size for all send/recv, or you can
differentiate them with another Size.


>>I looked at it very quickly and the buffer "message" is size_mem 
>>characters, but the MPI operations send/recv size_mem integers (MPI_INT).

> for a char[size], the type is what (MPI_CHAR) ?

Yes.

> i don't find much docs on MPI except the one with MPICH but it's not a real guide to programming MPI.

Read http://www.mpi-forum.org/docs/mpi-11-html/mpi-report.html, it's 
quite well written :-) Tres pedagogique.

> For gm, the myrinet doc is good and the example helps but is not 
> sufficient (i don't understand all)(not a real guide programming 
> too ;)

If you want my opinion, the GM's API and its doc sucks both. But when 
you work-around the traps, it's usefull :-)

Patrick

----------------------------------------------------------
|   Patrick Geoffray, Ph.D.      [email protected]
|   Myricom, Inc.                http://www.myri.com
|   Cell:  865-389-8852          685 Emory Valley Rd (B)
|   Phone: 865-425-0978          Oak Ridge, TN 37830
----------------------------------------------------------