Re: [myrinet] Re: [Myricom help #8219] Fwd: Registered memory in MPICH over GM

Bill,
 I have attached a program that measures GM (de)registration time over a
loop. I used this a while back -- hopefully it still compiles :) and is
useful to you.

Ecap
 

 Srigurunath Chakravarthi (E-cap), MPI Software Technology India.
 Email: [email protected], [email protected]
 Phone (India): Work: +91-80-6346485, 6557998 Home: +91-80-6667423, 6662288

On Tue, 13 Nov 2001, Bill Allard wrote:

> Thanx, Patrick!  It seemed to me that MPICH-GM would <not> register memory
> each time a send or receive is issued and that some sort of caching
> mechanism would be necessary to avoid this.  This is what you affirmed and
> you even told me what source file to check it out in.
> 
> In my own stuff, for the time being, I register a big block at startup,
> never register it again, and manage this block as needed.  It would be
> nice not to grab a big block in the beginning.  In order to know how to
> deal with all this, one needs to know just how expensive it is to register
> memory.  Is there a short answer to this?  If push comes to shove I can
> experiment but life is short...
> 
> William K. Allard
> Professor of Mathematics
> Box 90320
> Durham, NC 27708-0320
> (919) 660-2861 Fax:(919) 660-2864
> 
> On Mon, 12 Nov 2001, Patrick Geoffray wrote:
> 
> > Karen Wang wrote:
> >
> > > > I have the following question.  If you call MPI_Send or MPI_Receive, are
> > > > the respective buffers copied into DMA-able memory with
> > > > gm_register_memory or is it assumed that the buffers are already
> > > > gm_registered?
> > >
> > > GM send/receive functions need to operate memory that is DMAable.
> > > It is user's responsibility to prepare the buffer to be DMAable
> > > before calling GM send/receive functions.
> >
> > I am sure Bill knows that if his GM code works :-)
> >
> > > In your case, you will need to implement MPI_Send/MPI_Receive using
> > > GM functions.
> >
> > No, the implementation is already done in MPICH-GM. I would say this
> > is even the main interest of MPICH-GM :-))
> >
> > Seriously, to reply to the question: it depends upon the size
> > of the message.
> > For the Eager protocol (up to 16 KB), there is always a copy
> > on both sides, because it's cheap to copy (to and from a pre-allocated
> > registered area) and you need a non-blocking behaviour at the MPI level
> > (the send immediately returns, even if the receive is not posted)
> >
> > For large messages, the Rendez-vous protocole uses gm_directed_send()s
> > with a synchronization between sender and receiver. In this case, the
> > application buffer on both sides have to be registered (zero-copy),
> > but it's quite expensive to register/unregister each time. So we use
> > a registration cache in MPICH-GM (file mpid/ch_gm/regcache.c) to keep
> > memory registered. It will unregister by large blocks when you use
> > all of the amount of DMA-able memory (80% of the RAM with GM-1.5) or
> > when you free() a malloc()ed buffer.
> >
> > For a description of the Eager and Rendez-vous protocols:
> >    http://www.myri.com/scs/GM_FAQ.html#mpich21
> >
> > Hope it helps.
> >
> > Patrick
> >
> > ----------------------------------------------------------
> > |   Patrick Geoffray, Ph.D.      [email protected]
> > |   Myricom, Inc.                http://www.myri.com
> > |   Cell:  865-389-8852          685 Emory Valley Rd (B)
> > |   Phone: 865-425-0978          Oak Ridge, TN 37830
> > ----------------------------------------------------------
> >
>

/* Program to measure registration and deregistration of memory for GM */
/* Author: [email protected] */

#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>
#include <unistd.h>
#include <limits.h>

#include "gm.h"

#define maxDataSize	(8 * 1024 * 1024)
#define MILLION		(1.0e+6)
#define DMAX		(double) (INT_MAX)

int ITERS = 100;

int GetNextSize(int DataSize)
{
  if(DataSize == 0)
    return 4;
  else
    return DataSize * 2;
}

double GetTimeInSec(struct timeval tv)
{
  return ( (double)(tv.tv_sec) + (double)(tv.tv_usec)*1e-6 ) ;
}

int
main(int argc, char **argv)
{
  struct gm_port *p;
  unsigned char *buffer;
  gm_status_t status;
  unsigned int times;
  long dataSize;
  struct timeval start, middle, end;
  double start_sec, middle_sec, end_sec;
  double reg_time, dereg_time, total_reg_time=0, total_dereg_time=0;
  double min_reg_time = DMAX, min_dereg_time = DMAX;
  double max_reg_time = 0.0, max_dereg_time = 0.0;


  if (gm_open(&p, 0, 2, "gm_time_reg", GM_API_VERSION_1_0) != GM_SUCCESS) {
    printf("Couldn't open port 'p1' = %d\n",2);
    exit(-1);
  }
  

  if (argc>1) { /* Use the argument as num times to average over */
    ITERS = atoi(argv[1]);
  }

  printf("%s: Averaging over %5d iterations...\n", argv[0], ITERS);
  printf("Buffer\t\tgm_register_memory\t\tgm_deregister_memory\n");
  printf("Length\t\t[min]  [avg] [max]\t\t[min]  [avg] [max]\n");
  printf("[Bytes]\t\t\t[usec]\t\t\t[usec]\n");
  fflush(stdout);


  for(dataSize = 1; dataSize <= maxDataSize;
      dataSize = GetNextSize(dataSize))
    {

      total_dereg_time = total_reg_time = 0;
      min_reg_time = min_dereg_time = DMAX;
      max_reg_time = max_dereg_time = 0.0;
      
      for (times=0; times < ITERS; times++) {

	buffer = malloc(dataSize);

	if (!buffer) {
	  printf("Couldn't allocate a buffer\n");
	  gm_close(p);
	}

	gettimeofday(&start, NULL);

	status = gm_register_memory(p,buffer,dataSize);

	if (status != GM_SUCCESS) {
	  printf("Couldn't register memory. times=%d. dataSize=%ld\n",
		 times, dataSize);
	  gm_close(p);
	  exit(-1);
	}

	gettimeofday(&middle, NULL);

	status = gm_deregister_memory(p,buffer,dataSize);
	if (status != GM_SUCCESS) {
	  printf("Couldn't deregister memory. times=%d. dataSize=%ld\n",
		 times, dataSize);
	  gm_close(p);
	  exit(-1);
	}

	gettimeofday(&end, NULL);


	start_sec = GetTimeInSec(start);
	middle_sec = GetTimeInSec(middle);
	end_sec = GetTimeInSec(end);

	reg_time = middle_sec - start_sec;
	dereg_time = end_sec - middle_sec;

	/* Record min and max */
	if (reg_time < min_reg_time)
	  min_reg_time = reg_time;

	if (dereg_time < min_dereg_time)
	  min_dereg_time = dereg_time;

	if (reg_time > max_reg_time)
	  max_reg_time = reg_time;

	if (dereg_time > max_dereg_time)
	  max_dereg_time = dereg_time;

	/* Compute total  time */
	total_reg_time += reg_time;
	total_dereg_time += dereg_time;

	free(buffer);
      } /* for ITERS */

      printf("%10ld    %8.2f %8.2f %8.2f    %8.2f %8.2f %8.2f\n",
	     dataSize,
	     min_reg_time*MILLION,
	     ((total_reg_time/ITERS)*MILLION), /* avg */
	     max_reg_time*MILLION,
	     min_dereg_time*MILLION,
	     ((total_dereg_time/ITERS)*MILLION), /* avg */
	     max_dereg_time*MILLION);

      fflush(stdout);
	
    } /* for dataSize */

  gm_close(p);
}