Re: [myrinet] Why DMA instead of PIO in GM ?

  * geoge_zhang wrote on Fri Sep 28, 2001 21:34:34...
  > geoge_zhang Hello:
  > geoge_zhang   I've seen in GM document that GM uses DMA to
  > geoge_zhang   transfer messages from user space to board SRAM.
  > geoge_zhang   As is known that PIO transfer for small messages
  > geoge_zhang   achieves much better performance than DMA, then why
  > geoge_zhang   GM uses DMA for all messages ? Compared with GM,
  > geoge_zhang   many other user-level communication protocols such
  > geoge_zhang   as BIP use PIO for small messages and DMA for bigger
  > geoge_zhang   ones, and so it achieves less latency for small
  > geoge_zhang   messages (about 4.3us for BIP, and 9.6us for GM )
  > geoge_zhang   and better bandwidth for bigger ones. Then what's
  > geoge_zhang   the consideration of GM just using DMA ? Any reply
  > geoge_zhang   would be appreciated !
  > geoge_zhang
  > geoge_zhang   Best regards Geoge.

Not to defend GM (I still think a hardware company should not
write software ;-) but simply looking at point-to-point latency and
bandwidth tells you very little about how scalable and efficient
a message passing layer is.

In the PIO vs. DMA debate you also need to consider that you are
devoting the host CPU to doing such a mundane task as pushing bits
across the PCI bus, while it could be doing something important,
such as multiplying floating-point numbers. Seriously, having
communication going on in the background while the host CPU does
other tasks is very important. Of course, the message passing layer
must allow for that and not simply make the host CPU spin-wait for
the DMA to complete. In that case you might as well have the host
CPU do the PIO.

Rolf


+++-+--+----+-------+------------+--------------------+------------------------
Rolf Riesen                              Email: [email protected] [email protected]
Scalable Computing Systems               [email protected]
Sandia National Laboratories             WWW:   http://www.cs.sandia.gov/~rolf
Org. 9223,  MS 1110                      Work:  (505) 845-7363
Albuquerque, NM  87185-1110              Fax:   (505) 845-7442