Re: [myrinet] Re: Why DMA instead of PIO in GM ?

Rolf Riesen wrote:

>   > geoge_zhang   I've seen in GM document that GM uses DMA to
>   > geoge_zhang   transfer messages from user space to board SRAM.
>   > geoge_zhang   As is known that PIO transfer for small messages
>   > geoge_zhang   achieves much better performance than DMA, then why

> Not to defend GM (I still think a hardware company should not
> write software ;-)

That's an interesting statement.
 
Myricom was a pure hardware company at the beginning, and a lot of 
people were surprised to receive hardware and no software.
If you sell hardware only, you need to have at least another company 
to write the software. You cannot ask customers to write the drivers 
and the middlewares.
Some will do, because they are academic or US gvt (where money and 
time are fuzzy constraints), but you cannot have a market with only 
this type of customers. And very often, the software produced is 
not optimal, very limited and not supported. 

The software group at Myricom has increased a lot lately, with more 
people for testing, documentation, support, etc. And it's normal. 
If you visit Cisco or 3Com or any hardware vendor, you will see 
more people working for software than pure hardware.

GM works well because it's done by Myricom. What's better input 
than talking to the chip designer next door ? Who is more qualified 
to write MCP code than the guy who designed the micro-code of the 
LANai and wrote the compiler ? You will never have this type of 
symbiosis if you separate hardware and software.

Sorry if you sense some passion in my speech, it's understandable as 
a software guy at Myricom :-)



To come back on PIO vs DMA, I think both need to be used. PIO is 
the only way to achieve low latency. Yes, it wastes CPU cycles but 
I know a lot of people that would accept it to gain 2 us (even if 
it's completely useless for 90% of their application).

Why GM is still using DMA for everything ? Well, for various 
reason. The first one is that the MCP is doing a lot of things, 
to insure reliability (retransmission), fairness, etc. A lot 
of firmwares out there do not provide this level of service. 
In the case of BIP, where the code path in the MCP is minimal, 
the cost of starting a DMA is important regarding the total 
time spent in the MCP. For GM, the gain would be lower, because 
the time spent to do the DMA is overlaped by other processing. 
The second one is that Glenn (the "G" in GM) tried it a few years 
ago and it was not improving much. At that time, CPU cyles were 
still expensive, so DMA-only was the way to go.
These days, CPU cycles are cheaper, and LANai cycles are much 
cheaper, so it would make sense to try PIO again.

The fact, as Rolf noticed rightly, is that the few microseconds 
you will gain will certainely be useless, because the software 
on top of it will waste it. So it's not worth it.
I prefer to see the GM development efforts working to support 
efficiently the message passing semantic, the only way to 
really uses the Myrinet technology at the right capacity. 
You will see it in Embedded MPI, but you will have to wait 
a few more months for that.

Regards

Patrick

-----------------------------------------------------------
|   Patrick Geoffray, Ph.D.      [email protected]         |
|   Myricom, Inc.                http://www.myri.com      |
|   Cell: 865-389-8852           685 Emory Valley Rd (B)  |
|   Fax:  865-425-0978           Oak Ridge, TN 37830      |
-----------------------------------------------------------