Re: Opteron Vs. Athlon X2

On Thu, Dec 08, 2005 at 05:20:47PM -0800, Bryan J. Smith wrote:
> FYI, I was under the impression that each x1 channel is
> 0.125GBps bi-directional (0.25GBps effective), meaning an x8
> is 1.0GBps bi-directional (2.0GBps effective).

Your impression is wrong.  For a pretty reasonable explanation check:
	http://en.wikipedia.org/wiki/Pci-e

A few quotes:
  First-generation PCIe is often quoted to support a data-rate of 250
  MB/s in each direction, per lane. This figure is a calculation from the
  physical signalling-rate (2500 Mbaud) divided by the encoding overhead
  (10bits/byte.)
...

Note that the datarate already includes the overhead (10 bits per byte).

As for real world practical bandwidth:
  Like other high-speed serial interconnect systems, PCIe has significant
  protocol and processing overhead. Long continuous unidirectional
  transfers (such as those typical in high-performance storage
  controllers) can approach >95% of PCIe's raw (channel) data-rate.

Because the channel is full duplex it's easier to get a higher percentage
of the full bandwidth (unlike PCI-x.)

> They kick serious @$$ and prevent all that redundant load
> going back and forth from I/O to memory to CPU back to memory
> and down I/O again, just to do a XOR calculation (the XOR
> itself is nothing, the CPU can do them quickly).  If you're
> building a system that is a dedicated storage device, that's
> one thing, and you use software RAID -- but if you're
> building a server that is servicing clients, it's nice to
> keep the storage processing from hogging I/O that could be
> used for network and other services.

We have had this discussion before.  Not sure how a few hundred MB/sec
of I/O is supposed to eat up 4GB/sec of Pci-e bandwidth.  Keep in mind
that PCI-e isn't shared.  To talk to for instance another PCI-e device
you are using seperate lines (again unlike pci-x).

So PCI-e is a point to point duplex connection, reads and writes from
multiple devices do not compete for bandwidth, nor do they have to
negotiate for the bus.

To avoid a repeat of previous arguments please post ACTUAL numbers
showing the superiority of hardware RAID.  I don't deny it's possible,
but without real numbers the rest is hand waving.  I've sustained well
over a GB/sec of I/O with an opteron system, I've not experienced the
"hogging I/O" problem.

> How are the Linux drivers and user-space support?  I've never

The linux drivers seem fine, I've not played with the user-space
tools, so far just the web interface.  I know they are out there but
I've not used them.  Can anyone else on the list comment?

> used the Areca so I'm very interested.  My use of 3Ware
> 7000/8000 series is more about the proven drivers and
> user-space support.  I'd be happy to find a PCIe solution
> that is equal to the 3Ware for PCI-64/X.

With software RAID I can't tell the difference between 3ware and areca,
I've don't have any extensive production use of hardware RAID on either.
Not since I lost a few filesystems to a buggy 3ware hardware raid driver
back in the 6800 days.  Of course 3ware has gotten much better since then.

> Good to know.  But how is the user-space support?

Sorry, no experience.

-- 
Bill Broadley
Computational Science and Engineering
UC Davis