Re: Load balancing across multiple network paths

In article <[email protected]> you wrote:
> Uhm, 40% CPU utilization means exactly what you said, ie CPU not
> executing HLTs or waits for 60% of the time, no? If, on the other hand,
> you mean that this 40% usage happens in large, less frequent chunks, as
> opposed to smaller, more frequent ones, and that somehow freezes one or
> more enbd threads thus reducing the throughput by reducing concurrency,
> then (phew) that definitely concurs with and explains what I see. 
> 
> I've created a small graph at
> http://www.cems.uwe.ac.uk/~ngkorogi/graph.png

Very interesting. What I see is:

   one disk is worse than four disks, so something is happening in
   parallel!

   performance is independent of blocksize, so the kernel is
   aggregating  requests correctly (you should look at the spectrum
   line in nbdinfo to get a real measure of the distribution).

   Performance peaks at 48KB request size limits. You probably want to
   change make f card to see if it is a limit of the card.

   Peak performance is about 10.5MB/s, so more than one physical 100BT
   ethernet connection is being used simultaneously.

   The measured maximum for one disk is just under 10MB/s, which is
   about maximum at 100BT since only one ethernet device is being used
   and there are overheads from TCP and from enbd. I have just measured
   my BT100 connection using netperf for raw TCP speed, and it is
   93.62Mb/s (eepro100 to eepro100 across a switch) which is about
   10.7MB/s, and then there is the enbd overhead to come.

   Performance declines with increasing request size beyond 48KB even 
   with ONE disk, down to about 6MB/s at 128KB. You cannot attribute
   that to anything else except the behaviour of the network and the
   NIC, I think, since preumablyteh disks do not care whether you take
   data off them in small or large lumps! Well, maybe one could
   attribute it to the server having to read the data into buffers and
   then send it. If so, try running the server with the -M switch,
   will will tell it to send data to the net directly from a mapped area
   of the disk.

I'm a little concerned about that last point. I should check to see how
the server send data without -M. I suspect it needs to read a bit write
a bit read a bit write a bit ... rather than doing whatever it is doing
now. 

The client does use as receiving tcp buffer the area that is known to
the kernel already but I think one copy could be avoided there too.

> X is max number of blocks per requests (which I believe is
> merge_requests+1, no?) and Y is total throughput. All points correspond
> to avg of two 1Gb sustained reads with variance less than 1%. The first

How much RAM do you have?

> three series are done with a server exporting an MD RAID-0 device, with
> the 3 possible block sizes. The fourth one is with a single disk and 4k
> block size. There is an interesting downhill slope that occurs after
> around 50kb/req regardless of the value of merge_requests - some buffer
> in the NIC's driver is around 50k maybe? (actually the driver keeps 32
> receive buffers of 1532 bytes each - maybe that's it?). Weirder is the

Looks like it.

> bump-up at the end too!

Possibly some particular kernel efficiency at 128KB. Perhaps rapid
assignment of memory.

> 
> > Perhaps the problem is that the NICs are busmastering and they take the
> > bus for themselves, for long periods, locking out the other NIC? If each
> > NIC locks the bus for longer than it needs, just in case more packets
> > come in, that may explain some of it.
> > 
> > Can you look in the NIC driver source comments to see if there is
> > something about that sort of thing?
> 
> The cards are 3C905Bs and the driver is 3c59x. I did not find obvious
> references to bus mastering in the code, and, annoyingly, the module has
> an argument that purportedly controls bus-mastering, but the module
> ignores it :) The cards indeed do bus-mastering. I will try a different
> make and see what happens. Thanks for all the help!

If you cannot actually put them on different pci buses, then yes,
please try other makes of NIC and see what changes when you vary the
experimental parameters.


> On an unrelated issue: are there any suggestions, comments, warnings etc
> with respect to the following scenario?
> 
> * N computers with 4 ide disks each, in a raid0 array, running enbd-server. 
> * one computer running N enbd-clients, and a raid5 device on top of them.

Err, striped raid exported to clients running raid5 over the exported
devices. Yes - raid5 over the net will be a nightmare unless you use
FR5 instead of raid5 (fr5.sf.net). And in general, putting all your
eggs in one basket (the net) is not a robust solution!

A more standard configuration would be something like: arrange clients
in groups of 3. Let each machine run raid5 over one local disk and two
remote disks imported via ENBD, and let it export two local disks via
ENBD to the other raids.  That lets each machine export one raided
"disk" to the world, and so you get three exported (raided) disks of
twice the size of a physical disk out of nine physical disks.  And if
any one of the three machine dies, the other two both will survive its
loss in their arrays, and can recreate its array from the data they have
between them.


> Are there any guidelines or rules of thumb for module or filesystem
> parameters etc?

If you are running raid5, you are interested in achieving stability and 
rapid recovery, not speed!

Peter