Re: Re: Load balancing across multiple network paths

On Fri, 2004-12-10 at 17:23, Peter T. Breuer wrote:
> > > Well, it looks as though one thread is almost constantly frozen
out,
> > > which implies a resource limit somewhere. What was the cpu
utilization
> > > during this test? It may be the cpus being saturated, or a bus
....
> > 
> > The CPUs of both server and client never exceed 40% utilization. It
is
> 
> 40% is very high. I would expect on the order of 3 or 4%. The cpus
must
> be relatively slow compared to the transfer speed.

They are slow (PIIIs @ 650 MHz) but they should not be slow compared to
a transfer speed of 9Mb/s.

> > difficult to see whether the bus is saturated though. 
> 
> At 40% it is quite possible that the cpu is saturated to 100% for 40%
> of the time. Some resonance may keep it that way, though I don't see
> what!

Uhm, 40% CPU utilization means exactly what you said, ie CPU not
executing HLTs or waits for 60% of the time, no? If, on the other hand,
you mean that this 40% usage happens in large, less frequent chunks, as
opposed to smaller, more frequent ones, and that somehow freezes one or
more enbd threads thus reducing the throughput by reducing concurrency,
then (phew) that definitely concurs with and explains what I see. 

I've created a small graph at
http://www.cems.uwe.ac.uk/~ngkorogi/graph.png

X is max number of blocks per requests (which I believe is
merge_requests+1, no?) and Y is total throughput. All points correspond
to avg of two 1Gb sustained reads with variance less than 1%. The first
three series are done with a server exporting an MD RAID-0 device, with
the 3 possible block sizes. The fourth one is with a single disk and 4k
block size. There is an interesting downhill slope that occurs after
around 50kb/req regardless of the value of merge_requests - some buffer
in the NIC's driver is around 50k maybe? (actually the driver keeps 32
receive buffers of 1532 bytes each - maybe that's it?). Weirder is the
bump-up at the end too!

> Perhaps the problem is that the NICs are busmastering and they take
the
> bus for themselves, for long periods, locking out the other NIC? If
each
> NIC locks the bus for longer than it needs, just in case more packets
> come in, that may explain some of it.
> 
> Can you look in the NIC driver source comments to see if there is
> something about that sort of thing?

The cards are 3C905Bs and the driver is 3c59x. I did not find obvious
references to bus mastering in the code, and, annoyingly, the module has
an argument that purportedly controls bus-mastering, but the module
ignores it :) The cards indeed do bus-mastering. I will try a different
make and see what happens. Thanks for all the help!

On an unrelated issue: are there any suggestions, comments, warnings etc
with respect to the following scenario?

* N computers with 4 ide disks each, in a raid0 array, running
enbd-server. 
* one computer running N enbd-clients, and a raid5 device on top of
them.

Are there any guidelines or rules of thumb for module or filesystem
parameters etc?

Thanks
Nikos




This email has been independently scanned for viruses and any virus detected has been removed using McAfee anti-virus software