Re: Load balancing across multiple network paths

On Fri, 2004-12-10 at 14:43, Peter T. Breuer wrote:
> "Also sprach Nikos Gkorogiannis:"
> > I am not sure what you mean by that, but a look at /proc/interrupts
> > after a reboot and a dd shows roughly same numbers of interrupts for the
> > NICs. Also, doing a paraller dd from /proc/kcore through netcat from
> 
> That's impossible - the NIC were imbalanced 2:1 in terms of how much work
> they did! They cannot really have the same number of interrupts.

You are right of course, I was looking at a different machine's
/proc/interrupts.

> All the reads are at 32K. What happens if you lower (or increase)
> merge_requests limit. To me it looks as though there is serialisation
> through something, and it may simply "the CPU", perehaps in copying.

Indeed, lowering merge_requests to 16 spreads out evenly the load. So,
after a clean start and one dd, /proc/nbdinfo looks like

Device a:       Open
[a] State:      verify, rw, merge requests, enabled, validated,
show_errs, last error 0, lives 0, bp 0
[a] Queued:     +0R/0W curr (check 0R/0W) +2R/0W max
[a] Buffersize: 262144  (sectors=512, blocks=64)
[a] Blocksize:  4096    (log=12)
[a] Size:       16811008KB
[a] Blocks:     4202752
[a] Sockets:    4 (*)   (+)     (+)     (+)
[a] Requested:  64.048K (15.9K) (15.8K) (15.8K) (16.3K) 64.04KR/0W     
max 18
[a] Despatched: 64.048K (15.9K) (15.8K) (15.8K) (16.3K) 64.04KR/0W     
md5 0W (0 eq, 0 ne, 0 dn)
[a] Errored:    0 (0)   (0)     (0)     (0)     0+0
[a] Pending:    0 (0)   (0)     (0)     (0)     0R/0W+0R/0W
[a] B/s now:    152K    (152KR+0W)
[a] B/s ave:    6.11M   (6.11MR+0W)
[a] B/s max:    9.25M   (9.25MR+0W)
[a] Spectrum:   49%14   49%18
[a] Kthreads:   0 (0 waiting/0 running/1 max)
[a] Cthreads:   4 (+)   (+)     (+)     (+)
[a] Cpids:      4 (2066)        (2067)  (2068)  (2069)
Device b-p:     Closed

and indeed the dd throughput is slightly higher, at 9.4Mb/s.

> If so, shorter reads and relatively more overhead per channel may allow
> for better pipelining between the competing threads.

At the application level, in this case just dd, reading smaller chunks
does not make a difference on throughput. The load across processes (in
terms of requests in nbdinfo) seems evenly spread as above.


> Well, it looks as though one thread is almost constantly frozen out,
> which implies a resource limit somewhere. What was the cpu utilization
> during this test? It may be the cpus being saturated, or a bus ....

The CPUs of both server and client never exceed 40% utilization. It is
difficult to see whether the bus is saturated though. 

> I think of you include one more line, the Cpids, we could see if they
> have changed and hence if indeed one thread has has been starved till
> death.

I run around 16 tests of these and looked at Cpids - no deaths
whatsoever. Maybe they would occur if I run more tests.

I will try smaller block sizes than 4096 and get back to you.

Nikos





This email has been independently scanned for viruses and any virus detected has been removed using McAfee anti-virus software