Re: Load balancing across multiple network paths

On Thu, 2004-12-09 at 16:50, Peter T. Breuer wrote:

> There  must be some reason why one is consistently slower than the
> other. Which order are the interrupts attended in?

I am not sure what you mean by that, but a look at /proc/interrupts
after a reboot and a dd shows roughly same numbers of interrupts for the
NICs. Also, doing a paraller dd from /proc/kcore through netcat from
server to client machine maxes out both NICs at the same time, so it
cannot be simply a NIC issue... Now, since the server machine is an old
pII :) I thought that the PCI/IDE monstrosity might do it, but that does
not stand either after testing it (and it would not explain easily the
difference of throughput, only its reduction).

As another test I started four processes with
enbd-client 192.168.0.1:12345 192.168.0.1 192.168.1.1 192.168.0.1
192.168.1.1 -b 4096 -e /dev/nda &

After the first dd (256Mb), /proc/nbdinfo is 
[a] Sockets:    4 (+)   (+)     (*)     (+)
[a] Requested:  64.048K (3)     (34.5K) (2.47K) (27.0K) 64.04KR/0W     
max 32
[a] Despatched: 64.048K (3)     (34.5K) (2.47K) (27.0K) 64.04KR/0W     
md5 0W (0 eq, 0 ne, 0 dn)
[a] Errored:    0 (0)   (0)     (0)     (0)     0+0
[a] Pending:    0 (0)   (0)     (0)     (0)     0R/0W+0R/0W
[a] B/s now:    0 (0R+0W)
[a] B/s ave:    3.75M   (3.75MR+0W)
[a] B/s max:    8.24M   (8.24MR+0W)
[a] Spectrum:   99%32
[a] Kthreads:   0 (0 waiting/0 running/1 max)
[a] Cthreads:   4 (+)   (+)     (+)     (+)

After a second dd, /proc/nbdinfo is
[a] Sockets:    4 (+)   (+)     (*)     (+)
[a] Requested:  128.09K (22.6K) (34.5K) (2.56K) (68.3K) 128.0KR/0W     
max 32
[a] Despatched: 128.09K (22.6K) (34.5K) (2.56K) (68.3K) 128.0KR/0W     
md5 0W (0 eq, 0 ne, 0 dn)
[a] Errored:    0 (0)   (0)     (0)     (0)     0+0
[a] Pending:    0 (0)   (0)     (0)     (0)     0R/0W+0R/0W
[a] B/s now:    5.43M   (5.43MR+0W)
[a] B/s ave:    15.3M   (15.3MR+0W)
[a] B/s max:    82.2M   (82.2MR+0W)
[a] Spectrum:   99%32
[a] Kthreads:   0 (0 waiting/0 running/1 max)
[a] Cthreads:   4 (+)   (+)     (+)     (+)

And after a third,
[a] Sockets:    4 (+)   (+)     (*)     (+)
[a] Requested:  192.14K (22.6K) (34.5K) (35.5K) (99.3K) 192.1KR/0W     
max 32
[a] Despatched: 192.14K (22.6K) (34.5K) (35.5K) (99.3K) 192.1KR/0W     
md5 0W (0 eq, 0 ne, 0 dn)
[a] Errored:    0 (0)   (0)     (0)     (0)     0+0
[a] Pending:    0 (0)   (0)     (0)     (0)     0R/0W+0R/0W
[a] B/s now:    8.13M   (8.13MR+0W)
[a] B/s ave:    24.5M   (24.5MR+0W)
[a] B/s max:    414M    (414MR+0W)
[a] Spectrum:   99%32
[a] Kthreads:   0 (0 waiting/0 running/1 max)
[a] Cthreads:   4 (+)   (+)     (+)     (+)

All of these run at 8.5Mb/s. In the first run, processes 2 and 4 do most
of the work. In run 2, it's processes 1 and 4 and in the 3rd it's 3 and
4. Weird! How is the decision as to which process serves a request made?
Is it random? I did more runs and there is no obvious correlation, ie in
one run process number 1 served almost all requests. Also, in the runs
where it seems that processes on different NICs do most of the work, the
throughput is again 8.5Mb/s... 

Where should I look in the source, for the point where the kernel module
hands out a request to one of the client processes? 

Nikos






This email has been independently scanned for viruses and any virus detected has been removed using McAfee anti-virus software