Re: Load balancing across multiple network paths

On Wed, 2004-12-08 at 18:01, Peter T. Breuer wrote:
> > enbd-client 192.168.0.1:12345 192.168.0.1 192.168.1.1 -b 4096 /dev/nda
> 
> I'd forgotten the syntax! I suppose it is correct. The first must be
> where the master server that launches the slaves is to be found, and the
> other two must be the IPs on which the slaves are.

Exactly. Starting it with
# enbd-client 192.168.0.1:12345 192.168.1.1 -b 4096 /dev/nda
does not do the trick as it only creates one daemon (as per
/proc/nbdinfo).

> And these NICs are on different subnets. The really have different
> cables and different routes to the target?

Yes, they really connect to different switches, which in turn connect to
different NICs on the server machine.

> > > The daemons are independent. They do not alternate, because they do not
> > > (in effect) know the other exists. So there is some other explanation.
> > 
> > So, from what you saying I should expect almost double throughput? Hm,
> > curious...
> 
> Yes, are you able to measure the traffic through the NICs? Can you show
> the ifconfig lines? I want to look at the packet counts. Better if you
> can zero them first by removing and relaoding drivers, then running a test.

After a reboot, and zero counters on the two interfaces (eth1 and eth2),
I run on the client 

# dd if=/dev/nda of=/dev/null bs=64k count=8k
8192+0 records in
8192+0 records out
536870912 bytes transferred in 62.390865 seconds (8604960 bytes/sec)

Running ifconfig 

<snip>
eth1      Link encap:Ethernet  HWaddr 00:04:75:AF:A1:C3
          inet addr:192.168.0.254  Bcast:192.168.0.255 
Mask:255.255.255.0
          inet6 addr: fe80::204:75ff:feaf:a1c3/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:255815 errors:0 dropped:0 overruns:0 frame:0
          TX packets:130671 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:381316317 (363.6 MiB)  TX bytes:8981356 (8.5 MiB)
          Interrupt:10 Base address:0x1400

eth2      Link encap:Ethernet  HWaddr 00:04:75:AF:A5:C9
          inet addr:192.168.1.254  Bcast:192.168.1.255 
Mask:255.255.255.0
          inet6 addr: fe80::204:75ff:feaf:a5c9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:121458 errors:0 dropped:0 overruns:0 frame:0
          TX packets:62040 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:180925040 (172.5 MiB)  TX bytes:4265598 (4.0 MiB)
          Interrupt:5 Base address:0x1480

which surprisingly shows a 2:1 ratio! /proc/nbdinfo says 

Device a:       Open
[a] State:      verify, rw, merge requests, enabled, validated,
show_errs, last error 0, lives 0, bp 0
[a] Queued:     +0R/0W curr (check 0R/0W) +2R/0W max
[a] Buffersize: 262144  (sectors=512, blocks=64)
[a] Blocksize:  4096    (log=12)
[a] Size:       16811008KB
[a] Blocks:     4202752
[a] Sockets:    2       (*)     (+)
[a] Requested:  128.04K (86.8K) (41.2K) 128.0KR/0W      max 32
[a] Despatched: 128.04K (86.8K) (41.2K) 128.0KR/0W      md5 0W (0 eq, 0
ne, 0 dn)
[a] Errored:    0       (0)     (0)     0+0
[a] Pending:    0       (0)     (0)     0R/0W+0R/0W
[a] B/s now:    0       (0R+0W)
[a] B/s ave:    2.08M   (2.08MR+0W)
[a] B/s max:    80.8M   (80.8MR+0W)
[a] Spectrum:   99%32
[a] Kthreads:   0       (0 waiting/0 running/1 max)
[a] Cthreads:   2       (+)     (+)
[a] Cpids:      2       (710)   (711)
Device b-p:     Closed

Indeed the daemons cannot be alternating... Also the B/s max seems
spurious as these are 100Mbit/s NICs.

> (Btw, you can use ethernet binding).

Yes, though the real reason I want dual pathing is redundancy (in cards,
switches, cables etc) so ethernet binding is not really an option. Oh, I
should have said that this will be part of a networked raid5, so I
cannot use crossovers, hence the switches.

Nikos





This email has been independently scanned for viruses and any virus detected has been removed using McAfee anti-virus software