Re: Load balancing across multiple network paths

"Also sprach Nikos Gkorogiannis:"
> After a reboot, and zero counters on the two interfaces (eth1 and eth2),
> I run on the client 
> 
> # dd if=/dev/nda of=/dev/null bs=64k count=8k
> 8192+0 records in
> 8192+0 records out

That would be 64MB? Or about 48K full-sized ethernet packets.


> 536870912 bytes transferred in 62.390865 seconds (8604960 bytes/sec)
> 
> Running ifconfig 
> 
> <snip>
> eth1      Link encap:Ethernet  HWaddr 00:04:75:AF:A1:C3
>           inet addr:192.168.0.254  Bcast:192.168.0.255 Mask:255.255.255.0
>           inet6 addr: fe80::204:75ff:feaf:a1c3/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:255815 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:130671 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000

I suppose the MTU is 1500.

>           RX bytes:381316317 (363.6 MiB)  TX bytes:8981356 (8.5 MiB)
>           Interrupt:10 Base address:0x1400
> 
> eth2      Link encap:Ethernet  HWaddr 00:04:75:AF:A5:C9
>           inet addr:192.168.1.254  Bcast:192.168.1.255 Mask:255.255.255.0
>           inet6 addr: fe80::204:75ff:feaf:a5c9/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:121458 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:62040 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:180925040 (172.5 MiB)  TX bytes:4265598 (4.0 MiB)
>           Interrupt:5 Base address:0x1480
> 
> which surprisingly shows a 2:1 ratio! /proc/nbdinfo says 

yes, this conforms with what is visible below too.

> 
> Device a:       Open
> [a] State:      verify, rw, merge requests, enabled, validated,
> show_errs, last error 0, lives 0, bp 0
> [a] Queued:     +0R/0W curr (check 0R/0W) +2R/0W max
> [a] Buffersize: 262144  (sectors=512, blocks=64)
> [a] Blocksize:  4096    (log=12)
> [a] Size:       16811008KB
> [a] Blocks:     4202752
> [a] Sockets:    2       (*)     (+)
> [a] Requested:  128.04K (86.8K) (41.2K) 128.0KR/0W      max 32
> [a] Despatched: 128.04K (86.8K) (41.2K) 128.0KR/0W      md5 0W (0 eq, 0
> ne, 0 dn)
> [a] Errored:    0       (0)     (0)     0+0
> [a] Pending:    0       (0)     (0)     0R/0W+0R/0W
> [a] B/s now:    0       (0R+0W)
> [a] B/s ave:    2.08M   (2.08MR+0W)
> [a] B/s max:    80.8M   (80.8MR+0W)
> [a] Spectrum:   99%32
> [a] Kthreads:   0       (0 waiting/0 running/1 max)
> [a] Cthreads:   2       (+)     (+)
> [a] Cpids:      2       (710)   (711)
> Device b-p:     Closed
> 
> Indeed the daemons cannot be alternating... Also the B/s max seems
> spurious as these are 100Mbit/s NICs.

The problem is that that measurement is a bit too "spontaneous". The
time interval over which it is measured can be short.  Many such
intervals are averaged (and smoothed) to give the average measurement.

Actually, I see that the measurements are updated at least once
every second, in theory, and that the raw data is folded into an
existing average to give a smoothed rolling average speed. As far as I
can see, the new data gets weighted according to the amount of time
that has passed since the last measurement, so old data should have a
half-life of one second. The "maximum" is merely the greatest such
smoothed average that has been recorded. I suppose when the device
started up.

Feel free to improve the calculations (in enbd_speed).

If you look while the read is happening, over a long run, the "average" 
speed should be close to the current speed. 


> > (Btw, you can use ethernet binding).
> 
> Yes, though the real reason I want dual pathing is redundancy (in cards,
> switches, cables etc) so ethernet binding is not really an option. Oh, I
> should have said that this will be part of a networked raid5, so I
> cannot use crossovers, hence the switches.

Have you tried FR5 (fr5.sf.net)? I am hoping for testers.

You are not writing, so I don't think I could even suppose there is any
serialisation through preservation of write order!

And I think that if you do ethernet binding one of the NICs can fail
without harming the joint device. But I have not studied the issue.

I'm sorry, I don't see any clear evidence here one way or the other.
How about simply _timing_ a read with one NIC only, and then with two
NICs operating.


Peter