Re: Load balancing across multiple network paths

"Also sprach Nikos Gkorogiannis:"
> I am not sure what you mean by that, but a look at /proc/interrupts
> after a reboot and a dd shows roughly same numbers of interrupts for the
> NICs. Also, doing a paraller dd from /proc/kcore through netcat from

That's impossible - the NIC were imbalanced 2:1 in terms of how much work
they did! They cannot really have the same number of interrupts.

> As another test I started four processes with
> enbd-client 192.168.0.1:12345 192.168.0.1 192.168.1.1 192.168.0.1
> 192.168.1.1 -b 4096 -e /dev/nda &
> 
> After the first dd (256Mb), /proc/nbdinfo is 
> [a] Sockets:    4 (+)   (+)     (*)     (+)
> [a] Requested:  64.048K (3)     (34.5K) (2.47K) (27.0K) 64.04KR/0W     
> max 32
> [a] Despatched: 64.048K (3)     (34.5K) (2.47K) (27.0K) 64.04KR/0W     
> md5 0W (0 eq, 0 ne, 0 dn)
> [a] Errored:    0 (0)   (0)     (0)     (0)     0+0
> [a] Pending:    0 (0)   (0)     (0)     (0)     0R/0W+0R/0W
> [a] B/s now:    0 (0R+0W)
> [a] B/s ave:    3.75M   (3.75MR+0W)
> [a] B/s max:    8.24M   (8.24MR+0W)
> [a] Spectrum:   99%32

All the reads are at 32K. What happens if you lower (or increase)
merge_requests limit. To me it looks as though there is serialisation
through something, and it may simply "the CPU", perehaps in copying.

If so, shorter reads and relatively more overhead per channel may allow
for better pipelining between the competing threads.

> 
> After a second dd, /proc/nbdinfo is
> [a] Sockets:    4 (+)   (+)     (*)     (+)
> [a] Requested:  128.09K (22.6K) (34.5K) (2.56K) (68.3K) 128.0KR/0W     max 32
> [a] Despatched: 128.09K (22.6K) (34.5K) (2.56K) (68.3K) 128.0KR/0W     

Well, it looks as though one thread is almost constantly frozen out,
which implies a resource limit somewhere. What was the cpu utilization
during this test? It may be the cpus being saturated, or a bus ....

> md5 0W (0 eq, 0 ne, 0 dn)
> [a] Errored:    0 (0)   (0)     (0)     (0)     0+0
> [a] Pending:    0 (0)   (0)     (0)     (0)     0R/0W+0R/0W
> [a] B/s now:    5.43M   (5.43MR+0W)
> [a] B/s ave:    15.3M   (15.3MR+0W)
> [a] B/s max:    82.2M   (82.2MR+0W)
> [a] Spectrum:   99%32
> [a] Kthreads:   0 (0 waiting/0 running/1 max)
> [a] Cthreads:   4 (+)   (+)     (+)     (+)

I think of you include one more line, the Cpids, we could see if they
have changed and hence if indeed one thread has has been starved till
death.

> And after a third,
> [a] Sockets:    4 (+)   (+)     (*)     (+)
> [a] Requested:  192.14K (22.6K) (34.5K) (35.5K) (99.3K) 192.1KR/0W     
> max 32
> [a] Despatched: 192.14K (22.6K) (34.5K) (35.5K) (99.3K) 192.1KR/0W     
> md5 0W (0 eq, 0 ne, 0 dn)
> [a] Errored:    0 (0)   (0)     (0)     (0)     0+0
> [a] Pending:    0 (0)   (0)     (0)     (0)     0R/0W+0R/0W
> [a] B/s now:    8.13M   (8.13MR+0W)
> [a] B/s ave:    24.5M   (24.5MR+0W)
> [a] B/s max:    414M    (414MR+0W)
> [a] Spectrum:   99%32
> [a] Kthreads:   0 (0 waiting/0 running/1 max)
> [a] Cthreads:   4 (+)   (+)     (+)     (+)
> 
> All of these run at 8.5Mb/s. In the first run, processes 2 and 4 do most

Total, you mean.

> of the work. In run 2, it's processes 1 and 4 and in the 3rd it's 3 and
> 4. Weird! How is the decision as to which process serves a request made?

No decision is made. All the threads dive into the kernel independently
when they want a request. They take the first unattended request
available and handle it.

> Is it random? I did more runs and there is no obvious correlation, ie in
> one run process number 1 served almost all requests. Also, in the runs

It looks like a scheduler/starvation problem.

> where it seems that processes on different NICs do most of the work, the
> throughput is again 8.5Mb/s... 
> 
> Where should I look in the source, for the point where the kernel module
> hands out a request to one of the client processes? 

It doesn't.  The client processes run independently and ask the kernel
driver for a request when they are available for work.  They use the
GET_REQ ioctl. They will block in a semaphore until a request is
available, individually.  Each of course is in exactly the same code.

Peter