Re: Load balancing across multiple network paths

Nikos Gkorogiannis <[email protected]> wrote:
> Indeed, lowering merge_requests to 16 spreads out evenly the load. So,
> after a clean start and one dd, /proc/nbdinfo looks like

OK.

> and indeed the dd throughput is slightly higher, at 9.4Mb/s.
> 
> > If so, shorter reads and relatively more overhead per channel may allow
> > for better pipelining between the competing threads.
> 
> At the application level, in this case just dd, reading smaller chunks
> does not make a difference on throughput. The load across processes (in

It wouldn't make a difference.  The kernel will recombine different
requests up to the merge_requests limit, whetever parameters you give
to dd. Any effect should be negligible.

> terms of requests in nbdinfo) seems evenly spread as above.
> 
> 
> > Well, it looks as though one thread is almost constantly frozen out,
> > which implies a resource limit somewhere. What was the cpu utilization
> > during this test? It may be the cpus being saturated, or a bus ....
> 
> The CPUs of both server and client never exceed 40% utilization. It is

40% is very high. I would expect on the order of 3 or 4%. The cpus must
be relatively slow compared to the transfer speed.

> difficult to see whether the bus is saturated though. 

At 40% it is quite possible that the cpu is saturated to 100% for 40%
of the time. Some resonance may keep it that way, though I don't see
what!

> 
> > I think of you include one more line, the Cpids, we could see if they
> > have changed and hence if indeed one thread has has been starved till
> > death.
> 
> I run around 16 tests of these and looked at Cpids - no deaths
> whatsoever. Maybe they would occur if I run more tests.

OK, well the starvation that does occur is not absolute.

Assuming that the blockage is in the cpu, for large requests the cou
would first copy one request at 100% cpu, blocking out the other client
thread, then the other client would enter and copying would proceed
from it, occupying the cpu, perhaps blocking the original thread from
acking or going back to userpace with the results. This could result in
some kind of serialization, but I don't see how the cpus could run at
"only" 40% that way.

Perhaps the problem is that the NICs are busmastering and they take the
bus for themselves, for long periods, locking out the other NIC? If each
NIC locks the bus for longer than it needs, just in case more packets
come in, that may explain some of it.

Can you look in the NIC driver source comments to see if there is
something about that sort of thing?

> I will try smaller block sizes than 4096 and get back to you.

Peter