Re: [jgroups-dev] Questions on flow control

On 10/21/13 8:33 PM, Jiadong Wu wrote:
> Hi All,
>
> This is Jiadong. I'm new to jgroups, and I have some questions on the
> flow control mechanism of jgroups (3.4.0). When conducting the MPerf
> test on a 3-node cluster with GbE interconnection, I noticed that the
> default settings in udp.xml performed poorly under congestion. In 1-to-3
> multicasting scenario, default udp.xml delivered an amazing throughput
> of 120MB/sec/node with 5K-sized message. However, in 3-to-3 multicasting
> scenario, default udp.xml only delivered 45MB/sec/node. The throughput
> would drop further to single digit if I injected a continues TCP flow
> onto one of the nodes as interference.


On my home cluster of 4 nodes (1 GB Network) I can get 150MB/sec. 
However, I noticed that in our lab I get sub-optimal performance for 
multicasts. I suspect that this has to do with ethernet flow control 
(802.x), but have yet to confirm this.

In any case, I created JGRP-1507 [1] some time ago, to look into a new 
multicast flow control algorithm, which (1) takes the amount of traffic 
into account that's sent by a member P when it is *sent* not when it is 
*received* and (2) provides an adaptive algorithm (the current alg has a 
fixed number of bytes).

This is the biggest new feature in 3.5, but it'll take some time 
investigating and testing. There are surprisingly few papers on 
multicast flow control out there.

> At first I suspected that the problem was due to sub-optimal MFC
> parameters, but after trying several combinations I found out that
> tuning the credit parameters could become a dilemma case even without
> interference traffic.
>
> It seems to me that lowering the credit pool will increase xmit
> efficiency under congestion but will also decrease the best-effort
> throughput in a congestion-free environment.

Correct

> With max_credits=50K
> min_threshold=0.4 and frag_size=8K, I get a 3-to-3 multicasting
> throughput of 90MB/sec/node which is acceptable. However, with such a
> small credit pool the 1-to-3 multicasting throughput drops to only
> 48MB/sec/node.

Yes - that's something that needs to be addressed in [1]. Note that 
there's also an (experimental) RATE_LIMITER protocol which I wrote to 
prevent any member from sending more that its share of data to prevent 
message loss due to the switch getting overwhelmed.

> Is it true that the MFC implementation in jgroups 3.4.0 can only handle
> congestion statically?

Yes

>  If true, then is it viable to improve MFC with some feedback mechanism (e.g. AMID in TCP)?

Yes. The general idea of [1] (beyond the taking into account of bytes 
when they're sent) is to adjust the credit size based on the loss rate 
and possibly feedback from members. If you have suggestions / feedback 
re [1], please feel free to add them to [1].
Cheers,


[1] https://issues.jboss.org/browse/JGRP-1507

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)

------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk
_______________________________________________
Javagroups-development mailing list