Re: [jgroups-dev] JGroups 2.11: problem to discover members if more than 100 nodes

On 12/8/10 7:22 PM, Benoît wrote:
>
> With JG 2.6.15, and we have passed tests with 300 hosts.
> With JG 2.11 we're experiencing problems if we have more than 100 nodes
> although this version includes fixes for large scale clusters.
> Problems arise during the discovery step. See below the configuration we
> used.


First off, why do you use FLUSH ? It isn't recommended for large 
clusters, as it doesn't scale: when a new view has to be installed, or a 
merge happens, all of your 300 nodes have to be suspended, the view 
installed, and the resumed.

Also, VIEW_SYNC isn't part of 2.11, as a matter of fact it was removed 
some versions ago. Make sure your stack is similar to the one shipped 
with 2.11.

Note that I've got a JIRA on best practices for large clusters, take a 
look at [1].


> What we discovered (and thus many questions):
> - Initial investigations shown that there are a lot of errors in the
> reception of UDP packets on sockets created by JGroups. Within seconds there
> may be up to 7-8 Mbit received by a host. The maximum size of the UDP RX
> buffer must be set to at least 1MB to avoid having packet loss due to the
> startup burst. It is what is indeed recommanded in the performance tuning
> guideline, but we fear the size is too much tightly dependent on the number
> of members


Having a large buffer for datagram (net.core.rmem_max) makes sense, as 
it reduces the chances of loss. Loss has to be repaired by JGroups via 
retransmission, and that delays delivery of already received messages.

I don't think the max buffer size is a function of the cluster size; IMO 
it is rather a function of the max data received on the link, which is 
the link size, e.g. 1GBit. The buffer can probably be sized according to 
the bandwidth - delay product, however, that's only for 2 peers. Well, 
in any case, you can only receive 125MByte/sec for a 1GBit link, so the 
max buffer size should be ~ 130MByte. Of course, this doesn't take the 
delay in processing into account; if you app cannot handle 125MBytes 
/sec, then the buffer needs to take that into account.

In practice, this should be necessary, as flow control (UFC and MFC) 
throttle a fast sender, if a slower receiver cannot keep up with the 
rate at which the sender is sending messages.


> - We understand that when a member joins to the group, a ping request is
> multicasted and each member sends back a response if PING.max_rank=0.


Yes


>  In
> this case the cache is populated from both the ping requests and the ping
> responses.


You mean the logical address cache ? Yes, whenever there is a discovery 
request or response, the information shipped with them about logical 
address/name/UUID/IP address information is added to the cache.


>  If PING.max_rank=1, only one member replies (the coordinator) but
> a problem is that there is as many responses as cache entries, which still
> represents a big number of packets and responses received. It seems there is
> no benefit because PING (coordinator) sends responses on behalf of the other
> members.


Don't understand...


> - The cache is only filled during the membership discovery: it is what we
> seen when we've reviewed the code; and if we connect several channels and
> empty the cache,


Why would you want to empty the cache ? Don't do this !


> if no action is done, the cache remains empty or not complete.


That shouldn't happen: the merge protocol periodically kicks off a 
discovery process, which populates the cache, but of course only in the 
coordinator.

If a member Q needs to send a unicast to P, and there is no data for P 
in the cache, it'll also invoke the discovery process, which will fill 
Q's cache.

But, again, do not empty a cache !


>  If we patch the code to avoid the coordinator sends the entire
> cache (return_entire_cache is not set automatically to true when max_rank>0:
> the 'if' statement is removed in Discover.init()), no protocol is in charge
> of the cache filling afterwards. A test has been performed to check if a
> member can send an unicast message even with a cache empty or not complete
> and to know if the cache entry can be fetched "on demand". And the test
> fails: if the cache is empty, it is impossible to do any unicast call.


Wait, see my explanation above: the first send should fail, yes, but the 
second should pass ! Unicast retransmission will make sure the mesage is 
indeed retransmitted, so that there *is* a second (or subsequent) send.


> - At last, another patch has been done in Discover.up() (~line 339) to
> insert a Thread.sleep(100ms) just after sendDiscoveryResponse(): with
> max_rank=1, only the coordinator sends the cache, but for each peer, the
> entries are sent with tempo to avoid the burst. And it works much better.

You mean a random delay to stagger discovery responses ?


> However, sometimes, certain caches are not filled properly: several entries
> are missing and the view has the correct number of members but contains both
> physical addresses and still (unresolved) logical address. If we explicitly
> call PING.findInitialMembersAsString() trough the JConsole, the discovery
> protocol is run and JGroups manages to fill entirely the cache.
>
> What do you think about it ? Do you have any clue ?



>          <config>
>              <UDP
> 		        mcast_addr="239.195.11.0"
>                  mcast_port="40000"
>
>                  bind_port="40000"
>                  port_range="10"
>
>                  tos="8"
>
>                  ucast_recv_buf_size="1024000"
>                  ucast_send_buf_size="64000"
>                  mcast_recv_buf_size="1024000"
>                  mcast_send_buf_size="64000"
>
>                  loopback="false"
>
>                  ip_mcast="true"
>
>                  discard_incompatible_packets="true"
>
>                  enable_bundling="false"
>
>                  ip_ttl="8"
>
>                  thread_naming_pattern="cl"
>
>                  enable_diagnostics="false"
>
>                  thread_pool.enabled="true"
>                  thread_pool.min_threads="5"
>                  thread_pool.max_threads="100"
>                  thread_pool.keep_alive_time="10000"
>                  thread_pool.queue_enabled="true"
>                  thread_pool.queue_max_size="100"
>                  thread_pool.rejection_policy="Run"
>
>                  oob_thread_pool.enabled="true"
>                  oob_thread_pool.min_threads="5"
>                  oob_thread_pool.max_threads="20"
>                  oob_thread_pool.keep_alive_time="5000"
>                  oob_thread_pool.queue_enabled="true"
>                  oob_thread_pool.queue_max_size="100"
>                  oob_thread_pool.rejection_policy="Run"
> 				logical_addr_cache_max_size="200"
> 				
>                  stats="true"/>
> 		<PING timeout="5000" return_entire_cache="true" max_rank="1"
> ergonomics="false" num_initial_members="200" break_on_coord_rsp="true"
> num_ping_requests="2" stats="true"/>
>             <MERGE2 min_interval="10000" max_interval="20000" stats="true"/>
>         	<FD_ALL interval="5000" timeout="11000" stats="true"/>
>             <VERIFY_SUSPECT timeout="1500" stats="true"/>
>             <pbcast.NAKACK use_stats_for_retransmission="false"
> use_mcast_xmit="true" gc_lag="50" max_xmit_buf_size="10000"
>                         retransmit_timeout="300,600,1200,2400,3600"
> discard_delivered_msgs="true"
>                         stats="true"/>
>             <UNICAST timeout="300,600,1200,2400,3600" stats="true"/>
>             <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
> max_bytes="500000" stats="true"/>
>             <VIEW_SYNC avg_send_interval="60000" stats="true"/>
> 	<pbcast.GMS print_local_addr="false" join_timeout="5000"
> merge_timeout="5000" view_bundling="true" max_bundling_time="2000"
> stats="true"/>
>             <FRAG2 frag_size="1300" stats="true"/>
>             <!--<COMPRESS min_size="1300" compression_level="9"
> stats="true"/>  -->
>             <pbcast.FLUSH timeout="0" stats="true"/>
>           </config>


[1] https://jira.jboss.org/browse/JGRP-100

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss

------------------------------------------------------------------------------
This SF Dev2Dev email is sponsored by:

WikiLeaks The End of the Free Internet
http://p.sf.net/sfu/therealnews-com
_______________________________________________
Javagroups-development mailing list