[jgroups-dev] JGroups 2.11: problem to discover members if more than 100 nodes

With JG 2.6.15, and we have passed tests with 300 hosts.
With JG 2.11 we're experiencing problems if we have more than 100 nodes
although this version includes fixes for large scale clusters.
Problems arise during the discovery step. See below the configuration we
used.

What we discovered (and thus many questions):
- Initial investigations shown that there are a lot of errors in the
reception of UDP packets on sockets created by JGroups. Within seconds there
may be up to 7-8 Mbit received by a host. The maximum size of the UDP RX 
buffer must be set to at least 1MB to avoid having packet loss due to the
startup burst. It is what is indeed recommanded in the performance tuning
guideline, but we fear the size is too much tightly dependent on the number
of members.
- We understand that when a member joins to the group, a ping request is
multicasted and each member sends back a response if PING.max_rank=0. In
this case the cache is populated from both the ping requests and the ping
responses. If PING.max_rank=1, only one member replies (the coordinator) but
a problem is that there is as many responses as cache entries, which still
represents a big number of packets and responses received. It seems there is
no benefit because PING (coordinator) sends responses on behalf of the other
members.
- The cache is only filled during the membership discovery: it is what we
seen when we've reviewed the code; and if we connect several channels and
empty the cache, if no action is done, the cache remains empty or not
complete. If we patch the code to avoid the coordinator sends the entire
cache (return_entire_cache is not set automatically to true when max_rank>0:
the 'if' statement is removed in Discover.init()), no protocol is in charge
of the cache filling afterwards. A test has been performed to check if a
member can send an unicast message even with a cache empty or not complete
and to know if the cache entry can be fetched "on demand". And the test
fails: if the cache is empty, it is impossible to do any unicast call.
- At last, another patch has been done in Discover.up() (~line 339) to
insert a Thread.sleep(100ms) just after sendDiscoveryResponse(): with
max_rank=1, only the coordinator sends the cache, but for each peer, the
entries are sent with tempo to avoid the burst. And it works much better.
However, sometimes, certain caches are not filled properly: several entries
are missing and the view has the correct number of members but contains both
physical addresses and still (unresolved) logical address. If we explicitly
call PING.findInitialMembersAsString() trough the JConsole, the discovery
protocol is run and JGroups manages to fill entirely the cache. 

What do you think about it ? Do you have any clue ?

-----------------------------------------

<protocol_stacks>
    <stack name="c0" description="test">
        <config>
            <UDP
		        mcast_addr="239.195.11.0"
                mcast_port="40000"

                bind_port="40000"
                port_range="10"
                                
                tos="8"
                
                ucast_recv_buf_size="1024000"
                ucast_send_buf_size="64000"
                mcast_recv_buf_size="1024000"
                mcast_send_buf_size="64000"
                
                loopback="false"
                
                ip_mcast="true"

                discard_incompatible_packets="true"
                
                enable_bundling="false"

                ip_ttl="8"

                thread_naming_pattern="cl"
             
                enable_diagnostics="false"

                thread_pool.enabled="true"
                thread_pool.min_threads="5"
                thread_pool.max_threads="100"
                thread_pool.keep_alive_time="10000"
                thread_pool.queue_enabled="true"
                thread_pool.queue_max_size="100"
                thread_pool.rejection_policy="Run"
    
                oob_thread_pool.enabled="true"
                oob_thread_pool.min_threads="5"
                oob_thread_pool.max_threads="20"
                oob_thread_pool.keep_alive_time="5000"
                oob_thread_pool.queue_enabled="true"
                oob_thread_pool.queue_max_size="100"
                oob_thread_pool.rejection_policy="Run"
				logical_addr_cache_max_size="200"
				
                stats="true"/>
		   <PING timeout="5000" return_entire_cache="true" max_rank="1"
ergonomics="false" num_initial_members="200" break_on_coord_rsp="true"
num_ping_requests="2" stats="true"/>
           <MERGE2 min_interval="10000" max_interval="20000" stats="true"/>
       	   <FD_ALL interval="5000" timeout="11000" stats="true"/>
           <VERIFY_SUSPECT timeout="1500" stats="true"/>
           <pbcast.NAKACK use_stats_for_retransmission="false"
use_mcast_xmit="true" gc_lag="50" max_xmit_buf_size="10000"
                       retransmit_timeout="300,600,1200,2400,3600"
discard_delivered_msgs="true"
                       stats="true"/>
           <UNICAST timeout="300,600,1200,2400,3600" stats="true"/>           
           <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
max_bytes="500000" stats="true"/>
           <VIEW_SYNC avg_send_interval="60000" stats="true"/>
	       <pbcast.GMS print_local_addr="false" join_timeout="5000"
merge_timeout="5000" view_bundling="true" max_bundling_time="2000"
stats="true"/>
           <FRAG2 frag_size="1300" stats="true"/>
           <!-- <COMPRESS min_size="1300" compression_level="9"
stats="true"/> -->
           <pbcast.FLUSH timeout="0" stats="true"/>        
         </config>
    </stack>        
</protocol_stacks>


-- 
View this message in context: http://old.nabble.com/JGroups-2.11%3A-problem-to-discover-members-if-more-than-100-nodes-tp30408262p30408262.html
Sent from the JGroups - Dev mailing list archive at Nabble.com.


------------------------------------------------------------------------------
This SF Dev2Dev email is sponsored by:

WikiLeaks The End of the Free Internet
http://p.sf.net/sfu/therealnews-com
_______________________________________________
Javagroups-development mailing list