[jgroups-users] BUG / Nodes not determining the same merge leader when merging 2 clusters

Hello,

First, thanks to JGroups community for all the great work done !

I've been working on a distributed system using JGroups 2.12.1.Final to solve some issues.
And I suspect a bug on "Discovery" class/protocol.


*         Environment context

-          JGroups version : 2.12.1.Final

-          11 nodes

-          Tested environment : Ability to split and isolate the network into 2 sub networks

o   Sub network A of 3 nodes

o   Sub network B of 8 nodes

-          JGroups configuration using TCPPING -> see attached file


*         Issue : Nodes not determining the same merge leader when merging 2 clusters

*         Problem analysis :

MERGE2 triggers a scheduled task with a fixed delay to discover new groups in regular basis. It sends an event 'FIND_ALL_VIEWS' down from the protocol stack. It is handled by TCPPING which is responsible to build and send a PING to all initial hosts set in jgroups configuration files. PING is a 'GET_MBRS_REQ' message to discover new groups. As it is implemented in JGroups 2.12.1  and as it is shown below in "Discovery.java" class, the 'GET_MBRS_REQ' message is handled by the distant "initial hosts" only if the message is coming from a member in different view. Otherwise, the message is discarded.

    public Object up(Event evt) {

        switch(evt.getType()) {

            case Event.MSG:
                ...
                switch(hdr.type) {

                    case PingHeader.GET_MBRS_REQ:   // return Rsp(local_addr, coord)
                        ...
                        if(return_entire_cache && hdr.view_id == null && rank != 0) {
                            ...
                        }
                        else {
                            if(hdr.view_id != null) {

                                // If the discovery request is merge-triggered, and we the ViewId shipped with it
                                // is the same as ours, we don't respond (JGRP-1315).
                                ViewId my_view_id=view != null? view.getViewId() : null;
                                if(my_view_id != null && Util.sameViewId(my_view_id, hdr.view_id))
                                    return null;
                            }
                        ...

When the new subgroups discovered is returned back to the MERGE2 protocol stack, and if the current node is coordinator, it adds its group to the discovered ones to determine the "merge leader".

        private List<PingData> findAllViews() {
            List<PingData> retval=(List<PingData>)down_prot.down(new Event(Event.FIND_ALL_VIEWS));
            if(retval == null) return Collections.emptyList();
            if(is_coord && local_addr != null) {
                PingData tmp=new PingData(local_addr, view, true);
                //let's make sure that we add ourself as a coordinator
                if(!retval.contains(tmp))
                    retval.add(tmp);
            }
            return retval;
        }


*** The issue *** is that it happens that a non-coordinator node triggered the FindSubgroupsTask in MERGE2 protocol stack when merging clusters. The result is that it ends off a different groups discovered and different merge leader determined, which may cause merge to fail !



*         Solution : Fix the regression introduced by JGRP-1315 by responding to the GET_MBRS_REQ when the 'GET_MBRS_REQ' message receiver is coordinator.

*         Below is the proposed patch in Discovery.java :

                            if(hdr.view_id != null) {
                                // If the discovery request is merge-triggered, and we the ViewId shipped with it
                                // is the same as ours, we don't respond (JGRP-1315).
                                ViewId my_view_id=view != null? view.getViewId() : null;

-                                if(my_view_id != null && Util.sameViewId(my_view_id, hdr.view_id)) {
+                                if(my_view_id != null && Util.sameViewId(my_view_id, hdr.view_id) && !Util.isCoordinator(view, local_addr)) {
                                    log.debug("... Not responding to 'GET_MBRS_REQ' [ping_view_id=" + hdr.view_id + ", my_view_id=" + my_view_id + ", rank=" + rank + "]");
                                    return null;
                                }
                            }

Many thanks,
Thomas UNG

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users

<config xmlns="urn:org:jgroups"
		xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
		xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/JGroups-2.12.xsd">

	<TCP
		bind_port="${rdf.jgroups.tcp_protocol.bind_port:7800}"
		bind_addr="${rdf.jgroups.all_protocols.bind_address:localhost}"
		singleton_name="RDF_TCP_TP"
		loopback="true"
		recv_buf_size="20M"
		send_buf_size="640K"
		discard_incompatible_packets="true" 
		max_bundle_size="64K"
		max_bundle_timeout="30" 
		enable_bundling="false" 
		use_send_queues="true"
		sock_conn_timeout="300"

		timer_type="new"
		timer.min_threads="4"
		timer.max_threads="10"
		timer.keep_alive_time="3000"
		timer.queue_max_size="500"

		thread_pool.enabled="true"
		thread_pool.min_threads="1" 
		thread_pool.max_threads="20"
		thread_pool.keep_alive_time="10000" 
		thread_pool.queue_enabled="false"
		thread_pool.queue_max_size="100" 
		thread_pool.rejection_policy="discard"

		oob_thread_pool.enabled="true" 
		oob_thread_pool.min_threads="1"
		oob_thread_pool.max_threads="8" 
		oob_thread_pool.keep_alive_time="5000"
		oob_thread_pool.queue_enabled="false" 
		oob_thread_pool.queue_max_size="100"
		oob_thread_pool.rejection_policy="discard" 

		log_discard_msgs="false"/>

	<TCPPING
		timeout="3000"
		initial_hosts="${rdf.jgroups.tcpping_protocol.initial_hosts}"
		port_range="0"
		num_initial_members="2"
		num_ping_requests="1"/>

	<MERGE2 
		max_interval="30000" 
		min_interval="10000" />

	<FD_SOCK/>

	<!--
		<FD timeout="10000" max_tries="5" />
	-->

	<FD_ALL/>

	<VERIFY_SUSPECT 
		timeout="1500" />

	<pbcast.NAKACK 
		use_mcast_xmit="false" 
		gc_lag="100"
		retransmit_timeout="600,1200,2400,4800"
		discard_delivered_msgs="true"/>

	<UNICAST 
		timeout="300,600,1200" />

	<pbcast.STABLE 
		stability_delay="1000" 
		desired_avg_gossip="20000"
		max_bytes="4m" />

	<pbcast.GMS 
		print_local_addr="true" 
		join_timeout="5000"
		view_bundling="true"/>

	<UFC max_credits="2M"
		min_threshold="0.4"/>

	<MFC max_credits="2M"
		min_threshold="0.4"/>

	<FRAG2
		frag_size="60000" />

	<pbcast.STREAMING_STATE_TRANSFER />

	<pbcast.FLUSH/>

	<CENTRAL_LOCK
		num_backups="3" />

</config>