Re: [jgroups-users] Can probe be used to recover network splits?

On Thu, Mar 17, 2016 at 10:59 AM, Questions/problems related to using
JGroups <[email protected]> wrote:

>
>
> On 17/03/16 15:47, Questions/problems related to using JGroups wrote:
> > HI Bela,
> > Thank you for your response. This is very helpful. I will try it out !!
> >
> > The cluster does recover automatically with MERGE3. Only thing it took 6
> > hours in one case and 2 hours in other case to finally merge into 1
> > cluster.
>
> This should not be the case. Did you look at TRACE logs for GMS and MERGE3?
> MERGE3 with min and max around 10 and 30 seconds should take 1-2 rounds,
> so ca. 60 secs max to merge when the network is fine.
>
> I have good enough min_interval and max_intervals. See below:
<MERGE3
max_interval="200000"
max_participants_in_merge="200"
min_interval="20000" />

[PoojaK]: I could not enable TRACE for GMS and MERGE3 since it joined in
late after network had recovered.


> I'm sure there's something else going on, preventing a merge. Have you
> checked your thread pools? Do they have queues enabled perhaps? Do you
> configure the internal and/or timer pools as well?
>
> Thread pool queue is enabled  for Regular thread pools but disabled for
OOB.
I have internal and timer threads configured too.


> What's you config?
>

Below is my config:


> <?xml version="1.0" encoding="UTF-8"?>
>
>
>> <!-- JGroups protocol stack properties. -->
>
> <config
>
> xmlns="urn:org:jgroups"
>
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>
> xsi:schemaLocation="urn:org:jgroups
>> http://www.jgroups.org/schema/JGroups-3.0.xsd" >
>
> <TCP
>
> bind_port="8802"
>
> port_range="0"
>
>   loopback="true"
>
>   recv_buf_size="20000000"
>
> send_buf_size="640000"
>
> max_bundle_size="64000"
>
> max_bundle_timeout="30"
>
> use_send_queues="true"
>
> sock_conn_timeout="3000"
>
> peer_addr_read_timeout="3000"
>
> logical_addr_cache_expiration="600000"
>
> logical_addr_cache_max_size="4000"
>
> diagnostics_port="8801"
>
> thread_naming_pattern="pc1"
>
> thread_pool.enabled="true"
>
> thread_pool.min_threads="50"
>
> thread_pool.max_threads="100"
>
> thread_pool.keep_alive_time="5000"
>
> thread_pool.queue_enabled="true"
>
> thread_pool.queue_max_size="50000"
>
> thread_pool.rejection_policy="Discard"
>
> oob_thread_pool.enabled="true"
>
> oob_thread_pool.min_threads="50"
>
> oob_thread_pool.max_threads="100"
>
> oob_thread_pool.keep_alive_time="50000"
>
> oob_thread_pool.queue_enabled="false"
>
> oob_thread_pool.queue_max_size="1000"
>
> oob_thread_pool.rejection_policy="Discard"
>
> internal_thread_pool.max_threads="75"
>
> internal_thread_pool.min_threads="50"
>
>
>
> timer_type="new3"
>
> timer.min_threads="2"
>
> timer.max_threads="500"
>
>                                         timer.keep_alive_time="5000"
>
> timer.queue_max_size="2000"
>
> timer.rejection_policy="Discard"/>
>
> <TCPPING
>
> timeout="200000"
>
> initial_hosts=""
>
> port_range="0"
>
> num_initial_members="1" />
>
> <MERGE3
>
> max_interval="200000"
>
> max_participants_in_merge="200"
>
> min_interval="20000" />
>
> <FD_ALL
>
> timeout="600000"
>
> interval="60000" />
>
> <VERIFY_SUSPECT
>
> timeout="10000" />
>
> <BARRIER />
>
> <pbcast.NAKACK2
>
>                     xmit_interval="1000"
>
>                     use_mcast_xmit_req="false"
>
>                     use_mcast_xmit="false"
>
>                     discard_delivered_msgs="true" />
>
>
>>     <UNICAST3
>
>                     xmit_interval="1000" />
>
> <UNIGY/>
>
> <pbcast.STABLE
>
> stability_delay="2000"
>
> desired_avg_gossip="60000"
>
> max_bytes="400000" />
>
> <pbcast.GMS
>
> print_local_addr="true"
>
> join_timeout="6000"
>
> view_bundling="true"
>
> merge_timeout="60000"
>
> view_ack_collection_timeout="10000" />
>
> <FRAG2
>
> frag_size="60000" />
>
>
>> <pbcast.STATE_TRANSFER/>
>
> </config>
>
>

> > This is a cluster of 30 Nodes running with TCP as transport.
> > The nodes are geographically co-located across the globe.
> >
> >
> > On Thu, Mar 17, 2016 at 4:27 AM, JGroups - General mailing list [via
> > JGroups] <[hidden email] </user/SendEmail.jtp?type=node&node=10995&i=0>>
> > wrote:
> >
> >     Hi Pooja,
> >
> >     On 16/03/16 21:25, Questions/problems related to using JGroups wrote:
> >     > Hi ,
> >     >
> >     > Say if there is a split in the cluster and it is not recovering
> >     > automatically inspite of no network issues, has any one tried any
> probe
> >     > commands related to merge3 to heal the cluster automatically?
> >
> >     There's a @ManagedOperation MERGE3.sendInfo() that can be triggered,
> so
> >     everyone in a cluster sends their information, allowing the
> coordinator
> >     to start a merge.
> >
> >     However, this doesn't automatically merge the cluster, e.g. if
> everyone
> >     has the same view, then nothing will happen.
> >
> >     If there is no network problem, and MERGE3 does *not* recover the
> >     cluster, then that would be a bug. In such a case, what would be
> >     required for diagnosis is:
> >     - TRACE logs for all members of GMS and MERGE3
> >     - Views of all members. This could be used for a reproducer
> >
> >     I haven't yet come across a scenario with MERGE3 (*not* MERGE2!) that
> >     doesn't heal a network partition when the network is functioning ok
> >     again.
> >
> >      > Thanks
> >      > Pooja
> >      >
> >      >
> >
>  ------------------------------------------------------------------------
> >
> >     > View this message in context: Can probe be used to recover network
> >     > splits?
> >      >
> >     <
> http://jgroups.1086181.n5.nabble.com/Can-probe-be-used-to-recover-network-splits-tp10985.html
> >
> >
> >     > Sent from the JGroups - General mailing list archive
> >      > <http://jgroups.1086181.n5.nabble.com/JGroups-General-f3.html> at
> >      > Nabble.com.
> >     _____________________________________
> >     > javagroups-users mailing list
> >     >[hidden email] <http://
> /user/SendEmail.jtp?type=node&node=10991&i=0>
> >     >https://lists.sourceforge.net/lists/listinfo/javagroups-users
> >
> >     --
> >     Bela Ban, JGroups lead (http://www.jgroups.org)
> >
> >
> >
>  ------------------------------------------------------------------------------
> >
> >     Transform Data into Opportunity.
> >     Accelerate data analysis in your applications with
> >     Intel Data Analytics Acceleration Library.
> >     Click to learn more.
> >     http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
> >     _______________________________________________
> >     javagroups-users mailing list
> >     [hidden email] <http:///user/SendEmail.jtp?type=node&node=10991&i=1>
> >     https://lists.sourceforge.net/lists/listinfo/javagroups-users
> >
> >
> >
>  ------------------------------------------------------------------------
> >     If you reply to this email, your message will be added to the
> >     discussion below:
> >
> http://jgroups.1086181.n5.nabble.com/Can-probe-be-used-to-recover-network-splits-tp10985p10991.html
> >
> >     To start a new topic under JGroups - General, email [hidden email]
> >     </user/SendEmail.jtp?type=node&node=10995&i=1>
> >     To unsubscribe from JGroups, click here.
> >     NAML
> >     <
> http://jgroups.1086181.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >
> >
> >
> >
> >
> > ------------------------------------------------------------------------
> > View this message in context: Re: [jgroups-users] Can probe be used to
> > recover network splits?
> > <
> http://jgroups.1086181.n5.nabble.com/Can-probe-be-used-to-recover-network-splits-tp10985p10995.html
> >
> > Sent from the JGroups - General mailing list archive
> > <http://jgroups.1086181.n5.nabble.com/JGroups-General-f3.html> at
> > Nabble.com.
> >
> >
> >
> ------------------------------------------------------------------------------
> > Transform Data into Opportunity.
> > Accelerate data analysis in your applications with
> > Intel Data Analytics Acceleration Library.
> > Click to learn more.
> > http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
> >
> >
> >
> > _______________________________________________
> > javagroups-users mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/javagroups-users
> >
>
> --
> Bela Ban, JGroups lead (http://www.jgroups.org)
>
>
>
> ------------------------------------------------------------------------------
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>

------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users