Re: [jgroups-users] [ Every day cluster splits ]

Questions/problems related to using JGroups <[email protected]>
Newsgroups gmane.comp.java.javagroups.general
Message-ID <CAB1b6fF67NwZwo4UhY65+OfWQaySOfuo_T3nKVPnyG6QApf_wg@mail.gmail.com> 
Hi, community!

The long history is over. We tried everything and nothing helps. The root
cause was in buggy open switch virtual network interface. After a migration
to the physical interface the problem has gone. It's about two months of a
stable work without an accident cluster splitting. Thanks for helping!
On Apr 22, 2016 15:55, "Questions/problems related to using JGroups" <
[email protected]> wrote:

> Perhaps also try setting TCPPING.send_cache_on_join to true, to see if
> it fixes the issue
>
> On 22/04/16 13:01, Questions/problems related to using JGroups wrote:
> > Ok. Here's the new config:
> >
> > <config xmlns="urn:org:jgroups"
> >          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> >          xsi:schemaLocation="urn:org:jgroups
> > http://www.jgroups.org/schema/JGroups-3.6.1.xsd">
> >      <TCP_NIO2
> >          bind_port="7800"
> >          timer_type="new3"
> >          timer.min_threads="4"
> >          timer.max_threads="10"
> >          timer.keep_alive_time="3000"
> >          timer.queue_max_size="500"
> >          thread_pool.enabled="true"
> >          thread_pool.min_threads="4"
> >          thread_pool.max_threads="64"
> >          thread_pool.keep_alive_time="20000"
> >          thread_pool.queue_enabled="false"
> >          thread_pool.queue_max_size="100"
> >          thread_pool.rejection_policy="discard"
> >
> >          oob_thread_pool.enabled="true"
> >          oob_thread_pool.min_threads="2"
> >          oob_thread_pool.max_threads="8"
> >          oob_thread_pool.keep_alive_time="20000"
> >          oob_thread_pool.queue_enabled="false"
> >          oob_thread_pool.queue_max_size="100"
> >          oob_thread_pool.rejection_policy="discard"
> >          port_range="0"
> >      />
> >      <TCPPING
> >
> initial_hosts="${jgroups.tcpping.initial_hosts:172.16.42.3[7800],172.16.42.4[7800],172.16.42.5[7800]}"
> > port_range="0"/>
> >      <MERGE3 min_interval="5000" max_interval="20000"/>
> >      <FD_SOCK/>
> >      <FD_ALL2 timeout="90000"/>
> >      <VERIFY_SUSPECT/>
> >      <pbcast.NAKACK2 use_mcast_xmit="false"/>
> >      <UNICAST3/>
> >      <pbcast.STABLE/>
> >      <pbcast.GMS/>
> >      <MFC/>
> >      <FRAG2/>
> >      <pbcast.STATE_TRANSFER />
> >      <CENTRAL_LOCK/>
> > </config>
> >
> >
> > I've changed only a port_range to 0 in TCP_NIO2.
> > If it doesn't help, will try another ping protocols.
> >
> > BTW, Another cluster (production), with 2 nodes has no problems.
> > The configuration is similar.
> >
> > <config xmlns="urn:org:jgroups"
> >          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> >          xsi:schemaLocation="urn:org:jgroups
> > http://www.jgroups.org/schema/JGroups-3.6.1.xsd">
> >      <TCP
> >          bind_port="7800"
> >          timer_type="new3"
> >          timer.min_threads="4"
> >          timer.max_threads="10"
> >          timer.keep_alive_time="3000"
> >          timer.queue_max_size="500"
> >          thread_pool.enabled="true"
> >          thread_pool.min_threads="8"
> >          thread_pool.max_threads="128"
> >          thread_pool.keep_alive_time="20000"
> >          thread_pool.queue_enabled="false"
> >          thread_pool.queue_max_size="256"
> >          thread_pool.rejection_policy="discard"
> >
> >          oob_thread_pool.enabled="true"
> >          oob_thread_pool.min_threads="2"
> >          oob_thread_pool.max_threads="8"
> >          oob_thread_pool.keep_alive_time="20000"
> >          oob_thread_pool.queue_enabled="false"
> >          oob_thread_pool.queue_max_size="256"
> >          oob_thread_pool.rejection_policy="discard"
> >      />
> >      <TCPPING
> >
> initial_hosts="${jgroups.tcpping.initial_hosts:10.0.1.49[7800],10.0.1.50[7800]}"
> > port_range="0"/>
> >      <MERGE3/>
> >      <FD_SOCK/>
> >      <FD_ALL timeout="90000"/>
> >      <VERIFY_SUSPECT/>
> >      <pbcast.NAKACK2 use_mcast_xmit="false"/>
> >      <UNICAST3/>
> >      <pbcast.STABLE/>
> >      <pbcast.GMS/>
> >      <MFC/>
> >      <FRAG2/>
> >      <pbcast.STATE_TRANSFER />
> >      <CENTRAL_LOCK/>
> > </config>
> >
> > But, jgroups 3.6.1 is used (and another DC is used)
> > The previous time, before a restart, it was up since (Semptember,
> > 2015!!) with no one disconnection.
> >
> > We also tried to find a network problem (were dumping packet headers).
> > No packet loss was detected before cluster reconnects.
> >
> >
> > On Fri, Apr 22, 2016 at 12:05 PM, Questions/problems related to using
> > JGroups <[email protected]
> > <mailto:[email protected]>> wrote:
> >
> >     The second log shows that node3 consistently discovers a coordinator,
> >     wheres the first log almost never does this.
> >
> >     Can you verify that there are indeed nodes running on each of these
> >     addresses: 172.16.42.3[7800],172.16.42.4[7800],172.16.42.5[7800]?
> >
> >     Also make sure that all ports are 7800: if a member starts ans 7800
> is
> >     taken, it might pick 7801. You can prevent this by setting
> >     TCP_NIO2.port_range="0"; then a node would not start up, but throw an
> >     exception instead, when 7800 is taken.
> >
> >     You could also try to replace TCPPING with FILE_PING (requires a
> shared
> >     file system), TCPGOSSIP or MPING (requires IP mcasting), just to see
> if
> >     TCPPING is the culprit.
> >
> >
> >     On 20/04/16 13:41, Questions/problems related to using JGroups wrote:
> >     > A log from another node for the same period of time:
> >     >
> >     >https://gist.github.com/sherman/e17c39d0cfbdfc5bc8cb2241e7f47f3b
> >     >
> >     > On Wed, Apr 20, 2016 at 2:28 PM, Denis Gabaydulin <
> [email protected] <mailto:[email protected]>
> >      > <mailto:[email protected] <mailto:[email protected]>>> wrote:
> >      >
> >      >     Can't find a reason, why a cluster splits every day :-(
> >      >     The cluster has merged back after a dozen of seconds.
> >      >
> >      >     Have 3 nodes configuration.
> >      >
> >      >     <config xmlns="urn:org:jgroups"
> >      >              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
> "
> >      >              xsi:schemaLocation="urn:org:jgroups
> >      > http://www.jgroups.org/schema/JGroups-3.6.9.xsd">
> >      >           <TCP_NIO2
> >      >              bind_port="7800"
> >      >              timer_type="new3"
> >      >              timer.min_threads="4"
> >      >              timer.max_threads="10"
> >      >              timer.keep_alive_time="3000"
> >      >              timer.queue_max_size="500"
> >      >              thread_pool.enabled="true"
> >      >              thread_pool.min_threads="4"
> >      >              thread_pool.max_threads="64"
> >      >              thread_pool.keep_alive_time="20000"
> >      >              thread_pool.queue_enabled="false"
> >      >              thread_pool.queue_max_size="100"
> >      >              thread_pool.rejection_policy="discard"
> >      >
> >      >              oob_thread_pool.enabled="true"
> >      >              oob_thread_pool.min_threads="2"
> >      >              oob_thread_pool.max_threads="8"
> >      >              oob_thread_pool.keep_alive_time="20000"
> >      >              oob_thread_pool.queue_enabled="false"
> >      >              oob_thread_pool.queue_max_size="100"
> >      >              oob_thread_pool.rejection_policy="discard"
> >      >           />
> >      >          <TCPPING
> >      >
> >
>  initial_hosts="${jgroups.tcpping.initial_hosts:172.16.42.3[7800],172.16.42.4[7800],172.16.42.5[7800]}"
> >      >     port_range="0"/>
> >      >          <MERGE3 min_interval="5000" max_interval="20000"/>
> >      >          <FD_SOCK/>
> >      >          <FD_ALL timeout="90000"/>
> >      >          <VERIFY_SUSPECT/>
> >      >          <pbcast.NAKACK2 use_mcast_xmit="false"/>
> >      >          <UNICAST3/>
> >      >          <pbcast.STABLE/>
> >      >          <pbcast.GMS/>
> >      >          <MFC/>
> >      >          <FRAG2/>
> >      >          <pbcast.STATE_TRANSFER />
> >      >          <CENTRAL_LOCK/>
> >      >     </config>
> >      >
> >      >     Same were on TCP.
> >      >
> >      >     For example a log file from one of the nodes:
> >      >
> >      > https://gist.github.com/sherman/026114ad85d5b2a10db591c5d39cb9ac
> >      >
> >      >
> >      >
> >      >
> >      >
> >      >
> >      >
> >      >
> >
>  ------------------------------------------------------------------------------
> >      > Find and fix application performance issues faster with
> >     Applications Manager
> >      > Applications Manager provides deep performance insights into
> >     multiple tiers of
> >      > your business applications. It resolves application problems
> >     quickly and
> >      > reduces your MTTR. Get your free trial!
> >      > https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
> >      >
> >      >
> >      >
> >      > _______________________________________________
> >      > javagroups-users mailing list
> >      > [email protected]
> >     <mailto:[email protected]>
> >      > https://lists.sourceforge.net/lists/listinfo/javagroups-users
> >      >
> >
> >     --
> >     Bela Ban, JGroups lead (http://www.jgroups.org)
> >
> >
> >
>  ------------------------------------------------------------------------------
> >     Find and fix application performance issues faster with Applications
> >     Manager
> >     Applications Manager provides deep performance insights into
> >     multiple tiers of
> >     your business applications. It resolves application problems quickly
> and
> >     reduces your MTTR. Get your free trial!
> >     https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
> >     _______________________________________________
> >     javagroups-users mailing list
> >     [email protected]
> >     <mailto:[email protected]>
> >     https://lists.sourceforge.net/lists/listinfo/javagroups-users
> >
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > Find and fix application performance issues faster with Applications
> Manager
> > Applications Manager provides deep performance insights into multiple
> tiers of
> > your business applications. It resolves application problems quickly and
> > reduces your MTTR. Get your free trial!
> > https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
> >
> >
> >
> > _______________________________________________
> > javagroups-users mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/javagroups-users
> >
>
> --
> Bela Ban, JGroups lead (http://www.jgroups.org)
>
>
>
> ------------------------------------------------------------------------------
> Find and fix application performance issues faster with Applications
> Manager
> Applications Manager provides deep performance insights into multiple
> tiers of
> your business applications. It resolves application problems quickly and
> reduces your MTTR. Get your free trial!
> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.