Re: [jgroups-users] [ Every day cluster splits ]
Questions/problems related to using JGroups <[email protected]>
| Newsgroups | gmane.comp.java.javagroups.general |
|---|---|
| Message-ID | <[email protected]> |
Great you found the root cause; those switch issues are very hard to find when giving advice on a mailing list! Cheers, On 21/06/16 09:22, Questions/problems related to using JGroups wrote: > Hi, community! > > The long history is over. We tried everything and nothing helps. The > root cause was in buggy open switch virtual network interface. After a > migration to the physical interface the problem has gone. It's about two > months of a stable work without an accident cluster splitting. Thanks > for helping! > > On Apr 22, 2016 15:55, "Questions/problems related to using JGroups" > <[email protected] > <mailto:[email protected]>> wrote: > > Perhaps also try setting TCPPING.send_cache_on_join to true, to see if > it fixes the issue > > On 22/04/16 13:01, Questions/problems related to using JGroups wrote: > > Ok. Here's the new config: > > > > <config xmlns="urn:org:jgroups" > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > > xsi:schemaLocation="urn:org:jgroups > > http://www.jgroups.org/schema/JGroups-3.6.1.xsd"> > > <TCP_NIO2 > > bind_port="7800" > > timer_type="new3" > > timer.min_threads="4" > > timer.max_threads="10" > > timer.keep_alive_time="3000" > > timer.queue_max_size="500" > > thread_pool.enabled="true" > > thread_pool.min_threads="4" > > thread_pool.max_threads="64" > > thread_pool.keep_alive_time="20000" > > thread_pool.queue_enabled="false" > > thread_pool.queue_max_size="100" > > thread_pool.rejection_policy="discard" > > > > oob_thread_pool.enabled="true" > > oob_thread_pool.min_threads="2" > > oob_thread_pool.max_threads="8" > > oob_thread_pool.keep_alive_time="20000" > > oob_thread_pool.queue_enabled="false" > > oob_thread_pool.queue_max_size="100" > > oob_thread_pool.rejection_policy="discard" > > port_range="0" > > /> > > <TCPPING > > > initial_hosts="${jgroups.tcpping.initial_hosts:172.16.42.3[7800],172.16.42.4[7800],172.16.42.5[7800]}" > > port_range="0"/> > > <MERGE3 min_interval="5000" max_interval="20000"/> > > <FD_SOCK/> > > <FD_ALL2 timeout="90000"/> > > <VERIFY_SUSPECT/> > > <pbcast.NAKACK2 use_mcast_xmit="false"/> > > <UNICAST3/> > > <pbcast.STABLE/> > > <pbcast.GMS/> > > <MFC/> > > <FRAG2/> > > <pbcast.STATE_TRANSFER /> > > <CENTRAL_LOCK/> > > </config> > > > > > > I've changed only a port_range to 0 in TCP_NIO2. > > If it doesn't help, will try another ping protocols. > > > > BTW, Another cluster (production), with 2 nodes has no problems. > > The configuration is similar. > > > > <config xmlns="urn:org:jgroups" > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > > xsi:schemaLocation="urn:org:jgroups > > http://www.jgroups.org/schema/JGroups-3.6.1.xsd"> > > <TCP > > bind_port="7800" > > timer_type="new3" > > timer.min_threads="4" > > timer.max_threads="10" > > timer.keep_alive_time="3000" > > timer.queue_max_size="500" > > thread_pool.enabled="true" > > thread_pool.min_threads="8" > > thread_pool.max_threads="128" > > thread_pool.keep_alive_time="20000" > > thread_pool.queue_enabled="false" > > thread_pool.queue_max_size="256" > > thread_pool.rejection_policy="discard" > > > > oob_thread_pool.enabled="true" > > oob_thread_pool.min_threads="2" > > oob_thread_pool.max_threads="8" > > oob_thread_pool.keep_alive_time="20000" > > oob_thread_pool.queue_enabled="false" > > oob_thread_pool.queue_max_size="256" > > oob_thread_pool.rejection_policy="discard" > > /> > > <TCPPING > > > initial_hosts="${jgroups.tcpping.initial_hosts:10.0.1.49[7800],10.0.1.50[7800]}" > > port_range="0"/> > > <MERGE3/> > > <FD_SOCK/> > > <FD_ALL timeout="90000"/> > > <VERIFY_SUSPECT/> > > <pbcast.NAKACK2 use_mcast_xmit="false"/> > > <UNICAST3/> > > <pbcast.STABLE/> > > <pbcast.GMS/> > > <MFC/> > > <FRAG2/> > > <pbcast.STATE_TRANSFER /> > > <CENTRAL_LOCK/> > > </config> > > > > But, jgroups 3.6.1 is used (and another DC is used) > > The previous time, before a restart, it was up since (Semptember, > > 2015!!) with no one disconnection. > > > > We also tried to find a network problem (were dumping packet > headers). > > No packet loss was detected before cluster reconnects. > > > > > > On Fri, Apr 22, 2016 at 12:05 PM, Questions/problems related to using > > JGroups <[email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>>> wrote: > > > > The second log shows that node3 consistently discovers a > coordinator, > > wheres the first log almost never does this. > > > > Can you verify that there are indeed nodes running on each of > these > > addresses: 172.16.42.3[7800],172.16.42.4[7800],172.16.42.5[7800]? > > > > Also make sure that all ports are 7800: if a member starts > ans 7800 is > > taken, it might pick 7801. You can prevent this by setting > > TCP_NIO2.port_range="0"; then a node would not start up, but > throw an > > exception instead, when 7800 is taken. > > > > You could also try to replace TCPPING with FILE_PING > (requires a shared > > file system), TCPGOSSIP or MPING (requires IP mcasting), just > to see if > > TCPPING is the culprit. > > > > > > On 20/04/16 13:41, Questions/problems related to using > JGroups wrote: > > > A log from another node for the same period of time: > > > > > >https://gist.github.com/sherman/e17c39d0cfbdfc5bc8cb2241e7f47f3b > > > > > > On Wed, Apr 20, 2016 at 2:28 PM, Denis Gabaydulin > <[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> > > > <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>>> wrote: > > > > > > Can't find a reason, why a cluster splits every day :-( > > > The cluster has merged back after a dozen of seconds. > > > > > > Have 3 nodes configuration. > > > > > > <config xmlns="urn:org:jgroups" > > > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > > > xsi:schemaLocation="urn:org:jgroups > > > http://www.jgroups.org/schema/JGroups-3.6.9.xsd"> > > > <TCP_NIO2 > > > bind_port="7800" > > > timer_type="new3" > > > timer.min_threads="4" > > > timer.max_threads="10" > > > timer.keep_alive_time="3000" > > > timer.queue_max_size="500" > > > thread_pool.enabled="true" > > > thread_pool.min_threads="4" > > > thread_pool.max_threads="64" > > > thread_pool.keep_alive_time="20000" > > > thread_pool.queue_enabled="false" > > > thread_pool.queue_max_size="100" > > > thread_pool.rejection_policy="discard" > > > > > > oob_thread_pool.enabled="true" > > > oob_thread_pool.min_threads="2" > > > oob_thread_pool.max_threads="8" > > > oob_thread_pool.keep_alive_time="20000" > > > oob_thread_pool.queue_enabled="false" > > > oob_thread_pool.queue_max_size="100" > > > oob_thread_pool.rejection_policy="discard" > > > /> > > > <TCPPING > > > > > > initial_hosts="${jgroups.tcpping.initial_hosts:172.16.42.3[7800],172.16.42.4[7800],172.16.42.5[7800]}" > > > port_range="0"/> > > > <MERGE3 min_interval="5000" max_interval="20000"/> > > > <FD_SOCK/> > > > <FD_ALL timeout="90000"/> > > > <VERIFY_SUSPECT/> > > > <pbcast.NAKACK2 use_mcast_xmit="false"/> > > > <UNICAST3/> > > > <pbcast.STABLE/> > > > <pbcast.GMS/> > > > <MFC/> > > > <FRAG2/> > > > <pbcast.STATE_TRANSFER /> > > > <CENTRAL_LOCK/> > > > </config> > > > > > > Same were on TCP. > > > > > > For example a log file from one of the nodes: > > > > > > > https://gist.github.com/sherman/026114ad85d5b2a10db591c5d39cb9ac -- Bela Ban, JGroups lead (http://www.jgroups.org) ------------------------------------------------------------------------------ Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San Francisco, CA to explore cutting-edge tech and listen to tech luminaries present their vision of the future. This family event has something for everyone, including kids. Get more information and register today. http://sdm.link/attshape