Re: [jgroups-users] JGroups cluster being disturbed over the weekend (couple of days of inactivity probably)
Questions/problems related to using JGroups <[email protected]> Wed, 14 Dec 2016 07:39:27 +0100
| Newsgroups | gmane.comp.java.javagroups.general |
|---|---|
| Message-ID | <[email protected]> |
On 13/12/16 23:46, Questions/problems related to using JGroups wrote: > Hi Bela Ban, > > Sorry- I should have been more clear. > > By cluster being disturbed, I meant: > Among the 25 nodes, about 12 to 15 nodes report 'Cluster Modified' and > show that the cluster has been down-sized to that many nodes. > The remaining 13 or 10 nodes do not report any 'Cluster Modified' and > they are unable to transfer any messages to any other node in the cluster. > Just by restarting the application on these faulty 13 or 10 nodes, they > seem to fall back into the cluster with other nodes. > > We've mainly noticed this happening over the weekend. For instance, we > see he cluster working fine until Friday evening. Monday morning it is > broken. Looking at the application logs, we see 'Cluster Modified' > messages appeared Sunday evening. Perhaps some of your machines go into standby after a certain time of inactivity? Or perhaps the switch starts dropping packets? Are all nodes connected to the same switch? Do you have any firewall rules that might kick in over the weekend? Or does your application do something in the background which causes a leak and a subsequent long GC cycle? You can check who triggered the suspicions by invoking probe (or using JMX): probe.sh jmx=FD.num_ jmx=FD_SOCK.num_ probe.sh op=FD.printSuspectHistory probe.sh op=FD_SOCK.printSuspectHistory If this is FD, you may for example increase the timeout. > We can certainly try out 3.6.11 and see if this is resolved. However, we > are not JGroups experts :) - so we are curious to see if there is > something wrong with the configuration we've used and/or there is some > inactivity timeout involved here. I've never heard of such a problem during my 16 years working on JGroups, and JGroups is used in a lot of production settings, so I'm skeptical the upgrade will fix this... I think it's your env causing this, e.g. a broken switch etc... > On Tue, Dec 13, 2016 at 3:04 PM, Questions/problems related to using > JGroups <[email protected] > <mailto:[email protected]>> wrote: > > > > On 13/12/16 18:21, Questions/problems related to using JGroups wrote: > > Hi Bela Ban, > > > > We've made a lot of progress with JGroups for our use case, so far. > > > > We currently have a JGroups cluster with about 25 nodes. We actively use > > the cluster during the week and there is very little to no use for the > > cluster during the weekend. > > > > We've been noticing lately that the cluster gets disturbed over the > > weekend. We are guessing probably this happens over some period of > > inactivity. > > What's disturbed? Does the cluster fall apart? Do members block? Stack > trace? > > > > Is there any configuration parameter around inactivity in the config XML > > file that we should take a closer look? > > > > *More details about our JGroups cluster / config:* > > We are using JGroups 3.6.6.Final. > > /*<groupId>org.jgroups</groupId>*/ > > /* <artifactId>jgroups</artifactId>*/ > > /* <version>3.6.6.Final</version>*/ > > > Can you try this with 3.6.11? > > > > We are using MPING for discovery. > > We also have FD and FD_SOCK configured for failure detection. > > > > The following is our configuration file. Please let us know if you find > > any discrepancies in the configuration. > > > > /*<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance > <http://www.w3.org/2001/XMLSchema-instance>"*/ > > /* xmlns="urn:org:jgroups"*/ > > /* xsi:schemaLocation="urn:org:jgroups > > http://www.jgroups.org/schema/jgroups.xsd > <http://www.jgroups.org/schema/jgroups.xsd>">*/ > > /* <TCP bind_port="7800"*/ > > /* recv_buf_size="${tcp.recv_buf_size:5M}"*/ > > /* send_buf_size="${tcp.send_buf_size:5M}"*/ > > /* max_bundle_size="64K"*/ > > /* max_bundle_timeout="30"*/ > > /* use_send_queues="true"*/ > > /* sock_conn_timeout="300"*/ > > /* > > */ > > /* timer_type="new3"*/ > > /* timer.min_threads="4"*/ > > /* timer.max_threads="10"*/ > > /* timer.keep_alive_time="3000"*/ > > /* timer.queue_max_size="500"*/ > > /* > > */ > > /* thread_pool.enabled="true"*/ > > /* thread_pool.min_threads="2"*/ > > /* thread_pool.max_threads="8"*/ > > /* thread_pool.keep_alive_time="5000"*/ > > /* thread_pool.queue_enabled="true"*/ > > /* thread_pool.queue_max_size="10000"*/ > > /* thread_pool.rejection_policy="discard"*/ > > /* > > */ > > /* oob_thread_pool.enabled="true"*/ > > /* oob_thread_pool.min_threads="1"*/ > > /* oob_thread_pool.max_threads="8"*/ > > /* oob_thread_pool.keep_alive_time="5000"*/ > > /* oob_thread_pool.queue_enabled="false"*/ > > /* oob_thread_pool.queue_max_size="100"*/ > > /* oob_thread_pool.rejection_policy="discard"/>*/ > > /* > > */ > > /* <MPING ip_ttl="8"*/ > > /* mcast_addr="224.1.2.3"*/ > > /* mcast_port="7801"*/ > > /* send_on_all_interfaces="true"*/ > > /* receive_on_all_interfaces="true"/>*/ > > /* > > */ > > /* <MERGE3 min_interval="10000"*/ > > /* max_interval="30000"/>*/ > > /* <FD_SOCK client_bind_port="40718" start_port="40718"/>*/ > > /* <FD timeout="3000" max_tries="3" />*/ > > /* <VERIFY_SUSPECT timeout="1500" />*/ > > /* <BARRIER />*/ > > /* <pbcast.NAKACK2 use_mcast_xmit="false"*/ > > /* discard_delivered_msgs="true"/>*/ > > /* <UNICAST3 />*/ > > /* <pbcast.STABLE stability_delay="1000" > desired_avg_gossip="50000"*/ > > /* max_bytes="4M"/>*/ > > /* <pbcast.GMS print_local_addr="true" join_timeout="5000"*/ > > /* view_bundling="true"/>*/ > > /* <MFC max_credits="2M"*/ > > /* min_threshold="0.4"/>*/ > > /* <FRAG2 frag_size="60K" />*/ > > /* <!--RSVP resend_interval="2000" timeout="10000"/-->*/ > > /* <pbcast.STATE_TRANSFER/>*/ > > /*</config>*/ > > /* > > */ > > /* > > */ > > Please let me know if you need more information. > > -- > > *Regards, > > *Venkat > > -- > Bela Ban, JGroups lead (http://www.jgroups.org) > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > _______________________________________________ > javagroups-users mailing list > [email protected] > <mailto:[email protected]> > https://lists.sourceforge.net/lists/listinfo/javagroups-users > <https://lists.sourceforge.net/lists/listinfo/javagroups-users> > > > > > -- > *Regards, > *Venkat > > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > > > _______________________________________________ > javagroups-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/javagroups-users > -- Bela Ban, JGroups lead (http://www.jgroups.org) ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot