Re: [jgroups-users] JGroups cluster being disturbed over the weekend (couple of days of inactivity probably)
Questions/problems related to using JGroups <[email protected]> Thu, 15 Dec 2016 18:00:38 -0600
| Newsgroups | gmane.comp.java.javagroups.general |
|---|---|
| Message-ID | <CAAYFd2HkBc22j-tMrJXCA93EcFGCVhs=nVBvv8_QfqbW+yiK_g@mail.gmail.com> |
Hi Bela Ban, Yes, all nodes are connected to the same switch, with no intermediate firewall. However, we will double check with all your good suggestions/pointers here. Thank You. On Wed, Dec 14, 2016 at 12:39 AM, Questions/problems related to using JGroups <[email protected]> wrote: > > > On 13/12/16 23:46, Questions/problems related to using JGroups wrote: > > Hi Bela Ban, > > > > Sorry- I should have been more clear. > > > > By cluster being disturbed, I meant: > > Among the 25 nodes, about 12 to 15 nodes report 'Cluster Modified' and > > show that the cluster has been down-sized to that many nodes. > > The remaining 13 or 10 nodes do not report any 'Cluster Modified' and > > they are unable to transfer any messages to any other node in the > cluster. > > Just by restarting the application on these faulty 13 or 10 nodes, they > > seem to fall back into the cluster with other nodes. > > > > We've mainly noticed this happening over the weekend. For instance, we > > see he cluster working fine until Friday evening. Monday morning it is > > broken. Looking at the application logs, we see 'Cluster Modified' > > messages appeared Sunday evening. > > Perhaps some of your machines go into standby after a certain time of > inactivity? Or perhaps the switch starts dropping packets? > > Are all nodes connected to the same switch? Do you have any firewall > rules that might kick in over the weekend? > > Or does your application do something in the background which causes a > leak and a subsequent long GC cycle? > > You can check who triggered the suspicions by invoking probe (or using > JMX): > > probe.sh jmx=FD.num_ jmx=FD_SOCK.num_ > probe.sh op=FD.printSuspectHistory > probe.sh op=FD_SOCK.printSuspectHistory > > If this is FD, you may for example increase the timeout. > > > We can certainly try out 3.6.11 and see if this is resolved. However, we > > are not JGroups experts :) - so we are curious to see if there is > > something wrong with the configuration we've used and/or there is some > > inactivity timeout involved here. > > I've never heard of such a problem during my 16 years working on > JGroups, and JGroups is used in a lot of production settings, so I'm > skeptical the upgrade will fix this... I think it's your env causing > this, e.g. a broken switch etc... > > > > On Tue, Dec 13, 2016 at 3:04 PM, Questions/problems related to using > > JGroups <[email protected] > > <mailto:[email protected]>> wrote: > > > > > > > > On 13/12/16 18:21, Questions/problems related to using JGroups wrote: > > > Hi Bela Ban, > > > > > > We've made a lot of progress with JGroups for our use case, so far. > > > > > > We currently have a JGroups cluster with about 25 nodes. We > actively use > > > the cluster during the week and there is very little to no use for > the > > > cluster during the weekend. > > > > > > We've been noticing lately that the cluster gets disturbed over the > > > weekend. We are guessing probably this happens over some period of > > > inactivity. > > > > What's disturbed? Does the cluster fall apart? Do members block? > Stack > > trace? > > > > > > > Is there any configuration parameter around inactivity in the > config XML > > > file that we should take a closer look? > > > > > > *More details about our JGroups cluster / config:* > > > We are using JGroups 3.6.6.Final. > > > /*<groupId>org.jgroups</groupId>*/ > > > /* <artifactId>jgroups</artifactId>*/ > > > /* <version>3.6.6.Final</version>*/ > > > > > > Can you try this with 3.6.11? > > > > > > > We are using MPING for discovery. > > > We also have FD and FD_SOCK configured for failure detection. > > > > > > The following is our configuration file. Please let us know if you > find > > > any discrepancies in the configuration. > > > > > > /*<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance > > <http://www.w3.org/2001/XMLSchema-instance>"*/ > > > /* xmlns="urn:org:jgroups"*/ > > > /* xsi:schemaLocation="urn:org:jgroups > > > http://www.jgroups.org/schema/jgroups.xsd > > <http://www.jgroups.org/schema/jgroups.xsd>">*/ > > > /* <TCP bind_port="7800"*/ > > > /* recv_buf_size="${tcp.recv_buf_size:5M}"*/ > > > /* send_buf_size="${tcp.send_buf_size:5M}"*/ > > > /* max_bundle_size="64K"*/ > > > /* max_bundle_timeout="30"*/ > > > /* use_send_queues="true"*/ > > > /* sock_conn_timeout="300"*/ > > > /* > > > */ > > > /* timer_type="new3"*/ > > > /* timer.min_threads="4"*/ > > > /* timer.max_threads="10"*/ > > > /* timer.keep_alive_time="3000"*/ > > > /* timer.queue_max_size="500"*/ > > > /* > > > */ > > > /* thread_pool.enabled="true"*/ > > > /* thread_pool.min_threads="2"*/ > > > /* thread_pool.max_threads="8"*/ > > > /* thread_pool.keep_alive_time="5000"*/ > > > /* thread_pool.queue_enabled="true"*/ > > > /* thread_pool.queue_max_size="10000"*/ > > > /* thread_pool.rejection_policy="discard"*/ > > > /* > > > */ > > > /* oob_thread_pool.enabled="true"*/ > > > /* oob_thread_pool.min_threads="1"*/ > > > /* oob_thread_pool.max_threads="8"*/ > > > /* oob_thread_pool.keep_alive_time="5000"*/ > > > /* oob_thread_pool.queue_enabled="false"*/ > > > /* oob_thread_pool.queue_max_size="100"*/ > > > /* oob_thread_pool.rejection_policy="discard"/>*/ > > > /* > > > */ > > > /* <MPING ip_ttl="8"*/ > > > /* mcast_addr="224.1.2.3"*/ > > > /* mcast_port="7801"*/ > > > /* send_on_all_interfaces="true"*/ > > > /* receive_on_all_interfaces="true"/>*/ > > > /* > > > */ > > > /* <MERGE3 min_interval="10000"*/ > > > /* max_interval="30000"/>*/ > > > /* <FD_SOCK client_bind_port="40718" start_port="40718"/>*/ > > > /* <FD timeout="3000" max_tries="3" />*/ > > > /* <VERIFY_SUSPECT timeout="1500" />*/ > > > /* <BARRIER />*/ > > > /* <pbcast.NAKACK2 use_mcast_xmit="false"*/ > > > /* discard_delivered_msgs="true"/>*/ > > > /* <UNICAST3 />*/ > > > /* <pbcast.STABLE stability_delay="1000" > > desired_avg_gossip="50000"*/ > > > /* max_bytes="4M"/>*/ > > > /* <pbcast.GMS print_local_addr="true" join_timeout="5000"*/ > > > /* view_bundling="true"/>*/ > > > /* <MFC max_credits="2M"*/ > > > /* min_threshold="0.4"/>*/ > > > /* <FRAG2 frag_size="60K" />*/ > > > /* <!--RSVP resend_interval="2000" timeout="10000"/-->*/ > > > /* <pbcast.STATE_TRANSFER/>*/ > > > /*</config>*/ > > > /* > > > */ > > > /* > > > */ > > > Please let me know if you need more information. > > > -- > > > *Regards, > > > *Venkat > > > > -- > > Bela Ban, JGroups lead (http://www.jgroups.org) > > > > > > ------------------------------------------------------------ > ------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > _______________________________________________ > > javagroups-users mailing list > > [email protected] > > <mailto:[email protected]> > > https://lists.sourceforge.net/lists/listinfo/javagroups-users > > <https://lists.sourceforge.net/lists/listinfo/javagroups-users> > > > > > > > > > > -- > > *Regards, > > *Venkat > > > > > > > > ------------------------------------------------------------ > ------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > > > > > > > _______________________________________________ > > javagroups-users mailing list > > [email protected] > > https://lists.sourceforge.net/lists/listinfo/javagroups-users > > > > -- > Bela Ban, JGroups lead (http://www.jgroups.org) > > > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > _______________________________________________ > javagroups-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/javagroups-users > -- *Regards,*Venkat ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot _______________________________________________ javagroups-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/javagroups-users