Re: [jgroups-users] JGroups cluster being disturbed over the weekend (couple of days of inactivity probably)

Hi Bela Ban,

Sorry- I should have been more clear.

By cluster being disturbed, I meant:
Among the 25 nodes, about 12 to 15 nodes report 'Cluster Modified' and show
that the cluster has been down-sized to that many nodes.
The remaining 13 or 10 nodes do not report any 'Cluster Modified' and they
are unable to transfer any messages to any other node in the cluster.
Just by restarting the application on these faulty 13 or 10 nodes, they
seem to fall back into the cluster with other nodes.

We've mainly noticed this happening over the weekend. For instance, we see
he cluster working fine until Friday evening. Monday morning it is broken.
Looking at the application logs, we see 'Cluster Modified' messages
appeared Sunday evening.

We can certainly try out 3.6.11 and see if this is resolved. However, we
are not JGroups experts :) - so we are curious to see if there is something
wrong with the configuration we've used and/or there is some inactivity
timeout involved here.



On Tue, Dec 13, 2016 at 3:04 PM, Questions/problems related to using
JGroups <[email protected]> wrote:

>
>
> On 13/12/16 18:21, Questions/problems related to using JGroups wrote:
> > Hi Bela Ban,
> >
> > We've made a lot of progress with JGroups for our use case, so far.
> >
> > We currently have a JGroups cluster with about 25 nodes. We actively use
> > the cluster during the week and there is very little to no use for the
> > cluster during the weekend.
> >
> > We've been noticing lately that the cluster gets disturbed over the
> > weekend. We are guessing probably this happens over some period of
> > inactivity.
>
> What's disturbed? Does the cluster fall apart? Do members block? Stack
> trace?
>
>
> > Is there any configuration parameter around inactivity in the config XML
> > file that we should take a closer look?
> >
> > *More details about our JGroups cluster / config:*
> > We are using JGroups 3.6.6.Final.
> >             /*<groupId>org.jgroups</groupId>*/
> > /*            <artifactId>jgroups</artifactId>*/
> > /*            <version>3.6.6.Final</version>*/
>
>
> Can you try this with 3.6.11?
>
>
> > We are using MPING for discovery.
> > We also have FD and FD_SOCK configured for failure detection.
> >
> > The following is our configuration file. Please let us know if you find
> > any discrepancies in the configuration.
> >
> > /*<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"*/
> > /*        xmlns="urn:org:jgroups"*/
> > /*        xsi:schemaLocation="urn:org:jgroups
> > http://www.jgroups.org/schema/jgroups.xsd">*/
> > /*    <TCP bind_port="7800"*/
> > /*         recv_buf_size="${tcp.recv_buf_size:5M}"*/
> > /*         send_buf_size="${tcp.send_buf_size:5M}"*/
> > /*         max_bundle_size="64K"*/
> > /*         max_bundle_timeout="30"*/
> > /*         use_send_queues="true"*/
> > /*         sock_conn_timeout="300"*/
> > /*
> > */
> > /*         timer_type="new3"*/
> > /*         timer.min_threads="4"*/
> > /*         timer.max_threads="10"*/
> > /*         timer.keep_alive_time="3000"*/
> > /*         timer.queue_max_size="500"*/
> > /*
> > */
> > /*         thread_pool.enabled="true"*/
> > /*         thread_pool.min_threads="2"*/
> > /*         thread_pool.max_threads="8"*/
> > /*         thread_pool.keep_alive_time="5000"*/
> > /*         thread_pool.queue_enabled="true"*/
> > /*         thread_pool.queue_max_size="10000"*/
> > /*         thread_pool.rejection_policy="discard"*/
> > /*
> > */
> > /*         oob_thread_pool.enabled="true"*/
> > /*         oob_thread_pool.min_threads="1"*/
> > /*         oob_thread_pool.max_threads="8"*/
> > /*         oob_thread_pool.keep_alive_time="5000"*/
> > /*         oob_thread_pool.queue_enabled="false"*/
> > /*         oob_thread_pool.queue_max_size="100"*/
> > /*         oob_thread_pool.rejection_policy="discard"/>*/
> > /*
> > */
> > /*    <MPING ip_ttl="8"*/
> > /*           mcast_addr="224.1.2.3"*/
> > /*           mcast_port="7801"*/
> > /*           send_on_all_interfaces="true"*/
> > /*           receive_on_all_interfaces="true"/>*/
> > /*
> > */
> > /*    <MERGE3  min_interval="10000"*/
> > /*             max_interval="30000"/>*/
> > /*    <FD_SOCK client_bind_port="40718" start_port="40718"/>*/
> > /*    <FD timeout="3000" max_tries="3" />*/
> > /*    <VERIFY_SUSPECT timeout="1500"  />*/
> > /*    <BARRIER />*/
> > /*    <pbcast.NAKACK2 use_mcast_xmit="false"*/
> > /*                    discard_delivered_msgs="true"/>*/
> > /*    <UNICAST3 />*/
> > /*    <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"*/
> > /*                   max_bytes="4M"/>*/
> > /*    <pbcast.GMS print_local_addr="true" join_timeout="5000"*/
> > /*                view_bundling="true"/>*/
> > /*    <MFC max_credits="2M"*/
> > /*         min_threshold="0.4"/>*/
> > /*    <FRAG2 frag_size="60K"  />*/
> > /*    <!--RSVP resend_interval="2000" timeout="10000"/-->*/
> > /*    <pbcast.STATE_TRANSFER/>*/
> > /*</config>*/
> > /*
> > */
> > /*
> > */
> > Please let me know if you need more information.
> > --
> > *Regards,
> > *Venkat
>
> --
> Bela Ban, JGroups lead (http://www.jgroups.org)
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>



-- 

*Regards,*Venkat

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users