Re: [jgroups-users] jgroup cluster - member evictions

On 19/04/18 12:15, Questions/problems related to using JGroups wrote:
> Hello,
> 
> We are using an application which uses jgroups(version 3.6.2) for 
> cluster communication. This application a 4-node cluster deployed on 
> Oracle Weblogic 12.1.3 with Oracle database.
> 
> In this, we recently started noticing below messages.
> 
> /Possible VM pause detected of between 108676ms and 109676ms/

Does this mean that the VM was paused for 1-2 minutes?

> /Failed invocation, cluster member [XXXXX-14772] timed out/
> 
> //
> 
> /Unable to retrieve the status of all cluster nodes/

Well, this means that there was a blocking RPC to all cluster nodes, and 
some members did not reply within the timeout, probably because they 
were paused by GC and couldn't reply in time.


> /(received 14 identical messages from XXXXXXXXXXX-57924 in the last 
> 60074 ms)/
> 
> //
> 
> /org.jgroups.protocols.pbcast.NAKACK2 - JGRP000011: XXXXXXXXX-56540: 
> dropped message 3104 from non-member XXXXXXXXXXX-57924/

Obviously, FD or FD_ALL suspected member 57924 and is now dropping 
messages from it as a result. MERGE{2/3} will bring that member back 
into the cluster once the pause is over.


> Any idea what’s causing these messages?

See above. The ultimate solution is to fix the cause of the pauses, or 
at least be able to handle them, e.g. by reducing the RPC timeout and 
handling missing responses accordingly.

> Regards
> 
> Sree
> 
> **********************************************************************************************************************************************************************************************************************************************************************************************************
*******************************************************************************************************************************************************************************************************************************************************************************************************
> 
> 
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> 
> 
> 
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
> 

-- 
Bela Ban | http://www.jgroups.org


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot