Re: [jgroups-users] JGroups failing to give up on crashed coordinator

On 16/12/16 22:12, Questions/problems related to using JGroups wrote:
> Now that I have removed flush I occasionally get state transfer failures
> on the initial connection to the cluster because I'm getting a timeout
> from BARRIER.

It seems that you have OOB messages (or RPCs) which were received but 
haven't returned after 5 seconds. When BARRIER closes its latch, all new 
incoming messages are blocked, but existing messages are allowed to 
complete within 5s, then the state is transferred and the latch released 
again.

Take a look at the threads which don't complete within 5s and see if you 
can fix them, e.g. by making the max RPC timeout lower than the BARRIER 
timeout.

Alternatively, you could remove BARRIER and instead sync on your state 
when doing a state transfer.


> 12-16 17:52:54.330 E/Messaging( 1738): Caused by:
> org.jgroups.StateTransferException: state transfer failed
> 12-16 17:52:54.330 E/Messaging( 1738):    at
> org.jgroups.JChannel.getState(JChannel.java:614)
> 12-16 17:52:54.330 E/Messaging( 1738):    at
> org.jgroups.JChannel.getState(JChannel.java:540)
> 12-16 17:52:54.330 E/Messaging( 1738):    at
> org.jgroups.JChannel.connect(JChannel.java:344)
> 12-16 17:52:54.330 E/Messaging( 1738):    at
> org.jgroups.JChannel.connect(JChannel.java:301)
> 12-16 17:52:54.330 E/Messaging( 1738):    at
> com.novawurks.hsos.core.messaging.MessagingManager.start(MessagingManager.java:668)
> 12-16 17:52:54.330 E/Messaging( 1738):    ... 8 more
> 12-16 17:52:54.330 E/Messaging( 1738): Caused by:
> org.jgroups.TimeoutException: node_3: failed flushing pending threads in
> 5000 ms; threads:
> 12-16 17:52:54.330 E/Messaging( 1738):
> Thread[OOB-5,MAIN_CHANNEL,node_3,5,main]
> 12-16 17:52:54.330 E/Messaging( 1738):
> Thread[OOB-22,MAIN_CHANNEL,node_3,5,main]
> 12-16 17:52:54.330 E/Messaging( 1738):
> 12-16 17:52:54.330 E/Messaging( 1738):    at
> org.jgroups.protocols.BARRIER.closeBarrier(BARRIER.java:244)
> 12-16 17:52:54.330 E/Messaging( 1738):    at
> org.jgroups.protocols.BARRIER.down(BARRIER.java:115)
> 12-16 17:52:54.330 E/Messaging( 1738):    at
> org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:582)
> 12-16 17:52:54.330 E/Messaging( 1738):    at
> org.jgroups.protocols.UNICAST3.down(UNICAST3.java:682)
> 12-16 17:52:54.330 E/Messaging( 1738):    at
> org.jgroups.protocols.FRAG2.down(FRAG2.java:136)
> 12-16 17:52:54.330 E/Messaging( 1738):    at
> org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:347)
> 12-16 17:52:54.330 E/Messaging( 1738):    at
> org.jgroups.protocols.pbcast.GMS.down(GMS.java:1111)
> 12-16 17:52:54.330 E/Messaging( 1738):    at
> org.jgroups.protocols.pbcast.StreamingStateTransfer.closeBarrierAndSuspendStable(StreamingStateTransfer.java:307)
> 12-16 17:52:54.330 E/Messaging( 1738):    at
> org.jgroups.protocols.pbcast.StreamingStateTransfer.handleStateReq(StreamingStateTransfer.java:400)
>
>
> Would those OOB threads be for incoming or outgoing messages?


Incoming

> I get the callback prior to this error which may cause me to start sending
> some messages out.  On other nodes the view change will cause them to
> send messages to the new node right way.
>
> If I don't figure out what is causing this / how to avoid it soon I will
> be forced to put FLUSH back in.
>
> JT
>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>
>
>
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot