Re: [jgroups-users] JGroups failing to give up on crashed coordinator

On 29/01/16 19:03, Questions/problems related to using JGroups wrote:



>     [bela]
>     Let's focus on the killing first, without even looking at restarting a
>     process. FD_ALL should suspect the killed members including coords,
>     next-in-lines etc, and eventually establish a new view. In your config
>     this should be approximately done after 12 seconds.
>
> [jt]
> Based on the logs FD_ALL is working correctly.

[bela]
OK


>     I assume the non-killed members have correct communication to each
>     other? This is needed to disseminate the SUSPECT messages and to install
>     new views.
>
> [jt]
> No when the condition occurs they don't have properly functioning
> communication


[bela]

I fail to see the causal relation between killing a member and this 
affecting the network, can you shed some light on why this is the case?



>     [bela]
>     Well, at least thats consistent; or else there'd be a merge...
>     Do you have full connectivity from every healthy member to all other
>     healthy members, after a power cycle on multiple members?
>
> [jt]
> subsets of non restarted nodes appear have full connectivity to each
> other and form a cluster, others seem completely isolated for unicast
> but maybe not for multicast and they fail to replace their old view with
> a new one.

[bela]
So I assume you're killing the members _while there is a network 
condition_. As mentioned above, I can't see any reason for killed 
members to affect the network.

The above scenario you describe will of course affect operation and 
cause all sorts of effects, e.g. members suspecting others, views not 
getting disseminated and so on.
However, once the network heals, JGroups should be able to regroup and 
form a cluster consisting of all members.


>     [bela]
>     Well, a view V5 created and installed by cord C is delivered to all
>     members. Say member S doesn't get V5. What can happen:
>     - *If* S detects its missing V5, it asks C for retransmission and C
>     retransmits V5 to S
>     - C keeps retransmitting V5 until (a) V5 has been received by S or (b) S
>     is suspected and excluded
>     - So if we have intermittend connectivity, that isn't long enough to
>     exclude S, the the chances are V5 will be delivered to S
>
> [jt]
> Yeah so I'm thinking maybe it is not intermittent connectivity rather it
> is multicast works and unicast does not.


[bela] Any idea what's causing this? Is this a condition that's true for 
the entire duration of your test, or does it kick in some time after you 
kill your members?


>  That should be a scenario that
> it is easy to implement with iptables.  It also makes some sense given
> the way the wireless works.

[bela]
I'm no expert in wireless networks, but why would unicasts not work? 
I've never seen this...


>     > Do parts of the system work on UUID and other parts on ip address or name?
>
>     [bela]
>     No, members are always identified by UUID, never by IP address:port or
>     even name.
>
> [jt]
> So if this is the case then I think there may be a problem with
> FD_SOCK.  Either FD_SOCK is unsuspecting the new A UUID which is not in
> our view (because our view is still the one generated before A ever
> rebooted) or it is unsuspecting the original A UUID which has crashed
> and is not around anymore.  But removing FD_SOCK from the stack did not
> solve the problem so this is mainly a curiosity.

[bela]
I looked at the code, and this isn't the case: FD_SOCK operates on UUIDs 
and reincarnation is not possible.


>     So above you wrote that member do get suspected and removed correctly?
>     Is this the case? If so, we can focus on the restarting part, but I
>     thought you said that not even this works in your previous email...
>
> [jt]
> on some of the nodes it works on others it does not.  Some sort of
> network isolation is occurring.  So if I'm a node which cannot perform a
> successful unicast to nodes that i can (multicast) ping will I be stuck
> or can I install a view?

[bela]
You will be running into all sorts of problems, to say the least.

I once had a problem reported where they could not use unicasts and had 
to send multicasts all the time. The solution was to create a protocol 
which wrapped unicasts (e.g. the dest address) as multicasts and 
unwrapped them at the destination.
This fixed the problem, but of course comes with overhead, as every 
receiver needs to do the unwrapping and then check the message's dest 
address against its own, to reject and accept the message.
That wasn't you by any chance?

>  If I'm stuck can we detect this and decide that I must be coordinator with no members?

That's what FD_ALL is supposed to do.

>  A long time ago I set the GMS max_join_attempts to a very large number because I had nodes giving
> up on joining an existing cluster because I had some timeouts set too
> short.  I'm wondering if that is having an impact in this scenario.

Could be. Sometimes it's better to give up and become a singleton coord, 
rather than trying to join a cluster whose coord is shot.

> I appreciate the help, even if it is a bit academic at this point.
> After today I'm going to be away for a few weeks so I don't expect to
> figure out much more until later in Feb.

sure


-- 
Bela Ban, JGroups lead (http://www.jgroups.org)


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140