Re: [jgroups-users] JGroups failing to give up on crashed coordinator

I marked my inline responses with [bela]

On 28/01/16 23:07, Questions/problems related to using JGroups wrote:

>     On 26/01/16 02:51, Questions/problems related to using JGroups wrote:
>     > This is on 3.6.4 final.
>     >
>     > This problem originally occurred as a coincidence with suspect hardware
>     > but it is  reproducible by resetting/cycling power on multiple nodes
>     > including the coordinator in quick succession.
>
>     I assume not a 'shutdown -r now'?
>
> The scenario starts with a hard crash or power cycle, which when I
> instigated it I'm directly resetting the processor.  I've not be able to
> reproduce it without the hardware going down which makes me think the
> network state / network hardware is a contributing factor to the problem.


[bela]
What's directly resetting the processor? This is just unplugging the 
power cable, isn't it?
Do you think a reproducer could done with virtualized instances, e.g. 
VirtualBox?


>     > If I reset only the  coordinator or if I reset a group of nodes not including the coordinator
>     > I don't have a problem.
>
>     The problem is? What's giving up on a crashed coordinator? Still trying
>     to ask a crashed coord for cluster admission? Failing to elect a new
>     coord?
>
>
> The problem is that on some nodes they do not get a new view.  The
> coordinator has crashed and is restarting / rejoining the cluster.  Once
> all systems are up and running again they do not have common views.  So
> if I have A|3{A,B,C,D,E} and kill A and B then I could have (all letters
> correspond to the node names not the UUIDs):
> on A/B/C (after A and B restart):
> C|5(3){C,A,B}
> on D and E:
> A|3(5){A, B, C, D, E}
>
> There will be logs like this on Nodes D and E:
> W/NAKACK2: JGRP000011: D: dropped message 177 from non-member A
> (view=[A|5] (5) [A, B, C, D, E]) (received 29 identical messages from A
> in the last 67134 ms)
> FD_ALL: haven't received a heartbeat from A for 183291 ms, adding it to
> suspect list
> FD_SOCK: D: unsuspecting A
> I'm tempted to change the naming of my nodes to include a sequence
> number or something so that I can see the difference between A and A-prime


[bela]
Let's focus on the killing first, without even looking at restarting a 
process. FD_ALL should suspect the killed members including coords, 
next-in-lines etc, and eventually establish a new view. In your config 
this should be approximately done after 12 seconds.

I assume the non-killed members have correct communication to each 
other? This is needed to disseminate the SUSPECT messages and to install 
new views.

Re restarted members: the restarted members have a new UUID as address, 
so they should never get unsuspected. Or do you by any chance set 
prefabricated UUIDs into nodes?


>     > My expectation would be that JGroups would install a new view
>     > without the crashed nodes or even a view containing just the individual
>     > nodes.
>     So if you have {A,B,C,D} and power down A, then B, C and D should all
>     install view {B,C,D}, after roughly 12 seconds. Is this not the case?
>     Does this happen in a wired network, or only in the wireless network?
>
> The network is a wireless peer-to-peer so anything can happen.  When
> things were broken I manually pinged from the nodes with the old view
> and the multicast ping worked somewhat and the unicast ping not at all.

[bela]
What's a multicast and unicast ping ? FD_ALL and FD, respectively?

> Nodes that have managed to form a cluster with a new view can ping each
> other but not the nodes not in their view.

[bela]
Well, at least thats consistent; or else there'd be a merge...
Do you have full connectivity from every healthy member to all other 
healthy members, after a power cycle on multiple members?


>     > I have included some logs captured  on node 11 well after the resets
>     > occurred.  In this case it was node 7,13, and 14 that I reset where 7
>     > was coordinator, node 14 was the next node in the view, node 13 was in
>     > about the middle, and node 11 was near the end.   I have a log in the
>     > suspect callback that does not ever get generated in this scenario.
>
>     I don't see GMS in here, can you enable it? We should see either the
>     coord, or the next-in-line to become coord if the coord crashed try to
>     install a new view...
>
> I have pbcast.GMS in the stack, but I don't see any logs being generated
> by it.

[bela]
If you have {A,B,C,D,E} and crash A and B, then C should handle the 
suspicions, remove A and B from its eligible members set and become new 
coordinator. You should see it handling the suspects, and subsequently 
creating and disseminating the new view.
Are you sure you have trace logging enabled for GMS? Check with 
'probe.sh jmx=GMS.level'.



>     OK, here's how I'd like to proceed. You need to come up with the
>     simplest *reproducible* scenario that I can test on my end (with your
>     config), e.g. powering down A in {A,B,C,D}.
>
> I seriously doubt I can since I don't even know exactly what is happening to the network.

[bela]
Is this reproducible, e.g. every time you power down the current coord 
(A), you run into this issue?

> I'm pretty sure that the network is failing in a way that is confusing JGroups

[bela]
Wrt failure detection (FD_ALL), this is pretty simple: either members 
get heartbeats, or they don't. In the latter case, members from which we 
have't received updates for a while will get excluded and a new view 
installed. But since a view is also a message, views can get dropped, 
and JGroups will keep retransmitting them until they successfully get 
delivered, or until that member crashes.

So if you've got partial communication problems, JGroups will happily 
retry doing things, or stop doing them when the target member(s) leaves 
the cluster. Very simple and deterministic model.

> and it seems that installing a new view, even if it is a view of one, requires the network to either work
> correctly or not at all -- not something in-between.

[bela]
Well, a view V5 created and installed by cord C is delivered to all 
members. Say member S doesn't get V5. What can happen:
- *If* S detects its missing V5, it asks C for retransmission and C 
retransmits V5 to S
- C keeps retransmitting V5 until (a) V5 has been received by S or (b) S 
is suspected and excluded
- So if we have intermittend connectivity, that isn't long enough to 
exclude S, the the chances are V5 will be delivered to S



> After A and B cycle FD_ALL is correctly suspecting them due to lack of
> heartbeat.

[bela] OK, so at least this part works! :-)

> On node E from my example above, given that it thinks old A
> is the coordinator and it's Address/UUID cannot still be running I don't
> understand how FD_SOCK can unsuspect A.

[bela]
Yes, this is strange. You could try removing VERIFY_SUSPECT to see what 
this does.

> Do parts of the system work on UUID and other parts on ip address or name?

[bela]
No, members are always identified by UUID, never by IP address:port or 
even name.

So above you wrote that member do get suspected and removed correctly? 
Is this the case? If so, we can focus on the restarting part, but I 
thought you said that not even this works in your previous email...

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140