Re: [jgroups-users] JGroups failing to give up on crashed coordinator

On Fri, Jan 29, 2016 at 3:32 AM, Questions/problems related to using
JGroups <[email protected]> wrote:

> I marked my inline responses with [bela]
>
> On 28/01/16 23:07, Questions/problems related to using JGroups wrote:
>
> >     On 26/01/16 02:51, Questions/problems related to using JGroups wrote:
> >     > This is on 3.6.4 final.
> >     >
> >     > This problem originally occurred as a coincidence with suspect
> hardware
> >     > but it is  reproducible by resetting/cycling power on multiple
> nodes
> >     > including the coordinator in quick succession.
> >
> >     I assume not a 'shutdown -r now'?
> >
> > The scenario starts with a hard crash or power cycle, which when I
> > instigated it I'm directly resetting the processor.  I've not be able to
> > reproduce it without the hardware going down which makes me think the
> > network state / network hardware is a contributing factor to the problem.
>
>
> [bela]
> What's directly resetting the processor? This is just unplugging the
> power cable, isn't it?
> Do you think a reproducer could done with virtualized instances, e.g.
> VirtualBox?


[jt]
To reproduce the scenario I assert the reset pin on the processor which in
turn causes a power cycle.  Virtual instances would make sense if I can
come up with an iptables emulation of the network behavior.  The original
cause of the crashing was identified and corrected making the likelihood of
this happening again very low, so now I'm testing when I can to see if I
can identify/mitigate the odd behavior on the off chance we get
simultaneous crashing due to some other cause.



> >     > If I reset only the  coordinator or if I reset a group of nodes
> not including the coordinator
> >     > I don't have a problem.
> >
> >     The problem is? What's giving up on a crashed coordinator? Still
> trying
> >     to ask a crashed coord for cluster admission? Failing to elect a new
> >     coord?
> >
> >
> > The problem is that on some nodes they do not get a new view.  The
> > coordinator has crashed and is restarting / rejoining the cluster.  Once
> > all systems are up and running again they do not have common views.  So
> > if I have A|3{A,B,C,D,E} and kill A and B then I could have (all letters
> > correspond to the node names not the UUIDs):
> > on A/B/C (after A and B restart):
> > C|5(3){C,A,B}
> > on D and E:
> > A|3(5){A, B, C, D, E}
> >
> > There will be logs like this on Nodes D and E:
> > W/NAKACK2: JGRP000011: D: dropped message 177 from non-member A
> > (view=[A|5] (5) [A, B, C, D, E]) (received 29 identical messages from A
> > in the last 67134 ms)
> > FD_ALL: haven't received a heartbeat from A for 183291 ms, adding it to
> > suspect list
> > FD_SOCK: D: unsuspecting A
> > I'm tempted to change the naming of my nodes to include a sequence
> > number or something so that I can see the difference between A and
> A-prime
>
>
> [bela]
> Let's focus on the killing first, without even looking at restarting a
> process. FD_ALL should suspect the killed members including coords,
> next-in-lines etc, and eventually establish a new view. In your config
> this should be approximately done after 12 seconds.
>
> [jt]
Based on the logs FD_ALL is working correctly.



> I assume the non-killed members have correct communication to each
> other? This is needed to disseminate the SUSPECT messages and to install
> new views.
>
> [jt]
No when the condition occurs they don't have properly functioning
communication



> Re restarted members: the restarted members have a new UUID as address,
> so they should never get unsuspected. Or do you by any chance set
> prefabricated UUIDs into nodes?
>
> [jt]
I assign a name but the UUID is generated automatically in the normal way.


>
> >     > My expectation would be that JGroups would install a new view
> >     > without the crashed nodes or even a view containing just the
> individual
> >     > nodes.
> >     So if you have {A,B,C,D} and power down A, then B, C and D should all
> >     install view {B,C,D}, after roughly 12 seconds. Is this not the case?
> >     Does this happen in a wired network, or only in the wireless network?
> >
> > The network is a wireless peer-to-peer so anything can happen.  When
> > things were broken I manually pinged from the nodes with the old view
> > and the multicast ping worked somewhat and the unicast ping not at all.
>
> [bela]
> What's a multicast and unicast ping ? FD_ALL and FD, respectively?
>
> [jt]
On a console with the linux command line i use ping to the ip addresses of
other members and to the multicast ip address.



> > Nodes that have managed to form a cluster with a new view can ping each
> > other but not the nodes not in their view.
>
> [bela]
> Well, at least thats consistent; or else there'd be a merge...
> Do you have full connectivity from every healthy member to all other
> healthy members, after a power cycle on multiple members?
>
> [jt]
subsets of non restarted nodes appear have full connectivity to each other
and form a cluster, others seem completely isolated for unicast but maybe
not for multicast and they fail to replace their old view with a new one.


>
> >     > I have included some logs captured  on node 11 well after the
> resets
> >     > occurred.  In this case it was node 7,13, and 14 that I reset
> where 7
> >     > was coordinator, node 14 was the next node in the view, node 13
> was in
> >     > about the middle, and node 11 was near the end.   I have a log in
> the
> >     > suspect callback that does not ever get generated in this scenario.
> >
> >     I don't see GMS in here, can you enable it? We should see either the
> >     coord, or the next-in-line to become coord if the coord crashed try
> to
> >     install a new view...
> >
> > I have pbcast.GMS in the stack, but I don't see any logs being generated
> > by it.
>
> [bela]
> If you have {A,B,C,D,E} and crash A and B, then C should handle the
> suspicions, remove A and B from its eligible members set and become new
> coordinator. You should see it handling the suspects, and subsequently
> creating and disseminating the new view.
> Are you sure you have trace logging enabled for GMS? Check with
> 'probe.sh jmx=GMS.level'.
>
> [jt]
Ah no I didn't have trace enabled on it.


>
>
> >     OK, here's how I'd like to proceed. You need to come up with the
> >     simplest *reproducible* scenario that I can test on my end (with your
> >     config), e.g. powering down A in {A,B,C,D}.
> >
> > I seriously doubt I can since I don't even know exactly what is
> happening to the network.
>
> [bela]
> Is this reproducible, e.g. every time you power down the current coord
> (A), you run into this issue?
>
> [jt]
Not fully reproducible.  I don't think the the problem has occurred when
only the coordinator goes down or when a group that does not include the
coordinator goes down.  I think it has always required 3 or more out of 13
nodes, possibly needing the coordinator and next in line to go down.



> > I'm pretty sure that the network is failing in a way that is confusing
> JGroups
>
> [bela]
> Wrt failure detection (FD_ALL), this is pretty simple: either members
> get heartbeats, or they don't. In the latter case, members from which we
> have't received updates for a while will get excluded and a new view
> installed. But since a view is also a message, views can get dropped,
> and JGroups will keep retransmitting them until they successfully get
> delivered, or until that member crashes.
>
> So if you've got partial communication problems, JGroups will happily
> retry doing things, or stop doing them when the target member(s) leaves
> the cluster. Very simple and deterministic model.
>
> > and it seems that installing a new view, even if it is a view of one,
> requires the network to either work
> > correctly or not at all -- not something in-between.
>
> [bela]
> Well, a view V5 created and installed by cord C is delivered to all
> members. Say member S doesn't get V5. What can happen:
> - *If* S detects its missing V5, it asks C for retransmission and C
> retransmits V5 to S
> - C keeps retransmitting V5 until (a) V5 has been received by S or (b) S
> is suspected and excluded
> - So if we have intermittend connectivity, that isn't long enough to
> exclude S, the the chances are V5 will be delivered to S
>
> [jt]
Yeah so I'm thinking maybe it is not intermittent connectivity rather it is
multicast works and unicast does not.  That should be a scenario that it is
easy to implement with iptables.  It also makes some sense given the way
the wireless works.


>
> > After A and B cycle FD_ALL is correctly suspecting them due to lack of
> > heartbeat.
>
> [bela] OK, so at least this part works! :-)
>
> > On node E from my example above, given that it thinks old A
> > is the coordinator and it's Address/UUID cannot still be running I don't
> > understand how FD_SOCK can unsuspect A.
>
> [bela]
> Yes, this is strange. You could try removing VERIFY_SUSPECT to see what
> this does.
>
> [jt]
One of the first things I tried was removing VERIFY_SUSPECT and/or FD_SOCK
and it did not seem to make a difference.


> > Do parts of the system work on UUID and other parts on ip address or
> name?
>
> [bela]
> No, members are always identified by UUID, never by IP address:port or
> even name.
>
> [jt]
So if this is the case then I think there may be a problem with FD_SOCK.
Either FD_SOCK is unsuspecting the new A UUID which is not in our view
(because our view is still the one generated before A ever rebooted) or it
is unsuspecting the original A UUID which has crashed and is not around
anymore.  But removing FD_SOCK from the stack did not solve the problem so
this is mainly a curiosity.


> So above you wrote that member do get suspected and removed correctly?
> Is this the case? If so, we can focus on the restarting part, but I
> thought you said that not even this works in your previous email...
>
> [jt]
on some of the nodes it works on others it does not.  Some sort of network
isolation is occurring.  So if I'm a node which cannot perform a successful
unicast to nodes that i can (multicast) ping will I be stuck or can I
install a view?  If I'm stuck can we detect this and decide that I must be
coordinator with no members?  A long time ago I set the
GMS max_join_attempts to a very large number because I had nodes giving up
on joining an existing cluster because I had some timeouts set too short.
I'm wondering if that is having an impact in this scenario.

I appreciate the help, even if it is a bit academic at this point.  After
today I'm going to be away for a few weeks so I don't expect to figure out
much more until later in Feb.


> --
> Bela Ban, JGroups lead (http://www.jgroups.org)
>
>
>
> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users