Re: [jgroups-users] JGroups failing to give up on crashed coordinator

On Thu, Jan 28, 2016 at 9:55 AM, Questions/problems related to using
JGroups <[email protected]> wrote:

>
>
> On 26/01/16 02:51, Questions/problems related to using JGroups wrote:
> > This is on 3.6.4 final.
> >
> > This problem originally occurred as a coincidence with suspect hardware
> > but it is  reproducible by resetting/cycling power on multiple nodes
> > including the coordinator in quick succession.
>
> I assume not a 'shutdown -r now'?
>
The scenario starts with a hard crash or power cycle, which when I
instigated it I'm directly resetting the processor.  I've not be able to
reproduce it without the hardware going down which makes me think the
network state / network hardware is a contributing factor to the problem.


>
> > If I reset only the  coordinator or if I reset a group of nodes not
> including the coordinator
> > I don't have a problem.
>
> The problem is? What's giving up on a crashed coordinator? Still trying
> to ask a crashed coord for cluster admission? Failing to elect a new coord?
>

The problem is that on some nodes they do not get a new view.  The
coordinator has crashed and is restarting / rejoining the cluster.  Once
all systems are up and running again they do not have common views.  So if
I have A|3{A,B,C,D,E} and kill A and B then I could have (all letters
correspond to the node names not the UUIDs):
on A/B/C (after A and B restart):
C|5(3){C,A,B}
on D and E:
A|3(5){A, B, C, D, E}

There will be logs like this on Nodes D and E:
W/NAKACK2: JGRP000011: D: dropped message 177 from non-member A (view=[A|5]
(5) [A, B, C, D, E]) (received 29 identical messages from A in the last
67134 ms)
FD_ALL: haven't received a heartbeat from A for 183291 ms, adding it to
suspect list
FD_SOCK: D: unsuspecting A
I'm tempted to change the naming of my nodes to include a sequence number
or something so that I can see the difference between A and A-prime

In this state nodes D and E will generate enough traffic that they keep the
network dysfunctional.  If I stop the software on enough nodes the network
functionality will restore and any nodes still running, including ones
previously stuck on the old view, will form a cluster.


> > It also doesn't happen when I killed the
> > software processes but don't reset the hardware so the actual loss of
> > the wireless network destination is an important contributor to the
> > problem.
>
> So can this be reproduced by iptables, mimicking a network loss?
>

I'll have to look into that.


>
> > My expectation would be that JGroups would install a new view
> > without the crashed nodes or even a view containing just the individual
> > nodes.
>
>
> So if you have {A,B,C,D} and power down A, then B, C and D should all
> install view {B,C,D}, after roughly 12 seconds. Is this not the case?
> Does this happen in a wired network, or only in the wireless network?
>
> The network is a wireless peer-to-peer so anything can happen.  When
things were broken I manually pinged from the nodes with the old view and
the multicast ping worked somewhat and the unicast ping not at all.  Nodes
that have managed to form a cluster with a new view can ping each other but
not the nodes not in their view.


> >  As best as I can tell the nodes in this stuck state are creating
> > a self-perpetuating network problem that never clears most likely due to
> > the resulting traffic storm (I see a huge spike in network layer packet
> > retransmissions -- to as much as 50% of all traffic).  If I subsequently
> > stop JGroups on all nodes I can then restart with no problems.
>
>
> This shouldn't happen: a powered-down node should get excluded.
> I can try to reproduce this in my own environment, but I'm pretty sure
> it will work. So do you have a special env? Do you want me to try?
>
> I'll have to see if I can come up with an iptables config that behaves
similarly otherwise I'm sure you can't reproduce it.


>
> > I have included some logs captured  on node 11 well after the resets
> > occurred.  In this case it was node 7,13, and 14 that I reset where 7
> > was coordinator, node 14 was the next node in the view, node 13 was in
> > about the middle, and node 11 was near the end.   I have a log in the
> > suspect callback that does not ever get generated in this scenario.
>
> I don't see GMS in here, can you enable it? We should see either the
> coord, or the next-in-line to become coord if the coord crashed try to
> install a new view...
>
> I have pbcast.GMS in the stack, but I don't see any logs being generated
by it.  It seems like the problem only occurs if I crash the coordinator
and the next in line.  Is the third in line prepared to step in?   Is there
some data that is being lost since I crash both?



>
> > This is what the stack looks like and it seems to work pretty well most
> > of the time.  I've tried varying timeouts, removing VERIFY and FD_SOCK,
> > increasing bundle and frag sizes and nothing seems to prevent this.
> > I've run with abort as the rejection policy to see if maybe I'm
> > overflowing my queues but I never saw an abort.
>
> Can you try disabling the queue in all 3 thread pools?
> Also, remove the timer section as the default is usually pretty good.
>
> I'll give that a try to see if that makes a difference.



> OK, here's how I'd like to proceed. You need to come up with the
> simplest *reproducible* scenario that I can test on my end (with your
> config), e.g. powering down A in {A,B,C,D}.
>
> I seriously doubt I can since I don't even know exactly what is happening
to the network.  I'm pretty sure that the network is failing in a way that
is confusing JGroups and it seems that installing a new view, even if it is
a view of one, requires the network to either work correctly or not at all
-- not something in-between.

After A and B cycle FD_ALL is correctly suspecting them due to lack of
heartbeat.  On node E from my example above, given that it thinks old A is
the coordinator and it's Address/UUID cannot still be running I don't
understand how FD_SOCK can ususpect A.  Do parts of the system work on UUID
and other parts on ip address or name?  When FD_SOCK unsuspects the crashed
coordinator presumably it is doing it on the basis of seeing the new
A-prime on the network.

I think that if the stuck nodes would install a view of one member then
start the join over again the cluster might have a chance of reforming.  I
could try to detect the condition but I've not designed my software to be
able to disconnect and reconnect to the channel, I would most likely have
to reboot the node which could easily perpetuate the problem.  If I had a
way of telling JGroups to force-install a new view of just the one node
then that might work.


>
> > <!--
> >    Stack which matches the one setup programmatically.
> >    Variable substitution is used for cluster/node specific bind
> > address/port values.
> > -->
> >
> > <config xmlns="urn:org:jgroups"
> >          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> >          xsi:schemaLocation="urn:org:jgroups
> > http://www.jgroups.org/schema/jgroups.xsd">
> >      <UDP
> >          enable_diagnostics="true"
> >          ip_mcast="true"
> >          ip_ttl="8"
> >          tos="8"
> >          mcast_port="${jgroups.udp.mcast_port:45588}"
> >          bind_addr="${jgroups.udp.bind_addr:127.0.0.1}"
> >          bind_port="0"
> >
> >          ucast_recv_buf_size="200000"
> >          ucast_send_buf_size="200000"
> >          mcast_recv_buf_size="200000"
> >          mcast_send_buf_size="200000"
> >
> >          max_bundle_size="1400"
> >          max_bundle_timeout="25"
> >
> >          thread_naming_pattern="pcl"
> >
> >          thread_pool_enabled="true"
> >          thread_pool_min_threads="1"
> >          thread_pool_max_threads="16"
> >          thread_pool_keep_alive_time="10000"
> >          thread_pool_queue_enabled="true"
> >          thread_pool_queue_max_size="50"
> >          thread_pool_rejection_policy="discard"
> >
> >          internal_thread_pool_enabled="true"
> >          internal_thread_pool_min_threads="2"
> >          internal_thread_pool_max_threads="16"
> >          internal_thread_pool_keep_alive_time="10000"
> >          internal_thread_pool_queue_enabled="true"
> >          internal_thread_pool_queue_max_size="50"
> >          internal_thread_pool_rejection_policy="discard"
> >
> >          oob_thread_pool_enabled="true"
> >          oob_thread_pool_min_threads="1"
> >          oob_thread_pool_max_threads="32"
> >          oob_thread_pool_keep_alive_time="10000"
> >          oob_thread_pool_queue_enabled="true"
> >          oob_thread_pool_queue_max_size="50"
> >          oob_thread_pool_rejection_policy="discard"
> >
> >          timer_type="new3"
> >          timer_min_threads="2"
> >          timer_max_threads="8"
> >          timer_keep_alive_time="3000"
> >          timer_queue_max_size="50"
> >          timer_rejection_policy="discard"
> >      />
> >      <PING />
> >      <MERGE3
> >          min_interval="2500"
> >          max_interval="5000"
> >          check_interval="16000"
> >      />
> >      <FD_SOCK
> >           bind_addr="${jgroups.udp.bind_addr:127.0.0.1}"
> >      />
> >      <FD_ALL
> >         interval="3000"
> >         timeout="10000"
> >         msg_counts_as_heartbeat="false"
> >      />
> >      <VERIFY_SUSPECT
> >          timeout="1500"
> >          use_mcast_rsps="false"
> >      />
> >      <BARRIER />
> >      <pbcast.NAKACK2
> >          xmit_interval="500"
> >          xmit_table_num_rows="100"
> >          xmit_table_msgs_per_row="2000"
> >          xmit_table_max_compaction_time="30000"
> >          max_msg_batch_size="500"
> >          use_mcast_xmit="false"
> >          discard_delivered_msgs="true"
> >      />
> >      <UNICAST3
> >          conn_expiry_timeout="0"
> >          xmit_interval="500"
> >          xmit_table_num_rows="100"
> >          xmit_table_msgs_per_row="2000"
> >          xmit_table_max_compaction_time="60000"
> >          max_msg_batch_size="500"
> >      />
> >      <FRAG2
> >          frag_size="1250"
> >      />
> >      <RSVP
> >          ack_on_delivery="false"
> >          resend_interval="250"
> >          timeout="5000"
> >          throw_exception_on_timeout="true"
> >      />
> >      <pbcast.STABLE
> >          stability_delay="1000"
> >          desired_avg_gossip="15000"
> >          max_bytes="4M"
> >          send_stable_msgs_to_coord_only="false"
> >      />
> >      <pbcast.GMS
> >          max_join_attempts="100000"
> >          print_local_addr="true"
> >          view_bundling="true"
> >          join_timeout="15000"
> >          leave_timeout="15000"
> >          use_flush_if_present="true"
> >          use_merger2="true"
> >          print_physical_addrs="true"
> >          view_ack_collection_timeout="10000"
> >          merge_timeout="10000"
> >      />
> >      <pbcast.STATE
> >         buffer_size="10000"
> >         max_pool="5"
> >         pool_thread_keep_alive="10000"
> >      />
> > </config>
> >
> > Any ideas?
> >
> > Thanks,
> > JT
>
>
>
> --
> Bela Ban, JGroups lead (http://www.jgroups.org)
>
>
>
> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users