Re: [jgroups-users] JGroups failing to give up on crashed coordinator

On 26/01/16 02:51, Questions/problems related to using JGroups wrote:
> This is on 3.6.4 final.
>
> This problem originally occurred as a coincidence with suspect hardware
> but it is  reproducible by resetting/cycling power on multiple nodes
> including the coordinator in quick succession.

I assume not a 'shutdown -r now'?

> If I reset only the  coordinator or if I reset a group of nodes not including the coordinator
> I don't have a problem.

The problem is? What's giving up on a crashed coordinator? Still trying 
to ask a crashed coord for cluster admission? Failing to elect a new coord?

> It also doesn't happen when I killed the
> software processes but don't reset the hardware so the actual loss of
> the wireless network destination is an important contributor to the
> problem.

So can this be reproduced by iptables, mimicking a network loss?

> My expectation would be that JGroups would install a new view
> without the crashed nodes or even a view containing just the individual
> nodes.


So if you have {A,B,C,D} and power down A, then B, C and D should all 
install view {B,C,D}, after roughly 12 seconds. Is this not the case?
Does this happen in a wired network, or only in the wireless network?


>  As best as I can tell the nodes in this stuck state are creating
> a self-perpetuating network problem that never clears most likely due to
> the resulting traffic storm (I see a huge spike in network layer packet
> retransmissions -- to as much as 50% of all traffic).  If I subsequently
> stop JGroups on all nodes I can then restart with no problems.


This shouldn't happen: a powered-down node should get excluded.
I can try to reproduce this in my own environment, but I'm pretty sure 
it will work. So do you have a special env? Do you want me to try?


> I have included some logs captured  on node 11 well after the resets
> occurred.  In this case it was node 7,13, and 14 that I reset where 7
> was coordinator, node 14 was the next node in the view, node 13 was in
> about the middle, and node 11 was near the end.   I have a log in the
> suspect callback that does not ever get generated in this scenario.

I don't see GMS in here, can you enable it? We should see either the 
coord, or the next-in-line to become coord if the coord crashed try to 
install a new view...


> This is what the stack looks like and it seems to work pretty well most
> of the time.  I've tried varying timeouts, removing VERIFY and FD_SOCK,
> increasing bundle and frag sizes and nothing seems to prevent this.
> I've run with abort as the rejection policy to see if maybe I'm
> overflowing my queues but I never saw an abort.

Can you try disabling the queue in all 3 thread pools?
Also, remove the timer section as the default is usually pretty good.

OK, here's how I'd like to proceed. You need to come up with the 
simplest *reproducible* scenario that I can test on my end (with your 
config), e.g. powering down A in {A,B,C,D}.


> <!--
>    Stack which matches the one setup programmatically.
>    Variable substitution is used for cluster/node specific bind
> address/port values.
> -->
>
> <config xmlns="urn:org:jgroups"
>          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>          xsi:schemaLocation="urn:org:jgroups
> http://www.jgroups.org/schema/jgroups.xsd">
>      <UDP
>          enable_diagnostics="true"
>          ip_mcast="true"
>          ip_ttl="8"
>          tos="8"
>          mcast_port="${jgroups.udp.mcast_port:45588}"
>          bind_addr="${jgroups.udp.bind_addr:127.0.0.1}"
>          bind_port="0"
>
>          ucast_recv_buf_size="200000"
>          ucast_send_buf_size="200000"
>          mcast_recv_buf_size="200000"
>          mcast_send_buf_size="200000"
>
>          max_bundle_size="1400"
>          max_bundle_timeout="25"
>
>          thread_naming_pattern="pcl"
>
>          thread_pool_enabled="true"
>          thread_pool_min_threads="1"
>          thread_pool_max_threads="16"
>          thread_pool_keep_alive_time="10000"
>          thread_pool_queue_enabled="true"
>          thread_pool_queue_max_size="50"
>          thread_pool_rejection_policy="discard"
>
>          internal_thread_pool_enabled="true"
>          internal_thread_pool_min_threads="2"
>          internal_thread_pool_max_threads="16"
>          internal_thread_pool_keep_alive_time="10000"
>          internal_thread_pool_queue_enabled="true"
>          internal_thread_pool_queue_max_size="50"
>          internal_thread_pool_rejection_policy="discard"
>
>          oob_thread_pool_enabled="true"
>          oob_thread_pool_min_threads="1"
>          oob_thread_pool_max_threads="32"
>          oob_thread_pool_keep_alive_time="10000"
>          oob_thread_pool_queue_enabled="true"
>          oob_thread_pool_queue_max_size="50"
>          oob_thread_pool_rejection_policy="discard"
>
>          timer_type="new3"
>          timer_min_threads="2"
>          timer_max_threads="8"
>          timer_keep_alive_time="3000"
>          timer_queue_max_size="50"
>          timer_rejection_policy="discard"
>      />
>      <PING />
>      <MERGE3
>          min_interval="2500"
>          max_interval="5000"
>          check_interval="16000"
>      />
>      <FD_SOCK
>           bind_addr="${jgroups.udp.bind_addr:127.0.0.1}"
>      />
>      <FD_ALL
>         interval="3000"
>         timeout="10000"
>         msg_counts_as_heartbeat="false"
>      />
>      <VERIFY_SUSPECT
>          timeout="1500"
>          use_mcast_rsps="false"
>      />
>      <BARRIER />
>      <pbcast.NAKACK2
>          xmit_interval="500"
>          xmit_table_num_rows="100"
>          xmit_table_msgs_per_row="2000"
>          xmit_table_max_compaction_time="30000"
>          max_msg_batch_size="500"
>          use_mcast_xmit="false"
>          discard_delivered_msgs="true"
>      />
>      <UNICAST3
>          conn_expiry_timeout="0"
>          xmit_interval="500"
>          xmit_table_num_rows="100"
>          xmit_table_msgs_per_row="2000"
>          xmit_table_max_compaction_time="60000"
>          max_msg_batch_size="500"
>      />
>      <FRAG2
>          frag_size="1250"
>      />
>      <RSVP
>          ack_on_delivery="false"
>          resend_interval="250"
>          timeout="5000"
>          throw_exception_on_timeout="true"
>      />
>      <pbcast.STABLE
>          stability_delay="1000"
>          desired_avg_gossip="15000"
>          max_bytes="4M"
>          send_stable_msgs_to_coord_only="false"
>      />
>      <pbcast.GMS
>          max_join_attempts="100000"
>          print_local_addr="true"
>          view_bundling="true"
>          join_timeout="15000"
>          leave_timeout="15000"
>          use_flush_if_present="true"
>          use_merger2="true"
>          print_physical_addrs="true"
>          view_ack_collection_timeout="10000"
>          merge_timeout="10000"
>      />
>      <pbcast.STATE
>         buffer_size="10000"
>         max_pool="5"
>         pool_thread_keep_alive="10000"
>      />
> </config>
>
> Any ideas?
>
> Thanks,
> JT



-- 
Bela Ban, JGroups lead (http://www.jgroups.org)


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140