[jgroups-users] Large number of messages in NAKACK retransmission table

I am having issues with JGroups where a large number of messages are
sitting in the NAKACK retransmission table and I'd like some suggestions as
to possible ways to investigate the issue and reconfiguration options to
get the best performance out of JGroups given the current network
limitations.

The environment is:

   - JGroups 2.12.1 (can't change this - within a 3rd party product)
   - Network:
      - Approx 14 Nodes on the main subnet at 1Gbps
      - 2 Nodes on satellite subnet routed via a 2Mbps link
      - 2 Nodes on satellite subnet routed via a 256Kbps link
      - Host OS:
      - 1 x RHEL (sends the majority of the messages)
      - 17 x Windows 7
      - Current config:
       <UDP
           mcast_addr="..."
           mcast_port="..."
           discard_incompatible_packets="true"
           max_bundle_size="60000"
           max_bundle_timeout="30"
           ip_ttl="32"
           thread_pool.enabled="true"
           thread_pool.min_threads="1"
           thread_pool.max_threads="25"
           thread_pool.keep_alive_time="5000"
           thread_pool.queue_enabled="false"
           thread_pool.queue_max_size="100"
           thread_pool.rejection_policy="Run"
           oob_thread_pool.enabled="true"
           oob_thread_pool.min_threads="1"
           oob_thread_pool.max_threads="8"
           oob_thread_pool.keep_alive_time="5000"
           oob_thread_pool.queue_enabled="false"
           oob_thread_pool.queue_max_size="100"
           oob_thread_pool.rejection_policy="Run"
           mcast_send_buf_size="150000"
           mcast_recv_buf_size="80000"
           loopback="false"
           enable_bundling="true"
           enable_diagnostics="true"
       />
       <PING
           timeout="2000"
           num_initial_members="3"
       />
       <MERGE2
           min_interval="5000"
           max_interval="10000"
       />
       <FD
           timeout="2000"
           max_tries="3"
       />
       <VERIFY_SUSPECT
           timeout="1500"
       />
       <pbcast.NAKACK
           gc_lag="50"
           retransmit_timeout="300,600,1200,2400,4800"
           use_mcast_xmit="false"
           discard_delivered_msgs="true"
       />
       <UNICAST
           timeout="1200,2400,3600"
       />
       <pbcast.STABLE
           desired_avg_gossip="20000"
           stability_delay="1000"
           max_bytes="0"
       />
       <FRAG
           frag_size="8192"
       />
       <pbcast.GMS
           join_timeout="3000"
           print_local_addr="false"
           view_bundling="true"
       />
       <FC
           max_credits="20000000"
           min_threshold="0.10"
       />

If hosts on the satellite networks are connected, then we see the NAKACK
retransmission table in the sender (the RHEL box) grow until a Java OOM
exception is thrown.  We're thinking that this is most likely due to slow
response times caused by the 256Kbps link, but this has yet to be confirmed.

If no hosts on the satellite networks are connected, then we see the NAKACK
retransmission table in the sender (as seen by probe) sits between
2000-5000 messages.  I'm yet to graph the progression, but it appears to
grow for around 10-20 minutes then drop down, but it never gets close to
0.  For example, the digest output might look like:

mct001-24908:
WKS005-857: [55450 : 55552 (55552)]
WKS006-64076: [39473 : 39575 (39575)]
WKS009-5180: [268164 : 268264 (268264)]
WKS018-33904: [39344 : 39445 (39445)]
mct001-24908: [1651826 : 1654956 (1654956) (size=3130, missing=1827,
highest stability=1651826)

I can see STABLE and STABILITY messages being sent (using TRACE on
STABLE).  The frequency changes, being from 1 minute to over 10 minute
intervals.

The first thing I'd love to know is whether the above retransmission table
behavior is within what would be considered "normal" behavior.  If not,
what other things should I start to look at?

Cheers,
-- 
Tony Gedge

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users