Re: [jgroups-users] Large number of messages in NAKACK retransmission table

On 20/02/16 14:48, Questions/problems related to using JGroups wrote:
> I am having issues with JGroups where a large number of messages are
> sitting in the NAKACK retransmission table and I'd like some suggestions
> as to possible ways to investigate the issue and reconfiguration options
> to get the best performance out of JGroups given the current network
> limitations.
>
> The environment is:
>
>   * JGroups 2.12.1 (can't change this - within a 3rd party product)

Very bad - I don't usually support questions on such old versions...

>   * Network:
>       o Approx 14 Nodes on the main subnet at 1Gbps
>       o 2 Nodes on satellite subnet routed via a 2Mbps link
>       o 2 Nodes on satellite subnet routed via a 256Kbps link

Not optimal; JGroups is a peer-to-peer solution where peers are supposed 
to be running on the same network, or at least have the same 
characteristics (bandwidth, latency etc).

If anyhow possible, I suggest a core cluster consisting of the 14 nodes 
on the main subnet, and _client peers_ on the satellite subnets 
conncting to one of the core members via separate means (e.g. TCP). The 
client members would not be part of the core cluster.

>   * Host OS:
>       o 1 x RHEL (sends the majority of the messages)
>       o 17 x Windows 7
>   * Current config:
>     <UDP
>              mcast_addr="..."
>              mcast_port="..."
>              discard_incompatible_packets="true"
>              max_bundle_size="60000"
>              max_bundle_timeout="30"
>              ip_ttl="32"
>              thread_pool.enabled="true"
>              thread_pool.min_threads="1"
>              thread_pool.max_threads="25"
>              thread_pool.keep_alive_time="5000"
>              thread_pool.queue_enabled="false"
>              thread_pool.queue_max_size="100"
>              thread_pool.rejection_policy="Run"

I suggest "abort" or "discard" ad "run" can block the receiver thread.

>              oob_thread_pool.enabled="true"
>              oob_thread_pool.min_threads="1"
>              oob_thread_pool.max_threads="8"
>              oob_thread_pool.keep_alive_time="5000"
>              oob_thread_pool.queue_enabled="false"
>              oob_thread_pool.queue_max_size="100"
>              oob_thread_pool.rejection_policy="Run"

Same here

>              mcast_send_buf_size="150000"
>              mcast_recv_buf_size="80000"
>              loopback="false"
>              enable_bundling="true"
>              enable_diagnostics="true"
>          />
>          <PING
>              timeout="2000"
>              num_initial_members="3"
>          />
>          <MERGE2
>              min_interval="5000"
>              max_interval="10000"
>          />
>          <FD
>              timeout="2000"
>              max_tries="3"
>          />

Use FD_ALL which is much better suited for UDP

>          <VERIFY_SUSPECT
>              timeout="1500"
>          />
>          <pbcast.NAKACK
>              gc_lag="50"
>              retransmit_timeout="300,600,1200,2400,4800"
>              use_mcast_xmit="false"
>              discard_delivered_msgs="true"
>          />

I guess NAKACK2 didn't exist back in 2.12 (haven't checked)?

>          <UNICAST
>              timeout="1200,2400,3600"
>          />

Hmm, today we're at UNICAST3... :-)

>          <pbcast.STABLE
>              desired_avg_gossip="20000"
>              stability_delay="1000"
>              max_bytes="0"
>          />

Setting max_bytes to 0 is bad, I suggest set it to 1MB. This is 1 of the 
2 things to fix your issue (discussion further down).

>          <FRAG
>              frag_size="8192"
>          />

FRAG2 if available

>          <pbcast.GMS
>              join_timeout="3000"
>              print_local_addr="false"
>              view_bundling="true"
>          />
>          <FC
>              max_credits="20000000"
>              min_threshold="0.10"
>          />

Reduce max_credits, e.g. to 2M. This will slow senders down and prevent 
OOME, this is part 2 of the solution. Also, if available, use MFC/UFC.


> If hosts on the satellite networks are connected, then we see the NAKACK
> retransmission table in the sender (the RHEL box) grow until a Java OOM
> exception is thrown.  We're thinking that this is most likely due to
> slow response times caused by the 256Kbps link, but this has yet to be
> confirmed.

If you set max_bytes to a value > 0, then message garbage collection 
will be done based on the accumulated number of bytes and _not only_ on 
a constant interval (20s). This, coupled with a reduced max_credits 
threshold, should prevent OOMEs.


> If no hosts on the satellite networks are connected, then we see the
> NAKACK retransmission table in the sender (as seen by probe) sits
> between 2000-5000 messages.  I'm yet to graph the progression, but it
> appears to grow for around 10-20 minutes then drop down, but it never
> gets close to 0.

It should!

STABLE runs every 20s max, and once the coord has STABLE messages from 
all members, it computes the min seqnos of all received messages and 
broadcasts a STABILITY message which causes all messages below the min 
vector to get purged in all senders.

Of course, if a member is really slow, yet doesn't get suspected and 
excluded, it will prevent progress in STABLE. However, it will also be 
slow in replenishing credits (in FC), so senders will slow down as well, 
but unfortunately FC.max_credits if 20MB defeat this!

> For example, the digest output might look like:
>
> mct001-24908:
> WKS005-857: [55450 : 55552 (55552)]
> WKS006-64076: [39473 : 39575 (39575)]
> WKS009-5180: [268164 : 268264 (268264)]
> WKS018-33904: [39344 : 39445 (39445)]
> mct001-24908: [1651826 : 1654956 (1654956) (size=3130, missing=1827,
> highest stability=1651826)
>
> I can see STABLE and STABILITY messages being sent (using TRACE on
> STABLE).  The frequency changes, being from 1 minute to over 10 minute
> intervals.

Yes, STABILITY needs consensus from *all* members so if a STABLE message 
is slow or dropped then we'll have to wait for the next msg from that 
member.

> The first thing I'd love to know is whether the above retransmission
> table behavior is within what would be considered "normal" behavior.  If
> not, what other things should I start to look at?

No, absolutely not. I suggest apply my changes and re-run this. Note 
that you can trigger STABLE to send a message via probe.sh op=STABLE.gc.

> Cheers,
> --
> Tony Gedge
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140