Re: [jgroups-users] javagroups-users Digest, Vol 117, Issue 2

Hi Bela (and anyone else watching),

It's me again :).  I've been trying to get this system working reliably,
but with little success.  I understand that you can't spend time on this as
the underlying version is quite old.  However, I have a few general
questions which might be able to point me in the correct direction for my
own investigation.

   - Is it possible to run code built against a 2.X JGroups with 3.X
   jgroups?
   In other words, is the API compatible/can we do a drop-in replacement or
   is recompilation required?
   - What would the effect be of placing the FRAG before GMS?
   The config we have has the ordering:
   "UDP, PING, MERGE2, FD, VERIFY_SUSPECT, NAKACK, UNICAST, STABLE, FRAG,
   GMS, FC",
   while the example config (udp.xml) has the ordering:
   "UDP, PING, MERGE2, FD_SOCK, FD_ALL, VERIFY_SUSPECT, NAKACK, UNICAST,
   STABLE, GMS, UFC, MFC, FRAG2"
   - Is it the case that anything under FC is unaffected by flow control -
   that is, retransmissions (from NAKACK), pings, suspects and group
   membership messages will all not be flow controlled?
   - Is it possible to have the coordinator selected from a particular
   subnet (if present)?
   - Are there rules of thumb that can be used for initial sizing of
   credits and buffer sizes for different network latencies and message rates?

We are seeing the system operate for a while, then something happens to
cause a steadily increasing UDP message output until it reaches about a
constant 3Mbps of sent UDP data.  In this state, I can see a large number
of FC "CREDIT_REQUEST" messages are being sent, but I don't see any
REPLENISH messages, so I was wondering if the relative placement of FRAG
and FC was important.

Cheers,

Tony.

On Fri, Feb 26, 2016 at 10:29 PM, Questions/problems related to using
JGroups <[email protected]> wrote:

>
>
> On 26/02/16 05:10, Questions/problems related to using JGroups wrote:
> > Thanks Bela for the suggestions on configuration change.  The low-speed
> > ADSL link will be replaced soon, which should get both remote site
> > connections up to 2M symmetric.  Our observed latentcy on these
> > connections is stable at about 32ms.  Unfortunately given the product is
> > 3rd party we can't change the design to make the remote connections not
> > be directly part of the cluster.
>
> OK. But - as I said - I'm hesitant to support such an old version of
> JGroups. I'll most likely waste my time on bugs that were fixed long
> ago... :-(
>
>
> > In the meantime we have identified issues with out multicast network
> > configuration (had a flapping multicast tree) which appears to have
> > rectified a lot of the issues now that it's fixed (not surprisingly).
>
> OK, good.
>
> > I haven't applied any of your config changes yet, but am hoping to get
> > an opportunity to do so this week (the system is in a customer site).
> >
> > Looking at the logs, we're still occasionally seeing the following, even
> > after the multicast issue was fixed:
>
> (logs mitted for brevity)
>
> > [WKS006-24228]: MergeView::[mct001-7563|856] [mct001-7563, WKS018-5937,
> > WKS006-24228], subgroups=[[mct001-7563|854] [mct001-7563, WKS018-5937],
> > [WKS006-24228|855] [WKS006-24228]]
> > [WKS006-25659]: [863745c9-9c90-a887-df13-f3634c9433c2|846]
> > [863745c9-9c90-a887-df13-f3634c9433c2,
> > b9d231b2-fe50-2abe-db9a-0712bb277397,
> > d0ae7d6a-7674-264e-4d17-cb01ecdcc069, WKS018-5937,
> > 2d09d356-be1f-e555-58aa-db4c55f467a2,
> > 30004312-dc1c-812e-8254-ff7d81ee5796, WKS009-8772, WKS006-25659]
>
> Looks as if the system detected that a merge was in order, but failed
> actually performing the merge. This could be caused by the multicast
> issues (packets dropped), or by the discovery protocol not workig
> correctly because of dropped packets (another evidence of this is the
> members showing their UUIDs rather than logical names).
>
> > This goes on for several hours and only seems to stop if we shut down
> > all the members and restart them.  I've got everything logging at WARN
> > except for STABLE, GMS and MERGE2 which are at TRACE and NAKACK which is
> > a DEBUG.
> >
> > It looks to me like several issues are going on:
> >
> >  1. There's (at least) two controllers who are trying to merge, but it
> >     looks like the merge isn't completing.
>
> Yes - possibly because of dropped datagram or multicast packets
>
> >  2. There's a number of machines who are being continually suspected but
> >     not excluded.
>
> This could also be caused by packet loss
>
> >  3. Machines are being found, but their actual machine name is unknown
> >     (hence so many UUIDs)
>
> Yes, I noticed, too.
>
>
> > Do you think my analysis is correct?  If I'm right, I expect that I need
> > to increase some timeouts in MERGE2, FD and VERIFY_SUSPECT to give the
> > members more time to respond.  That in combination with your FC changes
> > to help with throttling should hopefully get it a bit more stable.
> >
> > Cheers, and thanks for taking the time to look at this.  It has been
> > extremely helpful!
>
> I suggest run the latest 3.6.8 JGroups version to see if you'll get the
> same issues. Run the Chat or View demo with the udp.xml or fast.xml
> shipped with JGroups (change the bind address, mcast addr:port etc).
>
> If you get similar issues, then the culprit is most likely the network
> dropping packets. Although 3.6.8 is much better suited to handle these
> cases; e.g. MERGE3 handles more edge cases than MERGE2.
>
>
> > --
> > Tony Gedge
> > _______________________________________________
> > javagroups-users mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/javagroups-users
> >
>
> --
> Bela Ban, JGroups lead (http://www.jgroups.org)
>
>
>
> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>



-- 
Tony Gedge
54 Heliopolis Pde
Mitchelton 4053

------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users