Re: [jgroups-users] javagroups-users Digest, Vol 117, Issue 2

On 26/02/16 05:10, Questions/problems related to using JGroups wrote:
> Thanks Bela for the suggestions on configuration change.  The low-speed
> ADSL link will be replaced soon, which should get both remote site
> connections up to 2M symmetric.  Our observed latentcy on these
> connections is stable at about 32ms.  Unfortunately given the product is
> 3rd party we can't change the design to make the remote connections not
> be directly part of the cluster.

OK. But - as I said - I'm hesitant to support such an old version of 
JGroups. I'll most likely waste my time on bugs that were fixed long 
ago... :-(


> In the meantime we have identified issues with out multicast network
> configuration (had a flapping multicast tree) which appears to have
> rectified a lot of the issues now that it's fixed (not surprisingly).

OK, good.

> I haven't applied any of your config changes yet, but am hoping to get
> an opportunity to do so this week (the system is in a customer site).
>
> Looking at the logs, we're still occasionally seeing the following, even
> after the multicast issue was fixed:

(logs mitted for brevity)

> [WKS006-24228]: MergeView::[mct001-7563|856] [mct001-7563, WKS018-5937,
> WKS006-24228], subgroups=[[mct001-7563|854] [mct001-7563, WKS018-5937],
> [WKS006-24228|855] [WKS006-24228]]
> [WKS006-25659]: [863745c9-9c90-a887-df13-f3634c9433c2|846]
> [863745c9-9c90-a887-df13-f3634c9433c2,
> b9d231b2-fe50-2abe-db9a-0712bb277397,
> d0ae7d6a-7674-264e-4d17-cb01ecdcc069, WKS018-5937,
> 2d09d356-be1f-e555-58aa-db4c55f467a2,
> 30004312-dc1c-812e-8254-ff7d81ee5796, WKS009-8772, WKS006-25659]

Looks as if the system detected that a merge was in order, but failed 
actually performing the merge. This could be caused by the multicast 
issues (packets dropped), or by the discovery protocol not workig 
correctly because of dropped packets (another evidence of this is the 
members showing their UUIDs rather than logical names).

> This goes on for several hours and only seems to stop if we shut down
> all the members and restart them.  I've got everything logging at WARN
> except for STABLE, GMS and MERGE2 which are at TRACE and NAKACK which is
> a DEBUG.
>
> It looks to me like several issues are going on:
>
>  1. There's (at least) two controllers who are trying to merge, but it
>     looks like the merge isn't completing.

Yes - possibly because of dropped datagram or multicast packets

>  2. There's a number of machines who are being continually suspected but
>     not excluded.

This could also be caused by packet loss

>  3. Machines are being found, but their actual machine name is unknown
>     (hence so many UUIDs)

Yes, I noticed, too.


> Do you think my analysis is correct?  If I'm right, I expect that I need
> to increase some timeouts in MERGE2, FD and VERIFY_SUSPECT to give the
> members more time to respond.  That in combination with your FC changes
> to help with throttling should hopefully get it a bit more stable.
>
> Cheers, and thanks for taking the time to look at this.  It has been
> extremely helpful!

I suggest run the latest 3.6.8 JGroups version to see if you'll get the 
same issues. Run the Chat or View demo with the udp.xml or fast.xml 
shipped with JGroups (change the bind address, mcast addr:port etc).

If you get similar issues, then the culprit is most likely the network 
dropping packets. Although 3.6.8 is much better suited to handle these 
cases; e.g. MERGE3 handles more edge cases than MERGE2.


> --
> Tony Gedge
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140