Re: [jgroups-users] javagroups-users Digest, Vol 117, Issue 2

Thanks Bela for the suggestions on configuration change.  The low-speed
ADSL link will be replaced soon, which should get both remote site
connections up to 2M symmetric.  Our observed latentcy on these connections
is stable at about 32ms.  Unfortunately given the product is 3rd party we
can't change the design to make the remote connections not be directly part
of the cluster.

In the meantime we have identified issues with out multicast network
configuration (had a flapping multicast tree) which appears to have
rectified a lot of the issues now that it's fixed (not surprisingly).

I haven't applied any of your config changes yet, but am hoping to get an
opportunity to do so this week (the system is in a customer site).

Looking at the logs, we're still occasionally seeing the following, even
after the multicast issue was fixed:
20160224 21:08:05 DEBUG [pbcast.GMS]: determining merge leader from
[mct001-7563, 30004312-dc1c-812e-8254-ff7d81ee5796, WKS004-20453,
WKS009-8772, 2d09d356-be1f-e555-58aa-db4c55f467a2,
b9d231b2-fe50-2abe-db9a-0712bb277397, WKS006-25659, WKS005-9818,
d0ae7d6a-7674-264e-4d17-cb01ecdcc069, 863745c9-9c90-a887-df13-f3634c9433c2]
20160224 21:08:05 DEBUG [pbcast.GMS]: I (mct001-7563) am not the merge
leader, waiting for merge leader (863745c9-9c90-a887-df13-f3634c9433c2) to
initiate merge
20160224 21:08:06 DEBUG [pbcast.GMS]: suspected
members=[6ca893ac-7c0d-4da9-d536-caf0f309d858],
suspected_mbrs=[6ca893ac-7c0d-4da9-d536-caf0f309d858,
a6fcb7f5-8c90-bbb2-a71f-d97ffe91322f, 8a659a2e-0312-652d-1fa2-f457341416cd,
c0010eb8-07b5-c5a1-8bf5-566a1326f921, 544c0635-785d-5ab1-364d-c5fbd54d957f]
20160224 21:08:06 DEBUG [pbcast.GMS]: suspected
members=[544c0635-785d-5ab1-364d-c5fbd54d957f],
suspected_mbrs=[6ca893ac-7c0d-4da9-d536-caf0f309d858,
a6fcb7f5-8c90-bbb2-a71f-d97ffe91322f, 8a659a2e-0312-652d-1fa2-f457341416cd,
c0010eb8-07b5-c5a1-8bf5-566a1326f921, 544c0635-785d-5ab1-364d-c5fbd54d957f]
20160224 21:08:07 TRACE [pbcast.GMS]: found no members to add or remove,
will not create new view
20160224 21:08:07 TRACE [pbcast.GMS]: found no members to add or remove,
will not create new view
20160224 21:08:08 DEBUG [pbcast.GMS]: suspected
members=[a6fcb7f5-8c90-bbb2-a71f-d97ffe91322f],
suspected_mbrs=[6ca893ac-7c0d-4da9-d536-caf0f309d858,
a6fcb7f5-8c90-bbb2-a71f-d97ffe91322f, 8a659a2e-0312-652d-1fa2-f457341416cd,
c0010eb8-07b5-c5a1-8bf5-566a1326f921, 544c0635-785d-5ab1-364d-c5fbd54d957f]
20160224 21:08:08 DEBUG [pbcast.GMS]: suspected
members=[c0010eb8-07b5-c5a1-8bf5-566a1326f921,
8a659a2e-0312-652d-1fa2-f457341416cd],
suspected_mbrs=[6ca893ac-7c0d-4da9-d536-caf0f309d858,
a6fcb7f5-8c90-bbb2-a71f-d97ffe91322f, 8a659a2e-0312-652d-1fa2-f457341416cd,
c0010eb8-07b5-c5a1-8bf5-566a1326f921, 544c0635-785d-5ab1-364d-c5fbd54d957f]
20160224 21:08:09 DEBUG [pbcast.GMS]: suspected
members=[6ca893ac-7c0d-4da9-d536-caf0f309d858],
suspected_mbrs=[6ca893ac-7c0d-4da9-d536-caf0f309d858,
a6fcb7f5-8c90-bbb2-a71f-d97ffe91322f, 8a659a2e-0312-652d-1fa2-f457341416cd,
c0010eb8-07b5-c5a1-8bf5-566a1326f921, 544c0635-785d-5ab1-364d-c5fbd54d957f]
20160224 21:08:09 DEBUG [pbcast.GMS]: suspected
members=[544c0635-785d-5ab1-364d-c5fbd54d957f],
suspected_mbrs=[6ca893ac-7c0d-4da9-d536-caf0f309d858,
a6fcb7f5-8c90-bbb2-a71f-d97ffe91322f, 8a659a2e-0312-652d-1fa2-f457341416cd,
c0010eb8-07b5-c5a1-8bf5-566a1326f921, 544c0635-785d-5ab1-364d-c5fbd54d957f]
20160224 21:08:09 WARN [pbcast.NAKACK]: mct001-7563: dropped message from
WKS004-20453 (not in table [WKS018-5937, mct001-7563, WKS006-24228]),
view=MergeView::[mct001-7563|856] [mct001-7563, WKS018-5937, WKS006-24228],
subgroups=[[mct001-7563|854] [mct001-7563, WKS018-5937], [WKS006-24228|855]
[WKS006-24228]]
20160224 21:08:10 TRACE [pbcast.GMS]: found no members to add or remove,
will not create new view
20160224 21:08:10 TRACE [pbcast.GMS]: found no members to add or remove,
will not create new view
20160224 21:08:11 DEBUG [pbcast.GMS]: suspected
members=[a6fcb7f5-8c90-bbb2-a71f-d97ffe91322f],
suspected_mbrs=[6ca893ac-7c0d-4da9-d536-caf0f309d858,
a6fcb7f5-8c90-bbb2-a71f-d97ffe91322f, 8a659a2e-0312-652d-1fa2-f457341416cd,
c0010eb8-07b5-c5a1-8bf5-566a1326f921, 544c0635-785d-5ab1-364d-c5fbd54d957f]
20160224 21:08:11 DEBUG [pbcast.GMS]: suspected
members=[c0010eb8-07b5-c5a1-8bf5-566a1326f921,
8a659a2e-0312-652d-1fa2-f457341416cd],
suspected_mbrs=[6ca893ac-7c0d-4da9-d536-caf0f309d858,
a6fcb7f5-8c90-bbb2-a71f-d97ffe91322f, 8a659a2e-0312-652d-1fa2-f457341416cd,
c0010eb8-07b5-c5a1-8bf5-566a1326f921, 544c0635-785d-5ab1-364d-c5fbd54d957f]
20160224 21:08:12 DEBUG [pbcast.GMS]: suspected
members=[6ca893ac-7c0d-4da9-d536-caf0f309d858],
suspected_mbrs=[6ca893ac-7c0d-4da9-d536-caf0f309d858,
a6fcb7f5-8c90-bbb2-a71f-d97ffe91322f, 8a659a2e-0312-652d-1fa2-f457341416cd,
c0010eb8-07b5-c5a1-8bf5-566a1326f921, 544c0635-785d-5ab1-364d-c5fbd54d957f]
20160224 21:08:12 DEBUG [pbcast.GMS]: suspected
members=[544c0635-785d-5ab1-364d-c5fbd54d957f],
suspected_mbrs=[6ca893ac-7c0d-4da9-d536-caf0f309d858,
a6fcb7f5-8c90-bbb2-a71f-d97ffe91322f, 8a659a2e-0312-652d-1fa2-f457341416cd,
c0010eb8-07b5-c5a1-8bf5-566a1326f921, 544c0635-785d-5ab1-364d-c5fbd54d957f]
20160224 21:08:12 TRACE [protocols.MERGE2]: Discovery results:
[mct001-7563]: MergeView::[mct001-7563|856] [mct001-7563, WKS018-5937,
WKS006-24228], subgroups=[[mct001-7563|854] [mct001-7563, WKS018-5937],
[WKS006-24228|855] [WKS006-24228]]
[WKS006-24228]: MergeView::[mct001-7563|856] [mct001-7563, WKS018-5937,
WKS006-24228], subgroups=[[mct001-7563|854] [mct001-7563, WKS018-5937],
[WKS006-24228|855] [WKS006-24228]]
[WKS006-25659]: [863745c9-9c90-a887-df13-f3634c9433c2|846]
[863745c9-9c90-a887-df13-f3634c9433c2,
b9d231b2-fe50-2abe-db9a-0712bb277397, d0ae7d6a-7674-264e-4d17-cb01ecdcc069,
WKS018-5937, 2d09d356-be1f-e555-58aa-db4c55f467a2,
30004312-dc1c-812e-8254-ff7d81ee5796, WKS009-8772, WKS006-25659]
20160224 21:08:12 DEBUG [protocols.MERGE2]: mct001-7563 found different
views : [mct001-7563|856], [863745c9-9c90-a887-df13-f3634c9433c2|846];
sending up MERGE event with merge participants [mct001-7563,
30004312-dc1c-812e-8254-ff7d81ee5796, WKS009-8772,
2d09d356-be1f-e555-58aa-db4c55f467a2, b9d231b2-fe50-2abe-db9a-0712bb277397,
WKS006-25659, d0ae7d6a-7674-264e-4d17-cb01ecdcc069,
863745c9-9c90-a887-df13-f3634c9433c2].
Discovery results:
[mct001-7563]: MergeView::[mct001-7563|856] [mct001-7563, WKS018-5937,
WKS006-24228], subgroups=[[mct001-7563|854] [mct001-7563, WKS018-5937],
[WKS006-24228|855] [WKS006-24228]]
[WKS006-24228]: MergeView::[mct001-7563|856] [mct001-7563, WKS018-5937,
WKS006-24228], subgroups=[[mct001-7563|854] [mct001-7563, WKS018-5937],
[WKS006-24228|855] [WKS006-24228]]
[WKS006-25659]: [863745c9-9c90-a887-df13-f3634c9433c2|846]
[863745c9-9c90-a887-df13-f3634c9433c2,
b9d231b2-fe50-2abe-db9a-0712bb277397, d0ae7d6a-7674-264e-4d17-cb01ecdcc069,
WKS018-5937, 2d09d356-be1f-e555-58aa-db4c55f467a2,
30004312-dc1c-812e-8254-ff7d81ee5796, WKS009-8772, WKS006-25659]

This goes on for several hours and only seems to stop if we shut down all
the members and restart them.  I've got everything logging at WARN except
for STABLE, GMS and MERGE2 which are at TRACE and NAKACK which is a DEBUG.

It looks to me like several issues are going on:

   1. There's (at least) two controllers who are trying to merge, but it
   looks like the merge isn't completing.
   2. There's a number of machines who are being continually suspected but
   not excluded.
   3. Machines are being found, but their actual machine name is unknown
   (hence so many UUIDs)

Do you think my analysis is correct?  If I'm right, I expect that I need to
increase some timeouts in MERGE2, FD and VERIFY_SUSPECT to give the members
more time to respond.  That in combination with your FC changes to help
with throttling should hopefully get it a bit more stable.

Cheers, and thanks for taking the time to look at this.  It has been
extremely helpful!

-- 
Tony Gedge

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users