Re: [jgroups-dev] MERGE3 issue
Development issues <[email protected]> Mon, 25 Jul 2016 06:58:09 +0000
| Newsgroups | gmane.comp.java.javagroups.devel |
|---|---|
| Message-ID | <E8918F7C66E85B4A926C4C3ED29AD7FA500191EB@sacmail1> |
This is a multi-part message in MIME format.
--===============6985522401799170309==
Content-Language: en-US
Content-Type: multipart/alternative;
boundary="_000_E8918F7C66E85B4A926C4C3ED29AD7FA500191EBsacmail1_"
This is a multi-part message in MIME format.
--_000_E8918F7C66E85B4A926C4C3ED29AD7FA500191EBsacmail1_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
X-EC0D2A8E-5CB7-4969-9C36-46D859D137BE-PartID: 66340435-D7F4-4E09-8982-8D6AB3DD8A2D
Hi All.. moving the conversation to the list for all to - um - ponder ;)
This is interesting, as the merge code is *exactly the same*; MERGE2 or
MERGE3 only trigger a merge, but don't actually execute the merge themselves=
! The merge is done by Merger{2} (in GMS).
Did not mean to imply the cause was MERGE3. Yes, they both call Merger.
Or are you suggesting that the merge is actually never triggered by MERGE3?
No, the merge appears to be triggered. The results appear to be going wrong=
. All off the former Coordinators are pointing to other former coordinators=
.
Any chance you can write a unit test similar to MergeTest{2-4}? These are te=
sts that reproduce a given scenario and then check that the system reacts co=
rrectly.
If I knew where it was breaking down I would be happy to do so. I am not th=
ere yet. :(
> I believe the sequence of events is as follows:
>
> 1.Merge needed between N coordinators
>
> 2.Network congestion occurs during merge process
I assume by network congestion you don't mean flow control?
No. I mean packet loss due to contention. I am working on smoothing out th=
e traffic load during Merger, but it is still pretty peaky. We shoot for un=
der 128k, and take a haircut above 256k.
What's the effect of this? Dropped packets? That could be reproduced via DIS=
CARD...
Yes, dropped packets. I am not familiar with DISCARD. I will look into it.=
> 3.Coordinators "think" that one of the other coordinators won the merge
> and change their impl.
To what? From Coord to Participant?
Precisely.
Where do you see coords changing their impl?
This is a network of 150ish hosts. When testing under situations where (for=
example) many hosts start at once, there will often be subgroups. This is,=
of course, expected. When the merge[s] completes, sometimes we are left in=
this problematic state.
It is evidenced by:
* No additional hosts can join
* No existing hosts can leave
* If you turn on logging for MERGE3 and watch the SendInfo messages =
flowing by, you will see that all of the hosts that should be coordinators h=
ave Sendinfo output that shows they think they are participants.
For example, the output would indicate something like:
Host A view: B, X, Y, Z, A (where B should be coordinator)
Host B view: C, Q, R, S, B (where C should be coordinator)
Host C view: A, M, N, O, C (where A should be coordinator)
In Merger.merge(), if a
coordinator determines that it is not the merge leader, it doesn't do
anything, let alone change its impl.
Understood. I am trying to understand whether there is something - FD_ALL2/=
SOCK, GMS, MERGE3 - that should rectify the situation that we are in by pick=
ing a new coordinator - or prevent it [by requiring 100% acks].
> I believe this is the result of congestion during the merge publish. This=
is clearly visible if you watch the
> results from sendInfo() rolling in on any host in the group.
> The result is a steady state of merge needed between N networks. Those
> that are in a network are healthy, but no one can join.. or leave (which
> in many ways is worse). We are warning up to debug this and I thought I
> might reach out to you to see if you had any thoughts or suggestions.
If you describe how to reproduce the problem, or send me a unit tests,
I'll take a look.
I am happy to describe the scenario, but it would probably be tough for you =
to reproduce. We have, for this example, 90 hosts on network A, and on netw=
orks B-H we have varying numbers, though at least 3 are on all of them, and =
at most 14. I believe this will reproduce with as few as three networks, th=
ough we have not seen it with less than 5 of them running.
There are bandwidth limits, 250k each way, between the networks. Over 250k =
will queue in typical Cisco fashion for a bit, but quickly start dropping pa=
ckets...
rate-limit input 256000 35200 38400 confirm-action set-prec-transmit 4 excee=
d-action drop
rate-limit output 256000 35200 38400 confirm-action set-prec-transmit 4 exce=
ed-action drop
Fire up one node, then fire up the rest, and sometimes it gets into the stat=
e I described above.
> On a somewhat related note, in order to smooth out network usage [and
> avoid the aforementioned congestion] we have added a pause to the
> response to view changes.
The VIEW_ACK?
Several places. sendViewAck(), response to GET_DIGEST_REQ, and SendJoinResp=
onse() in GMS. Response to VIEW_REQ in MERGE3. sendMergeResponse() in Merg=
er. Changes are currently limited to GMS, MERGE3, and Merger. Just to recl=
arify, the problems described above happened before, these changes, and they=
have reduced the incidence, but not yet eliminated it. While I believe the=
smoothing of responses from "bulk" helps prevent contention, there is no wa=
y to have complete control over the network, so something else could cause t=
he packet loss and put us into the same situation.
> This has vastly improved things and has made
> the occurrence of the MERGE3 issue I described less common.
First I don't see how delaying VIEW_ACK would make a diff and second, I
don't see a causal relation between this and MERGE3.
> I am looking at whether MERGE3 might need the same type of pause (likely
> based on what I am seeing on wireshark). We could certainly create GMS2
> and MERGE4, but I believe this sort of change is of general value.
> Again, any comments on the implantation are appreciated.
I must confess I failed to understand both the problem and the solution
to it... :-)
Sorry about that. Two things going on here: 1) When we lost packets "at the=
wrong time" we can end up in the situation described above where no one thi=
nks they are coordinator (and there is no merge being run, as there are no c=
oordinators). 2) When merges do occur, high utilization occurs which can le=
ad to (1).
I am trying to address (2) by smoothing out responses to multicasted "change=
s". For (1) I am not sure what should be happening, so I am not sure where =
I should start debugging. Should something be "cleaning up" the situation, =
or is there code somewhere that should make it impossible to occur.
Regards,
Neal
--_000_E8918F7C66E85B4A926C4C3ED29AD7FA500191EBsacmail1_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
X-EC0D2A8E-5CB7-4969-9C36-46D859D137BE-PartID: C4CE82C6-3965-4937-9048-6468B5A150CA
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii">=
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
@font-face
{font-family:wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"cambria math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"lucida console";
panose-1:2 11 6 9 4 5 4 2 2 4;}
p.msonormal, li.msonormal, div.msonormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"calibri",sans-serif;}
a:link, span.msohyperlink
{mso-style-priority:99;
color:#0563c1;
text-decoration:underline;}
a:visited, span.msohyperlinkfollowed
{mso-style-priority:99;
color:#954f72;
text-decoration:underline;}
p.msoplaintext, li.msoplaintext, div.msoplaintext
{mso-style-priority:99;
mso-style-link:"plain text char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"calibri",sans-serif;}
span.plaintextchar
{mso-style-name:"plain text char";
mso-style-priority:99;
mso-style-link:"plain text";
font-family:"calibri",sans-serif;}
.msochpdefault
{mso-style-type:export-only;
font-family:"calibri",sans-serif;}
@page wordsection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.wordsection1
{page:wordsection1;}
@list l0
{mso-list-id:1784961303;
mso-list-type:hybrid;
mso-list-template-ids:827104140 67698689 67698691 67698693 67698689 6769869=
1 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:=B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"courier new";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:=A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:=B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"courier new";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:=A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:=B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"courier new";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:=A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body link=3D"#0563C1" vlink=3D"#954F72">
<div class=3D"WordSection1">
<p class=3D"MsoPlainText">Hi All.. moving the conversation to the list for a=
ll to – um – ponder ;) </p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText">This is interesting, as the merge code is *exactly=
the same*; MERGE2 or </p>
<p class=3D"MsoPlainText">MERGE3 only trigger a merge, but don't actually ex=
ecute the merge themselves! The merge is done by Merger{2} (in GMS). </p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText"><i>Did not mean to imply the cause was MERGE3.&nbs=
p; Yes, they both call Merger. </i></p>
<p class=3D"MsoPlainText"><span style=3D"color:black"> </span></p>
<p class=3D"MsoPlainText">Or are you suggesting that the merge is actually n=
ever triggered by MERGE3? </p>
<p class=3D"MsoPlainText"><span style=3D"color:black"> </span></p>
<p class=3D"MsoPlainText"><i><span style=3D"color:black">No, the merge appea=
rs to be triggered. The results appear to be going wrong. All of=
f the former Coordinators are pointing to other former coordinators. </span=
></i></p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText">Any chance you can write a unit test similar to Me=
rgeTest{2-4}? These are tests that reproduce a given scenario and then check=
that the system reacts correctly. </p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText"><i><span style=3D"color:black">If I knew where it =
was breaking down I would be happy to do so. I am not there yet.
</span></i><i><span style=3D"font-family:Wingdings;color:black">L</span><spa=
n style=3D"color:black"> </span></i></p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText">> I believe the sequence of events is as =
follows: </p>
<p class=3D"MsoPlainText">> </p>
<p class=3D"MsoPlainText">> 1.Merge needed between N coordinators </p>
<p class=3D"MsoPlainText">> </p>
<p class=3D"MsoPlainText">> 2.Network congestion occurs during merge proc=
ess </p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText">I assume by network congestion you don't mean flow=
control? </p>
<p class=3D"MsoPlainText"><span style=3D"color:black"> </span></p>
<p class=3D"MsoPlainText"><i><span style=3D"color:black">No. I mean pa=
cket loss due to contention. I am working on smoothing out the traffic=
load during Merger, but it is still pretty peaky. We shoot for under =
128k, and take a haircut above 256k. </span></i></p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText">What's the effect of this? Dropped packets? That c=
ould be reproduced via DISCARD... </p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText"><i><span style=3D"color:black">Yes, dropped packet=
s. I am not familiar with DISCARD. I will look into it. </span>=
</i></p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText">> 3.Coordinators “think” that one o=
f the other coordinators won the merge </p>
<p class=3D"MsoPlainText">> and change their impl. </p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText">To what? From Coord to Participant? </p>
<p class=3D"MsoPlainText"><span style=3D"color:black"> </span></p>
<p class=3D"MsoPlainText"><i><span style=3D"color:black">Precisely. </span>=
</i></p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText">Where do you see coords changing their impl? </p=
>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText"><i>This is a network of 150ish hosts. When t=
esting under situations where (for example) many hosts start at once, there =
will often be subgroups. This is, of course, expected. When the =
merge[s] completes, sometimes we are left in this
problematic state. </i></p>
<p class=3D"MsoPlainText"><i> </i></p>
<p class=3D"MsoPlainText"><i>It is evidenced by: </i></p>
<p class=3D"MsoPlainText" style=3D"margin-left:.5in;text-indent:-.25in;mso-l=
ist:l0 level1 lfo1">
<![if !supportLists]><span style=3D"font-family:Symbol"><span style=3D"mso-l=
ist:Ignore">·<span style=3D"font:7.0pt "Times New Roman"">&=
nbsp;
</span></span></span><![endif]><i>No additional hosts can join </i></p>
<p class=3D"MsoPlainText" style=3D"margin-left:.5in;text-indent:-.25in;mso-l=
ist:l0 level1 lfo1">
<![if !supportLists]><span style=3D"font-family:Symbol"><span style=3D"mso-l=
ist:Ignore">·<span style=3D"font:7.0pt "Times New Roman"">&=
nbsp;
</span></span></span><![endif]><i>No existing hosts can leave </i></p>
<p class=3D"MsoPlainText" style=3D"margin-left:.5in;text-indent:-.25in;mso-l=
ist:l0 level1 lfo1">
<![if !supportLists]><span style=3D"font-family:Symbol"><span style=3D"mso-l=
ist:Ignore">·<span style=3D"font:7.0pt "Times New Roman"">&=
nbsp;
</span></span></span><![endif]><i>If you turn on logging for MERGE3 and watc=
h the SendInfo messages flowing by, you will see that all of the hosts that =
should be coordinators have Sendinfo output that shows they think they are p=
articipants. </i></p>
<p class=3D"MsoPlainText"><i> </i></p>
<p class=3D"MsoPlainText"><i>For example, the output would indicate somethin=
g like: </i></p>
<p class=3D"MsoPlainText" style=3D"text-align:justify"><i>Host A view: B, X,=
Y, Z, A (where B should be coordinator) </i></p>
<p class=3D"MsoPlainText"><i>Host B view: C, Q, R, S, B (where C should be c=
oordinator) </i></p>
<p class=3D"MsoPlainText"><i>Host C view: A, M, N, O, C (where A should be c=
oordinator) </i></p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText">In Merger.merge(), if a </p>
<p class=3D"MsoPlainText">coordinator determines that it is not the merge le=
ader, it doesn't do
</p>
<p class=3D"MsoPlainText">anything, let alone change its impl. </p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText"><i><span style=3D"color:black">Understood. I=
am trying to understand whether there is something – FD_ALL2/SOCK, GM=
S, MERGE3 – that should rectify the situation that we are in by pickin=
g a new coordinator – or prevent it [by requiring
100% acks]. </span></i></p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText">> I believe this is the result of congestion du=
ring the merge publish. This is clearly visible if you watch the </p>=
<p class=3D"MsoPlainText">> results from sendInfo() rolling in on any hos=
t in the group. </p>
<p class=3D"MsoPlainText">> The result is a steady state of merge needed =
between N networks. Those </p>
<p class=3D"MsoPlainText">> that are in a network are healthy, but no one=
can join.. or leave (which </p>
<p class=3D"MsoPlainText">> in many ways is worse). We are warning =
up to debug this and I thought I </p>
<p class=3D"MsoPlainText">> might reach out to you to see if you had any =
thoughts or suggestions. </p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText">If you describe how to reproduce the problem, or s=
end me a unit tests,
</p>
<p class=3D"MsoPlainText">I'll take a look. </p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText"><i>I am happy to describe the scenario, but it wou=
ld probably be tough for you to reproduce. We have, for this example, =
90 hosts on network A, and on networks B-H we have varying numbers, though a=
t least 3 are on all of them, and at
most 14. I believe this will reproduce with as few as three networks,=
though we have not seen it with less than 5 of them running. </i></p>
<p class=3D"MsoPlainText"><i> </i></p>
<p class=3D"MsoPlainText"><i>There are bandwidth limits, 250k each way, betw=
een the networks. Over 250k will queue in typical Cisco fashion for a =
bit, but quickly start dropping packets… </i></p>
<p class=3D"MsoPlainText"><i><span style=3D"font-size:8.0pt;font-family:&quo=
t;Lucida Console";color:#1F4E79">rate-limit input 256000 35200 38400 co=
nfirm-action set-prec-transmit 4 exceed-action drop</span><span style=3D"col=
or:#1F4E79"> </span></i></p>
<p class=3D"MsoPlainText"><i><span style=3D"font-size:8.0pt;font-family:&quo=
t;Lucida Console";color:#1F4E79">rate-limit output 256000 35200 38400 c=
onfirm-action set-prec-transmit 4 exceed-action drop</span><span style=3D"co=
lor:#1F4E79"> </span></i><i> </i></p>
<p class=3D"MsoPlainText"><i><span style=3D"color:black"> </span></i>=
</p>
<p class=3D"MsoPlainText"><i><span style=3D"color:black">Fire up one node, t=
hen fire up the rest, and sometimes it gets into the state I described above=
. </span></i></p>
<p class=3D"MsoPlainText"><span style=3D"color:black"> </span></p>
<p class=3D"MsoPlainText">> On a somewhat related note, in order to smoot=
h out network usage [and </p>
<p class=3D"MsoPlainText">> avoid the aforementioned congestion] we have =
added a pause to the </p>
<p class=3D"MsoPlainText">> response to view changes. </p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText">The VIEW_ACK? </p>
<p class=3D"MsoPlainText"><span style=3D"color:black"> </span></p>
<p class=3D"MsoPlainText"><i><span style=3D"color:black">Several places.&nbs=
p; sendViewAck(), response to GET_DIGEST_REQ, and SendJoinResponse() in GMS.=
Response to VIEW_REQ in MERGE3. sendMergeResponse() in Merger.&=
nbsp; Changes are currently limited to GMS, MERGE3, and
Merger. Just to reclarify, the problems described above happened befo=
re, these changes, and they have reduced the incidence, but not yet eliminat=
ed it. While I believe the smoothing of responses from “bulkR=
21; helps prevent contention, there is no way to have
complete control over the network, so something else could cause the packet=
loss and put us into the same situation.
</span></i></p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText">> This has vastly improved things and has made =
</p>
<p class=3D"MsoPlainText">> the occurrence of the MERGE3 issue I describe=
d less common. </p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText">First I don't see how delaying VIEW_ACK would make=
a diff and second, I
</p>
<p class=3D"MsoPlainText">don't see a causal relation between this and MERGE=
3. </p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText">> I am looking at whether MERGE3 might need the=
same type of pause (likely </p>
<p class=3D"MsoPlainText">> based on what I am seeing on wireshark). =
; We could certainly create GMS2 </p>
<p class=3D"MsoPlainText">> and MERGE4, but I believe this sort of change=
is of general value. </p>
<p class=3D"MsoPlainText">> Again, any comments on the implantation are a=
ppreciated. </p>
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText">I must confess I failed to understand both the pro=
blem and the solution
</p>
<p class=3D"MsoPlainText">to it... :-) </p>
<p class=3D"MsoPlainText"><span style=3D"color:black"> </span></p>
<p class=3D"MsoPlainText"><i><span style=3D"color:black">Sorry about that.&n=
bsp; Two things going on here: 1) When we lost packets “at the wrong t=
ime” we can end up in the situation described above where no one think=
s they are coordinator (and there is no merge being
run, as there are no coordinators). 2) When merges do occur, high uti=
lization occurs which can lead to (1). </span></i></p>
<p class=3D"MsoPlainText"><i><span style=3D"color:black"> </span></i>=
</p>
<p class=3D"MsoPlainText"><i><span style=3D"color:black">I am trying to addr=
ess (2) by smoothing out responses to multicasted “changes”.&nbs=
p; For (1) I am not sure what
<b>should</b> be happening, so I am not sure where I should start debugging.=
Should something be “cleaning up” the situation, or is th=
ere code somewhere that should make it impossible to occur. </span></i></p>=
<p class=3D"MsoPlainText"> </p>
<p class=3D"MsoPlainText">Regards, </p>
<p class=3D"MsoPlainText"><span style=3D"color:black">Neal </span></p>
</div>
</body>
</html>
--_000_E8918F7C66E85B4A926C4C3ED29AD7FA500191EBsacmail1_--
--===============6985522401799170309==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports.http://sdm.link/zohodev2dev
--===============6985522401799170309==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Javagroups-development mailing list
--===============6985522401799170309==--