[jgroups-dev] Muti node crash
"Sundararajan, Mukund" <[email protected]> Wed, 25 Mar 2015 01:42:54 +0000
| Newsgroups | gmane.comp.java.javagroups.devel |
|---|---|
| Message-ID | <1159DDC505B322408341C5F00AB950DF89EC3A47@NWKNJEXMBX1.corp.root.ipc.com> |
--===============4336346404816952913==
Content-Language: en-US
Content-Type: multipart/alternative;
boundary="_000_1159DDC505B322408341C5F00AB950DF89EC3A47NWKNJEXMBX1corp_"
--_000_1159DDC505B322408341C5F00AB950DF89EC3A47NWKNJEXMBX1corp_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
All,
We are facing a unique problem. I have a 30 node cluster, 15 nodes in one d=
ata center and the rest in another. The coordinator is in the second data c=
enter. When the first data center loses power, thereby resulting in the abn=
ormal shutdown of the 15 nodes, the following happens -
1. The coordinator puts out several view changes, seconds apart from =
each other. The total number of view changes is 15. This is good.
2. While the view changes are being broadcasted, MERGE2 on the coordi=
nator starts finding all views. This sends a GET_MBRS_REQ to go out. This g=
oes out with View ID =3D X.
3. The coordinator gets a response to GET_MBRS_REQ from all members. =
However different members report different views. By the time the response =
reaches the coordinator, it itself has installed a new View Y because the v=
iew changes as a result of step 1 are happening in parallel.
4. The coordinator then initiates a merge. There are four sub groups =
to merge. All four sub groups have the same coordinator ( just different vi=
ew IDs). As a net result the coordinator ends up creating a final view with=
just one member, itself. It never recovers from this condition.
Does anyone have any idea why GET_MBRS_REQ triggered by MERGE2 and the view=
changes are not synchronized? Is my configuration missing something? Why d=
oes not the coordinator recover, rediscover members and install and full vi=
ew of 15 members after it incorrectly merges with itself in step 4 and redu=
ces the view size to 1?
Thanks
Mukund
---------------------------------------------------------------------------=
---------------------------------------------------------------------------=
------------------------------------
DISCLAIMER: This e-mail may contain information that is confidential, privi=
leged or otherwise protected from disclosure. If you are not an intended re=
cipient of this e-mail, do not duplicate or redistribute it by any means. P=
lease delete it and any attachments and notify the sender that you have rec=
eived it in error. Unintended recipients are prohibited from taking action =
on the basis of information in this e-mail.E-mail messages may contain comp=
uter viruses or other defects, may not be accurately replicated on other sy=
stems, or may be intercepted, deleted or interfered with without the knowle=
dge of the sender or the intended recipient. If you are not comfortable wit=
h the risks associated with e-mail messages, you may decide not to use e-ma=
il to communicate with IPC. IPC reserves the right, to the extent and under=
circumstances permitted by applicable law, to retain, monitor and intercep=
t e-mail messages to and from its systems.
--_000_1159DDC505B322408341C5F00AB950DF89EC3A47NWKNJEXMBX1corp_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:656224335;
mso-list-type:hybrid;
mso-list-template-ids:872433692 67698703 67698713 67698715 67698703 676987=
13 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-US" link=3D"blue" vlink=3D"purple">
<div class=3D"WordSection1">
<p class=3D"MsoNormal">All,<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">We are facing a unique problem. I have a 30 node clu=
ster, 15 nodes in one data center and the rest in another. The coordinator =
is in the second data center. When the first data center loses power, there=
by resulting in the abnormal shutdown
of the 15 nodes, the following happens -<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoListParagraph" style=3D"text-indent:-.25in;mso-list:l0 level=
1 lfo1"><![if !supportLists]><span style=3D"mso-list:Ignore">1.<span style=
=3D"font:7.0pt "Times New Roman""> &=
nbsp;
</span></span><![endif]>The coordinator puts out several view changes, seco=
nds apart from each other. The total number of view changes is 15. This is =
good.<o:p></o:p></p>
<p class=3D"MsoListParagraph" style=3D"text-indent:-.25in;mso-list:l0 level=
1 lfo1"><![if !supportLists]><span style=3D"mso-list:Ignore">2.<span style=
=3D"font:7.0pt "Times New Roman""> &=
nbsp;
</span></span><![endif]>While the view changes are being broadcasted, MERGE=
2 on the coordinator starts finding all views. This sends a GET_MBRS_REQ to=
go out. This goes out with View ID =3D X.<o:p></o:p></p>
<p class=3D"MsoListParagraph" style=3D"text-indent:-.25in;mso-list:l0 level=
1 lfo1"><![if !supportLists]><span style=3D"mso-list:Ignore">3.<span style=
=3D"font:7.0pt "Times New Roman""> &=
nbsp;
</span></span><![endif]>The coordinator gets a response to GET_MBRS_REQ fro=
m all members. However different members report different views. By the tim=
e the response reaches the coordinator, it itself has installed a new View =
Y because the view changes as a
result of step 1 are happening in parallel.<o:p></o:p></p>
<p class=3D"MsoListParagraph" style=3D"text-indent:-.25in;mso-list:l0 level=
1 lfo1"><![if !supportLists]><span style=3D"mso-list:Ignore">4.<span style=
=3D"font:7.0pt "Times New Roman""> &=
nbsp;
</span></span><![endif]>The coordinator then initiates a merge. There are f=
our sub groups to merge. All four sub groups have the same coordinator ( ju=
st different view IDs). As a net result the coordinator ends up creating a =
final view with just one member,
itself. It never recovers from this condition.<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Does anyone have any idea why GET_MBRS_REQ triggered=
by MERGE2 and the view changes are not synchronized? Is my configuration m=
issing something? Why does not the coordinator recover, rediscover members =
and install and full view of 15 members
after it incorrectly merges with itself in step 4 and reduces the view siz=
e to 1?<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Thanks<o:p></o:p></p>
<p class=3D"MsoNormal">Mukund<o:p></o:p></p>
</div>
<p><font color=3D"#808080" size=3D"2" face=3D"Arial"><font color=3D"#808080=
" size=3D"2" face=3D"Arial">-----------------------------------------------=
---------------------------------------------------------------------------=
----------------------------------------------------------------</font></p>
<p><font color=3D"#808080" size=3D"2" face=3D"Arial">DISCLAIMER: This e-mai=
l may contain information that is confidential, privileged or otherwise pro=
tected from disclosure. If you are not an intended recipient of this e-mail=
, do not duplicate or redistribute it
by any means. Please delete it and any attachments and notify the sender t=
hat you have received it in error. Unintended recipients are prohibited fro=
m taking action on the basis of information in this e-mail.E-mail messages =
may contain computer viruses or
other defects, may not be accurately replicated on other systems, or may b=
e intercepted, deleted or interfered with without the knowledge of the send=
er or the intended recipient. If you are not comfortable with the risks ass=
ociated with e-mail messages, you
may decide not to use e-mail to communicate with IPC. IPC reserves the rig=
ht, to the extent and under circumstances permitted by applicable law, to r=
etain, monitor and intercept e-mail messages to and from its systems.</font=
></font><br>
</p>
<br>
</body>
</html>
--_000_1159DDC505B322408341C5F00AB950DF89EC3A47NWKNJEXMBX1corp_--
--===============4336346404816952913==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
--===============4336346404816952913==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Javagroups-development mailing list
--===============4336346404816952913==--