Re: [jgroups-dev] Problem with FD_SOCK failure detection
"mayank.agarwal" <[email protected]> Thu, 22 Aug 2013 20:31:25 -0700 (PDT)
| Newsgroups | gmane.comp.java.javagroups.devel |
|---|---|
| Message-ID | <CE3CD522.ABA5%[email protected]> |
--===============1269102849848514144== Content-Type: multipart/alternative; boundary="----=_Part_175838_13916355.1377228685558" ------=_Part_175838_13916355.1377228685558 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit My Comments were inline. In case, you missed this. From: Mayank Agarwal <[email protected]<mailto:[email protected]>> Date: Wednesday 21 August 2013 9:09 PM To: "Bela Ban [via JGroups]" <[email protected]<mailto:[email protected]>> Subject: Re: Problem with FD_SOCK failure detection From: "Bela Ban [via JGroups]" <[email protected]<mailto:[email protected]>> Date: Wednesday 21 August 2013 8:43 PM To: Mayank Agarwal <[email protected]<mailto:[email protected]>> Subject: Re: Problem with FD_SOCK failure detection Works for me: I used udp.xml, commented FD_ALL and increased the timeout in VERIFY_SUSPECT to 5000. The I started members A and B. Then I killed A (will get removed ca. 5 s later) and immediately started C. C looped around the failed JOIN until A was removed, then B joined and the new view was {B,C}. Mayank: The exact case is first start C and than kill A immediately. This happens intermittentlty (If you repeat the process many times) Comments inine. On 8/21/13 12:29 PM, mayank.agarwal wrote: > yes, this is reproducible. > FD_SOCK start() method is not called as JChannel.connect call wasn't > succeeded because it was trying to JOIN killed member(co-ordinator) This should not be the case as the killed coordinator will be removed from the view, so a discovery should not return it anymore. Do you use return_entire_cache="true" by any chance ? Mayank: return_entire_cache is by default false and we are not modifying this in xml > there are three nodes in cluster. (member-89,member-101,member-88,member-87) > we are getting lot of warnings mentioned below. it is trying to contact > member-89(which is killed) > [pool-3-thread-1] WARN o.jgroups.protocols.pbcast.GMS - > JOIN(member-87-22015) sent to member-89-55050 timed out (after 7000 ms), on > try 2852 It attempted to join 2852 times ? Note that you can set GMS.max_join_attempts to put a ceiling to the number of tries. Mayank: By default GMS.max_join_attempts is 0 which means 'never give up'. Can you suggest optimal value for this? > On discovery request from member-87: member-88 is returning view > [member-89-55050|10]. This shouldn't be the case, as member-89 should have been removed from the view ! What's the config you use ? Mayank: config is attached > So it is always sending JOIN request to member-89(killed). > > member-88,89,101 has view {member-89,member-101,member-88,member-87} > member-87 does't have any view. > Scenario is: let say 2 A,B,C is in cluster new node D try to join and at > the same time A goes down. than C has already closed the socket with A > gracefully, assuming D will create the socket with A . D's FD_SOCK pinger > thread is not started. ( i am not seeing the msg "pinger_thread started" in > logs) Yes, that would be the case if D always gets discovery response(s) which still indicate A is the coordinator. This is the core issue, and we need to find out why anyone's still returning A in its discovery response. > We are not using combo of FD_SOCK with FD/FD_ALL. we are using only in > FD_SOCK. Why ? The recommended config includes both FD_SOCK *and* either FD (TCP) or FD_ALL (UDP). Mayank: we were using FD some months ago. But there are many false suspicions because of GC so we removed it. If you can come up with exact steps to reproduce this, or a small program, than that would be good. Then I could create a JIRA. Mayank: I haven't created a program to reproduce this. We are repeating this cases by deploying our app to system and killing member in this manner. I will try to reproduce this through a small program -- Bela Ban, JGroups lead (http://www.jgroups.org) ------------------------------------------------------------------------------ Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk _______________________________________________ Javagroups-development mailing list ________________________________ If you reply to this email, your message will be added to the discussion below: http://jgroups.1086181.n5.nabble.com/Problem-with-FD-SOCK-failure-detection-tp9847p9852.html To unsubscribe from Problem with FD_SOCK failure detection, click here<http://jgroups.1086181.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=9847&code=bWF5YW5rLmFnYXJ3YWxAZ3VhdnVzLmNvbXw5ODQ3fDI4NzIzNzg4Ng==>. NAML<http://jgroups.1086181.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> -- View this message in context: http://jgroups.1086181.n5.nabble.com/Problem-with-FD-SOCK-failure-detection-tp9847p9855.html Sent from the JGroups - Dev mailing list archive at Nabble.com. ------=_Part_175838_13916355.1377228685558 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <div>My Comments were inline. In case, you missed this.</div> <div><br> </div> <span id="OLK_SRC_BODY_SECTION"> <div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt"> <span style="font-weight:bold">From: </span>Mayank Agarwal <<a href="/user/SendEmail.jtp?type=node&node=9855&i=0" target="_top" rel="nofollow" link="external">[hidden email]</a>><br> <span style="font-weight:bold">Date: </span>Wednesday 21 August 2013 9:09 PM<br> <span style="font-weight:bold">To: </span>"Bela Ban [via JGroups]" <<a href="/user/SendEmail.jtp?type=node&node=9855&i=1" target="_top" rel="nofollow" link="external">[hidden email]</a>><br> <span style="font-weight:bold">Subject: </span>Re: Problem with FD_SOCK failure detection<br> </div> <div><br> </div> <div> <div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif; "> <div style="color: rgb(0, 0, 0); "><br> </div> <div style="color: rgb(0, 0, 0); "><br> </div> <span id="OLK_SRC_BODY_SECTION" style="color: rgb(0, 0, 0); "> <div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt"> <span style="font-weight:bold">From: </span>"Bela Ban [via JGroups]" <<a href="/user/SendEmail.jtp?type=node&node=9855&i=2" target="_top" rel="nofollow" link="external">[hidden email]</a>><br> <span style="font-weight:bold">Date: </span>Wednesday 21 August 2013 8:43 PM<br> <span style="font-weight:bold">To: </span>Mayank Agarwal <<a href="/user/SendEmail.jtp?type=node&node=9855&i=3" target="_top" rel="nofollow" link="external">[hidden email]</a>><br> <span style="font-weight:bold">Subject: </span>Re: Problem with FD_SOCK failure detection<br> </div> <div><br> </div> <div> <div>Works for me: I used udp.xml, commented FD_ALL and increased the timeout <br> in VERIFY_SUSPECT to 5000. <br> <br> The I started members A and B. Then I killed A (will get removed ca. 5 s <br> later) and immediately started C. C looped around the failed JOIN until <br> A was removed, then B joined and the new view was {B,C}. <br> </div> </div> </span> <div style="color: rgb(0, 0, 0); "><br> </div> <span id="OLK_SRC_BODY_SECTION"> <div> <div><font class="Apple-style-span" color="#7f0000">Mayank: The exact case is first start C and than kill A immediately. This happens intermittentlty (If you repeat the process many times)</font></div> </div> </span> <div><br> </div> <span id="OLK_SRC_BODY_SECTION"> <div> <div><br> Comments inine. <br> <br> On 8/21/13 12:29 PM, mayank.agarwal wrote: <br> > yes, this is reproducible. <br> > FD_SOCK start() method is not called as JChannel.connect call wasn't <br> > succeeded because it was trying to JOIN killed member(co-ordinator) <br> <br> This should not be the case as the killed coordinator will be removed <br> from the view, so a discovery should not return it anymore. <br> Do you use return_entire_cache="true" by any chance ? <br> </div> </div> </span> <div><br> </div> <div><font class="Apple-style-span" color="#7f0000">Mayank: return_entire_cache is by default false and we are not modifying this in xml</font></div> <span id="OLK_SRC_BODY_SECTION"> <div> <div><br> > there are three nodes in cluster. (member-89,member-101,member-88,member-87) <br> > we are getting lot of warnings mentioned below. it is trying to contact <br> > member-89(which is killed) <br> > [pool-3-thread-1] WARN o.jgroups.protocols.pbcast.GMS - <br> > JOIN(member-87-22015) sent to member-89-55050 timed out (after 7000 ms), on <br> > try 2852 <br> <br> It attempted to join 2852 times ? <br> Note that you can set GMS.max_join_attempts to put a ceiling to the <br> number of tries. <br> </div> </div> </span> <div><br> </div> <div><font class="Apple-style-span" color="#7f0000">Mayank: By default GMS.max_join_attempts is 0 which means 'never give up'. Can you suggest optimal value for this?</font></div> <span id="OLK_SRC_BODY_SECTION"> <div> <div><br> > On discovery request from member-87: member-88 is returning view <br> > [member-89-55050|10]. <br> <br> This shouldn't be the case, as member-89 should have been removed from <br> the view ! <br> What's the config you use ? <br> </div> </div> </span> <div><font class="Apple-style-span" color="#7f0000">Mayank: config is attached </font></div> <span id="OLK_SRC_BODY_SECTION"> <div> <div><br> > So it is always sending JOIN request to member-89(killed). <br> > <br> > member-88,89,101 has view {member-89,member-101,member-88,member-87} <br> > member-87 does't have any view. <br> > Scenario is: let say 2 A,B,C is in cluster new node D try to join and at <br> > the same time A goes down. than C has already closed the socket with A <br> > gracefully, assuming D will create the socket with A . D's FD_SOCK pinger <br> > thread is not started. ( i am not seeing the msg "pinger_thread started" in <br> > logs) <br> <br> <br> Yes, that would be the case if D always gets discovery response(s) which <br> still indicate A is the coordinator. This is the core issue, and we need <br> to find out why anyone's still returning A in its discovery response. <br> <br> <br> > We are not using combo of FD_SOCK with FD/FD_ALL. we are using only in <br> > FD_SOCK. <br> <br> Why ? The recommended config includes both FD_SOCK *and* either FD (TCP) <br> or FD_ALL (UDP). <br> </div> </div> </span> <div><font class="Apple-style-span" color="#7f0000"><br> </font></div> <div><font class="Apple-style-span" color="#7f0000">Mayank: we were using FD some months ago. But there are many false suspicions because of GC so we removed it.</font></div> <span id="OLK_SRC_BODY_SECTION"> <div> <div><br> If you can come up with exact steps to reproduce this, or a small <br> program, than that would be good. Then I could create a JIRA. </div> </div> </span> <div><br> </div> <div><font class="Apple-style-span" color="#7f0000">Mayank: I haven't created a program to reproduce this. We are repeating this cases by deploying our app to system and killing member in this manner.</font></div> <div><font class="Apple-style-span" color="#7f0000">I will try to reproduce this through a small program</font></div> <span id="OLK_SRC_BODY_SECTION"> <div> <div><br> -- <br> Bela Ban, JGroups lead (<a href="http://www.jgroups.org" target="_top" rel="nofollow" link="external" style="color: rgb(0, 0, 0); ">http://www.jgroups.org</a>) <br> <br> ------------------------------------------------------------------------------ <br> Introducing Performance Central, a new site from SourceForge and <br> AppDynamics. Performance Central is your source for news, insights, <br> analysis and resources for efficient Application Performance Management. <br> Visit us today! <br> <a href="http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk" target="_top" rel="nofollow" link="external" style="color: rgb(0, 0, 0); ">http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk</a><br> _______________________________________________ <br> Javagroups-development mailing list <br> <br> <br> <br> <hr noshade="noshade" size="1" color="#cccccc" style="color: rgb(0, 0, 0); "> <div style="color: rgb(68, 68, 68); font: normal normal normal 12px/normal tahoma, geneva, helvetica, arial, sans-serif; "> <div style="font-weight:bold">If you reply to this email, your message will be added to the discussion below:</div> <a href="http://jgroups.1086181.n5.nabble.com/Problem-with-FD-SOCK-failure-detection-tp9847p9852.html" target="_top" rel="nofollow" link="external">http://jgroups.1086181.n5.nabble.com/Problem-with-FD-SOCK-failure-detection-tp9847p9852.html</a></div> <div style="color: rgb(102, 102, 102); font: normal normal normal 11px/normal tahoma, geneva, helvetica, arial, sans-serif; margin-top: 0.4em; line-height: 1.5em; "> To unsubscribe from Problem with FD_SOCK failure detection, <a href="" target="_top" rel="nofollow" link="external"> click here</a>.<br> <a href="http://jgroups.1086181.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml" rel="nofollow" style="font:9px serif" target="_top" link="external">NAML</a></div> </div> </div> </span></div> </div> </span> <br/><hr align="left" width="300" /> View this message in context: <a href="http://jgroups.1086181.n5.nabble.com/Problem-with-FD-SOCK-failure-detection-tp9847p9855.html">Re: Problem with FD_SOCK failure detection</a><br/> Sent from the <a href="http://jgroups.1086181.n5.nabble.com/JGroups-Dev-f6604.html">JGroups - Dev mailing list archive</a> at Nabble.com.<br/> ------=_Part_175838_13916355.1377228685558-- --===============1269102849848514144== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk --===============1269102849848514144== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Javagroups-development mailing list --===============1269102849848514144==--