Re: Odd weight and healthcheck issue?
Daniel Selans <[email protected]> Tue, 11 Sep 2007 12:18:02 -0400
| Newsgroups | gmane.linux.keepalived.announce |
|---|---|
| Message-ID | <[email protected]> |
--===============0306018634== Content-Type: multipart/alternative; boundary=Apple-Mail-2-766885423 --Apple-Mail-2-766885423 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed I am certain it wasn't a DoS; I ran tcpdump on the director, and all looked fine. As a matter of fact, if the amount of requests that the .49 box was receiving would be spread out according by weight across all of the servers, it would all match up to regular use. Also, the recursor answers queries only to our internal network, so chances of a DoS are very low. I have a MISC_CHECK doing a dig with a timeout of 8s for each host; looking at the logs and ps output, keepalived was executing it, but never reported it to be down, even though the DNS service was unreachable. Granted this hasn't happened since then, but I certainly would like to make sure that this doesn't happen again either. TIA On Sep 10, 2007, at 2:01 AM, Brad Chang wrote: > is it possible that you where getting all these requests froma > single host > (like a DOS)? did you check the connection table to see if its > random hosts? > > thanks > -Brad > > Quoting Daniel Selans <[email protected]>: > >> Hello list, >> >> This morning I woke up to a call from one of one of our techs at the >> company saying that our DNS cluster was down. After quickly checking >> the ipvs table on the director, I saw something really odd: >> >> -> RemoteAddress:Port Forward Weight ActiveConn >> InActConn >> UDP dns1.dimenoc.com:domain wlc persistent 30 >> -> 10.85.83.49:domain Masq 1 0 450573 >> -> 10.85.83.48:domain Masq 2 0 10504 >> -> 10.85.83.47:domain Masq 2 0 2574 >> -> 10.85.83.46:domain Masq 2 0 1842 >> >> Let me give you some insight on the setup - .46, .47, .48 are dual >> xeons, thus the weight of 2 and .49 is a P4. All of the dual xeon >> machines were up and responding, although the .49 was clearly >> overloaded and not even responding to ssh. Only after clearing the >> ipvs tables, and restarting keepalived was I able to get those >> numbers to start balancing out. >> >> How is this possible? Keepalived should've removed the .49 machine, >> and should've started distributing load across the machines with >> barely any connections. Yet .49 was left around and continued taking >> on connections while the dual xeons didn't get any work at all. >> >> If anyone has any input on this issue, I'd greatly appreciate it. >> >> The LVS director is on a dual xeon as well, on a 2.6.9-smp kernel, if >> this matters, and same goes for the rest of the machines. >> >> Daniel Selans >> Sr. Systems Administrator/Network Engineer >> Hostdime.com, Inc. | http://www.hostdime.com/ >> >> >> >> --------------------------------------------------------------------- >> ---- >> This SF.net email is sponsored by: Splunk Inc. >> Still grepping through log files to find problems? Stop. >> Now Search log events and configuration files using AJAX and a >> browser. >> Download your FREE copy of Splunk now >> http://get.splunk.com/ >> _______________________________________________ >> Keepalived-announce mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/keepalived-announce >> > > > > > > > Thanks > -Brad Chang > -http://www.dotnoc.com > > > ---------------------------------------------------------------- > http://www.dotnoc.com hosting the way it should be. > Daniel Selans Sr. Systems Administrator/Network Engineer Hostdime.com, Inc. | http://www.hostdime.com/ --Apple-Mail-2-766885423 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=ISO-8859-1 <html><body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space; ">I am certain it wasn't a DoS; I = ran tcpdump on the director, and all looked fine.<div><br = class=3D"webkit-block-placeholder"></div><div>As a matter of fact, if = the amount of requests that the .49 box was receiving=A0would be spread = out according by weight across all of the servers, it would all match up = to regular use.</div><div>Also, the recursor answers queries only to our = internal network, so chances of a DoS are very = low.</div><div><br></div><div>I have a MISC_CHECK doing a dig with a = timeout of 8s for each host; looking at the logs and ps output, = =A0keepalived was executing it, but never reported it to be down, even = though the DNS service was unreachable.</div><div><br = class=3D"webkit-block-placeholder"></div><div>Granted this hasn't = happened since then, but I certainly would like to make sure that this = doesn't happen again either.</div><div><br = class=3D"webkit-block-placeholder"></div><div>TIA</div><div><br = class=3D"webkit-block-placeholder"></div><div><br = class=3D"webkit-block-placeholder"></div><div><br = class=3D"webkit-block-placeholder"></div><div><div><div>On Sep 10, 2007, = at 2:01 AM, Brad Chang wrote:</div><br = class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; ">is it possible that you where getting all these = requests froma<span class=3D"Apple-converted-space">=A0 </span>single = host</div><div style=3D"margin-top: 0px; margin-right: 0px; = margin-bottom: 0px; margin-left: 0px; ">(like a DOS)? did you check the = connection table to see if its random hosts?</div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; min-height: 14px; "><br></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; = ">thanks</div><div style=3D"margin-top: 0px; margin-right: 0px; = margin-bottom: 0px; margin-left: 0px; ">-Brad</div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; min-height: 14px; "><br></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Quoting = Daniel Selans <<a = href=3D"mailto:[email protected]">[email protected]</a>>:</div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; min-height: 14px; "><br></div> <blockquote = type=3D"cite"><div style=3D"margin-top: 0px; margin-right: 0px; = margin-bottom: 0px; margin-left: 0px; ">Hello list,</div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; min-height: 14px; "><br></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">This = morning I woke up to a call from one of one of our techs at the <span = class=3D"Apple-converted-space">=A0</span></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">company = saying that our DNS cluster was down. After quickly checking <span = class=3D"Apple-converted-space">=A0</span></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">the ipvs = table on the director, I saw something really odd:</div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; min-height: 14px; "><br></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><span = class=3D"Apple-converted-space">=A0=A0 </span>-> RemoteAddress:Port = <span class=3D"Apple-converted-space">=A0 =A0 =A0 =A0 =A0 </span>Forward = Weight ActiveConn InActConn</div><div style=3D"margin-top: 0px; = margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">UDP<span = class=3D"Apple-converted-space">=A0 </span>dns1.dimenoc.com:domain wlc = persistent 30</div><div style=3D"margin-top: 0px; margin-right: 0px; = margin-bottom: 0px; margin-left: 0px; "><span = class=3D"Apple-converted-space">=A0=A0 </span>-> 10.85.83.49:domain = <span class=3D"Apple-converted-space">=A0 =A0 =A0 =A0 =A0 = </span>Masq<span class=3D"Apple-converted-space">=A0 =A0 </span>1<span = class=3D"Apple-converted-space">=A0 =A0 =A0 </span>0<span = class=3D"Apple-converted-space">=A0 =A0 =A0 =A0 =A0 = </span>450573</div><div style=3D"margin-top: 0px; margin-right: 0px; = margin-bottom: 0px; margin-left: 0px; "><span = class=3D"Apple-converted-space">=A0=A0 </span>-> 10.85.83.48:domain = <span class=3D"Apple-converted-space">=A0 =A0 =A0 =A0 =A0 = </span>Masq<span class=3D"Apple-converted-space">=A0 =A0 </span>2<span = class=3D"Apple-converted-space">=A0 =A0 =A0 </span>0<span = class=3D"Apple-converted-space">=A0 =A0 =A0 =A0 =A0 = </span>10504</div><div style=3D"margin-top: 0px; margin-right: 0px; = margin-bottom: 0px; margin-left: 0px; "><span = class=3D"Apple-converted-space">=A0=A0 </span>-> 10.85.83.47:domain = <span class=3D"Apple-converted-space">=A0 =A0 =A0 =A0 =A0 = </span>Masq<span class=3D"Apple-converted-space">=A0 =A0 </span>2<span = class=3D"Apple-converted-space">=A0 =A0 =A0 </span>0<span = class=3D"Apple-converted-space">=A0 =A0 =A0 =A0 =A0 = </span>2574</div><div style=3D"margin-top: 0px; margin-right: 0px; = margin-bottom: 0px; margin-left: 0px; "><span = class=3D"Apple-converted-space">=A0=A0 </span>-> 10.85.83.46:domain = <span class=3D"Apple-converted-space">=A0 =A0 =A0 =A0 =A0 = </span>Masq<span class=3D"Apple-converted-space">=A0 =A0 </span>2<span = class=3D"Apple-converted-space">=A0 =A0 =A0 </span>0<span = class=3D"Apple-converted-space">=A0 =A0 =A0 =A0 =A0 = </span>1842</div><div style=3D"margin-top: 0px; margin-right: 0px; = margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; ">Let me give you some insight on the setup - .46, = .47, .48 are dual <span class=3D"Apple-converted-space">=A0</span></div><d= iv style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; ">xeons, thus the weight of 2 and .49 is a P4. All of = the dual xeon <span class=3D"Apple-converted-space">=A0</span></div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; ">machines were up and responding, although the .49 = was clearly <span class=3D"Apple-converted-space">=A0</span></div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; ">overloaded and not even responding to ssh. Only = after clearing the <span = class=3D"Apple-converted-space">=A0</span></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">ipvs = tables, and restarting keepalived was I able to get those <span = class=3D"Apple-converted-space">=A0</span></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">numbers = to start balancing out.</div><div style=3D"margin-top: 0px; = margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: = 14px; "><br></div><div style=3D"margin-top: 0px; margin-right: 0px; = margin-bottom: 0px; margin-left: 0px; ">How is this possible? Keepalived = should've removed the .49 machine, <span = class=3D"Apple-converted-space">=A0</span></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">and = should've started distributing load across the machines with <span = class=3D"Apple-converted-space">=A0</span></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">barely = any connections. Yet .49 was left around and continued taking <span = class=3D"Apple-converted-space">=A0</span></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">on = connections while the dual xeons didn't get any work at all.</div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; min-height: 14px; "><br></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">If = anyone has any input on this issue, I'd greatly appreciate it.</div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; min-height: 14px; "><br></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">The LVS = director is on a dual xeon as well, on a 2.6.9-smp kernel, if <span = class=3D"Apple-converted-space">=A0</span></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">this = matters, and same goes for the rest of the machines.</div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; min-height: 14px; "><br></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Daniel = Selans</div><div style=3D"margin-top: 0px; margin-right: 0px; = margin-bottom: 0px; margin-left: 0px; ">Sr. Systems = Administrator/Network Engineer</div><div style=3D"margin-top: 0px; = margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Hostdime.com, = Inc. | <a = href=3D"http://www.hostdime.com">http://www.hostdime.com</a>/</div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; min-height: 14px; "><br></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; = min-height: 14px; "><br></div><div style=3D"margin-top: 0px; = margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: = 14px; "><br></div><div style=3D"margin-top: 0px; margin-right: 0px; = margin-bottom: 0px; margin-left: 0px; = ">------------------------------------------------------------------------= -</div><div style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: = 0px; margin-left: 0px; ">This SF.net email is sponsored by: Splunk = Inc.</div><div style=3D"margin-top: 0px; margin-right: 0px; = margin-bottom: 0px; margin-left: 0px; ">Still grepping through log files = to find problems?<span class=3D"Apple-converted-space">=A0 = </span>Stop.</div><div style=3D"margin-top: 0px; margin-right: 0px; = margin-bottom: 0px; margin-left: 0px; ">Now Search log events and = configuration files using AJAX and a browser.</div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; ">Download your FREE copy of Splunk now >><span = class=3D"Apple-converted-space">=A0 </span><a = href=3D"http://get.splunk.com">http://get.splunk.com</a>/</div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; = ">_______________________________________________</div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; ">Keepalived-announce mailing list</div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; "><a = href=3D"mailto:[email protected]">Keepalived-annou= [email protected]</a></div><div style=3D"margin-top: 0px; = margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><a = href=3D"https://lists.sourceforge.net/lists/listinfo/keepalived-announce">= https://lists.sourceforge.net/lists/listinfo/keepalived-announce</a></div>= <div style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; min-height: 14px; "><br></div> </blockquote><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; min-height: 14px; "><br></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; = min-height: 14px; "><br></div><div style=3D"margin-top: 0px; = margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: = 14px; "><br></div><div style=3D"margin-top: 0px; margin-right: 0px; = margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; min-height: 14px; "><br></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; = min-height: 14px; "><br></div><div style=3D"margin-top: 0px; = margin-right: 0px; margin-bottom: 0px; margin-left: 0px; = ">Thanks</div><div style=3D"margin-top: 0px; margin-right: 0px; = margin-bottom: 0px; margin-left: 0px; ">-Brad Chang</div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; ">-http://<a = href=3D"http://www.dotnoc.com">www.dotnoc.com</a></div><div = style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; min-height: 14px; "><br></div><div style=3D"margin-top: = 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; = min-height: 14px; "><br></div><div style=3D"margin-top: 0px; = margin-right: 0px; margin-bottom: 0px; margin-left: 0px; = ">----------------------------------------------------------------</div><d= iv style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = margin-left: 0px; "><a = href=3D"http://www.dotnoc.com">http://www.dotnoc.com</a> hosting the way = it should be.</div><div style=3D"margin-top: 0px; margin-right: 0px; = margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div> = </blockquote></div><br><div> <span class=3D"Apple-style-span" = style=3D"border-collapse: separate; border-spacing: 0px 0px; color: = rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; text-align: auto; = -khtml-text-decorations-in-effect: none; text-indent: 0px; = -apple-text-size-adjust: auto; text-transform: none; orphans: 2; = white-space: normal; widows: 2; word-spacing: 0px; "><div>Daniel = Selans</div><div>Sr. Systems Administrator/Network = Engineer</div><div>Hostdime.com, Inc. | <a = href=3D"http://www.hostdime.com">http://www.hostdime.com</a>/</div><br = class=3D"Apple-interchange-newline"></span> = </div><br></div></body></html>= --Apple-Mail-2-766885423-- --===============0306018634== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ --===============0306018634== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Keepalived-announce mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/keepalived-announce --===============0306018634==--