LVS start to malfunction after few hours
Michael Ben-Nes <[email protected]> Tue, 19 Jan 2010 10:10:56 +0200
| Newsgroups | gmane.linux.redhat.piranha |
|---|---|
| Message-ID | <[email protected]> |
--===============7300019653140716704==
Content-Type: multipart/alternative; boundary=001485f63170670170047d80021a
--001485f63170670170047d80021a
Content-Type: text/plain; charset=ISO-8859-1
Hi,
My fresh LVS installation start to malfunction after few hours.
I use piranha / pulse on CentOS 5.4 to RR between two Nginx server ( only
static files, not persistent ).
When I start the service everything work as expected. After few hours one of
the real servers become unavailable ( randomly ) even though ipvsadm show it
is ok.
The real server which is not answering is continuously been accessed by the
nanny process every 6 sec and is accessible through its real IP.
The only clue I found is that instead of 2 nanny process there are 4 nanny
process ( 2 for each server ).
logs show nothing of interest while pulse ruining.
When I stop pulse all the related process are terminated beside the 4
nannies which I need to kill by hand:
Here is a snip from the logs:
Jan 18 09:57:21 blb1 pulse[5795]: Terminating due to signal 15
Jan 18 09:57:21 blb1 lvs[5798]: shutting down due to signal 15
Jan 18 09:57:21 blb1 lvs[5798]: shutting down virtual service Nginx
Jan 18 09:59:15 blb1 nanny[2668]: Terminating due to signal 15
Jan 18 09:59:15 blb1 nanny[2670]: Terminating due to signal 15
Jan 18 09:59:15 blb1 nanny[5812]: Terminating due to signal 15
Jan 18 09:59:15 blb1 nanny[2668]: /sbin/ipvsadm command failed!
Jan 18 09:59:15 blb1 nanny[5812]: /sbin/ipvsadm command failed!
Jan 18 09:59:15 blb1 nanny[2670]: /sbin/ipvsadm command failed!
Jan 18 09:59:15 blb1 nanny[5813]: Terminating due to signal 15
Jan 18 09:59:15 blb1 nanny[5813]: /sbin/ipvsadm command failed!
Here is the data I gathered while the build malfunction:
####### LVS server ( no backup server )
piranha-0.8.4-13.el5 - ipvsadm-1.24-10
# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
# cat /etc/sysconfig/ha/lvs.cf
serial_no = 37
primary = 82.81.215.137
service = lvs
backup = 0.0.0.0
heartbeat = 1
heartbeat_port = 539
keepalive = 6
deadtime = 18
network = direct
nat_nmask = 255.255.255.255
debug_level = NONE
virtual Nginx {
active = 1
address = 82.81.215.141 eth0:1
vip_nmask = 255.255.255.224
port = 80
send = "GET / HTTP/1.0\r\n\r\n"
expect = "HTTP"
use_regex = 0
load_monitor = none
scheduler = wlc # Suppose to be RR - changed only to test if the
scheduler is the problem - same effect
protocol = tcp
timeout = 6
reentry = 15
quiesce_server = 1
server bweb1.my-domain.com {
address = 82.81.215.138
active = 1
weight = 1
}
server bweb2.my-domain.com {
address = 82.81.215.139
active = 1
weight = 1
}
server bweb3.my-domain.com {
address = 82.81.215.140
active = 0
weight = 1
}
}
# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 82.81.215.141:80 wlc
-> 82.81.215.139:80 Route 1 0 0
-> 82.81.215.138:80 Route 1 0 0
# ps auxw|egrep "nanny|ipv|lvs|pulse"
root 2668 0.0 0.0 8456 692 ? Ss Jan16 0:00
/usr/sbin/nanny -c -h 82.81.215.138 -p 80 -r 80 -s GET / HTTP/1.0\r\n\r\n -x
HTTP -q -a 15 -I /sbin/ipvsadm -t 6 -w 1 -V 82.81.215.141 -M g -U none --lvs
root 2670 0.0 0.0 8456 688 ? Ss Jan16 0:00
/usr/sbin/nanny -c -h 82.81.215.139 -p 80 -r 80 -s GET / HTTP/1.0\r\n\r\n -x
HTTP -q -a 15 -I /sbin/ipvsadm -t 6 -w 1 -V 82.81.215.141 -M g -U none --lvs
root 5795 0.0 0.0 8488 372 ? Ss Jan17 0:00 pulse
root 5798 0.0 0.0 8476 656 ? Ss Jan17 0:00
/usr/sbin/lvsd --nofork -c /etc/sysconfig/ha/lvs.cf
root 5812 0.0 0.0 8456 692 ? Ss Jan17 0:00
/usr/sbin/nanny -c -h 82.81.215.138 -p 80 -r 80 -s GET / HTTP/1.0\r\n\r\n -x
HTTP -q -a 15 -I /sbin/ipvsadm -t 6 -w 1 -V 82.81.215.141 -M g -U none --lvs
root 5813 0.0 0.0 8456 692 ? Ss Jan17 0:00
/usr/sbin/nanny -c -h 82.81.215.139 -p 80 -r 80 -s GET / HTTP/1.0\r\n\r\n -x
HTTP -q -a 15 -I /sbin/ipvsadm -t 6 -w 1 -V 82.81.215.141 -M g -U none --lvs
####### One of the servers ( the one that does not answer. though its
identical to the other )
# arptables -L -n
Chain IN (policy ACCEPT)
target source-ip destination-ip source-hw
destination-hw hlen op hrd pro
DROP 0.0.0.0/0 82.81.215.141 00/00
00/00 any 0000/0000 0000/0000 0000/0000
Chain OUT (policy ACCEPT)
target source-ip destination-ip source-hw
destination-hw hlen op hrd pro
mangle 0.0.0.0/0 82.81.215.141 00/00
00/00 any 0000/0000 0000/0000 0000/0000 --mangle-ip-s
82.81.215.139
Chain FORWARD (policy ACCEPT)
target source-ip destination-ip source-hw
destination-hw hlen op hrd pro
# ifconfig
eth0 Link encap:Ethernet HWaddr 00:11:25:41:69:A4
inet addr:82.81.215.139 Bcast:82.81.215.159 Mask:255.255.255.224
inet6 addr: fe80::211:25ff:fe41:69a4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:602454 errors:0 dropped:0 overruns:0 frame:0
TX packets:514536 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:51144864 (48.7 MiB) TX bytes:251901147 (240.2 MiB)
Interrupt:169 Memory:dcff0000-dd000000
eth0:1 Link encap:Ethernet HWaddr 00:11:25:41:69:A4
inet addr:82.81.215.141 Bcast:82.81.215.159 Mask:255.255.255.224
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:169 Memory:dcff0000-dd000000
Thanks for any idea that might shade some light on this topic :)
Best,
Miki
--------------------------------------------------
Michael Ben-Nes - Internet Consultant and Director.
http://www.epoch.co.il - weaving the Net.
Cellular: 054-4848113
--------------------------------------------------
--001485f63170670170047d80021a
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><span style=3D"font-family:arial, sans-serif;font-size:13p=
x;border-collapse:collapse"><div>Hi,</div><div><br></div><div>My fresh LVS =
installation start to malfunction after few hours.</div>
<div><br></div><div>I use piranha / pulse on CentOS 5.4 to RR between two N=
ginx server ( only static files, not persistent ).</div><div><br></div><div=
>When I start the service everything work as expected. After few hours one =
of the real servers become unavailable ( randomly ) even though ipvsadm sho=
w it is ok.</div>
<div>The real server which is not answering is continuously been accessed b=
y the nanny process every 6 sec and is accessible through its real IP.</div=
><div><br></div><div>The only clue I found is that instead of 2 nanny proce=
ss there are 4 nanny process ( 2 for each server ).</div>
<div>logs show nothing of interest while pulse ruining.<br></div><div><br><=
/div><div>When I stop pulse all the related process are terminated beside t=
he 4 nannies which I need to kill by hand:</div><div><br></div><div>Here is=
a snip from the logs:</div>
<div><div>Jan 18 09:57:21 blb1 pulse[5795]: Terminating due to signal 15</d=
iv><div>Jan 18 09:57:21 blb1 lvs[5798]: shutting down due to signal 15</div=
><div>Jan 18 09:57:21 blb1 lvs[5798]: shutting down virtual service Nginx</=
div>
<div>Jan 18 09:59:15 blb1 nanny[2668]: Terminating due to signal 15</div><d=
iv>Jan 18 09:59:15 blb1 nanny[2670]: Terminating due to signal 15</div><div=
>Jan 18 09:59:15 blb1 nanny[5812]: Terminating due to signal 15</div><div>
Jan 18 09:59:15 blb1 nanny[2668]: /sbin/ipvsadm command failed!</div><div>J=
an 18 09:59:15 blb1 nanny[5812]: /sbin/ipvsadm command failed!</div><div>Ja=
n 18 09:59:15 blb1 nanny[2670]: /sbin/ipvsadm command failed!</div><div>
Jan 18 09:59:15 blb1 nanny[5813]: Terminating due to signal 15</div><div>Ja=
n 18 09:59:15 blb1 nanny[5813]: /sbin/ipvsadm command failed!</div><div><br=
></div></div><div><br></div><div>Here is the data I gathered while the buil=
d malfunction:</div>
<div><br></div><div>####### LVS server ( no backup server )</div><div><br><=
/div><div>piranha-0.8.4-13.el5 - ipvsadm-1.24-10</div><div><br></div><div><=
div># sysctl net.ipv4.ip_forward</div><div>net.ipv4.ip_forward =3D 1</div>
<div><br></div></div><div># cat /etc/sysconfig/ha/<a href=3D"http://lvs.cf"=
style=3D"color:rgb(42, 93, 176)" target=3D"_blank">lvs.cf</a>=A0</div><div=
>serial_no =3D 37</div><div>primary =3D 82.81.215.137</div><div>service =3D=
lvs</div>
<div>backup =3D 0.0.0.0</div><div>heartbeat =3D 1</div><div>heartbeat_port =
=3D 539</div><div>keepalive =3D 6</div><div>deadtime =3D 18</div><div>netwo=
rk =3D direct</div><div>nat_nmask =3D 255.255.255.255</div><div>debug_level=
=3D NONE</div>
<div>virtual Nginx {</div><div>=A0=A0 =A0 active =3D 1</div><div>=A0=A0 =A0=
address =3D 82.81.215.141 eth0:1</div><div>=A0=A0 =A0 vip_nmask =3D 255.25=
5.255.224</div><div>=A0=A0 =A0 port =3D 80</div><div>=A0=A0 =A0 send =3D &q=
uot;GET / HTTP/1.0\r\n\r\n"</div>
<div>=A0=A0 =A0 expect =3D "HTTP"</div><div>=A0=A0 =A0 use_regex =
=3D 0</div><div>=A0=A0 =A0 load_monitor =3D none</div><div>=A0=A0 =A0 sched=
uler =3D wlc =A0# Suppose to be RR - changed only to test if the scheduler =
is the problem - same effect</div>
<div>=A0=A0 =A0 protocol =3D tcp</div><div>=A0=A0 =A0 timeout =3D 6</div><d=
iv>=A0=A0 =A0 reentry =3D 15</div><div>=A0=A0 =A0 quiesce_server =3D 1</div=
><div>=A0=A0 =A0 server=A0<a href=3D"http://bweb1.my-domain.com" style=3D"c=
olor:rgb(42, 93, 176)" target=3D"_blank">bweb1.my-domain.com</a>=A0{</div>
<div>=A0=A0 =A0 =A0 =A0 address =3D 82.81.215.138</div><div>=A0=A0 =A0 =A0 =
=A0 active =3D 1</div><div>=A0=A0 =A0 =A0 =A0 weight =3D 1</div><div>=A0=A0=
=A0 }</div><div>=A0=A0 =A0 server=A0<a href=3D"http://bweb2.my-domain.com"=
style=3D"color:rgb(42, 93, 176)" target=3D"_blank">bweb2.my-domain.com</a>=
=A0{</div>
<div>=A0=A0 =A0 =A0 =A0 address =3D 82.81.215.139</div><div>=A0=A0 =A0 =A0 =
=A0 active =3D 1</div><div>=A0=A0 =A0 =A0 =A0 weight =3D 1</div><div>=A0=A0=
=A0 }</div><div>=A0=A0 =A0 server=A0<a href=3D"http://bweb3.my-domain.com"=
style=3D"color:rgb(42, 93, 176)" target=3D"_blank">bweb3.my-domain.com</a>=
=A0{</div>
<div>=A0=A0 =A0 =A0 =A0 address =3D 82.81.215.140</div><div>=A0=A0 =A0 =A0 =
=A0 active =3D 0</div><div>=A0=A0 =A0 =A0 =A0 weight =3D 1</div><div>=A0=A0=
=A0 }</div><div>}</div><div><br></div><div># ipvsadm -L -n</div><div>IP Vi=
rtual Server version 1.2.1 (size=3D4096)</div>
<div>Prot LocalAddress:Port Scheduler Flags</div><div>=A0=A0-> RemoteAdd=
ress:Port =A0 =A0 =A0 =A0 =A0 Forward Weight ActiveConn InActConn</div><div=
>TCP =A0<a href=3D"http://82.81.215.141:80" style=3D"color:rgb(42, 93, 176)=
" target=3D"_blank">82.81.215.141:80</a>=A0wlc</div>
<div>=A0=A0->=A0<a href=3D"http://82.81.215.139:80" style=3D"color:rgb(4=
2, 93, 176)" target=3D"_blank">82.81.215.139:80</a>=A0=A0 =A0 =A0 =A0 =A0 =
=A0 Route =A0 1 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0=A0</div><=
div>=A0=A0->=A0<a href=3D"http://82.81.215.138:80" style=3D"color:rgb(42=
, 93, 176)" target=3D"_blank">82.81.215.138:80</a>=A0=A0 =A0 =A0 =A0 =A0 =
=A0 Route =A0 1 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0=A0</div>
<div><br></div><div><br></div><div># ps auxw|egrep "nanny|ipv|lvs|puls=
e"</div><div>root =A0 =A0 =A02668 =A00.0 =A00.0 =A0 8456 =A0 692 ? =A0=
=A0 =A0 =A0Ss =A0 Jan16 =A0 0:00 /usr/sbin/nanny -c -h 82.81.215.138 -p 80=
-r 80 -s GET / HTTP/1.0\r\n\r\n -x HTTP -q -a 15 -I /sbin/ipvsadm -t 6 -w =
1 -V 82.81.215.141 -M g -U none --lvs</div>
<div>root =A0 =A0 =A02670 =A00.0 =A00.0 =A0 8456 =A0 688 ? =A0 =A0 =A0 =A0S=
s =A0 Jan16 =A0 0:00 /usr/sbin/nanny -c -h 82.81.215.139 -p 80 -r 80 -s GET=
/ HTTP/1.0\r\n\r\n -x HTTP -q -a 15 -I /sbin/ipvsadm -t 6 -w 1 -V 82.81.21=
5.141 -M g -U none --lvs</div>
<div>root =A0 =A0 =A05795 =A00.0 =A00.0 =A0 8488 =A0 372 ? =A0 =A0 =A0 =A0S=
s =A0 Jan17 =A0 0:00 pulse</div><div>root =A0 =A0 =A05798 =A00.0 =A00.0 =A0=
8476 =A0 656 ? =A0 =A0 =A0 =A0Ss =A0 Jan17 =A0 0:00 /usr/sbin/lvsd --nofor=
k -c /etc/sysconfig/ha/<a href=3D"http://lvs.cf" style=3D"color:rgb(42, 93,=
176)" target=3D"_blank">lvs.cf</a></div>
<div>root =A0 =A0 =A05812 =A00.0 =A00.0 =A0 8456 =A0 692 ? =A0 =A0 =A0 =A0S=
s =A0 Jan17 =A0 0:00 /usr/sbin/nanny -c -h 82.81.215.138 -p 80 -r 80 -s GET=
/ HTTP/1.0\r\n\r\n -x HTTP -q -a 15 -I /sbin/ipvsadm -t 6 -w 1 -V 82.81.21=
5.141 -M g -U none --lvs</div>
<div>root =A0 =A0 =A05813 =A00.0 =A00.0 =A0 8456 =A0 692 ? =A0 =A0 =A0 =A0S=
s =A0 Jan17 =A0 0:00 /usr/sbin/nanny -c -h 82.81.215.139 -p 80 -r 80 -s GET=
/ HTTP/1.0\r\n\r\n -x HTTP -q -a 15 -I /sbin/ipvsadm -t 6 -w 1 -V 82.81.21=
5.141 -M g -U none --lvs</div>
<div><br></div><div><br></div><div><br></div><div>####### One of the server=
s ( the one that does not answer. though its identical to the other )</div>=
<div><br></div><div># arptables -L -n</div><div>Chain IN (policy ACCEPT)</d=
iv>
<div>target =A0 =A0 source-ip =A0 =A0 =A0 =A0 =A0 =A0destination-ip =A0 =A0=
=A0 source-hw =A0 =A0 =A0 =A0 =A0destination-hw =A0 =A0 hlen =A0 op =A0 =
=A0 =A0 =A0 hrd =A0 =A0 =A0 =A0pro =A0 =A0 =A0=A0</div><div>DROP =A0 =A0 =
=A0=A0<a href=3D"http://0.0.0.0/0" style=3D"color:rgb(42, 93, 176)" target=
=3D"_blank">0.0.0.0/0</a>=A0=A0 =A0 =A0 =A0 =A0 =A082.81.215.141 =A0 =A0 =
=A0 =A000/00 =A0 =A0 =A0 =A0 =A0 =A0 =A000/00 =A0 =A0 =A0 =A0 =A0 =A0 =A0an=
y =A0 =A00000/0000 =A00000/0000 =A00000/0000=A0</div>
<div><br></div><div>Chain OUT (policy ACCEPT)</div><div>target =A0 =A0 sour=
ce-ip =A0 =A0 =A0 =A0 =A0 =A0destination-ip =A0 =A0 =A0 source-hw =A0 =A0 =
=A0 =A0 =A0destination-hw =A0 =A0 hlen =A0 op =A0 =A0 =A0 =A0 hrd =A0 =A0 =
=A0 =A0pro =A0 =A0 =A0=A0</div><div>mangle =A0 =A0=A0<a href=3D"http://0.0.=
0.0/0" style=3D"color:rgb(42, 93, 176)" target=3D"_blank">0.0.0.0/0</a>=A0=
=A0 =A0 =A0 =A0 =A0 =A082.81.215.141 =A0 =A0 =A0 =A000/00 =A0 =A0 =A0 =A0 =
=A0 =A0 =A000/00 =A0 =A0 =A0 =A0 =A0 =A0 =A0any =A0 =A00000/0000 =A00000/00=
00 =A00000/0000 --mangle-ip-s 82.81.215.139=A0</div>
<div><br></div><div>Chain FORWARD (policy ACCEPT)</div><div>target =A0 =A0 =
source-ip =A0 =A0 =A0 =A0 =A0 =A0destination-ip =A0 =A0 =A0 source-hw =A0 =
=A0 =A0 =A0 =A0destination-hw =A0 =A0 hlen =A0 op =A0 =A0 =A0 =A0 hrd =A0 =
=A0 =A0 =A0pro =A0 =A0 =A0=A0</div><div><br></div><div><br>
</div>
<div># ifconfig=A0</div><div>eth0 =A0 =A0 =A0Link encap:Ethernet =A0HWaddr =
00:11:25:41:69:A4 =A0</div><div>=A0=A0 =A0 =A0 =A0 =A0inet addr:82.81.215.1=
39 =A0Bcast:82.81.215.159 =A0Mask:255.255.255.224</div><div>=A0=A0 =A0 =A0 =
=A0 =A0inet6 addr: fe80::211:25ff:fe41:69a4/64 Scope:Link</div>
<div>=A0=A0 =A0 =A0 =A0 =A0UP BROADCAST RUNNING MULTICAST =A0MTU:1500 =A0Me=
tric:1</div><div>=A0=A0 =A0 =A0 =A0 =A0RX packets:602454 errors:0 dropped:0=
overruns:0 frame:0</div><div>=A0=A0 =A0 =A0 =A0 =A0TX packets:514536 error=
s:0 dropped:0 overruns:0 carrier:0</div>
<div>=A0=A0 =A0 =A0 =A0 =A0collisions:0 txqueuelen:1000=A0</div><div>=A0=A0=
=A0 =A0 =A0 =A0RX bytes:51144864 (48.7 MiB) =A0TX bytes:251901147 (240.2 M=
iB)</div><div>=A0=A0 =A0 =A0 =A0 =A0Interrupt:169 Memory:dcff0000-dd000000=
=A0</div><div><br></div><div>eth0:1 =A0 =A0Link encap:Ethernet =A0HWaddr 00=
:11:25:41:69:A4 =A0</div>
<div>=A0=A0 =A0 =A0 =A0 =A0inet addr:82.81.215.141 =A0Bcast:82.81.215.159 =
=A0Mask:255.255.255.224</div><div>=A0=A0 =A0 =A0 =A0 =A0UP BROADCAST RUNNIN=
G MULTICAST =A0MTU:1500 =A0Metric:1</div><div>=A0=A0 =A0 =A0 =A0 =A0Interru=
pt:169 Memory:dcff0000-dd000000=A0</div><div>
<br></div><div><br></div><div>Thanks for any idea that might shade some lig=
ht on this topic :)</div><div><br></div><div>Best,</div><div>Miki</div><div=
><br></div></span>--------------------------------------------------<br>
Michael Ben-Nes - Internet Consultant and Director.<br><a href=3D"http://ww=
w.epoch.co.il" target=3D"_blank">http://www.epoch.co.il</a> - weaving the N=
et.<br>Cellular: 054-4848113<br>-------------------------------------------=
-------<br>
</div>
--001485f63170670170047d80021a--
--===============7300019653140716704==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Piranha-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/piranha-list
--===============7300019653140716704==--