Re: LVS start to malfunction after few hours
Tapan Thapa <[email protected]> Tue, 19 Jan 2010 13:53:02 +0530
| Newsgroups | gmane.linux.redhat.piranha |
|---|---|
| Message-ID | <[email protected]> |
--===============5901058044263250296==
Content-Type: multipart/alternative; boundary=00151758856cb004a3047d802dc6
--00151758856cb004a3047d802dc6
Content-Type: text/plain; charset=ISO-8859-1
Hello,
I am not an expert but i have implemented piranha in my setup and it is
working fine so i am trying to help.
>From your configuration it seems that you have not enabled your second
server.
server bweb3.my-domain.com {
address = 82.81.215.140
*active = 0*
weight = 1
Please check this.
Regards
Tapan Thapa
On Tue, Jan 19, 2010 at 1:40 PM, Michael Ben-Nes <[email protected]>wrote:
> Hi,
>
> My fresh LVS installation start to malfunction after few hours.
>
> I use piranha / pulse on CentOS 5.4 to RR between two Nginx server ( only
> static files, not persistent ).
>
> When I start the service everything work as expected. After few hours one
> of the real servers become unavailable ( randomly ) even though ipvsadm show
> it is ok.
> The real server which is not answering is continuously been accessed by the
> nanny process every 6 sec and is accessible through its real IP.
>
> The only clue I found is that instead of 2 nanny process there are 4 nanny
> process ( 2 for each server ).
> logs show nothing of interest while pulse ruining.
>
> When I stop pulse all the related process are terminated beside the 4
> nannies which I need to kill by hand:
>
> Here is a snip from the logs:
> Jan 18 09:57:21 blb1 pulse[5795]: Terminating due to signal 15
> Jan 18 09:57:21 blb1 lvs[5798]: shutting down due to signal 15
> Jan 18 09:57:21 blb1 lvs[5798]: shutting down virtual service Nginx
> Jan 18 09:59:15 blb1 nanny[2668]: Terminating due to signal 15
> Jan 18 09:59:15 blb1 nanny[2670]: Terminating due to signal 15
> Jan 18 09:59:15 blb1 nanny[5812]: Terminating due to signal 15
> Jan 18 09:59:15 blb1 nanny[2668]: /sbin/ipvsadm command failed!
> Jan 18 09:59:15 blb1 nanny[5812]: /sbin/ipvsadm command failed!
> Jan 18 09:59:15 blb1 nanny[2670]: /sbin/ipvsadm command failed!
> Jan 18 09:59:15 blb1 nanny[5813]: Terminating due to signal 15
> Jan 18 09:59:15 blb1 nanny[5813]: /sbin/ipvsadm command failed!
>
>
> Here is the data I gathered while the build malfunction:
>
> ####### LVS server ( no backup server )
>
> piranha-0.8.4-13.el5 - ipvsadm-1.24-10
>
> # sysctl net.ipv4.ip_forward
> net.ipv4.ip_forward = 1
>
> # cat /etc/sysconfig/ha/lvs.cf
> serial_no = 37
> primary = 82.81.215.137
> service = lvs
> backup = 0.0.0.0
> heartbeat = 1
> heartbeat_port = 539
> keepalive = 6
> deadtime = 18
> network = direct
> nat_nmask = 255.255.255.255
> debug_level = NONE
> virtual Nginx {
> active = 1
> address = 82.81.215.141 eth0:1
> vip_nmask = 255.255.255.224
> port = 80
> send = "GET / HTTP/1.0\r\n\r\n"
> expect = "HTTP"
> use_regex = 0
> load_monitor = none
> scheduler = wlc # Suppose to be RR - changed only to test if the
> scheduler is the problem - same effect
> protocol = tcp
> timeout = 6
> reentry = 15
> quiesce_server = 1
> server bweb1.my-domain.com {
> address = 82.81.215.138
> active = 1
> weight = 1
> }
> server bweb2.my-domain.com {
> address = 82.81.215.139
> active = 1
> weight = 1
> }
> server bweb3.my-domain.com {
> address = 82.81.215.140
> active = 0
> weight = 1
> }
> }
>
> # ipvsadm -L -n
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 82.81.215.141:80 wlc
> -> 82.81.215.139:80 Route 1 0 0
> -> 82.81.215.138:80 Route 1 0 0
>
>
> # ps auxw|egrep "nanny|ipv|lvs|pulse"
> root 2668 0.0 0.0 8456 692 ? Ss Jan16 0:00
> /usr/sbin/nanny -c -h 82.81.215.138 -p 80 -r 80 -s GET / HTTP/1.0\r\n\r\n -x
> HTTP -q -a 15 -I /sbin/ipvsadm -t 6 -w 1 -V 82.81.215.141 -M g -U none --lvs
> root 2670 0.0 0.0 8456 688 ? Ss Jan16 0:00
> /usr/sbin/nanny -c -h 82.81.215.139 -p 80 -r 80 -s GET / HTTP/1.0\r\n\r\n -x
> HTTP -q -a 15 -I /sbin/ipvsadm -t 6 -w 1 -V 82.81.215.141 -M g -U none --lvs
> root 5795 0.0 0.0 8488 372 ? Ss Jan17 0:00 pulse
> root 5798 0.0 0.0 8476 656 ? Ss Jan17 0:00
> /usr/sbin/lvsd --nofork -c /etc/sysconfig/ha/lvs.cf
> root 5812 0.0 0.0 8456 692 ? Ss Jan17 0:00
> /usr/sbin/nanny -c -h 82.81.215.138 -p 80 -r 80 -s GET / HTTP/1.0\r\n\r\n -x
> HTTP -q -a 15 -I /sbin/ipvsadm -t 6 -w 1 -V 82.81.215.141 -M g -U none --lvs
> root 5813 0.0 0.0 8456 692 ? Ss Jan17 0:00
> /usr/sbin/nanny -c -h 82.81.215.139 -p 80 -r 80 -s GET / HTTP/1.0\r\n\r\n -x
> HTTP -q -a 15 -I /sbin/ipvsadm -t 6 -w 1 -V 82.81.215.141 -M g -U none --lvs
>
>
>
> ####### One of the servers ( the one that does not answer. though its
> identical to the other )
>
> # arptables -L -n
> Chain IN (policy ACCEPT)
> target source-ip destination-ip source-hw
> destination-hw hlen op hrd pro
> DROP 0.0.0.0/0 82.81.215.141 00/00
> 00/00 any 0000/0000 0000/0000 0000/0000
>
> Chain OUT (policy ACCEPT)
> target source-ip destination-ip source-hw
> destination-hw hlen op hrd pro
> mangle 0.0.0.0/0 82.81.215.141 00/00
> 00/00 any 0000/0000 0000/0000 0000/0000 --mangle-ip-s
> 82.81.215.139
>
> Chain FORWARD (policy ACCEPT)
> target source-ip destination-ip source-hw
> destination-hw hlen op hrd pro
>
>
> # ifconfig
> eth0 Link encap:Ethernet HWaddr 00:11:25:41:69:A4
> inet addr:82.81.215.139 Bcast:82.81.215.159
> Mask:255.255.255.224
> inet6 addr: fe80::211:25ff:fe41:69a4/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:602454 errors:0 dropped:0 overruns:0 frame:0
> TX packets:514536 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:51144864 (48.7 MiB) TX bytes:251901147 (240.2 MiB)
> Interrupt:169 Memory:dcff0000-dd000000
>
> eth0:1 Link encap:Ethernet HWaddr 00:11:25:41:69:A4
> inet addr:82.81.215.141 Bcast:82.81.215.159
> Mask:255.255.255.224
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> Interrupt:169 Memory:dcff0000-dd000000
>
>
> Thanks for any idea that might shade some light on this topic :)
>
> Best,
> Miki
>
> --------------------------------------------------
> Michael Ben-Nes - Internet Consultant and Director.
> http://www.epoch.co.il - weaving the Net.
> Cellular: 054-4848113
> --------------------------------------------------
>
> _______________________________________________
> Piranha-list mailing list
> [email protected]
> https://www.redhat.com/mailman/listinfo/piranha-list
>
--00151758856cb004a3047d802dc6
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Hello,<br><br>I am not an expert but i have implemented piranha in my setup=
and it is working fine so i am trying to help.<br><br>From your configurat=
ion it seems that you have not enabled your second server.<br><br><div>serv=
er=A0<a href=3D"http://bweb3.my-domain.com/" style=3D"color: rgb(42, 93, 17=
6);" target=3D"_blank">bweb3.my-domain.com</a>=A0{</div>
<div>=A0=A0 =A0 =A0 =A0 address =3D 82.81.215.140</div><div>=A0=A0 =A0 =A0 =
=A0 <b><span style=3D"color: rgb(255, 0, 0);">active =3D 0</span></b></div>=
<div>=A0=A0 =A0 =A0 =A0 weight =3D 1</div><br>Please check this.<br><br>Reg=
ards<br>Tapan Thapa<br><br><div class=3D"gmail_quote">
On Tue, Jan 19, 2010 at 1:40 PM, Michael Ben-Nes <span dir=3D"ltr"><<a h=
ref=3D"mailto:[email protected]">[email protected]</a>></span> wrote=
:<br><blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(=
204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div dir=3D"ltr"><span style=3D"font-family: arial,sans-serif; font-size: 1=
3px; border-collapse: collapse;"><div>Hi,</div><div><br></div><div>My fresh=
LVS installation start to malfunction after few hours.</div>
<div><br></div><div>I use piranha / pulse on CentOS 5.4 to RR between two N=
ginx server ( only static files, not persistent ).</div><div><br></div><div=
>When I start the service everything work as expected. After few hours one =
of the real servers become unavailable ( randomly ) even though ipvsadm sho=
w it is ok.</div>
<div>The real server which is not answering is continuously been accessed b=
y the nanny process every 6 sec and is accessible through its real IP.</div=
><div><br></div><div>The only clue I found is that instead of 2 nanny proce=
ss there are 4 nanny process ( 2 for each server ).</div>
<div>logs show nothing of interest while pulse ruining.<br></div><div><br><=
/div><div>When I stop pulse all the related process are terminated beside t=
he 4 nannies which I need to kill by hand:</div><div><br></div><div>Here is=
a snip from the logs:</div>
<div><div>Jan 18 09:57:21 blb1 pulse[5795]: Terminating due to signal 15</d=
iv><div>Jan 18 09:57:21 blb1 lvs[5798]: shutting down due to signal 15</div=
><div>Jan 18 09:57:21 blb1 lvs[5798]: shutting down virtual service Nginx</=
div>
<div>Jan 18 09:59:15 blb1 nanny[2668]: Terminating due to signal 15</div><d=
iv>Jan 18 09:59:15 blb1 nanny[2670]: Terminating due to signal 15</div><div=
>Jan 18 09:59:15 blb1 nanny[5812]: Terminating due to signal 15</div><div>
Jan 18 09:59:15 blb1 nanny[2668]: /sbin/ipvsadm command failed!</div><div>J=
an 18 09:59:15 blb1 nanny[5812]: /sbin/ipvsadm command failed!</div><div>Ja=
n 18 09:59:15 blb1 nanny[2670]: /sbin/ipvsadm command failed!</div><div>
Jan 18 09:59:15 blb1 nanny[5813]: Terminating due to signal 15</div><div>Ja=
n 18 09:59:15 blb1 nanny[5813]: /sbin/ipvsadm command failed!</div><div><br=
></div></div><div><br></div><div>Here is the data I gathered while the buil=
d malfunction:</div>
<div><br></div><div>####### LVS server ( no backup server )</div><div><br><=
/div><div>piranha-0.8.4-13.el5 - ipvsadm-1.24-10</div><div><br></div><div><=
div># sysctl net.ipv4.ip_forward</div><div>net.ipv4.ip_forward =3D 1</div>
<div><br></div></div><div># cat /etc/sysconfig/ha/<a href=3D"http://lvs.cf"=
style=3D"color: rgb(42, 93, 176);" target=3D"_blank">lvs.cf</a>=A0</div><d=
iv>serial_no =3D 37</div><div>primary =3D 82.81.215.137</div><div>service =
=3D lvs</div>
<div>backup =3D 0.0.0.0</div><div>heartbeat =3D 1</div><div>heartbeat_port =
=3D 539</div><div>keepalive =3D 6</div><div>deadtime =3D 18</div><div>netwo=
rk =3D direct</div><div>nat_nmask =3D 255.255.255.255</div><div>debug_level=
=3D NONE</div>
<div>virtual Nginx {</div><div>=A0=A0 =A0 active =3D 1</div><div>=A0=A0 =A0=
address =3D 82.81.215.141 eth0:1</div><div>=A0=A0 =A0 vip_nmask =3D 255.25=
5.255.224</div><div>=A0=A0 =A0 port =3D 80</div><div>=A0=A0 =A0 send =3D &q=
uot;GET / HTTP/1.0\r\n\r\n"</div>
<div>=A0=A0 =A0 expect =3D "HTTP"</div><div>=A0=A0 =A0 use_regex =
=3D 0</div><div>=A0=A0 =A0 load_monitor =3D none</div><div>=A0=A0 =A0 sched=
uler =3D wlc =A0# Suppose to be RR - changed only to test if the scheduler =
is the problem - same effect</div>
<div>=A0=A0 =A0 protocol =3D tcp</div><div>=A0=A0 =A0 timeout =3D 6</div><d=
iv>=A0=A0 =A0 reentry =3D 15</div><div>=A0=A0 =A0 quiesce_server =3D 1</div=
><div>=A0=A0 =A0 server=A0<a href=3D"http://bweb1.my-domain.com" style=3D"c=
olor: rgb(42, 93, 176);" target=3D"_blank">bweb1.my-domain.com</a>=A0{</div=
>
<div>=A0=A0 =A0 =A0 =A0 address =3D 82.81.215.138</div><div>=A0=A0 =A0 =A0 =
=A0 active =3D 1</div><div>=A0=A0 =A0 =A0 =A0 weight =3D 1</div><div>=A0=A0=
=A0 }</div><div>=A0=A0 =A0 server=A0<a href=3D"http://bweb2.my-domain.com"=
style=3D"color: rgb(42, 93, 176);" target=3D"_blank">bweb2.my-domain.com</=
a>=A0{</div>
<div>=A0=A0 =A0 =A0 =A0 address =3D 82.81.215.139</div><div>=A0=A0 =A0 =A0 =
=A0 active =3D 1</div><div>=A0=A0 =A0 =A0 =A0 weight =3D 1</div><div>=A0=A0=
=A0 }</div><div>=A0=A0 =A0 server=A0<a href=3D"http://bweb3.my-domain.com"=
style=3D"color: rgb(42, 93, 176);" target=3D"_blank">bweb3.my-domain.com</=
a>=A0{</div>
<div>=A0=A0 =A0 =A0 =A0 address =3D 82.81.215.140</div><div>=A0=A0 =A0 =A0 =
=A0 active =3D 0</div><div>=A0=A0 =A0 =A0 =A0 weight =3D 1</div><div>=A0=A0=
=A0 }</div><div>}</div><div><br></div><div># ipvsadm -L -n</div><div>IP Vi=
rtual Server version 1.2.1 (size=3D4096)</div>
<div>Prot LocalAddress:Port Scheduler Flags</div><div>=A0=A0-> RemoteAdd=
ress:Port =A0 =A0 =A0 =A0 =A0 Forward Weight ActiveConn InActConn</div><div=
>TCP =A0<a href=3D"http://82.81.215.141:80" style=3D"color: rgb(42, 93, 176=
);" target=3D"_blank">82.81.215.141:80</a>=A0wlc</div>
<div>=A0=A0->=A0<a href=3D"http://82.81.215.139:80" style=3D"color: rgb(=
42, 93, 176);" target=3D"_blank">82.81.215.139:80</a>=A0=A0 =A0 =A0 =A0 =A0=
=A0 Route =A0 1 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0=A0</div>=
<div>=A0=A0->=A0<a href=3D"http://82.81.215.138:80" style=3D"color: rgb(=
42, 93, 176);" target=3D"_blank">82.81.215.138:80</a>=A0=A0 =A0 =A0 =A0 =A0=
=A0 Route =A0 1 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0=A0</div>
<div><br></div><div><br></div><div># ps auxw|egrep "nanny|ipv|lvs|puls=
e"</div><div>root =A0 =A0 =A02668 =A00.0 =A00.0 =A0 8456 =A0 692 ? =A0=
=A0 =A0 =A0Ss =A0 Jan16 =A0 0:00 /usr/sbin/nanny -c -h 82.81.215.138 -p 80=
-r 80 -s GET / HTTP/1.0\r\n\r\n -x HTTP -q -a 15 -I /sbin/ipvsadm -t 6 -w =
1 -V 82.81.215.141 -M g -U none --lvs</div>
<div>root =A0 =A0 =A02670 =A00.0 =A00.0 =A0 8456 =A0 688 ? =A0 =A0 =A0 =A0S=
s =A0 Jan16 =A0 0:00 /usr/sbin/nanny -c -h 82.81.215.139 -p 80 -r 80 -s GET=
/ HTTP/1.0\r\n\r\n -x HTTP -q -a 15 -I /sbin/ipvsadm -t 6 -w 1 -V 82.81.21=
5.141 -M g -U none --lvs</div>
<div>root =A0 =A0 =A05795 =A00.0 =A00.0 =A0 8488 =A0 372 ? =A0 =A0 =A0 =A0S=
s =A0 Jan17 =A0 0:00 pulse</div><div>root =A0 =A0 =A05798 =A00.0 =A00.0 =A0=
8476 =A0 656 ? =A0 =A0 =A0 =A0Ss =A0 Jan17 =A0 0:00 /usr/sbin/lvsd --nofor=
k -c /etc/sysconfig/ha/<a href=3D"http://lvs.cf" style=3D"color: rgb(42, 93=
, 176);" target=3D"_blank">lvs.cf</a></div>
<div>root =A0 =A0 =A05812 =A00.0 =A00.0 =A0 8456 =A0 692 ? =A0 =A0 =A0 =A0S=
s =A0 Jan17 =A0 0:00 /usr/sbin/nanny -c -h 82.81.215.138 -p 80 -r 80 -s GET=
/ HTTP/1.0\r\n\r\n -x HTTP -q -a 15 -I /sbin/ipvsadm -t 6 -w 1 -V 82.81.21=
5.141 -M g -U none --lvs</div>
<div>root =A0 =A0 =A05813 =A00.0 =A00.0 =A0 8456 =A0 692 ? =A0 =A0 =A0 =A0S=
s =A0 Jan17 =A0 0:00 /usr/sbin/nanny -c -h 82.81.215.139 -p 80 -r 80 -s GET=
/ HTTP/1.0\r\n\r\n -x HTTP -q -a 15 -I /sbin/ipvsadm -t 6 -w 1 -V 82.81.21=
5.141 -M g -U none --lvs</div>
<div><br></div><div><br></div><div><br></div><div>####### One of the server=
s ( the one that does not answer. though its identical to the other )</div>=
<div><br></div><div># arptables -L -n</div><div>Chain IN (policy ACCEPT)</d=
iv>
<div>target =A0 =A0 source-ip =A0 =A0 =A0 =A0 =A0 =A0destination-ip =A0 =A0=
=A0 source-hw =A0 =A0 =A0 =A0 =A0destination-hw =A0 =A0 hlen =A0 op =A0 =
=A0 =A0 =A0 hrd =A0 =A0 =A0 =A0pro =A0 =A0 =A0=A0</div><div>DROP =A0 =A0 =
=A0=A0<a href=3D"http://0.0.0.0/0" style=3D"color: rgb(42, 93, 176);" targe=
t=3D"_blank">0.0.0.0/0</a>=A0=A0 =A0 =A0 =A0 =A0 =A082.81.215.141 =A0 =A0 =
=A0 =A000/00 =A0 =A0 =A0 =A0 =A0 =A0 =A000/00 =A0 =A0 =A0 =A0 =A0 =A0 =A0an=
y =A0 =A00000/0000 =A00000/0000 =A00000/0000=A0</div>
<div><br></div><div>Chain OUT (policy ACCEPT)</div><div>target =A0 =A0 sour=
ce-ip =A0 =A0 =A0 =A0 =A0 =A0destination-ip =A0 =A0 =A0 source-hw =A0 =A0 =
=A0 =A0 =A0destination-hw =A0 =A0 hlen =A0 op =A0 =A0 =A0 =A0 hrd =A0 =A0 =
=A0 =A0pro =A0 =A0 =A0=A0</div><div>mangle =A0 =A0=A0<a href=3D"http://0.0.=
0.0/0" style=3D"color: rgb(42, 93, 176);" target=3D"_blank">0.0.0.0/0</a>=
=A0=A0 =A0 =A0 =A0 =A0 =A082.81.215.141 =A0 =A0 =A0 =A000/00 =A0 =A0 =A0 =
=A0 =A0 =A0 =A000/00 =A0 =A0 =A0 =A0 =A0 =A0 =A0any =A0 =A00000/0000 =A0000=
0/0000 =A00000/0000 --mangle-ip-s 82.81.215.139=A0</div>
<div><br></div><div>Chain FORWARD (policy ACCEPT)</div><div>target =A0 =A0 =
source-ip =A0 =A0 =A0 =A0 =A0 =A0destination-ip =A0 =A0 =A0 source-hw =A0 =
=A0 =A0 =A0 =A0destination-hw =A0 =A0 hlen =A0 op =A0 =A0 =A0 =A0 hrd =A0 =
=A0 =A0 =A0pro =A0 =A0 =A0=A0</div><div><br></div><div><br>
</div>
<div># ifconfig=A0</div><div>eth0 =A0 =A0 =A0Link encap:Ethernet =A0HWaddr =
00:11:25:41:69:A4 =A0</div><div>=A0=A0 =A0 =A0 =A0 =A0inet addr:82.81.215.1=
39 =A0Bcast:82.81.215.159 =A0Mask:255.255.255.224</div><div>=A0=A0 =A0 =A0 =
=A0 =A0inet6 addr: fe80::211:25ff:fe41:69a4/64 Scope:Link</div>
<div>=A0=A0 =A0 =A0 =A0 =A0UP BROADCAST RUNNING MULTICAST =A0MTU:1500 =A0Me=
tric:1</div><div>=A0=A0 =A0 =A0 =A0 =A0RX packets:602454 errors:0 dropped:0=
overruns:0 frame:0</div><div>=A0=A0 =A0 =A0 =A0 =A0TX packets:514536 error=
s:0 dropped:0 overruns:0 carrier:0</div>
<div>=A0=A0 =A0 =A0 =A0 =A0collisions:0 txqueuelen:1000=A0</div><div>=A0=A0=
=A0 =A0 =A0 =A0RX bytes:51144864 (48.7 MiB) =A0TX bytes:251901147 (240.2 M=
iB)</div><div>=A0=A0 =A0 =A0 =A0 =A0Interrupt:169 Memory:dcff0000-dd000000=
=A0</div><div><br></div><div>eth0:1 =A0 =A0Link encap:Ethernet =A0HWaddr 00=
:11:25:41:69:A4 =A0</div>
<div>=A0=A0 =A0 =A0 =A0 =A0inet addr:82.81.215.141 =A0Bcast:82.81.215.159 =
=A0Mask:255.255.255.224</div><div>=A0=A0 =A0 =A0 =A0 =A0UP BROADCAST RUNNIN=
G MULTICAST =A0MTU:1500 =A0Metric:1</div><div>=A0=A0 =A0 =A0 =A0 =A0Interru=
pt:169 Memory:dcff0000-dd000000=A0</div><div>
<br></div><div><br></div><div>Thanks for any idea that might shade some lig=
ht on this topic :)</div><div><br></div><div>Best,</div><div>Miki</div><div=
><br></div></span>--------------------------------------------------<br>
Michael Ben-Nes - Internet Consultant and Director.<br><a href=3D"http://ww=
w.epoch.co.il" target=3D"_blank">http://www.epoch.co.il</a> - weaving the N=
et.<br>Cellular: 054-4848113<br>-------------------------------------------=
-------<br>
</div>
<br>_______________________________________________<br>
Piranha-list mailing list<br>
<a href=3D"mailto:[email protected]">[email protected]</a><br>
<a href=3D"https://www.redhat.com/mailman/listinfo/piranha-list" target=3D"=
_blank">https://www.redhat.com/mailman/listinfo/piranha-list</a><br></block=
quote></div><br>
--00151758856cb004a3047d802dc6--
--===============5901058044263250296==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Piranha-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/piranha-list
--===============5901058044263250296==--