Re: [ANNOUNCE] keepalived 2.0.1
Dan Arena <[email protected]> Fri, 15 Jun 2018 11:34:58 -0400
| Newsgroups | gmane.linux.keepalived.devel |
|---|---|
| Message-ID | <CAKnMEmHEZOfOUG09Av8YBbjbFXe90_cXMv9FbB1tgJ64k7aPog@mail.gmail.com> |
--===============8166506195954029585== Content-Type: multipart/alternative; boundary="00000000000062f714056eaff52b" --00000000000062f714056eaff52b Content-Type: text/plain; charset="UTF-8" Damien and Quentin, I suspect the 1024 limit is because of the system default of 1024 open files per process, which can be easily increased. You can look at soft/hard limits in a shell with `ulimit -Sa` and `ulimit -Ha`, and then for example if the hard limit for open files is 4096 but soft is at 1024, increase the soft to 2048 with `ulimit -Sn 2048`. Using ulimit will only be change it for that shell and child processes though. You can greatly increase the default system-wide hard and soft limits by editing "/etc/security/limits.conf". On a modern linux OS it is usually safe to increase the open files limit to 50k or even 100k or more(`cat /proc/sys/fs/file-max` for the max system-wide open files), though I am not an experienced C coder so I don't know how select() will handle that many sockets, probably best to use epoll() if you want more than like 10k. Thanks, Dan On Fri, Jun 15, 2018 at 10:26 AM, Damien Clabaut < [email protected]> wrote: > Hello Quentin and thank you for your answer. > > You have been quick - I was going to send you an email to report that > keepalived kept crashing when going over 500 instances. > > When you say the limit is at 1024 file descriptors, is this for the entire > OS, or will launching Keepalived in separate netns+mntns allow for 1024 per > netns+mntns ? > > Regarding the move from select() to epoll(), may I ask you to elaborate ? > How much work would it be ? > > Regards, > > On Fri, 2018-06-15 at 15:20 +0100, Quentin Armitage wrote: > > Damien, > > > > Many thanks for the update. It's good to hear that keepalived is holding > up well, and very helpful to have some intense independent testing. > > > > At the moment keepalived uses select() for sending and receiving on > sockets, and keepalived opens two sockets per interface that it is using for > > vrrp (one send, one receive). When using VMACs that means two sockets > per vrrp instance, and since select() has a limit of 1024 file descriptors, > > that means the limit of the number of vrrp instances is about 500 > (allowing for the other sockets that keepalived uses). > > > > One way to reduce the number of sockets used is not to use VMACs, since > it will open only two sockets per interface, so if you have 250 vrrp > > instances all on the same interface it will only use 2 sockets, as > opposed to the 500 it would use if using VMACs. > > > > Please let us know if you need to be able to run more than 500 vrrp > instances using VMACs, in which case we can look at merging a scheduler > using > > epoll() instead of select(). > > > > Quentin > > > > On Fri, 2018-06-15 at 12:03 +0000, Damien Clabaut wrote: > > > Hello Alexandre, Quentin, > > > > > > We finally have had time to start our tests on Keepalived 2.0.2, with > our usual patches (mostly, enabling json at compilation). > > > > > > The test setup is as follows: > > > 2 servers (xeon e5 + Intel 82599ES) are connected together through a > switch (2*10G LACP from each server to the switch). > > > > > > We use the template we pushed to the Keepalived repo to spawn 398 VRRP > instances (vlans 2-200, one instance per vlan per IP version) with the > > > track_file option (each instance track a different file) on each > server. > > > > > > Then, we have an isolation tool, that will basically collect the state > of the system from JSON, then write "-100" in the files tracked by each > > > instance (we do 1 instance every second to have a smooth migration). > The script then waits for all instances to be in the BACKUP state, before > > > reversing everything. > > > > > > We left that running in a loop overnight (75 runs of the loop). Each > run takes around 800 seconds (~400 instances, 2 movements per instance, 1 > > > movement per second). > > > > > > Results: > > > - Used CPU/RAM do not increase over time > > > - We use about 20% of 1 core for softirq > > > - No instance ever went to FAULT state > > > - Even after so many runs of the loop, each run takes the same time > > > > > > Congratulations on making something this stable, we will be happy to > keep testing before putting Keepalived 2.0.2 in production. > > > > > > Next test, which will take place over the weekend, will be the same as > described above but with more instances (probably over 2000, so we can see > > > what > > > happens when we have more than 100% of 1 core used by softirq). > > > > > > Regards, > > > Damien > > > OVH vRouter team > > > > > > On Tue, 2018-06-05 at 13:34 +0000, Damien Clabaut wrote: > > > > Thank you for your answer, I will test this thoroughly. > > > > Damien > > > > On Tue, 2018-06-05 at 14:02 +0100, Quentin Armitage wrote: > > > > > Damien, > > > > > > > > > > The move from v1.4.5 to v2.0.0 was exactly merging the Beta branch > in Master, i.e. I executed > > > > > git merge beta > > > > > and pushed the merged branch. > > > > > > > > > > The main changes since 2018-05-11 which are included in v2.0.0 are: > > > > > Making notifications consistent across SNMP, notify scripts, > notify FIFOs and SMTP messages > > > > > Detecting virtual ip address deletions and transitioning to backup > (it will subsequently transition to master unless another instance becomes > > > > > master) > > > > > Handling interface deletion, transitioning to fault, and then > backup once interface restored. > > > > > Sundry fixes/improvements > > > > > > > > > > Additions for v2.0.1 > > > > > Handling VMAC deletion and restoring VMACs > > > > > Handing virtual route deletion either due to the route being > explicitly deleted or and interface going down > > > > > Support for new ip rule options in Linux 4.17 > > > > > Sundry fixes/improvements > > > > > > > > > > Coming soon > > > > > Handling virtual rule deletions > > > > > Handling static address, route and rule deletions > > > > > > > > > > Once the handling of all address, route and rule deletions is > completed, then keepalived should be able to fully support hot-swapping of > NICs. > > > > > > > > > > Quentin > > > > > > > > > > On Tue, 2018-06-05 at 11:46 +0000, Damien Clabaut wrote: > > > > > > Hello Alexandre, > > > > > > > > > > > > This is great news! > > > > > > > > > > > > Can you confirm that the move from 1.X to 2.0 is "just" merging > the Beta branch to Master ? > > > > > > > > > > > > We have tested Git Beta branch from 2018-05-11, should we expect > a lot of change ? > > > > > > > > > > > > Regards, > > > > > > On Mon, 2018-06-04 at 19:23 +0200, Alexandre Cassen wrote: > > > > > > > Hello, > > > > > > > > > > > > > > New release, with some updates and extensions. > > > > > > > > > > > > > > This code can be found on www.keepalived.org website or on > > > > > > > https://github.com/acassen/keepalived. > > > > > > > > > > > > > > ChangeLog for this release is : > > > > > > > > > > > > > > 2018-06-04 Quentin Armitage <[email protected]> > > > > > > > * keepalived-2.0.1 > > > > > > > * Remove '\n' characters from log_message() text. > > > > > > > * Allow IPv6 ip rules to be specified using fwmarks. > > > > > > > * Fix configure generation of keepalived.spec file. > > > > > > > * Stop rebuilding scheduler.o every make. > > > > > > > * Remove ' characters from configure args in keepalived -v > > > > > > > output. > > > > > > > * Remove duplicate reporting of network namespace in config > > > > > > > dumps > > > > > > > * Add ${_INSTANCE} config parameter. > > > > > > > * Remove debugging log message. > > > > > > > * Recalculate max_fd used for select if it should reduce. > > > > > > > * Add tracking of virtual routes. > > > > > > > If a virtual route is deleted, by default to vrrp > instance > > > > > > > will now transition to backup mode, and if it > transitions to > > > > > > > master again the route will be re-added. If an interface > on > > > > > > > which a route is configured is down, then the instance > will go > > > > > > > to fault state, since the route cannot be added. > > > > > > > This commit also adds a no-track option for routes, which > > > > > > > means that deletion of the route will not cause the vrrp > > > > > > > instance to transition to backup. > > > > > > > * Handle interface down at startup with tracked route > configured > > > > > > > on it. If a virtual route which is tracked is configured > on an > > > > > > > interface that is down at startup, then the vrrp instance > > > > > > > needs to start in fault state. > > > > > > > * Rename netlink_reflect_filter() to netlink_link_filter() > > > > > > > The function only handles RTM_NEWLINK/RTM_DELLINK > messages and > > > > > > > there are other functions to handle other message types. > > > > > > > * Fix compilation warning. > > > > > > > * Make recreating deleted VMACs work. > > > > > > > * Fix Travis-CI compilation failure and warning. > > > > > > > * Stop duplicate definition and duplicate include in > > > > > > > vrrp_iproute.c. > > > > > > > * Add new ip rule options for Linux 4.17 > > > > > > > FRA_PROTOCOL, FRA_IP_PROTO, FRA_SPORT_RANGE and > > > > > > > FRA_DPORT_RANGE have been added in Linux 4.17. > > > > > > > > > > > > > > > > > > > > > regs, > > > > > > > Alexandre > > > > > > > > > > > > > > ------------------------------------------------------------ > ------------------ > > > > > > > Check out the vibrant tech community on one of the world's most > > > > > > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > > > > > _______________________________________________ > > > > > > > Keepalived-devel mailing list > > > > > > > [email protected] > > > > > > > https://lists.sourceforge.net/lists/listinfo/keepalived-devel > -- > Damien Clabaut > R&D vRouter > OVH > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > Keepalived-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/keepalived-devel > --00000000000062f714056eaff52b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Damien and Quentin,</div><div><br></div><div>I suspec= t the 1024 limit is because of the system default of 1024 open files per pr= ocess, which can be easily increased. You can look at soft/hard limits in a= shell with `ulimit -Sa` and `ulimit -Ha`, and then for example if the hard= limit for open files is 4096 but soft is at 1024,=C2=A0 increase the soft = to 2048 with `ulimit -Sn 2048`. Using ulimit will only be change it for tha= t shell and child processes though. You can greatly increase the default sy= stem-wide hard and soft limits by editing "/etc/security/limits.conf&q= uot;. On a modern linux OS it is usually safe to increase the open files li= mit to 50k or even 100k or more(`cat /proc/sys/fs/file-max` for the max sys= tem-wide open files), though I am not an experienced C coder so I don't= know how select() will handle that many sockets, probably best to use epol= l() if you want more than like 10k.<br></div><div><br></div><div>Thanks,</d= iv><div>Dan<br></div><div><br></div><div class=3D"gmail_extra"><br><div cla= ss=3D"gmail_quote">On Fri, Jun 15, 2018 at 10:26 AM, Damien Clabaut <span d= ir=3D"ltr"><<a href=3D"mailto:[email protected]" target=3D"_bl= ank">[email protected]</a>></span> wrote:<br><blockquote class= =3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg= b(204,204,204);padding-left:1ex">Hello Quentin and thank=C2=A0 you for your= answer.<br> <br> You have been quick - I was going to send you an email to report that keepa= lived kept crashing when going over 500 instances.<br> <br> When you say the limit is at 1024 file descriptors, is this for the=20 entire OS, or will launching Keepalived in separate netns+mntns allow=20 for 1024 per<br> netns+mntns ?<br> <br> Regarding the move from select() to epoll(), may I ask you to elaborate ? H= ow much work would it be ?<br> <br> Regards,<br> <div class=3D"gmail-HOEnZb"><div class=3D"gmail-h5"><br> On Fri, 2018-06-15 at 15:20 +0100, Quentin Armitage wrote:<br> > Damien,<br> > <br> > Many thanks for the update. It's good to hear that keepalived is= =20 holding up well, and very helpful to have some intense independent=20 testing.<br> > <br> > At the moment keepalived uses select() for sending and receiving on sockets, and keepalived opens two sockets per interface that it is=20 using for<br> > vrrp (one send, one receive). When using VMACs that means two=20 sockets per vrrp instance, and since select() has a limit of 1024 file=20 descriptors,<br> > that means the limit of the number of vrrp instances is about 500 (all= owing for the other sockets that keepalived uses).<br> > <br> > One way to reduce the number of sockets used is not to use VMACs,=20 since it will open only two sockets per interface, so if you have 250=20 vrrp<br> > instances all on the same interface it will only use 2 sockets, as opp= osed to the 500 it would use if using VMACs.<br> > <br> > Please let us know if you need to be able to run more than 500 vrrp instances using VMACs, in which case we can look at merging a scheduler using<br> > epoll() instead of select().<br> > <br> > Quentin<br> > <br> > On Fri, 2018-06-15 at 12:03 +0000, Damien Clabaut wrote:<br> > > Hello Alexandre, Quentin,<br> > > <br> > > We finally have had time to start our tests on Keepalived=20 2.0.2, with our usual patches (mostly, enabling json at compilation).<br> > > <br> > > The test setup is as follows:<br> > > 2 servers (xeon e5 + Intel 82599ES) are connected together=20 through a switch (2*10G LACP from each server to the switch).<br> > > <br> > > We use the template we pushed to the Keepalived repo to spawn=20 398 VRRP instances (vlans 2-200, one instance per vlan per IP version)=20 with the<br> > > track_file option (each instance track a different file) on each = server.<br> > > <br> > > Then, we have an isolation tool, that will basically collect=20 the state of the system from JSON, then write "-100" in the files= =20 tracked by each<br> > > instance (we do 1 instance every second to have a smooth=20 migration). The script then waits for all instances to be in the BACKUP=20 state, before<br> > > reversing everything.<br> > > <br> > > We left that running in a loop overnight (75 runs of the=20 loop). Each run takes around 800 seconds (~400 instances, 2 movements=20 per instance, 1<br> > > movement per second).<br> > > <br> > > Results:<br> > > - Used CPU/RAM do not increase over time<br> > > - We use about 20% of 1 core for softirq<br> > > - No instance ever went to FAULT state<br> > > - Even after so many runs of the loop, each run takes the same ti= me<br> > > <br> > > Congratulations on making something this stable, we will be=20 happy to keep testing before putting Keepalived 2.0.2 in production.<br> > > <br> > > Next test, which will take place over the weekend, will be the same as described above but with more instances (probably over 2000, so we can see<br> > > what<br> > > happens when we have more than 100% of 1 core used by softirq).<b= r> > > <br> > > Regards,<br> > > Damien<br> > > OVH vRouter team<br> > > <br> > > On Tue, 2018-06-05 at 13:34 +0000, Damien Clabaut wrote:<br> > > > Thank you for your answer, I will test this thoroughly.<br> > > > Damien<br> > > > On Tue, 2018-06-05 at 14:02 +0100, Quentin Armitage wrote:<b= r> > > > > Damien,<br> > > > > <br> > > > > The move from v1.4.5 to v2.0.0 was exactly merging the = Beta branch in Master, i.e. I executed<br> > > > > git merge beta<br> > > > > and pushed the merged branch.<br> > > > > <br> > > > > The main changes since 2018-05-11 which are included in= v2.0.0 are:<br> > > > > Making notifications consistent across SNMP, notify scr= ipts, notify FIFOs and SMTP messages<br> > > > > Detecting virtual ip address deletions and=20 transitioning to backup (it will subsequently transition to master=20 unless another instance becomes<br> > > > > master)<br> > > > > Handling interface deletion, transitioning to fault, an= d then backup once interface restored.<br> > > > > Sundry fixes/improvements<br> > > > > <br> > > > > Additions for v2.0.1<br> > > > > Handling VMAC deletion and restoring VMACs<br> > > > > Handing virtual route deletion either due to the route = being explicitly deleted or and interface going down<br> > > > > Support for new ip rule options in Linux 4.17<br> > > > > Sundry fixes/improvements<br> > > > > <br> > > > > Coming soon<br> > > > > Handling virtual rule deletions<br> > > > > Handling static address, route and rule deletions<br> > > > > <br> > > > > Once the handling of all address, route and rule=20 deletions is completed, then keepalived should be able to fully support=20 hot-swapping of NICs.<br> > > > > <br> > > > > Quentin<br> > > > > <br> > > > > On Tue, 2018-06-05 at 11:46 +0000, Damien Clabaut wrote= :<br> > > > > > Hello Alexandre,<br> > > > > > <br> > > > > > This is great news! <br> > > > > > <br> > > > > > Can you confirm that the move from 1.X to 2.0 is &= quot;just" merging the Beta branch to Master ?<br> > > > > > <br> > > > > > We have tested Git Beta branch from 2018-05-11, sh= ould we expect a lot of change ?<br> > > > > > <br> > > > > > Regards,<br> > > > > > On Mon, 2018-06-04 at 19:23 +0200, Alexandre Casse= n wrote:<br> > > > > > > Hello,<br> > > > > > > <br> > > > > > > New release, with some updates and extensions= .<br> > > > > > > <br> > > > > > > This code can be found on <a href=3D"http://w= ww.keepalived.org" rel=3D"noreferrer" target=3D"_blank">www.keepalived.org<= /a> website or on <br> > > > > > > <a href=3D"https://github.com/acassen/keepali= ved" rel=3D"noreferrer" target=3D"_blank">https://github.com/acassen/<wbr>k= eepalived</a>.<br> > > > > > > <br> > > > > > > ChangeLog for this release is :<br> > > > > > > <br> > > > > > > 2018-06-04 Quentin Armitage <<a href=3D"ma= ilto:[email protected]">[email protected]</a>><br> > > > > > >=C2=A0 =C2=A0 =C2=A0* keepalived-2.0.1<br> > > > > > >=C2=A0 =C2=A0 =C2=A0* Remove '\n' char= acters from log_message() text.<br> > > > > > >=C2=A0 =C2=A0 =C2=A0* Allow IPv6 ip rules to b= e specified using fwmarks.<br> > > > > > >=C2=A0 =C2=A0 =C2=A0* Fix configure generation= of keepalived.spec file.<br> > > > > > >=C2=A0 =C2=A0 =C2=A0* Stop rebuilding schedule= r.o every make.<br> > > > > > >=C2=A0 =C2=A0 =C2=A0* Remove ' characters = from configure args in keepalived -v<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0output.<br> > > > > > >=C2=A0 =C2=A0 =C2=A0* Remove duplicate reporti= ng of network namespace in config<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0dumps<br> > > > > > >=C2=A0 =C2=A0 =C2=A0* Add ${_INSTANCE} config = parameter.<br> > > > > > >=C2=A0 =C2=A0 =C2=A0* Remove debugging log mes= sage.<br> > > > > > >=C2=A0 =C2=A0 =C2=A0* Recalculate max_fd used = for select if it should reduce.<br> > > > > > >=C2=A0 =C2=A0 =C2=A0* Add tracking of virtual = routes.<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0If a virtual route = is deleted, by default to vrrp instance<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0will now transition= to backup mode, and if it transitions to<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0master again the ro= ute will be re-added. If an interface on<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0which a route is co= nfigured is down, then the instance will go<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0to fault state, sin= ce the route cannot be added.<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0This commit also ad= ds a no-track option for routes, which<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0means that deletion= of the route will not cause the vrrp<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0instance to transit= ion to backup.<br> > > > > > >=C2=A0 =C2=A0 =C2=A0* Handle interface down at= startup with tracked route configured<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0on it. If a virtual= route which is tracked is configured on an<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0interface that is d= own at startup, then the vrrp instance<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0needs to start in f= ault state.<br> > > > > > >=C2=A0 =C2=A0 =C2=A0* Rename netlink_reflect_f= ilter() to netlink_link_filter()<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0The function only h= andles RTM_NEWLINK/RTM_DELLINK messages and<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0there are other fun= ctions to handle other message types.<br> > > > > > >=C2=A0 =C2=A0 =C2=A0* Fix compilation warning.= <br> > > > > > >=C2=A0 =C2=A0 =C2=A0* Make recreating deleted = VMACs work.<br> > > > > > >=C2=A0 =C2=A0 =C2=A0* Fix Travis-CI compilatio= n failure and warning.<br> > > > > > >=C2=A0 =C2=A0 =C2=A0* Stop duplicate definitio= n and duplicate include in<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0vrrp_iproute.c.<br> > > > > > >=C2=A0 =C2=A0 =C2=A0* Add new ip rule options = for Linux 4.17<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0FRA_PROTOCOL, FRA_I= P_PROTO, FRA_SPORT_RANGE and<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0FRA_DPORT_RANGE hav= e been added in Linux 4.17.<br> > > > > > > <br> > > > > > > <br> > > > > > > regs,<br> > > > > > > Alexandre<br> > > > > > > <br> > > > > > > ------------------------------<wbr>----------= --------------------<wbr>------------------<br> > > > > > > Check out the vibrant tech community on one o= f the world's most<br> > > > > > > engaging tech sites, Slashdot.org! <a href=3D= "http://sdm.link/slashdot" rel=3D"noreferrer" target=3D"_blank">http://sdm.= link/slashdot</a><br> > > > > > > ______________________________<wbr>__________= _______<br> > > > > > > Keepalived-devel mailing list<br> > > > > > > <a href=3D"mailto:[email protected]= ceforge.net">Keepalived-devel@lists.<wbr>sourceforge.net</a><br> > > > > > > <a href=3D"https://lists.sourceforge.net/list= s/listinfo/keepalived-devel" rel=3D"noreferrer" target=3D"_blank">https://l= ists.sourceforge.net/<wbr>lists/listinfo/keepalived-<wbr>devel</a><br> </div></div><span class=3D"gmail-im gmail-HOEnZb">-- <br> Damien Clabaut<br> R&D vRouter<br> OVH<br> </span><div class=3D"gmail-HOEnZb"><div class=3D"gmail-h5">----------------= --------------<wbr>------------------------------<wbr>------------------<br= > Check out the vibrant tech community on one of the world's most<br> engaging tech sites, Slashdot.org! <a href=3D"http://sdm.link/slashdot" rel= =3D"noreferrer" target=3D"_blank">http://sdm.link/slashdot</a><br> ______________________________<wbr>_________________<br> Keepalived-devel mailing list<br> <a href=3D"mailto:[email protected]">Keepalived-devel@= lists.<wbr>sourceforge.net</a><br> <a href=3D"https://lists.sourceforge.net/lists/listinfo/keepalived-devel" r= el=3D"noreferrer" target=3D"_blank">https://lists.sourceforge.net/<wbr>list= s/listinfo/keepalived-<wbr>devel</a><br> </div></div></blockquote></div><br></div><br></div> --00000000000062f714056eaff52b-- --===============8166506195954029585== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot --===============8166506195954029585== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Keepalived-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/keepalived-devel --===============8166506195954029585==--