Re: [ANNOUNCE] keepalived 2.0.1
Quentin Armitage <[email protected]> Fri, 15 Jun 2018 15:20:33 +0100
| Newsgroups | gmane.linux.keepalived.devel |
|---|---|
| Organization | The Armitage family |
| Message-ID | <[email protected]> |
--===============7540491253558976128== Content-Type: multipart/alternative; boundary="=-zDsdKbLO0mXOPuDAsiFJ" --=-zDsdKbLO0mXOPuDAsiFJ Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Damien, Many thanks for the update. It's good to hear that keepalived is holding up well, and very helpful to have some intense independent testing. At the moment keepalived uses select() for sending and receiving on sockets, and keepalived opens two sockets per interface that it is using for vrrp (one send, one receive). When using VMACs that means two sockets per vrrp instance, and since select() has a limit of 1024 file descriptors, that means the limit of the number of vrrp instances is about 500 (allowing for the other sockets that keepalived uses). One way to reduce the number of sockets used is not to use VMACs, since it will open only two sockets per interface, so if you have 250 vrrp instances all on the same interface it will only use 2 sockets, as opposed to the 500 it would use if using VMACs. Please let us know if you need to be able to run more than 500 vrrp instances using VMACs, in which case we can look at merging a scheduler using epoll() instead of select(). Quentin On Fri, 2018-06-15 at 12:03 +0000, Damien Clabaut wrote: > Hello Alexandre, Quentin, > > We finally have had time to start our tests on Keepalived 2.0.2, with > our usual patches (mostly, enabling json at compilation). > > The test setup is as follows: > 2 servers (xeon e5 + Intel 82599ES) are connected together through a > switch (2*10G LACP from each server to the switch). > > We use the template we pushed to the Keepalived repo to spawn 398 > VRRP instances (vlans 2-200, one instance per vlan per IP version) > with the > track_file option (each instance track a different file) on each > server. > > Then, we have an isolation tool, that will basically collect the > state of the system from JSON, then write "-100" in the files tracked > by each > instance (we do 1 instance every second to have a smooth migration). > The script then waits for all instances to be in the BACKUP state, > before > reversing everything. > > We left that running in a loop overnight (75 runs of the loop). Each > run takes around 800 seconds (~400 instances, 2 movements per > instance, 1 > movement per second). > > Results: > - Used CPU/RAM do not increase over time > - We use about 20% of 1 core for softirq > - No instance ever went to FAULT state > - Even after so many runs of the loop, each run takes the same time > > Congratulations on making something this stable, we will be happy to > keep testing before putting Keepalived 2.0.2 in production. > > Next test, which will take place over the weekend, will be the same > as described above but with more instances (probably over 2000, so we > can see what > happens when we have more than 100% of 1 core used by softirq). > > Regards, > Damien > OVH vRouter team > > On Tue, 2018-06-05 at 13:34 +0000, Damien Clabaut wrote: > > Thank you for your answer, I will test this thoroughly. > > Damien > > On Tue, 2018-06-05 at 14:02 +0100, Quentin Armitage wrote: > > > Damien, > > > > > > The move from v1.4.5 to v2.0.0 was exactly merging the Beta > > > branch in Master, i.e. I executed > > > git merge beta > > > and pushed the merged branch. > > > > > > The main changes since 2018-05-11 which are included in v2.0.0 > > > are: > > > Making notifications consistent across SNMP, notify scripts, > > > notify FIFOs and SMTP messages > > > Detecting virtual ip address deletions and transitioning to > > > backup (it will subsequently transition to master unless another > > > instance becomes > > > master) > > > Handling interface deletion, transitioning to fault, and then > > > backup once interface restored. > > > Sundry fixes/improvements > > > > > > Additions for v2.0.1 > > > Handling VMAC deletion and restoring VMACs > > > Handing virtual route deletion either due to the route being > > > explicitly deleted or and interface going down > > > Support for new ip rule options in Linux 4.17 > > > Sundry fixes/improvements > > > > > > Coming soon > > > Handling virtual rule deletions > > > Handling static address, route and rule deletions > > > > > > Once the handling of all address, route and rule deletions is > > > completed, then keepalived should be able to fully support hot- > > > swapping of NICs. > > > > > > Quentin > > > > > > On Tue, 2018-06-05 at 11:46 +0000, Damien Clabaut wrote: > > > > Hello Alexandre, > > > > > > > > This is great news! > > > > > > > > Can you confirm that the move from 1.X to 2.0 is "just" merging > > > > the Beta branch to Master ? > > > > > > > > We have tested Git Beta branch from 2018-05-11, should we > > > > expect a lot of change ? > > > > > > > > Regards, > > > > On Mon, 2018-06-04 at 19:23 +0200, Alexandre Cassen wrote: > > > > > Hello, > > > > > > > > > > New release, with some updates and extensions. > > > > > > > > > > This code can be found on www.keepalived.org website or on > > > > > https://github.com/acassen/keepalived. > > > > > > > > > > ChangeLog for this release is : > > > > > > > > > > 2018-06-04 Quentin Armitage <[email protected]> > > > > > * keepalived-2.0.1 > > > > > * Remove '\n' characters from log_message() text. > > > > > * Allow IPv6 ip rules to be specified using fwmarks. > > > > > * Fix configure generation of keepalived.spec file. > > > > > * Stop rebuilding scheduler.o every make. > > > > > * Remove ' characters from configure args in keepalived > > > > > -v > > > > > output. > > > > > * Remove duplicate reporting of network namespace in > > > > > config > > > > > dumps > > > > > * Add ${_INSTANCE} config parameter. > > > > > * Remove debugging log message. > > > > > * Recalculate max_fd used for select if it should > > > > > reduce. > > > > > * Add tracking of virtual routes. > > > > > If a virtual route is deleted, by default to vrrp > > > > > instance > > > > > will now transition to backup mode, and if it > > > > > transitions to > > > > > master again the route will be re-added. If an > > > > > interface on > > > > > which a route is configured is down, then the > > > > > instance will go > > > > > to fault state, since the route cannot be added. > > > > > This commit also adds a no-track option for routes, > > > > > which > > > > > means that deletion of the route will not cause the > > > > > vrrp > > > > > instance to transition to backup. > > > > > * Handle interface down at startup with tracked route > > > > > configured > > > > > on it. If a virtual route which is tracked is > > > > > configured on an > > > > > interface that is down at startup, then the vrrp > > > > > instance > > > > > needs to start in fault state. > > > > > * Rename netlink_reflect_filter() to > > > > > netlink_link_filter() > > > > > The function only handles RTM_NEWLINK/RTM_DELLINK > > > > > messages and > > > > > there are other functions to handle other message > > > > > types. > > > > > * Fix compilation warning. > > > > > * Make recreating deleted VMACs work. > > > > > * Fix Travis-CI compilation failure and warning. > > > > > * Stop duplicate definition and duplicate include in > > > > > vrrp_iproute.c. > > > > > * Add new ip rule options for Linux 4.17 > > > > > FRA_PROTOCOL, FRA_IP_PROTO, FRA_SPORT_RANGE and > > > > > FRA_DPORT_RANGE have been added in Linux 4.17. > > > > > > > > > > > > > > > regs, > > > > > Alexandre > > > > > > > > > > ----------------------------------------------------------- > > > > > ------------------- > > > > > Check out the vibrant tech community on one of the world's > > > > > most > > > > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > > > _______________________________________________ > > > > > Keepalived-devel mailing list > > > > > [email protected] > > > > > https://lists.sourceforge.net/lists/listinfo/keepalived-devel --=-zDsdKbLO0mXOPuDAsiFJ Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable <html><head></head><body bgcolor=3D"#ffffff" text=3D"#2e3436" link=3D"#2a76= c6" vlink=3D"#2e3436"><div>Damien,</div><div><br></div><div>Many thanks for= the update. It's good to hear that keepalived is holding up well, and very= helpful to have some intense independent testing.</div><div><br></div><div= >At the moment keepalived uses select() for sending and receiving on socket= s, and keepalived opens two sockets per interface that it is using for vrrp= (one send, one receive). When using VMACs that means two sockets per vrrp = instance, and since select() has a limit of 1024 file descriptors, that mea= ns the limit of the number of vrrp instances is about 500 (allowing for the= other sockets that keepalived uses).</div><div><br></div><div>One way to r= educe the number of sockets used is not to use VMACs, since it will open on= ly two sockets per interface, so if you have 250 vrrp instances all on the = same interface it will only use 2 sockets, as opposed to the 500 it would u= se if using VMACs.</div><div><br></div><div>Please let us know if you need = to be able to run more than 500 vrrp instances using VMACs, in which case w= e can look at merging a scheduler using epoll() instead of select().</div><= div><br></div><div>Quentin</div><div><br></div><div>On Fri, 2018-06-15 at 1= 2:03 +0000, Damien Clabaut wrote:</div><blockquote type=3D"cite" style=3D"m= argin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><pre>Hell= o Alexandre, Quentin, We finally have had time to start our tests on Keepalived 2.0.2, with our u= sual patches (mostly, enabling json at compilation). The test setup is as follows: 2 servers (xeon e5 + Intel 82599ES) are connected together through a switch= (2*10G LACP from each server to the switch). We use the template we pushed to the Keepalived repo to spawn 398 VRRP inst= ances (vlans 2-200, one instance per vlan per IP version) with the track_file option (each instance track a different file) on each server. Then, we have an isolation tool, that will basically collect the state of t= he system from JSON, then write "-100" in the files tracked by each instance (we do 1 instance every second to have a smooth migration). The sc= ript then waits for all instances to be in the BACKUP state, before reversing everything. We left that running in a loop overnight (75 runs of the loop). Each run ta= kes around 800 seconds (~400 instances, 2 movements per instance, 1 movement per second). Results: - Used CPU/RAM do not increase over time - We use about 20% of 1 core for softirq - No instance ever went to FAULT state - Even after so many runs of the loop, each run takes the same time Congratulations on making something this stable, we will be happy to keep t= esting before putting Keepalived 2.0.2 in production. Next test, which will take place over the weekend, will be the same as desc= ribed above but with more instances (probably over 2000, so we can see what happens when we have more than 100% of 1 core used by softirq). Regards, Damien OVH vRouter team On Tue, 2018-06-05 at 13:34 +0000, Damien Clabaut wrote: <blockquote type=3D"cite" style=3D"margin:0 0 0 .8ex; border-left:2px #729f= cf solid;padding-left:1ex"> Thank you for your answer, I will test this thoroughly. Damien On Tue, 2018-06-05 at 14:02 +0100, Quentin Armitage wrote: <blockquote type=3D"cite" style=3D"margin:0 0 0 .8ex; border-left:2px #729f= cf solid;padding-left:1ex"> Damien, The move from v1.4.5 to v2.0.0 was exactly merging the Beta branch in Maste= r, i.e. I executed git merge beta and pushed the merged branch. The main changes since 2018-05-11 which are included in v2.0.0 are: Making notifications consistent across SNMP, notify scripts, notify FIFOs a= nd SMTP messages Detecting virtual ip address deletions and transitioning to backup (it will= subsequently transition to master unless another instance becomes master) Handling interface deletion, transitioning to fault, and then backup once i= nterface restored. Sundry fixes/improvements Additions for v2.0.1 Handling VMAC deletion and restoring VMACs Handing virtual route deletion either due to the route being explicitly del= eted or and interface going down Support for new ip rule options in Linux 4.17 Sundry fixes/improvements Coming soon Handling virtual rule deletions Handling static address, route and rule deletions Once the handling of all address, route and rule deletions is completed, th= en keepalived should be able to fully support hot-swapping of NICs. Quentin On Tue, 2018-06-05 at 11:46 +0000, Damien Clabaut wrote: <blockquote type=3D"cite" style=3D"margin:0 0 0 .8ex; border-left:2px #729f= cf solid;padding-left:1ex"> Hello Alexandre, This is great news!=20 Can you confirm that the move from 1.X to 2.0 is "just" merging the Beta br= anch to Master ? We have tested Git Beta branch from 2018-05-11, should we expect a lot of c= hange ? Regards, On Mon, 2018-06-04 at 19:23 +0200, Alexandre Cassen wrote: <blockquote type=3D"cite" style=3D"margin:0 0 0 .8ex; border-left:2px #729f= cf solid;padding-left:1ex"> Hello, New release, with some updates and extensions. This code can be found on <a href=3D"http://www.keepalived.org">www.keepali= ved.org</a> website or on=20 <a href=3D"https://github.com/acassen/keepalived">https://github.com/acasse= n/keepalived</a>. ChangeLog for this release is : 2018-06-04 Quentin Armitage <<a href=3D"mailto:[email protected]">= [email protected]</a>> * keepalived-2.0.1 * Remove '\n' characters from log_message() text. * Allow IPv6 ip rules to be specified using fwmarks. * Fix configure generation of keepalived.spec file. * Stop rebuilding scheduler.o every make. * Remove ' characters from configure args in keepalived -v output. * Remove duplicate reporting of network namespace in config dumps * Add ${_INSTANCE} config parameter. * Remove debugging log message. * Recalculate max_fd used for select if it should reduce. * Add tracking of virtual routes. If a virtual route is deleted, by default to vrrp instance will now transition to backup mode, and if it transitions to master again the route will be re-added. If an interface on which a route is configured is down, then the instance will go to fault state, since the route cannot be added. This commit also adds a no-track option for routes, which means that deletion of the route will not cause the vrrp instance to transition to backup. * Handle interface down at startup with tracked route configured on it. If a virtual route which is tracked is configured on an interface that is down at startup, then the vrrp instance needs to start in fault state. * Rename netlink_reflect_filter() to netlink_link_filter() The function only handles RTM_NEWLINK/RTM_DELLINK messages and there are other functions to handle other message types. * Fix compilation warning. * Make recreating deleted VMACs work. * Fix Travis-CI compilation failure and warning. * Stop duplicate definition and duplicate include in vrrp_iproute.c. * Add new ip rule options for Linux 4.17 FRA_PROTOCOL, FRA_IP_PROTO, FRA_SPORT_RANGE and FRA_DPORT_RANGE have been added in Linux 4.17. regs, Alexandre ---------------------------------------------------------------------------= --- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! <a href=3D"http://sdm.link/slashdot">htt= p://sdm.link/slashdot</a> _______________________________________________ Keepalived-devel mailing list <a href=3D"mailto:[email protected]">Keepalived-devel@= lists.sourceforge.net</a> <a href=3D"https://lists.sourceforge.net/lists/listinfo/keepalived-devel">h= ttps://lists.sourceforge.net/lists/listinfo/keepalived-devel</a> </blockquote></blockquote></blockquote></blockquote></pre></blockquote></bo= dy></html> --=-zDsdKbLO0mXOPuDAsiFJ-- --===============7540491253558976128== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot --===============7540491253558976128== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Keepalived-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/keepalived-devel --===============7540491253558976128==--