Re: [ANNOUNCE] keepalived 2.0.1

Hello Quentin and thank  you for your answer.

You have been quick - I was going to send you an email to report that keepalived kept crashing when going over 500 instances.

When you say the limit is at 1024 file descriptors, is this for the entire OS, or will launching Keepalived in separate netns+mntns allow for 1024 per
netns+mntns ?

Regarding the move from select() to epoll(), may I ask you to elaborate ? How much work would it be ?

Regards,

On Fri, 2018-06-15 at 15:20 +0100, Quentin Armitage wrote:
> Damien,
> 
> Many thanks for the update. It's good to hear that keepalived is holding up well, and very helpful to have some intense independent testing.
> 
> At the moment keepalived uses select() for sending and receiving on sockets, and keepalived opens two sockets per interface that it is using for
> vrrp (one send, one receive). When using VMACs that means two sockets per vrrp instance, and since select() has a limit of 1024 file descriptors,
> that means the limit of the number of vrrp instances is about 500 (allowing for the other sockets that keepalived uses).
> 
> One way to reduce the number of sockets used is not to use VMACs, since it will open only two sockets per interface, so if you have 250 vrrp
> instances all on the same interface it will only use 2 sockets, as opposed to the 500 it would use if using VMACs.
> 
> Please let us know if you need to be able to run more than 500 vrrp instances using VMACs, in which case we can look at merging a scheduler using
> epoll() instead of select().
> 
> Quentin
> 
> On Fri, 2018-06-15 at 12:03 +0000, Damien Clabaut wrote:
> > Hello Alexandre, Quentin,
> > 
> > We finally have had time to start our tests on Keepalived 2.0.2, with our usual patches (mostly, enabling json at compilation).
> > 
> > The test setup is as follows:
> > 2 servers (xeon e5 + Intel 82599ES) are connected together through a switch (2*10G LACP from each server to the switch).
> > 
> > We use the template we pushed to the Keepalived repo to spawn 398 VRRP instances (vlans 2-200, one instance per vlan per IP version) with the
> > track_file option (each instance track a different file) on each server.
> > 
> > Then, we have an isolation tool, that will basically collect the state of the system from JSON, then write "-100" in the files tracked by each
> > instance (we do 1 instance every second to have a smooth migration). The script then waits for all instances to be in the BACKUP state, before
> > reversing everything.
> > 
> > We left that running in a loop overnight (75 runs of the loop). Each run takes around 800 seconds (~400 instances, 2 movements per instance, 1
> > movement per second).
> > 
> > Results:
> > - Used CPU/RAM do not increase over time
> > - We use about 20% of 1 core for softirq
> > - No instance ever went to FAULT state
> > - Even after so many runs of the loop, each run takes the same time
> > 
> > Congratulations on making something this stable, we will be happy to keep testing before putting Keepalived 2.0.2 in production.
> > 
> > Next test, which will take place over the weekend, will be the same as described above but with more instances (probably over 2000, so we can see
> > what
> > happens when we have more than 100% of 1 core used by softirq).
> > 
> > Regards,
> > Damien
> > OVH vRouter team
> > 
> > On Tue, 2018-06-05 at 13:34 +0000, Damien Clabaut wrote:
> > > Thank you for your answer, I will test this thoroughly.
> > > Damien
> > > On Tue, 2018-06-05 at 14:02 +0100, Quentin Armitage wrote:
> > > > Damien,
> > > > 
> > > > The move from v1.4.5 to v2.0.0 was exactly merging the Beta branch in Master, i.e. I executed
> > > > git merge beta
> > > > and pushed the merged branch.
> > > > 
> > > > The main changes since 2018-05-11 which are included in v2.0.0 are:
> > > > Making notifications consistent across SNMP, notify scripts, notify FIFOs and SMTP messages
> > > > Detecting virtual ip address deletions and transitioning to backup (it will subsequently transition to master unless another instance becomes
> > > > master)
> > > > Handling interface deletion, transitioning to fault, and then backup once interface restored.
> > > > Sundry fixes/improvements
> > > > 
> > > > Additions for v2.0.1
> > > > Handling VMAC deletion and restoring VMACs
> > > > Handing virtual route deletion either due to the route being explicitly deleted or and interface going down
> > > > Support for new ip rule options in Linux 4.17
> > > > Sundry fixes/improvements
> > > > 
> > > > Coming soon
> > > > Handling virtual rule deletions
> > > > Handling static address, route and rule deletions
> > > > 
> > > > Once the handling of all address, route and rule deletions is completed, then keepalived should be able to fully support hot-swapping of NICs.
> > > > 
> > > > Quentin
> > > > 
> > > > On Tue, 2018-06-05 at 11:46 +0000, Damien Clabaut wrote:
> > > > > Hello Alexandre,
> > > > > 
> > > > > This is great news! 
> > > > > 
> > > > > Can you confirm that the move from 1.X to 2.0 is "just" merging the Beta branch to Master ?
> > > > > 
> > > > > We have tested Git Beta branch from 2018-05-11, should we expect a lot of change ?
> > > > > 
> > > > > Regards,
> > > > > On Mon, 2018-06-04 at 19:23 +0200, Alexandre Cassen wrote:
> > > > > > Hello,
> > > > > > 
> > > > > > New release, with some updates and extensions.
> > > > > > 
> > > > > > This code can be found on www.keepalived.org website or on 
> > > > > > https://github.com/acassen/keepalived.
> > > > > > 
> > > > > > ChangeLog for this release is :
> > > > > > 
> > > > > > 2018-06-04 Quentin Armitage <[email protected]>
> > > > > > 	* keepalived-2.0.1
> > > > > > 	* Remove '\n' characters from log_message() text.
> > > > > > 	* Allow IPv6 ip rules to be specified using fwmarks.
> > > > > > 	* Fix configure generation of keepalived.spec file.
> > > > > > 	* Stop rebuilding scheduler.o every make.
> > > > > > 	* Remove ' characters from configure args in keepalived -v
> > > > > > 	  output.
> > > > > > 	* Remove duplicate reporting of network namespace in config
> > > > > > 	  dumps
> > > > > > 	* Add ${_INSTANCE} config parameter.
> > > > > > 	* Remove debugging log message.
> > > > > > 	* Recalculate max_fd used for select if it should reduce.
> > > > > > 	* Add tracking of virtual routes.
> > > > > > 	  If a virtual route is deleted, by default to vrrp instance
> > > > > > 	  will now transition to backup mode, and if it transitions to
> > > > > > 	  master again the route will be re-added. If an interface on
> > > > > > 	  which a route is configured is down, then the instance will go
> > > > > > 	  to fault state, since the route cannot be added.
> > > > > > 	  This commit also adds a no-track option for routes, which
> > > > > > 	  means that deletion of the route will not cause the vrrp
> > > > > > 	  instance to transition to backup.
> > > > > > 	* Handle interface down at startup with tracked route configured
> > > > > > 	  on it. If a virtual route which is tracked is configured on an
> > > > > > 	  interface that is down at startup, then the vrrp instance
> > > > > > 	  needs to start in fault state.
> > > > > > 	* Rename netlink_reflect_filter() to netlink_link_filter()
> > > > > > 	  The function only handles RTM_NEWLINK/RTM_DELLINK messages and
> > > > > > 	  there are other functions to handle other message types.
> > > > > > 	* Fix compilation warning.
> > > > > > 	* Make recreating deleted VMACs work.
> > > > > > 	* Fix Travis-CI compilation failure and warning.
> > > > > > 	* Stop duplicate definition and duplicate include in
> > > > > > 	  vrrp_iproute.c.
> > > > > > 	* Add new ip rule options for Linux 4.17
> > > > > > 	  FRA_PROTOCOL, FRA_IP_PROTO, FRA_SPORT_RANGE and
> > > > > > 	  FRA_DPORT_RANGE have been added in Linux 4.17.
> > > > > > 
> > > > > > 
> > > > > > regs,
> > > > > > Alexandre
> > > > > > 
> > > > > > ------------------------------------------------------------------------------
> > > > > > Check out the vibrant tech community on one of the world's most
> > > > > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> > > > > > _______________________________________________
> > > > > > Keepalived-devel mailing list
> > > > > > [email protected]
> > > > > > https://lists.sourceforge.net/lists/listinfo/keepalived-devel
-- 
Damien Clabaut
R&D vRouter
OVH
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot