Re: [ANNOUNCE] keepalived 2.0.1
Damien Clabaut <[email protected]> Fri, 15 Jun 2018 15:14:14 +0000
| Newsgroups | gmane.linux.keepalived.devel |
|---|---|
| Message-ID | <[email protected]> |
Quentin,
Thank you for the precision.
So far I do not see a usecase where we would have more than 500 VRRP instances in the same netns.
We have our own implementation of VRFs and we chose to start one Keepalived process in each - allowing for one of them to crash or restart without
affecting the others.
Regards,
On Fri, 2018-06-15 at 16:01 +0100, Quentin Armitage wrote:
> Damien,
>
> Regarding implementing epoll() I suspect that would take two to four weeks to implement and fully test, but it may well be that the work that
> Alexandre has already done with scheduler-ng might reduce that.
>
> The limit of 1024 file descriptors is, in effect, per keepalived instance. Yes, you can certainly run different instances of keepalived in separate
> netns+mntns and each one will be able to handle 500 or so VRRP instances. Are you aware that keepalived already has support for namespaces? If you
> specify "net_namespace NET_NS_NAME" at the beginning of your configuration file, then keepalived will run in that network namespace (it also creates
> a mntns so that the pid files (to it) appear to be in /var/run/keepalived, but in the default mntns they are in /var/run/keepalived/NET_NS_NAME/.
>
> I use network namespaces extensively so that I can test multiple keepalived instances all talking to each other, but all in a single machine. An
> example configuration I would use, with netns names low and high, and keepalived run with either the '-i low' or '-i high' option is:
>
> @low net_namespace low
> @high net_namespace high
>
> @low $US=3
> @low $OTHER=1
> @high $US=1
> @high $OTHER=3
> @low $ID=low
> @high $ID=high
>
> global_defs {
> router_id $ID
> }
>
> vrrp_instance VI_1 {
> interface eth0
> virtual_router_id 1
> @high priority 90
> @low priority 80
> version 3
>
> virtual_ipaddress {
> 2xxx:xxx:xxxx::10/128 dev eth0
> }
> }
>
> This allows that same configuration file to be used for both the low and high namespaces, with the conditional configuration handling the
> differences between the two configs. It is also using replaceable parameters ($ID).
>
> A more interesting use of replaceable parameters/templates is below. $VI= ... defines a template, and then $VI is an instantiation of that template.
> Using this would significantly reduce the size of the config files you are creating for running your several hundred vrrp instance tests.
>
> Quentin
>
>
> $VI= \
> vrrp_track_file TF_${NUM} { \
> file /tmp/track_file_${NUM}.trk \
> weight 0 \
> init_file \
> } \
> \
> vrrp_instance VI_$NUM { \
> interface $IF \
> state $INIT_STATE \
> virtual_router_id $VRID \
> version $VRRP_VERSION \
> priority $PRIORITY \
> advert_int 1 \
> # use_vmac \
> virtual_ipaddress { \
> 10.2.${NUM}.100/32 \
> 10.2.${NUM}.101/32 \
> } \
> track_script { \
> scr${NUM} \
> } \
> track_interface { \
> $IF_TRACK \
> } \
> track_file { \
> TF_${NUM} \
> } \
> notify_master "${_PWD}/scripts/notify_event.sh VI_${NUM}.$INST master" \
> notify_backup "${_PWD}/scripts/notify_event.sh VI_${NUM}.$INST backup" \
> notify_fault "${_PWD}/scripts/notify_event.sh VI_${NUM}.$INST fault" \
> notify_stop "${_PWD}/scripts/notify_event.sh VI_${NUM}.$INST stop" \
> notify ${_PWD}/scripts/notify_all.${INST}.sh \
> smtp_alert \
> }
>
> $INIT_STATE=BACKUP
> $IF=wlp3s0 # For when not running in a namespace
> @low $IF=eth$NUM
> @high $IF=eth$NUM
> $IF_TRACK=wlp3s0
> @low $IF_TRACK=eth$TRACK_NUM
> @high $IF_TRACK=eth$TRACK_NUM
> $VRID=253
>
> $NUM=0
> $TRACK_NUM=4
> $SCRIPT
> $VI # create VI_0
>
> $NUM=1
> $TRACK_NUM=5
> $SCRIPT
> $VI # create VI_1
>
>
> On Fri, 2018-06-15 at 14:26 +0000, Damien Clabaut wrote:
> > Hello Quentin and thank you for your answer.
> >
> > You have been quick - I was going to send you an email to report that keepalived kept crashing when going over 500 instances.
> >
> > When you say the limit is at 1024 file descriptors, is this for the entire OS, or will launching Keepalived in separate netns+mntns allow for 1024
> > per
> > netns+mntns ?
> >
> > Regarding the move from select() to epoll(), may I ask you to elaborate ? How much work would it be ?
> >
> > Regards,
> >
> > On Fri, 2018-06-15 at 15:20 +0100, Quentin Armitage wrote:
> > > Damien,
> > >
> > > Many thanks for the update. It's good to hear that keepalived is holding up well, and very helpful to have some intense independent testing.
> > >
> > > At the moment keepalived uses select() for sending and receiving on sockets, and keepalived opens two sockets per interface that it is using for
> > > vrrp (one send, one receive). When using VMACs that means two sockets per vrrp instance, and since select() has a limit of 1024 file
> > > descriptors,
> > > that means the limit of the number of vrrp instances is about 500 (allowing for the other sockets that keepalived uses).
> > >
> > > One way to reduce the number of sockets used is not to use VMACs, since it will open only two sockets per interface, so if you have 250 vrrp
> > > instances all on the same interface it will only use 2 sockets, as opposed to the 500 it would use if using VMACs.
> > >
> > > Please let us know if you need to be able to run more than 500 vrrp instances using VMACs, in which case we can look at merging a scheduler
> > > using
> > > epoll() instead of select().
> > >
> > > Quentin
> > >
> > > On Fri, 2018-06-15 at 12:03 +0000, Damien Clabaut wrote:
> > > > Hello Alexandre, Quentin,
> > > >
> > > > We finally have had time to start our tests on Keepalived 2.0.2, with our usual patches (mostly, enabling json at compilation).
> > > >
> > > > The test setup is as follows:
> > > > 2 servers (xeon e5 + Intel 82599ES) are connected together through a switch (2*10G LACP from each server to the switch).
> > > >
> > > > We use the template we pushed to the Keepalived repo to spawn 398 VRRP instances (vlans 2-200, one instance per vlan per IP version) with the
> > > > track_file option (each instance track a different file) on each server.
> > > >
> > > > Then, we have an isolation tool, that will basically collect the state of the system from JSON, then write "-100" in the files tracked by each
> > > > instance (we do 1 instance every second to have a smooth migration). The script then waits for all instances to be in the BACKUP state, before
> > > > reversing everything.
> > > >
> > > > We left that running in a loop overnight (75 runs of the loop). Each run takes around 800 seconds (~400 instances, 2 movements per instance, 1
> > > > movement per second).
> > > >
> > > > Results:
> > > > - Used CPU/RAM do not increase over time
> > > > - We use about 20% of 1 core for softirq
> > > > - No instance ever went to FAULT state
> > > > - Even after so many runs of the loop, each run takes the same time
> > > >
> > > > Congratulations on making something this stable, we will be happy to keep testing before putting Keepalived 2.0.2 in production.
> > > >
> > > > Next test, which will take place over the weekend, will be the same as described above but with more instances (probably over 2000, so we can
> > > > see
> > > > what
> > > > happens when we have more than 100% of 1 core used by softirq).
> > > >
> > > > Regards,
> > > > Damien
> > > > OVH vRouter team
> > > >
> > > > On Tue, 2018-06-05 at 13:34 +0000, Damien Clabaut wrote:
> > > > > Thank you for your answer, I will test this thoroughly.
> > > > > Damien
> > > > > On Tue, 2018-06-05 at 14:02 +0100, Quentin Armitage wrote:
> > > > > > Damien,
> > > > > >
> > > > > > The move from v1.4.5 to v2.0.0 was exactly merging the Beta branch in Master, i.e. I executed
> > > > > > git merge beta
> > > > > > and pushed the merged branch.
> > > > > >
> > > > > > The main changes since 2018-05-11 which are included in v2.0.0 are:
> > > > > > Making notifications consistent across SNMP, notify scripts, notify FIFOs and SMTP messages
> > > > > > Detecting virtual ip address deletions and transitioning to backup (it will subsequently transition to master unless another instance
> > > > > > becomes
> > > > > > master)
> > > > > > Handling interface deletion, transitioning to fault, and then backup once interface restored.
> > > > > > Sundry fixes/improvements
> > > > > >
> > > > > > Additions for v2.0.1
> > > > > > Handling VMAC deletion and restoring VMACs
> > > > > > Handing virtual route deletion either due to the route being explicitly deleted or and interface going down
> > > > > > Support for new ip rule options in Linux 4.17
> > > > > > Sundry fixes/improvements
> > > > > >
> > > > > > Coming soon
> > > > > > Handling virtual rule deletions
> > > > > > Handling static address, route and rule deletions
> > > > > >
> > > > > > Once the handling of all address, route and rule deletions is completed, then keepalived should be able to fully support hot-swapping of
> > > > > > NICs.
> > > > > >
> > > > > > Quentin
> > > > > >
> > > > > > On Tue, 2018-06-05 at 11:46 +0000, Damien Clabaut wrote:
> > > > > > > Hello Alexandre,
> > > > > > >
> > > > > > > This is great news!
> > > > > > >
> > > > > > > Can you confirm that the move from 1.X to 2.0 is "just" merging the Beta branch to Master ?
> > > > > > >
> > > > > > > We have tested Git Beta branch from 2018-05-11, should we expect a lot of change ?
> > > > > > >
> > > > > > > Regards,
> > > > > > > On Mon, 2018-06-04 at 19:23 +0200, Alexandre Cassen wrote:
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > New release, with some updates and extensions.
> > > > > > > >
> > > > > > > > This code can be found on www.keepalived.org website or on
> > > > > > > > https://github.com/acassen/keepalived.
> > > > > > > >
> > > > > > > > ChangeLog for this release is :
> > > > > > > >
> > > > > > > > 2018-06-04 Quentin Armitage <[email protected]>
> > > > > > > > * keepalived-2.0.1
> > > > > > > > * Remove '\n' characters from log_message() text.
> > > > > > > > * Allow IPv6 ip rules to be specified using fwmarks.
> > > > > > > > * Fix configure generation of keepalived.spec file.
> > > > > > > > * Stop rebuilding scheduler.o every make.
> > > > > > > > * Remove ' characters from configure args in keepalived -v
> > > > > > > > output.
> > > > > > > > * Remove duplicate reporting of network namespace in config
> > > > > > > > dumps
> > > > > > > > * Add ${_INSTANCE} config parameter.
> > > > > > > > * Remove debugging log message.
> > > > > > > > * Recalculate max_fd used for select if it should reduce.
> > > > > > > > * Add tracking of virtual routes.
> > > > > > > > If a virtual route is deleted, by default to vrrp instance
> > > > > > > > will now transition to backup mode, and if it transitions to
> > > > > > > > master again the route will be re-added. If an interface on
> > > > > > > > which a route is configured is down, then the instance will go
> > > > > > > > to fault state, since the route cannot be added.
> > > > > > > > This commit also adds a no-track option for routes, which
> > > > > > > > means that deletion of the route will not cause the vrrp
> > > > > > > > instance to transition to backup.
> > > > > > > > * Handle interface down at startup with tracked route configured
> > > > > > > > on it. If a virtual route which is tracked is configured on an
> > > > > > > > interface that is down at startup, then the vrrp instance
> > > > > > > > needs to start in fault state.
> > > > > > > > * Rename netlink_reflect_filter() to netlink_link_filter()
> > > > > > > > The function only handles RTM_NEWLINK/RTM_DELLINK messages and
> > > > > > > > there are other functions to handle other message types.
> > > > > > > > * Fix compilation warning.
> > > > > > > > * Make recreating deleted VMACs work.
> > > > > > > > * Fix Travis-CI compilation failure and warning.
> > > > > > > > * Stop duplicate definition and duplicate include in
> > > > > > > > vrrp_iproute.c.
> > > > > > > > * Add new ip rule options for Linux 4.17
> > > > > > > > FRA_PROTOCOL, FRA_IP_PROTO, FRA_SPORT_RANGE and
> > > > > > > > FRA_DPORT_RANGE have been added in Linux 4.17.
> > > > > > > >
> > > > > > > >
> > > > > > > > regs,
> > > > > > > > Alexandre
> > > > > > > >
> > > > > > > > ------------------------------------------------------------------------------
> > > > > > > > Check out the vibrant tech community on one of the world's most
> > > > > > > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> > > > > > > > _______________________________________________
> > > > > > > > Keepalived-devel mailing list
> > > > > > > > [email protected]
> > > > > > > > https://lists.sourceforge.net/lists/listinfo/keepalived-devel
--
Damien Clabaut
R&D vRouter
OVH
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot