Re: [ANNOUNCE] keepalived-1.2.13

* Alexandre Cassen

>         * keepalived-1.2.13 released.

Thanks! :-) My initial testing tells me I can go in production with this
version. I noticed some things though:

>         * ipvs : ipvs_syncd_cmd uses memset() to zero the daemonrule
>           buffer before populating it and sending it up. daemonrule is
>           malloc()ed by ipvs_start(). ipvs_start() can bail early if it
>           can't communicate with ipvs. Neither place which call
>           ipvs_start() check the return value, allowing them to walk
>           straight into a NULL pointer deref. (jsgh)

I am only using VRRP, not IPVS. So with version v1.2.12 and earlier,
when starting up, I would get a harmless error:

[...]
Opening file '/etc/keepalived/keepalived.conf'.
IPVS: Can't initialize ipvs: Protocol not available
Configuration is using : 225633 Bytes
[...]

However, with v1.2.13, this turns into a log-spamming tight loop where
the healthcheck is respawned over and over and over again:

[...]
Initializing ipvs 2.6
IPVS: Can't initialize ipvs: Protocol not available
Healthcheck child process(28566) died: Respawning
Starting Healthcheck child process, pid=28570
Initializing ipvs 2.6
IPVS: Can't initialize ipvs: Protocol not available
Healthcheck child process(28570) died: Respawning
Starting Healthcheck child process, pid=28574
Initializing ipvs 2.6
[...and this goes on and on and on...]

This is due to the following change:

diff --git a/keepalived/check/check_daemon.c b/keepalived/check/check_daemon.c
index 29dd6d2..240ad00 100644
--- a/keepalived/check/check_daemon.c
+++ b/keepalived/check/check_daemon.c
@@ -90,7 +90,10 @@ static void
 start_check(void)
 {
        /* Initialize sub-system */
-       ipvs_start();
+       if (ipvs_start() != IPVS_SUCCESS) {
+               stop_check();
+               return;
+       }
        init_checkers_queue();
 #ifdef _WITH_VRRP_
        init_interface_queue();

Before ipvs_start() failing wasn't a big deal, it would just continue
without. In v1.2.13 ipvs_start() ends up killing the healthchecker
process, which cases it to be restarted - repeat ad infinitum.

The fix is easy, just start keepalived with "--vrrp" to prevent the
healtcheck thread from being started in the first place. So this is
mostly a JFYI.

>         * vrrp : fix double close issue (DROP_MEMBERSHIP & netlink
>           channel). This is a old pending 'bug', not arming at all
>           but just frustrating to see again and again this log message :
> 
>           "cant do IP_DROP_MEMBERSHIP errno=Bad file descriptor (9)"
> 
>           What the hell ! it was due to a double close during reload &
>           stop procedure. VRRP fd are stored in a socket pool and use
>           the I/O MUX to handle VRRP traffic. While reloading or stopping
>           the daemon the I/O MUX was released first and secondly socket
>           pool.  The issue spotted here, in thread_destroy_master() all
>           pending thread are canceled and read/write fds related are
>           close(). Well OK a close on a mcast socket perform kernel side
>           the DROP_MEMBERSHIP when needed, but it is much more clean to
>           perform proper operations userspace ! This patch sequencely
>           cancel pending thread, release socket pool and finally
>           destroy master thread. Same 'issue' appear in netlink channel.

This patch doesn't fully get rid of these errors. When I reload or stop
keepalived v1.2.13, the following messages gets logged:

[...]
Keepalived_vrrp[20000]: cant do IP_DROP_MEMBERSHIP errno=Cannot assign requested address (99)
Keepalived_vrrp[20000]: last message repeated 52 times
[...]

I have 53 vrrp_instances, so it is logged once per instance.

Also worth noting is that it only happens once, so if I reload twice,
the messages only show up on the first reload, or similarly, if I reload
and then stop, they don't show up on the stop.

That said, the issue appears to be solely cosmetic to me.

Thanks again!

Tore




------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs