notification scrips executed in wrong order

Using keepalived 1.2.10 on a CentOS 6 Linux.

I have a backup instance with notification scripts:

 notify_master "/root/test.sh master"
 notify_backup "/root/test.sh backup"

script is at the bottom of the mail.
Now I simulate a missed VRRP packet as regularly seen at a site in the field:

# iptables -I INPUT -d 224.0.0.0/8 -j DROP

... wait until I see something happen in /var/log/message

# iptables -I INPUT -d 224.0.0.0/8 -j ACCEPT

I see the following output from keepalived (as expected):

VRRP_Instance({) Transition to MASTER STATE
VRRP_Instance({) Entering MASTER STATE
VRRP_Instance({) setting protocol VIPs.
VRRP_Instance({) Sending gratuitous ARPs on eth1 for 192.168.3.155
Opening script file /root/test.sh
Netlink reflector reports IP 192.168.3.155 added
VRRP_Instance({) Received higher prio advert
VRRP_Instance({) Entering BACKUP STATE
VRRP_Instance({) removing protocol VIPs.
Opening script file /root/test.sh
Netlink reflector reports IP 192.168.3.155 removed

But the scripts are executed in the wrong order (1 out of 3 attempts):
[12:01:00.776829933] [13695 -> 13696] (./keepalived-l-D-d-n-f/root/keepalived.conf) backup to backup
[12:01:00.779875164] [13694 -> 13697] (./keepalived-l-D-d-n-f/root/keepalived.conf) backup to master

First number is the PPID of the script, second the PID. So the process that calls the script to go into backup state overtakes the script that is called to go into the master state.
This can also nicely be seen when attaching strace.

As the real script controls an application that uses this state information, the result is that the application is in master state, but should be in backup state.

I believe some rework for the notification mechanism needs to be done to ensure the correct order of script execution. What is the correct process? I'd be happy to contribute to that (though I have no easy fix in mind, yet).

Cheers,
Martin

[root@cluster2 ~]# cat test.sh
#!/bin/bash

BASEDIR=`dirname $0`
LOGFILE="$BASEDIR/test.log"
LOCKFILE="$BASEDIR/test.lock"
STATE="$BASEDIR/state"

do_log()
{
        echo -n "[`date +%T.%N`] " >> $LOGFILE
        if [ -n "$PPID" ] ; then
                echo -n "[$PPID -> $$] " >> $LOGFILE
                echo -n "(`cat /proc/$PPID/cmdline`) " >> $LOGFILE
        fi
        echo "$*" >> $LOGFILE
}

(
        flock -x 200

        if [ -f $STATE ] ; then
                state=`cat $STATE`
        else
                state="backup"
        fi

case "$1" in
        master)
        do_log "$state to master"
        echo "master" > $STATE
        ;;
        backup)
        do_log "$state to backup"
        echo "backup" > $STATE
        ;;
        fault)
        do_log "fault"
        ;;
esac
) 200> $LOCKFILE


Firmensitz: Muenchen Amtsgericht: AG Muenchen Handelsregister: HRB 144340 Geschaeftsfuehrer: Timothy James Daly, Jonathan Park, Martin Stecher Bankverbindung: ABN-Amro Bank N.V. Konto 671 211 9006 UST-ID: DE168122444

------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today. 
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk