Re: VRRP packets with duplicate AH sequence number

On Fri, 2017-03-24 at 12:49 +0200, Aleksey wrote:

> Hi guys!
> 
> I have strange behavior of my keepalived running only as VRRP daemon. 
> Sometimes it sends advertisements with the same sequence number and then 
> errors on log appear:
> 
> 11:03:44.993382 IP 10.1.64.156 > 224.0.0.18: 
> AH(spi=0x0a01409c,seq=0x10f9e47): VRRPv2, Advertisement, vrid 7, prio 
> 200, authtype ah, intvl 1s, length 20
> 11:03:44.993393 IP 10.1.64.156 > 224.0.0.18: 
> AH(spi=0x0a01409c,seq=0x10f9e47): VRRPv2, Advertisement, vrid 7, prio 
> 200, authtype ah, intvl 1s, length 20
> 11:03:44.993414 IP 10.1.64.156 > 224.0.0.18: 
> AH(spi=0x0a01409c,seq=0x10f9e47): VRRPv2, Advertisement, vrid 7, prio 
> 200, authtype ah, intvl 1s, length 20
> 11:03:44.993422 IP 10.1.64.156 > 224.0.0.18: 
> AH(spi=0x0a01409c,seq=0x10f9e47): VRRPv2, Advertisement, vrid 7, prio 
> 200, authtype ah, intvl 1s, length 20
> 
> Mar 24 11:03:44 Zero-white Keepalived_vrrp[17338]: bogus VRRP packet 
> received on eth0 !!!
> Mar 24 11:03:44 Zero-white Keepalived_vrrp[17338]: VRRP_Instance(LAN) 
> Dropping received VRRP packet...
> Mar 24 11:03:44 Zero-white Keepalived_vrrp[17338]: VRRP_Instance(LAN) 
> IPSEC-AH : sequence number 17800775 already proceeded. Packet dropped. 
> Local(17800812)
> Mar 24 11:03:44 Zero-white Keepalived_vrrp[17338]: bogus VRRP packet 
> received on eth0 !!!
> Mar 24 11:03:44 Zero-white Keepalived_vrrp[17338]: VRRP_Instance(LAN) 
> Dropping received VRRP packet...
> Mar 24 11:03:44 Zero-white Keepalived_vrrp[17338]: VRRP_Instance(LAN) 
> IPSEC-AH : sequence number 17800775 already proceeded. Packet dropped. 
> Local(17800813)
> 
> These are just samples, so don't mind the "Local" sequence number stated 
> in logs - I've got plenty of them. Though number stated on tcpdump and 
> "already processed number" in logs matches.
> The problem is floating - but happens frequently (couple of times per 
> day). I can't figure out what triggers it and why does keepalived 
> consider it's own sent packets as received (I can see them received on 
> my box in iptables raw table when doing trace - however tcpdump doesn't 
> show them as received).
> I've noticed this problem a couple of weeks ago, was nothing similar 
> before. If to talk about what was done with the box recently - I 
> installed openvpn server on it. Later I tried searching the web and 
> found some info that keepalived is not really capable of working with 
> openvpn on one box cause it doesn't like hotplugging interfaces which 
> openvpn does (tun interfaces). However even if I stop openvpn server the 
> problem still persists. Is this really so and openvpn can't be used on 
> the same box with keepalived?
> I'm using keepalived 1.2.13 on Debian 7 with 3.16 kernel from backports. 
> Openvpn version is 2.3.2 (if it is important).
> My keepalived is below:
> vrrp_instance LAN {
>          state BACKUP
>          interface eth0
>          virtual_router_id 7
>          priority 200
>          advert_int 1
>          garp_master_delay 1
>          #nopreempt
>          preempt
>          preempt_delay 2
>          notify_fault "/etc/keepalived/gw_vrrp_master_down"
>          notify_backup "/etc/keepalived/gw_vrrp_master_down"
>          notify_master "/etc/keepalived/gw_vrrp_master_up"
> authentication {
>          auth_type AH
>          auth_pass 12345
> }
> virtual_ipaddress {
>          10.1.64.155
> }
> }
> 
> Any help is appreciated. Thanks in advance.
> 

It appears that the packets are the same packet that for some reason is
being reported as being received 4 times (there is a 60 micro-second
interval from the first to the last). This suggests to me that it is not
keepalived sending the packets 4 times, but for some reason 1 packet is
being reported 4 times as being received, which is presumably either due
to some networking configuration or something happening in your kernel.
It's probably worth looking in all your system logs at the time this
problem happens to see if there is anything else happening at the same
time. Did installing openvpn server change some other aspect of your
system configuration so that even when not running the server you have a
different configuration from before?

Your version of keepalived is very old - some 16 versions behind the
current version. If would be worth upgrading to version 1.3.5, since
there have been hundreds of patches since your version.

There is indeed an issue of keepalived not fully supporting hotplugging;
it is something that I plan to add in the near future. However, if
keepalived isn't configured to use any of the openvpn interfaces, then
it shouldn't be a problem for keepalived (I'm talking about recent
versions of keepalived here; I don't know about v1.2.13).

Are you in a position to try uninstalling openvpn and seeing if the
problem persists?

Quentin Armitage

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
Keepalived-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/keepalived-devel