Re: VRRP packets with duplicate AH sequence number

On 2017-03-24 14:18, Quentin Armitage wrote:
> On Fri, 2017-03-24 at 12:49 +0200, Aleksey wrote:
> 
> Hi guys!
> 
> I have strange behavior of my keepalived running only as VRRP daemon.
> Sometimes it sends advertisements with the same sequence number and
> then
> errors on log appear:
> 
> 11:03:44.993382 IP 10.1.64.156 > 224.0.0.18:
> AH(spi=0x0a01409c,seq=0x10f9e47): VRRPv2, Advertisement, vrid 7, prio
> 200, authtype ah, intvl 1s, length 20
> 11:03:44.993393 IP 10.1.64.156 > 224.0.0.18:
> AH(spi=0x0a01409c,seq=0x10f9e47): VRRPv2, Advertisement, vrid 7, prio
> 200, authtype ah, intvl 1s, length 20
> 11:03:44.993414 IP 10.1.64.156 > 224.0.0.18:
> AH(spi=0x0a01409c,seq=0x10f9e47): VRRPv2, Advertisement, vrid 7, prio
> 200, authtype ah, intvl 1s, length 20
> 11:03:44.993422 IP 10.1.64.156 > 224.0.0.18:
> AH(spi=0x0a01409c,seq=0x10f9e47): VRRPv2, Advertisement, vrid 7, prio
> 200, authtype ah, intvl 1s, length 20
> 
> Mar 24 11:03:44 Zero-white Keepalived_vrrp[17338]: bogus VRRP packet
> received on eth0 !!!
> Mar 24 11:03:44 Zero-white Keepalived_vrrp[17338]: VRRP_Instance(LAN)
> Dropping received VRRP packet...
> Mar 24 11:03:44 Zero-white Keepalived_vrrp[17338]: VRRP_Instance(LAN)
> IPSEC-AH : sequence number 17800775 already proceeded. Packet dropped.
> 
> Local(17800812)
> Mar 24 11:03:44 Zero-white Keepalived_vrrp[17338]: bogus VRRP packet
> received on eth0 !!!
> Mar 24 11:03:44 Zero-white Keepalived_vrrp[17338]: VRRP_Instance(LAN)
> Dropping received VRRP packet...
> Mar 24 11:03:44 Zero-white Keepalived_vrrp[17338]: VRRP_Instance(LAN)
> IPSEC-AH : sequence number 17800775 already proceeded. Packet dropped.
> 
> Local(17800813)
> 
> These are just samples, so don't mind the "Local" sequence number
> stated
> in logs - I've got plenty of them. Though number stated on tcpdump and
> 
> "already processed number" in logs matches.
> The problem is floating - but happens frequently (couple of times per
> day). I can't figure out what triggers it and why does keepalived
> consider it's own sent packets as received (I can see them received on
> 
> my box in iptables raw table when doing trace - however tcpdump
> doesn't
> show them as received).
> I've noticed this problem a couple of weeks ago, was nothing similar
> before. If to talk about what was done with the box recently - I
> installed openvpn server on it. Later I tried searching the web and
> found some info that keepalived is not really capable of working with
> openvpn on one box cause it doesn't like hotplugging interfaces which
> openvpn does (tun interfaces). However even if I stop openvpn server
> the
> problem still persists. Is this really so and openvpn can't be used on
> 
> the same box with keepalived?
> I'm using keepalived 1.2.13 on Debian 7 with 3.16 kernel from
> backports.
> Openvpn version is 2.3.2 (if it is important).
> My keepalived is below:
> vrrp_instance LAN {
>  state BACKUP
>  interface eth0
>  virtual_router_id 7
>  priority 200
>  advert_int 1
>  garp_master_delay 1
>  #nopreempt
>  preempt
>  preempt_delay 2
>  notify_fault "/etc/keepalived/gw_vrrp_master_down"
>  notify_backup "/etc/keepalived/gw_vrrp_master_down"
>  notify_master "/etc/keepalived/gw_vrrp_master_up"
> authentication {
>  auth_type AH
>  auth_pass 12345
> }
> virtual_ipaddress {
>  10.1.64.155
> }
> }
> 
> Any help is appreciated. Thanks in advance.
> 
>  It appears that the packets are the same packet that for some reason
> is being reported as being received 4 times (there is a 60
> micro-second interval from the first to the last). This suggests to me
> that it is not keepalived sending the packets 4 times, but for some
> reason 1 packet is being reported 4 times as being received, which is
> presumably either due to some networking configuration or something
> happening in your kernel. It's probably worth looking in all your
> system logs at the time this problem happens to see if there is
> anything else happening at the same time. Did installing openvpn
> server change some other aspect of your system configuration so that
> even when not running the server you have a different configuration
> from before?
> 
>  Your version of keepalived is very old - some 16 versions behind the
> current version. If would be worth upgrading to version 1.3.5, since
> there have been hundreds of patches since your version.
> 
>  There is indeed an issue of keepalived not fully supporting
> hotplugging; it is something that I plan to add in the near future.
> However, if keepalived isn't configured to use any of the openvpn
> interfaces, then it shouldn't be a problem for keepalived (I'm talking
> about recent versions of keepalived here; I don't know about v1.2.13).
> 
>  Are you in a position to try uninstalling openvpn and seeing if the
> problem persists?
> 
>  Quentin Armitage


Hi Quentin!

Thanks for your reply.

Well, the box serves as the router/firewall, has asterisk and dhcpd 
running as well as it had ipsec (strongswan) which is now turned off. 
And openvpn now of course.

System logs don't show something special exactly prior to the issue - 
some dhcp leases, tftp requests from IP phones and so on. Turning off 
openvpn doesn't seem to help. Except for firewall configuration (adding 
accept udp/1194 on input + adding rules between LAN and openvpn IP pool) 
no other systems settings were altered during openvpn setup.

Keepalived runs on physical interface, it doesn't run on tun interfaces 
of course. I can try uninstalling openvpn in fact it is now some kind of 
testing on my box. So i'll try it and report.

Also I'll try to upgrade keepalived - 1.2.13 is just the version in 
Debian 8 repositories (also installed it from backports just as the 3.16 
kernel).

Probably I'll also try to upgrade my whole debian box to debian 8 - I 
have seen problems with some software when some backported packages 
(especially kernel) are installed in the system. But this option will 
probably be the last - cause it obviously inflicts the services the box 
provides and I'll need to get a maintenance window for it.

Thanks a lot for clues, I'll try and get back with the results.
-- 
With kind regards,
Aleksey

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot