Re: VRRP packets with duplicate AH sequence number

Aleksey,

This appears to confirm that keepalived is only sending the packet once,
but for some reason the packet is being looped back to the keepalived
process multiple times (in this case 86 times) in the space of 2.248
milli-seconds, and for the tcpdump trace you show the packets are being
received at approximately 4 micro-second intervals.

Before the problem of duplicate packets occurs keepalived is sending
packets of intervals of approximately 1.001 seconds; after the duplicate
packets issue is resolved the interval is approx 1.0005 seconds (i.e.
the extra delay over the configured 1 second has halved). The packet
that is duplicated (seq no 996 = 0x3e4) is first shown with a timestamp
of 14:53:36.066046 (and final timestamp 14:53:36.068294), which suggests
that the next packet should have been seen at 14:53:37.067
approximately, whereas it wasn't seen until 14:53:37.460675,
approximately 390 milli-seconds later than expected, and it then
continues from there at 1.0005 second intervals. This suggests that
something has happened to NODE2 that caused a delay in the scheduling of
keepalived.

Whether it is something within the kernel on NODE2, or whether it is
something happening within your network, I cannot say from the
information provided. I suspect, however, it is something in your
network that, having received the VRRP packet is retransmitting it
multiple times, which would explain why keepalived is receiving the
packet when it normally doesn't, and also the machine where you are
running the tcpdump is seeing the packet multiple times. On the other
hand, something is happening on NODE2 to cause a 1.39s interval between
sending the packet with sequence no 996, and the next packet, with
sequence no 1083; this could be down to processing a very high number of
packets from the network or for some reason looping back messages
itself.

I do not see how keepalived can be causing the problem, since the log
shows that there is only 1 call to sendmsg(), so the only conclusion I
can see is that the problem is either occurring within the networking
stack on NODE2, or elsewhere on your network. To me, this conclusion is
reinforced by their being no other reports of similar problems, so far
as I am aware.

As an aside, I don't believe Keepalived is behaving as it should do in
that when it receives a packet that it considers is invalid, since it is
still incrementing the sequence number. It should be noted that IPSEC
(or indeed any authentication) has not been part of the VRRP
specification since 2004, when RFC3768 superseded RFC2338, but I
understand some people still find authentication useful. RFC2338 didn't
fully specify how IPSEC should be used, and so Alexandre wrote
http://www.keepalived.org/draft-ietf-vrrp-ipsecah-spec-00.txt  in 2003.
It might be worth checking whether Keepalived still conforms to that
document, and indeed whether when a master receives a packet that isn't
of higher priority, it should not increment the next sequence number.

Quentin

On Wed, 2017-05-17 at 15:03 +0300, Aleksey wrote:

> well, here is the output:
> 
> 25127 May 17 14:53:33 NODE2 Keepalived_vrrp[15515]: Sending advert with 
> seq no 993
> 25128 May 17 14:53:34 NODE2 Keepalived_vrrp[15515]: Sending advert with 
> seq no 994
> 25129 May 17 14:53:35 NODE2 Keepalived_vrrp[15515]: Sending advert with 
> seq no 995
> 25130 May 17 14:53:36 NODE2 Keepalived_vrrp[15515]: Sending advert with 
> seq no 996
> 25131 May 17 14:53:36 NODE2 Keepalived_vrrp[15515]: VRRP_Instance(TEST) 
> IPSEC-AH : sequence number 996 already proceeded. Packet dropped. 
> Local(997)
> 25132 May 17 14:53:36 NODE2 Keepalived_vrrp[15515]: bogus VRRP packet 
> received on eth0 !!!
> 25133 May 17 14:53:36 NODE2 Keepalived_vrrp[15515]: VRRP_Instance(TEST) 
> Dropping received VRRP packet...
> 25134 May 17 14:53:36 NODE2 Keepalived_vrrp[15515]: VRRP_Instance(TEST) 
> IPSEC-AH : sequence number 996 already proceeded. Packet dropped. 
> Local(998)
> 25135 May 17 14:53:36 NODE2 Keepalived_vrrp[15515]: bogus VRRP packet 
> received on eth0 !!!
> 25136 May 17 14:53:36 NODE2 Keepalived_vrrp[15515]: VRRP_Instance(TEST) 
> Dropping received VRRP packet...
> 25137 May 17 14:53:36 NODE2 Keepalived_vrrp[15515]: VRRP_Instance(TEST) 
> IPSEC-AH : sequence number 996 already proceeded. Packet dropped. 
> Local(999)
> ---some lines omitted---
> 25386 May 17 14:53:36 NODE2 Keepalived_vrrp[15515]: VRRP_Instance(TEST) 
> IPSEC-AH : sequence number 996 already proceeded. Packet dropped. 
> Local(1082)
> 25387 May 17 14:53:36 NODE2 Keepalived_vrrp[15515]: bogus VRRP packet 
> received on eth0 !!!
> 25388 May 17 14:53:36 NODE2 Keepalived_vrrp[15515]: VRRP_Instance(TEST) 
> Dropping received VRRP packet...
> 25389 May 17 14:53:37 NODE2 Keepalived_vrrp[15515]: Sending advert with 
> seq no 1083
> 25390 May 17 14:53:38 NODE2 Keepalived_vrrp[15515]: Sending advert with 
> seq no 1084
> 25391 May 17 14:53:39 NODE2 Keepalived_vrrp[15515]: Sending advert with 
> seq no 1085
> 25392 May 17 14:53:40 NODE2 Keepalived_vrrp[15515]: Sending advert with 
> seq no 1086
> 
> 
> Here is the tcpdump:
> 14:53:31.060704 IP 10.1.64.21 > 224.0.0.18: 
> AH(spi=0x0a014015,seq=0x3df): VRRPv2, Advertisement, vrid 73, prio 200, 
> authtype ah, intvl 1s, length 20
> 14:53:32.061823 IP 10.1.64.21 > 224.0.0.18: 
> AH(spi=0x0a014015,seq=0x3e0): VRRPv2, Advertisement, vrid 73, prio 200, 
> authtype ah, intvl 1s, length 20
> 14:53:33.062940 IP 10.1.64.21 > 224.0.0.18: 
> AH(spi=0x0a014015,seq=0x3e1): VRRPv2, Advertisement, vrid 73, prio 200, 
> authtype ah, intvl 1s, length 20
> 14:53:34.063809 IP 10.1.64.21 > 224.0.0.18: 
> AH(spi=0x0a014015,seq=0x3e2): VRRPv2, Advertisement, vrid 73, prio 200, 
> authtype ah, intvl 1s, length 20
> 14:53:35.064916 IP 10.1.64.21 > 224.0.0.18: 
> AH(spi=0x0a014015,seq=0x3e3): VRRPv2, Advertisement, vrid 73, prio 200, 
> authtype ah, intvl 1s, length 20
> 14:53:36.066046 IP 10.1.64.21 > 224.0.0.18: 
> AH(spi=0x0a014015,seq=0x3e4): VRRPv2, Advertisement, vrid 73, prio 200, 
> authtype ah, intvl 1s, length 20
> 14:53:36.066072 IP 10.1.64.21 > 224.0.0.18: 
> AH(spi=0x0a014015,seq=0x3e4): VRRPv2, Advertisement, vrid 73, prio 200, 
> authtype ah, intvl 1s, length 20
> 14:53:36.066076 IP 10.1.64.21 > 224.0.0.18: 
> AH(spi=0x0a014015,seq=0x3e4): VRRPv2, Advertisement, vrid 73, prio 200, 
> authtype ah, intvl 1s, length 20
> 14:53:36.066085 IP 10.1.64.21 > 224.0.0.18: 
> AH(spi=0x0a014015,seq=0x3e4): VRRPv2, Advertisement, vrid 73, prio 200, 
> authtype ah, intvl 1s, length 20
> ---some lines omitted---
> 14:53:36.068288 IP 10.1.64.21 > 224.0.0.18: 
> AH(spi=0x0a014015,seq=0x3e4): VRRPv2, Advertisement, vrid 73, prio 200, 
> authtype ah, intvl 1s, length 20
> 14:53:36.068292 IP 10.1.64.21 > 224.0.0.18: 
> AH(spi=0x0a014015,seq=0x3e4): VRRPv2, Advertisement, vrid 73, prio 200, 
> authtype ah, intvl 1s, length 20
> 14:53:36.068294 IP 10.1.64.21 > 224.0.0.18: 
> AH(spi=0x0a014015,seq=0x3e4): VRRPv2, Advertisement, vrid 73, prio 200, 
> authtype ah, intvl 1s, length 20
> 14:53:37.460675 IP 10.1.64.21 > 224.0.0.18: 
> AH(spi=0x0a014015,seq=0x43b): VRRPv2, Advertisement, vrid 73, prio 200, 
> authtype ah, intvl 1s, length 20
> 14:53:38.461012 IP 10.1.64.21 > 224.0.0.18: 
> AH(spi=0x0a014015,seq=0x43c): VRRPv2, Advertisement, vrid 73, prio 200, 
> authtype ah, intvl 1s, length 20
> 14:53:39.461632 IP 10.1.64.21 > 224.0.0.18: 
> AH(spi=0x0a014015,seq=0x43d): VRRPv2, Advertisement vrid 73, prio 200, 
> authtype ah, intvl 1s, length 20
> 14:53:40.461996 IP 10.1.64.21 > 224.0.0.18: 
> AH(spi=0x0a014015,seq=0x43e): VRRPv2, Advertisement, vrid 73, prio 200, 
> authtype ah, intvl 1s, length 20
> 14:53:41.462617 IP 10.1.64.21 > 224.0.0.18: 
> AH(spi=0x0a014015,seq=0x43f): VRRPv2, Advertisement, vrid 73, prio 200, 
> authtype ah, intvl 1s, length 20
> 
> 
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
Keepalived-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/keepalived-devel