Re: [PATCH] Increase the size of the netlink recvmsg buffer

On 2016-11-14 03:25, Quentin Armitage wrote:
> On Sat, 2016-11-12 at 16:23 -0800, dwilder wrote:
> 
> On systems with a large number of network interfaces keepalived may
> report:
> "Netlink: error: message truncated messages"
> <and>
> "VRRP is trying to assign VIP to unknown qr-dc4f0313-ca interface !!!
> go
> out
>  and fix your conf !!!"
> 
> This was reported on a system running OpenStack/Nutron.
> 
> I found that the netlink recvmsg buffer was too small causing messages
> 
> to be
> truncated. Increasing the buffer from 4K to 8K prevents the problem.
> (linux/netlink.h suggests using an 8K buffer).
> 
> I tested the fix by creating 100 veth interfaces and verifying the
> errors
> did not return.
> 
> Signed-off-by: David Wilder <[email protected]>
> 
> ---
>  keepalived/vrrp/vrrp_netlink.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/keepalived/vrrp/vrrp_netlink.c
> b/keepalived/vrrp/vrrp_netlink.c
> index d7adffa..7897b4c 100644
> --- a/keepalived/vrrp/vrrp_netlink.c
> +++ b/keepalived/vrrp/vrrp_netlink.c
> @@ -273,7 +273,7 @@ netlink_parse_info(int (*filter) (struct
> sockaddr_nl
> *, struct nlmsghdr *),
>  int error;
> 
>  while (1) {
> - char buf[4096];
> + char buf[8192];
>  struct iovec iov = { buf, sizeof buf };
>  struct sockaddr_nl snl;
>  struct msghdr msg =
> 
>  Hi David,
> 
>  I'm very happy to apply your patch, but since I have been unable to
> reproduce the problem, it would be helpful if you could provide some
> further information.
> 
>  1. I haven't been able to see any reference in linux/netlink.h
> suggesting an 8k buffer (or any other buffer size). Could you please
> let me know where the reference is and/or send a copy of your
> linux/netlink.h
>  2. Could you send a copy of the script you used to create the veth
> interfaces, so that I can reproduce it here.
>  3. What version of Linux are you using?
> 
>  With many thanks,
>  Quentin Armitage

Hi Quentin

Here is the reference from linux/include/linux/netlink.h (the kernel 
header not the user header)
<snip>
/*
  *      skb should fit one page. This choice is good for headerless 
malloc.
  *      But we should limit to 8K so that userspace does not have to
  *      use enormous buffer sizes on recvmsg() calls just to avoid
  *      MSG_TRUNC when PAGE_SIZE is very large.
  */
#if PAGE_SIZE < 8192UL
#define NLMSG_GOODSIZE  SKB_WITH_OVERHEAD(PAGE_SIZE)
#else
#define NLMSG_GOODSIZE  SKB_WITH_OVERHEAD(8192UL)
#endif

#define NLMSG_DEFAULT_SIZE (NLMSG_GOODSIZE - NLMSG_HDRLEN)
<snip>

I forgot to mention that the problem was reported on on ppc64le, it is 
not reproducible on x86-64.  The page size used on ppc64le is 64K, the 
page size on x86-64 is only 4k, from the code above I can infer why the 
problem is not seen with a 4k page size.

The kernel I am running is: 3.10.0-514.el7.ppc64le

We have verified the fix on our original openstack setup.

This is the keepalived config I used in my testing (generated by 
openstack).

vrrp_instance VR_1 {
     state BACKUP
     interface ha-99a3cb02-dc
     virtual_router_id 1
     priority 50
     garp_master_delay 60
     nopreempt
     advert_int 2
     track_interface {
         ha-99a3cb02-dc
     }
     virtual_ipaddress {
         169.254.0.1/24 dev ha-99a3cb02-dc
     }
     virtual_ipaddress_excluded {
         27.0.0.1/24 dev qr-bc6c9831-52
         9.47.64.9/20 dev qg-9b2d21c4-59
         fe80::f816:3eff:fe95:9c41/64 dev qg-9b2d21c4-59 scope link
         fe80::f816:3eff:febb:d0ff/64 dev qr-bc6c9831-52 scope link
     }
     virtual_routes {
         0.0.0.0/0 via 9.47.79.254 dev qg-9b2d21c4-59
     }
}


My test script.  Note: this script will not generate a working 
keepalived setup but it is sufficient to demonstrate the bug.

#!/bin/bash

# List of interfaces to create
Interfaces="ha-99a3cb02-dc qr-bc6c9831-52 qg-9b2d21c4-59"

# Un-comment to generate 100 extra veth pairs
# Interfaces="`seq 1 100` ha-99a3cb02-dc qr-bc6c9831-52 qg-9b2d21c4-59"

for i in $Interfaces; do
         echo Creaating $i
         ip link add $i type veth peer name v-$PEER
         ifconfig $i up
         PEER=$(($PEER+1))
done

# 
KEEPALIVED="/home/wilder/scratch/keepalived-1.2.24/keepalived/keepalived"
KEEPALIVED="keepalived"

CONF=$PWD/test.conf

$KEEPALIVED -d -n -l -D  -f $CONF

echo Done, cleaning up
for i in $Interfaces; do
         ip link del $i
done

-------
( test run showing the problem )
# ./setup
Creaating ha-99a3cb02-dc
Creaating qr-bc6c9831-52
Creaating qg-9b2d21c4-59
Starting Keepalived v1.2.24 (11/11,2016)
Opening file '/home/wilder/scratch/test.conf'.
Starting Healthcheck child process, pid=1130854
Starting VRRP child process, pid=1130855
Initializing ipvs
Netlink: error: message truncated   <<<<< ****
Netlink: error: message truncated   <<<<<<
Netlink: error: message truncated
Netlink: error: message truncated
Netlink: error: message truncated
Netlink: error: message truncated
Netlink reflector reports IP 192.168.0.2 added
Netlink reflector reports IP 192.168.0.2 added
Netlink reflector reports IP 172.17.0.1 added
Netlink reflector reports IP 172.17.0.1 added
Netlink reflector reports IP fe80::9abe:94ff:fe0d:f2f4 added
Netlink reflector reports IP fe80::9abe:94ff:fe0d:f2f4 added
Netlink reflector reports IP fe80::42:dbff:fe53:c725 added
Netlink reflector reports IP fe80::42:dbff:fe53:c725 added
Registering Kernel netlink reflector
Registering Kernel netlink reflector
Registering Kernel netlink command channel
Registering Kernel netlink command channel
Registering gratuitous ARP shared channel
Opening file '/home/wilder/scratch/test.conf'.
Opening file '/home/wilder/scratch/test.conf'.
Cant find interface ha-99a3cb02-dc for vrrp_instance VR_1 !!!
      ha-99a3cb02-dc no match, ignoring...
VRRP is trying to assign ip address 169.254.0.1/24 to unknown 
ha-99a3cb02-dc interface !!! go out and fix your conf !!!
VRRP is trying to assign ip address 9.47.64.9/20 to unknown 
qg-9b2d21c4-59 interface !!! go out and fix your conf !!!
VRRP is trying to assign ip address fe80::f816:3eff:fe95:9c41/64 to 
unknown qg-9b2d21c4-59 interface !!! go out and fix your conf !!!  
<<<<<<<<*****
Cannot specify scope for IPv6 addresses (fe80::f816:3eff:febb:d0ff/64) - 
ignoring scope
VRRP is trying to assign VROUTE to unknown qg-9b2d21c4-59 interface !!! 
go out and fix your conf !!!
VRRP_Instance(VR_1) Unknown interface !
------< Global definitions >------
  Router ID = docker
  Default interface = eth0
  LVS flush = false
  VRRP IPv4 mcast group = 224.0.0.18
  VRRP IPv6 mcast group = ff02::12
  Gratuitous ARP delay = 5
  Gratuitous ARP repeat = 5
  Gratuitous ARP refresh timer = 0
  Gratuitous ARP refresh repeat = 1
  Gratuitous ARP lower priority delay = 0
  Gratuitous ARP lower priority repeat = -1
  Send advert after receive lower priority advert = true
  Gratuitous ARP interval = 0
  Gratuitous NA interval = 0
  VRRP default protocol version = 2
  Iptables input chain = INPUT
  VRRP check unicast_src = false
  VRRP skip check advert addresses = false
  VRRP strict mode = false
  VRRP process priority = 0
  VRRP don't swap = false
  Checker process priority = 0
  Checker don't swap = false
  Network namespace = (default)
------< SSL definitions >------
  Using autogen SSL context
Using LinkWatch kernel netlink reflector...
Stopped
pid 11308 exited with permanent error CONFIG. Terminating
Stopping
Stopped
Stopped Keepalived v1.2.24 (11/11,2016)

Here is a test run with my change.

# ./setup
Creaating ha-99a3cb02-dc
Creaating qr-bc6c9831-52
Creaating qg-9b2d21c4-59
Starting Keepalived v1.2.24 (11/11,2016)
Opening file '/home/wilder/scratch/test.conf'.
Starting Healthcheck child process, pid=1131124
Starting VRRP child process, pid=1131125
Initializing ipvs
Netlink reflector reports IP 192.168.0.2 added
Netlink reflector reports IP 192.168.0.2 added
Netlink reflector reports IP 9.114.216.147 added
Netlink reflector reports IP 9.114.216.147 added
Netlink reflector reports IP 192.168.0.2 added
Netlink reflector reports IP 192.168.0.2 added
Netlink reflector reports IP 172.17.0.1 added
Netlink reflector reports IP 172.17.0.1 added
Netlink reflector reports IP 172.18.0.1 added
Netlink reflector reports IP 172.18.0.1 added
Netlink reflector reports IP fe80::9abe:94ff:fe0d:f2f4 added
Netlink reflector reports IP fd55:faaf:e1ab:3df:9abe:94ff:fe0d:f2f6 
added
Netlink reflector reports IP fe80::9abe:94ff:fe0d:f2f6 added
Netlink reflector reports IP fe80::42:dbff:fe53:c725 added
Netlink reflector reports IP fe80::9abe:94ff:fe0d:f2f4 added
Netlink reflector reports IP fe80::42:93ff:fe45:7d65 added
Netlink reflector reports IP fd55:faaf:e1ab:3df:9abe:94ff:fe0d:f2f6 
added
Registering Kernel netlink reflector
Netlink reflector reports IP fe80::9abe:94ff:fe0d:f2f6 added
Registering Kernel netlink command channel
Netlink reflector reports IP fe80::42:dbff:fe53:c725 added
Netlink reflector reports IP fe80::42:93ff:fe45:7d65 added
Registering gratuitous ARP shared channel
Registering Kernel netlink reflector
Registering Kernel netlink command channel
Opening file '/home/wilder/scratch/test.conf'.
Opening file '/home/wilder/scratch/test.conf'.
Cannot specify scope for IPv6 addresses (fe80::f816:3eff:fe95:9c41/64) - 
ignoring scope
Cannot specify scope for IPv6 addresses (fe80::f816:3eff:febb:d0ff/64) - 
ignoring scope
(VR_1): Cannot find an IP address to use for interface ha-99a3cb02-dc
------< Global definitions >------
  Router ID = docker
  Default interface = eth0
  LVS flush = false
  VRRP IPv4 mcast group = 224.0.0.18
  VRRP IPv6 mcast group = ff02::12
  Gratuitous ARP delay = 5
  Gratuitous ARP repeat = 5
  Gratuitous ARP refresh timer = 0
  Gratuitous ARP refresh repeat = 1
  Gratuitous ARP lower priority delay = 0
  Gratuitous ARP lower priority repeat = -1
  Send advert after receive lower priority advert = true
  Gratuitous ARP interval = 0
  Gratuitous NA interval = 0
  VRRP default protocol version = 2
  Iptables input chain = INPUT
  VRRP check unicast_src = false
  VRRP skip check advert addresses = false
  VRRP strict mode = false
  VRRP process priority = 0
  VRRP don't swap = false
  Checker process priority = 0
  Checker don't swap = false
  Network namespace = (default)

Regards
David Wilder


------------------------------------------------------------------------------