Re: [PATCH] Increase the size of the netlink recvmsg buffer

On 2016-11-14 15:37, Quentin Armitage wrote:
> On Mon, 2016-11-14 at 11:42 -0800, dwilder wrote:
> 
> On 2016-11-14 03:25, Quentin Armitage wrote:
>> On Sat, 2016-11-12 at 16:23 -0800, dwilder wrote:
>> 
>> On systems with a large number of network interfaces keepalived may
>> report:
>> "Netlink: error: message truncated messages"
>> <and>
>> "VRRP is trying to assign VIP to unknown qr-dc4f0313-ca interface
> !!!
>> go
>> out
>> and fix your conf !!!"
>> 
>> This was reported on a system running OpenStack/Nutron.
>> 
>> I found that the netlink recvmsg buffer was too small causing
> messages
>> 
>> to be
>> truncated. Increasing the buffer from 4K to 8K prevents the problem.
>> (linux/netlink.h suggests using an 8K buffer).
>> 
>> I tested the fix by creating 100 veth interfaces and verifying the
>> errors
>> did not return.
>> 
>> Signed-off-by: David Wilder <[email protected]>
>> 
>> ---
>> keepalived/vrrp/vrrp_netlink.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>> 
>> diff --git a/keepalived/vrrp/vrrp_netlink.c
>> b/keepalived/vrrp/vrrp_netlink.c
>> index d7adffa..7897b4c 100644
>> --- a/keepalived/vrrp/vrrp_netlink.c
>> +++ b/keepalived/vrrp/vrrp_netlink.c
>> @@ -273,7 +273,7 @@ netlink_parse_info(int (*filter) (struct
>> sockaddr_nl
>> *, struct nlmsghdr *),
>> int error;
>> 
>> while (1) {
>> - char buf[4096];
>> + char buf[8192];
>> struct iovec iov = { buf, sizeof buf };
>> struct sockaddr_nl snl;
>> struct msghdr msg =
>> 
>> Hi David,
>> 
>> I'm very happy to apply your patch, but since I have been unable to
>> reproduce the problem, it would be helpful if you could provide some
>> further information.
>> 
>> 1. I haven't been able to see any reference in linux/netlink.h
>> suggesting an 8k buffer (or any other buffer size). Could you please
>> let me know where the reference is and/or send a copy of your
>> linux/netlink.h
>> 2. Could you send a copy of the script you used to create the veth
>> interfaces, so that I can reproduce it here.
>> 3. What version of Linux are you using?
>> 
>> With many thanks,
>> Quentin Armitage
> 
> Hi Quentin
> 
> Here is the reference from linux/include/linux/netlink.h (the kernel
> header not the user header)
> <snip>
> /*
>  * skb should fit one page. This choice is good for headerless
> malloc.
>  * But we should limit to 8K so that userspace does not have to
>  * use enormous buffer sizes on recvmsg() calls just to avoid
>  * MSG_TRUNC when PAGE_SIZE is very large.
>  */
> #if PAGE_SIZE < 8192UL
> #define NLMSG_GOODSIZE SKB_WITH_OVERHEAD(PAGE_SIZE)
> #else
> #define NLMSG_GOODSIZE SKB_WITH_OVERHEAD(8192UL)
> #endif
> 
> #define NLMSG_DEFAULT_SIZE (NLMSG_GOODSIZE - NLMSG_HDRLEN)
> <snip>
> 
> I forgot to mention that the problem was reported on on ppc64le, it is
> 
> not reproducible on x86-64. The page size used on ppc64le is 64K, the
> page size on x86-64 is only 4k, from the code above I can infer why
> the
> problem is not seen with a 4k page size.
> 
> The kernel I am running is: 3.10.0-514.el7.ppc64le
> 
> We have verified the fix on our original openstack setup.
> 
>  David,
> 
>  Rather than increase the size of the buf[] array on all systems,
> would something like the following approach be reasonable?
> 
>  int error;
> 
>  size_t nlmsg_buf_size = getpagesize();
>  if (nlmsg_buf_size > 8192)
>  nlmsg_buf_size = 8192;
> 
>  while (1) {
>  char buf[nlmsg_buf_size];
>  struct iovec iov = { buf, sizeof buf };
>  struct sockaddr_nl snl;
>  struct msghdr msg =
> 
>  If we go down this route, then I would move the setting of
> nlmsg_buf_size into some initialisation code.
> 
>  Does this seem reasonable to you?
> 
>  With regards,
>  Quentin Armitage

Yup,  I am ok with it. Do you want a new patch?


------------------------------------------------------------------------------