Re: [PATCH] Increase the size of the netlink recvmsg buffer

On Mon, 2016-11-14 at 11:42 -0800, dwilder wrote:

> On 2016-11-14 03:25, Quentin Armitage wrote:
> > On Sat, 2016-11-12 at 16:23 -0800, dwilder wrote:
> > 
> > On systems with a large number of network interfaces keepalived may
> > report:
> > "Netlink: error: message truncated messages"
> > <and>
> > "VRRP is trying to assign VIP to unknown qr-dc4f0313-ca interface !!!
> > go
> > out
> >  and fix your conf !!!"
> > 
> > This was reported on a system running OpenStack/Nutron.
> > 
> > I found that the netlink recvmsg buffer was too small causing messages
> > 
> > to be
> > truncated. Increasing the buffer from 4K to 8K prevents the problem.
> > (linux/netlink.h suggests using an 8K buffer).
> > 
> > I tested the fix by creating 100 veth interfaces and verifying the
> > errors
> > did not return.
> > 
> > Signed-off-by: David Wilder <[email protected]>
> > 
> > ---
> >  keepalived/vrrp/vrrp_netlink.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/keepalived/vrrp/vrrp_netlink.c
> > b/keepalived/vrrp/vrrp_netlink.c
> > index d7adffa..7897b4c 100644
> > --- a/keepalived/vrrp/vrrp_netlink.c
> > +++ b/keepalived/vrrp/vrrp_netlink.c
> > @@ -273,7 +273,7 @@ netlink_parse_info(int (*filter) (struct
> > sockaddr_nl
> > *, struct nlmsghdr *),
> >  int error;
> > 
> >  while (1) {
> > - char buf[4096];
> > + char buf[8192];
> >  struct iovec iov = { buf, sizeof buf };
> >  struct sockaddr_nl snl;
> >  struct msghdr msg =
> > 
> >  Hi David,
> > 
> >  I'm very happy to apply your patch, but since I have been unable to
> > reproduce the problem, it would be helpful if you could provide some
> > further information.
> > 
> >  1. I haven't been able to see any reference in linux/netlink.h
> > suggesting an 8k buffer (or any other buffer size). Could you please
> > let me know where the reference is and/or send a copy of your
> > linux/netlink.h
> >  2. Could you send a copy of the script you used to create the veth
> > interfaces, so that I can reproduce it here.
> >  3. What version of Linux are you using?
> > 
> >  With many thanks,
> >  Quentin Armitage
> 
> Hi Quentin
> 
> Here is the reference from linux/include/linux/netlink.h (the kernel 
> header not the user header)
> <snip>
> /*
>   *      skb should fit one page. This choice is good for headerless 
> malloc.
>   *      But we should limit to 8K so that userspace does not have to
>   *      use enormous buffer sizes on recvmsg() calls just to avoid
>   *      MSG_TRUNC when PAGE_SIZE is very large.
>   */
> #if PAGE_SIZE < 8192UL
> #define NLMSG_GOODSIZE  SKB_WITH_OVERHEAD(PAGE_SIZE)
> #else
> #define NLMSG_GOODSIZE  SKB_WITH_OVERHEAD(8192UL)
> #endif
> 
> #define NLMSG_DEFAULT_SIZE (NLMSG_GOODSIZE - NLMSG_HDRLEN)
> <snip>
> 
> I forgot to mention that the problem was reported on on ppc64le, it is 
> not reproducible on x86-64.  The page size used on ppc64le is 64K, the 
> page size on x86-64 is only 4k, from the code above I can infer why the 
> problem is not seen with a 4k page size.
> 
> The kernel I am running is: 3.10.0-514.el7.ppc64le
> 
> We have verified the fix on our original openstack setup.
> 

David,

Rather than increase the size of the buf[] array on all systems, would
something like the following approach be reasonable?

int error;

size_t nlmsg_buf_size = getpagesize();
if (nlmsg_buf_size > 8192)
    nlmsg_buf_size = 8192;

while (1) {
    char buf[nlmsg_buf_size];
    struct iovec iov = { buf, sizeof buf };
    struct sockaddr_nl snl;
    struct msghdr msg =
 
If we go down this route, then I would move the setting of
nlmsg_buf_size into some initialisation code.

Does this seem reasonable to you?

With regards,
Quentin Armitage

------------------------------------------------------------------------------

_______________________________________________
Keepalived-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/keepalived-devel