[quagga-users 14963] Re: Duplication of kernel routes
Stefan Bethke <[email protected]> Thu, 8 Mar 2018 13:08:53 +0100
| Newsgroups | gmane.network.quagga.user |
|---|---|
| Message-ID | <[email protected]> |
--===============3549367592191308694== Content-Type: multipart/alternative; boundary="Apple-Mail=_685AC509-AB14-48CB-9AA0-65ACE5EEFE85" --Apple-Mail=_685AC509-AB14-48CB-9AA0-65ACE5EEFE85 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 It appears the issue was triggered by configuring a GUA on the router's = interface, and pointing the default route of the hosts in the VLAN to = that GUA. Using RAs and having the default route point at the LLA of the router's = interface stopped the weird routing table entries. > Am 16.01.2018 um 19:48 schrieb Stefan Bethke <[email protected]>: >=20 > Hello, >=20 > I=E2=80=99m running into a peculiar problem I=E2=80=99m hoping someone = can give me some pointers for further diagnostics, or even ways to = mitigate. >=20 > I=E2=80=99m running quagga-1.1.1-68.1 on SLES11SP4, = 3.0.101-108.21-default #1 SMP Thu Jan 11 12:58:47 CET 2018 x86_64, with = current patches. zebra, ospfd, ospf6d and bpd are running. >=20 > I can reproduce the issue on a VSphere VM with 3 vmxnet interfaces; = two of which are bonded as a active/backup pair. On the production = machine, it=E2=80=99s two bonded interfaces with LACP. The production = setup is a pair of machines with keepalived, where VLANs are split = between both nodes (odd/even VLAN numbers). >=20 > The router receives a full table. > ip -4 route show | wc -l: 669651 > ip -6 route show | wc -l: 43739 >=20 > There=E2=80=99s quite a number of customer machines connected to = various VLANs; only a single VLAN is producing this effect. >=20 > The problem presents like this: whenever one of the customer FreeBSD = 11.1 machines has an active IPv6 configuration, the number of IPv6 = kernel routes in the router balloons. Every about 30 seconds, the number = of routes increases by one full table size (so adding about 45,000); at = some point, the entire IPv6 table gets reset and the process starts = over. IPv4 appears to be completely unaffected. Other customers machines = with FreeBSD do not exhibit this problem, nor do any other Ones. >=20 > The FreeBSD configuration appears to be standard: NDP and auto link = local are active and working seemingly OK; no routing protocol is = active. With NUD active, there are about 5-10 NDP packets per minute, = all confirming reachability with solicited requests. With NUD off, the = frequency drops to about 2 NDP packets per minute. The customer VLAN has = about 4 MACs active, two of which are locally administered. > # ifconfig bridge0 inet6 > bridge0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 = mtu 1500 > description: vm-bridge0 > inet6 fe80::3c:9fff:fe37:2500%bridge0/64 scopeid 0x7 > inet6 2a00:14b0:4200:xyza::1e1/64 > inet6 2a00:14b0:4200:xyza::1e2/128 > inet6 2a00:14b0:4200:xyza::1fd/128 > inet6 2a00:14b0:4200:xyza::1e5/128 > nd6 options=3D8020<AUTO_LINKLOCAL,DEFAULTIF> > # ndp -i bridge0 > linkmtu=3D0, maxmtu=3D0, curhlim=3D64, basereachable=3D30s0ms, = reachable=3D43s, retrans=3D1s0ms > Flags: auto_linklocal >=20 > The effect stops within minutes once both machines=E2=80=99 IPv6 = interfaces are disabled. >=20 > tcpdump/wireshark traces don=E2=80=99t seem to be showing anything out = of the ordinary. Traffic on the VLAN is very moderate. >=20 > What=E2=80=99s puzzling to me is that there are lots of identical = duplicate routes installed in the kernel: > # ip -6 route show | grep 2a00:14b0:4200:xyza > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > 2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel metric 256 > ^C >=20 > I=E2=80=99m at a loss how and why these duplicate routes should be = installed. I would appreciate any hints on how to diagnose this = further. >=20 > The following versions of quagga were tried, without any change in = behaviour: > # 2012-10-02 17:21:35 quagga-0.99.15-0.10.1.x86_64.rpm installed ok > # 2013-01-14 18:55:55 quagga-0.99.15-0.12.1.x86_64.rpm installed ok > # 2013-09-27 18:27:25 quagga-0.99.15-0.14.1.x86_64.rpm installed ok > # 2015-08-26 16:40:56 quagga-0.99.15-0.14.11.x86_64.rpm installed ok > # 2016-04-06 15:40:30 quagga-0.99.15-0.21.1.x86_64.rpm installed ok > # 2016-06-08 10:45:33 quagga-0.99.15-0.24.2.x86_64.rpm installed ok > # 2016-10-26 14:55:44 quagga-0.99.15-0.29.1.x86_64.rpm installed ok > # 2017-04-18 15:48:51 quagga-1.1.1-65.1.x86_64.rpm installed ok > # 2017-12-12 16:41:30 quagga-1.1.1-68.1.x86_64.rpm installed ok >=20 >=20 > Thanks, > Stefan >=20 > -- > Stefan Bethke <[email protected]> Fon +49 151 14070811 >=20 >=20 > _______________________________________________ > Quagga-users mailing list > Quagga-users-UOy77sIEA+cAd7ICUelF/[email protected] > https://lists.quagga.net/mailman/listinfo/quagga-users --=20 Stefan Bethke <[email protected]> Fon +49 151 14070811 --Apple-Mail=_685AC509-AB14-48CB-9AA0-65ACE5EEFE85 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D"">It appears the issue was triggered by configuring a GUA on = the router's interface, and pointing the default route of the hosts in = the VLAN to that GUA.<div class=3D""><br class=3D""></div><div = class=3D"">Using RAs and having the default route point at the LLA of = the router's interface stopped the weird routing table = entries.</div><div class=3D""><br class=3D""><div><blockquote = type=3D"cite" class=3D""><div class=3D"">Am 16.01.2018 um 19:48 schrieb = Stefan Bethke <<a href=3D"mailto:[email protected]" = class=3D"">[email protected]</a>>:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><div = class=3D"">Hello,<br class=3D""><br class=3D"">I=E2=80=99m running into = a peculiar problem I=E2=80=99m hoping someone can give me some pointers = for further diagnostics, or even ways to mitigate.<br class=3D""><br = class=3D"">I=E2=80=99m running quagga-1.1.1-68.1 on SLES11SP4, = 3.0.101-108.21-default #1 SMP Thu Jan 11 12:58:47 CET 2018 x86_64, with = current patches. zebra, ospfd, ospf6d and bpd are running.<br = class=3D""><br class=3D"">I can reproduce the issue on a VSphere VM with = 3 vmxnet interfaces; two of which are bonded as a active/backup pair. On = the production machine, it=E2=80=99s two bonded interfaces with LACP. = The production setup is a pair of machines with keepalived, where VLANs = are split between both nodes (odd/even VLAN numbers).<br class=3D""><br = class=3D"">The router receives a full table.<br class=3D"">ip -4 route = show | wc -l: 669651<br class=3D"">ip -6 route show | wc -l: 43739<br = class=3D""><br class=3D"">There=E2=80=99s quite a number of customer = machines connected to various VLANs; only a single VLAN is producing = this effect.<br class=3D""><br class=3D"">The problem presents like = this: whenever one of the customer FreeBSD 11.1 machines has an active = IPv6 configuration, the number of IPv6 kernel routes in the router = balloons. Every about 30 seconds, the number of routes increases by one = full table size (so adding about 45,000); at some point, the entire IPv6 = table gets reset and the process starts over. IPv4 appears to be = completely unaffected. Other customers machines with FreeBSD do not = exhibit this problem, nor do any other Ones.<br class=3D""><br = class=3D"">The FreeBSD configuration appears to be standard: NDP and = auto link local are active and working seemingly OK; no routing protocol = is active. With NUD active, there are about 5-10 NDP packets per minute, = all confirming reachability with solicited requests. With NUD off, the = frequency drops to about 2 NDP packets per minute. The customer VLAN has = about 4 MACs active, two of which are locally administered.<br = class=3D""># ifconfig bridge0 inet6<br class=3D"">bridge0: = flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu = 1500<br class=3D""><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>description: vm-bridge0<br = class=3D""><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span>inet6 fe80::3c:9fff:fe37:2500%bridge0/64 scopeid 0x7<br = class=3D""><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span>inet6 2a00:14b0:4200:xyza::1e1/64<br class=3D""><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>inet6 = 2a00:14b0:4200:xyza::1e2/128<br class=3D""><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>inet6 = 2a00:14b0:4200:xyza::1fd/128<br class=3D""><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>inet6 = 2a00:14b0:4200:xyza::1e5/128<br class=3D""><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>nd6 = options=3D8020<AUTO_LINKLOCAL,DEFAULTIF><br class=3D""># ndp -i = bridge0<br class=3D"">linkmtu=3D0, maxmtu=3D0, curhlim=3D64, = basereachable=3D30s0ms, reachable=3D43s, retrans=3D1s0ms<br = class=3D"">Flags: auto_linklocal<br class=3D""><br class=3D"">The effect = stops within minutes once both machines=E2=80=99 IPv6 interfaces are = disabled.<br class=3D""><br class=3D"">tcpdump/wireshark traces don=E2=80=99= t seem to be showing anything out of the ordinary. Traffic on the VLAN = is very moderate.<br class=3D""><br class=3D"">What=E2=80=99s puzzling = to me is that there are lots of identical duplicate routes installed in = the kernel:<br class=3D""># ip -6 route show | grep = 2a00:14b0:4200:xyza<br class=3D"">2a00:14b0:4200:xyza::/64 dev vlan503 = proto kernel metric 256<br class=3D"">2a00:14b0:4200:xyza::/64= dev vlan503 proto kernel metric 256<br = class=3D"">2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel = metric 256<br class=3D"">2a00:14b0:4200:xyza::/64 dev vlan503 = proto kernel metric 256<br class=3D"">2a00:14b0:4200:xyza::/64= dev vlan503 proto kernel metric 256<br = class=3D"">2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel = metric 256<br class=3D"">2a00:14b0:4200:xyza::/64 dev vlan503 = proto kernel metric 256<br class=3D"">2a00:14b0:4200:xyza::/64= dev vlan503 proto kernel metric 256<br = class=3D"">2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel = metric 256<br class=3D"">2a00:14b0:4200:xyza::/64 dev vlan503 = proto kernel metric 256<br class=3D"">2a00:14b0:4200:xyza::/64= dev vlan503 proto kernel metric 256<br = class=3D"">2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel = metric 256<br class=3D"">2a00:14b0:4200:xyza::/64 dev vlan503 = proto kernel metric 256<br class=3D"">2a00:14b0:4200:xyza::/64= dev vlan503 proto kernel metric 256<br = class=3D"">2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel = metric 256<br class=3D"">2a00:14b0:4200:xyza::/64 dev vlan503 = proto kernel metric 256<br class=3D"">2a00:14b0:4200:xyza::/64= dev vlan503 proto kernel metric 256<br = class=3D"">2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel = metric 256<br class=3D"">2a00:14b0:4200:xyza::/64 dev vlan503 = proto kernel metric 256<br class=3D"">2a00:14b0:4200:xyza::/64= dev vlan503 proto kernel metric 256<br = class=3D"">2a00:14b0:4200:xyza::/64 dev vlan503 proto kernel = metric 256<br class=3D"">^C<br class=3D""><br class=3D"">I=E2=80=99m= at a loss how and why these duplicate routes should be installed. = I would appreciate any hints on how to diagnose this further.<br = class=3D""><br class=3D"">The following versions of quagga were tried, = without any change in behaviour:<br class=3D""># 2012-10-02 17:21:35 = quagga-0.99.15-0.10.1.x86_64.rpm installed ok<br class=3D""># 2013-01-14 = 18:55:55 quagga-0.99.15-0.12.1.x86_64.rpm installed ok<br class=3D""># = 2013-09-27 18:27:25 quagga-0.99.15-0.14.1.x86_64.rpm installed ok<br = class=3D""># 2015-08-26 16:40:56 quagga-0.99.15-0.14.11.x86_64.rpm = installed ok<br class=3D""># 2016-04-06 15:40:30 = quagga-0.99.15-0.21.1.x86_64.rpm installed ok<br class=3D""># 2016-06-08 = 10:45:33 quagga-0.99.15-0.24.2.x86_64.rpm installed ok<br class=3D""># = 2016-10-26 14:55:44 quagga-0.99.15-0.29.1.x86_64.rpm installed ok<br = class=3D""># 2017-04-18 15:48:51 quagga-1.1.1-65.1.x86_64.rpm installed = ok<br class=3D""># 2017-12-12 16:41:30 quagga-1.1.1-68.1.x86_64.rpm = installed ok<br class=3D""><br class=3D""><br class=3D"">Thanks,<br = class=3D"">Stefan<br class=3D""><br class=3D"">--<br class=3D"">Stefan = Bethke <<a href=3D"mailto:[email protected]" = class=3D"">[email protected]</a>> Fon +49 151 14070811<br = class=3D""><br class=3D""><br = class=3D"">_______________________________________________<br = class=3D"">Quagga-users mailing list<br class=3D""><a = href=3D"mailto:Quagga-users-UOy77sIEA+cAd7ICUelF/[email protected]" = class=3D"">Quagga-users-UOy77sIEA+cAd7ICUelF/[email protected]</a><br = class=3D"">https://lists.quagga.net/mailman/listinfo/quagga-users<br = class=3D""></div></div></blockquote></div><br class=3D""><div class=3D""> <div style=3D"color: rgb(0, 0, 0); font-family: 'Andale Mono'; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px;">-- <br class=3D"">Stefan Bethke = <<a href=3D"mailto:[email protected]" class=3D"">[email protected]</a>> = Fon +49 151 14070811<br class=3D""><br class=3D""></div> </div> <br class=3D""></div></body></html>= --Apple-Mail=_685AC509-AB14-48CB-9AA0-65ACE5EEFE85-- --===============3549367592191308694== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Quagga-users mailing list Quagga-users-UOy77sIEA+cAd7ICUelF/[email protected] https://lists.quagga.net/mailman/listinfo/quagga-users --===============3549367592191308694==--