[quagga-users 14539] Re: Serious BGP issue
jan hugo prins <[email protected]>
| Newsgroups | gmane.network.quagga.user |
|---|---|
| Message-ID | <[email protected]> |
I don't think this is a memory issue. At least not system base memory. My router has 32G of memory, can't imagine that I filled that memory. I actually have stats that tell me I don't live under any kind of memory pressure, except when it is something inside the daemon. Jan Hugo On 11/16/2016 09:29 PM, Mike Tancsa wrote: > xx.yy.zz.118 4 11647 513123 433373 0 0 0 13w1d08h 18134 > and then after a soft clear > 19910 > > I dont see this on my other ibgp peers. The problem one has a few dozen > ebgp peers as well as some route maps and filter lists. But then again, > other routers (with fewer ebgp peers) have similar configs. How can I > tell if I am running into memory pressure ? > > >> sh mem > Temporary memory : 3 > String vector : 2 > Vector : 2311 > Vector index : 2311 > Link List : 86 > Link Node : 263 > Thread : 391 > Thread master : 1 > Thread stats : 23 > VTY : 2 > Interface : 10 > Connected : 16 > Buffer : 3 > Buffer data : 1 > Stream : 294 > Stream data : 294 > Stream FIFO : 96 > Prefix : 24 > Hash : 1352 > Hash Bucket : 1133638 > Hash Index : 1352 > Route table : 50 > Route node : 1 > Prefix List : 117 > Prefix List Entry : 5802 > Prefix List Str : 117 > Route map : 6 > Route map name : 6 > Route map index : 9 > Route map rule : 19 > Route map rule str : 19 > Route map compiled : 22 > Command desc : 27898 > Socket union : 240 > Logging : 1 > Zclient : 2 > Work queue : 65 > Work queue name string : 65 > Priority queue : 2 > Priority queue data : 2 > Host config : 3 > VRF : 1 > VRF name : 1 > VRF bit-map : 17 > ----------------------------- > Nexthop : 14 > ----------------------------- > BGP instance : 1 > BGP listen socket details : 2 > BGP peer : 96 > BGP peer hostname : 96 > Peer group : 5 > Peer description : 78 > Peer password string : 13 > BGP attribute : 629265 > BGP extra attributes : 629265 > BGP aspath : 200384 > BGP aspath seg : 200698 > BGP aspath segment data : 200698 > BGP aspath str : 200384 > ----------------------------- > BGP table : 49 > BGP node : 1186260 > BGP route : 2592777 > BGP ancillary route info : 2438838 > BGP connected : 6 > BGP static : 3 > BGP adv attr : 303272 > BGP adv : 1654505 > BGP synchronise : 1344 > BGP adj in : 121250 > BGP adj out : 2272503 > ----------------------------- > BGP AS list : 15 > BGP AS filter : 129 > BGP AS filter str : 129 > ----------------------------- > community : 676 > community val : 676 > community str : 676 > ----------------------------- > extcommunity : 6 > extcommunity val : 6 > extcommunity str : 6 > ----------------------------- > community-list : 2 > community-list name : 2 > community-list entry : 2 > community-list handler : 1 > ----------------------------- > BGP transit attr : 10 > BGP transit val : 10 > ----------------------------- > BGP nexthop : 14 > BGP regexp : 129 > BGP own address : 4 > ----------------------------- > > Temporary memory : 18 > String vector : 2 > Vector : 800 > Vector index : 800 > Link List : 49 > Link Node : 49 > Thread : 10 > Thread master : 1 > Thread stats : 8 > VTY : 2 > Interface : 17 > Connected : 16 > Buffer : 3 > Buffer data : 1 > Stream : 4 > Stream data : 4 > Prefix : 24 > Hash : 1 > Hash Bucket : 8 > Hash Index : 1 > Route table : 26 > Route node : 1186369 > Command desc : 8948 > Logging : 1 > Work queue : 2 > Work queue name string : 1 > Priority queue : 2 > Priority queue data : 2 > Host config : 2 > VRF : 1 > VRF name : 1 > VRF bit-map : 37 > ----------------------------- > ZEBRA VRF : 1 > Nexthop : 645208 > RIB : 645207 > Static route : 21 > RIB destination : 645201 > RIB table info : 4 > ----------------------------- > > On 11/16/2016 9:22 AM, Mike Tancsa wrote: >> The other thing I noticed, is that over time, the problem iBGP peer >> sends less and less routes to its peers. So at start up time, it will >> advertise (x) prefixes to its iBGP peers. Over the course of a day, it >> will go to (x-y) and eventually to almost nothing until I do a soft >> clear out and it will return to (x). But (x) is still a number lower >> than it should be. I have artificially set the local pref to a higher >> value, so it should be passing far more routes to the iBGP peers than >> just (x). >> >> ---Mike >> >> On 11/16/2016 5:00 AM, Paul Thornton wrote: >>> Hi all >>> >>> I also saw something very similar in January: >>> >>> https://lists.quagga.net/pipermail/quagga-users/2016-February/014275.html >>> >>> I've not managed to reproduce this - in lab testing I couldn't make it >>> fail, I could break it on production but that isn't the best place to be >>> troubleshooting and debugging! >>> >>> It seemed to be related to certain prefixes that just weren't advertised >>> between routers within our network; but there wasn't any real rhyme or >>> reason to what these were (comparing attributes etc). >>> >>> (Apologies if this goes twice to the list, I replied first time from the >>> wrong address) >>> >>> Paul. >>> >>> >>> On 15/11/2016 21:40, Mike Tancsa wrote: >>>> This sounds like the problem I saw last Jan. >>>> >>>> https://lists.quagga.net/pipermail/quagga-users/2016-January/014248.html >>>> >>>> >>>> >>>> On 11/3/2016 8:31 PM, jan hugo prins wrote: >>>>> I'm happy to provide you with the complete config of my 3 routers. >>>>> I will send them to you personaly after stripping the passwords. >>>>> >>>>> Jan Hugo >>>>> >>>>> >>>>> On 11/04/2016 12:32 AM, Martin Winter wrote: >>>>>> Jan, >>>>>> >>>>>> can you clarify if there are any route-maps, filter-list, prefix-lists >>>>>> or anything else? >>>>>> Also, are you doing any “neighbor x.x.x.x next-hop-self” or >>>>>> peer groups? >>>>>> >>>>>> (I’m not sure interested in config which would match, but also filters >>>>>> etc on the relevant peers in case they are misbehaving) >>>>>> >>>>>> My testing on filter-lists etc is a bit lacking, so I would assume the >>>>>> bug most likely to be there. >>>>>> >>>>>> - Martin >>>>>> >>>>>> >>>>>> On 3 Nov 2016, at 9:41, jan hugo prins wrote: >>>>>> >>>>>>> Some extra side note: >>>>>>> >>>>>>> A random prefix on the BGP02: >>>>>>> >>>>>>> bgp02.as48972.net# sh ip bgp 23.226.189.0/24 >>>>>>> BGP routing table entry for 23.226.189.0/24 >>>>>>> Paths: (3 available, best #2, table Default-IP-Routing-Table) >>>>>>> Advertised to non peer-group peers: >>>>>>> 95.130.232.4 95.130.232.6 >>>>>>> 5580 5580 1299 4809 38197 >>>>>>> 78.152.40.92 (metric 11) from 95.130.232.6 (95.130.232.6) >>>>>>> Origin IGP, metric 0, localpref 90, valid, internal >>>>>>> Last update: Thu Nov 3 16:09:45 2016 >>>>>>> >>>>>>> 16243 47886 3356 4809 38197 >>>>>>> 77.222.66.177 from 77.222.66.177 (87.249.109.240) >>>>>>> Origin IGP, localpref 90, valid, external, best >>>>>>> Last update: Thu Oct 27 00:56:04 2016 >>>>>>> >>>>>>> 5580 5580 1299 4809 38197 >>>>>>> 78.152.49.109 from 78.152.49.109 (80.94.64.124) >>>>>>> Origin IGP, metric 0, localpref 90, valid, external >>>>>>> Community: 1299:30000 5580:21220 >>>>>>> Last update: Mon Oct 31 17:51:58 2016 >>>>>>> >>>>>>> Is available on the BGP03: >>>>>>> >>>>>>> bgp03.as48972.net# sh ip bgp 23.226.189.0/24 >>>>>>> BGP routing table entry for 23.226.189.0/24 >>>>>>> Paths: (2 available, best #2, table Default-IP-Routing-Table) >>>>>>> Advertised to non peer-group peers: >>>>>>> 95.130.232.3 95.130.232.4 >>>>>>> 16243 47886 3356 4809 38197 >>>>>>> 77.222.66.181 from 77.222.66.181 (87.249.109.241) >>>>>>> Origin IGP, localpref 90, valid, external >>>>>>> Last update: Thu Nov 3 16:09:38 2016 >>>>>>> >>>>>>> 5580 5580 1299 4809 38197 >>>>>>> 78.152.40.92 from 78.152.40.92 (80.94.64.124) >>>>>>> Origin IGP, metric 0, localpref 90, valid, external, best >>>>>>> Community: 1299:30000 5580:21220 >>>>>>> Last update: Thu Nov 3 16:09:21 2016 >>>>>>> >>>>>>> But even though the BGP02 claims to have it advertised to both the >>>>>>> bgp03 >>>>>>> and the routeserver / lookingglass: >>>>>>> >>>>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.4 adv |grep 23.226.189.0/24 >>>>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.6 adv |grep 23.226.189.0/24 >>>>>>> >>>>>>> It did not. >>>>>>> >>>>>>> Jan Hugo Prins >>>>>>> >>>>>>> On 11/03/2016 05:08 PM, jan hugo prins wrote: >>>>>>>> As a sidenote, a second router that runs my LookingGlass and is >>>>>>>> used to >>>>>>>> inject blackhole routes, is also not receiving the full table from >>>>>>>> the >>>>>>>> BGP02. >>>>>>>> This can be observed at http://lg.as48972.net/ >>>>>>>> >>>>>>>> Jan Hugo Prins >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 11/03/2016 04:36 PM, jan hugo prins wrote: >>>>>>>>> Hi everyone, >>>>>>>>> >>>>>>>>> I have a strange issue. >>>>>>>>> I have 2 external BGP routers. They both have 2 uplinks to my >>>>>>>>> transit >>>>>>>>> providers and this works fine. >>>>>>>>> Until a while ago, when I wanted to take one router out for >>>>>>>>> maintenance, >>>>>>>>> I would shut it's transit links, remove "default-information >>>>>>>>> originate >>>>>>>>> always" from OSPF, set higher costs on the uplink and / or set >>>>>>>>> "max-metric router-lsa administrative" wait for most of the >>>>>>>>> traffic to >>>>>>>>> shift to the second router and do my maintenance. >>>>>>>>> >>>>>>>>> Because the router would always have an iBGP link to the second >>>>>>>>> router >>>>>>>>> it would receive all routes from this router and be fine in the >>>>>>>>> case of >>>>>>>>> some traffic coming in anyway. But today I wanted to upgrade >>>>>>>>> Quagga to >>>>>>>>> version 1.10 and I saw something really strange: >>>>>>>>> >>>>>>>>> bgp02.as48972.net# sh ip bgp sum >>>>>>>>> BGP router identifier 95.130.232.3, local AS number 48972 >>>>>>>>> RIB entries 1120940, using 120 MiB of memory >>>>>>>>> Peers 29, using 129 KiB of memory >>>>>>>>> Peer groups 3, using 96 bytes of memory >>>>>>>>> >>>>>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ >>>>>>>>> Up/Down >>>>>>>>> State/P >>>>>>>>> fxRcd >>>>>>>>> 77.222.66.177 4 16243 1434500 11053 0 0 0 >>>>>>>>> 01w0d16h 609362 >>>>>>>>> 78.152.49.109 4 5580 3162545 66219 0 0 0 >>>>>>>>> 01w0d16h 609733 >>>>>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never >>>>>>>>> Idle >>>>>>>>> (Admin) >>>>>>>>> 95.130.232.4 4 48972 10983 58016 0 0 0 >>>>>>>>> 02:28:03 0 >>>>>>>>> 95.130.232.6 4 48972 2536718 65789 0 0 0 >>>>>>>>> 02:21:53 137022 >>>>>>>>> 193.108.98.2 4 20828 208151 11056 0 0 0 >>>>>>>>> 2d09h49m 137235 >>>>>>>>> 193.108.98.3 4 20828 204274 11056 0 0 0 >>>>>>>>> 2d09h39m 137227 >>>>>>>>> >>>>>>>>> bgp03.as48972.net(config-router)# do sh ip bgp sum >>>>>>>>> BGP router identifier 95.130.232.6, local AS number 48972 >>>>>>>>> RIB entries 311015, using 33 MiB of memory >>>>>>>>> Peers 21, using 94 KiB of memory >>>>>>>>> Peer groups 3, using 96 bytes of memory >>>>>>>>> >>>>>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ >>>>>>>>> Up/Down >>>>>>>>> State/P >>>>>>>>> fxRcd >>>>>>>>> 77.222.66.181 4 16243 298044 882 0 0 0 00:01:21 >>>>>>>>> Idle >>>>>>>>> (Admin) >>>>>>>>> 78.152.40.92 4 5580 559289 5234 0 0 0 00:01:14 >>>>>>>>> Idle >>>>>>>>> (Admin) >>>>>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never >>>>>>>>> Idle >>>>>>>>> (Admin) >>>>>>>>> 95.130.232.3 4 48972 7643 867546 0 0 0 >>>>>>>>> 02:21:10 39002 >>>>>>>>> 95.130.232.4 4 48972 837 558695 0 0 0 >>>>>>>>> 02:27:21 0 >>>>>>>>> 193.108.98.2 4 20828 68276 883 0 0 0 >>>>>>>>> 03:30:44 137235 >>>>>>>>> 193.108.98.3 4 20828 64031 883 0 0 0 >>>>>>>>> 03:30:35 137227 >>>>>>>>> >>>>>>>>> I would expect that the BGP03 would get at least 580.000 routes >>>>>>>>> from the >>>>>>>>> BGP02, but this doesn't happen for some strange reason. >>>>>>>>> Also resetting the iBGP link between the BGP02 and BGP03 does not >>>>>>>>> fix >>>>>>>>> the problem. >>>>>>>>> >>>>>>>>> Anyone any idea? >>>>>>>>> >>>>>>>>> jan Hugo Prins >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Quagga-users mailing list >>>>>>>>> [email protected] >>>>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users >>>>>>>> _______________________________________________ >>>>>>>> Quagga-users mailing list >>>>>>>> [email protected] >>>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users >>>>>>> _______________________________________________ >>>>>>> Quagga-users mailing list >>>>>>> [email protected] >>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users >>>>>> _______________________________________________ >>>>>> Quagga-users mailing list >>>>>> [email protected] >>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users >>>>> _______________________________________________ >>>>> Quagga-users mailing list >>>>> [email protected] >>>>> https://lists.quagga.net/mailman/listinfo/quagga-users >>>>> >>>>> >>> >>> _______________________________________________ >>> Quagga-users mailing list >>> [email protected] >>> https://lists.quagga.net/mailman/listinfo/quagga-users >>> >>> >> > _______________________________________________ Quagga-users mailing list [email protected] https://lists.quagga.net/mailman/listinfo/quagga-users