[quagga-users 14534] Re: Serious BGP issue
Mike Tancsa <[email protected]>
| Newsgroups | gmane.network.quagga.user |
|---|---|
| Organization | Sentex Communications |
| Message-ID | <[email protected]> |
xx.yy.zz.118 4 11647 513123 433373 0 0 0 13w1d08h 18134 and then after a soft clear 19910 I dont see this on my other ibgp peers. The problem one has a few dozen ebgp peers as well as some route maps and filter lists. But then again, other routers (with fewer ebgp peers) have similar configs. How can I tell if I am running into memory pressure ? > sh mem Temporary memory : 3 String vector : 2 Vector : 2311 Vector index : 2311 Link List : 86 Link Node : 263 Thread : 391 Thread master : 1 Thread stats : 23 VTY : 2 Interface : 10 Connected : 16 Buffer : 3 Buffer data : 1 Stream : 294 Stream data : 294 Stream FIFO : 96 Prefix : 24 Hash : 1352 Hash Bucket : 1133638 Hash Index : 1352 Route table : 50 Route node : 1 Prefix List : 117 Prefix List Entry : 5802 Prefix List Str : 117 Route map : 6 Route map name : 6 Route map index : 9 Route map rule : 19 Route map rule str : 19 Route map compiled : 22 Command desc : 27898 Socket union : 240 Logging : 1 Zclient : 2 Work queue : 65 Work queue name string : 65 Priority queue : 2 Priority queue data : 2 Host config : 3 VRF : 1 VRF name : 1 VRF bit-map : 17 ----------------------------- Nexthop : 14 ----------------------------- BGP instance : 1 BGP listen socket details : 2 BGP peer : 96 BGP peer hostname : 96 Peer group : 5 Peer description : 78 Peer password string : 13 BGP attribute : 629265 BGP extra attributes : 629265 BGP aspath : 200384 BGP aspath seg : 200698 BGP aspath segment data : 200698 BGP aspath str : 200384 ----------------------------- BGP table : 49 BGP node : 1186260 BGP route : 2592777 BGP ancillary route info : 2438838 BGP connected : 6 BGP static : 3 BGP adv attr : 303272 BGP adv : 1654505 BGP synchronise : 1344 BGP adj in : 121250 BGP adj out : 2272503 ----------------------------- BGP AS list : 15 BGP AS filter : 129 BGP AS filter str : 129 ----------------------------- community : 676 community val : 676 community str : 676 ----------------------------- extcommunity : 6 extcommunity val : 6 extcommunity str : 6 ----------------------------- community-list : 2 community-list name : 2 community-list entry : 2 community-list handler : 1 ----------------------------- BGP transit attr : 10 BGP transit val : 10 ----------------------------- BGP nexthop : 14 BGP regexp : 129 BGP own address : 4 ----------------------------- Temporary memory : 18 String vector : 2 Vector : 800 Vector index : 800 Link List : 49 Link Node : 49 Thread : 10 Thread master : 1 Thread stats : 8 VTY : 2 Interface : 17 Connected : 16 Buffer : 3 Buffer data : 1 Stream : 4 Stream data : 4 Prefix : 24 Hash : 1 Hash Bucket : 8 Hash Index : 1 Route table : 26 Route node : 1186369 Command desc : 8948 Logging : 1 Work queue : 2 Work queue name string : 1 Priority queue : 2 Priority queue data : 2 Host config : 2 VRF : 1 VRF name : 1 VRF bit-map : 37 ----------------------------- ZEBRA VRF : 1 Nexthop : 645208 RIB : 645207 Static route : 21 RIB destination : 645201 RIB table info : 4 ----------------------------- On 11/16/2016 9:22 AM, Mike Tancsa wrote: > The other thing I noticed, is that over time, the problem iBGP peer > sends less and less routes to its peers. So at start up time, it will > advertise (x) prefixes to its iBGP peers. Over the course of a day, it > will go to (x-y) and eventually to almost nothing until I do a soft > clear out and it will return to (x). But (x) is still a number lower > than it should be. I have artificially set the local pref to a higher > value, so it should be passing far more routes to the iBGP peers than > just (x). > > ---Mike > > On 11/16/2016 5:00 AM, Paul Thornton wrote: >> Hi all >> >> I also saw something very similar in January: >> >> https://lists.quagga.net/pipermail/quagga-users/2016-February/014275.html >> >> I've not managed to reproduce this - in lab testing I couldn't make it >> fail, I could break it on production but that isn't the best place to be >> troubleshooting and debugging! >> >> It seemed to be related to certain prefixes that just weren't advertised >> between routers within our network; but there wasn't any real rhyme or >> reason to what these were (comparing attributes etc). >> >> (Apologies if this goes twice to the list, I replied first time from the >> wrong address) >> >> Paul. >> >> >> On 15/11/2016 21:40, Mike Tancsa wrote: >>> This sounds like the problem I saw last Jan. >>> >>> https://lists.quagga.net/pipermail/quagga-users/2016-January/014248.html >>> >>> >>> >>> On 11/3/2016 8:31 PM, jan hugo prins wrote: >>>> I'm happy to provide you with the complete config of my 3 routers. >>>> I will send them to you personaly after stripping the passwords. >>>> >>>> Jan Hugo >>>> >>>> >>>> On 11/04/2016 12:32 AM, Martin Winter wrote: >>>>> Jan, >>>>> >>>>> can you clarify if there are any route-maps, filter-list, prefix-lists >>>>> or anything else? >>>>> Also, are you doing any “neighbor x.x.x.x next-hop-self” or >>>>> peer groups? >>>>> >>>>> (I’m not sure interested in config which would match, but also filters >>>>> etc on the relevant peers in case they are misbehaving) >>>>> >>>>> My testing on filter-lists etc is a bit lacking, so I would assume the >>>>> bug most likely to be there. >>>>> >>>>> - Martin >>>>> >>>>> >>>>> On 3 Nov 2016, at 9:41, jan hugo prins wrote: >>>>> >>>>>> Some extra side note: >>>>>> >>>>>> A random prefix on the BGP02: >>>>>> >>>>>> bgp02.as48972.net# sh ip bgp 23.226.189.0/24 >>>>>> BGP routing table entry for 23.226.189.0/24 >>>>>> Paths: (3 available, best #2, table Default-IP-Routing-Table) >>>>>> Advertised to non peer-group peers: >>>>>> 95.130.232.4 95.130.232.6 >>>>>> 5580 5580 1299 4809 38197 >>>>>> 78.152.40.92 (metric 11) from 95.130.232.6 (95.130.232.6) >>>>>> Origin IGP, metric 0, localpref 90, valid, internal >>>>>> Last update: Thu Nov 3 16:09:45 2016 >>>>>> >>>>>> 16243 47886 3356 4809 38197 >>>>>> 77.222.66.177 from 77.222.66.177 (87.249.109.240) >>>>>> Origin IGP, localpref 90, valid, external, best >>>>>> Last update: Thu Oct 27 00:56:04 2016 >>>>>> >>>>>> 5580 5580 1299 4809 38197 >>>>>> 78.152.49.109 from 78.152.49.109 (80.94.64.124) >>>>>> Origin IGP, metric 0, localpref 90, valid, external >>>>>> Community: 1299:30000 5580:21220 >>>>>> Last update: Mon Oct 31 17:51:58 2016 >>>>>> >>>>>> Is available on the BGP03: >>>>>> >>>>>> bgp03.as48972.net# sh ip bgp 23.226.189.0/24 >>>>>> BGP routing table entry for 23.226.189.0/24 >>>>>> Paths: (2 available, best #2, table Default-IP-Routing-Table) >>>>>> Advertised to non peer-group peers: >>>>>> 95.130.232.3 95.130.232.4 >>>>>> 16243 47886 3356 4809 38197 >>>>>> 77.222.66.181 from 77.222.66.181 (87.249.109.241) >>>>>> Origin IGP, localpref 90, valid, external >>>>>> Last update: Thu Nov 3 16:09:38 2016 >>>>>> >>>>>> 5580 5580 1299 4809 38197 >>>>>> 78.152.40.92 from 78.152.40.92 (80.94.64.124) >>>>>> Origin IGP, metric 0, localpref 90, valid, external, best >>>>>> Community: 1299:30000 5580:21220 >>>>>> Last update: Thu Nov 3 16:09:21 2016 >>>>>> >>>>>> But even though the BGP02 claims to have it advertised to both the >>>>>> bgp03 >>>>>> and the routeserver / lookingglass: >>>>>> >>>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.4 adv |grep 23.226.189.0/24 >>>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.6 adv |grep 23.226.189.0/24 >>>>>> >>>>>> It did not. >>>>>> >>>>>> Jan Hugo Prins >>>>>> >>>>>> On 11/03/2016 05:08 PM, jan hugo prins wrote: >>>>>>> As a sidenote, a second router that runs my LookingGlass and is >>>>>>> used to >>>>>>> inject blackhole routes, is also not receiving the full table from >>>>>>> the >>>>>>> BGP02. >>>>>>> This can be observed at http://lg.as48972.net/ >>>>>>> >>>>>>> Jan Hugo Prins >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 11/03/2016 04:36 PM, jan hugo prins wrote: >>>>>>>> Hi everyone, >>>>>>>> >>>>>>>> I have a strange issue. >>>>>>>> I have 2 external BGP routers. They both have 2 uplinks to my >>>>>>>> transit >>>>>>>> providers and this works fine. >>>>>>>> Until a while ago, when I wanted to take one router out for >>>>>>>> maintenance, >>>>>>>> I would shut it's transit links, remove "default-information >>>>>>>> originate >>>>>>>> always" from OSPF, set higher costs on the uplink and / or set >>>>>>>> "max-metric router-lsa administrative" wait for most of the >>>>>>>> traffic to >>>>>>>> shift to the second router and do my maintenance. >>>>>>>> >>>>>>>> Because the router would always have an iBGP link to the second >>>>>>>> router >>>>>>>> it would receive all routes from this router and be fine in the >>>>>>>> case of >>>>>>>> some traffic coming in anyway. But today I wanted to upgrade >>>>>>>> Quagga to >>>>>>>> version 1.10 and I saw something really strange: >>>>>>>> >>>>>>>> bgp02.as48972.net# sh ip bgp sum >>>>>>>> BGP router identifier 95.130.232.3, local AS number 48972 >>>>>>>> RIB entries 1120940, using 120 MiB of memory >>>>>>>> Peers 29, using 129 KiB of memory >>>>>>>> Peer groups 3, using 96 bytes of memory >>>>>>>> >>>>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ >>>>>>>> Up/Down >>>>>>>> State/P >>>>>>>> fxRcd >>>>>>>> 77.222.66.177 4 16243 1434500 11053 0 0 0 >>>>>>>> 01w0d16h 609362 >>>>>>>> 78.152.49.109 4 5580 3162545 66219 0 0 0 >>>>>>>> 01w0d16h 609733 >>>>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never >>>>>>>> Idle >>>>>>>> (Admin) >>>>>>>> 95.130.232.4 4 48972 10983 58016 0 0 0 >>>>>>>> 02:28:03 0 >>>>>>>> 95.130.232.6 4 48972 2536718 65789 0 0 0 >>>>>>>> 02:21:53 137022 >>>>>>>> 193.108.98.2 4 20828 208151 11056 0 0 0 >>>>>>>> 2d09h49m 137235 >>>>>>>> 193.108.98.3 4 20828 204274 11056 0 0 0 >>>>>>>> 2d09h39m 137227 >>>>>>>> >>>>>>>> bgp03.as48972.net(config-router)# do sh ip bgp sum >>>>>>>> BGP router identifier 95.130.232.6, local AS number 48972 >>>>>>>> RIB entries 311015, using 33 MiB of memory >>>>>>>> Peers 21, using 94 KiB of memory >>>>>>>> Peer groups 3, using 96 bytes of memory >>>>>>>> >>>>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ >>>>>>>> Up/Down >>>>>>>> State/P >>>>>>>> fxRcd >>>>>>>> 77.222.66.181 4 16243 298044 882 0 0 0 00:01:21 >>>>>>>> Idle >>>>>>>> (Admin) >>>>>>>> 78.152.40.92 4 5580 559289 5234 0 0 0 00:01:14 >>>>>>>> Idle >>>>>>>> (Admin) >>>>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never >>>>>>>> Idle >>>>>>>> (Admin) >>>>>>>> 95.130.232.3 4 48972 7643 867546 0 0 0 >>>>>>>> 02:21:10 39002 >>>>>>>> 95.130.232.4 4 48972 837 558695 0 0 0 >>>>>>>> 02:27:21 0 >>>>>>>> 193.108.98.2 4 20828 68276 883 0 0 0 >>>>>>>> 03:30:44 137235 >>>>>>>> 193.108.98.3 4 20828 64031 883 0 0 0 >>>>>>>> 03:30:35 137227 >>>>>>>> >>>>>>>> I would expect that the BGP03 would get at least 580.000 routes >>>>>>>> from the >>>>>>>> BGP02, but this doesn't happen for some strange reason. >>>>>>>> Also resetting the iBGP link between the BGP02 and BGP03 does not >>>>>>>> fix >>>>>>>> the problem. >>>>>>>> >>>>>>>> Anyone any idea? >>>>>>>> >>>>>>>> jan Hugo Prins >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Quagga-users mailing list >>>>>>>> [email protected] >>>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users >>>>>>> _______________________________________________ >>>>>>> Quagga-users mailing list >>>>>>> [email protected] >>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users >>>>>> _______________________________________________ >>>>>> Quagga-users mailing list >>>>>> [email protected] >>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users >>>>> _______________________________________________ >>>>> Quagga-users mailing list >>>>> [email protected] >>>>> https://lists.quagga.net/mailman/listinfo/quagga-users >>>> >>>> _______________________________________________ >>>> Quagga-users mailing list >>>> [email protected] >>>> https://lists.quagga.net/mailman/listinfo/quagga-users >>>> >>>> >>> >> >> >> _______________________________________________ >> Quagga-users mailing list >> [email protected] >> https://lists.quagga.net/mailman/listinfo/quagga-users >> >> > > -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, [email protected] Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ _______________________________________________ Quagga-users mailing list [email protected] https://lists.quagga.net/mailman/listinfo/quagga-users