[quagga-users 14538] Re: Serious BGP issue

I have seen this as well. What I noticed is that initially a set of x
routes is pushed from router A to router B. After that point you would
expect that a route coming is on router A would be pushed as update to
it's iBGP peers. But what I notice is that this doesn't happen (debug
bgp updates out). After the initial connection only updates about
unreachable networks are send to the iBGP peers.

I don't know what happened yesterday evening because it looks like the
iBGP session between my 2 frontend routers has been reset.



But you can see that the number of routes is going down again on the
iBGP session.



This while the number of routes from my upsteams is rather stable:



I'm really curious what kind of bug this is.

Jan Hugo




On 11/16/2016 03:22 PM, Mike Tancsa wrote:
> The other thing I noticed, is that over time, the problem iBGP peer
> sends less and less routes to its peers.  So at start up time, it will
> advertise (x) prefixes to its iBGP peers. Over the course of a day, it
> will go to (x-y) and eventually to almost nothing until I do a soft
> clear out and it will return to (x).  But (x) is still a number lower
> than it should be.  I have artificially set the local pref to a higher
> value, so it should be passing far more routes to the iBGP peers than
> just (x).
>
> 	---Mike
>
> On 11/16/2016 5:00 AM, Paul Thornton wrote:
>> Hi all
>>
>> I also saw something very similar in January:
>>
>> https://lists.quagga.net/pipermail/quagga-users/2016-February/014275.html
>>
>> I've not managed to reproduce this - in lab testing I couldn't make it
>> fail, I could break it on production but that isn't the best place to be
>> troubleshooting and debugging!
>>
>> It seemed to be related to certain prefixes that just weren't advertised
>> between routers within our network; but there wasn't any real rhyme or
>> reason to what these were (comparing attributes etc).
>>
>> (Apologies if this goes twice to the list, I replied first time from the
>> wrong address)
>>
>> Paul.
>>
>>
>> On 15/11/2016 21:40, Mike Tancsa wrote:
>>> This sounds like the problem I saw last Jan.
>>>
>>> https://lists.quagga.net/pipermail/quagga-users/2016-January/014248.html
>>>
>>>
>>>
>>> On 11/3/2016 8:31 PM, jan hugo prins wrote:
>>>> I'm happy to provide you with the complete config of my 3 routers.
>>>> I will send them to you personaly after stripping the passwords.
>>>>
>>>> Jan Hugo
>>>>
>>>>
>>>> On 11/04/2016 12:32 AM, Martin Winter wrote:
>>>>> Jan,
>>>>>
>>>>> can you clarify if there are any route-maps, filter-list, prefix-lists
>>>>> or anything else?
>>>>> Also, are you doing any “neighbor x.x.x.x next-hop-self” or
>>>>> peer groups?
>>>>>
>>>>> (I’m not sure interested in config which would match, but also filters
>>>>> etc on the relevant peers in case they are misbehaving)
>>>>>
>>>>> My testing on filter-lists etc is a bit lacking, so I would assume the
>>>>> bug most likely to be there.
>>>>>
>>>>> - Martin
>>>>>
>>>>>
>>>>> On 3 Nov 2016, at 9:41, jan hugo prins wrote:
>>>>>
>>>>>> Some extra side note:
>>>>>>
>>>>>> A random prefix on the BGP02:
>>>>>>
>>>>>> bgp02.as48972.net# sh ip bgp 23.226.189.0/24
>>>>>> BGP routing table entry for 23.226.189.0/24
>>>>>> Paths: (3 available, best #2, table Default-IP-Routing-Table)
>>>>>>    Advertised to non peer-group peers:
>>>>>>    95.130.232.4 95.130.232.6
>>>>>>    5580 5580 1299 4809 38197
>>>>>>      78.152.40.92 (metric 11) from 95.130.232.6 (95.130.232.6)
>>>>>>        Origin IGP, metric 0, localpref 90, valid, internal
>>>>>>        Last update: Thu Nov  3 16:09:45 2016
>>>>>>
>>>>>>    16243 47886 3356 4809 38197
>>>>>>      77.222.66.177 from 77.222.66.177 (87.249.109.240)
>>>>>>        Origin IGP, localpref 90, valid, external, best
>>>>>>        Last update: Thu Oct 27 00:56:04 2016
>>>>>>
>>>>>>    5580 5580 1299 4809 38197
>>>>>>      78.152.49.109 from 78.152.49.109 (80.94.64.124)
>>>>>>        Origin IGP, metric 0, localpref 90, valid, external
>>>>>>        Community: 1299:30000 5580:21220
>>>>>>        Last update: Mon Oct 31 17:51:58 2016
>>>>>>
>>>>>> Is available on the BGP03:
>>>>>>
>>>>>> bgp03.as48972.net# sh ip bgp 23.226.189.0/24
>>>>>> BGP routing table entry for 23.226.189.0/24
>>>>>> Paths: (2 available, best #2, table Default-IP-Routing-Table)
>>>>>>    Advertised to non peer-group peers:
>>>>>>    95.130.232.3 95.130.232.4
>>>>>>    16243 47886 3356 4809 38197
>>>>>>      77.222.66.181 from 77.222.66.181 (87.249.109.241)
>>>>>>        Origin IGP, localpref 90, valid, external
>>>>>>        Last update: Thu Nov  3 16:09:38 2016
>>>>>>
>>>>>>    5580 5580 1299 4809 38197
>>>>>>      78.152.40.92 from 78.152.40.92 (80.94.64.124)
>>>>>>        Origin IGP, metric 0, localpref 90, valid, external, best
>>>>>>        Community: 1299:30000 5580:21220
>>>>>>        Last update: Thu Nov  3 16:09:21 2016
>>>>>>
>>>>>> But even though the BGP02 claims to have it advertised to both the
>>>>>> bgp03
>>>>>> and the routeserver / lookingglass:
>>>>>>
>>>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.4 adv |grep 23.226.189.0/24
>>>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.6 adv |grep 23.226.189.0/24
>>>>>>
>>>>>> It did not.
>>>>>>
>>>>>> Jan Hugo Prins
>>>>>>
>>>>>> On 11/03/2016 05:08 PM, jan hugo prins wrote:
>>>>>>> As a sidenote, a second router that runs my LookingGlass and is
>>>>>>> used to
>>>>>>> inject blackhole routes, is also not receiving the full table from
>>>>>>> the
>>>>>>> BGP02.
>>>>>>> This can be observed at http://lg.as48972.net/
>>>>>>>
>>>>>>> Jan Hugo Prins
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 11/03/2016 04:36 PM, jan hugo prins wrote:
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> I have a strange issue.
>>>>>>>> I have 2 external BGP routers. They both have 2 uplinks to my
>>>>>>>> transit
>>>>>>>> providers and this works fine.
>>>>>>>> Until a while ago, when I wanted to take one router out for
>>>>>>>> maintenance,
>>>>>>>> I would shut it's transit links, remove "default-information
>>>>>>>> originate
>>>>>>>> always" from OSPF, set higher costs on the uplink and / or set
>>>>>>>> "max-metric router-lsa administrative"  wait for most of the
>>>>>>>> traffic to
>>>>>>>> shift to the second router and do my maintenance.
>>>>>>>>
>>>>>>>> Because the router would always have an iBGP link to the second
>>>>>>>> router
>>>>>>>> it would receive all routes from this router and be fine in the
>>>>>>>> case of
>>>>>>>> some traffic coming in anyway. But today I wanted to upgrade
>>>>>>>> Quagga to
>>>>>>>> version 1.10 and I saw something really strange:
>>>>>>>>
>>>>>>>> bgp02.as48972.net# sh ip bgp sum
>>>>>>>> BGP router identifier 95.130.232.3, local AS number 48972
>>>>>>>> RIB entries 1120940, using 120 MiB of memory
>>>>>>>> Peers 29, using 129 KiB of memory
>>>>>>>> Peer groups 3, using 96 bytes of memory
>>>>>>>>
>>>>>>>> Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ
>>>>>>>> Up/Down
>>>>>>>> State/P
>>>>>>>> fxRcd
>>>>>>>> 77.222.66.177   4 16243 1434500   11053        0    0    0
>>>>>>>> 01w0d16h   609362
>>>>>>>> 78.152.49.109   4  5580 3162545   66219        0    0    0
>>>>>>>> 01w0d16h   609733
>>>>>>>> 95.130.232.2    4 48972       0       0        0    0    0 never
>>>>>>>> Idle
>>>>>>>> (Admin)
>>>>>>>> 95.130.232.4    4 48972   10983   58016        0    0    0
>>>>>>>> 02:28:03        0
>>>>>>>> 95.130.232.6    4 48972 2536718   65789        0    0    0
>>>>>>>> 02:21:53   137022
>>>>>>>> 193.108.98.2    4 20828  208151   11056        0    0    0
>>>>>>>> 2d09h49m   137235
>>>>>>>> 193.108.98.3    4 20828  204274   11056        0    0    0
>>>>>>>> 2d09h39m   137227
>>>>>>>>
>>>>>>>> bgp03.as48972.net(config-router)# do sh ip bgp sum
>>>>>>>> BGP router identifier 95.130.232.6, local AS number 48972
>>>>>>>> RIB entries 311015, using 33 MiB of memory
>>>>>>>> Peers 21, using 94 KiB of memory
>>>>>>>> Peer groups 3, using 96 bytes of memory
>>>>>>>>
>>>>>>>> Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ
>>>>>>>> Up/Down
>>>>>>>> State/P
>>>>>>>> fxRcd
>>>>>>>> 77.222.66.181   4 16243  298044     882        0    0    0 00:01:21
>>>>>>>> Idle
>>>>>>>> (Admin)
>>>>>>>> 78.152.40.92    4  5580  559289    5234        0    0    0 00:01:14
>>>>>>>> Idle
>>>>>>>> (Admin)
>>>>>>>> 95.130.232.2    4 48972       0       0        0    0    0 never
>>>>>>>> Idle
>>>>>>>> (Admin)
>>>>>>>> 95.130.232.3    4 48972    7643  867546        0    0    0
>>>>>>>> 02:21:10    39002
>>>>>>>> 95.130.232.4    4 48972     837  558695        0    0    0
>>>>>>>> 02:27:21        0
>>>>>>>> 193.108.98.2    4 20828   68276     883        0    0    0
>>>>>>>> 03:30:44   137235
>>>>>>>> 193.108.98.3    4 20828   64031     883        0    0    0
>>>>>>>> 03:30:35   137227
>>>>>>>>
>>>>>>>> I would expect that the BGP03 would get at least 580.000 routes
>>>>>>>> from the
>>>>>>>> BGP02, but this doesn't happen for some strange reason.
>>>>>>>> Also resetting the iBGP link between the BGP02 and BGP03 does not
>>>>>>>> fix
>>>>>>>> the problem.
>>>>>>>>
>>>>>>>> Anyone any idea?
>>>>>>>>
>>>>>>>> jan Hugo Prins
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Quagga-users mailing list
>>>>>>>> Quagga-users-UOy77sIEA+cAd7ICUelF/[email protected]
>>>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>>>> _______________________________________________
>>>>>>> Quagga-users mailing list
>>>>>>> Quagga-users-UOy77sIEA+cAd7ICUelF/[email protected]
>>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>>> _______________________________________________
>>>>>> Quagga-users mailing list
>>>>>> Quagga-users-UOy77sIEA+cAd7ICUelF/[email protected]
>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>> _______________________________________________
>>>>> Quagga-users mailing list
>>>>> Quagga-users-UOy77sIEA+cAd7ICUelF/[email protected]
>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>> _______________________________________________
>>>> Quagga-users mailing list
>>>> Quagga-users-UOy77sIEA+cAd7ICUelF/[email protected]
>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>
>>>>
>>
>> _______________________________________________
>> Quagga-users mailing list
>> Quagga-users-UOy77sIEA+cAd7ICUelF/[email protected]
>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>
>>
>

_______________________________________________
Quagga-users mailing list
Quagga-users-UOy77sIEA+cAd7ICUelF/[email protected]
https://lists.quagga.net/mailman/listinfo/quagga-users