Re: [f-nsp] FCX High CPU Diagnostics
Derek Maxwell <[email protected]> Wed, 3 Apr 2019 19:49:26 +0000
| Newsgroups | gmane.network.nsp.foundry |
|---|---|
| Message-ID | <[email protected]> |
Output of dm cpu: idle pid 0 id 536918812 cpu 0 usage 0 main pid 1 id 536918812 cpu 0 usage 0 flash pid 0 id 536918812 cpu 0 usage 0 arrays at 2238d894 and 2238dd6c from entry 78 to entry 35 loop cnt: 3395 3626 -> 231 read cnt: 57670 61592 -> 3922 loop rate = 231/430 = 0 rcode = 100 - (100*0/2550) = 100 99/100 busy, from 43 sec ago sho ip bgp sum does not show any FIB programming We have fewer than 10 routes in so we're definitely not exceeding any limits. We have icmp redirects disabled On 4/3/19, 12:35 PM, "foundry-nsp on behalf of Hunter Fuller" <[email protected] on behalf of [email protected]> wrote: I don't know FCX, but on the ICX we have the command "dm cpu" that gives more details. Lists the 5 most busy tasks on the system. Run it two times 1 minute apart, if the first time gave you bogus info. SSH@LAB-7450STACK#dm cpu appl-task pid 1322 tid 0xa5270460 cpu usage 84% console-task pid 1369 tid 0xa4a42460 cpu usage 0% flash-task pid 1281 tid 0xb5fd6460 cpu usage 0% sys-tmr-task pid 1280 tid 0xb5ff6460 cpu usage 0% itc-task pid 1285 tid 0xb5f76460 cpu usage 0% loop cnt: 567307594 569243809 -> 1936215 read cnt: 136318039 136318293 -> 254 loop rate = 1936215/270 = 7171 rcode_old = 100 - (100*7171/3900) = -83 rcode_log2= 100-(100*log2(7171)/log2(3900))=-7 1/100 busy, from 27 sec ago -- Hunter Fuller Router Jockey VBH Annex B-5 +1 256 824 5331 Office of Information Technology The University of Alabama in Huntsville Network Engineering On Wed, Apr 3, 2019 at 2:28 PM Tim Warnock <[email protected]> wrote: > > Are you trying to put more routes than can go into TCAM? > > Does 'show ip bgp sum' show any of your BGP sessions doing FIB programming? > > I don't know the FCX but a command from mlx/ce[s-r] is 'rcon lp', then run 'show task'. That will give you an idea if the LP is busy, if the command exists. > > > -----Original Message----- > > From: foundry-nsp [mailto:[email protected]] On > > Behalf Of Derek Maxwell > > Sent: Thursday, 4 April 2019 5:14 AM > > To: Jörg Kost <[email protected]>; foundry-nsp <foundry- > > [email protected]> > > Subject: Re: [f-nsp] FCX High CPU Diagnostics > > > > So, something doesn't add up here: > > > > During a high CPU event, this is the output of show processes cpu: > > > > show processes cpu > > Process Name 5Sec(%) 1Min(%) 5Min(%) 15Min(%) Runtime(ms) > > ARP 0.00 0.00 0.00 0.00 1989 > > BGP 0.00 0.00 0.00 0.00 2809 > > DOT1X 0.00 0.00 0.00 0.00 0 > > GVRP 0.00 0.00 0.00 0.00 0 > > ICMP 0.00 0.00 0.00 0.00 1690 > > IP 1.10 1.37 1.21 0.58 128381 > > OSPF 0.00 0.00 0.00 0.00 0 > > RIP 0.00 0.00 0.00 0.00 20 > > STP 0.00 0.00 0.00 0.00 0 > > VRRP 0.00 0.00 0.00 0.00 0 > > Process Name 5Sec(%) 1Min(%) 5Min(%) 15Min(%) Runtime(ms) > > IPv6 0.00 0.00 0.00 0.01 33559 > > ICMP6 0.00 0.01 0.01 0.01 31180 > > ND6 0.00 0.00 0.00 0.00 1461 > > RIPng 0.00 0.00 0.00 0.00 41 > > OSPFv3 0.00 0.00 0.00 0.00 0 > > IPV6_RX 0.00 0.00 0.00 0.00 181 > > > > IP is the only high CPU process, but it's only at 1.1-1.37%. > > > > > > This is the output of show cpu: > > show cpu > > 99 percent busy, from 289 sec ago > > 1 sec avg: 99 percent busy > > 5 sec avg: 99 percent busy > > 60 sec avg: 99 percent busy > > 300 sec avg: 99 percent busy > > > > It's 99% busy and the control plane is barely responding. > > > > Any thoughts on a hidden process that may be monopolizing the system? > > > > > > On 4/3/19, 11:36 AM, "foundry-nsp on behalf of Derek Maxwell" <foundry- > > [email protected] on behalf of > > [email protected]> wrote: > > > > Yes and that's exactly what I was looking for. Thanks! > > > > Will continue to debug from here when it pops up again and see what I > > find. > > > > -----Original Message----- > > From: Jörg Kost <[email protected]> > > Sent: Wednesday, April 3, 2019 11:26 AM > > To: Derek Maxwell <[email protected]>; foundry-nsp > > <[email protected]> > > Subject: Re: [f-nsp] FCX High CPU Diagnostics > > > > Does > > show processes cpu > > work on the FCX? > > > > On 3 Apr 2019, at 19:51, Derek Maxwell wrote: > > > > > Anyone have any thoughts on how to diagnose intermittent 99% CPU > > > utilization on a FCX624? We're running Layer 3 code and high CPU usage > > > seems to be causing BGP sessions to drop. > > > > > _______________________________________________ > > foundry-nsp mailing list > > [email protected] > > http://puck.nether.net/mailman/listinfo/foundry-nsp > > > > > > _______________________________________________ > > foundry-nsp mailing list > > [email protected] > > http://puck.nether.net/mailman/listinfo/foundry-nsp > _______________________________________________ > foundry-nsp mailing list > [email protected] > http://puck.nether.net/mailman/listinfo/foundry-nsp _______________________________________________ foundry-nsp mailing list [email protected] http://puck.nether.net/mailman/listinfo/foundry-nsp _______________________________________________ foundry-nsp mailing list [email protected] http://puck.nether.net/mailman/listinfo/foundry-nsp