[f-nsp] High CPU MLX-4

Hello List

My colleague posted on this list last month about a LP CPU issue experienced on MLX routers with GRE tunnels

The issue did not resolve itself instead we asked our customers to not send outbound traffic through us

However a new issue has arised

Our topography is as follows

CARRIER A -----> MLX-4-1 ---- MLX-4-2 ----> CARRIER B .. Carrier B connection is specifically designed to tank attacks, carrier A backhauls clean/protected traffic

MLX-4-2 holds our GRE tunnels

Now we are seeing 95% LP CPU on MLX-4-1 and a packet capture shows only GRE packets from MLX-4-2 destined for the customers GRE endpoint

SLOT #: LP CPU UTILIZATION in %:

in 1 second: in 5 seconds: in 60 seconds: in 300 seconds:

1: 94 94 94 94

LP-1#show tasks

Task Name Pri State PC Stack Size CPU Usage(%) task vid

-------------- --- ----- -------- -------- ------ ------------ --------

con 27 wait 0005c710 040c5dc8 32768 0 0

mon 31 wait 0005c710 041b7f10 8192 0 0

flash 20 wait 0005c710 041c6f40 8192 0 0

dbg 30 wait 0005c710 041beec0 16384 0 0

main 3 wait 0005c710 23cc6f40 262144 1 101

LP-I2C 3 wait 0005c710 27d70ee0 4096 0 101

LP-Assist 3 wait 0005c710 29bbef00 32768 0 101

LP-FCopy 3 wait 0005c710 29bc3f00 16384 0 101

LP-VPLS-Offld 3 wait 0005c710 29bc8f00 16384 0 101

LP-OF-Offld 3 wait 0005c710 29bcdf00 16384 0 101

LP-TM-Offld 3 wait 0005c710 29bd2f00 16384 0 101

LP-Stats 3 wait 0005c710 29bd7f60 16384 0 101

LP-IPC 3 wait 0005c710 29c18f00 262144 0 101

LP-TX-Pak 3 wait 0005c710 29c21f00 32768 0 101

LP-RX-Pak 3 wait 0005c710 29c42f38 131072 97 101

LP-SYS-Mon 3 wait 0005c710 29c47f28 16384 0 101

LP-RTD-Mon 3 wait 0005c710 29c4cf08 16384 0 101

LP-Console 3 ready 20b636c0 29c6df78 131072 0 101

LP-CPU-Mon 3 wait 0005c710 29c96f40 163840 0 101

MLX-4-2 Client GRE endpoint

xxxxxxxx -> xxxxx [Protocol:47]

**********************************************************************

[ppcr_rx_packet]: Packet received

Time stamp : 00 day(s) 00h 14m 33s:,

TM Header: [ 8026 2000 0000 ]

Type: Fabric Unicast(0x00000008) Size: 152 Parity: 2 Src IF: 0

Src Fap: 0 Dest Port: 0 Src Type: 0 Class: 0x00000000

**********************************************************************

Packet size: 146, XPP reason code: 0x00004747

Traffic levels are very low , the connection to carrier A shows approximately 40Mbps

LP CPU on MLX-4-2 is

SLOT #: LP CPU UTILIZATION in %:

in 1 second: in 5 seconds: in 60 seconds: in 300 seconds:

1: 1 1 1 1

As a test I shut the port between MLX-4-1 and MLX-4-2 immediately CPU usage dropped to 1% on MLX-4-1

No protocols over GRE tunnel we announce /24 and such and route throught the tunnel using static route

Show port on MLX-4-1 to MLX-4-2

Port is not enabled to receive all vlan packets for pbr

MTU 1548 bytes, encapsulation ethernet

Openflow: Disabled, Openflow Index 1

Cluster L2 protocol forwarding enabled

300 second input rate: 64599535 bits/sec, 61494 packets/sec, 0.74% utilization

300 second output rate: 2468 bits/sec, 4 packets/sec, 0.00% utilization

82862765 packets input, 10844340289 bytes, 0 no buffer

Received 25656 broadcasts, 27667 multicasts, 82809442 unicasts

0 input errors, 0 CRC, 0 frame, 0 ignored

0 runts, 0 giants

NP received 82871502 packets, Sent to TM 82860777 packets

NP Ingress dropped 10729 packets

9484 packets output, 726421 bytes, 0 underruns

Transmitted 127 broadcasts, 553 multicasts, 8804 unicasts

0 output errors, 0 collisions

NP transmitted 9485 packets, Received from TM 48717 packets

Show port on MLX-4-2 to MLX-4-1

Port is not enabled to receive all vlan packets for pbr

MTU 1548 bytes, encapsulation ethernet

Openflow: Disabled, Openflow Index 1

Cluster L2 protocol forwarding enabled

300 second input rate: 2416 bits/sec, 3 packets/sec, 0.00% utilization

300 second output rate: 64189791 bits/sec, 61109 packets/sec, 0.74% utilization

5105571056 packets input, 760042160157 bytes, 0 no buffer

Received 1874232 broadcasts, 5287030 multicasts, 5098409794 unicasts

0 input errors, 0 CRC, 0 frame, 0 ignored

0 runts, 0 giants

NP received 5105571056 packets, Sent to TM 5105113719 packets

NP Ingress dropped 457337 packets

590086066756 packets output, 81697023432476 bytes, 0 underruns

Transmitted 129784095 broadcasts, 208762136 multicasts, 589747520525 unicasts

0 output errors, 0 collisions

NP transmitted 590086072891 packets, Received from TM 590091974310 packets

Cheers

_______________________________________________
foundry-nsp mailing list
[email protected]
http://puck.nether.net/mailman/listinfo/foundry-nsp