Re: clogging qdisc

On 27.12.2018 19:23, Grant Taylor wrote:
> On 12/27/18 10:15 AM, Grzegorz Gwóźdź wrote:
>> This solution worked for few years in several networks but in one 
>> network since few weeks, in peak hours that mechanism clogs.
>
> Okay.  It sounds to me like the methodology works well enough. But it 
> might have scaling problems.


Previous system it was 2 x 6-core xeon (24 threads)
Now it's threadripper 2990 (64 threads)

No core is loaded over 40% (max 25% software interrupts per core only 
because NIC has 8 queues per interface so it can be operated by 8 cores 
per interface)
Few ksoftirqd on about 0.7% cpu

How can I find choke point if no parameter indicates it?

>
>> Pings to all local hosts grows to hundreds ms (even to hosts without 
>> any traffic) and throughtput drops.
>
> Ouch.
>
>> The only solution is:
>> tc qdisc del root dev eth0
>
> That doesn't seem like a solution.  Maybe a workaround, if you're lucky.
>
>> If I immediately add rules again problem immediately starts too.
>
> That sounds like the workaround doesn't even work.
>
>> But after some time even though traffic is bigger I load queues and 
>> everything works until next attack
>
> I'm thinking that "attack" might be the proper word.
>
> I'm wondering if this is a number of packets per second vs the size of 
> packets per second issue.
>
> Specifically if the "attack" is considerably more smaller packets than 
> normal.  I'm guessing normal traffic is fewer but bigger packets.
>
> Take a packet capture during normal traffic periods and a separate 
> packet capture during attack traffic periods.
>
> Then open each of the captures in Wireshark and pull up the Packet 
> Lengths report from the Statistics menu.  I'm guessing that you will 
> see a significant difference between the two captures.
>

I put separate machine on mirrored interface just for logging traffic - 
statistically there is no difference both from LAN and from Internet.
After clogging, number of small packets rises (mostly SYN packets) and 
overall number of packets drops but just before clogging I cannot see 
anything strange. But it's over 1Gbps and 100k pps
I'm almost sure the cause is in the incoming packets but I'ts hard to find

10s statistics under "attack", roughly the same as normal
=================================================================
Packet Lengths:
Topic / Item       Count         Average       Min val       Max 
val       Rate (ms)     Percent       Burst rate    Burst start
----------------------------------------------------------------------------------------------------------------------------------
Packet Lengths     3416240       866,96        60 1518          
341,6367      100%          443,1200 3,735
  0-19              0             -             - -             
0,0000        0,00%         - -
  20-39             0             -             - -             
0,0000        0,00%         - -
  40-79             987398        65,82         60 79            
98,7435       28,90%        129,1300 2,865
  80-159            323407        103,61        80 159           
32,3419       9,47%         48,5500 1,905
  160-319           98202         218,05        160 319           
9,8206        2,87%         12,1800 3,360
  320-639           72602         467,25        320 639           
7,2605        2,13%         16,1800 4,680
  640-1279          57417         969,00        640 1279          
5,7419        1,68%         7,6000 1,510
  1280-2559         1877214       1466,15       1280 1518          
187,7284      54,95%        257,1100 3,705
  2560-5119         0             -             - -             
0,0000        0,00%         - -
  5120 and greater  0             -             - -             
0,0000        0,00%         - - 
----------------------------------------------------------------------------------------------------------------------------------

Statistics above shows 300k pps because its whole interface IN + OUT for 
both containers

Default queue for unclassified packets is 2Gbps so everything that 
doesn't have defined filter fits there.

Smaller or bigger packets - every packet should fall into designed queue 
if it is normal packet but maybe it's maliciously prepared frame that 
breaks rules and algorithms


>> I don't think it is hardware issue because this system works in LXC 
>> container and on the same NIC in other container (doing the same work 
>> for other clients) everything works fine.
>
> I think containers and VMs are good for some things.  I don't think 
> that they are (more specifically their overhead is) good for high 
> throughput traffic.  Particularly large numbers of small packets per 
> second (PPS). High PPS with small traffic requires quite a bit of 
> optimization.  I also think it's actually rare outside of specific 
> situations.  What's I've seen more frequently is fewer (by one or more 
> orders of magnitude) packets that are larger (by one or more orders of 
> magnitude).  Overall the amount of data is roughly the same.  But 
> /how/ it's done can cause considerable load on equipment.  Especially 
> equipment that is not optimized for high PPS, much less additional 
> overhead like containers or VMs.
>
It's lxc so there is almost no overhead on network traffic
I estimate this machine could handle over 40Gbps

There are 2 lxc containers on this machine. One is handling 1.7Gbps and 
is working fine, second has 1.2Gbps and has problems.
Both work on the same physical interface.
I've moved one container to other machine - no difference - one, that 
was clogging - clogged still


>> Load on system is low, there is no hardware problem, whole hardware 
>> has been replaced, on new hardware I've installed new system (Ubuntu 
>> 18.04)
>> No dropped packets in interface statistics. dmesg is clear.
>
> What messages were you seeing in dmesg before?

I mean there is nothing new in dmesg


>
>> As a result conntrack table grows until overflow (if I don't delete 
>> qdisc)
>> I even sniffed all traffic and tried to analyze it but it's hard 
>> since it's over 1Gbps (on 10Gb interface)
>
> The connection tracking table overflowing tells me one of two things. 
> That you are truly dealing with a high PPS condition -or- that you 
> don't have enough memory in the system and the size of the conntrack 
> table is restricted.
>
> I once took a system that comfortable ran with ~512 MB of memory up to 
> 4 GB to allow the conntrack table to be large enough for what the 
> system was doing.  (I think the conntrack table was a fixed percentage 
> of memory in that kernel.  Maybe it's a tune able now.)
>
conntrack table overflows because clients try to establish connections 
but can not wait for reply (clogged TC blocks them). It fills up for 
about 10 minutes and then overflows


>> What can I check?
>
> Check the Packet Lengths graph as suggested above.
>
> If your problem is indeed high PPS, you might also be having problem 
> outside of your Linux machine.  It's quite possible that there is 
> enough traffic / PPS that things like switches and / or wireless 
> access points are also being negatively effected.  It's possible that 
> they are your choke point and not the actual Linux system.

But deleting qdisc causes traffic growth since clients have no limits. 
Choke point is certainly in my Linux box.

And if it was NIC issue it would go worse after deleting qdisc (more 
traffic)


>
>> Where to look for a cause?
>
> I think you need to get a good understanding of what your two typical 
> traffic patterns are, normal, and attack.  Including if this is 
> legitimate traffic or if it is someone conducting an attack and the 
> network is buckling under the stress.
>
> You might also consider changing out network cards.  I've been around 
> people that like to poo poo some Realtek cards and other non-Intel / 
> non-Broadcom NICs.  Admittedly, some of the better NICs have more CPU 
> / memory / I/O on them to handle more traffic.

I use 10Gb Mellanox NIC. I can try Intel's but Mellanox always performed 
better for me.


Are there any "hacks" in TC allowing to look in the guts?

It looks like it's changing state to "clogged" but

tc -s class ls dev eth0

looks completely normal (only grows number of sfq queues created 
dynamically for every connection since more and more connections are 
created but not closed)

If I don't delete qdisc it recovers from that state after about 15 
minutes (but earlier traffic is crushed to 1/3, conntrack overflows, etc)

At the same time system itself responds pretty well, there is no sign of 
anything going wrong, similar system in other container works ok...


GG