Re: clogging qdisc

On 12/27/18 10:15 AM, Grzegorz Gwóźdź wrote:
> This solution worked for few years in several networks but in one 
> network since few weeks, in peak hours that mechanism clogs.

Okay.  It sounds to me like the methodology works well enough.  But it 
might have scaling problems.

> Pings to all local hosts grows to hundreds ms (even to hosts without any 
> traffic) and throughtput drops.

Ouch.

> The only solution is:
> tc qdisc del root dev eth0

That doesn't seem like a solution.  Maybe a workaround, if you're lucky.

> If I immediately add rules again problem immediately starts too.

That sounds like the workaround doesn't even work.

> But after some time even though traffic is bigger I load queues and 
> everything works until next attack

I'm thinking that "attack" might be the proper word.

I'm wondering if this is a number of packets per second vs the size of 
packets per second issue.

Specifically if the "attack" is considerably more smaller packets than 
normal.  I'm guessing normal traffic is fewer but bigger packets.

Take a packet capture during normal traffic periods and a separate 
packet capture during attack traffic periods.

Then open each of the captures in Wireshark and pull up the Packet 
Lengths report from the Statistics menu.  I'm guessing that you will see 
a significant difference between the two captures.

> I don't think it is hardware issue because this system works in LXC 
> container and on the same NIC in other container (doing the same work 
> for other clients) everything works fine.

I think containers and VMs are good for some things.  I don't think that 
they are (more specifically their overhead is) good for high throughput 
traffic.  Particularly large numbers of small packets per second (PPS). 
High PPS with small traffic requires quite a bit of optimization.  I 
also think it's actually rare outside of specific situations.  What's 
I've seen more frequently is fewer (by one or more orders of magnitude) 
packets that are larger (by one or more orders of magnitude).  Overall 
the amount of data is roughly the same.  But /how/ it's done can cause 
considerable load on equipment.  Especially equipment that is not 
optimized for high PPS, much less additional overhead like containers or 
VMs.

> Load on system is low, there is no hardware problem, whole hardware has 
> been replaced, on new hardware I've installed new system (Ubuntu 18.04)
> No dropped packets in interface statistics. dmesg is clear.

What messages were you seeing in dmesg before?

> As a result conntrack table grows until overflow (if I don't delete qdisc)
> I even sniffed all traffic and tried to analyze it but it's hard since 
> it's over 1Gbps (on 10Gb interface)

The connection tracking table overflowing tells me one of two things. 
That you are truly dealing with a high PPS condition -or- that you don't 
have enough memory in the system and the size of the conntrack table is 
restricted.

I once took a system that comfortable ran with ~512 MB of memory up to 4 
GB to allow the conntrack table to be large enough for what the system 
was doing.  (I think the conntrack table was a fixed percentage of 
memory in that kernel.  Maybe it's a tune able now.)

> What can I check?

Check the Packet Lengths graph as suggested above.

If your problem is indeed high PPS, you might also be having problem 
outside of your Linux machine.  It's quite possible that there is enough 
traffic / PPS that things like switches and / or wireless access points 
are also being negatively effected.  It's possible that they are your 
choke point and not the actual Linux system.

> Where to look for a cause?

I think you need to get a good understanding of what your two typical 
traffic patterns are, normal, and attack.  Including if this is 
legitimate traffic or if it is someone conducting an attack and the 
network is buckling under the stress.

You might also consider changing out network cards.  I've been around 
people that like to poo poo some Realtek cards and other non-Intel / 
non-Broadcom NICs.  Admittedly, some of the better NICs have more CPU / 
memory / I/O on them to handle more traffic.

You might want to evaluate your container network configuration.  I've 
read a few places that there are performance differences in Linux native 
bridges, OvS, {MAC,IP}VLAN.  It's possible that based on shear numbers, 
the difference in performance is adding up and starting to cause 
problems under the higher traffic load.

Unfortunately, I think you have more investigating to do to be able to 
identify what the problem might be.  Simply trading out hardware is 
likely to be expensive and come with a lot of annoyances (or worse).



-- 
Grant. . . .
unix || die