Performance for unclassified packets when using HTB

Hey,

We are using HTB for a project and observing some performance
degradation for network transfers. I could not find any documentation
anywhere specifying what happens to unclassified packets - is there
some rate limit on them due to filtering overhead or the addition of
filters should not affect the transfer rate of unclassified packets ?

This is what our configuration looks like -

sudo tc qdisc add dev bond0 root handle 1: htb
sudo tc class add dev bond0 parent 1: classid 1:10 htb rate
34359738360  burst 34359738360  cburst 34359738360
sudo tc class add dev bond0 parent 1: classid 1:20 htb rate
34359738360  burst 34359738360  cburst 34359738360
sudo tc filter add dev bond0 parent 1: prio 99 protocol ip u32 match
ip dport 7700 0xffff classid 1:10
sudo tc filter add dev bond0 parent 1: prio 100 protocol ip u32 match
ip sport 7701 0xffff classid 1:20

The limits for the classes are dynamic and we update them periodically
based on the configuration. After adding these filters and classes, we
are seeing 50% performance drop for traffic not going to/from any of
the above ports (7700 and 7701).

Is there anything wrong with our configuration ?