Re: TBF Seems broken in recent X86-64 kernels

Hi,

I had been noticing similar behaviour recently and came to a similar
conclusion. So I took a look into it and in my case it was actually
came down to the fact that TSO/GSO is now implemented (and enabled) in
most NIC drivers that support it. TSO/GSO has the effect of increasing
the 'internal mtu'. tbf has a check in its enqueue function that
checks see if the incoming packet is over the tc's mtu (which defaults
to 2047) - but with TSO/GSO the 'packet size' grows larger than that.
If you are seeing this issue then the simple fix was just to add the
appropriate mtu parameter to the tc command - so in your case it would
be (yu can try playing with the value of the mtu):

tc qdisc add dev eth4 root handle 1: tbf rate 6300kbit burst 100kb
latency 200ms mpu 64 mtu 10000

To verify whether it is the TSO/GSO causing the issue you could try
disabling them (though you will lose some of the performance benefits)
using ethtool -K eth4 tso off gso off - if you do this then your
original tbf configuration should work ok.

Incidentally this effect is much worse when tbf is used on the
loopback interface which has a default interface mtu of 16K - thus you
need to set the tc mtu to about 100k. For a bit more on it see my
post:
http://timesinker.blogspot.com/2011/02/using-linux-tbf-qdisc-for-rate-limiting.html

Cheers,

Piers O'Hanlon



On 12 March 2011 16:59, Tom Eastep <[email protected]> wrote:
> It appears that the breakage occurred somewhere between 2.6.26 and
> 2.6.32. I've tried this on recent X86 Fedora, OpenSuSE and Debian
> releases and the results are similar. In all cases, X86 works as expected.
>
> Here's a test on 2.6.26:
>
> sami:~# uname -r
> 2.6.26-2-vserver-amd64
> sami:~# scp shorewall-docs-xml-4.4.17.tgz gateway:
> shorewall-docs-xml-4.4.17.tgz     100% 6306KB   6.2MB/s   00:00
>
> sami:~# tc qdisc add dev eth4 root handle 1: tbf rate 6300kbit \
>                               burst 100kb latency 200ms mpu 64
>
> sami:~# scp shorewall-docs-xml-4.4.17.tgz gateway:
> shorewall-docs-xml-4.4.17.tgz     100% 6306KB   1.0MB/s   00:06
>
> sami:~# tc -s qdisc ls dev eth4; tc -s class ls dev eth4
> qdisc tbf 1: root rate 6300Kbit burst 100Kb lat 200.0ms
>  Sent 6919034 bytes 6366 pkt (dropped 4, overlimits 8825 requeues 0)
>  rate 0bit 0pps backlog 0b 0p requeues 0
> class tbf 1:1 parent 1:
> sami:~#
>
> This test is from 2.6.32:
>
> #gateway:~# uname -r
> 2.6.32-5-vserver-amd64
>
> root@gateway:~# scp shorewall-docs-xml-4.4.17.tgz sami:
> shorewall-docs-xml-4.4.17.tgz     100% 6306KB   6.2MB/s   00:00
>
> root@gateway:~# scp shorewall-docs-xml-4.4.17.tgz sami:
> shorewall-docs-xml-4.4.17.tgz        100% 6306KB   6.2MB/s   00:00
> root@gateway:~# tc qdisc add dev eth4 root handle 1: tbf rate \
>                         6300kbit burst 100kb latency 200ms mpu 64
> root@gateway:~# scp shorewall-docs-xml-4.4.17.tgz sami:
> shorewall-docs-xml-4.4.17.tgz         38% 2448KB  30.8KB/s - stalled -
> shorewall-docs-xml-4.4.17.tgz         38% 2448KB  27.7KB/s - stalled -
>
> At this point I killed scp.
>
> root@gateway:~# tc -s qdisc ls dev eth1; tc -s class ls dev eth1
> qdisc tbf 1: root refcnt 6 rate 6300Kbit burst 100Kb lat 200.0ms
>  Sent 1097211 bytes 1347 pkt (dropped 285, overlimits 0 requeues 0)
>  rate 0bit 0pps backlog 0b 0p requeues 0
> class tbf 1:1 parent 1:
>
> root@gateway:~#
>
> -Tom
> --
> Tom Eastep        \ When I die, I want to go like my Grandfather who
> Shoreline,         \ died peacefully in his sleep. Not screaming like
> Washington, USA     \ all of the passengers in his car
> http://shorewall.net \________________________________________________
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html