Re: making HTB work...

> Chain NoCat (1 references)
> target     prot opt source               destination
> MARK       all  --  0.0.0.0/0            0.0.0.0/0          MARK set 0x4
> MARK       all  --  192.168.89.12        0.0.0.0/0          MAC
> 00:D0:B7:1C:BE:50 MARK set 0x1
> MARK       all  --  0.0.0.0/0            192.168.89.12      MARK set 0x1
> MARK       all  --  192.168.89.10        0.0.0.0/0          MAC
> 00:02:2D:1B:50:67 MARK set 0x1
> MARK       all  --  0.0.0.0/0            192.168.89.10      MARK set 0x1
> MARK       all  --  192.168.89.11        0.0.0.0/0          MAC
> 00:E0:00:D3:47:13 MARK set 0x3
> MARK       all  --  0.0.0.0/0            192.168.89.11      MARK set 0x3
>
>
> Looks like my packets are getting marked. The MARK does indeed change
> based on who I log in as. So far, so good.
>
> Now, my questions:
>
> How do I know that this will actually work? How do I mark packets going
> both in and out, so that throttling happens both ways?
> Tests with speakeasy's tool show throttling of data going out (I can
> clamp it down to 56K for public category) but nothing makes it throttle
> for the incoming stream.
>
> Anybody have a clue ?

It isn't easy, and the way it comes out of the box isn't even close to right
for download traffic.

First, note that NAT occurs AFTER the mangle table.  This means that on your
packets headed from the Internet to your clients, you are not going to be
able to match on a destination of 192.168.x.x like you are trying to do in
the mangle table you pasted.  You can, however, match with tc filters, as tc
filters are done AFTER NAT.  However, you can't modify tc filters, you can
only delete the entire thing and start over.  This makes it even more
complicated.

In my case, I was attempting to implement per-user throttling, not per-group
throttling.  My goal was for normal users (anonymous users that skipped
login) to each receive 512kbit in each direction, and logged-in users to be
exempt from all throttling.  I managed to do this, though it isn't a pretty
method.  Also, I'm a network engineer, not a programmer, so don't expect my
code to be great, but it does seem to work in my production environment, so
I pass this on in the hopes it will be useful for others.  Hopefully I
remembered to include all the relevant code.  If people have questions or
need more code, just ask.

I know this isn't what you're working on (group throttling), but perhaps
some of the methods I used to accomplish per-user throttling will give you
ideas for what you are trying to do.



For download throttling:

For downloading, we can match the destination IP with tc filters, since tc
filters occur after NAT where we can see the private internal destination
IPs.  So, I don't use iptables marking at all.

system "tc qdisc del dev $ENV{'InternalDevice'} root handle 10:";
system "tc qdisc add dev $ENV{'InternalDevice'} root handle 10: cbq
bandwidth 10Mbit avpkt 1000";
system "tc class add dev $ENV{'InternalDevice'} parent 10:0 classid 10:1 cbq
bandwidth 10Mbit rate 10mbit allot 1514 maxburst 20 avpkt 1000 prio 1";
system "tc filter add dev $ENV{'InternalDevice'} protocol ip parent 10: prio
1 u32 match ip src 64.106.128.101 classid 10:1";

for($x=$IP_low; $x <= $IP_high; $x++) {
        system "tc class add dev $ENV{'InternalDevice'} parent 10:1 classid
10:$x cbq bandwidth 10Mbit rate 512kbit allot 1514 maxburst 20 avpkt 1000
prio 1 bounded";
        system "tc qdisc add dev $ENV{'InternalDevice'} parent 10:$x sfq
quantum 1514b perturb 15";
        system "tc filter add dev $ENV{'InternalDevice'} protocol ip parent
10: prio 1 u32 match ip dst $IP_block.$x classid 10:$x";
}

I create a queue and a filter for every possible IP in my DHCP pool.  I
match users into the throttle based on tc filters to the destination private
IP.  That wasn't so hard.  But now things get more complicated.




For upload throttling:

Remember that here we can use iptables mark, but can't use tc filters to
match on source IPs.  I need to somehow pass the source IP (only available
during the mangle table) on to tc in order to sort users into per-IP queues.
But how can I when NoCat is already using the mark field?  What I do is
store them both in the mark field.

I modified every "match" in NoCat to use a bitmask.  For example, "--mark 3"
becomes "--mark 3/7".  Now nocat is only using the least significant 3 bits,
and I'm free to use the higher bits for my own use.  I bitshift the last
octet of my internal IP into the mark field.

In access.fw:

shiftedmark=`perl -e "print $mark+($lastoctet*(2**3))"`
iptables -t mangle $cmd NoCat $match_mac -s $ip -j MARK --set-mark
$shiftedmark

So now every packet is marked with both the last octet of the source IP and
the group.  NoCat uses the group for its own purposes, and I can match on
the last octet part to sort users into the proper outbound queue with tc.
But there's ANOTHER catch.  tc doesn't support bitmasks.  So, for every IP I
have to put in 4 tc filters to match on all 4 possible nocat groups:


system "tc qdisc del dev $ENV{'ExternalDevice'} root handle 20:";
system "tc qdisc add dev $ENV{'ExternalDevice'} root handle 20: cbq
bandwidth 10Mbit avpkt 1000";
system "tc class add dev $ENV{'ExternalDevice'} parent 20:0 classid 20:1 cbq
bandwidth 10Mbit rate 10mbit allot 1514 maxburst 20 avpkt 1000 prio 1";

for($x=$IP_low; $x <= $IP_high; $x++) {
        system "tc class add dev $ENV{'ExternalDevice'} parent 20:1 classid
20:$x cbq bandwidth 10Mbit rate 512kbit allot 1514 maxburst 20 avpkt 1000
prio 1 bounded";
        system "tc qdisc add dev $ENV{'ExternalDevice'} parent 20:$x sfq
quantum 1514b perturb 15";
        for($y=1; $y<4; $y++) {
                $mark = ($x * (2**3)) + $y;
                system "tc filter add dev $ENV{'ExternalDevice'} protocol ip
parent 20: prio 1 handle $mark fw classid 20:$x";
        }
}

So, now we've sorted every IP into a private queue for upload and a private
queue for download.  The final step is to modify the queue rate every time
there is a change.  In access.conf:

rate="10Mbit"
if [ "$class" = "Public" ]; then
    rate="512kbit"
fi
if [ "$action" = "deny" ]; then
    rate="512kbit"
fi

tc class change dev eth1 parent 10:1 classid 10:$lastoctet cbq bandwidth
10Mbit rate $rate allot 1514 maxburst 20 avpkt 1000 prio 1 bounded
tc class change dev eth0 parent 20:1 classid 20:$lastoctet cbq bandwidth
10Mbit rate $rate allot 1514 maxburst 20 avpkt 1000 prio 1 bounded

What this does is set the rate to 10mbit default, but down to 512kbit
whenever an IP is moved back to deny state or when a "public" group user
logs in.

The end result is that I have a gateway where people can log in with their
radius account for unrestricted access, or the general public can just click
"free Internet access" to get on the Internet for free with no account
required but with limited 512kbit speed.