Re: Correctly calculating overheads on unknown connections

Hi Dave, hi Andy,


On Sep 20, 2014, at 19:55 , Dave Taht <[email protected]> wrote:

> We'd had a very long thread on cerowrt-devel and in the end sebastian
> (I think) had developed some scripts to exaustively (it took hours)
> derive the right encapsulation frame size on a link. I can't find the
> relevant link right now, ccing that list…

	I am certainly not the first to have looked at ATM encapsulation effects on DSL-links, e.g. Jesper Dangaard Brouer wrote a thesis about this topic (see http://www.adsl-optimizer.dk) and together with Russel Stuart (http://ace-host.stuart.id.au/russell/files/tc/tc-atm/)  I believe they taught the linux kernel about how to account for encapsulation. What you need to tell the kernel is whether or not you have ATM encapsulation (ATM is weird in that each ip Packet gets chopped into 48 byte cells, with the last partially full cell padded) and the per packet overhead on your link. You can either get this information from your ISP and/or from the DSL-modem’s information page, but both are not guaranteed to be available/useful. So I set  out to empirically deduce this information from measurements on my own link. I naively started out with using ICMP echo requests as probes (as I easily could generate probe packets with different sizes with the linux/macosx ping binary), as it turned out, this works well enough, at least for relatively slow ADSL-links. So ping_sweeper6.sh (attached) is the program I use (on an otherwise idle link, typically over night) to collect ~1000 repetitions of time stamped ping packets spanning two (potential) ATM cells. I then use tc_stab_parameter_guide.m (a matlab/octave program) to read in the output of the ping_sweeper script and process the data. In short if the link runs ATM encapsulation the plot of the data needs to look like a stair with 48 byte step width, if it is just smoothly increasing the carrier is not ATM. For ATM links and only ATM links, the script also tries to figure out the per packet overhead which always worked well for me. (My home-link got recently a silent upgrade where the encapsulation changed from 40 bytes to 44 bytes (probably due to the introduction of VLAN tags), which caused some disturbances in link capacity measurements I was running at the time; so I ran my code again and lo and behold the overhead had increased, which caused the issues with the measurements, as after taking the real overhead into account the disturbances went away, but I guess I digress ;) )


Best Regards
	Sebastian


> 
> On Sat, Sep 20, 2014 at 7:17 PM, Andy Furniss <[email protected]> wrote:
>> Alan Goodman wrote:
>>> 
>>> Hi,
>>> 
>>> I am looking to figure out the most fool proof way to calculate stab
>>> overheads for ADSL/VDSL connections.
>>> 
>>> ppp0      Link encap:Point-to-Point Protocol inet addr:81.149.38.69
>>> P-t-P:81.139.160.1 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP
>>> MULTICAST  MTU:1492  Metric:1 RX packets:17368223 errors:0 dropped:0
>>> overruns:0 frame:0 TX packets:12040295 errors:0 dropped:0 overruns:0
>>> carrier:0 collisions:0 txqueuelen:100 RX bytes:17420109286 (16.2 GiB)
>>> TX bytes:3611007028 (3.3 GiB)
>>> 
>>> I am setting a longer txqueuelen as I am not currently using any fair
>>> queuing (buffer bloat issues with sfq)
>> 
>> 
>> Whatever is txqlen is on ppp there is likely some other buffer after it
>> - the default can hurt with eg, htb as if you don't add qdiscs to
>> classes it takes (last time I looked) its qlen from that.
>> 
>> Sfq was only ever meant for bulk, so should really be in addition to
>> some classification to separate interactive - I don't really get the
> 
> Hmm? sfq separates bulk from interactive pretty nicely. It tends to do
> bad things to bulk as it doesn't manage queue length.
> 
> A little bit of prioritization or deprioritization for some traffic is
> helpful, but most traffic is hard to classify.
> 
>> bufferbloat bit, you could make the default 128 limit lower if you wanted.
> 
> htb + fq_codel, if available, is the right thing here....
> 
> http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die
> 
>>> The connection is a BT Infinity FTTC VDSL connection synced at
>>> 80mbit/20mbit.  The modem is connected directly to the ethernet port
>>> on a server running a slightly tweaked HFSC setup that you folks
>>> helped me set up in July - back when I was on ADSL.  I am still
>>> running pppoe I believe from my server.
>> 
>> 
>> I have similar since May 2013 and I still haven't got round to reading
>> up on everything yet :-)
>> 
>> I have extra geek score for using mini jumbos = running pppoe with mtu
>> 1500 which works for me on plusnet. You need a recent pppd for this and
>> a nic that works with mtu >= 1508.
>> 
>> As for overheads, initial searching indicated that it's not easy or
>> maybe even truly possible like adsl.
>> 
>>> The largest ping packet that I can fit out onto the wire is 1464
>>> bytes:
>>> 
>>> # ping -c 2 -s 1464 -M do google.com PING google.com (31.55.166.216)
>>> 1464(1492) bytes of data. 1472 bytes from 31.55.166.216: icmp_seq=1
>>> ttl=58 time=11.7 ms 1472 bytes from 31.55.166.216: icmp_seq=2 ttl=58
>>> time=11.9 ms
>>> 
>>> # ping -c 2 -s 1465 -M do google.com PING google.com (31.55.166.212)
>>> 1465(1493) bytes of data. From
>>> host81-149-38-69.in-addr.btopenworld.com (81.149.38.69) icmp_seq=1
>>> Frag needed and DF set (mtu = 1492) From
>>> host81-149-38-69.in-addr.btopenworld.com (81.149.38.69) icmp_seq=1
>>> Frag needed and DF set (mtu = 1492)
>> 
>> 
>> You can't work out your overheads like this.
>> 
>> On slow uplink adsl it was possible with ping to infer the fixed part
>> but you needed to send loads of pings increasing in size and plot the
>> best time for each to make a stepped graph.
>> 
>> 
>>> Based on this I believe overhead should be set to 28, however with 28
>>> set as my overhead and hfsc ls m2 20000kbit ul m2 20000kbit I seem
>>> to be loosing about 1.5mbit of upload...
>> 
>> 
>> Even if you could do things perfectly I would back off a few kbit just
>> to be safe. Timers may be different or there may be OAM/Reporting data
>> going up, albeit rarely.
>> 
>>> 
>>> No traffic manager enabled:
>>> 
>>> http://www.thinkbroadband.com/speedtest/results.html?id=141116089424883990118
>>> 
>>> 
>>> HFSC traffic manager:
>>> 
>>> http://www.thinkbroadband.com/speedtest/results.html?id=141116216621093133034
>>> 
>>> 
>>> 
>>> Am I calculating overhead incorrectly?
>> 
>> 
>> VDSL doesn't use ATM I think the PTM it uses is 64/65 - so don't specify
>> atm with stab. Unfortunately stab doesn't do 64/65.
>> 
>> As for the fixed part - I am not sure, but roughly starting with IP as
>> that's what tc sees on ppp (as opposed to ip + 14 on eth)
>> 
>> IP
>> +8 for PPPOE
>> +14 for ethertype and macs
>> +4 because Openreach modem uses vlan
>> +2 CRC ??
>> + "a few" 64/65
>> 
>> That's it for fixed - of course 64/65 adds another one for every 64 TBH
>> I didn't get the precice detail from the spec and not having looked
>> recently I can't remember.
>> 
>> BT Sin 498 does give some of this info and a couple of examples of
>> throughput for different frame sizes - but it's rounded to kbit which
>> means I couldn't work out to the byte what the overheads were.
>> 
>> Worse still VDSL can use link layer retransmits and the sin says that
>> though currently (2013) not enabled, they would be in due course. I have
>> no clue how these work.
>> 
>> 
>> 
>> --
>> To unsubscribe from this list: send the line "unsubscribe lartc" in
>> the body of a message to [email protected]
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 
> -- 
> Dave Täht
> 
> https://www.bufferbloat.net/projects/make-wifi-fast
> _______________________________________________
> Cerowrt-devel mailing list
> [email protected]
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

#! /bin/bash
# TODO use seq or bash to generate a list of the requested sizes (to alow for non-equdistantly spaced sizes)

# just an identifier for the ping log
TECH=ADSL
# finding a proper target IP is somewhat of an art, just traceroute a remote site 
# and find the nearest host reliably responding to pings showing the smallet variation of pingtimes
# for this I typically run "traceroute 8.8.8.8", and then select the first host on the ISP side (typically after 
# the first large RTT increment) and test its response by "ping -c 10 -s 16 NNN.NNN.NNN.NNN", if this host does not repsond 
# I pick the next host along the route to 8.8.8.8. I assume the closer the host the less disturbed by other traffic the 
# response will be.


if [ ! $# == 1 ]; then
    echo "To run measurements supply the TARGET IP address as first agument to ${0} this script."
    echo "Use traceroute 8.8.8.8 to get a list of increasingly distant hosts, pick the first host out of your network (ideally the DSLAM)."
    echo "Test whether the selected host responds to ping: 'ping -s16 -c 1 target.IP.address.quad' : this needs to actually return non zero RTTs."
    echo "If the hosts does not reply to the pings take the next host from the traceroute (movin closer to 8.8.8.8), repeat until you find a replying host."
    echo "Once the main script is started have a quick look at the logfile, to see whether the RTTs stay close to the initial test RTT."
    echo "If the RTTs have increased a lot, the PINGPERIOD might be too short, and the host might have put us on a slow path; either increase PINGPERIOD or try the next host..."
    echo ""
    echo "Here is the traceroute (might take a while):"
    echo ""
    traceroute 8.8.8.8
    
    exit 0
else
    TARGET=${1}		# Replace by an appropriate host
fi


DATESTR=`date +%Y%m%d_%H%M%S`	# to allow multiple sequential records
LOG=ping_sweep_${TECH}_${DATESTR}.txt

MAX_PREIP_OVERHEAD_SIZE=44	# as far as I can tell 44 bytes is the maximum pre IP header overhead for an ATM based carrier
IP4_HEADER_SIZE=20		# 20 bytes
IDEAL_MTU=1500			#  what the MTU should look like

# by default non-root ping will only end one packet per second, so work around that by calling ping independently for each package
# empirically figure out the shortest period still giving the standard ping time (to avoid being slow-pathed by our host)
# at 100 packets/s of 116 + 28 + 40 we would need 4 ATM cells = 192byte * 100/s = 150kbit/s
# at 100 packets/s of 16 + 28 + 40nwe would need 2 ATM cells = 96byte * 100/s = 75kbit/s
# on average we need 150 + 75 * 0.5 = 112.5 Kbit/s, increase the ping period if uplinh < 112.5 Kbit/s
PINGPERIOD=0.01		# reduce if uplink slower than roughly 200Kbit/s
PINGSPERSIZE=10000	# the higher the link rate the more samples we need to reliably detect the increasingly smaller ATM quantisation steps. Can be reduced for slower links

# Start, needed to find the per packet overhead dependent on the ATM encapsulation
# to reiably show ATM quantization one would like to see at least two steps, so cover a range > 2 ATM cells (so > 96 bytes)
SWEEPMINSIZE=16		# 64bit systems seem to require 16 bytes of payload to include a timestamp...
SWEEPMAXSIZE=116
SWEEPMAXSIZE=216
    

n_SWEEPS=`expr ${SWEEPMAXSIZE} - ${SWEEPMINSIZE}`


i_sweep=0
i_size=0

while [ ${i_sweep} -lt ${PINGSPERSIZE} ]
do
    (( i_sweep++ ))
    echo "Current iteration: ${i_sweep}"
    # now loop from sweepmin to sweepmax
    i_size=${SWEEPMINSIZE}
    while [ ${i_size} -le ${SWEEPMAXSIZE} ]
    do
	echo "${i_sweep}. repetition of ping size ${i_size}"
	ping -c 1 -s ${i_size} ${TARGET} >> ${LOG} &
	(( i_size++ ))
	# we need a sleep binary that allows non integer times (GNU sleep is fine as is sleep of macosx 10.8.4)
	sleep ${PINGPERIOD}
    done
done

#tail -f ${LOG}

echo "Done... ($0)
"