Re: comments on draft-dcbench-def-00 (part 1)

David,
I agree, the main concern is to have the single definition, but let me address why we are proposing a histogram and what that histogram means too.

The biggest reason we wanted to clarify jitter is it is a term that is widely used when talking about benchmarking switches and it is used in a wide variety of ways. We did see RFC3393 saying avoid the term, but it hasn't been followed and the common uses have expanded past what is called out in RFC3393. Rather than continuing to avoid using the term, which hasn't worked, we need to take the time to define it. 

The confusion I usually see when someone uses the term jitter, they mean one of two things. 
1. Average delay variation - the average of the delta of packet arrival  times of the same size packets(in some cases I've heard people talk about jitter between different size packets, but that would have inherent different latency). Say the test consists of 100 packets sent all at 64 bytes. Each packet latency is measured to be a fixed value. The jitter is usually expressed as say, average jitter of 5ns, meaning the delay variation average of each packet is 5ns on average.
2. Some people mean jitter to be in the form of max, min and average latency of the same packet arrival. With our same example of 100 packets sent all at 64 bytes, the min latency is 10ns average is 12ns max was 15ns.

Looking back at the draft again, I see how it needs some refinement.
In the end, to clear up the confusion, we settled on proposing:
1. Jitter is valid only when sending packets of the same size.
2. There are two important consideration.
	2a. Packet to Packet delay variation
	2b. The min and max
3. Ultimately displaying a histogram of packet-to-packet latency change throughout the entire test gives the best view. Maybe back in 2000 this wasn't entirely possible, but now it is. Displaying jitter in the histogram form alleviates the need to abstract out information by using averages. 

--
Jacob Rapp


-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of David Newman
Sent: Thursday, October 24, 2013 10:56 AM
Cc: [email protected]
Subject: Re: [bmwg] comments on draft-dcbench-def-00 (part 1)

On 10/24/13, 10:27 AM, MORTON, ALFRED C (AL) wrote:
> I can make a little time to pick-up this discussion again, so maybe we 
> can close on some advice for the draft authors.
> 
> If as David agreed below, the e2e jitter estimation is a valuable 
> feature, then it helps to narrow the field of applicable definitions 
> for delay variation.  We know how to do this (albeit with some 
> mathematical
> complexity) for the form of delay variation distribution that is 
> essentially the histogram of delays with the minimum delay subtracted from all values.
> http://tools.ietf.org/html/rfc6049#section-6
> 
> We don't know how to do comparable e2e estimation math for an 
> alternate form of delay variation, usually called jitter, which 
> emphasizes the change in delay between successive packets.
> 
> Both these forms are defined and compared in
> http://tools.ietf.org/html/rfc5481#section-4
> as PDV and IPDV, respectively.
> 
> Over the last two months, I've had some discussions with folks who 
> want to support Data Center trouble-shooting with IPv6 optional header 
> time-stamp assistance. These folks have worked to broadly inform 
> various IETF working groups of their goals and proposed solutions 
> (v6ops, ippm, and even tictoc for some synchronization related 
> aspects). One thing I've learned from the exchange is that the 
> histogram view of delay variation can assist their efforts. This may 
> be more of an operational aspect than something we need to measure in 
> device benchmarking, but the synergy with e2e estimation is worth noting.

I've found histograms to be very useful in lab as well as production settings. There's still the question of "histograms of what?" though.

My main concern is having enough specifity in the term "jitter" (and indeed all terms in all bmwg docs) so that everyone implementing the test will make the same measurement.

The current reference, to the second definition in RFC 3393 section 1.1, avoids giving a specific measurement (is it average or minimum?) and even goes on to say the term jitter should be avoided.

Though I have a bias toward the definition produced by this WG, in RFC 4689, I'm less concerned with what definition we choose (3393, 4689, 6049, other) than with having that definition clearly spelled out.

Underspecification can lead to different RFC-compliant test sets producing different results, and no one wants that.

dn

> 
> regards,
> Al
> (as a participant)
> 
> 
>> -----Original Message-----
>> From: [email protected] [mailto:[email protected]] On Behalf 
>> Of David Newman
>> Sent: Wednesday, July 31, 2013 2:38 PM
>> Cc: [email protected]
>> Subject: Re: [bmwg] comments on draft-dcbench-def-00 (part 1)
>>
>> On 7/31/13 11:07 AM, MORTON JR., ALFRED C (AL) wrote:
>>
>>> I assumed we have benchmarked a single device, and wanted to 
>>> estimate how some number of these devices would affect delay 
>>> variation when deployed on the path. Clearly, if we can benchmark a 
>>> particular black-box SUT with a specific combination of these 
>>> devices, there's no need to do the estimation for that case.
>>> But having the ability to estimate e2e variation for an unknown 
>>> number
>> on the path
>>> seems valuable to me, and I wonder if Data Center Engineers agree.
>>>
>>> it's been a long day here, hope this makes sense,
>>
>> Mostly. Agreed 100 percent on the requirement for e2e estimation.
>>
>> Unclear how that estimate would differ from a single-box estimate 
>> times some number of identical boxes, modulo cable lengths.
>>
>> And regardless of box count, there's still the question of which  
>> method of jitter measurement to use.
>>
>> That was my original point: Given so many definitions to choose from, 
>> it'd be nice to spell out one definition in this document, either 
>> referencing something else, or defining something new (as in the use 
>> of FILO for latency measurement).
>>
>> dn
>>
>> _______________________________________________
>> bmwg mailing list
>> [email protected]
>> https://www.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/bmwg