Re: comments on draft-dcbench-def-00 (part 1)

Hi Al,

I'd like to explain further, please see inline:


On 10/24/13, 12:35 PM, MORTON, ALFRED C (AL) wrote:
> Hi Jacob, thanks for joining in.
>
> you wrote:
>> 2. There are two important consideration.
>> 	2a. Packet to Packet delay variation
>
> Just to be clear, you take the one-way delay of packet "x",
> and subtract the one-way delay of packet "x-1", and that
> yields a single value of delay variation (one of many definitions).
>
> The sequence of packets and their delays is important to this metric,
> because we never compare the one-way delay of packet "x" to
> any other packet except "x-1".

absolutely, it's we should compare two packets of the 'same' size and x 
compared to x-1, we will clarify this. the metric here measured is the 
'absolute delta=[x-(x-1)]'=jitter.


>
> So, if I have your definition right, what performance related question
> does the 2a metric help to answer?  What physical quantity does it define?

It tries to define in my opinion a more precise jitter variation were we 
state that jitter is only significant between x and x-1 packets, when x 
and x-1 have the *same* packet characteristics.

>
> I'd prefer that we had some compelling reasons to choose one definition
> over the others.  If e2e estimation of n-DUT delay variation is one of our
> goals, this definition won't help us, AFAIK.
>
>> 	2b. The min and max
>
> I think we are on the same page with 2b, min and max,
> but not quite the same paragraph yet.
> These are the extremes of the one-way delay
> histogram, which exhibits some spread between min and max one-way delay...
>
> In your example, the min and max were the one-way delay values (12 and 15ns),
> so the delay *variation* was 3ns maximum (min is zero, that's the basis for
> variation in the definition I'm using here, every one-way delay value has the
> minimum delay subtracted to yield the variation).
>
> This definition of delay variation (subtract the minimum) can support the
> e2e estimation of n-DUT delay variation.

just to be sure I am clear, what we try to say is that the DUT can yield 
actually difference results for the variation 'absolute delta=[x-(x-1)]' 
and there will be a lowest [min] value at the end of the test and a max.

The idea between the histogram is to represent the statistical 
population of the results so we can see what the spread is around min 
and max.


>
> So, would you agree that's a useful thing to be able to do,
> and something that data center architects might want to know?
>
> Al
> (as participant)
>
>> -----Original Message-----
>> From: [email protected] [mailto:[email protected]] On Behalf Of
>> Rapp, Jacob (HP Networking)
>> Sent: Thursday, October 24, 2013 2:50 PM
>> To: David Newman
>> Cc: [email protected]
>> Subject: Re: [bmwg] comments on draft-dcbench-def-00 (part 1)
>>
>> David,
>> I agree, the main concern is to have the single definition, but let me
>> address why we are proposing a histogram and what that histogram means
>> too.
>>
>> The biggest reason we wanted to clarify jitter is it is a term that is
>> widely used when talking about benchmarking switches and it is used in a
>> wide variety of ways. We did see RFC3393 saying avoid the term, but it
>> hasn't been followed and the common uses have expanded past what is called
>> out in RFC3393. Rather than continuing to avoid using the term, which
>> hasn't worked, we need to take the time to define it.
>>
>> The confusion I usually see when someone uses the term jitter, they mean
>> one of two things.
>> 1. Average delay variation - the average of the delta of packet arrival
>> times of the same size packets(in some cases I've heard people talk about
>> jitter between different size packets, but that would have inherent
>> different latency). Say the test consists of 100 packets sent all at 64
>> bytes. Each packet latency is measured to be a fixed value. The jitter is
>> usually expressed as say, average jitter of 5ns, meaning the delay
>> variation average of each packet is 5ns on average.
>> 2. Some people mean jitter to be in the form of max, min and average
>> latency of the same packet arrival. With our same example of 100 packets
>> sent all at 64 bytes, the min latency is 10ns average is 12ns max was
>> 15ns.
>>
>> Looking back at the draft again, I see how it needs some refinement.
>> In the end, to clear up the confusion, we settled on proposing:
>> 1. Jitter is valid only when sending packets of the same size.
>> 2. There are two important consideration.
>> 	2a. Packet to Packet delay variation
>> 	2b. The min and max
>> 3. Ultimately displaying a histogram of packet-to-packet latency change
>> throughout the entire test gives the best view. Maybe back in 2000 this
>> wasn't entirely possible, but now it is. Displaying jitter in the
>> histogram form alleviates the need to abstract out information by using
>> averages.
>>
>> --
>> Jacob Rapp
>>
>>
>> -----Original Message-----
>> From: [email protected] [mailto:[email protected]] On Behalf Of
>> David Newman
>> Sent: Thursday, October 24, 2013 10:56 AM
>> Cc: [email protected]
>> Subject: Re: [bmwg] comments on draft-dcbench-def-00 (part 1)
>>
>> On 10/24/13, 10:27 AM, MORTON, ALFRED C (AL) wrote:
>>> I can make a little time to pick-up this discussion again, so maybe we
>>> can close on some advice for the draft authors.
>>>
>>> If as David agreed below, the e2e jitter estimation is a valuable
>>> feature, then it helps to narrow the field of applicable definitions
>>> for delay variation.  We know how to do this (albeit with some
>>> mathematical
>>> complexity) for the form of delay variation distribution that is
>>> essentially the histogram of delays with the minimum delay subtracted
>> from all values.
>>> http://tools.ietf.org/html/rfc6049#section-6
>>>
>>> We don't know how to do comparable e2e estimation math for an
>>> alternate form of delay variation, usually called jitter, which
>>> emphasizes the change in delay between successive packets.
>>>
>>> Both these forms are defined and compared in
>>> http://tools.ietf.org/html/rfc5481#section-4
>>> as PDV and IPDV, respectively.
>>>
>>> Over the last two months, I've had some discussions with folks who
>>> want to support Data Center trouble-shooting with IPv6 optional header
>>> time-stamp assistance. These folks have worked to broadly inform
>>> various IETF working groups of their goals and proposed solutions
>>> (v6ops, ippm, and even tictoc for some synchronization related
>>> aspects). One thing I've learned from the exchange is that the
>>> histogram view of delay variation can assist their efforts. This may
>>> be more of an operational aspect than something we need to measure in
>>> device benchmarking, but the synergy with e2e estimation is worth
>> noting.
>>
>> I've found histograms to be very useful in lab as well as production
>> settings. There's still the question of "histograms of what?" though.
>>
>> My main concern is having enough specifity in the term "jitter" (and
>> indeed all terms in all bmwg docs) so that everyone implementing the test
>> will make the same measurement.
>>
>> The current reference, to the second definition in RFC 3393 section 1.1,
>> avoids giving a specific measurement (is it average or minimum?) and even
>> goes on to say the term jitter should be avoided.
>>
>> Though I have a bias toward the definition produced by this WG, in RFC
>> 4689, I'm less concerned with what definition we choose (3393, 4689, 6049,
>> other) than with having that definition clearly spelled out.
>>
>> Underspecification can lead to different RFC-compliant test sets producing
>> different results, and no one wants that.
>>
>> dn
>>
>>>
>>> regards,
>>> Al
>>> (as a participant)
>>>
>>>
>>>> -----Original Message-----
>>>> From: [email protected] [mailto:[email protected]] On Behalf
>>>> Of David Newman
>>>> Sent: Wednesday, July 31, 2013 2:38 PM
>>>> Cc: [email protected]
>>>> Subject: Re: [bmwg] comments on draft-dcbench-def-00 (part 1)
>>>>
>>>> On 7/31/13 11:07 AM, MORTON JR., ALFRED C (AL) wrote:
>>>>
>>>>> I assumed we have benchmarked a single device, and wanted to
>>>>> estimate how some number of these devices would affect delay
>>>>> variation when deployed on the path. Clearly, if we can benchmark a
>>>>> particular black-box SUT with a specific combination of these
>>>>> devices, there's no need to do the estimation for that case.
>>>>> But having the ability to estimate e2e variation for an unknown
>>>>> number
>>>> on the path
>>>>> seems valuable to me, and I wonder if Data Center Engineers agree.
>>>>>
>>>>> it's been a long day here, hope this makes sense,
>>>>
>>>> Mostly. Agreed 100 percent on the requirement for e2e estimation.
>>>>
>>>> Unclear how that estimate would differ from a single-box estimate
>>>> times some number of identical boxes, modulo cable lengths.
>>>>
>>>> And regardless of box count, there's still the question of which
>>>> method of jitter measurement to use.
>>>>
>>>> That was my original point: Given so many definitions to choose from,
>>>> it'd be nice to spell out one definition in this document, either
>>>> referencing something else, or defining something new (as in the use
>>>> of FILO for latency measurement).
>>>>
>>>> dn
>>>>
>>>> _______________________________________________
>>>> bmwg mailing list
>>>> [email protected]
>>>> https://www.ietf.org/mailman/listinfo/bmwg
>>
>> _______________________________________________
>> bmwg mailing list
>> [email protected]
>> https://www.ietf.org/mailman/listinfo/bmwg
>> _______________________________________________
>> bmwg mailing list
>> [email protected]
>> https://www.ietf.org/mailman/listinfo/bmwg
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg
>