Re: comments on draft-dcbench-def-00 (part 1)

Hi Jacob, thanks for joining in.

you wrote:
> 2. There are two important consideration.
> 	2a. Packet to Packet delay variation

Just to be clear, you take the one-way delay of packet "x",
and subtract the one-way delay of packet "x-1", and that 
yields a single value of delay variation (one of many definitions).

The sequence of packets and their delays is important to this metric,
because we never compare the one-way delay of packet "x" to 
any other packet except "x-1".

So, if I have your definition right, what performance related question
does the 2a metric help to answer?  What physical quantity does it define?

I'd prefer that we had some compelling reasons to choose one definition
over the others.  If e2e estimation of n-DUT delay variation is one of our
goals, this definition won't help us, AFAIK.

> 	2b. The min and max

I think we are on the same page with 2b, min and max,
but not quite the same paragraph yet. 
These are the extremes of the one-way delay
histogram, which exhibits some spread between min and max one-way delay...

In your example, the min and max were the one-way delay values (12 and 15ns),
so the delay *variation* was 3ns maximum (min is zero, that's the basis for
variation in the definition I'm using here, every one-way delay value has the 
minimum delay subtracted to yield the variation).

This definition of delay variation (subtract the minimum) can support the
e2e estimation of n-DUT delay variation.

So, would you agree that's a useful thing to be able to do,
and something that data center architects might want to know?

Al
(as participant)

> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf Of
> Rapp, Jacob (HP Networking)
> Sent: Thursday, October 24, 2013 2:50 PM
> To: David Newman
> Cc: [email protected]
> Subject: Re: [bmwg] comments on draft-dcbench-def-00 (part 1)
> 
> David,
> I agree, the main concern is to have the single definition, but let me
> address why we are proposing a histogram and what that histogram means
> too.
> 
> The biggest reason we wanted to clarify jitter is it is a term that is
> widely used when talking about benchmarking switches and it is used in a
> wide variety of ways. We did see RFC3393 saying avoid the term, but it
> hasn't been followed and the common uses have expanded past what is called
> out in RFC3393. Rather than continuing to avoid using the term, which
> hasn't worked, we need to take the time to define it.
> 
> The confusion I usually see when someone uses the term jitter, they mean
> one of two things.
> 1. Average delay variation - the average of the delta of packet arrival
> times of the same size packets(in some cases I've heard people talk about
> jitter between different size packets, but that would have inherent
> different latency). Say the test consists of 100 packets sent all at 64
> bytes. Each packet latency is measured to be a fixed value. The jitter is
> usually expressed as say, average jitter of 5ns, meaning the delay
> variation average of each packet is 5ns on average.
> 2. Some people mean jitter to be in the form of max, min and average
> latency of the same packet arrival. With our same example of 100 packets
> sent all at 64 bytes, the min latency is 10ns average is 12ns max was
> 15ns.
> 
> Looking back at the draft again, I see how it needs some refinement.
> In the end, to clear up the confusion, we settled on proposing:
> 1. Jitter is valid only when sending packets of the same size.
> 2. There are two important consideration.
> 	2a. Packet to Packet delay variation
> 	2b. The min and max
> 3. Ultimately displaying a histogram of packet-to-packet latency change
> throughout the entire test gives the best view. Maybe back in 2000 this
> wasn't entirely possible, but now it is. Displaying jitter in the
> histogram form alleviates the need to abstract out information by using
> averages.
> 
> --
> Jacob Rapp
> 
> 
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf Of
> David Newman
> Sent: Thursday, October 24, 2013 10:56 AM
> Cc: [email protected]
> Subject: Re: [bmwg] comments on draft-dcbench-def-00 (part 1)
> 
> On 10/24/13, 10:27 AM, MORTON, ALFRED C (AL) wrote:
> > I can make a little time to pick-up this discussion again, so maybe we
> > can close on some advice for the draft authors.
> >
> > If as David agreed below, the e2e jitter estimation is a valuable
> > feature, then it helps to narrow the field of applicable definitions
> > for delay variation.  We know how to do this (albeit with some
> > mathematical
> > complexity) for the form of delay variation distribution that is
> > essentially the histogram of delays with the minimum delay subtracted
> from all values.
> > http://tools.ietf.org/html/rfc6049#section-6
> >
> > We don't know how to do comparable e2e estimation math for an
> > alternate form of delay variation, usually called jitter, which
> > emphasizes the change in delay between successive packets.
> >
> > Both these forms are defined and compared in
> > http://tools.ietf.org/html/rfc5481#section-4
> > as PDV and IPDV, respectively.
> >
> > Over the last two months, I've had some discussions with folks who
> > want to support Data Center trouble-shooting with IPv6 optional header
> > time-stamp assistance. These folks have worked to broadly inform
> > various IETF working groups of their goals and proposed solutions
> > (v6ops, ippm, and even tictoc for some synchronization related
> > aspects). One thing I've learned from the exchange is that the
> > histogram view of delay variation can assist their efforts. This may
> > be more of an operational aspect than something we need to measure in
> > device benchmarking, but the synergy with e2e estimation is worth
> noting.
> 
> I've found histograms to be very useful in lab as well as production
> settings. There's still the question of "histograms of what?" though.
> 
> My main concern is having enough specifity in the term "jitter" (and
> indeed all terms in all bmwg docs) so that everyone implementing the test
> will make the same measurement.
> 
> The current reference, to the second definition in RFC 3393 section 1.1,
> avoids giving a specific measurement (is it average or minimum?) and even
> goes on to say the term jitter should be avoided.
> 
> Though I have a bias toward the definition produced by this WG, in RFC
> 4689, I'm less concerned with what definition we choose (3393, 4689, 6049,
> other) than with having that definition clearly spelled out.
> 
> Underspecification can lead to different RFC-compliant test sets producing
> different results, and no one wants that.
> 
> dn
> 
> >
> > regards,
> > Al
> > (as a participant)
> >
> >
> >> -----Original Message-----
> >> From: [email protected] [mailto:[email protected]] On Behalf
> >> Of David Newman
> >> Sent: Wednesday, July 31, 2013 2:38 PM
> >> Cc: [email protected]
> >> Subject: Re: [bmwg] comments on draft-dcbench-def-00 (part 1)
> >>
> >> On 7/31/13 11:07 AM, MORTON JR., ALFRED C (AL) wrote:
> >>
> >>> I assumed we have benchmarked a single device, and wanted to
> >>> estimate how some number of these devices would affect delay
> >>> variation when deployed on the path. Clearly, if we can benchmark a
> >>> particular black-box SUT with a specific combination of these
> >>> devices, there's no need to do the estimation for that case.
> >>> But having the ability to estimate e2e variation for an unknown
> >>> number
> >> on the path
> >>> seems valuable to me, and I wonder if Data Center Engineers agree.
> >>>
> >>> it's been a long day here, hope this makes sense,
> >>
> >> Mostly. Agreed 100 percent on the requirement for e2e estimation.
> >>
> >> Unclear how that estimate would differ from a single-box estimate
> >> times some number of identical boxes, modulo cable lengths.
> >>
> >> And regardless of box count, there's still the question of which
> >> method of jitter measurement to use.
> >>
> >> That was my original point: Given so many definitions to choose from,
> >> it'd be nice to spell out one definition in this document, either
> >> referencing something else, or defining something new (as in the use
> >> of FILO for latency measurement).
> >>
> >> dn
> >>
> >> _______________________________________________
> >> bmwg mailing list
> >> [email protected]
> >> https://www.ietf.org/mailman/listinfo/bmwg
> 
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg