Re: I-D Action: draft-ietf-bmwg-dcbench-terminology-02.txt

BMWG,

The document is now updated with also Al's comment about section 3) 
Jitter / Delay variation. As Jacob said as authors we strongly feel 
about wanting to still call this type of measurement jitter as this is 
how we see it charecterized in the data center. So I tried to reflect it 
by the first sentence in the definition after Al's comments.

https://www.ietf.org/internet-drafts/draft-ietf-bmwg-dcbench-terminology-03.txt


On 1/6/16 5:01 PM, Jacob Rapp wrote:
> First thank you everyone for the detailed feedback! Below is the consolidated reply to the feedback.
>
> The draft draft-ietf-bmwg-dcbench-terminology-02 reflects all the changes.
>
> Detail:
>
> We have updated the drafts with the feedback as well per this updated draft.
>
> Scott’s Review and feedback: Attached is the doc with the reply with all the feedback and updates we plan to make.
>
> Al’s Feedback:
> <LA/JR> The suggestions below are all great and we will make the updates accordingly. The only thing, is we would like to leave it called “Jitter”. The original purpose inclusion was to specifically define the term “Jitter”. We have seen a lot of cases where “jitter” is used in many different ways which has lead to a lot of confusion.
>
> In the latest draft of Data Center Terminology -01:
> - using the term Jitter, should be delay variation now
> - still citing RFC 3393 almost exclusively, which is too flexible
> - only citing IPDV from RFC 5481, <JR/LA> Agreed, will update with your suggestion below.
> - in the past meeting we agreed on RFC 5481 PDV as primary, IPDV only if you  below.
>    want to understand if traffic has become more bursty in transfer
> - Metric Units section is the place to specify Units of Measure (seconds
>    in this case, not much else)
> I suggest to revise section 3 as follows:
> NEW:
>
> 3 Delay Variation
>
> 3.1 Definition
>
> Delay variation is derived from multiple measurements
> of one-way delay, as described in RFC 3393.
> The mandatory definition of Delay Variation is the PDV form
> from section 4.2 of RFC 5481. When considering a stream of packets,
> the delays of all packets are subtracted from the minimum delay
> over all packets in the stream. This facilitates assessment of
> the range of delay variation (Max - Min), or a high percentile
> of PDV (99th percentile, for robustness against outliers).
>
> If First-bit to Last-bit timestamps are used for Delay measurement, then
> Delay Variation MUST be measured using packets or frames of the
> same size, since the definition of latency includes the serialization
> time for each packet. Otherwise if using First-bit to First-bit,
> the size restriction does not apply.
>
>
> 3.2 Discussion
>     In addition to PDV Range and or a high percentile of PDV,
>     Inter-Packet Delay Variation (IPDV) as defined in section 4.1 of
>     RFC5481 (differences between two consecutive packets) MAY be used
>     for the purpose of determining how packet spacing has changed during
>     transfer, for example to see if packet stream has become closely-spaced
>     or "bursty". However, the Absolute Value of IPDV SHOULD NOT be used, as
>     this collapses the "busrty" and "dispersed" sides of the IPDV distribution
>     together.
>
> 3.3 Measurement Units
>
> The measurement of delay variation is expressed in units of seconds.
>
> A PDV histogram MAY be provided for the population of packets measured.
>
> -=-=-=-=-=-=-=-=-=-
> OLD:
> 3 Jitter
>
> 3.1 Definition
>
>     The definition of Jitter is covered extensively in RFC 3393. This
>     definition is not meant to replace that definition, but it is meant
>     to provide guidance of use for data center network devices.
>
>     The use of Jitter is in according with the variation delay definition
>     from RFC 3393:
>
>     The second meaning has to do with the variation of a metric (e.g.,
>     delay) with respect to some reference metric (e.g., average delay or
>     minimum delay).  This meaning is frequently used by computer
>     scientists and frequently (but not always) refers to variation in
>     delay.
>
>     Even with the reference to RFC 3393, there are many definitions of
>     "jitter" possible. The one selected for Data Center Benchmarking is
>     closest to RFC 3393.
>
>
> 3.2 Discussion
>
>     Jitter can be measured in different scenarios:-packet to packet delay
>     variation-delta between min and max packet delay variation for all
>     packets sent.
>
> 3.3 Measurement Units
>
>     The jitter MUST be measured when sending packets of the same size.
>     Jitter MUST be measured as packet to packet delay variation and delta
>     between min and max packet delay variation of all packets sent. A
>     histogram MAY be provided as a population of packets measured per
>     latency or latency buckets. Inter-Packet Delay Variation as defined
>     in RFC5481 (differences between two consecutive packets) MAY be used
>     for this purpose.
>
>
> Marius Feedback:<LA/JR> Agreed. This was some of Scott’s early feedback as well. We added in specific language to cover number of trials as well.
> Example: "•For each test methodology described, it is key to obtain repeatability of the results. The recommendation is to perform enough iterations of the given test to make sure the result is accurate, this is especially important for section 3) as the buffering testing has been historically the least reliable.”
>
> Marius Georgescu: I just have a short comment about repeatability.
> Essentially they talk about variation. I think they should be talking
> about variance. It's good to have min and max, but the variance is
> also accounting for the number of trials. Also, about the summarizing
> function, I think we should have that discussion as a working group
> and will try to cover it in my presentation.
>
> —Jacob Rapp & Lucien Avramov
>
>
>
>
>
>
>
>
> On 1/6/16, 4:58 PM, "bmwg on behalf of [email protected]" <[email protected] on behalf of [email protected]> wrote:
>
>>
>> A New Internet-Draft is available from the on-line Internet-Drafts directories.
>> This draft is a work item of the Benchmarking Methodology Working Group of the IETF.
>>
>>         Title           : Data Center Benchmarking Terminology
>>         Authors         : Lucien Avramov
>>                           Jacob Rapp
>> 	Filename        : draft-ietf-bmwg-dcbench-terminology-02.txt
>> 	Pages           : 15
>> 	Date            : 2016-01-06
>>
>> Abstract:
>> The purpose of this informational document is to establish definitions,
>> discussion and measurement techniques for data center benchmarking.
>> Also, it is to introduce new terminologies applicable to data center
>> performance evaluations. The purpose of this document is not to define
>> the test methodology, but rather establish the important concepts when
>> one is interested in benchmarking network switches and routers in the
>> data center.
>>
>>
>> The IETF datatracker status page for this draft is:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Dietf-2Dbmwg-2Ddcbench-2Dterminology_&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8tU&s=KBX60CyHXvKQA-6xRIrVa_MkB8r_w2NR0LES2kjyeno&e=
>>
>> There's also a htmlized version available at:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Ddcbench-2Dterminology-2D02&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8tU&s=MpAy2_x5iROY5UauQybc8zbhfenLpqX0fe1bSdk-1Mw&e=
>>
>> A diff from the previous version is available at:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_rfcdiff-3Furl2-3Ddraft-2Dietf-2Dbmwg-2Ddcbench-2Dterminology-2D02&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8tU&s=9aTPOGU6rnv5EV3nfIJ7_CzB2NE7Bpi32M3Z039PaNM&e=
>>
>>
>> Please note that it may take a couple of minutes from the time of submission
>> until the htmlized version and diff are available at tools.ietf.org.
>>
>> Internet-Drafts are also available by anonymous FTP at:
>> https://urldefense.proofpoint.com/v2/url?u=ftp-3A__ftp.ietf.org_internet-2Ddrafts_&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8tU&s=Jz3dA8OqSDgiHOalJ8c8WuAl2XuIK4pxYq4014WinKo&e=
>>
>> _______________________________________________
>> bmwg mailing list
>> [email protected]
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_bmwg&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8tU&s=7S-JdXwgqeVkDUxz6DxWc8oPEXFYi7mrWSy1A7DEcKQ&e=
>>
>>
>> _______________________________________________
>> bmwg mailing list
>> [email protected]
>> https://www.ietf.org/mailman/listinfo/bmwg