Re: I-D Action: draft-ietf-bmwg-dcbench-terminology-02.txt

Regarding the Definition of Goodput, BWMG has one in RFC2647 already,
and we had this discussion about it last year:

https://www.ietf.org/mail-archive/web/bmwg/current/msg03094.html

how time flies,
Al

> -----Original Message-----
> From: bmwg [mailto:[email protected]] On Behalf Of MORTON, ALFRED C
> (AL)
> Sent: Tuesday, February 23, 2016 12:35 PM
> To: Lucien Avramov (lavramov); Jacob Rapp; [email protected]
> Subject: Re: [bmwg] I-D Action: draft-ietf-bmwg-dcbench-terminology-
> 02.txt
>
> *** Security Advisory: This Message Originated Outside of AT&T ***.
> Reference http://cso.att.com/EmailSecurity/IDSP.html for more
> information.
>
> Hi Lucien and Jacob,
>
> Thanks for the quick update to the terminology draft.
>
> Are there now places where the Methodology draft needs
> to be updated to stay in sync with terminology?
>
>
> I have a couple of comments on the methodology, when I scanned now:
>
> Nit:
>    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
>    "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
>    document are to be interpreted as described in RFC 2119 [6].
> Ref [6] is missing.
> -=-=-=-=-=-=-=-
>
> 6.3 Reporting Format
>
>    The report MUST include the following:
>
>    - Number of ingress and egress ports along with designation of
>    stateful or stateless.
>
>    - Stateful goodput
>
> I think you are referring to the "Data Center Goodput" defined here:
> https://tools.ietf.org/html/draft-ietf-bmwg-dcbench-terminology-
> 04#section-7
>
> so at least we need a reference, and it's likely that BMWG'ers
> (including me) will have a few comments on the Goodput Definition in
> the terminology draft. (It seems to me we might already have such a
> definition, and you can just refer to it, if it is suitable).
>
> -=-=-=-=-=-=-=-=-=-
>
>
> Also, this reference:
> 7.3.  URL References
>
>    [5]  Yanpei Chen, Rean Griffith, Junda Liu, Randy H. Katz, Anthony D.
>          Joseph, "Understanding TCP Incast Throughput Collapse in
>          Datacenter Networks",
>
> http://www.eecs.berkeley.edu/~ychen2/professional/TCPIncastWREN2009.pdf"
> .
>
> should be listed in the Informative References section, IF you plan to
> use it
> in the draft - I didn't find a reference to [5] in the text.
>
> Al
>
>
> > -----Original Message-----
> > From: Lucien Avramov (lavramov) [mailto:[email protected]]
> > Sent: Tuesday, February 23, 2016 2:31 AM
> > To: MORTON, ALFRED C (AL); Jacob Rapp; [email protected]
> > Subject: Re: [bmwg] I-D Action: draft-ietf-bmwg-dcbench-terminology-
> > 02.txt
> >
> > Thank you very much Al!
> >
> > I just incorporated your suggestion in the -04 version I posted:
> > https://www.ietf.org/rfcdiff?url2=draft-ietf-bmwg-dcbench-terminology-
> 04
> >
> > Yes, we did spend time on Scott's review word document and we
> > incorporated changes in the -03 version and send our thoughts also on
> a
> > the review of his word document to the mailer on 7/1/16 for
> traceability
> > purposes and to not loose all the valuable time Scott spend on
> reviewing
> > our draft. (thank you for checking)
> >
> > We want to thank BMWG for being eager to provide us so much support
> and
> > reviews over the years on our data center benchmarking work.
> >
> > The authors
> >
> >
> > On 2/23/16 8:14 AM, MORTON, ALFRED C (AL) wrote:
> > > Hi Lucien and Jacob,
> > >
> > > Thanks for your updates to the dcbench terminology draft,
> > > both updates in fact, 02 on Jan 6 and 03 on Jan 7.
> > >
> > > I have one editorial suggestion which you can address in a future
> > version:
> > > In section 3.1, the word "assimilated" doesn't fit this sentence (to
> > me).
> > >
> > > OLD
> > > Jitter in the data center context is also assimilated to common
> > term...
> > > suggest
> > > Jitter in the data center context is synonymous with the common
> > term...
> > >
> > > Otherwise, section 3 looks good.
> > >
> > > I think you indicated that Scott Bradner's comments on the draft
> have
> > been
> > > addressed? Scott sent his notes in an MSword file to the list on Nov
> > 3,
> > > and the discussion of his comments is captured in the IETF-94
> meeting
> > minutes:
> > > https://www.ietf.org/proceedings/94/minutes/minutes-94-bmwg
> > > (right up front). There may be more details in the minutes.
> > >
> > > Would you please confirm that Scott's comments have been addressed
> > > or otherwise answered?
> > >
> > > thanks, and sorry for delay on this reply,
> > > Al
> > >
> > >
> > >> -----Original Message-----
> > >> From: bmwg [mailto:[email protected]] On Behalf Of Lucien
> Avramov
> > >> (lavramov)
> > >> Sent: Thursday, January 07, 2016 5:42 PM
> > >> To: Jacob Rapp; [email protected]
> > >> Subject: Re: [bmwg] I-D Action: draft-ietf-bmwg-dcbench-
> terminology-
> > >> 02.txt
> > >>
> > >> BMWG,
> > >>
> > >> The document is now updated with also Al's comment about section 3)
> > >> Jitter / Delay variation. As Jacob said as authors we strongly feel
> > >> about wanting to still call this type of measurement jitter as this
> > is
> > >> how we see it charecterized in the data center. So I tried to
> reflect
> > it
> > >> by the first sentence in the definition after Al's comments.
> > >>
> > >> https://www.ietf.org/internet-drafts/draft-ietf-bmwg-dcbench-
> > >> terminology-03.txt
> > >>
> > >>
> > >> On 1/6/16 5:01 PM, Jacob Rapp wrote:
> > >>> First thank you everyone for the detailed feedback! Below is the
> > >> consolidated reply to the feedback.
> > >>>
> > >>> The draft draft-ietf-bmwg-dcbench-terminology-02 reflects all the
> > >> changes.
> > >>>
> > >>> Detail:
> > >>>
> > >>> We have updated the drafts with the feedback as well per this
> > updated
> > >> draft.
> > >>>
> > >>> Scott's Review and feedback: Attached is the doc with the reply
> with
> > >> all the feedback and updates we plan to make.
> > >>>
> > >>> Al's Feedback:
> > >>> <LA/JR> The suggestions below are all great and we will make the
> > >> updates accordingly. The only thing, is we would like to leave it
> > called
> > >> "Jitter". The original purpose inclusion was to specifically define
> > the
> > >> term "Jitter". We have seen a lot of cases where "jitter" is used
> in
> > >> many different ways which has lead to a lot of confusion.
> > >>>
> > >>> In the latest draft of Data Center Terminology -01:
> > >>> - using the term Jitter, should be delay variation now
> > >>> - still citing RFC 3393 almost exclusively, which is too flexible
> > >>> - only citing IPDV from RFC 5481, <JR/LA> Agreed, will update with
> > >> your suggestion below.
> > >>> - in the past meeting we agreed on RFC 5481 PDV as primary, IPDV
> > only
> > >> if you  below.
> > >>>     want to understand if traffic has become more bursty in
> transfer
> > >>> - Metric Units section is the place to specify Units of Measure
> > >> (seconds
> > >>>     in this case, not much else)
> > >>> I suggest to revise section 3 as follows:
> > >>> NEW:
> > >>>
> > >>> 3 Delay Variation
> > >>>
> > >>> 3.1 Definition
> > >>>
> > >>> Delay variation is derived from multiple measurements
> > >>> of one-way delay, as described in RFC 3393.
> > >>> The mandatory definition of Delay Variation is the PDV form
> > >>> from section 4.2 of RFC 5481. When considering a stream of
> packets,
> > >>> the delays of all packets are subtracted from the minimum delay
> > >>> over all packets in the stream. This facilitates assessment of
> > >>> the range of delay variation (Max - Min), or a high percentile
> > >>> of PDV (99th percentile, for robustness against outliers).
> > >>>
> > >>> If First-bit to Last-bit timestamps are used for Delay
> measurement,
> > >> then
> > >>> Delay Variation MUST be measured using packets or frames of the
> > >>> same size, since the definition of latency includes the
> > serialization
> > >>> time for each packet. Otherwise if using First-bit to First-bit,
> > >>> the size restriction does not apply.
> > >>>
> > >>>
> > >>> 3.2 Discussion
> > >>>      In addition to PDV Range and or a high percentile of PDV,
> > >>>      Inter-Packet Delay Variation (IPDV) as defined in section 4.1
> > of
> > >>>      RFC5481 (differences between two consecutive packets) MAY be
> > used
> > >>>      for the purpose of determining how packet spacing has changed
> > >> during
> > >>>      transfer, for example to see if packet stream has become
> > closely-
> > >> spaced
> > >>>      or "bursty". However, the Absolute Value of IPDV SHOULD NOT
> be
> > >> used, as
> > >>>      this collapses the "busrty" and "dispersed" sides of the IPDV
> > >> distribution
> > >>>      together.
> > >>>
> > >>> 3.3 Measurement Units
> > >>>
> > >>> The measurement of delay variation is expressed in units of
> seconds.
> > >>>
> > >>> A PDV histogram MAY be provided for the population of packets
> > >> measured.
> > >>>
> > >>> -=-=-=-=-=-=-=-=-=-
> > >>> OLD:
> > >>> 3 Jitter
> > >>>
> > >>> 3.1 Definition
> > >>>
> > >>>      The definition of Jitter is covered extensively in RFC 3393.
> > This
> > >>>      definition is not meant to replace that definition, but it is
> > >> meant
> > >>>      to provide guidance of use for data center network devices.
> > >>>
> > >>>      The use of Jitter is in according with the variation delay
> > >> definition
> > >>>      from RFC 3393:
> > >>>
> > >>>      The second meaning has to do with the variation of a metric
> > (e.g.,
> > >>>      delay) with respect to some reference metric (e.g., average
> > delay
> > >> or
> > >>>      minimum delay).  This meaning is frequently used by computer
> > >>>      scientists and frequently (but not always) refers to
> variation
> > in
> > >>>      delay.
> > >>>
> > >>>      Even with the reference to RFC 3393, there are many
> definitions
> > of
> > >>>      "jitter" possible. The one selected for Data Center
> > Benchmarking
> > >> is
> > >>>      closest to RFC 3393.
> > >>>
> > >>>
> > >>> 3.2 Discussion
> > >>>
> > >>>      Jitter can be measured in different scenarios:-packet to
> packet
> > >> delay
> > >>>      variation-delta between min and max packet delay variation
> for
> > all
> > >>>      packets sent.
> > >>>
> > >>> 3.3 Measurement Units
> > >>>
> > >>>      The jitter MUST be measured when sending packets of the same
> > size.
> > >>>      Jitter MUST be measured as packet to packet delay variation
> and
> > >> delta
> > >>>      between min and max packet delay variation of all packets
> sent.
> > A
> > >>>      histogram MAY be provided as a population of packets measured
> > per
> > >>>      latency or latency buckets. Inter-Packet Delay Variation as
> > >> defined
> > >>>      in RFC5481 (differences between two consecutive packets) MAY
> be
> > >> used
> > >>>      for this purpose.
> > >>>
> > >>>
> > >>> Marius Feedback:<LA/JR> Agreed. This was some of Scott's early
> > >> feedback as well. We added in specific language to cover number of
> > >> trials as well.
> > >>> Example: "*For each test methodology described, it is key to
> obtain
> > >> repeatability of the results. The recommendation is to perform
> enough
> > >> iterations of the given test to make sure the result is accurate,
> > this
> > >> is especially important for section 3) as the buffering testing has
> > been
> > >> historically the least reliable."
> > >>>
> > >>> Marius Georgescu: I just have a short comment about repeatability.
> > >>> Essentially they talk about variation. I think they should be
> > talking
> > >>> about variance. It's good to have min and max, but the variance is
> > >>> also accounting for the number of trials. Also, about the
> > summarizing
> > >>> function, I think we should have that discussion as a working
> group
> > >>> and will try to cover it in my presentation.
> > >>>
> > >>> -Jacob Rapp & Lucien Avramov
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On 1/6/16, 4:58 PM, "bmwg on behalf of [email protected]"
> > >> <[email protected] on behalf of [email protected]>
> wrote:
> > >>>
> > >>>>
> > >>>> A New Internet-Draft is available from the on-line Internet-
> Drafts
> > >> directories.
> > >>>> This draft is a work item of the Benchmarking Methodology Working
> > >> Group of the IETF.
> > >>>>
> > >>>>          Title           : Data Center Benchmarking Terminology
> > >>>>          Authors         : Lucien Avramov
> > >>>>                            Jacob Rapp
> > >>>>        Filename        : draft-ietf-bmwg-dcbench-terminology-02.txt
> > >>>>        Pages           : 15
> > >>>>        Date            : 2016-01-06
> > >>>>
> > >>>> Abstract:
> > >>>> The purpose of this informational document is to establish
> > >> definitions,
> > >>>> discussion and measurement techniques for data center
> benchmarking.
> > >>>> Also, it is to introduce new terminologies applicable to data
> > center
> > >>>> performance evaluations. The purpose of this document is not to
> > >> define
> > >>>> the test methodology, but rather establish the important concepts
> > >> when
> > >>>> one is interested in benchmarking network switches and routers in
> > the
> > >>>> data center.
> > >>>>
> > >>>>
> > >>>> The IETF datatracker status page for this draft is:
> > >>>> https://urldefense.proofpoint.com/v2/url?u=https-
> > >> 3A__datatracker.ietf.org_doc_draft-2Dietf-2Dbmwg-2Ddcbench-
> > >> 2Dterminology_&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-
> > >>
> >
> uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8
> > >> tU&s=KBX60CyHXvKQA-6xRIrVa_MkB8r_w2NR0LES2kjyeno&e=
> > >>>>
> > >>>> There's also a htmlized version available at:
> > >>>> https://urldefense.proofpoint.com/v2/url?u=https-
> > >> 3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Ddcbench-
> 2Dterminology-
> > >> 2D02&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-
> > >>
> >
> uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8
> > >> tU&s=MpAy2_x5iROY5UauQybc8zbhfenLpqX0fe1bSdk-1Mw&e=
> > >>>>
> > >>>> A diff from the previous version is available at:
> > >>>> https://urldefense.proofpoint.com/v2/url?u=https-
> > >> 3A__www.ietf.org_rfcdiff-3Furl2-3Ddraft-2Dietf-2Dbmwg-2Ddcbench-
> > >> 2Dterminology-2D02&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-
> > YihVMNtXt-
> > >>
> >
> uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8
> > >> tU&s=9aTPOGU6rnv5EV3nfIJ7_CzB2NE7Bpi32M3Z039PaNM&e=
> > >>>>
> > >>>>
> > >>>> Please note that it may take a couple of minutes from the time of
> > >> submission
> > >>>> until the htmlized version and diff are available at
> > tools.ietf.org.
> > >>>>
> > >>>> Internet-Drafts are also available by anonymous FTP at:
> > >>>> https://urldefense.proofpoint.com/v2/url?u=ftp-
> > >> 3A__ftp.ietf.org_internet-
> > >> 2Ddrafts_&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-
> > >>
> >
> uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8
> > >> tU&s=Jz3dA8OqSDgiHOalJ8c8WuAl2XuIK4pxYq4014WinKo&e=
> > >>>>
> > >>>> _______________________________________________
> > >>>> bmwg mailing list
> > >>>> [email protected]
> > >>>> https://urldefense.proofpoint.com/v2/url?u=https-
> > >>
> >
> 3A__www.ietf.org_mailman_listinfo_bmwg&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiD
> > >> JAXVeAw-YihVMNtXt-
> > >>
> >
> uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8
> > >> tU&s=7S-JdXwgqeVkDUxz6DxWc8oPEXFYi7mrWSy1A7DEcKQ&e=
> > >>>>
> > >>>>
> > >>>> _______________________________________________
> > >>>> bmwg mailing list
> > >>>> [email protected]
> > >>>> https://www.ietf.org/mailman/listinfo/bmwg
> > >>
> > >> _______________________________________________
> > >> bmwg mailing list
> > >> [email protected]
> > >> https://www.ietf.org/mailman/listinfo/bmwg
>
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg