Re: I-D Action: draft-ietf-bmwg-dcbench-terminology-02.txt

One of my action items from IETF-95 was to follow-up on the
definition of Goodput :
> BMWG has a definition of Goodput in RFC 2647:
> http://tools.ietf.org/html/rfc2647#section-3.17
>
>    Definition:
>      The number of bits per unit of time forwarded to the correct
>      destination interface of the DUT/SUT, minus any bits lost or
>      retransmitted.
>

We could refer to this definition, or we might clarify slightly
By deleting "loss or" in the existing definition.

In any case, let's connect our present with our past,
through references.

Thanks,
Al, as a participant

________________________________________
From: bmwg [[email protected]] On Behalf Of MORTON, ALFRED C (AL)
Sent: Wednesday, February 24, 2016 10:56 AM
To: Lucien Avramov (lavramov); Jacob Rapp; [email protected]
Subject: Re: [bmwg] I-D Action: draft-ietf-bmwg-dcbench-terminology-02.txt

*** Security Advisory: This Message Originated Outside of AT&T ***.
Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.

Regarding the Definition of Goodput, BWMG has one in RFC2647 already,
and we had this discussion about it last year:

https://www.ietf.org/mail-archive/web/bmwg/current/msg03094.html

how time flies,
Al

> -----Original Message-----
> From: bmwg [mailto:[email protected]] On Behalf Of MORTON, ALFRED C
> (AL)
> Sent: Tuesday, February 23, 2016 12:35 PM
> To: Lucien Avramov (lavramov); Jacob Rapp; [email protected]
> Subject: Re: [bmwg] I-D Action: draft-ietf-bmwg-dcbench-terminology-
> 02.txt
>
> *** Security Advisory: This Message Originated Outside of AT&T ***.
> Reference http://cso.att.com/EmailSecurity/IDSP.html for more
> information.
>
> Hi Lucien and Jacob,
>
> Thanks for the quick update to the terminology draft.
>
> Are there now places where the Methodology draft needs
> to be updated to stay in sync with terminology?
>
>
> I have a couple of comments on the methodology, when I scanned now:
>
> Nit:
>    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
>    "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
>    document are to be interpreted as described in RFC 2119 [6].
> Ref [6] is missing.
> -=-=-=-=-=-=-=-
>
> 6.3 Reporting Format
>
>    The report MUST include the following:
>
>    - Number of ingress and egress ports along with designation of
>    stateful or stateless.
>
>    - Stateful goodput
>
> I think you are referring to the "Data Center Goodput" defined here:
> https://tools.ietf.org/html/draft-ietf-bmwg-dcbench-terminology-
> 04#section-7
>
> so at least we need a reference, and it's likely that BMWG'ers
> (including me) will have a few comments on the Goodput Definition in
> the terminology draft. (It seems to me we might already have such a
> definition, and you can just refer to it, if it is suitable).
>
> -=-=-=-=-=-=-=-=-=-
>
>
> Also, this reference:
> 7.3.  URL References
>
>    [5]  Yanpei Chen, Rean Griffith, Junda Liu, Randy H. Katz, Anthony D.
>          Joseph, "Understanding TCP Incast Throughput Collapse in
>          Datacenter Networks",
>
> http://www.eecs.berkeley.edu/~ychen2/professional/TCPIncastWREN2009.pdf"
> .
>
> should be listed in the Informative References section, IF you plan to
> use it
> in the draft - I didn't find a reference to [5] in the text.
>
> Al
>
>
> > -----Original Message-----
> > From: Lucien Avramov (lavramov) [mailto:[email protected]]
> > Sent: Tuesday, February 23, 2016 2:31 AM
> > To: MORTON, ALFRED C (AL); Jacob Rapp; [email protected]
> > Subject: Re: [bmwg] I-D Action: draft-ietf-bmwg-dcbench-terminology-
> > 02.txt
> >
> > Thank you very much Al!
> >
> > I just incorporated your suggestion in the -04 version I posted:
> > https://www.ietf.org/rfcdiff?url2=draft-ietf-bmwg-dcbench-terminology-
> 04
> >
> > Yes, we did spend time on Scott's review word document and we
> > incorporated changes in the -03 version and send our thoughts also on
> a
> > the review of his word document to the mailer on 7/1/16 for
> traceability
> > purposes and to not loose all the valuable time Scott spend on
> reviewing
> > our draft. (thank you for checking)
> >
> > We want to thank BMWG for being eager to provide us so much support
> and
> > reviews over the years on our data center benchmarking work.
> >
> > The authors
> >
> >
> > On 2/23/16 8:14 AM, MORTON, ALFRED C (AL) wrote:
> > > Hi Lucien and Jacob,
> > >
> > > Thanks for your updates to the dcbench terminology draft,
> > > both updates in fact, 02 on Jan 6 and 03 on Jan 7.
> > >
> > > I have one editorial suggestion which you can address in a future
> > version:
> > > In section 3.1, the word "assimilated" doesn't fit this sentence (to
> > me).
> > >
> > > OLD
> > > Jitter in the data center context is also assimilated to common
> > term...
> > > suggest
> > > Jitter in the data center context is synonymous with the common
> > term...
> > >
> > > Otherwise, section 3 looks good.
> > >
> > > I think you indicated that Scott Bradner's comments on the draft
> have
> > been
> > > addressed? Scott sent his notes in an MSword file to the list on Nov
> > 3,
> > > and the discussion of his comments is captured in the IETF-94
> meeting
> > minutes:
> > > https://www.ietf.org/proceedings/94/minutes/minutes-94-bmwg
> > > (right up front). There may be more details in the minutes.
> > >
> > > Would you please confirm that Scott's comments have been addressed
> > > or otherwise answered?
> > >
> > > thanks, and sorry for delay on this reply,
> > > Al
> > >
> > >
> > >> -----Original Message-----
> > >> From: bmwg [mailto:[email protected]] On Behalf Of Lucien
> Avramov
> > >> (lavramov)
> > >> Sent: Thursday, January 07, 2016 5:42 PM
> > >> To: Jacob Rapp; [email protected]
> > >> Subject: Re: [bmwg] I-D Action: draft-ietf-bmwg-dcbench-
> terminology-
> > >> 02.txt
> > >>
> > >> BMWG,
> > >>
> > >> The document is now updated with also Al's comment about section 3)
> > >> Jitter / Delay variation. As Jacob said as authors we strongly feel
> > >> about wanting to still call this type of measurement jitter as this
> > is
> > >> how we see it charecterized in the data center. So I tried to
> reflect
> > it
> > >> by the first sentence in the definition after Al's comments.
> > >>
> > >> https://www.ietf.org/internet-drafts/draft-ietf-bmwg-dcbench-
> > >> terminology-03.txt
> > >>
> > >>
> > >> On 1/6/16 5:01 PM, Jacob Rapp wrote:
> > >>> First thank you everyone for the detailed feedback! Below is the
> > >> consolidated reply to the feedback.
> > >>>
> > >>> The draft draft-ietf-bmwg-dcbench-terminology-02 reflects all the
> > >> changes.
> > >>>
> > >>> Detail:
> > >>>
> > >>> We have updated the drafts with the feedback as well per this
> > updated
> > >> draft.
> > >>>
> > >>> Scott's Review and feedback: Attached is the doc with the reply
> with
> > >> all the feedback and updates we plan to make.
> > >>>
> > >>> Al's Feedback:
> > >>> <LA/JR> The suggestions below are all great and we will make the
> > >> updates accordingly. The only thing, is we would like to leave it
> > called
> > >> "Jitter". The original purpose inclusion was to specifically define
> > the
> > >> term "Jitter". We have seen a lot of cases where "jitter" is used
> in
> > >> many different ways which has lead to a lot of confusion.
> > >>>
> > >>> In the latest draft of Data Center Terminology -01:
> > >>> - using the term Jitter, should be delay variation now
> > >>> - still citing RFC 3393 almost exclusively, which is too flexible
> > >>> - only citing IPDV from RFC 5481, <JR/LA> Agreed, will update with
> > >> your suggestion below.
> > >>> - in the past meeting we agreed on RFC 5481 PDV as primary, IPDV
> > only
> > >> if you  below.
> > >>>     want to understand if traffic has become more bursty in
> transfer
> > >>> - Metric Units section is the place to specify Units of Measure
> > >> (seconds
> > >>>     in this case, not much else)
> > >>> I suggest to revise section 3 as follows:
> > >>> NEW:
> > >>>
> > >>> 3 Delay Variation
> > >>>
> > >>> 3.1 Definition
> > >>>
> > >>> Delay variation is derived from multiple measurements
> > >>> of one-way delay, as described in RFC 3393.
> > >>> The mandatory definition of Delay Variation is the PDV form
> > >>> from section 4.2 of RFC 5481. When considering a stream of
> packets,
> > >>> the delays of all packets are subtracted from the minimum delay
> > >>> over all packets in the stream. This facilitates assessment of
> > >>> the range of delay variation (Max - Min), or a high percentile
> > >>> of PDV (99th percentile, for robustness against outliers).
> > >>>
> > >>> If First-bit to Last-bit timestamps are used for Delay
> measurement,
> > >> then
> > >>> Delay Variation MUST be measured using packets or frames of the
> > >>> same size, since the definition of latency includes the
> > serialization
> > >>> time for each packet. Otherwise if using First-bit to First-bit,
> > >>> the size restriction does not apply.
> > >>>
> > >>>
> > >>> 3.2 Discussion
> > >>>      In addition to PDV Range and or a high percentile of PDV,
> > >>>      Inter-Packet Delay Variation (IPDV) as defined in section 4.1
> > of
> > >>>      RFC5481 (differences between two consecutive packets) MAY be
> > used
> > >>>      for the purpose of determining how packet spacing has changed
> > >> during
> > >>>      transfer, for example to see if packet stream has become
> > closely-
> > >> spaced
> > >>>      or "bursty". However, the Absolute Value of IPDV SHOULD NOT
> be
> > >> used, as
> > >>>      this collapses the "busrty" and "dispersed" sides of the IPDV
> > >> distribution
> > >>>      together.
> > >>>
> > >>> 3.3 Measurement Units
> > >>>
> > >>> The measurement of delay variation is expressed in units of
> seconds.
> > >>>
> > >>> A PDV histogram MAY be provided for the population of packets
> > >> measured.
> > >>>
> > >>> -=-=-=-=-=-=-=-=-=-
> > >>> OLD:
> > >>> 3 Jitter
> > >>>
> > >>> 3.1 Definition
> > >>>
> > >>>      The definition of Jitter is covered extensively in RFC 3393.
> > This
> > >>>      definition is not meant to replace that definition, but it is
> > >> meant
> > >>>      to provide guidance of use for data center network devices.
> > >>>
> > >>>      The use of Jitter is in according with the variation delay
> > >> definition
> > >>>      from RFC 3393:
> > >>>
> > >>>      The second meaning has to do with the variation of a metric
> > (e.g.,
> > >>>      delay) with respect to some reference metric (e.g., average
> > delay
> > >> or
> > >>>      minimum delay).  This meaning is frequently used by computer
> > >>>      scientists and frequently (but not always) refers to
> variation
> > in
> > >>>      delay.
> > >>>
> > >>>      Even with the reference to RFC 3393, there are many
> definitions
> > of
> > >>>      "jitter" possible. The one selected for Data Center
> > Benchmarking
> > >> is
> > >>>      closest to RFC 3393.
> > >>>
> > >>>
> > >>> 3.2 Discussion
> > >>>
> > >>>      Jitter can be measured in different scenarios:-packet to
> packet
> > >> delay
> > >>>      variation-delta between min and max packet delay variation
> for
> > all
> > >>>      packets sent.
> > >>>
> > >>> 3.3 Measurement Units
> > >>>
> > >>>      The jitter MUST be measured when sending packets of the same
> > size.
> > >>>      Jitter MUST be measured as packet to packet delay variation
> and
> > >> delta
> > >>>      between min and max packet delay variation of all packets
> sent.
> > A
> > >>>      histogram MAY be provided as a population of packets measured
> > per
> > >>>      latency or latency buckets. Inter-Packet Delay Variation as
> > >> defined
> > >>>      in RFC5481 (differences between two consecutive packets) MAY
> be
> > >> used
> > >>>      for this purpose.
> > >>>
> > >>>
> > >>> Marius Feedback:<LA/JR> Agreed. This was some of Scott's early
> > >> feedback as well. We added in specific language to cover number of
> > >> trials as well.
> > >>> Example: "*For each test methodology described, it is key to
> obtain
> > >> repeatability of the results. The recommendation is to perform
> enough
> > >> iterations of the given test to make sure the result is accurate,
> > this
> > >> is especially important for section 3) as the buffering testing has
> > been
> > >> historically the least reliable."
> > >>>
> > >>> Marius Georgescu: I just have a short comment about repeatability.
> > >>> Essentially they talk about variation. I think they should be
> > talking
> > >>> about variance. It's good to have min and max, but the variance is
> > >>> also accounting for the number of trials. Also, about the
> > summarizing
> > >>> function, I think we should have that discussion as a working
> group
> > >>> and will try to cover it in my presentation.
> > >>>
> > >>> -Jacob Rapp & Lucien Avramov
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On 1/6/16, 4:58 PM, "bmwg on behalf of [email protected]"
> > >> <[email protected] on behalf of [email protected]>
> wrote:
> > >>>
> > >>>>
> > >>>> A New Internet-Draft is available from the on-line Internet-
> Drafts
> > >> directories.
> > >>>> This draft is a work item of the Benchmarking Methodology Working
> > >> Group of the IETF.
> > >>>>
> > >>>>          Title           : Data Center Benchmarking Terminology
> > >>>>          Authors         : Lucien Avramov
> > >>>>                            Jacob Rapp
> > >>>>        Filename        : draft-ietf-bmwg-dcbench-terminology-02.txt
> > >>>>        Pages           : 15
> > >>>>        Date            : 2016-01-06
> > >>>>
> > >>>> Abstract:
> > >>>> The purpose of this informational document is to establish
> > >> definitions,
> > >>>> discussion and measurement techniques for data center
> benchmarking.
> > >>>> Also, it is to introduce new terminologies applicable to data
> > center
> > >>>> performance evaluations. The purpose of this document is not to
> > >> define
> > >>>> the test methodology, but rather establish the important concepts
> > >> when
> > >>>> one is interested in benchmarking network switches and routers in
> > the
> > >>>> data center.
> > >>>>
> > >>>>
> > >>>> The IETF datatracker status page for this draft is:
> > >>>> https://urldefense.proofpoint.com/v2/url?u=https-
> > >> 3A__datatracker.ietf.org_doc_draft-2Dietf-2Dbmwg-2Ddcbench-
> > >> 2Dterminology_&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-
> > >>
> >
> uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8
> > >> tU&s=KBX60CyHXvKQA-6xRIrVa_MkB8r_w2NR0LES2kjyeno&e=
> > >>>>
> > >>>> There's also a htmlized version available at:
> > >>>> https://urldefense.proofpoint.com/v2/url?u=https-
> > >> 3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Ddcbench-
> 2Dterminology-
> > >> 2D02&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-
> > >>
> >
> uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8
> > >> tU&s=MpAy2_x5iROY5UauQybc8zbhfenLpqX0fe1bSdk-1Mw&e=
> > >>>>
> > >>>> A diff from the previous version is available at:
> > >>>> https://urldefense.proofpoint.com/v2/url?u=https-
> > >> 3A__www.ietf.org_rfcdiff-3Furl2-3Ddraft-2Dietf-2Dbmwg-2Ddcbench-
> > >> 2Dterminology-2D02&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-
> > YihVMNtXt-
> > >>
> >
> uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8
> > >> tU&s=9aTPOGU6rnv5EV3nfIJ7_CzB2NE7Bpi32M3Z039PaNM&e=
> > >>>>
> > >>>>
> > >>>> Please note that it may take a couple of minutes from the time of
> > >> submission
> > >>>> until the htmlized version and diff are available at
> > tools.ietf.org.
> > >>>>
> > >>>> Internet-Drafts are also available by anonymous FTP at:
> > >>>> https://urldefense.proofpoint.com/v2/url?u=ftp-
> > >> 3A__ftp.ietf.org_internet-
> > >> 2Ddrafts_&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-
> > >>
> >
> uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8
> > >> tU&s=Jz3dA8OqSDgiHOalJ8c8WuAl2XuIK4pxYq4014WinKo&e=
> > >>>>
> > >>>> _______________________________________________
> > >>>> bmwg mailing list
> > >>>> [email protected]
> > >>>> https://urldefense.proofpoint.com/v2/url?u=https-
> > >>
> >
> 3A__www.ietf.org_mailman_listinfo_bmwg&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiD
> > >> JAXVeAw-YihVMNtXt-
> > >>
> >
> uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8
> > >> tU&s=7S-JdXwgqeVkDUxz6DxWc8oPEXFYi7mrWSy1A7DEcKQ&e=
> > >>>>
> > >>>>
> > >>>> _______________________________________________
> > >>>> bmwg mailing list
> > >>>> [email protected]
> > >>>> https://www.ietf.org/mailman/listinfo/bmwg
> > >>
> > >> _______________________________________________
> > >> bmwg mailing list
> > >> [email protected]
> > >> https://www.ietf.org/mailman/listinfo/bmwg
>
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/bmwg