Re: I-D Action: draft-ietf-bmwg-dcbench-terminology-02.txt
"MORTON, ALFRED C (AL)" <[email protected]>
| Newsgroups | gmane.ietf.bmwg |
|---|---|
| Message-ID | <4AF73AA205019A4C8A1DDD32C034631D3FF3A31090@NJFPSRVEXG0.research.att.com> |
Hi Lucien and Jacob,
Thanks for the quick update to the terminology draft.
Are there now places where the Methodology draft needs
to be updated to stay in sync with terminology?
I have a couple of comments on the methodology, when I scanned now:
Nit:
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [6].
Ref [6] is missing.
-=-=-=-=-=-=-=-
6.3 Reporting Format
The report MUST include the following:
- Number of ingress and egress ports along with designation of
stateful or stateless.
- Stateful goodput
I think you are referring to the "Data Center Goodput" defined here:
https://tools.ietf.org/html/draft-ietf-bmwg-dcbench-terminology-04#section-7
so at least we need a reference, and it's likely that BMWG'ers
(including me) will have a few comments on the Goodput Definition in
the terminology draft. (It seems to me we might already have such a
definition, and you can just refer to it, if it is suitable).
-=-=-=-=-=-=-=-=-=-
Also, this reference:
7.3. URL References
[5] Yanpei Chen, Rean Griffith, Junda Liu, Randy H. Katz, Anthony D.
Joseph, "Understanding TCP Incast Throughput Collapse in
Datacenter Networks",
http://www.eecs.berkeley.edu/~ychen2/professional/TCPIncastWREN2009.pdf".
should be listed in the Informative References section, IF you plan to use it
in the draft - I didn't find a reference to [5] in the text.
Al
> -----Original Message-----
> From: Lucien Avramov (lavramov) [mailto:[email protected]]
> Sent: Tuesday, February 23, 2016 2:31 AM
> To: MORTON, ALFRED C (AL); Jacob Rapp; [email protected]
> Subject: Re: [bmwg] I-D Action: draft-ietf-bmwg-dcbench-terminology-
> 02.txt
>
> Thank you very much Al!
>
> I just incorporated your suggestion in the -04 version I posted:
> https://www.ietf.org/rfcdiff?url2=draft-ietf-bmwg-dcbench-terminology-04
>
> Yes, we did spend time on Scott's review word document and we
> incorporated changes in the -03 version and send our thoughts also on a
> the review of his word document to the mailer on 7/1/16 for traceability
> purposes and to not loose all the valuable time Scott spend on reviewing
> our draft. (thank you for checking)
>
> We want to thank BMWG for being eager to provide us so much support and
> reviews over the years on our data center benchmarking work.
>
> The authors
>
>
> On 2/23/16 8:14 AM, MORTON, ALFRED C (AL) wrote:
> > Hi Lucien and Jacob,
> >
> > Thanks for your updates to the dcbench terminology draft,
> > both updates in fact, 02 on Jan 6 and 03 on Jan 7.
> >
> > I have one editorial suggestion which you can address in a future
> version:
> > In section 3.1, the word "assimilated" doesn't fit this sentence (to
> me).
> >
> > OLD
> > Jitter in the data center context is also assimilated to common
> term...
> > suggest
> > Jitter in the data center context is synonymous with the common
> term...
> >
> > Otherwise, section 3 looks good.
> >
> > I think you indicated that Scott Bradner's comments on the draft have
> been
> > addressed? Scott sent his notes in an MSword file to the list on Nov
> 3,
> > and the discussion of his comments is captured in the IETF-94 meeting
> minutes:
> > https://www.ietf.org/proceedings/94/minutes/minutes-94-bmwg
> > (right up front). There may be more details in the minutes.
> >
> > Would you please confirm that Scott's comments have been addressed
> > or otherwise answered?
> >
> > thanks, and sorry for delay on this reply,
> > Al
> >
> >
> >> -----Original Message-----
> >> From: bmwg [mailto:[email protected]] On Behalf Of Lucien Avramov
> >> (lavramov)
> >> Sent: Thursday, January 07, 2016 5:42 PM
> >> To: Jacob Rapp; [email protected]
> >> Subject: Re: [bmwg] I-D Action: draft-ietf-bmwg-dcbench-terminology-
> >> 02.txt
> >>
> >> BMWG,
> >>
> >> The document is now updated with also Al's comment about section 3)
> >> Jitter / Delay variation. As Jacob said as authors we strongly feel
> >> about wanting to still call this type of measurement jitter as this
> is
> >> how we see it charecterized in the data center. So I tried to reflect
> it
> >> by the first sentence in the definition after Al's comments.
> >>
> >> https://www.ietf.org/internet-drafts/draft-ietf-bmwg-dcbench-
> >> terminology-03.txt
> >>
> >>
> >> On 1/6/16 5:01 PM, Jacob Rapp wrote:
> >>> First thank you everyone for the detailed feedback! Below is the
> >> consolidated reply to the feedback.
> >>>
> >>> The draft draft-ietf-bmwg-dcbench-terminology-02 reflects all the
> >> changes.
> >>>
> >>> Detail:
> >>>
> >>> We have updated the drafts with the feedback as well per this
> updated
> >> draft.
> >>>
> >>> Scott's Review and feedback: Attached is the doc with the reply with
> >> all the feedback and updates we plan to make.
> >>>
> >>> Al's Feedback:
> >>> <LA/JR> The suggestions below are all great and we will make the
> >> updates accordingly. The only thing, is we would like to leave it
> called
> >> "Jitter". The original purpose inclusion was to specifically define
> the
> >> term "Jitter". We have seen a lot of cases where "jitter" is used in
> >> many different ways which has lead to a lot of confusion.
> >>>
> >>> In the latest draft of Data Center Terminology -01:
> >>> - using the term Jitter, should be delay variation now
> >>> - still citing RFC 3393 almost exclusively, which is too flexible
> >>> - only citing IPDV from RFC 5481, <JR/LA> Agreed, will update with
> >> your suggestion below.
> >>> - in the past meeting we agreed on RFC 5481 PDV as primary, IPDV
> only
> >> if you below.
> >>> want to understand if traffic has become more bursty in transfer
> >>> - Metric Units section is the place to specify Units of Measure
> >> (seconds
> >>> in this case, not much else)
> >>> I suggest to revise section 3 as follows:
> >>> NEW:
> >>>
> >>> 3 Delay Variation
> >>>
> >>> 3.1 Definition
> >>>
> >>> Delay variation is derived from multiple measurements
> >>> of one-way delay, as described in RFC 3393.
> >>> The mandatory definition of Delay Variation is the PDV form
> >>> from section 4.2 of RFC 5481. When considering a stream of packets,
> >>> the delays of all packets are subtracted from the minimum delay
> >>> over all packets in the stream. This facilitates assessment of
> >>> the range of delay variation (Max - Min), or a high percentile
> >>> of PDV (99th percentile, for robustness against outliers).
> >>>
> >>> If First-bit to Last-bit timestamps are used for Delay measurement,
> >> then
> >>> Delay Variation MUST be measured using packets or frames of the
> >>> same size, since the definition of latency includes the
> serialization
> >>> time for each packet. Otherwise if using First-bit to First-bit,
> >>> the size restriction does not apply.
> >>>
> >>>
> >>> 3.2 Discussion
> >>> In addition to PDV Range and or a high percentile of PDV,
> >>> Inter-Packet Delay Variation (IPDV) as defined in section 4.1
> of
> >>> RFC5481 (differences between two consecutive packets) MAY be
> used
> >>> for the purpose of determining how packet spacing has changed
> >> during
> >>> transfer, for example to see if packet stream has become
> closely-
> >> spaced
> >>> or "bursty". However, the Absolute Value of IPDV SHOULD NOT be
> >> used, as
> >>> this collapses the "busrty" and "dispersed" sides of the IPDV
> >> distribution
> >>> together.
> >>>
> >>> 3.3 Measurement Units
> >>>
> >>> The measurement of delay variation is expressed in units of seconds.
> >>>
> >>> A PDV histogram MAY be provided for the population of packets
> >> measured.
> >>>
> >>> -=-=-=-=-=-=-=-=-=-
> >>> OLD:
> >>> 3 Jitter
> >>>
> >>> 3.1 Definition
> >>>
> >>> The definition of Jitter is covered extensively in RFC 3393.
> This
> >>> definition is not meant to replace that definition, but it is
> >> meant
> >>> to provide guidance of use for data center network devices.
> >>>
> >>> The use of Jitter is in according with the variation delay
> >> definition
> >>> from RFC 3393:
> >>>
> >>> The second meaning has to do with the variation of a metric
> (e.g.,
> >>> delay) with respect to some reference metric (e.g., average
> delay
> >> or
> >>> minimum delay). This meaning is frequently used by computer
> >>> scientists and frequently (but not always) refers to variation
> in
> >>> delay.
> >>>
> >>> Even with the reference to RFC 3393, there are many definitions
> of
> >>> "jitter" possible. The one selected for Data Center
> Benchmarking
> >> is
> >>> closest to RFC 3393.
> >>>
> >>>
> >>> 3.2 Discussion
> >>>
> >>> Jitter can be measured in different scenarios:-packet to packet
> >> delay
> >>> variation-delta between min and max packet delay variation for
> all
> >>> packets sent.
> >>>
> >>> 3.3 Measurement Units
> >>>
> >>> The jitter MUST be measured when sending packets of the same
> size.
> >>> Jitter MUST be measured as packet to packet delay variation and
> >> delta
> >>> between min and max packet delay variation of all packets sent.
> A
> >>> histogram MAY be provided as a population of packets measured
> per
> >>> latency or latency buckets. Inter-Packet Delay Variation as
> >> defined
> >>> in RFC5481 (differences between two consecutive packets) MAY be
> >> used
> >>> for this purpose.
> >>>
> >>>
> >>> Marius Feedback:<LA/JR> Agreed. This was some of Scott's early
> >> feedback as well. We added in specific language to cover number of
> >> trials as well.
> >>> Example: "*For each test methodology described, it is key to obtain
> >> repeatability of the results. The recommendation is to perform enough
> >> iterations of the given test to make sure the result is accurate,
> this
> >> is especially important for section 3) as the buffering testing has
> been
> >> historically the least reliable."
> >>>
> >>> Marius Georgescu: I just have a short comment about repeatability.
> >>> Essentially they talk about variation. I think they should be
> talking
> >>> about variance. It's good to have min and max, but the variance is
> >>> also accounting for the number of trials. Also, about the
> summarizing
> >>> function, I think we should have that discussion as a working group
> >>> and will try to cover it in my presentation.
> >>>
> >>> -Jacob Rapp & Lucien Avramov
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On 1/6/16, 4:58 PM, "bmwg on behalf of [email protected]"
> >> <[email protected] on behalf of [email protected]> wrote:
> >>>
> >>>>
> >>>> A New Internet-Draft is available from the on-line Internet-Drafts
> >> directories.
> >>>> This draft is a work item of the Benchmarking Methodology Working
> >> Group of the IETF.
> >>>>
> >>>> Title : Data Center Benchmarking Terminology
> >>>> Authors : Lucien Avramov
> >>>> Jacob Rapp
> >>>> Filename : draft-ietf-bmwg-dcbench-terminology-02.txt
> >>>> Pages : 15
> >>>> Date : 2016-01-06
> >>>>
> >>>> Abstract:
> >>>> The purpose of this informational document is to establish
> >> definitions,
> >>>> discussion and measurement techniques for data center benchmarking.
> >>>> Also, it is to introduce new terminologies applicable to data
> center
> >>>> performance evaluations. The purpose of this document is not to
> >> define
> >>>> the test methodology, but rather establish the important concepts
> >> when
> >>>> one is interested in benchmarking network switches and routers in
> the
> >>>> data center.
> >>>>
> >>>>
> >>>> The IETF datatracker status page for this draft is:
> >>>> https://urldefense.proofpoint.com/v2/url?u=https-
> >> 3A__datatracker.ietf.org_doc_draft-2Dietf-2Dbmwg-2Ddcbench-
> >> 2Dterminology_&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-
> >>
> uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8
> >> tU&s=KBX60CyHXvKQA-6xRIrVa_MkB8r_w2NR0LES2kjyeno&e=
> >>>>
> >>>> There's also a htmlized version available at:
> >>>> https://urldefense.proofpoint.com/v2/url?u=https-
> >> 3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Ddcbench-2Dterminology-
> >> 2D02&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-
> >>
> uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8
> >> tU&s=MpAy2_x5iROY5UauQybc8zbhfenLpqX0fe1bSdk-1Mw&e=
> >>>>
> >>>> A diff from the previous version is available at:
> >>>> https://urldefense.proofpoint.com/v2/url?u=https-
> >> 3A__www.ietf.org_rfcdiff-3Furl2-3Ddraft-2Dietf-2Dbmwg-2Ddcbench-
> >> 2Dterminology-2D02&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-
> YihVMNtXt-
> >>
> uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8
> >> tU&s=9aTPOGU6rnv5EV3nfIJ7_CzB2NE7Bpi32M3Z039PaNM&e=
> >>>>
> >>>>
> >>>> Please note that it may take a couple of minutes from the time of
> >> submission
> >>>> until the htmlized version and diff are available at
> tools.ietf.org.
> >>>>
> >>>> Internet-Drafts are also available by anonymous FTP at:
> >>>> https://urldefense.proofpoint.com/v2/url?u=ftp-
> >> 3A__ftp.ietf.org_internet-
> >> 2Ddrafts_&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-
> >>
> uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8
> >> tU&s=Jz3dA8OqSDgiHOalJ8c8WuAl2XuIK4pxYq4014WinKo&e=
> >>>>
> >>>> _______________________________________________
> >>>> bmwg mailing list
> >>>> [email protected]
> >>>> https://urldefense.proofpoint.com/v2/url?u=https-
> >>
> 3A__www.ietf.org_mailman_listinfo_bmwg&d=BQICAg&c=Sqcl0Ez6M0X8aeM67LKIiD
> >> JAXVeAw-YihVMNtXt-
> >>
> uEs&r=4eVZvh2C5Un5N5XXv1rh7g&m=Kx2LCUg_N6QScJ7FkeVYlF80cWWhT8QLn11PAaNQ8
> >> tU&s=7S-JdXwgqeVkDUxz6DxWc8oPEXFYi7mrWSy1A7DEcKQ&e=
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> bmwg mailing list
> >>>> [email protected]
> >>>> https://www.ietf.org/mailman/listinfo/bmwg
> >>
> >> _______________________________________________
> >> bmwg mailing list
> >> [email protected]
> >> https://www.ietf.org/mailman/listinfo/bmwg