comments on draft-dcbench-def-00 (part 1)

Hi Jacob and Lucien,

Thanks for your efforts to adopt BMWG's traditional template
for your draft on terminology. This draft appears to provide a good
foundation for your methodology, and also for some fundamental 
discussions on benchmarking, so I share a few specific comments below.

Al (as a participant)

-=-=-=-=-=-=-=-=-=-

1.2. Definition format

   Term to be defined. (e.g., Latency)

   Definition: The specific definition for the term.

   Discussion: A brief discussion about the term, it's application and
   any restrictions on measurement procedures.

   Measurement: Methodology for the measure and units used to report
   measurements of this term, if applicable.

Comment: The last item "Measurement" is usually just "Measurement Units" in our
traditional format, because we usually add many details in the course of developing
BMWG methodology drafts. In the past, we have included the headings of "Issues" and 
"See Also" for references to other terms. 

>>>>>Perhaps we should have a short discussion
     of the usefulness of our current Definition format and see if there is need 
     for change. We can also consider the RFC 6390 template which presents guidelines 
     for new metrics.

-=-=-=-=-=-=-=-=-=-

2.  Latency


2.1. Definition

   Latency is a the amount of time it takes a frame to transit the DUT.

   Latency can be measured with the following methods, irrespectively of
   the type of switching device (bit forwarding aka cut-through or store
   forward type of device)

   FILO (First In Last Out) The time interval starting when the end of
   the first bit of the input frame reaches the input port and ending
   when the last bit of the output frame is seen on the output port

Several comments and suggestions:

* The paragraphs for FILO, FIFO, LILO, LIFO are actually different definitions
of Latency (not methods per se), but this is easily fixed:

OLD
   Latency can be measured with the following methods,
NEW
   The Latency interval can be assessed between different combinations of events,

* Another way to summarize the 4 different definitions is to simply refer to
the bit position as they normally occur, input to output.  This would change the
names as follows:

FILO -> First bit, Last bit
FIFO -> First bit, First bit
LILO -> Last bit, Last bit
LIFO -> Last bit, First bit

and these can be summarized FbLb or just FL.

* FILO is the definition recommended, but this has been avoided in the past
(in BMWG) to exclude the frame serialization time from Latency. It makes for easier
comparison when you are testing with multiple frame sizes, especially between
large and small packets (and that was preferred in RFC 1242, even with the possibility
of negative delays).

* If BMWG agrees to make this change, it would mean that this draft updates RFC 1242,
which we should do very carefully, IMO.

>>>>>> Let's plan to discuss Latency further during the BMWG session.


-=-=-=-=-=-=-=-=-=-

3 Jitter

3.1 Definition

   The definition of Jitter is covered extensively in RFC 3393. This
   definition is not meant to replace that definition, but it is meant
   to provide guidance of use for data center network devices.

   The use of Jitter is in according with the variation delay definition
   from RFC 3393:

   The second meaning has to do with the variation of a metric (e.g.,
   delay) with respect to some reference metric (e.g., average delay or
   minimum delay).  This meaning is frequently used by computer
   scientists and frequently (but not always) refers to variation in
   delay.

Comment:  Even with the reference to RFC 3393, there are many definitions
of "jitter" possible. Fortunately this is not the first draft to examine 
the possibilities :-)

I find that a comment I made on the Traffic Management draft is applicable here:
>...
>   - Packet Delay Variation (PDV): the Packet Delay Variation metric is
>   the variation between the timestamp of the received egress port
>   packets and specified in RFC 1889.
>
> Comment: since you have One-way Delay (PD defined above PDV), why resort to the RFC 1889 definition?  
> What does this give you?
> Suggest: Specify PDV as defined in http://tools.ietf.org/html/rfc5481#section-4.2 instead, for the many 
> reasons given in http://tools.ietf.org/html/rfc5481#page-27
> (or call this IPDV to avoid confusion).

In the Isolated test lab environment, we don't have time clock drift issues that
we can't solve, in many cases the same clock is used for ingress and egress timestamps.
So, we are free to choose the delay variation metric(s) that serve our audience best.

>>>> I think "Jitter" is another topic that we can advance through face2face discussion.
     What do we really want to learn from the "jitter" metric?
     How will these results be used? (in network design, application planning, etc.)

(I'll have to do this review in multiple parts, it seems)