Re: I-D Action: draft-ietf-bmwg-dcbench-terminology-00.txt

Hi Jacob and Lucien,

Please find my comments on
> https://tools.ietf.org/html/draft-ietf-bmwg-dcbench-terminology-00
> 
below, Sections 1, 2, and 3, mostly on Latency section (ACM:)

Let's chat when we can, 
regards,
Al
(participant)




1.  Introduction

   Traffic patterns in the data center are not uniform and are contently
   changing. They are dictated by the nature and variety of applications
   utilized in the data center. It can be largely east-west traffic
   flows in one data center and north-south in another, while some may
   combine both. Traffic patterns can be bursty in nature and contain
   many-to-one, many-to-many, or one-to-many flows. Each flow may also
   be small and latency sensitive or large and throughput sensitive
   while containing a mix of UDP and TCP traffic. All of which can
   coexist in a single cluster and flow through a single network device
   all at the same time. Benchmarking of network devices have long used
   RFC1242, RFC2432, RFC2544, RFC2889 and RFC3918. These benchmarks have
   largely been focused around various latency attributes and max
   throughput of the Device Under Test being benchmarked. These
   standards are good at measuring theoretical max throughput,
   forwarding rates and latency under testing conditions, but to not
ACM: s/to/do/
   represent real traffic patterns that may affect these networking
ACM: s/real/production/
   devices.


   The following defines a set of definitions, metrics and terminologies
ACM: s/following/following document/
   including congestion scenarios, switch buffer analysis and redefines
   basic definitions in order to represent a wide mix of traffic
   conditions.


...
1.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [6].
ACM:
We usually add some text here to clarify how the these words apply to testing
specifications - see my comments on draft-georgescu-bmwg ...

1.2. Definition format

   Term to be defined. (e.g., Latency)

   Definition: The specific definition for the term.

   Discussion: A brief discussion about the term, it's application and
   any restrictions on measurement procedures.

   Measurement Units: Methodology for the measure and units used to
   report measurements of this term, if applicable.


2.  Latency


2.1. Definition
ACM:  The definition needs to be very concise, see below.

   Latency is a the amount of time it takes a frame to transit the DUT. 

   The Latency interval can be assessed between different combinations
   of events, irrespectively of the type of switching device (bit
   forwarding aka cut-through or store forward type of device)

   Traditionally the latency measurement definitions are: 

   	FILO (First In Last Out) The time interval starting when the end of
   the first bit of the input frame reaches the input port and ending
   when the last bit of the output frame is seen on the output port

   	FIFO (First In First Out) The time interval starting when the end of
   the first bit of the input frame reaches the input port and ending
   when the start of the first bit of the output frame is seen on the
   output port

   	LILO (Last In Last Out) The time interval starting when the last bit
   of the input frame reaches the input port and the last bit of the
   output frame is seen on the output port

   	LIFO (Last In First Out) The time interval starting when the last
   bit of the input frame reaches the input port and ending when the

   first bit of the output frame is seen on the output port.


   Another possibility to summarize the four different definitions above
   is to  refer to the bit position as they normally occur: input to
   output.


            FILO is FL (First bit Last bit) 	FIFO is FF (First bit First
   bit) 	LILO is LL (Last bit Last bit) 	LIFO is LF (Last bit First bit)

   This definition explained in this section in context of data center
   switching benchmarking is in lieu of the previous definition of
   Latency defined in RFC 1242, section 3.8 and is quoted here:

   For store and forward devices: The time interval starting when the
   last bit of the input frame reaches the input port and ending when
   the first bit of the output frame is seen on the output port.

   For bit forwarding devices: The time interval starting when the end
   of the first bit of the input frame reaches the input port and ending
   when the start of the first bit of the output frame is seen on the
   output port.

2.2 Discussion
ACM: Comment, in reality, most of the above section is background and should 
     appear in the Discussion section.  The Definition should briefly explain
	 FILO, and give most of the paragraph below. In other words, the 
	 definition needs to be concise, like RFC 1242, but without the options.

   FILO is the most important measuring definition. Any type of switches
ACM:  s/Any type/All types/
   MUST be measured with the FILO mechanism: FILO will include the
ACM:  s/mechanism/time stamps/
   latency of the switch and the latency of the frame as well as the
   serialization delay. 
ACM: Question: what's the difference between the latency of the frame
     and the serialization delay??
	
   It is a picture of the 'whole' latency going
   through the DUT. 
ACM: Suggest to replace this sentence with the first sentence of the section:
   "Latency is a the amount of time it takes a frame to transit the DUT."

   For applications, which are latency sensitive and
   can function with initial bytes of the frame, FIFO MAY be an
   additional type of measuring to supplement FILO.  
ACM: Need and example of the above, it might be better described some
other way (e.g., not using the term "application")

   LIFO mechanism can be used with store forward type of switches but
 ACM: s/store/store and/
   not with cut-through type of switches, as it will provide negative
   latency values for larger packet sizes. Therefore this mechanism MUST
   NOT be used when comparing latencies of two different DUTs. 
ACM: Even if the DUTs/switches are both store and forward?

2.3 Measurement Units

   The measuring methods to use for benchmarking purposes are as follow:

   1) FILO MUST be used as a measuring method, as this will include the
   latency of the packet; and today the application commonly need to
   read the whole packet to process the information and take an action. 

   2) FIFO MAY be used for certain applications able to proceed data as
 
   the first bits arrive (FPGA for example)

   3) LIFO MUST not be used, because it subtracts the latency of the
   packet; unlike all the other methods.


3 Jitter

3.1 Definition
ACM: Comment
The feedback at the last meeting was that we need BOTH definitions from
RFC 5481, that is PDV and IPDV. Let's use that reference RFC and build from
there.

   The definition of Jitter is covered extensively in RFC 3393. This
   definition is not meant to replace that definition, but it is meant
   to provide guidance of use for data center network devices. 

   The use of Jitter is in according with the variation delay definition
   from RFC 3393:

   The second meaning has to do with the variation of a metric (e.g.,
   delay) with respect to some reference metric (e.g., average delay or
   minimum delay).  This meaning is frequently used by computer
   scientists and frequently (but not always) refers to variation in
   delay.

   Even with the reference to RFC 3393, there are many definitions of
   "jitter" possible. The one selected for Data Center Benchmarking is
   closest to RFC 3393.

3.2 Discussion

   Jitter can be measured in different scenarios:-packet to packet delay
   variation-delta between min and max packet delay variation for all
   packets sent.

3.3 Measurement Units

   The jitter MUST be measured when sending packets of the same size.
   Jitter MUST be measured as packet to packet delay variation and delta
   between min and max packet delay variation of all packets sent. A
   histogram MAY be provided as a population of packets measured per
   latency or latency buckets.