Re: comments on draft-dcbench-def-00 (part 1)

Hi Al,

Thank you very much for taking the time to write this detailed feedback.
Comments inline


On 7/24/13 7:03 PM, MORTON JR., ALFRED C (AL) wrote:
> Hi Jacob and Lucien,
>
> Thanks for your efforts to adopt BMWG's traditional template
> for your draft on terminology. This draft appears to provide a good
> foundation for your methodology, and also for some fundamental
> discussions on benchmarking, so I share a few specific comments below.
>
> Al (as a participant)
>
> -=-=-=-=-=-=-=-=-=-
>
> 1.2. Definition format
>
>     Term to be defined. (e.g., Latency)
>
>     Definition: The specific definition for the term.
>
>     Discussion: A brief discussion about the term, it's application and
>     any restrictions on measurement procedures.
>
>     Measurement: Methodology for the measure and units used to report
>     measurements of this term, if applicable.
>
> Comment: The last item "Measurement" is usually just "Measurement Units" in our
> traditional format, because we usually add many details in the course of developing
> BMWG methodology drafts. In the past, we have included the headings of "Issues" and
> "See Also" for references to other terms.

We can rename the structure we use to Measurement Units, but we dont 
have always just a unit per say, we go beyond it. For example, for 
latency, the unit is time for the values, but in the measurement section 
we define what should be measured for this benchmarking: filo as must, 
fifo as may. If you feel appropriate that we call it measurement unit, 
then we can change each one to match.

Our intent was to have this paragraph to be the recommendation on how to 
measure a specific topic.

>
>>>>>> Perhaps we should have a short discussion
>       of the usefulness of our current Definition format and see if there is need
>       for change. We can also consider the RFC 6390 template which presents guidelines
>       for new metrics.
>
> -=-=-=-=-=-=-=-=-=-
>
> 2.  Latency
>
>
> 2.1. Definition
>
>     Latency is a the amount of time it takes a frame to transit the DUT.
>
>     Latency can be measured with the following methods, irrespectively of
>     the type of switching device (bit forwarding aka cut-through or store
>     forward type of device)
>
>     FILO (First In Last Out) The time interval starting when the end of
>     the first bit of the input frame reaches the input port and ending
>     when the last bit of the output frame is seen on the output port
>
> Several comments and suggestions:
>
> * The paragraphs for FILO, FIFO, LILO, LIFO are actually different definitions
> of Latency (not methods per se), but this is easily fixed:
>
> OLD
>     Latency can be measured with the following methods,
> NEW
>     The Latency interval can be assessed between different combinations of events,
>
> * Another way to summarize the 4 different definitions is to simply refer to
> the bit position as they normally occur, input to output.  This would change the
> names as follows:
>
> FILO -> First bit, Last bit
> FIFO -> First bit, First bit
> LILO -> Last bit, Last bit
> LIFO -> Last bit, First bit
>
> and these can be summarized FbLb or just FL.

That is good idea, it will add more clarity, we will add the above in 
the section


>
> * FILO is the definition recommended, but this has been avoided in the past
> (in BMWG) to exclude the frame serialization time from Latency. It makes for easier
> comparison when you are testing with multiple frame sizes, especially between
> large and small packets (and that was preferred in RFC 1242, even with the possibility
> of negative delays).

FILO would be the way to fairly compare a store forward switch to a 
cut-through switch, hence this recommendation, to get the users closer 
to an apple-apple comparison.

Also this will show the end consumers of this benchmarking, the real 
latency as seen per the applications. Usually applications can process 
data, once all the packet is been received. Only certain type of 
application, using for example an FPGA card, can start processing data 
with the first bits, without having to wait for the whole packet.

So we still strongly believe FIFO is very important, but we want to have 
the users to be given the FILO measure as well so they can later compare 
different method of switches and also see the performance from an 
'application' perspective versus just a 'network' latency point of view.

>
> * If BMWG agrees to make this change, it would mean that this draft updates RFC 1242,
> which we should do very carefully, IMO.
>
>>>>>>> Let's plan to discuss Latency further during the BMWG session.
>
>
> -=-=-=-=-=-=-=-=-=-
>
> 3 Jitter
>
> 3.1 Definition
>
>     The definition of Jitter is covered extensively in RFC 3393. This
>     definition is not meant to replace that definition, but it is meant
>     to provide guidance of use for data center network devices.
>
>     The use of Jitter is in according with the variation delay definition
>     from RFC 3393:
>
>     The second meaning has to do with the variation of a metric (e.g.,
>     delay) with respect to some reference metric (e.g., average delay or
>     minimum delay).  This meaning is frequently used by computer
>     scientists and frequently (but not always) refers to variation in
>     delay.
>
> Comment:  Even with the reference to RFC 3393, there are many definitions
> of "jitter" possible. Fortunately this is not the first draft to examine
> the possibilities :-)
>
> I find that a comment I made on the Traffic Management draft is applicable here:
>> ...
>>    - Packet Delay Variation (PDV): the Packet Delay Variation metric is
>>    the variation between the timestamp of the received egress port
>>    packets and specified in RFC 1889.
>>
>> Comment: since you have One-way Delay (PD defined above PDV), why resort to the RFC 1889 definition?
>> What does this give you?
>> Suggest: Specify PDV as defined in http://tools.ietf.org/html/rfc5481#section-4.2 instead, for the many
>> reasons given in http://tools.ietf.org/html/rfc5481#page-27
>> (or call this IPDV to avoid confusion).
>
> In the Isolated test lab environment, we don't have time clock drift issues that
> we can't solve, in many cases the same clock is used for ingress and egress timestamps.
> So, we are free to choose the delay variation metric(s) that serve our audience best.
>
>>>>> I think "Jitter" is another topic that we can advance through face2face discussion.
>       What do we really want to learn from the "jitter" metric?
>       How will these results be used? (in network design, application planning, etc.)

For the Data Center benchmarking, we found that the rfc 3393 definition 
was the easiest to consume for its applicability to the data center. we 
wanted to add to this that what is key is to have packets of the same 
size when jitter is measured. It's mentioned in the RFC 3393 but in a 
more complex method as follows:

'‘+ L, a packet length in bits. The packets of a Type P packet stream 
from which the singleton ipdv metric is taken MUST all be of the same 
length.’’

We want to make this simpler to the end data center user by calling it 
packets of the same size.

Also we add to this definition, by explaining the necessity for a 
histogram with latency buckets, this will help understand where there is 
higher jitter for instance.


>
> (I'll have to do this review in multiple parts, it seems)
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg
>