Re: comments on draft-dcbench-def-00 (part 1)

Thanks Al and David,

I also find the 4689 to add more details on jitter, but yet we don't 
have a simple explanation for the datacenter consumers of the 
benchmarking results.

Having been doing datacenter benchmarking and meeting customers to whom 
we show results, explain results or guide for product benchmarking for 
years, the problem we have faced the most is how to make jitter a simple 
and significant metric when evaluating a data center switch.

In the datacenter, where performance is often times quite critical [high 
performance compute clusters, high frequency trading datacenter 
environments; to name a few], the jitter is part of the performance for 
a DUT.

The problem we try to solve with jitter in terms of these two drafts is:
-how to make jitter a simple term for datacenter engineers to understand 
and to measure.

We don't have the ambition to redefine jitter or re-question previous 
drafts, rather place them in datacenter context with these two key parts 
of this draft-dcbench-def-00:

3.2 Discussion

    Jitter can be measured in different scenarios:
   -packet to packet delay variation
   -delta between min and max packet delay variation for all
    packets sent.

3.3 Measurement

    The jitter MUST be measured when sending packets of the same size.
    Jitter MUST be measured as packet to packet delay variation and delta
    between min and max packet delay variation of all packets sent. A
    histogram MAY be provided as a population of packets measured per
    latency or latency buckets.


We have observed certain benchmarking reports to abuse and not be RFC 
compliant when measuring jitter, not allowing anymore an apple-to-apple 
comparison between different DUTs.

The hope here is to have an informational document providing guidance 
for the datacenter benchmarking [including the jitter].

Al you raise a key point. When the jitter results are provided to a 
datacenter engineer, it will give him an idea of the variance of 
performance in a steady environment to understand what is the actual 
real latency addition that he may observe.

For example a 200 nanosecond latency for a DUT under FIFO, with a 10 
nanosecond of jitter at a given packet size will provide a 200+-10 
variance of latency the end user can expect. This gives an order of 
magnitude for the end performance one can actually have in production 
environment. If the jitter on a DUT is high [for the previous example 
say 100 ns] then a user will understand that the latency can vary up to 
50% instead of 5% [with the 10ns example].


Hope that clarifies it,

Cheers,
Lucien


On 7/30/13 7:39 PM, MORTON JR., ALFRED C (AL) wrote:
> Hi David and Lucien,
>
> Thanks for the additional reference, David.
>
> The definition in 4689 is like the Interpacket delay variation definition
> that can be formulated using 3393, but these definitions both depend on
> the sequence of delays. Also, the absolute value function used in 4689
> colapses the positive and negative inter-packet delays together,
> and sometimes these have different meanings or causes.
>
> I think the question I'm asking is
> "How with the Jitter results be used by Datacenter Engineers?"
>
> If, for example, they wish to know the overall jitter from a path
> with ten DUTs in series, we can estimate that value but we have to
> measure the right metric at the start to do it.
>
> hope this helps, or gets people thinking at least.
> regards,
> Al
>
> ________________________________________
> From: [email protected] [[email protected]] On Behalf Of David Newman [[email protected]]
> Sent: Monday, July 29, 2013 5:35 PM
> To: [email protected]
> Subject: Re: [bmwg] comments on draft-dcbench-def-00 (part 1)
>
> On 7/29/13 9:14 AM, Lucien Avramov (lavramov) wrote:
>
>>> 3 Jitter
>>>
>>> 3.1 Definition
>>>
>>>      The definition of Jitter is covered extensively in RFC 3393. This
>>>      definition is not meant to replace that definition, but it is meant
>>>      to provide guidance of use for data center network devices.
>>>
>>>      The use of Jitter is in according with the variation delay definition
>>>      from RFC 3393:
>>>
>>>      The second meaning has to do with the variation of a metric (e.g.,
>>>      delay) with respect to some reference metric (e.g., average delay or
>>>      minimum delay).  This meaning is frequently used by computer
>>>      scientists and frequently (but not always) refers to variation in
>>>      delay.
>>>
>>> Comment:  Even with the reference to RFC 3393, there are many definitions
>>> of "jitter" possible. Fortunately this is not the first draft to examine
>>> the possibilities :-)
>>>
>>> I find that a comment I made on the Traffic Management draft is
>>> applicable here:
>>>> ...
>>>>     - Packet Delay Variation (PDV): the Packet Delay Variation metric is
>>>>     the variation between the timestamp of the received egress port
>>>>     packets and specified in RFC 1889.
>>>>
>>>> Comment: since you have One-way Delay (PD defined above PDV), why
>>>> resort to the RFC 1889 definition?
>>>> What does this give you?
>>>> Suggest: Specify PDV as defined in
>>>> http://tools.ietf.org/html/rfc5481#section-4.2 instead, for the many
>>>> reasons given in http://tools.ietf.org/html/rfc5481#page-27
>>>> (or call this IPDV to avoid confusion).
>>>
>>> In the Isolated test lab environment, we don't have time clock drift
>>> issues that
>>> we can't solve, in many cases the same clock is used for ingress and
>>> egress timestamps.
>>> So, we are free to choose the delay variation metric(s) that serve our
>>> audience best.
>>>
>>>>>>> I think "Jitter" is another topic that we can advance through
>>>>>>> face2face discussion.
>>>        What do we really want to learn from the "jitter" metric?
>>>        How will these results be used? (in network design, application
>>> planning, etc.)
>>
>> For the Data Center benchmarking, we found that the rfc 3393 definition
>> was the easiest to consume for its applicability to the data center. we
>> wanted to add to this that what is key is to have packets of the same
>> size when jitter is measured. It's mentioned in the RFC 3393 but in a
>> more complex method as follows:
>>
>> '‘+ L, a packet length in bits. The packets of a Type P packet stream
>> from which the singleton ipdv metric is taken MUST all be of the same
>> length.’’
>>
>> We want to make this simpler to the end data center user by calling it
>> packets of the same size.
>>
>> Also we add to this definition, by explaining the necessity for a
>> histogram with latency buckets, this will help understand where there is
>> higher jitter for instance.
>
> RFC 4689, approved by this wg a few years ago, provides a simpler
> definition of jitter:
>
> http://tools.ietf.org/html/rfc4689#section-3.2.5
>
> |D(i) - D(i-1)|, where D equals the Forwarding Delay
>        and i is the order the packets were received.
>
> This paragraph compares the RFC 3393 and 4689 definitions:
>
> "Jitter is related to the IPDV (IP Delay Variation) taking the absolute
> value of the ipdv. The two metrics will produce different mean values.
> Mean Jitter will produce a positive value, where the mean ipdv is
> typically zero. Also, IPDV is undefined when one packet from a pair is
> lost."
>
> The RFC 4689 definition may be more applicable here.
>
> dn
>
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg
>