Re: Second WGLC: draft-ietf-bmwg-dcbench-terminology and methodology

"MORTON, ALFRED C (AL)" <[email protected]> Tue, 13 Sep 2016 15:09:00 -0400

Newsgroups	gmane.ietf.bmwg
Message-ID	<4AF73AA205019A4C8A1DDD32C034631D4598DC7370@NJFPSRVEXG0.research.att.com>

Hi Lucien and Jacob,

Here are my comments on the meth draft, below.

Al
(as a participant)

INTRODUCTION, paragraph 12:
OLD:

NEW:

Section 1., paragraph 1:
OLD:

Traffic patterns in the data center are not uniform and are
constantly changing. They are dictated by the nature and variety of
applications utilized in the data center. It can be largely east-west
traffic flows in one data center and north-south in another, while
some may combine both. Traffic patterns can be bursty in nature and
contain many-to-one, many-to-many, or one-to-many flows. Each flow
may also be small and latency sensitive or large and throughput
sensitive while containing a mix of UDP and TCP traffic. All of which
can coexist in a single cluster and flow through a single network
device all at the same time. Benchmarking of network devices have
long used RFC1242, RFC2432, RFC2544, RFC2889 and RFC3918. These
benchmarks have largely been focused around various latency
attributes and max throughput of the Device Under Test [DUT] being
benchmarked. These standards are good at measuring theoretical max
throughput, forwarding rates and latency under testing conditions
however, they do not represent real traffic patterns that may affect
these networking devices.

NEW:

Traffic patterns in the data center are not uniform and are
constantly changing. They are dictated by the nature and variety of
applications utilized in the data center. It can be largely east-west
traffic flows in one data center and north-south in another, while
some may combine both. Traffic patterns can be bursty in nature and
contain many-to-one, many-to-many, or one-to-many flows. Each flow
may also be small and latency sensitive or large and throughput
sensitive while containing a mix of UDP and TCP traffic. All of which
can coexist in a single cluster and flow through a single network
device all at the same time. Benchmarking of network devices have
AM long used RFC1242, RFC2432, RFC2544, RFC2889 and RFC3918. These <<<add [ref#s]
benchmarks have largely been focused around various latency
AM attributes and Throughput [2] of the Device Under Test (DUT) being
AM benchmarked. These standards are good at measuring theoretical
AM Throughput, forwarding rates and latency under testing conditions
however, they do not represent real traffic patterns that may affect
these networking devices.

Section 1.2., paragraph 4:
OLD:

-Reporting Format

NEW:

AM -Reporting Format: Additional interpretation of RFC2119 terms:

Section 1.2., paragraph 5:
OLD:

MUST: minimum test for the scenario described

NEW:

AM MUST: required metric or benchmark for the scenario described (minimum)

Section 1.2., paragraph 6:
OLD:

SHOULD: recommended test for the scenario described

NEW:

AM SHOULD or RECOMMENDED: strongly suggested metric for the scenario described

Section 1.2., paragraph 7:
OLD:

MAY: ideal test for the scenario described

NEW:

AM MAY: Comprehensive metric for the scenario described

Section 1.2., paragraph 8:
OLD:

For each test methodology described, it is key to obtain
repeatability of the results. The recommendation is to perform enough
iterations of the given test to make sure the result is accurate,
this is especially important for section 3) as the buffering testing
has been historically the least reliable.

NEW:

AM For each test methodology described, it is critical to obtain
AM repeatability in the results. The recommendation is to perform enough
AM iterations of the given test and to make sure the result is consistent,
AM this is especially important for section 3, as the buffering testing
has historically been the least reliable.

Section 2.1, paragraph 1:
OLD:

Provide at maximum rate test for the performance values for
throughput, latency and jitter. It is meant to provide the tests to
run and methodology to verify that a DUT is capable of forwarding
packets at line rate under non-congested conditions.

NEW:

AM Provide a maximum rate test for the performance values for
AM Throughput, latency and jitter. It is meant to provide the tests to
AM perform and methodology to verify that a DUT is capable of forwarding
packets at line rate under non-congested conditions.

Section 2.2, paragraph 1:
OLD:

A traffic generator SHOULD be connected to all ports on the DUT. Two
tests MUST be conducted: a port-pair test [RFC 2544/3918 compliant]
and also in a full mesh type of DUT test [RFC 2889/3918 compliant].

NEW:

A traffic generator SHOULD be connected to all ports on the DUT. Two
AM tests MUST be conducted: a port-pair test [RFC 2544/3918 section ?? compliant]
AM and also in a full mesh type of DUT test [RFC 2889/3918 section ?? compliant].

Section 2.2, paragraph 2:
OLD:

For all tests, the percentage of traffic per port capacity sent MUST
be 99.98% at most, with no PPM adjustment to ensure stressing the DUT
in worst case conditions. Tests results at a lower rate MAY be
provided for better understanding of performance increase in terms of
latency and jitter when the rate is lower than 99.98%. The receiving
rate of the traffic needs to be captured during this test in % of
line rate.

NEW:

For all tests, the percentage of traffic per port capacity sent MUST
be 99.98% at most, with no PPM adjustment to ensure stressing the DUT
in worst case conditions. Tests results at a lower rate MAY be
provided for better understanding of performance increase in terms of
latency and jitter when the rate is lower than 99.98%. The receiving
AM rate of the traffic should be reported during this test in % of
line rate.

Section 2.2, paragraph 3:
OLD:

The test MUST provide the latency values for minimum, average and
maximum, for the exact same iteration of the test.

NEW:

AM The test MUST provide the statistics of minimum, average and
AM maximum of the latency distribution, for the exact same iteration of the test.

Section 2.2, paragraph 4:
OLD:

The test MUST provide the jitter values for minimum, average and
maximum, for the exact same iteration of the test.

NEW:

AM The test MUST provide the statistics of minimum, average and
AM maximum of the jitter distribution, for the exact same iteration of the test.

Section 2.3, paragraph 2:
OLD:

-physical layer calibration information as defined into (Placeholder
for definitions draft)

NEW:

AM -physical layer calibration information as defined into (Placeholder
AM for definitions draft) ????? ref to terms draft section ???

Section 2.3, paragraph 4:
OLD:

-reading for throughput received in percentage of bandwidth, while
sending 99.98% of port capacity on each port, across packet size from
64 byte all the way to 9216. As guidance, an increment of 64 byte
packet size between each iteration being ideal, a 256 byte and 512
bytes being also often time used, the most common packets sizes order
for the report is: 64b,128b,256b,512b,1024b,1518b,4096,8000,9216b.

NEW:

AM -reading for Throughput received in percentage of bandwidth, while
AM sending 99.98% of port capacity on each port, for each packet size from
AM 64 bytes to 9216 bytes. As guidance, an increment of 64 byte
packet size between each iteration being ideal, a 256 byte and 512
bytes being also often time used, the most common packets sizes order
for the report is: 64b,128b,256b,512b,1024b,1518b,4096,8000,9216b.

Section 2.3, paragraph 5:
OLD:

The pattern for testing can be expressed using RFC 6985 [IMIX Genome:
Specification of Variable Packet Sizes for Additional Testing]

NEW:

AM For IMIX testing, the pattern for testing can be expressed using RFC 6985 [IMIX Genome:
AM Specification of Variable Packet Sizes for Additional Testing] << add to refs!

Section 2.3, paragraph 6:
OLD:

-throughput needs to be expressed in % of total transmitted frames
-for packet drops, they MUST be expressed in packet count value and
SHOULD be expressed in % of line rate

NEW:

-throughput needs to be expressed in % of total transmitted frames

AM -for packet drops, they MUST be expressed as a count of packets and
SHOULD be expressed in % of line rate

Section 2.3, paragraph 10:
OLD:

-The tests for throughput, latency and jitter MAY be conducted as
individual independent events, with proper documentation in the
report but SHOULD be conducted at the same time.

NEW:

AM -The tests for Throughput, latency and jitter MAY be conducted as
AM individual independent trials, with proper documentation in the
report but SHOULD be conducted at the same time.

Section 3.2, paragraph 2:
OLD:

The methodology for measuring buffering for a data-center switch is
based on using known congestion of known fixed packet size along with
maximum latency value measurements. The maximum latency will increase
until the first packet drop occurs. At this point, the maximum
latency value will remain constant. This is the point of inflexion of
this maximum latency change to a constant value. There MUST be
multiple ingress ports receiving known amount of frames at a known
fixed size, destined for the same egress port in order to create a
known congestion event. The total amount of packets sent from the
oversubscribed port minus one, multiplied by the packet size
represents the maximum port buffer size at the measured inflexion
point.

NEW:

The methodology for measuring buffering for a data-center switch is
based on using known congestion of known fixed packet size along with
maximum latency value measurements. The maximum latency will increase
until the first packet drop occurs. At this point, the maximum
latency value will remain constant. This is the point of inflexion of
this maximum latency change to a constant value. There MUST be
multiple ingress ports receiving known amount of frames at a known
fixed size, destined for the same egress port in order to create a
AM known congestion condition. The total amount of packets sent from the
oversubscribed port minus one, multiplied by the packet size
represents the maximum port buffer size at the measured inflexion
point.

Section 3.2, paragraph 4:
OLD:

First iteration: ingress port 1 sending line rate to egress port 2,
while port 3 sending a known low amount of over subscription traffic
(1% recommended) with a packet size of 64 bytes to egress port 2.
Measure the buffer size value of the number of frames sent from the
port sending the oversubscribed traffic up to the inflexion point
multiplied by the frame size.

NEW:

First iteration: ingress port 1 sending line rate to egress port 2,
AM while port 3 sending a known low amount of over-subscription traffic
(1% recommended) with a packet size of 64 bytes to egress port 2.
Measure the buffer size value of the number of frames sent from the
port sending the oversubscribed traffic up to the inflexion point
multiplied by the frame size.

Section 3.2, paragraph 5:
OLD:

Second iteration: ingress port 1 sending line rate to egress port 2,
while port 3 sending a known low amount of over subscription traffic
(1% recommended) with same packet size 65 bytes to egress port 2.
Measure the buffer size value of the number of frames sent from the
port sending the oversubscribed traffic up to the inflexion point
multiplied by the frame size.

NEW:

Second iteration: ingress port 1 sending line rate to egress port 2,
AM while port 3 sending a known low amount of over-subscription traffic
(1% recommended) with same packet size 65 bytes to egress port 2.
Measure the buffer size value of the number of frames sent from the
port sending the oversubscribed traffic up to the inflexion point
multiplied by the frame size.

Section 3.2, paragraph 6:
OLD:

Last iteration: ingress port 1 sending line rate to egress port 2,
while port 3 sending a known low amount of over subscription traffic
(1% recommended) with same packet size B bytes to egress port 2.
Measure the buffer size value of the number of frames sent from the
port sending the oversubscribed traffic up to the inflexion point
multiplied by the frame size..

NEW:

AM Continuing iterations: ingress port 1 sending line rate to egress port 2,
AM while port 3 sending a known low amount of over-subscription traffic
(1% recommended) with same packet size B bytes to egress port 2.
Measure the buffer size value of the number of frames sent from the
port sending the oversubscribed traffic up to the inflexion point
AM multiplied by the frame size.

Section 3.2, paragraph 7:
OLD:

When the B value is found to provide the highest buffer size, this is
the highest buffer efficiency

NEW:

AM When the B value is found to provide the largest buffer size, then
AM size B allows the highest buffer efficiency.

Section 3.2, paragraph 9:
OLD:

At fixed packet size B determined in 3.2.1, for a fixed default COS
value of 0 and for unicast traffic proceed with the following:

NEW:

AM At fixed packet size B determined in procedure 1), for a fixed default DSCP ??
value of 0 and for unicast traffic proceed with the following:

Section 3.2, paragraph 10:
OLD:

First iteration: ingress port 1 sending line rate to egress port 2,
while port 3 sending a known low amount of over subscription traffic
(1% recommended) with same packet size to the egress port 2. Measure
the buffer size value by multiplying the number of extra frames sent
by the frame size.

NEW:

First iteration: ingress port 1 sending line rate to egress port 2,
AM while port 3 sending a known low amount of over-subscription traffic
(1% recommended) with same packet size to the egress port 2. Measure
the buffer size value by multiplying the number of extra frames sent
by the frame size.

Section 3.2, paragraph 11:
OLD:

Second iteration: ingress port 2 sending line rate to egress port 3,
while port 4 sending a known low amount of over subscription traffic
(1% recommended) with same packet size to the egress port 3. Measure
the buffer size value by multiplying the number of extra frames sent
by the frame size.

NEW:

Second iteration: ingress port 2 sending line rate to egress port 3,
AM while port 4 sending a known low amount of over-subscription traffic
(1% recommended) with same packet size to the egress port 3. Measure
the buffer size value by multiplying the number of extra frames sent
by the frame size.

Section 3.2, paragraph 12:
OLD:

Last iteration: ingress port N-2 sending line rate traffic to egress
port N-1, while port N sending a known low amount of over
subscription traffic (1% recommended) with same packet size to the
egress port N Measure the buffer size value by multiplying the number
of extra frames sent by the frame size.

NEW:

Last iteration: ingress port N-2 sending line rate traffic to egress
AM port N-1, while port N sending a known low amount of over-
subscription traffic (1% recommended) with same packet size to the
egress port N Measure the buffer size value by multiplying the number
of extra frames sent by the frame size.

Section 3.2, paragraph 13:
OLD:

This test series MAY be repeated using all different COS values of
traffic and then using Multicast type of traffic, in order to find if
there is any COS impact on the buffer size.

NEW:

AM This test series MAY be repeated using all different DSCP? values of
traffic and then using Multicast type of traffic, in order to find if
AM there is any DSCP? impact on the buffer size.

Section 3.2, paragraph 18:
OLD:

This test series MAY be repeated using all different COS values of
traffic and then using Multicast type of traffic.

NEW:

AM This test series MAY be repeated using all different DSCP? values of
traffic and then using Multicast type of traffic.

Section 3.2, paragraph 23:
OLD:

This test series MAY be repeated using all different COS values of
traffic and then using Multicast type of traffic.

NEW:

AM This test series MAY be repeated using all different DSCP? values of
traffic and then using Multicast type of traffic.

Section 3.2, paragraph 25:
OLD:

Also the COS value for the packets SHOULD be provided for each test
iteration as the buffer allocation size MAY differ per COS value. It
is RECOMMENDED that the ingress and egress ports are varied in a
random, but documented fashion in multiple tests to measure the
buffer size for each port of the DUT.

NEW:

AM Also the DSCP? value for the packets SHOULD be provided for each test
iteration as the buffer allocation size MAY differ per COS value. It
is RECOMMENDED that the ingress and egress ports are varied in a
random, but documented fashion in multiple tests to measure the
buffer size for each port of the DUT.

Section 3.3, paragraph 2:
OLD:

- The packet size used for the most efficient buffer used, along
with COS value

NEW:

- The packet size used for the most efficient buffer used, along
AM with DSCP? value

Section 3.3, paragraph 6:
OLD:

- The amount of over subscription if different than 1%

NEW:

AM - The amount of over-subscription if different than 1%

Section 4.2, paragraph 1:
OLD:

A traffic generator MUST be connected to all ports on the DUT. In
order to cause congestion, two or more ingress ports MUST bursts
packets destined for the same egress port. The simplest of the setups
would be two ingress ports and one egress port (2-to-1).

NEW:

A traffic generator MUST be connected to all ports on the DUT. In
AM order to cause congestion, two or more ingress ports MUST send bursts of
packets destined for the same egress port. The simplest of the setups
would be two ingress ports and one egress port (2-to-1).

Section 4.2, paragraph 2:
OLD:

The burst MUST be measure with an intensity of 100%, meaning the
burst of packets will be sent with a minimum inter-packet gap. The
amount of packet contained in the burst will be variable and increase
until there is a non-zero packet loss measured. The aggregate amount
of packets from all the senders will be used to calculate the maximum
amount of microburst the DUT can sustain.

NEW:

AM The burst MUST be sent with an intensity of 100%, meaning the
burst of packets will be sent with a minimum inter-packet gap. The
AM amount of packets contained in the burst will be trial variable and increased
until there is a non-zero packet loss measured. The aggregate amount
of packets from all the senders will be used to calculate the maximum
amount of microburst the DUT can sustain.

Section 4.3, paragraph 2:
OLD:

- The maximum value of packets received per ingress port with the
maximum burst size obtained with zero packet loss

NEW:

AM - The maximum number of packets received per ingress port with the
maximum burst size obtained with zero packet loss

Section 4.3, paragraph 5:
OLD:

- The repeatability of the test needs to be indicated: number of
iteration of the same test and percentage of variation between
results (min, max, avg)

NEW:

- The repeatability of the test needs to be indicated: number of
AM iterations of the same test and percentage of variation between
results (min, max, avg)

Section 5.1, paragraph 1:
OLD:

Head-of-line blocking (HOL blocking) is a performance-limiting
phenomenon that occurs when packets are held-up by the first packet
ahead waiting to be transmitted to a different output port. This is
defined in RFC 2889 section 5.5. Congestion Control. This section
expands on RFC 2889 in the context of Data Center Benchmarking
The objective of this test is to understand the DUT behavior under
head of line blocking scenario and measure the packet loss.

NEW:

Head-of-line blocking (HOL blocking) is a performance-limiting
phenomenon that occurs when packets are held-up by the first packet
ahead waiting to be transmitted to a different output port. This is
AM defined in RFC 2889 section 5.5, Congestion Control. This section
expands on RFC 2889 in the context of Data Center Benchmarking

AM The objective of this test is to understand the DUT behavior under a
head of line blocking scenario and measure the packet loss.

Section 5.2, paragraph 1:
OLD:

In order to cause congestion, head of line blocking, groups of four
ports are used. A group has 2 ingress and 2 egress ports. The first
ingress port MUST have two flows configured each going to a different
egress port. The second ingress port will congest the second egress
port by sending line rate. The goal is to measure if there is loss
for the first egress port which is not not oversubscribed.

NEW:

AM In order to cause congestion in the form of head of line blocking, groups of four
ports are used. A group has 2 ingress and 2 egress ports. The first
ingress port MUST have two flows configured each going to a different
egress port. The second ingress port will congest the second egress
port by sending line rate. The goal is to measure if there is loss
AM on the flow for the first egress port which is not oversubscribed.

Section 9., paragraph 3:
OLD:

2) Measure with N/4 groups with N DUT ports

First iteration: Expand to fully utilize all the DUT ports in
increments of four. Repeat the methodology of 1) with all the group
of ports possible to achieve on the device and measure for each port
group the amount of traffic loss.

NEW:

2) Measure with N/4 groups with N DUT ports
AM QUESTION: Is the traffic from ingress split across 4 egress ports (25%)??
First iteration: Expand to fully utilize all the DUT ports in
increments of four. Repeat the methodology of 1) with all the group
of ports possible to achieve on the device and measure for each port
group the amount of traffic loss.

Section 5.3, paragraph 3:
OLD:

- If HOLB was observed

NEW:

AM - If HOLB was observed (Need to say what measurement supports this conclusion)

Section 6.1, paragraph 1:
OLD:

The objective of this test is to measure the effect of TCP Goodput
and latency with a mix of large and small flows. The test is designed
to simulate a mixed environment of stateful flows that require high
rates of goodput and stateless flows that require low latency.

NEW:

AM The objective of this test is to measure the values for TCP Goodput
and latency with a mix of large and small flows. The test is designed
to simulate a mixed environment of stateful flows that require high
rates of goodput and stateless flows that require low latency.

Section 6.2, paragraph 8:
OLD:

First Iteration: 1 Ingress port receiving stateful TCP traffic and 1
Ingress port receiving stateless traffic destined to 1 Egress Ports

NEW:

First Iteration: 1 Ingress port receiving stateful TCP traffic and 1
AM Ingress port receiving stateless traffic destined to 1 Egress Port

Section 6.2, paragraph 9:
OLD:

Second Iteration: 2 Ingress port receiving stateful TCP traffic and 1
Ingress port receiving stateless traffic destined to 1 Egress Ports

NEW:

AM Second Iteration: 2 Ingress ports receiving stateful TCP traffic and 1
AM Ingress port receiving stateless traffic destined to 1 Egress Port

Section 6.2, paragraph 10:
OLD:

Last Iteration: N-2 Ingress port receiving stateful TCP traffic and 1
Ingress port receiving stateless traffic destined to 1 Egress Ports

NEW:

AM Last Iteration: N-2 Ingress ports receiving stateful TCP traffic and 1
AM Ingress port receiving stateless traffic destined to 1 Egress Port

Section 6.2, paragraph 12:
OLD:

During Iterations number of Egress ports MAY vary as well. First
Iteration: 1 Ingress port receiving stateful TCP traffic and 1
Ingress port receiving stateless traffic destined to 1 Egress Ports

NEW:

During Iterations number of Egress ports MAY vary as well. First
Iteration: 1 Ingress port receiving stateful TCP traffic and 1
AM Ingress port receiving stateless traffic destined to 1 Egress Port

Section 6.2, paragraph 13:
OLD:

Second Iteration: 1 Ingress port receiving stateful TCP traffic and 2
Ingress port receiving stateless traffic destined to 1 Egress Ports

NEW:

Second Iteration: 1 Ingress port receiving stateful TCP traffic and 2
AM Ingress port receiving stateless traffic destined to 1 Egress Port

Section 6.2, paragraph 14:
OLD:

Last Iteration: 1 Ingress port receiving stateful TCP traffic and N-2
Ingress port receiving stateless traffic destined to 1 Egress Ports

NEW:

Last Iteration: 1 Ingress port receiving stateful TCP traffic and N-2

AM Ingress port receiving stateless traffic destined to 1 Egress Port

Section 6.3, paragraph 2:
OLD:

- Number of ingress and egress ports along with designation of
stateful or stateless.

NEW:

- Number of ingress and egress ports along with designation of
AM stateful or stateless flow assignment.

Section 6.3, paragraph 3:
OLD:

- Stateful goodput

NEW:

AM - Stateful flow goodput

Section 6.3, paragraph 4:
OLD:

- Stateless latency

NEW:

AM - Stateless flow latency

Section 7.2., paragraph 3:
OLD:

[5] Stopp D. and Hickman B., "Methodology for IP Multicast
Benchmarking", BCP 26, RFC 3918, October 2004.

7.3. URL References

NEW:

[5] Stopp D. and Hickman B., "Methodology for IP Multicast
AM Benchmarking", RFC 3918, October 2004.

Section 7.2., paragraph 4:
OLD:

[6] Yanpei Chen, Rean Griffith, Junda Liu, Randy H. Katz, Anthony D.
Joseph, "Understanding TCP Incast Throughput Collapse in
Datacenter Networks",
http://www.eecs.berkeley.edu/~ychen2/professional/TCPIncastWREN2009.pdf".

NEW:

AM (7.3 heading removed)
[6] Yanpei Chen, Rean Griffith, Junda Liu, Randy H. Katz, Anthony D.
Joseph, "Understanding TCP Incast Throughput Collapse in
Datacenter Networks",
http://www.eecs.berkeley.edu/~ychen2/professional/TCPIncastWREN2009.pdf".

> -----Original Message-----
> From: bmwg [mailto:[email protected]] On Behalf Of MORTON, ALFRED C
> (AL)
> Sent: Tuesday, September 13, 2016 2:43 PM
> To: [email protected]
> Subject: [bmwg] Second WGLC: draft-ietf-bmwg-dcbench-terminology and
> methodology
>
> *** Security Advisory: This Message Originated Outside of AT&T ***.
> Reference http://cso.att.com/EmailSecurity/IDSP.html for more
> information.
>
> BMWG:
>
> A WG Last Call period for the Internet-Drafts on
> Data Center Benchmarking Terminology and Methodology:
>
> https://tools.ietf.org/html/draft-ietf-bmwg-dcbench-terminology-05
> https://tools.ietf.org/html/draft-ietf-bmwg-dcbench-methodology-02
>
> will be open from 13 September 2016 through 27 September 2016.
>
> The first WGLC closed on 8 April 2016 with comments.
>
> These drafts are continuing the BMWG Last Call Process. See
> http://www1.ietf.org/mail-archive/web/bmwg/current/msg00846.html
>
> Please read and express your opinion on whether or not these
> Internet-Drafts should be forwarded to the Area Directors for
> publication as Informational RFCs. Send your comments
> to this list or to the co-chairs [email protected] and
> [email protected]
>
> for the co-chairs,
> Al
>
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/bmwg

draft-ietf-bmwg-dcbench-methodology-02acm.txt (text/plain, 30.2 KB)

Internet Engineering Task Force L. Avramov
Internet-Draft, Intended status: Informational Cisco Systems
Expires October 29, 2016 J. Rapp
April 27, 2016 VMware

Data Center Benchmarking Methodology
draft-ietf-bmwg-dcbench-methodology-02

Abstract

The purpose of this informational document is to establish test and
evaluation methodology and measurement techniques for physical
network equipment in the data center.

Status of this Memo

This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.

Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."

The list of current Internet-Drafts can be accessed at
http://www.ietf.org/1id-abstracts.html

The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html

This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents

Avramov & Rapp Expires October 29, 2016 [Page 1]

Internet-Draft Data Center Benchmarking Methodology April 27, 2016

(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.

Avramov & Rapp Expires October 29, 2016 [Page 2]

Internet-Draft Data Center Benchmarking Methodology April 27, 2016

Table of Contents

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Requirements Language . . . . . . . . . . . . . . . . . . 5
1.2. Methodology format and repeatability recommendation . . . . 5
2. Line Rate Testing . . . . . . . . . . . . . . . . . . . . . . . 5
2.1 Objective . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 Reporting Format . . . . . . . . . . . . . . . . . . . . . . 6
3. Buffering Testing . . . . . . . . . . . . . . . . . . . . . . . 7
3.1 Objective . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . 7
3.3 Reporting format . . . . . . . . . . . . . . . . . . . . . . 10
4 Microburst Testing . . . . . . . . . . . . . . . . . . . . . . . 10
4.1 Objective . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . 10
4.3 Reporting Format . . . . . . . . . . . . . . . . . . . . . . 11
5. Head of Line Blocking . . . . . . . . . . . . . . . . . . . . . 11
5.1 Objective . . . . . . . . . . . . . . . . . . . . . . . . . 11
5.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . 12
5.3 Reporting Format . . . . . . . . . . . . . . . . . . . . . . 13
6. Incast Stateful and Stateless Traffic . . . . . . . . . . . . . 13
6.1 Objective . . . . . . . . . . . . . . . . . . . . . . . . . 13
6.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . 13
6.3 Reporting Format . . . . . . . . . . . . . . . . . . . . . . 15
7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 15
7.1. Normative References . . . . . . . . . . . . . . . . . . . 16
7.2. Informative References . . . . . . . . . . . . . . . . . . 16
7.3. URL References . . . . . . . . . . . . . . . . . . . . . . 16
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 16

1. Introduction

Traffic patterns in the data center are not uniform and are
constantly changing. They are dictated by the nature and variety of
applications utilized in the data center. It can be largely east-west
traffic flows in one data center and north-south in another, while
some may combine both. Traffic patterns can be bursty in nature and
contain many-to-one, many-to-many, or one-to-many flows. Each flow
may also be small and latency sensitive or large and throughput
sensitive while containing a mix of UDP and TCP traffic. All of which
can coexist in a single cluster and flow through a single network
device all at the same time. Benchmarking of network devices have
AM long used RFC1242, RFC2432, RFC2544, RFC2889 and RFC3918. These <<<add [ref#s]
benchmarks have largely been focused around various latency
AM attributes and Throughput [2] of the Device Under Test (DUT) being

Avramov & Rapp Expires October 29, 2016 [Page 3]

Internet-Draft Data Center Benchmarking Methodology April 27, 2016

AM benchmarked. These standards are good at measuring theoretical
AM Throughput, forwarding rates and latency under testing conditions
however, they do not represent real traffic patterns that may affect
these networking devices.

The following provides a methodology for benchmarking Data Center DUT
including congestion scenarios, switch buffer analysis, microburst,
head of line blocking, while also using a wide mix of traffic
conditions.

Avramov & Rapp Expires October 29, 2016 [Page 4]

Internet-Draft Data Center Benchmarking Methodology April 27, 2016

1.1. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [6].

1.2. Methodology format and repeatability recommendation

The format used for each section of this document is the following:

-Objective

-Methodology

AM -Reporting Format: Additional interpretation of RFC2119 terms:

AM MUST: required metric or benchmark for the scenario described (minimum)

AM SHOULD or RECOMMENDED: strongly suggested metric for the scenario described

AM MAY: Comprehensive metric for the scenario described

2. Line Rate Testing

2.1 Objective

2.2 Methodology

For all tests, the percentage of traffic per port capacity sent MUST
be 99.98% at most, with no PPM adjustment to ensure stressing the DUT

Avramov & Rapp Expires October 29, 2016 [Page 5]

Internet-Draft Data Center Benchmarking Methodology April 27, 2016

in worst case conditions. Tests results at a lower rate MAY be
provided for better understanding of performance increase in terms of
latency and jitter when the rate is lower than 99.98%. The receiving
AM rate of the traffic should be reported during this test in % of
line rate.

AM The test MUST provide the statistics of minimum, average and
AM maximum of the latency distribution, for the exact same iteration of the test.

AM The test MUST provide the statistics of minimum, average and
AM maximum of the jitter distribution, for the exact same iteration of the test.

Alternatively when a traffic generator CAN NOT be connected to all
ports on the DUT, a snake test MUST be used for line rate testing,
excluding latency and jitter as those became then irrelevant. The
snake test consists in the following method: -connect the first and
last port of the DUT to a traffic generator-connect back to back
sequentially all the ports in between: port 2 to 3, port 4 to 5 etc
to port n-2 to port n-1; where n is the total number of ports of the
DUT-configure port 1 and 2 in the same vlan X, port 3 and 4 in the
same vlan Y, etc. port n-1 and port n in the same vlan ZZZ. This
snake test provides a capability to test line rate for Layer 2 and
Layer 3 RFC 2544/3918 in instance where a traffic generator with only
two ports is available. The latency and jitter are not to be
considered with this test.

2.3 Reporting Format

The report MUST include:

AM -physical layer calibration information as defined into (Placeholder
AM for definitions draft) ????? ref to terms draft section ???

-number of ports used

AM For IMIX testing, the pattern for testing can be expressed using RFC 6985 [IMIX Genome:
AM Specification of Variable Packet Sizes for Additional Testing] << add to refs!

-throughput needs to be expressed in % of total transmitted frames

Avramov & Rapp Expires October 29, 2016 [Page 6]

Internet-Draft Data Center Benchmarking Methodology April 27, 2016

AM -for packet drops, they MUST be expressed as a count of packets and
SHOULD be expressed in % of line rate

-for latency and jitter, values expressed in unit of time [usually
microsecond or nanosecond] reading across packet size from 64 bytes
to 9216 bytes

-for latency and jitter, provide minimum, average and maximum values.
if different iterations are done to gather the minimum, average and
maximum, it SHOULD be specified in the report along with a
justification on why the information could not have been gathered at
the same test iteration

-for jitter, a histogram describing the population of packets
measured per latency or latency buckets is RECOMMENDED

AM -The tests for Throughput, latency and jitter MAY be conducted as
AM individual independent trials, with proper documentation in the
report but SHOULD be conducted at the same time.

3. Buffering Testing

3.1 Objective

To measure the size of the buffer of a DUT under
typical|many|multiple conditions. Buffer architectures between
multiple DUTs can differ and include egress buffering, shared egress
buffering switch-on-chip [SoC], ingress buffering or a combination.
The test methodology covers the buffer measurement regardless of
buffer architecture used in the DUT.

3.2 Methodology

A traffic generator MUST be connected to all ports on the DUT.

The methodology for measuring buffering for a data-center switch is
based on using known congestion of known fixed packet size along with
maximum latency value measurements. The maximum latency will increase
until the first packet drop occurs. At this point, the maximum
latency value will remain constant. This is the point of inflexion of
this maximum latency change to a constant value. There MUST be
multiple ingress ports receiving known amount of frames at a known
fixed size, destined for the same egress port in order to create a
AM known congestion condition. The total amount of packets sent from the
oversubscribed port minus one, multiplied by the packet size

Avramov & Rapp Expires October 29, 2016 [Page 7]

Internet-Draft Data Center Benchmarking Methodology April 27, 2016

represents the maximum port buffer size at the measured inflexion
point.

1) Measure the highest buffer efficiency

AM When the B value is found to provide the largest buffer size, then
AM size B allows the highest buffer efficiency.

2) Measure maximum port buffer size

AM At fixed packet size B determined in procedure 1), for a fixed default DSCP ??
value of 0 and for unicast traffic proceed with the following:

Last iteration: ingress port N-2 sending line rate traffic to egress
AM port N-1, while port N sending a known low amount of over-

Avramov & Rapp Expires October 29, 2016 [Page 8]

Internet-Draft Data Center Benchmarking Methodology April 27, 2016

subscription traffic (1% recommended) with same packet size to the
egress port N Measure the buffer size value by multiplying the number
of extra frames sent by the frame size.

AM This test series MAY be repeated using all different DSCP? values of
traffic and then using Multicast type of traffic, in order to find if
AM there is any DSCP? impact on the buffer size.

3) Measure maximum port pair buffer sizes

First iteration: ingress port 1 sending line rate to egress port 2;
ingress port 3 sending line rate to egress port 4 etc. Ingress port
N-1 and N will respectively over subscribe at 1% of line rate egress
port 2 and port 3. Measure the buffer size value by multiplying the
number of extra frames sent by the frame size for each egress port.

Second iteration: ingress port 1 sending line rate to egress port 2;
ingress port 3 sending line rate to egress port 4 etc. Ingress port
N-1 and N will respectively over subscribe at 1% of line rate egress
port 4 and port 5. Measure the buffer size value by multiplying the
number of extra frames sent by the frame size for each egress port.

Last iteration: ingress port 1 sending line rate to egress port 2;
ingress port 3 sending line rate to egress port 4 etc. Ingress port
N-1 and N will respectively over subscribe at 1% of line rate egress
port N-3 and port N-2. Measure the buffer size value by multiplying
the number of extra frames sent by the frame size for each egress
port.

AM This test series MAY be repeated using all different DSCP? values of
traffic and then using Multicast type of traffic.

4) Measure maximum DUT buffer size with many to one ports

First iteration: ingress ports 1,2,... N-1 sending each [(1/[N-
1])*99.98]+[1/[N-1]] % of line rate per port to the N egress port.

Second iteration: ingress ports 2,... N sending each [(1/[N-
1])*99.98]+[1/[N-1]] % of line rate per port to the 1 egress port.

Last iteration: ingress ports N,1,2...N-2 sending each [(1/[N-
1])*99.98]+[1/[N-1]] % of line rate per port to the N-1 egress port.

AM This test series MAY be repeated using all different DSCP? values of
traffic and then using Multicast type of traffic.

Unicast traffic and then Multicast traffic SHOULD be used in order to
determine the proportion of buffer for documented selection of tests.

Avramov & Rapp Expires October 29, 2016 [Page 9]

Internet-Draft Data Center Benchmarking Methodology April 27, 2016

3.3 Reporting format

The report MUST include:

- The packet size used for the most efficient buffer used, along
AM with DSCP? value

- The maximum port buffer size for each port

- The maximum DUT buffer size

- The packet size used in the test

AM - The amount of over-subscription if different than 1%

- The number of ingress and egress ports along with their location
on the DUT.

- The repeatability of the test needs to be indicated: number of
iteration of the same test and percentage of variation between
results (min, max, avg)

4 Microburst Testing

4.1 Objective

To find the maximum amount of packet bursts a DUT can sustain under
various configurations.

4.2 Methodology

Avramov & Rapp Expires October 29, 2016 [Page 10]

Internet-Draft Data Center Benchmarking Methodology April 27, 2016

until there is a non-zero packet loss measured. The aggregate amount
of packets from all the senders will be used to calculate the maximum
amount of microburst the DUT can sustain.

It is RECOMMENDED that the ingress and egress ports are varied in
multiple tests to measure the maximum microburst capacity.

The intensity of a microburst MAY be varied in order to obtain the
microburst capacity at various ingress rates.

It is RECOMMENDED that all ports on the DUT will be tested
simultaneously and in various configurations in order to understand
all the combinations of ingress ports, egress ports and intensities.

An example would be:

First Iteration: N-1 Ingress ports sending to 1 Egress Ports

Second Iterations: N-2 Ingress ports sending to 2 Egress Ports

Last Iterations: 2 Ingress ports sending to N-2 Egress Ports

4.3 Reporting Format

The report MUST include:

AM - The maximum number of packets received per ingress port with the
maximum burst size obtained with zero packet loss

- The packet size used in the test

- The number of ingress and egress ports along with their location
on the DUT

- The repeatability of the test needs to be indicated: number of
AM iterations of the same test and percentage of variation between
results (min, max, avg)

5. Head of Line Blocking

5.1 Objective

Head-of-line blocking (HOL blocking) is a performance-limiting
phenomenon that occurs when packets are held-up by the first packet
ahead waiting to be transmitted to a different output port. This is
AM defined in RFC 2889 section 5.5, Congestion Control. This section
expands on RFC 2889 in the context of Data Center Benchmarking

Avramov & Rapp Expires October 29, 2016 [Page 11]

Internet-Draft Data Center Benchmarking Methodology April 27, 2016

AM The objective of this test is to understand the DUT behavior under a
head of line blocking scenario and measure the packet loss.

5.2 Methodology

A traffic generator MUST be connected to at least eight ports on the
DUT and SHOULD be connected using all the DUT ports.

1) Measure two groups with eight DUT ports

First iteration: measure the packet loss for two groups with
consecutive ports

The first group is composed of: ingress port 1 is sending 50% of
traffic to egress port 3 and ingress port 1 is sending 50% of traffic
to egress port 4. Ingress port 2 is sending line rate to egress port
4. Measure the amount of traffic loss for the traffic from ingress
port 1 to egress port 3.

The second group is composed of: ingress port 5 is sending 50% of
traffic to egress port 7 and ingress port 5 is sending 50% of traffic
to egress port 8. Ingress port 6 is sending line rate to egress port
8. Measure the amount of traffic loss for the traffic from ingress
port 5 to egress port 7.

Second iteration: repeat the first iteration by shifting all the
ports from N to N+1

the first group is composed of: ingress port 2 is sending 50% of
traffic to egress port 4 and ingress port 2 is sending 50% of traffic
to egress port 5. Ingress port 3 is sending line rate to egress port
5. Measure the amount of traffic loss for the traffic from ingress
port 2 to egress port 4.

the second group is composed of: ingress port 6 is sending 50% of
traffic to egress port 8 and ingress port 6 is sending 50% of traffic
to egress port 9. Ingress port 7 is sending line rate to egress port
9. Measure the amount of traffic loss for the traffic from ingress
port 6 to egress port 8.

Avramov & Rapp Expires October 29, 2016 [Page 12]

Internet-Draft Data Center Benchmarking Methodology April 27, 2016

Last iteration: when the first port of the first group is connected
on the last DUT port and the last port of the second group is
connected to the seventh port of the DUT

Measure the amount of traffic loss for the traffic from ingress port
N to egress port 2 and from ingress port 4 to egress port 6.

Second iteration: Shift by +1 the start of each consecutive ports of
groups

Last iteration: Shift by N-1 the start of each consecutive ports of
groups and measure the traffic loss for each port group.

5.3 Reporting Format

For each test the report MUST include:

- The port configuration including the number and location of ingress
and egress ports located on the DUT

AM - If HOLB was observed (Need to say what measurement supports this conclusion)

- Percent of traffic loss

- The repeatability of the test needs to be indicated: number of
iteration of the same test and percentage of variation between
results (min, max, avg)

6. Incast Stateful and Stateless Traffic

6.1 Objective

6.2 Methodology

Avramov & Rapp Expires October 29, 2016 [Page 13]

Internet-Draft Data Center Benchmarking Methodology April 27, 2016

In order to simulate the effects of stateless and stateful traffic on
the DUT there MUST be multiple ingress ports receiving traffic
destined for the same egress port. There also MAY be a mix of
stateful and stateless traffic arriving on a single ingress port. The
simplest setup would be 2 ingress ports receiving traffic destined to
the same egress port.

One ingress port MUST be maintaining a TCP connection trough the
ingress port to a receiver connected to an egress port. Traffic in
the TCP stream MUST be sent at the maximum rate allowed by the
traffic generator. At the same time the TCP traffic is flowing
through the DUT the stateless traffic is sent destined to a receiver
on the same egress port. The stateless traffic MUST be a microburst
of 100% intensity.

It is RECOMMENDED that the ingress and egress ports are varied in
multiple tests to measure the maximum microburst capacity.

The intensity of a microburst MAY be varied in order to obtain the
microburst capacity at various ingress rates.

It is RECOMMENDED that all ports on the DUT be used in the test.

For example:

Stateful Traffic port variation:

During Iterations number of Egress ports MAY vary as well.

First Iteration: 1 Ingress port receiving stateful TCP traffic and 1
AM Ingress port receiving stateless traffic destined to 1 Egress Port

AM Second Iteration: 2 Ingress ports receiving stateful TCP traffic and 1
AM Ingress port receiving stateless traffic destined to 1 Egress Port

AM Last Iteration: N-2 Ingress ports receiving stateful TCP traffic and 1
AM Ingress port receiving stateless traffic destined to 1 Egress Port

Stateless Traffic port variation:

During Iterations number of Egress ports MAY vary as well. First
Iteration: 1 Ingress port receiving stateful TCP traffic and 1
AM Ingress port receiving stateless traffic destined to 1 Egress Port

Second Iteration: 1 Ingress port receiving stateful TCP traffic and 2
AM Ingress port receiving stateless traffic destined to 1 Egress Port

Last Iteration: 1 Ingress port receiving stateful TCP traffic and N-2

Avramov & Rapp Expires October 29, 2016 [Page 14]

Internet-Draft Data Center Benchmarking Methodology April 27, 2016

AM Ingress port receiving stateless traffic destined to 1 Egress Port

6.3 Reporting Format

The report MUST include the following:

- Number of ingress and egress ports along with designation of
AM stateful or stateless flow assignment.

AM - Stateful flow goodput

AM - Stateless flow latency

- The repeatability of the test needs to be indicated: number of
iteration of the same test and percentage of variation between
results (min, max, avg)

7. References

Avramov & Rapp Expires October 29, 2016 [Page 15]

Internet-Draft Data Center Benchmarking Methodology April 27, 2016

7.1. Normative References

[1] Bradner, S. "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, July 1991.

[2] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999.

7.2. Informative References

[3] Avramov L. and Rapp J., "Data Center Benchmarking Terminology",
April 2016.

[4] Mandeville R. and Perser J., "Benchmarking Methodology for LAN
Switching Devices", RFC 2889, August 2000.

[5] Stopp D. and Hickman B., "Methodology for IP Multicast
AM Benchmarking", RFC 3918, October 2004.

Authors' Addresses

Lucien Avramov
Cisco Systems
170 West Tasman drive
San Jose, CA 95134
United States
Phone: +1 408 526 7686
Email: [email protected]

Jacob Rapp
VMware
3401 Hillview Ave
Palo Alto, CA
United States
Phone: +1 650 857 3367
Email: [email protected]

Avramov & Rapp Expires October 29, 2016 [Page 16]