AD review of draft-ietf-bmwg-dcbench-terminology.
Warren Kumari <[email protected]>
| Newsgroups | gmane.ietf.bmwg |
|---|---|
| Message-ID | <CAHw9_i+yE8a02bP=NQym6nxUhCJM6OOdhkRwv=2GamQjV+KS1A@mail.gmail.com> |
Hi there,
Apologies, I had forgotten to click the AD eval button... {DONE}
Thanks for a (generally) clear and useful document.
I have completed my AD review of draft-ietf-bmwg-dcbench-terminology and I
have a number of suggested edits to make the document even more readable /
clearer. In a few instances I wasn't quite sure what the document was
trying to say, so please let me know if any of the suggestions don't make
sense, or change the text meaning...
Also, please poke me once updated, so that I can progress it to IETF LC.
W
------
Abstract
The purpose of this informational document is to establish definitions,
discussion and measurement techniques for data center benchmarking.
[O] establish definitions, discussion and measurement techniques
[R] I'm not sure what is meant by "discussion" here. "Discussion and
measurement techniques" doesn't make much sense to me. Perhaps "to
establish definitions, foster discussion and describe measurement
techniques ..."?
Also, it is to introduce new terminologies applicable to data center
[O] benchmarking. Also, it is to introduce
[P] benchmarking, as well as introduce
[R] clarity/no need for a separate sentence.
performance evaluations. The purpose of this document is not to define
the test methodology, but rather establish the important concepts when
one is interested in benchmarking network switches and routers in the
[O] when one is interested in benchmarking
[P] for benchmarking
[R] readability
data center.
1. Introduction
Traffic patterns in the data center are not uniform and are contently
[O] contently
[P] constantly
[R] word choice/typo
changing. They are dictated by the nature and variety of applications
utilized in the data center. It can be largely east-west traffic
flows in one data center and north-south in another, while some may
combine both. Traffic patterns can be bursty in nature and contain
many-to-one, many-to-many, or one-to-many flows. Each flow may also
be small and latency sensitive or large and throughput sensitive
while containing a mix of UDP and TCP traffic. All of which can
coexist in a single cluster and flow through a single network device
all at the same time. Benchmarking of network devices have long used
[O] All of which can
coexist in a single cluster and flow through a single network device
all at the same time.
[P] One or more of these may coexist in a single cluster and flow through a
single network device simultaneously.
[R] grammar/readability - unless this changes the intent?
[RFC1242], [RFC2432], [RFC2544], [2] and [3]. These benchmarks have
largely been focused around various latency attributes and max
throughput of the Device Under Test being benchmarked. These
standards are good at measuring theoretical max throughput,
forwarding rates and latency under testing conditions, but to not
[O] , but to not
[P]; but they do not
[R] grammar
represent real traffic patterns that may affect these networking
devices. The data center networking devices covered are switches and
routers.
1.2. Definition format
Term to be defined. (e.g., Latency)
Definition: The specific definition for the term.
Discussion: A brief discussion about the term, it's application and
[O] it's
[P] its
[R] Grammar. Should be a possessive, not a contraction of "it is."
any restrictions on measurement procedures.
Measurement Units: Methodology for the measure and units used to
report measurements of this term, if applicable.
2. Latency
2.1. Definition
Latency is a the amount of time it takes a frame to transit the DUT.
Latency is measured in unit of time (seconds, milliseconds,
microseconds and so on). The purpose of measuring latency is to
understand what is the impact of adding a device in the communication
[O] understand what is the impact
[P] understand the impact
[R] grammar
path.
The Latency interval can be assessed between different combinations
of events, irrespectively of the type of switching device (bit
[O] irrespectively
[P] regardless
[R] grammar
forwarding aka cut-through or store forward type of device)
2.2 Discussion
FILO is the most important measuring definition. Any type of switches
[O] Any type of switches
[P] All switches
[R] readability
MUST be measured with the FILO mechanism: FILO will include the
latency of the switch and the latency of the frame as well as the
serialization delay. It is a picture of the 'whole' latency going
through the DUT. For applications, which are latency sensitive and
can function with initial bytes of the frame, FIFO MAY be an
additional type of measuring to supplement FILO.
3 Jitter
3.2 Discussion
In addition to PDV Range and or a high percentile of PDV, Inter-
[O] and or
[P] and/or
Packet Delay Variation (IPDV) as defined in section 4.1 of RFC5481
(differences between two consecutive packets) MAY be used for the
purpose of determining how packet spacing has changed during
transfer, for example to see if packet stream has become closely-
[O] for example to see
[P] for example, to see
[R] grammar
spaced or "bursty". However, the Absolute Value of IPDV SHOULD NOT be
used, as this collapses the "bursty" and "dispersed" sides of the
IPDV distribution together.
4 Physical Layer Calibration
4.2 Discussion
Physical layer calibration is part of the end to end latency, which
should be taken into acknowledgment while evaluating the DUT. Small
variations of the physical components of the test may impact the
latency being measure so they MUST be described when presenting
[O] latency being measure so
[P] latency being measured, so
[R] grammar
results.
5 Line rate
5.1 Definition
The transmit timing, or maximum transmitted data rate is controlled
by the "transmit clock" in the DUT. The receive timing (maximum
ingress data rate) is derived from the transmit clock of the
connected interface.
The line rate or physical layer frame rate is the maximum capacity to
send frames of a specific size at the transmit clock frequency of the
DUT.
The term port capacity term defines the maximum speed capability for
[O] The term port capacity term
[P] The term "port capacity"
[R] readability
the given port; for example 1GE, 10GE, 40GE, 100GE etc.
The frequency ("clock rate") of the transmit clock in any two
connected interfaces will never be precisely the same, therefore a
tolerance is needed, this will be expressed by Parts Per Million
[O] same, therefore a tolerance is needed, this will
[P] same; therefore, a tolerance is needed. This will
[R] grammar
5.2 Discussion
For a transmit clock source, most Ethernet switches use "clock
modules" (also called "oscillator modules") that are sealed,
internally temperature-compensated, and very accurate. The output
frequency of these modules is not adjustable because it is not
necessary. Many test sets, however, offer a software-controlled
adjustment of the transmit clock rate, which should be used to
compensate the test equipment to not send more than line rate of the
DUT.
[O] which should be used to
compensate the test equipment to not send more than line rate of the
DUT.
[R] please reword for clarity.
To allow for the minor variations typically found in the clock rate
of commercially-available clock modules and other crystal-based
Test set equipment manufacturers are well-aware of the standards, and
allows a software-controlled +/- 100 PPM "offset" (clock-rate
[O] allows
[P] allow
[R] grammar
adjustment) to compensate for normal variations in the clock speed of
"devices under test". This offset adjustment allows engineers to
determine the approximate speed the connected device is operating,
and verify that it is within parameters allowed by standards.
5.3 Measurement Units
In a production network, it is very unlikely to see precise line rate
over a very brief period. There is no observable difference between
dropping packets at 99% of line rate and 100% of line rate. -Line
rate CAN measured at 100% of line rate with a -100PPM adjustment. -
Line rate SHOULD be measured at 99,98% with 0 PPM adjustment.-The PPM
[O] .-The
[P] . The
[R] Typo Or was this intended to be a list?
adjustment SHOULD only be used for a line rate type of measurement
6 Buffering
6.1 Buffer
6.1.1 Definition
Buffer Size: the term buffer size, represents the total amount of
[O] the term buffer size, represents
[P] the term buffer size represents
[R] grammar
frame buffering memory available on a DUT. This size is expressed in
Byte; KB (kilobytes), MB (megabytes) or GB (gigabyte). When the
buffer size is expressed it SHOULD be defined by a size metric
defined above. When the buffer size is expressed, an indication of
[O] defined above
[P] listed above [OR] stated above
[R] the sizes are not "defined" above
the frame MTU used for that measurement is also necessary as well as
the cos or dscp value set; as often times the buffers are carved by
quality of service implementation. (please refer to the buffer
[O] . (please
[P] . (Please
[R] grammar
efficiency section for further details).
Example: Buffer Size of DUT when sending 1518 bytes frames is 18 Mb.
Port Buffer Size: the port buffer size is the amount of buffer a
single ingress port, egress port or combination of ingress and egress
buffering location for a single port. The reason of mentioning the
[O] reason of mentioning
[P] reason for mentioning
[R] grammar
three locations for the port buffer is, that the DUT buffering scheme
[O] is, that
[P] is because
[R] grammar
can be unknown or untested, and therefore the indication of where the
buffer is located helps understand the buffer architecture and
[O] and therefore the indication of where the
buffer is located helps understand the buffer architecture
[P] and so knowing the buffer location helps clarify the buffer architecture
[R] readability
therefore the total buffer size. The Port Buffer Size is an
informational value that MAY be provided from the DUT vendor. It is
not a value that is tested by benchmarking. Benchmarking will be done
using the Maximum Port Buffer Size or Maximum Buffer Size
methodology.
Maximum Port Buffer Size: this is in most cases the same as the Port
[O] this is in most cases the same
[P] in most cases, this is the same
[R] readability
Buffer Size. In certain switch architecture called SoC (switch on
chip), there is a concept of port buffer and shared buffer pool
[O] there is a concept
[R] ? in certain switch architecture there is a concept? Or is there an
actual port buffer?
available for all ports. Maximum Port Buffer, defines the scenario of
a SoC buffer, where this amount in B (byte), KB (kilobyte), MB
(megabyte) or GB (gigabyte) would represent the sum of the port
buffer along with the maximum value of shared buffer this given port
can take. The Maximum Port Buffer Size needs to be expressed along
[O] Maximum Port Buffer, defines the scenario of
a SoC buffer, where this amount in B (byte), KB (kilobyte), MB
(megabyte) or GB (gigabyte) would represent the sum of the port
buffer along with the maximum value of shared buffer this given port
can take.
[P] Maximum Port Buffer, in terms of an SoC buffer, represents the sum of
the port buffer and the maximum value of shared buffer allowed for this
port, defined in terms of B (byte), KB (kilobyte), MB (megabyte), or GB
(gigabyte).
[R] readability - I think. I has a hard time parsing the previous sentence.
with the frame MTU used for the measurement and the cos or dscp bit
value set for the test.
Example: a DUT has been measured to have 3KB of port buffer for 1518
frame size packets and a total of 4.7 MB of maximum port buffer for
1518 frame size packets and a cos of 0.
Maximum DUT Buffer Size: this is the total size of Buffer a DUT can
be measured to have. It is most likely different than the Maximum
[O] It is most likely different than
[P] either: It is, most likely, different to [OR] It is probably different
to
[R] grammar/readability
Port Buffer Size. It CAN also be different from the sum of Maximum
Port Buffer Size. The Maximum Buffer Size needs to be expressed along
with the frame MTU used for the measurement and along with the cos or
dscp value set during the test.
Example: a DUT has been measured to have 3KB of port buffer for 1518
frame size packets and a total of 4.7 MB of maximum port buffer for
1518 frame size packets. The DUT has a Maximum Buffer Size of 18 MB
at 1500 bytes and a cos of 0.
Burst: The burst is a fixed number of packets sent over a percentage
of linerate of a defined port speed. The amount of frames sent are
evenly distributed across the interval T. A constant C, can be
[O] interval T. A constant C,
[P] interval, T. A constant, C,
[R] clarity
defined to provide the average time between two consecutive packets
evenly spaced.
6.1.2 Discussion
When measuring buffering on a DUT, it is important to understand what
the behavior is for each port, and also for all ports as this will
provide an evidence of the total amount of buffering available on the
switch. The terms of buffer efficiency here helps one understand what
[O] When measuring buffering on a DUT, it is important to understand what
the behavior is for each port, and also for all ports as this will
provide an evidence of the total amount of buffering available on the
switch.
[P] When measuring buffering on a DUT, it is important to understand the
behavior for each and all ports. This provides data for the total amount of
buffering available on the switch.
is the optimum packet size for the buffer to be used, or what is the
real volume of buffer available for a specific packet size. This
[O] what is the optimum packet size for the buffer to be used, or what is
the
real volume of buffer available for a specific packet size.
[P] the optimum packet size for the buffer, or the real volume of the
buffer available for a specific packet size
[R] readability
section does not discuss how to conduct the test methodology, it
rather explains the buffer definitions and what metrics should be
[O] , it rather explains
[P] ; instead, it explains
provided for a comprehensive data center device buffering
benchmarking.
6.1.3 Measurement Units
When Buffer is measured:-the buffer size MUST be measured-the port
buffer size MAY be provided for each port-the maximum port buffer
size MUST be measured-the maximum DUT buffer size MUST be measured-
the intensity of microburst MAY be mentioned when a microburst test
is performed-the cos or dscp value set during the test SHOULD be
provided
[O] entire paragraph above
[R] this is very difficult to read with the hyphens. Guessing this was
intended to be a list, but the XML got munged?
6.2 Incast
6.2.1 Definition
The term Incast, very commonly utilized in the data center, refers to
the traffic pattern of many-to-one or many-to-many conversations.
Typically in the data center it would refer to many different ingress
server ports(many), sending traffic to a common uplink (one), or
[O] ports(many)
[P] ports (many)
multiple uplinks (many). This pattern is generalized for any network
as many incoming ports sending traffic to one or few uplinks. It can
also be found in many-to-many traffic patterns.
6.2.2 Discussion
In this scenario, buffers are solicited on the DUT. In a ingress
buffering mechanism, the ingress port buffers would be solicited
along with Virtual Output Queues, when available; whereas in an
egress buffer mechanism, the egress buffer of the one outgoing port
would be used.
In either cases, regardless of where the buffer memory is located on
the switch architecture; the Incast creates buffer utilization.
[O] In either cases, regardless of where the buffer memory is located on
the switch architecture; the Incast creates buffer utilization.
[P] In either case, regardless of where the buffer memory is located in the
switch architecture, the Incast creates the buffer utilization.
[R] grammar
When one or more frames having synchronous arrival times at the DUT
they are considered forming an incast.
[O] incast
[P] Incast
[R] consistency
7 Application Throughput: Data Center Goodput
7.3. Measurement Units
Example: a TCP file transfer over HTTP protocol on a 10Gb/s media.
The file cannot be transferred over Ethernet as a single continuous
stream. It must be broken down into individual frames of 1500 bytes
when the standard MTU [Maximum Transmission Unit] is used. Each
packet requires 20 bytes of IP header information and 20 bytes of TCP
header information, therefore 1460 byte are available per packet for
[O] information, therefore
[P] information; therefore,
[R] grammar
the file transfer. Linux based systems are further limited to 1448
bytes as they also carry a 12 byte timestamp. Finally, the date is
transmitted in this example over Ethernet which adds a 26 byte
overhead per packet.
--
I don't think the execution is relevant when it was obviously a bad idea in
the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair of
pants.
---maf
_______________________________________________
bmwg mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/bmwg