Comments on draft-constantine-bmwg-traffic-management-01

Barry, Tim, and Ram,

I have some comments to share on the first part of your draft,
below. There's no need to react or edit immediately, we can discuss
further in Berlin...

Al
(as a participant)

-=-=-=-=-=-=-=-=-=-

Section 1.1 Traffic Management Overview

This section contains a brief description of key functions of a
traffic management system. It would help to link these descriptions
with references that provide more details, since I'm sure you have
some at hand.

Also, BMWG has already defined some relevant terms in
http://tools.ietf.org/html/rfc4689
and it would be good to re-use that work where possible.

A specific suggestion:
OLD
  - Active Queue Management (AQM): monitors the status of internal 
    queues and actively drops packets, which causes the sending hosts
	to back-off and in turn can alleviate queue congestion.
NEW
  - Active Queue Management (AQM): monitors the status of internal 
    queues and actively drops (or re-marks) packets, which causes hosts using congestion-aware protocols
                              ^^^^^^^^^^^^^                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	to back-off and in turn can alleviate queue congestion.

and mention active queue management explicitly in Figure 1, like the other functions.
Re-marking or ECN and similar functions might be described separately. 


Section 1.2

...
   For an example: a device is specified to be capable of shaping on all
   of it's egress ports.  The individual verification test would first be
   conducted to benchmark the advertised shaping function to specification.
   Then the capacity test would be executed to test the shaping function
   concurrently on all interfaces and with maximum traffic load.

s/it's/its/

and 
   ...
   concurrently on all interfaces and with N instances of the shaping function and maximum traffic load.
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(is this a test that's planned?)


...
   Also note that the Network Delay Emulator (NDE) should be passive in
   nature such as a fiber spool.  This is recommended to eliminate the
   ...
   performance benchmarked independently.  This requirement will vary 
   from test to test on desired traffic speed and should be calibrated 
   before any test requiring delay, which can add a significant 
   additional amount of testing to each step.

Question: is there no high-end capacity test for the NDE that would 
cover all traffic levels below? (I suppose that the degree of packet inspection
performed by the emulator influences the result, but there must be some 
simplification possible here - fiber spools are not easily configured, etc.)


Section 3
...
   Within this framework, the metrics are defined for each traffic
   management test but do not include pass / fail criterion, which is
   not within the charter of BMWG.  This framework provides the test
   methods and metrics to conduct repeatable testing, which will
   provide the means to compare advertised performance between vendors.

Suggest:
   ...
   provide the means to compare measured performance between DUTs.
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
   As mentioned in section 1.2, this framework describes the individual
   tests and metrics for several management functions.  It is also within
   scope that this framework will benchmark each function in terms of overall
   rated capacity.  This involves concurrent testing of multiple 
   interfaces with the specific traffic management function enabled, and
   doing so to the capacity limit of each interface.

Suggest:
   ... 
   interfaces with the specific traffic management function enabled, 
   up to the capacity limit of each interface.
   ^^

...

   However, it is not within scope of this framework to test multiple 
   traffic management functions concurrently (e.g. police on ingress and
   shape on egress ports).  The multitudes of possible combinations is
   almost unbounded and the ability to identify functional "break points"
   would be most times impossible.

Comment: While avoiding all possible combinations, it seems that usual
profiles of concurrent functions would be useful to benchmark, 
IF the tests produce reliable and repeatable results. The wg participants
could help by suggesting feature combinations.

...
   A goal of this framework is to define specific stateless traffic
   ("packet blasting") tests to conduct the benchmark tests and also to
   derive stateful test patterns (TCP or application layer) that can
   also be used to further benchmark the performance of applicable
   traffic management techniques such as shaping and congestion
   management techniques such as RED/WRED.  In cases where the network
   device is stateful in nature (i.e. firewall, etc.), stateful test
   pattern traffic is important to test along with UDP traffic in 
   specific test scenarios (i.e. TCP application and UDP VoIP, etc.)

Suggest:
   ...
   specific test scenarios (i.e. applications using TCP transport and UDP VoIP, etc.)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Section 4.1 (for stateless/UDP)
...
   Stateless traffic measurements require that sequence number and 
   time-stamp be inserted into the payload for lost packet and delay
   analyses.  This framework does not specify the packet format to carry
   sequence number of timing information.

Comment: This is very implementation-specific. The requirement is to provide
some means to establish correspondence between sent and received packets,
which could be a sequence number alone with time stamps stored elsewhere.

...
   - Burst Size Achieved (BSA): for the traffic policing and network
   queue tests, the tester will be configured to send bursts to test
   either the Committed Burst Size (CBS) or Excess Burst Size (EBS) of
   a policer or the queue / buffer size configured in the DUT.  The
   Burst Size Achieved metric is a measure of the actual burst size
   received at the egress port of the DUT with no lost packets.  As an
   example, the configured CBS of a DUT is 64KB and after the burst test,
   only a 63 KB can be achieved without packet loss.  Then 63KB is the
   BSA.
Comments:
  - no requirement on Latency?
  - re-marking may replace loss

...
   - Lost Packets (LP): For all traffic management tests, the tester will
   transmit the test packets into the DUT ingress port and the number of
   packets received at the egress port will be measured.  The difference
   between packets transmitted into the ingress port and received at the
   egress port is the number of lost packets as measured at the egress
   port.  These packets must have unique identifiers such that only the
   test packets are measured.  RFC 4737 and RFC 2680 specify techniques
   to establish the threshold when a packet is lost versus out-of-order.
Suggest: Last sentence is too convoluted...

   ... RFC 2680 describes the need 
   to establish the threshold to designate a packet as lost, and the 
   threshold used MUST be reported with the results.

...
   - Out of Sequence (OOS): in additions to LF metric, the test
   packets must be monitored for sequence and the out-of-sequence (OOS)
   packets will be counted per RFC 4737 and RFC 2680.
Comment: Sequence is different from Order, and only Order evaluation is
orthogonal to Loss. See RFC 4737 for a simple definition of "Reordered Packet".

...
   - Packet Delay Variation (PDV): the Packet Delay Variation metric is
   the variation between the timestamp of the received egress port
   packets and specified in RFC 1889.

Comment: since you have One-way Delay (PD defined above PDV), why resort to
the RFC 1889 definition?  What does this give you?
Suggest: Specify PDV as defined in http://tools.ietf.org/html/rfc5481#section-4.2 instead,
for the many reasons given in http://tools.ietf.org/html/rfc5481#page-27
(or call this IPDV to avoid confusion).

Section 4.2 (for TCP)
...
   - TCP Test Pattern Execution Time (TTPET): RFC 6349 defined the TCP 
   Transfer Time for bulk transfers, which is simply the measured time 
   to transfer bytes across single or concurrent TCP connections. The
   TCP test patterns used in traffic management tests will be bulk
   transfer and interactive in nature; these test patterns simulate 
   delay-tolerant applications like FTP, streaming video etc. The TTPET
   will be the measure of the time for a single execution of a TCP Test
   Pattern (TTP).  Average, minimum, and maximum times will be measured.

Comment: the use of the term "Test Pattern" in connection with TCP strikes me as odd.
A test pattern usually specifies some aspects of the packet stream that will be 
repeated. Instead, this is a bulk transfer where I assume the eventual
traffic pattern will be determined dynamically by TCP's flow control.
Isn't this simply "TCP Bulk Transfer Time" ?

...
   - TCP Efficiency: after the execution of the TCP Test Pattern, TCP
   Efficiency represents the percentage of Bytes that were not
   retransmitted.

                          Transmitted Bytes - Retransmitted Bytes

      TCP Efficiency % =  ---------------------------------------  X 100

                                   Transmitted Bytes

   Transmitted Bytes are the total number of TCP Bytes to be transmitted
   including the original and the retransmitted Bytes.
Comment: Here, I think you need to carefully define "Retransmitted Bytes"
or the door will be open to some "specsmanship". For example, if bytes are 
only received once at the egress, were they retransmitted?  Perhaps they are
just bytes from Reordered packets...

...
     - Buffer Delay: represents the increase in RTT during a TCP test
     versus the baseline DUT RTT (non congested, inherent latency).  RTT 
	 and the technique to measure RTT (average versus baseline) are defined
	 in RFC 6349.  Referencing RFC 6349, the average RTT is derived from 
	 the total of all measured RTTs during the actual test at every second
	 divided by the test duration in seconds.

                                         Total RTTs during transfer
         Average RTT during transfer = -----------------------------
                                        Transfer duration in seconds


                        Average RTT during Transfer - Baseline RTT
       Buffer Delay % = ------------------------------------------ X 100
                                    Baseline RTT

Comment/Question:  Without checking RFC6349 (because I'm out of time),
are Retransmitted Packets included in the RTT calculations? It seems that
you want to avoid them in order to infer the physical buffer time.

Also, Buffer Delay is a bit odd expressed as a percentage of the minimum/baseline
(to me at least). Maybe use a different name for this.