Re: Scope of draft-constantine-bmwg-traffic-management-02

Thanks for introducing this topic on the list, Barry.

There are functions which must perform correctly is all devices
we've benchmarked, but that has not been the emphasis of the benchmarks.

you wrote:

But again, the lack of an objective way to compare these traffic management functions is a gaping hole in the industry

*and* the importance/interest in proper traffic management is at the fore front of many carriers / providers.


Say we have configured traffic shapers to perform a specification,
such as strict policing of <=20Mbps.  Maybe a test of transmission rate
accuracy would be appropriate here (in addition to the tests Barry mentioned **).
For example, a device that operated with 18 such policers with 99.9% accuracy,
but 2 policers with flow rate <95% of the configured value has reached
some form of capacity limit we can report.

Al
(as a participant)

From: [email protected] [mailto:[email protected]] On Behalf Of Barry Constantine
Sent: Saturday, September 21, 2013 4:46 PM
To: [email protected]
Subject: [bmwg] Scope of draft-constantine-bmwg-traffic-management-02

Hi BMWG,

Wanted to bring a topic onto the list from IETF 87 and this had to do with the concern that the traffic management benchmarking work was introducing aspects of conformance testing into the framework.

A summary of the discussion is provided below and we would like to get feedback and drive to closure with the working group.

To Scott Bradner's concern with the work:  "function verification sounds a lot like "conformance" and conformance is clearly out of scope for BMWG".  The specific context Scott referenced was that of a traffic shaper.

Here are some key paragraphs that we added to -02 to clarify the scope:


*** Section 1.2

"   The tests are divided into individual tests and rated capacity tests.

   The individual tests are intended to verify the traffic management

   function according to the specifications.  As an example, suppose a

   traffic shaper is to be tested at a CIR of 20 Mbps.  The intent of

   the individual test is to test one instance of the shaper and it's

   ability to properly shape according to the metrics** defined in

   section 4."

** These metrics include burst size achieved, packet loss / sequence,  jitter



*** Section 3

"   Also, it is not within scope to perform conformance testing. Tests

   defined in this framework benchmark the traffic management functions

   according to the metrics defined in section 4 and do not address any

   conformance to standards related to traffic management.  Traffic

   management specifications largely do not exist and this is a prime

   driver for this framework; to provide an objective means to compare

   vendor traffic management functions."


From the co-author's perspective, what we see in industry today is that vendors advertise functions such as policers, shapers, etc and there is no means to compare performance between them.



We definitely want to avoid this work being any form of conformance test and realize we must carefully specify the benchmark measurements so as not to cross that line.



But again, the lack of an objective way to compare these traffic management functions is a gaping hole in the industry *and* the importance/interest in proper traffic management is at the fore front of many carriers / providers.



So we're looking to the folks in BMWG to chime in, help sort out the scope, provide expertise in meaningful benchmark metrics, etc.

Thank you,
Barry Constantine

_______________________________________________
bmwg mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/bmwg