Re: Scope of draft-constantine-bmwg-traffic-management-02

So throwing in my 2 cents and the reason why I argued
to keep scheduling and queuing as a performance test
rather than a conformance test.  I think it goes a bit
beyond just configuration verification. I think the intent
is to verify that the vendor has correctly implemented the
different types of scheduling or queue modifications properly.
Which would in fact be a performance test.  Verifying that
the performance of vendor A's implementation of queue
management or Vendor A's implementation of Hardware
rather than Vendor's Bs queue management or hardware
choice.

I've had several vendors who shall remain nameless that
have decided for some odd reason, usually financial, that
they can reduce memory and save money.  Or stick with
an older algorithm that does Time slices in 125ms.  I had
hoped that this work will enable me to run performance
tests against these vendors and call out the performance of
their switch / router / whatever, and say your shapers
don't perform as well as the other vendor.

I'm not sure how that's a conformance test.

TimC


On Sun, Sep 22, 2013 at 6:35 AM, MORTON, ALFRED C (AL) <[email protected]>wrote:

> Thanks for introducing this topic on the list, Barry.****
>
> ** **
>
> There are functions which must perform correctly is all devices****
>
> we've benchmarked, but that has not been the emphasis of the benchmarks.**
> **
>
> ** **
>
> you wrote:****
>
> But again, the lack of an objective way to compare these traffic
> management functions is a gaping hole in the industry ****
>
> **and** the importance/interest in proper traffic management is at the
> fore front of many carriers / providers.****
>
> ** **
>
> ** **
>
> Say we have configured traffic shapers to perform a specification,****
>
> such as strict policing of <=20Mbps.  Maybe a test of transmission rate***
> *
>
> accuracy would be appropriate here (in addition to the tests Barry
> mentioned **).****
>
> For example, a device that operated with 18 such policers with 99.9%
> accuracy,****
>
> but 2 policers with flow rate <95% of the configured value has reached****
>
> some form of capacity limit we can report.****
>
> ** **
>
> Al****
>
> (as a participant)****
>
> ** **
>
> *From:* [email protected] [mailto:[email protected]] *On Behalf
> Of *Barry Constantine
> *Sent:* Saturday, September 21, 2013 4:46 PM
> *To:* [email protected]
> *Subject:* [bmwg] Scope of draft-constantine-bmwg-traffic-management-02***
> *
>
> ** **
>
> Hi BMWG,****
>
> ** **
>
> Wanted to bring a topic onto the list from IETF 87 and this had to do with
> the concern that the traffic management benchmarking work was introducing
> aspects of conformance testing into the framework.****
>
> ** **
>
> A summary of the discussion is provided below and we would like to get
> feedback and drive to closure with the working group.****
>
> ** **
>
> To Scott Bradner’s concern with the work:  “function verification sounds a
> lot like "conformance" and conformance is clearly out of scope for BMWG”.
> The specific context Scott referenced was that of a traffic shaper.****
>
> ** **
>
> Here are some key paragraphs that we added to -02 to clarify the scope:***
> *
>
> ** **
>
> *** Section 1.2****
>
> "   The tests are divided into individual tests and rated capacity tests.*
> ***
>
>    The individual tests are intended to verify the traffic management ****
>
>    function according to the specifications.  As an example, suppose a ***
> *
>
>    traffic shaper is to be tested at a CIR of 20 Mbps.  The intent of ****
>
>    the individual test is to test one instance of the shaper and it's ****
>
>    ability to properly shape according to the metrics** defined in ****
>
>    section 4."****
>
> ** These metrics include burst size achieved, packet loss / sequence,
> jitter ****
>
> ** **
>
> *** Section 3****
>
> "   Also, it is not within scope to perform conformance testing. Tests****
>
>    defined in this framework benchmark the traffic management functions***
> *
>
>    according to the metrics defined in section 4 and do not address any***
> *
>
>    conformance to standards related to traffic management.  Traffic ****
>
>    management specifications largely do not exist and this is a prime****
>
>    driver for this framework; to provide an objective means to compare****
>
>    vendor traffic management functions."****
>
> ** **
>
> From the co-author’s perspective, what we see in industry today is that
> vendors advertise functions such as policers, shapers, etc and there is no
> means to compare performance between them.****
>
> ** **
>
> We definitely want to avoid this work being any form of conformance test
> and realize we must carefully specify the benchmark measurements so as not
> to cross that line.  ****
>
> ** **
>
> But again, the lack of an objective way to compare these traffic
> management functions is a gaping hole in the industry **and** the
> importance/interest in proper traffic management is at the fore front of
> many carriers / providers.****
>
> ** **
>
> So we’re looking to the folks in BMWG to chime in, help sort out the
> scope, provide expertise in meaningful benchmark metrics, etc. ****
>
> ** **
>
> Thank you,****
>
> Barry Constantine****
>
> ** **
>
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg
>
>


-- 
"Speed is fine, but accuracy is final." Wyatt Earp, 1888

_______________________________________________
bmwg mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/bmwg