Re: Scope of draft-constantine-bmwg-traffic-management-02

On Sep 23, 2013, at 11:47 AM, tim copley <[email protected]> wrote:

> So throwing in my 2 cents and the reason why I argued 
> to keep scheduling and queuing as a performance test
> rather than a conformance test.  

we may be having a terminology conflict

it is not a question of calling something a conformance vs a performance test

the actual test is what determines if its a conformance or performance test

an example: 

	if you have  a test that checks to see what the output rate of a filter is
under varying input rates - that is a performance test

	if you have a test that checks to see that the pattern of output packets is round robin 
from multiple inputs - that is a conformance test

the fact that something is a conformance test does not make it bad but it makes it outside the current scope of BMWG

but as I said earlier, I think this is something that the chairs & the ADs need to look at

Scott


On Sep 23, 2013, at 11:47 AM, tim copley <[email protected]> wrote:

> So throwing in my 2 cents and the reason why I argued 
> to keep scheduling and queuing as a performance test
> rather than a conformance test.  I think it goes a bit 
> beyond just configuration verification. I think the intent 
> is to verify that the vendor has correctly implemented the 
> different types of scheduling or queue modifications properly.  
> Which would in fact be a performance test.  Verifying that
> the performance of vendor A's implementation of queue
> management or Vendor A's implementation of Hardware
> rather than Vendor's Bs queue management or hardware
> choice.
> 
> I've had several vendors who shall remain nameless that
> have decided for some odd reason, usually financial, that
> they can reduce memory and save money.  Or stick with
> an older algorithm that does Time slices in 125ms.  I had
> hoped that this work will enable me to run performance
> tests against these vendors and call out the performance of
> their switch / router / whatever, and say your shapers
> don't perform as well as the other vendor.
> 
> I'm not sure how that's a conformance test.
> 
> TimC
> 
> 
> On Sun, Sep 22, 2013 at 6:35 AM, MORTON, ALFRED C (AL) <[email protected]> wrote:
> Thanks for introducing this topic on the list, Barry.
> 
>  
> 
> There are functions which must perform correctly is all devices
> 
> we've benchmarked, but that has not been the emphasis of the benchmarks.
> 
>  
> 
> you wrote:
> 
> But again, the lack of an objective way to compare these traffic management functions is a gaping hole in the industry
> 
> *and* the importance/interest in proper traffic management is at the fore front of many carriers / providers.
> 
>  
> 
>  
> 
> Say we have configured traffic shapers to perform a specification,
> 
> such as strict policing of <=20Mbps.  Maybe a test of transmission rate
> 
> accuracy would be appropriate here (in addition to the tests Barry mentioned **).
> 
> For example, a device that operated with 18 such policers with 99.9% accuracy,
> 
> but 2 policers with flow rate <95% of the configured value has reached
> 
> some form of capacity limit we can report.
> 
>  
> 
> Al
> 
> (as a participant)
> 
>  
> 
> From: [email protected] [mailto:[email protected]] On Behalf Of Barry Constantine
> Sent: Saturday, September 21, 2013 4:46 PM
> To: [email protected]
> Subject: [bmwg] Scope of draft-constantine-bmwg-traffic-management-02
> 
>  
> 
> Hi BMWG,
> 
>  
> 
> Wanted to bring a topic onto the list from IETF 87 and this had to do with the concern that the traffic management benchmarking work was introducing aspects of conformance testing into the framework.
> 
>  
> 
> A summary of the discussion is provided below and we would like to get feedback and drive to closure with the working group.
> 
>  
> 
> To Scott Bradner’s concern with the work:  “function verification sounds a lot like "conformance" and conformance is clearly out of scope for BMWG”.  The specific context Scott referenced was that of a traffic shaper.
> 
>  
> 
> Here are some key paragraphs that we added to -02 to clarify the scope:
> 
>  
> 
> *** Section 1.2
> 
> "   The tests are divided into individual tests and rated capacity tests.
> 
>    The individual tests are intended to verify the traffic management
> 
>    function according to the specifications.  As an example, suppose a
> 
>    traffic shaper is to be tested at a CIR of 20 Mbps.  The intent of
> 
>    the individual test is to test one instance of the shaper and it's
> 
>    ability to properly shape according to the metrics** defined in
> 
>    section 4."
> 
> ** These metrics include burst size achieved, packet loss / sequence,  jitter
> 
>  
> 
> *** Section 3
> 
> "   Also, it is not within scope to perform conformance testing. Tests
> 
>    defined in this framework benchmark the traffic management functions
> 
>    according to the metrics defined in section 4 and do not address any
> 
>    conformance to standards related to traffic management.  Traffic
> 
>    management specifications largely do not exist and this is a prime
> 
>    driver for this framework; to provide an objective means to compare
> 
>    vendor traffic management functions."
> 
>  
> 
> From the co-author’s perspective, what we see in industry today is that vendors advertise functions such as policers, shapers, etc and there is no means to compare performance between them.
> 
>  
> 
> We definitely want to avoid this work being any form of conformance test and realize we must carefully specify the benchmark measurements so as not to cross that line. 
> 
>  
> 
> But again, the lack of an objective way to compare these traffic management functions is a gaping hole in the industry *and* the importance/interest in proper traffic management is at the fore front of many carriers / providers.
> 
>  
> 
> So we’re looking to the folks in BMWG to chime in, help sort out the scope, provide expertise in meaningful benchmark metrics, etc.
> 
>  
> 
> Thank you,
> 
> Barry Constantine
> 
>  
> 
> 
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg
> 
> 
> 
> 
> -- 
> "Speed is fine, but accuracy is final." Wyatt Earp, 1888
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg

Scott O Bradner
Senior Technology Consultant

Harvard University Information Technology
Innovation & Architecture
(P) +1 (617) 495 3864
1033 Mass Ave, room 462
Cambridge, MA 02138