Re: Scope of draft-constantine-bmwg-traffic-management-02

Hi Tim,

Tim Copley wrote:
	<some device developers> 
	have decided ... that
	they can reduce memory and save money.  Or stick with
	an older algorithm that does Time slices in 125ms.  I had
	hoped that this work will enable me to run performance
	tests against these vendors and call out the performance of
	their switch / router / whatever, and say your shapers
	don't perform as well as the other vendor.
	I'm not sure how that's a conformance test.

An algorithm that works on 125ms time slices is necessarily different
from another that uses 5ms, or 1 second slices, and it may be enough
to know the embedded time slice in use, or the range that can be 
configured - thus we may not need a test for this difference at all
(if I understand your example above).

OTOH, it's more difficult to predict how memory limits will manifest
in dynamic algorithm performance, and it certainly implies there will be a 
maximum number of simultaneous algorithms that can operate (capacity).
I think that's a regime of testing that may very well fit in our current
charter. The input load characteristics are a factor in this evaluation,
and also the timing of measurement samples at the DUT output.

IMO, we should attempt to develop benchmarks to usefully distinguish
performance/capacity of different devices performing traffic management alg.,
and see if they are sufficient to meet your needs and others interested
in this topic. 

Does that sound like a good (enough) starting place?
Al
(participant)

> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf Of
> Bradner, Scott
> Sent: Monday, September 23, 2013 1:47 PM
> To: [email protected]
> Subject: Re: [bmwg] Scope of draft-constantine-bmwg-traffic-management-02
> 
> On Sep 23, 2013, at 11:47 AM, tim copley <[email protected]> wrote:
> 
> > So throwing in my 2 cents and the reason why I argued
> > to keep scheduling and queuing as a performance test
> > rather than a conformance test.
> 
> we may be having a terminology conflict
> 
> it is not a question of calling something a conformance vs a performance
> test
> 
> the actual test is what determines if its a conformance or performance
> test
> 
> an example:
> 
> 	if you have  a test that checks to see what the output rate of a
> filter is
> under varying input rates - that is a performance test
> 
> 	if you have a test that checks to see that the pattern of output
> packets is round robin
> from multiple inputs - that is a conformance test
> 
> the fact that something is a conformance test does not make it bad but it
> makes it outside the current scope of BMWG
> 
> but as I said earlier, I think this is something that the chairs & the ADs
> need to look at
> 
> Scott
> 
> 
> On Sep 23, 2013, at 11:47 AM, tim copley <[email protected]> wrote:
> 
> > So throwing in my 2 cents and the reason why I argued
> > to keep scheduling and queuing as a performance test
> > rather than a conformance test.  I think it goes a bit
> > beyond just configuration verification. I think the intent
> > is to verify that the vendor has correctly implemented the
> > different types of scheduling or queue modifications properly.
> > Which would in fact be a performance test.  Verifying that
> > the performance of vendor A's implementation of queue
> > management or Vendor A's implementation of Hardware
> > rather than Vendor's Bs queue management or hardware
> > choice.
> >
> > I've had several vendors who shall remain nameless that
> > have decided for some odd reason, usually financial, that
> > they can reduce memory and save money.  Or stick with
> > an older algorithm that does Time slices in 125ms.  I had
> > hoped that this work will enable me to run performance
> > tests against these vendors and call out the performance of
> > their switch / router / whatever, and say your shapers
> > don't perform as well as the other vendor.
> >
> > I'm not sure how that's a conformance test.
> >
> > TimC
> >
> >
> > On Sun, Sep 22, 2013 at 6:35 AM, MORTON, ALFRED C (AL)
> <[email protected]> wrote:
> > Thanks for introducing this topic on the list, Barry.
> >
> >
> >
> > There are functions which must perform correctly is all devices
> >
> > we've benchmarked, but that has not been the emphasis of the benchmarks.
> >
> >
> >
> > you wrote:
> >
> > But again, the lack of an objective way to compare these traffic
> management functions is a gaping hole in the industry
> >
> > *and* the importance/interest in proper traffic management is at the
> fore front of many carriers / providers.
> >
> >
> >
> >
> >
> > Say we have configured traffic shapers to perform a specification,
> >
> > such as strict policing of <=20Mbps.  Maybe a test of transmission rate
> >
> > accuracy would be appropriate here (in addition to the tests Barry
> mentioned **).
> >
> > For example, a device that operated with 18 such policers with 99.9%
> accuracy,
> >
> > but 2 policers with flow rate <95% of the configured value has reached
> >
> > some form of capacity limit we can report.
> >
> >
> >
> > Al
> >
> > (as a participant)
> >
> >
> >
> > From: [email protected] [mailto:[email protected]] On Behalf Of
> Barry Constantine
> > Sent: Saturday, September 21, 2013 4:46 PM
> > To: [email protected]
> > Subject: [bmwg] Scope of draft-constantine-bmwg-traffic-management-02
> >
> >
> >
> > Hi BMWG,
> >
> >
> >
> > Wanted to bring a topic onto the list from IETF 87 and this had to do
> with the concern that the traffic management benchmarking work was
> introducing aspects of conformance testing into the framework.
> >
> >
> >
> > A summary of the discussion is provided below and we would like to get
> feedback and drive to closure with the working group.
> >
> >
> >
> > To Scott Bradner's concern with the work:  "function verification sounds
> a lot like "conformance" and conformance is clearly out of scope for
> BMWG".  The specific context Scott referenced was that of a traffic
> shaper.
> >
> >
> >
> > Here are some key paragraphs that we added to -02 to clarify the scope:
> >
> >
> >
> > *** Section 1.2
> >
> > "   The tests are divided into individual tests and rated capacity
> tests.
> >
> >    The individual tests are intended to verify the traffic management
> >
> >    function according to the specifications.  As an example, suppose a
> >
> >    traffic shaper is to be tested at a CIR of 20 Mbps.  The intent of
> >
> >    the individual test is to test one instance of the shaper and it's
> >
> >    ability to properly shape according to the metrics** defined in
> >
> >    section 4."
> >
> > ** These metrics include burst size achieved, packet loss / sequence,
> jitter
> >
> >
> >
> > *** Section 3
> >
> > "   Also, it is not within scope to perform conformance testing. Tests
> >
> >    defined in this framework benchmark the traffic management functions
> >
> >    according to the metrics defined in section 4 and do not address any
> >
> >    conformance to standards related to traffic management.  Traffic
> >
> >    management specifications largely do not exist and this is a prime
> >
> >    driver for this framework; to provide an objective means to compare
> >
> >    vendor traffic management functions."
> >
> >
> >
> > From the co-author's perspective, what we see in industry today is that
> vendors advertise functions such as policers, shapers, etc and there is no
> means to compare performance between them.
> >
> >
> >
> > We definitely want to avoid this work being any form of conformance test
> and realize we must carefully specify the benchmark measurements so as not
> to cross that line.
> >
> >
> >
> > But again, the lack of an objective way to compare these traffic
> management functions is a gaping hole in the industry *and* the
> importance/interest in proper traffic management is at the fore front of
> many carriers / providers.
> >
> >
> >
> > So we're looking to the folks in BMWG to chime in, help sort out the
> scope, provide expertise in meaningful benchmark metrics, etc.
> >
> >
> >
> > Thank you,
> >
> > Barry Constantine
> >
> >
> >
> >
> > _______________________________________________
> > bmwg mailing list
> > [email protected]
> > https://www.ietf.org/mailman/listinfo/bmwg
> >
> >
> >
> >
> > --
> > "Speed is fine, but accuracy is final." Wyatt Earp, 1888
> > _______________________________________________
> > bmwg mailing list
> > [email protected]
> > https://www.ietf.org/mailman/listinfo/bmwg
> 
> Scott O Bradner
> Senior Technology Consultant
> 
> Harvard University Information Technology
> Innovation & Architecture
> (P) +1 (617) 495 3864
> 1033 Mass Ave, room 462
> Cambridge, MA 02138
> 
> 
> 
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg