Re: Scope of draft-constantine-bmwg-traffic-management-02

Ideally, "now" we hear replies from people who develop devices
that would be measured according to the general procedure below.

Al
bmwg co-chair

> -----Original Message-----
> From: Barry Constantine [mailto:[email protected]]
> Sent: Tuesday, September 24, 2013 7:08 AM
> To: MORTON, ALFRED C (AL); Bradner, Scott; [email protected]
> Subject: RE: [bmwg] Scope of draft-constantine-bmwg-traffic-management-02
> 
> Hi Folks,
> 
> I think it is going to come down to a metric for measuring the
> transmission rate of the shaper (in addition to the other metrics already
> discussed including loss, jitter, etc.)
> 
> Prior to Berlin, our lab tests used this basic premise:
> 
> 1. Shapers "smooth" traffic to conform to the policer CIR and burst (Bc
> and Be) parameters; this way bursty traffic is shaped and not policed by
> the policer
> 2. So the shaper can only burst so many bytes in a time interval (Tc)
> 3. The draft would specify guidelines for specifying this time interval;
> Tc = Bc / CIR is an example and the one that we used in preliminary
> testing
> 4. The shaper tests would measure the how many bytes are transmitted at
> the egress per this time interval (along with the other metrics; loss,
> jitter, etc that are already specified in the draft)
> 
> This technique seems to provide an objective way to compare performance
> amongst various equipment shapers.
> 
> Thoughts?
> 
> Thank you,
> Barry Constantine
> 
> JDSU Communications Test
> Principal Member Technical Staff
> 301-325-7069
> 
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf Of
> MORTON, ALFRED C (AL)
> Sent: Monday, September 23, 2013 3:27 PM
> To: Bradner, Scott; [email protected]
> Subject: Re: [bmwg] Scope of draft-constantine-bmwg-traffic-management-02
> 
> Hi Tim,
> 
> Tim Copley wrote:
> 	<some device developers>
> 	have decided ... that
> 	they can reduce memory and save money.  Or stick with
> 	an older algorithm that does Time slices in 125ms.  I had
> 	hoped that this work will enable me to run performance
> 	tests against these vendors and call out the performance of
> 	their switch / router / whatever, and say your shapers
> 	don't perform as well as the other vendor.
> 	I'm not sure how that's a conformance test.
> 
> An algorithm that works on 125ms time slices is necessarily different from
> another that uses 5ms, or 1 second slices, and it may be enough to know
> the embedded time slice in use, or the range that can be configured - thus
> we may not need a test for this difference at all (if I understand your
> example above).
> 
> OTOH, it's more difficult to predict how memory limits will manifest in
> dynamic algorithm performance, and it certainly implies there will be a
> maximum number of simultaneous algorithms that can operate (capacity).
> I think that's a regime of testing that may very well fit in our current
> charter. The input load characteristics are a factor in this evaluation,
> and also the timing of measurement samples at the DUT output.
> 
> IMO, we should attempt to develop benchmarks to usefully distinguish
> performance/capacity of different devices performing traffic management
> alg., and see if they are sufficient to meet your needs and others
> interested in this topic.
> 
> Does that sound like a good (enough) starting place?
> Al
> (participant)
> 
> > -----Original Message-----
> > From: [email protected] [mailto:[email protected]] On Behalf
> > Of Bradner, Scott
> > Sent: Monday, September 23, 2013 1:47 PM
> > To: [email protected]
> > Subject: Re: [bmwg] Scope of
> > draft-constantine-bmwg-traffic-management-02
> >
> > On Sep 23, 2013, at 11:47 AM, tim copley <[email protected]> wrote:
> >
> > > So throwing in my 2 cents and the reason why I argued to keep
> > > scheduling and queuing as a performance test rather than a
> > > conformance test.
> >
> > we may be having a terminology conflict
> >
> > it is not a question of calling something a conformance vs a
> > performance test
> >
> > the actual test is what determines if its a conformance or performance
> > test
> >
> > an example:
> >
> > 	if you have  a test that checks to see what the output rate of a
> > filter is under varying input rates - that is a performance test
> >
> > 	if you have a test that checks to see that the pattern of output
> > packets is round robin from multiple inputs - that is a conformance
> > test
> >
> > the fact that something is a conformance test does not make it bad but
> > it makes it outside the current scope of BMWG
> >
> > but as I said earlier, I think this is something that the chairs & the
> > ADs need to look at
> >
> > Scott
> >
> >
> > On Sep 23, 2013, at 11:47 AM, tim copley <[email protected]> wrote:
> >
> > > So throwing in my 2 cents and the reason why I argued to keep
> > > scheduling and queuing as a performance test rather than a
> > > conformance test.  I think it goes a bit beyond just configuration
> > > verification. I think the intent is to verify that the vendor has
> > > correctly implemented the different types of scheduling or queue
> > > modifications properly.
> > > Which would in fact be a performance test.  Verifying that the
> > > performance of vendor A's implementation of queue management or
> > > Vendor A's implementation of Hardware rather than Vendor's Bs queue
> > > management or hardware choice.
> > >
> > > I've had several vendors who shall remain nameless that have decided
> > > for some odd reason, usually financial, that they can reduce memory
> > > and save money.  Or stick with an older algorithm that does Time
> > > slices in 125ms.  I had hoped that this work will enable me to run
> > > performance tests against these vendors and call out the performance
> > > of their switch / router / whatever, and say your shapers don't
> > > perform as well as the other vendor.
> > >
> > > I'm not sure how that's a conformance test.
> > >
> > > TimC
> > >
> > >
> > > On Sun, Sep 22, 2013 at 6:35 AM, MORTON, ALFRED C (AL)
> > <[email protected]> wrote:
> > > Thanks for introducing this topic on the list, Barry.
> > >
> > >
> > >
> > > There are functions which must perform correctly is all devices
> > >
> > > we've benchmarked, but that has not been the emphasis of the
> benchmarks.
> > >
> > >
> > >
> > > you wrote:
> > >
> > > But again, the lack of an objective way to compare these traffic
> > management functions is a gaping hole in the industry
> > >
> > > *and* the importance/interest in proper traffic management is at the
> > fore front of many carriers / providers.
> > >
> > >
> > >
> > >
> > >
> > > Say we have configured traffic shapers to perform a specification,
> > >
> > > such as strict policing of <=20Mbps.  Maybe a test of transmission
> > > rate
> > >
> > > accuracy would be appropriate here (in addition to the tests Barry
> > mentioned **).
> > >
> > > For example, a device that operated with 18 such policers with 99.9%
> > accuracy,
> > >
> > > but 2 policers with flow rate <95% of the configured value has
> > > reached
> > >
> > > some form of capacity limit we can report.
> > >
> > >
> > >
> > > Al
> > >
> > > (as a participant)
> > >
> > >
> > >
> > > From: [email protected] [mailto:[email protected]] On Behalf
> > > Of
> > Barry Constantine
> > > Sent: Saturday, September 21, 2013 4:46 PM
> > > To: [email protected]
> > > Subject: [bmwg] Scope of
> > > draft-constantine-bmwg-traffic-management-02
> > >
> > >
> > >
> > > Hi BMWG,
> > >
> > >
> > >
> > > Wanted to bring a topic onto the list from IETF 87 and this had to
> > > do
> > with the concern that the traffic management benchmarking work was
> > introducing aspects of conformance testing into the framework.
> > >
> > >
> > >
> > > A summary of the discussion is provided below and we would like to
> > > get
> > feedback and drive to closure with the working group.
> > >
> > >
> > >
> > > To Scott Bradner's concern with the work:  "function verification
> > > sounds
> > a lot like "conformance" and conformance is clearly out of scope for
> > BMWG".  The specific context Scott referenced was that of a traffic
> > shaper.
> > >
> > >
> > >
> > > Here are some key paragraphs that we added to -02 to clarify the
> scope:
> > >
> > >
> > >
> > > *** Section 1.2
> > >
> > > "   The tests are divided into individual tests and rated capacity
> > tests.
> > >
> > >    The individual tests are intended to verify the traffic
> > > management
> > >
> > >    function according to the specifications.  As an example, suppose
> > > a
> > >
> > >    traffic shaper is to be tested at a CIR of 20 Mbps.  The intent
> > > of
> > >
> > >    the individual test is to test one instance of the shaper and
> > > it's
> > >
> > >    ability to properly shape according to the metrics** defined in
> > >
> > >    section 4."
> > >
> > > ** These metrics include burst size achieved, packet loss /
> > > sequence,
> > jitter
> > >
> > >
> > >
> > > *** Section 3
> > >
> > > "   Also, it is not within scope to perform conformance testing. Tests
> > >
> > >    defined in this framework benchmark the traffic management
> > > functions
> > >
> > >    according to the metrics defined in section 4 and do not address
> > > any
> > >
> > >    conformance to standards related to traffic management.  Traffic
> > >
> > >    management specifications largely do not exist and this is a
> > > prime
> > >
> > >    driver for this framework; to provide an objective means to
> > > compare
> > >
> > >    vendor traffic management functions."
> > >
> > >
> > >
> > > From the co-author's perspective, what we see in industry today is
> > > that
> > vendors advertise functions such as policers, shapers, etc and there
> > is no means to compare performance between them.
> > >
> > >
> > >
> > > We definitely want to avoid this work being any form of conformance
> > > test
> > and realize we must carefully specify the benchmark measurements so as
> > not to cross that line.
> > >
> > >
> > >
> > > But again, the lack of an objective way to compare these traffic
> > management functions is a gaping hole in the industry *and* the
> > importance/interest in proper traffic management is at the fore front
> > of many carriers / providers.
> > >
> > >
> > >
> > > So we're looking to the folks in BMWG to chime in, help sort out the
> > scope, provide expertise in meaningful benchmark metrics, etc.
> > >
> > >
> > >
> > > Thank you,
> > >
> > > Barry Constantine
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > bmwg mailing list
> > > [email protected]
> > > https://www.ietf.org/mailman/listinfo/bmwg
> > >
> > >
> > >
> > >
> > > --
> > > "Speed is fine, but accuracy is final." Wyatt Earp, 1888
> > > _______________________________________________
> > > bmwg mailing list
> > > [email protected]
> > > https://www.ietf.org/mailman/listinfo/bmwg
> >
> > Scott O Bradner
> > Senior Technology Consultant
> >
> > Harvard University Information Technology Innovation & Architecture
> > (P) +1 (617) 495 3864
> > 1033 Mass Ave, room 462
> > Cambridge, MA 02138
> >
> >
> >
> > _______________________________________________
> > bmwg mailing list
> > [email protected]
> > https://www.ietf.org/mailman/listinfo/bmwg
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg