Re: Scope of draft-constantine-bmwg-traffic-management-02

Hello Barry and Al,

Thank you for the welcome and guidelines.  I'll do what I should have done and that's to read the draft first and then make suggestions.  Glad that there is stateless and stateful ingress specified.

All the best,

Nick
-----------------------------------------------------------------
Nick Lippis
Publisher of The Lippis Report; A Resource for IT & Network Business
Decision Makers.
Get your free subscription at www.lippis.com

Lippis Enterprises, Inc.
Direct: 781-740-7801
Email:  [email protected] 
Follow me on Twitter http://twitter.com/NickLippis
Skype: nicklippis 
Opt in to receive the Lippis Report
www.lippis.com
-----------------------------------------------------------------

This message and any attachments are intended only for the use of the
intended recipient and may contain information that is PRIVILEGED and
CONFIDENTIAL. If you are not the intended recipient, you are hereby notified
that any dissemination of this communication is strictly prohibited. If you
have received this communication in error, please delete all copies of this
message and any attachments and please notify us.  Thank you.




 Please consider power requirements before you buy IT.

On Sep 24, 2013, at 9:23 AM, Barry Constantine <[email protected]> wrote:

> Hi Nick,
>  
> In the draft, we specify stateless and stateful ingress sources in sections 6.3.1.1 and 6.3.1.2.
>  
> The stateless focuses upon the link speed and burst bytes per interval for a shaper tests; currently does not mention IMIX but we could discuss this as well. 
>  
> If you could take some time to review the document and provide comments back to the list, that would be great.
>  
> Thank you,
> Barry Constantine
>  
> JDSU Communications Test
> Principal Member Technical Staff
> 301-325-7069
>  
> From: Nick Lippis [mailto:[email protected]] 
> Sent: Tuesday, September 24, 2013 9:04 AM
> To: Barry Constantine
> Cc: MORTON, ALFRED C (AL); Bradner, Scott; [email protected]
> Subject: Re: [bmwg] Scope of draft-constantine-bmwg-traffic-management-02
>  
> Hello Everyone,
>  
> I'm a long time lister and first time caller.
>  
> I like Barry's approach to measuring burst and shaper performance.  Has the group given consideration to the traffic profile at ingress, that is, to specify an IMIX that could stress the shaper in addition to varying input rates and packet sizes?  I've been using an IxCloud IMIX developed with Ixia for our industry switch test and would offer that for consideration.  Please advise.
>  
> All the best,
>  
> Nick
>  
>  
> On Sep 24, 2013, at 7:08 AM, Barry Constantine <[email protected]> wrote:
> 
> 
> Hi Folks,
> 
> I think it is going to come down to a metric for measuring the transmission rate of the shaper (in addition to the other metrics already discussed including loss, jitter, etc.)
> 
> Prior to Berlin, our lab tests used this basic premise:
> 
> 1. Shapers "smooth" traffic to conform to the policer CIR and burst (Bc and Be) parameters; this way bursty traffic is shaped and not policed by the policer
> 2. So the shaper can only burst so many bytes in a time interval (Tc)
> 3. The draft would specify guidelines for specifying this time interval; Tc = Bc / CIR is an example and the one that we used in preliminary testing 
> 4. The shaper tests would measure the how many bytes are transmitted at the egress per this time interval (along with the other metrics; loss, jitter, etc that are already specified in the draft)
> 
> This technique seems to provide an objective way to compare performance amongst various equipment shapers.
> 
> Thoughts? 
> 
> Thank you,
> Barry Constantine
> 
> JDSU Communications Test
> Principal Member Technical Staff
> 301-325-7069
> 
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf Of MORTON, ALFRED C (AL)
> Sent: Monday, September 23, 2013 3:27 PM
> To: Bradner, Scott; [email protected]
> Subject: Re: [bmwg] Scope of draft-constantine-bmwg-traffic-management-02
> 
> Hi Tim,
> 
> Tim Copley wrote:
>             <some device developers> 
>             have decided ... that
>             they can reduce memory and save money.  Or stick with
>             an older algorithm that does Time slices in 125ms.  I had
>             hoped that this work will enable me to run performance
>             tests against these vendors and call out the performance of
>             their switch / router / whatever, and say your shapers
>             don't perform as well as the other vendor.
>             I'm not sure how that's a conformance test.
> 
> An algorithm that works on 125ms time slices is necessarily different from another that uses 5ms, or 1 second slices, and it may be enough to know the embedded time slice in use, or the range that can be configured - thus we may not need a test for this difference at all (if I understand your example above).
> 
> OTOH, it's more difficult to predict how memory limits will manifest in dynamic algorithm performance, and it certainly implies there will be a maximum number of simultaneous algorithms that can operate (capacity).
> I think that's a regime of testing that may very well fit in our current charter. The input load characteristics are a factor in this evaluation, and also the timing of measurement samples at the DUT output.
> 
> IMO, we should attempt to develop benchmarks to usefully distinguish performance/capacity of different devices performing traffic management alg., and see if they are sufficient to meet your needs and others interested in this topic. 
> 
> Does that sound like a good (enough) starting place?
> Al
> (participant)
> 
> 
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf 
> Of Bradner, Scott
> Sent: Monday, September 23, 2013 1:47 PM
> To: [email protected]
> Subject: Re: [bmwg] Scope of 
> draft-constantine-bmwg-traffic-management-02
> 
> On Sep 23, 2013, at 11:47 AM, tim copley <[email protected]> wrote:
> 
> 
> So throwing in my 2 cents and the reason why I argued to keep 
> scheduling and queuing as a performance test rather than a 
> conformance test.
> 
> we may be having a terminology conflict
> 
> it is not a question of calling something a conformance vs a 
> performance test
> 
> the actual test is what determines if its a conformance or performance 
> test
> 
> an example:
> 
>             if you have  a test that checks to see what the output rate of a 
> filter is under varying input rates - that is a performance test
> 
>             if you have a test that checks to see that the pattern of output 
> packets is round robin from multiple inputs - that is a conformance 
> test
> 
> the fact that something is a conformance test does not make it bad but 
> it makes it outside the current scope of BMWG
> 
> but as I said earlier, I think this is something that the chairs & the 
> ADs need to look at
> 
> Scott
> 
> 
> On Sep 23, 2013, at 11:47 AM, tim copley <[email protected]> wrote:
> 
> 
> So throwing in my 2 cents and the reason why I argued to keep 
> scheduling and queuing as a performance test rather than a 
> conformance test.  I think it goes a bit beyond just configuration 
> verification. I think the intent is to verify that the vendor has 
> correctly implemented the different types of scheduling or queue 
> modifications properly.
> Which would in fact be a performance test.  Verifying that the 
> performance of vendor A's implementation of queue management or 
> Vendor A's implementation of Hardware rather than Vendor's Bs queue 
> management or hardware choice.
> 
> I've had several vendors who shall remain nameless that have decided 
> for some odd reason, usually financial, that they can reduce memory 
> and save money.  Or stick with an older algorithm that does Time 
> slices in 125ms.  I had hoped that this work will enable me to run 
> performance tests against these vendors and call out the performance 
> of their switch / router / whatever, and say your shapers don't 
> perform as well as the other vendor.
> 
> I'm not sure how that's a conformance test.
> 
> TimC
> 
> 
> On Sun, Sep 22, 2013 at 6:35 AM, MORTON, ALFRED C (AL)
> <[email protected]> wrote:
> 
> Thanks for introducing this topic on the list, Barry.
> 
> 
> 
> There are functions which must perform correctly is all devices
> 
> we've benchmarked, but that has not been the emphasis of the benchmarks.
> 
> 
> 
> you wrote:
> 
> But again, the lack of an objective way to compare these traffic
> management functions is a gaping hole in the industry
> 
> 
> *and* the importance/interest in proper traffic management is at the
> fore front of many carriers / providers.
> 
> 
> 
> 
> 
> 
> Say we have configured traffic shapers to perform a specification,
> 
> such as strict policing of <=20Mbps.  Maybe a test of transmission 
> rate
> 
> accuracy would be appropriate here (in addition to the tests Barry
> mentioned **).
> 
> 
> For example, a device that operated with 18 such policers with 99.9%
> accuracy,
> 
> 
> but 2 policers with flow rate <95% of the configured value has 
> reached
> 
> some form of capacity limit we can report.
> 
> 
> 
> Al
> 
> (as a participant)
> 
> 
> 
> From: [email protected] [mailto:[email protected]] On Behalf 
> Of
> Barry Constantine
> 
> Sent: Saturday, September 21, 2013 4:46 PM
> To: [email protected]
> Subject: [bmwg] Scope of 
> draft-constantine-bmwg-traffic-management-02
> 
> 
> 
> Hi BMWG,
> 
> 
> 
> Wanted to bring a topic onto the list from IETF 87 and this had to 
> do
> with the concern that the traffic management benchmarking work was 
> introducing aspects of conformance testing into the framework.
> 
> 
> 
> 
> A summary of the discussion is provided below and we would like to 
> get
> feedback and drive to closure with the working group.
> 
> 
> 
> 
> To Scott Bradner's concern with the work:  "function verification 
> sounds
> a lot like "conformance" and conformance is clearly out of scope for 
> BMWG".  The specific context Scott referenced was that of a traffic 
> shaper.
> 
> 
> 
> 
> Here are some key paragraphs that we added to -02 to clarify the scope:
> 
> 
> 
> *** Section 1.2
> 
> "   The tests are divided into individual tests and rated capacity
> tests.
> 
> 
>   The individual tests are intended to verify the traffic 
> management
> 
>   function according to the specifications.  As an example, suppose 
> a
> 
>   traffic shaper is to be tested at a CIR of 20 Mbps.  The intent 
> of
> 
>   the individual test is to test one instance of the shaper and 
> it's
> 
>   ability to properly shape according to the metrics** defined in
> 
>   section 4."
> 
> ** These metrics include burst size achieved, packet loss / 
> sequence,
> jitter
> 
> 
> 
> 
> *** Section 3
> 
> "   Also, it is not within scope to perform conformance testing. Tests
> 
>   defined in this framework benchmark the traffic management 
> functions
> 
>   according to the metrics defined in section 4 and do not address 
> any
> 
>   conformance to standards related to traffic management.  Traffic
> 
>   management specifications largely do not exist and this is a 
> prime
> 
>   driver for this framework; to provide an objective means to 
> compare
> 
>   vendor traffic management functions."
> 
> 
> 
> From the co-author's perspective, what we see in industry today is 
> that
> vendors advertise functions such as policers, shapers, etc and there 
> is no means to compare performance between them.
> 
> 
> 
> 
> We definitely want to avoid this work being any form of conformance 
> test
> and realize we must carefully specify the benchmark measurements so as 
> not to cross that line.
> 
> 
> 
> 
> But again, the lack of an objective way to compare these traffic
> management functions is a gaping hole in the industry *and* the 
> importance/interest in proper traffic management is at the fore front 
> of many carriers / providers.
> 
> 
> 
> 
> So we're looking to the folks in BMWG to chime in, help sort out the
> scope, provide expertise in meaningful benchmark metrics, etc.
> 
> 
> 
> 
> Thank you,
> 
> Barry Constantine
> 
> 
> 
> 
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg
> 
> 
> 
> 
> --
> "Speed is fine, but accuracy is final." Wyatt Earp, 1888 
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg
> 
> Scott O Bradner
> Senior Technology Consultant
> 
> Harvard University Information Technology Innovation & Architecture
> (P) +1 (617) 495 3864
> 1033 Mass Ave, room 462
> Cambridge, MA 02138
> 
> 
> 
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/bmwg