Re: Scope of draft-constantine-bmwg-traffic-management-02

Barry
I think it would be much better if the draft shows how UUT calculates the packet length.
Whether UUT's calculation includes only L2 frame or it also includes Preamble + IFG.

Regards,
-Shishio

(2013/09/24 20:08), Barry Constantine wrote:
> Hi Folks,
> 
> I think it is going to come down to a metric for measuring the transmission rate of the shaper (in addition to the other metrics already discussed including loss, jitter, etc.)
> 
> Prior to Berlin, our lab tests used this basic premise:
> 
> 1. Shapers "smooth" traffic to conform to the policer CIR and burst (Bc and Be) parameters; this way bursty traffic is shaped and not policed by the policer
> 2. So the shaper can only burst so many bytes in a time interval (Tc)
> 3. The draft would specify guidelines for specifying this time interval; Tc = Bc / CIR is an example and the one that we used in preliminary testing
> 4. The shaper tests would measure the how many bytes are transmitted at the egress per this time interval (along with the other metrics; loss, jitter, etc that are already specified in the draft)
> 
> This technique seems to provide an objective way to compare performance amongst various equipment shapers.
> 
> Thoughts?
> 
> Thank you,
> Barry Constantine
> 
> JDSU Communications Test
> Principal Member Technical Staff
> 301-325-7069
> 
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf Of MORTON, ALFRED C (AL)
> Sent: Monday, September 23, 2013 3:27 PM
> To: Bradner, Scott; [email protected]
> Subject: Re: [bmwg] Scope of draft-constantine-bmwg-traffic-management-02
> 
> Hi Tim,
> 
> Tim Copley wrote:
> 	<some device developers>
> 	have decided ... that
> 	they can reduce memory and save money.  Or stick with
> 	an older algorithm that does Time slices in 125ms.  I had
> 	hoped that this work will enable me to run performance
> 	tests against these vendors and call out the performance of
> 	their switch / router / whatever, and say your shapers
> 	don't perform as well as the other vendor.
> 	I'm not sure how that's a conformance test.
> 
> An algorithm that works on 125ms time slices is necessarily different from another that uses 5ms, or 1 second slices, and it may be enough to know the embedded time slice in use, or the range that can be configured - thus we may not need a test for this difference at all (if I understand your example above).
> 
> OTOH, it's more difficult to predict how memory limits will manifest in dynamic algorithm performance, and it certainly implies there will be a maximum number of simultaneous algorithms that can operate (capacity).
> I think that's a regime of testing that may very well fit in our current charter. The input load characteristics are a factor in this evaluation, and also the timing of measurement samples at the DUT output.
> 
> IMO, we should attempt to develop benchmarks to usefully distinguish performance/capacity of different devices performing traffic management alg., and see if they are sufficient to meet your needs and others interested in this topic.
> 
> Does that sound like a good (enough) starting place?
> Al
> (participant)
> 
>> -----Original Message-----
>> From: [email protected] [mailto:[email protected]] On Behalf
>> Of Bradner, Scott
>> Sent: Monday, September 23, 2013 1:47 PM
>> To: [email protected]
>> Subject: Re: [bmwg] Scope of
>> draft-constantine-bmwg-traffic-management-02
>>
>> On Sep 23, 2013, at 11:47 AM, tim copley <[email protected]> wrote:
>>
>>> So throwing in my 2 cents and the reason why I argued to keep
>>> scheduling and queuing as a performance test rather than a
>>> conformance test.
>>
>> we may be having a terminology conflict
>>
>> it is not a question of calling something a conformance vs a
>> performance test
>>
>> the actual test is what determines if its a conformance or performance
>> test
>>
>> an example:
>>
>> 	if you have  a test that checks to see what the output rate of a
>> filter is under varying input rates - that is a performance test
>>
>> 	if you have a test that checks to see that the pattern of output
>> packets is round robin from multiple inputs - that is a conformance
>> test
>>
>> the fact that something is a conformance test does not make it bad but
>> it makes it outside the current scope of BMWG
>>
>> but as I said earlier, I think this is something that the chairs & the
>> ADs need to look at
>>
>> Scott
>>
>>
>> On Sep 23, 2013, at 11:47 AM, tim copley <[email protected]> wrote:
>>
>>> So throwing in my 2 cents and the reason why I argued to keep
>>> scheduling and queuing as a performance test rather than a
>>> conformance test.  I think it goes a bit beyond just configuration
>>> verification. I think the intent is to verify that the vendor has
>>> correctly implemented the different types of scheduling or queue
>>> modifications properly.
>>> Which would in fact be a performance test.  Verifying that the
>>> performance of vendor A's implementation of queue management or
>>> Vendor A's implementation of Hardware rather than Vendor's Bs queue
>>> management or hardware choice.
>>>
>>> I've had several vendors who shall remain nameless that have decided
>>> for some odd reason, usually financial, that they can reduce memory
>>> and save money.  Or stick with an older algorithm that does Time
>>> slices in 125ms.  I had hoped that this work will enable me to run
>>> performance tests against these vendors and call out the performance
>>> of their switch / router / whatever, and say your shapers don't
>>> perform as well as the other vendor.
>>>
>>> I'm not sure how that's a conformance test.
>>>
>>> TimC
>>>
>>>
>>> On Sun, Sep 22, 2013 at 6:35 AM, MORTON, ALFRED C (AL)
>> <[email protected]> wrote:
>>> Thanks for introducing this topic on the list, Barry.
>>>
>>>
>>>
>>> There are functions which must perform correctly is all devices
>>>
>>> we've benchmarked, but that has not been the emphasis of the benchmarks.
>>>
>>>
>>>
>>> you wrote:
>>>
>>> But again, the lack of an objective way to compare these traffic
>> management functions is a gaping hole in the industry
>>>
>>> *and* the importance/interest in proper traffic management is at the
>> fore front of many carriers / providers.
>>>
>>>
>>>
>>>
>>>
>>> Say we have configured traffic shapers to perform a specification,
>>>
>>> such as strict policing of <=20Mbps.  Maybe a test of transmission
>>> rate
>>>
>>> accuracy would be appropriate here (in addition to the tests Barry
>> mentioned **).
>>>
>>> For example, a device that operated with 18 such policers with 99.9%
>> accuracy,
>>>
>>> but 2 policers with flow rate <95% of the configured value has
>>> reached
>>>
>>> some form of capacity limit we can report.
>>>
>>>
>>>
>>> Al
>>>
>>> (as a participant)
>>>
>>>
>>>
>>> From: [email protected] [mailto:[email protected]] On Behalf
>>> Of
>> Barry Constantine
>>> Sent: Saturday, September 21, 2013 4:46 PM
>>> To: [email protected]
>>> Subject: [bmwg] Scope of
>>> draft-constantine-bmwg-traffic-management-02
>>>
>>>
>>>
>>> Hi BMWG,
>>>
>>>
>>>
>>> Wanted to bring a topic onto the list from IETF 87 and this had to
>>> do
>> with the concern that the traffic management benchmarking work was
>> introducing aspects of conformance testing into the framework.
>>>
>>>
>>>
>>> A summary of the discussion is provided below and we would like to
>>> get
>> feedback and drive to closure with the working group.
>>>
>>>
>>>
>>> To Scott Bradner's concern with the work:  "function verification
>>> sounds
>> a lot like "conformance" and conformance is clearly out of scope for
>> BMWG".  The specific context Scott referenced was that of a traffic
>> shaper.
>>>
>>>
>>>
>>> Here are some key paragraphs that we added to -02 to clarify the scope:
>>>
>>>
>>>
>>> *** Section 1.2
>>>
>>> "   The tests are divided into individual tests and rated capacity
>> tests.
>>>
>>>     The individual tests are intended to verify the traffic
>>> management
>>>
>>>     function according to the specifications.  As an example, suppose
>>> a
>>>
>>>     traffic shaper is to be tested at a CIR of 20 Mbps.  The intent
>>> of
>>>
>>>     the individual test is to test one instance of the shaper and
>>> it's
>>>
>>>     ability to properly shape according to the metrics** defined in
>>>
>>>     section 4."
>>>
>>> ** These metrics include burst size achieved, packet loss /
>>> sequence,
>> jitter
>>>
>>>
>>>
>>> *** Section 3
>>>
>>> "   Also, it is not within scope to perform conformance testing. Tests
>>>
>>>     defined in this framework benchmark the traffic management
>>> functions
>>>
>>>     according to the metrics defined in section 4 and do not address
>>> any
>>>
>>>     conformance to standards related to traffic management.  Traffic
>>>
>>>     management specifications largely do not exist and this is a
>>> prime
>>>
>>>     driver for this framework; to provide an objective means to
>>> compare
>>>
>>>     vendor traffic management functions."
>>>
>>>
>>>
>>>  From the co-author's perspective, what we see in industry today is
>>> that
>> vendors advertise functions such as policers, shapers, etc and there
>> is no means to compare performance between them.
>>>
>>>
>>>
>>> We definitely want to avoid this work being any form of conformance
>>> test
>> and realize we must carefully specify the benchmark measurements so as
>> not to cross that line.
>>>
>>>
>>>
>>> But again, the lack of an objective way to compare these traffic
>> management functions is a gaping hole in the industry *and* the
>> importance/interest in proper traffic management is at the fore front
>> of many carriers / providers.
>>>
>>>
>>>
>>> So we're looking to the folks in BMWG to chime in, help sort out the
>> scope, provide expertise in meaningful benchmark metrics, etc.
>>>
>>>
>>>
>>> Thank you,
>>>
>>> Barry Constantine
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> bmwg mailing list
>>> [email protected]
>>> https://www.ietf.org/mailman/listinfo/bmwg
>>>
>>>
>>>
>>>
>>> --
>>> "Speed is fine, but accuracy is final." Wyatt Earp, 1888
>>> _______________________________________________
>>> bmwg mailing list
>>> [email protected]
>>> https://www.ietf.org/mailman/listinfo/bmwg
>>
>> Scott O Bradner
>> Senior Technology Consultant
>>
>> Harvard University Information Technology Innovation & Architecture
>> (P) +1 (617) 495 3864
>> 1033 Mass Ave, room 462
>> Cambridge, MA 02138
>>
>>
>>
>> _______________________________________________
>> bmwg mailing list
>> [email protected]
>> https://www.ietf.org/mailman/listinfo/bmwg
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg
> .
>