Re: Scope of draft-constantine-bmwg-traffic-management-02

Hi Nick,

Welcome to "active state" on the list!

Feel free to share your relevant experiences here, though I think
it's fair to ask that you avoid naming the manufacturers of DUTs.
Test equipment source is worth mentioning when you have something
constructive to say about useful capabilities (as you did below).

regards,
Al
bmwg co-chair

From: Nick Lippis [mailto:[email protected]]
Sent: Tuesday, September 24, 2013 9:04 AM
To: Barry Constantine
Cc: MORTON, ALFRED C (AL); Bradner, Scott; [email protected]
Subject: Re: [bmwg] Scope of draft-constantine-bmwg-traffic-management-02

Hello Everyone,

I'm a long time lister and first time caller.

I like Barry's approach to measuring burst and shaper performance.  Has the group given consideration to the traffic profile at ingress, that is, to specify an IMIX that could stress the shaper in addition to varying input rates and packet sizes?  I've been using an IxCloud IMIX developed with Ixia for our industry switch test and would offer that for consideration.  Please advise.

All the best,

Nick


On Sep 24, 2013, at 7:08 AM, Barry Constantine <[email protected]<mailto:[email protected]>> wrote:


Hi Folks,

I think it is going to come down to a metric for measuring the transmission rate of the shaper (in addition to the other metrics already discussed including loss, jitter, etc.)

Prior to Berlin, our lab tests used this basic premise:

1. Shapers "smooth" traffic to conform to the policer CIR and burst (Bc and Be) parameters; this way bursty traffic is shaped and not policed by the policer
2. So the shaper can only burst so many bytes in a time interval (Tc)
3. The draft would specify guidelines for specifying this time interval; Tc = Bc / CIR is an example and the one that we used in preliminary testing
4. The shaper tests would measure the how many bytes are transmitted at the egress per this time interval (along with the other metrics; loss, jitter, etc that are already specified in the draft)

This technique seems to provide an objective way to compare performance amongst various equipment shapers.

Thoughts?

Thank you,
Barry Constantine

JDSU Communications Test
Principal Member Technical Staff
301-325-7069

-----Original Message-----
From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of MORTON, ALFRED C (AL)
Sent: Monday, September 23, 2013 3:27 PM
To: Bradner, Scott; [email protected]<mailto:[email protected]>
Subject: Re: [bmwg] Scope of draft-constantine-bmwg-traffic-management-02

Hi Tim,

Tim Copley wrote:
          <some device developers>
          have decided ... that
          they can reduce memory and save money.  Or stick with
          an older algorithm that does Time slices in 125ms.  I had
          hoped that this work will enable me to run performance
          tests against these vendors and call out the performance of
          their switch / router / whatever, and say your shapers
          don't perform as well as the other vendor.
          I'm not sure how that's a conformance test.

An algorithm that works on 125ms time slices is necessarily different from another that uses 5ms, or 1 second slices, and it may be enough to know the embedded time slice in use, or the range that can be configured - thus we may not need a test for this difference at all (if I understand your example above).

OTOH, it's more difficult to predict how memory limits will manifest in dynamic algorithm performance, and it certainly implies there will be a maximum number of simultaneous algorithms that can operate (capacity).
I think that's a regime of testing that may very well fit in our current charter. The input load characteristics are a factor in this evaluation, and also the timing of measurement samples at the DUT output.

IMO, we should attempt to develop benchmarks to usefully distinguish performance/capacity of different devices performing traffic management alg., and see if they are sufficient to meet your needs and others interested in this topic.

Does that sound like a good (enough) starting place?
Al
(participant)


-----Original Message-----
From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf
Of Bradner, Scott
Sent: Monday, September 23, 2013 1:47 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: [bmwg] Scope of
draft-constantine-bmwg-traffic-management-02

On Sep 23, 2013, at 11:47 AM, tim copley <[email protected]<mailto:[email protected]>> wrote:


So throwing in my 2 cents and the reason why I argued to keep
scheduling and queuing as a performance test rather than a
conformance test.

we may be having a terminology conflict

it is not a question of calling something a conformance vs a
performance test

the actual test is what determines if its a conformance or performance
test

an example:

          if you have  a test that checks to see what the output rate of a
filter is under varying input rates - that is a performance test

          if you have a test that checks to see that the pattern of output
packets is round robin from multiple inputs - that is a conformance
test

the fact that something is a conformance test does not make it bad but
it makes it outside the current scope of BMWG

but as I said earlier, I think this is something that the chairs & the
ADs need to look at

Scott


On Sep 23, 2013, at 11:47 AM, tim copley <[email protected]<mailto:[email protected]>> wrote:


So throwing in my 2 cents and the reason why I argued to keep
scheduling and queuing as a performance test rather than a
conformance test.  I think it goes a bit beyond just configuration
verification. I think the intent is to verify that the vendor has
correctly implemented the different types of scheduling or queue
modifications properly.
Which would in fact be a performance test.  Verifying that the
performance of vendor A's implementation of queue management or
Vendor A's implementation of Hardware rather than Vendor's Bs queue
management or hardware choice.

I've had several vendors who shall remain nameless that have decided
for some odd reason, usually financial, that they can reduce memory
and save money.  Or stick with an older algorithm that does Time
slices in 125ms.  I had hoped that this work will enable me to run
performance tests against these vendors and call out the performance
of their switch / router / whatever, and say your shapers don't
perform as well as the other vendor.

I'm not sure how that's a conformance test.

TimC


On Sun, Sep 22, 2013 at 6:35 AM, MORTON, ALFRED C (AL)
<[email protected]<mailto:[email protected]>> wrote:

Thanks for introducing this topic on the list, Barry.



There are functions which must perform correctly is all devices

we've benchmarked, but that has not been the emphasis of the benchmarks.



you wrote:

But again, the lack of an objective way to compare these traffic
management functions is a gaping hole in the industry


*and* the importance/interest in proper traffic management is at the
fore front of many carriers / providers.






Say we have configured traffic shapers to perform a specification,

such as strict policing of <=20Mbps.  Maybe a test of transmission
rate

accuracy would be appropriate here (in addition to the tests Barry
mentioned **).


For example, a device that operated with 18 such policers with 99.9%
accuracy,


but 2 policers with flow rate <95% of the configured value has
reached

some form of capacity limit we can report.



Al

(as a participant)



From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf
Of
Barry Constantine

Sent: Saturday, September 21, 2013 4:46 PM
To: [email protected]<mailto:[email protected]>
Subject: [bmwg] Scope of
draft-constantine-bmwg-traffic-management-02



Hi BMWG,



Wanted to bring a topic onto the list from IETF 87 and this had to
do
with the concern that the traffic management benchmarking work was
introducing aspects of conformance testing into the framework.




A summary of the discussion is provided below and we would like to
get
feedback and drive to closure with the working group.




To Scott Bradner's concern with the work:  "function verification
sounds
a lot like "conformance" and conformance is clearly out of scope for
BMWG".  The specific context Scott referenced was that of a traffic
shaper.




Here are some key paragraphs that we added to -02 to clarify the scope:



*** Section 1.2

"   The tests are divided into individual tests and rated capacity
tests.


  The individual tests are intended to verify the traffic
management

  function according to the specifications.  As an example, suppose
a

  traffic shaper is to be tested at a CIR of 20 Mbps.  The intent
of

  the individual test is to test one instance of the shaper and
it's

  ability to properly shape according to the metrics** defined in

  section 4."

** These metrics include burst size achieved, packet loss /
sequence,
jitter




*** Section 3

"   Also, it is not within scope to perform conformance testing. Tests

  defined in this framework benchmark the traffic management
functions

  according to the metrics defined in section 4 and do not address
any

  conformance to standards related to traffic management.  Traffic

  management specifications largely do not exist and this is a
prime

  driver for this framework; to provide an objective means to
compare

  vendor traffic management functions."



From the co-author's perspective, what we see in industry today is
that
vendors advertise functions such as policers, shapers, etc and there
is no means to compare performance between them.




We definitely want to avoid this work being any form of conformance
test
and realize we must carefully specify the benchmark measurements so as
not to cross that line.




But again, the lack of an objective way to compare these traffic
management functions is a gaping hole in the industry *and* the
importance/interest in proper traffic management is at the fore front
of many carriers / providers.




So we're looking to the folks in BMWG to chime in, help sort out the
scope, provide expertise in meaningful benchmark metrics, etc.




Thank you,

Barry Constantine




_______________________________________________
bmwg mailing list
[email protected]<mailto:[email protected]>
https://www.ietf.org/mailman/listinfo/bmwg




--
"Speed is fine, but accuracy is final." Wyatt Earp, 1888
_______________________________________________
bmwg mailing list
[email protected]<mailto:[email protected]>
https://www.ietf.org/mailman/listinfo/bmwg

Scott O Bradner
Senior Technology Consultant

Harvard University Information Technology Innovation & Architecture
(P) +1 (617) 495 3864
1033 Mass Ave, room 462
Cambridge, MA 02138



_______________________________________________
bmwg mailing list
[email protected]<mailto:[email protected]>
https://www.ietf.org/mailman/listinfo/bmwg
_______________________________________________
bmwg mailing list
[email protected]<mailto:[email protected]>
https://www.ietf.org/mailman/listinfo/bmwg
_______________________________________________
bmwg mailing list
[email protected]<mailto:[email protected]>
https://www.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/bmwg