Re: WG ADOPTION: draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)

...to add *some* of these metrics, I believe.  
(just to be clear, not sure what happened there,
something out of sync...)

Al

> -----Original Message-----
> From: bmwg [mailto:[email protected]] On Behalf Of MORTON, ALFRED C
> (AL)
> Sent: Monday, October 12, 2015 10:56 AM
> To: [email protected]
> Subject: Re: [bmwg] WG ADOPTION: draft-bhuvan-bmwg-of-controller-
> benchmarking (terms and meth)
> 
> *** Security Advisory: This Message Originated Outside of AT&T ***.
> Reference http://cso.att.com/EmailSecurity/IDSP.html for more
> information.
> 
> BMWG Folks, (this is the thread I was referring to in reply to Jacob)
> 
> I support the adoption of these drafts as WG items.
> 
> I'd also like to see some comments below addressed (e.g., Brian's), if
> they haven't already. We would have to look at controller-clusters to do
> add of these metrics, I believe.
> 
> regards,
> Al
> (as a participant)
> 
> -----Original Message-----
> From: Castelli, Brian [mailto:[email protected]]
> Sent: Monday, June 23, 2014 11:39 AM
> To: Bhuvan (Veryx Technologies); MORTON, ALFRED C (AL); 'Banks, Sarah';
> [email protected]
> Cc: 'Anton Basil'; 'Vishwas Manral'
> Subject: Re: [bmwg] Feedback for Benchmarking Methodology for OpenFlow
> SDN Controller Performance
> 
> I really like Al¹s suggestion to generalize SDN controller benchmarking
> because 1) the ONF effort is OpenFlow specific and 2) OpenFlow does not
> define SDN. OpenFlow is one of several southbound protocols that can be
> used between a controller function and network node functions.
> Regardless of the maturity of current OpenFlow implementations, I
> believe that we can define a family of benchmarks that have general
> applicability to SDN environments because there are common
> functions/services that will be provided independent of protocol. At the
> end of the day, protocols are commodities. It¹s the service they provide
> that adds value.
> 
> The current draft contains many of the elements required to sufficiently
> define a benchmarking methodology for a controller function in an SDN
> environment. We should follow Al¹s suggestion, define the taxonomy used
> in the draft, and consolidate wording to make the document easier to
> read and apply.
> 
> What follows is an attempt to define a higher-level set of SDN
> controller test cases. It builds on the work done in the draft. I will
> provide some detail on the first test case in order to establish a
> cadence. Subsequent test cases will not have the detail, but the ability
> to iterate over variables and so on is implied.
> 
> The SDN Controller test taxonomy would include:
> 
> - Asynchronous Message Response Rate. A test case to measure AMRR would
> cover a variety of conditions that are analogous to the following
> OpenFlow-specific events:
> 
>   o Packet_in received
>   o Flow expiration received
>   o Link down received
> 
> In other words, AMRR would be used to measure the ability of an SDN
> controller to respond to a variety of asynchronous (i.e. unexpected)
> conditions. One methodology for measurement would be specified by the
> benchmark. The tester would then apply that methodology to packet_ins,
> flow expiration, link down, and whatever other AMR testing is
> appropriate.
> 
> The AMRR methodology should include iteration variables and goal
> seeking.
> The tester ought to be able to use the methodology to determine, for
> example, the maximum packet_in AMRR for a variety of conditions, such as
> variations in the number of connected nodes, the complexity of the
> packet_ins, and so on. Variables also include negative testing with
> invalid packets, for example.
> 
> - Synchronous Message Response Rate. A test case for measuring SMRR
> would cover a variety of conditions analogous to the following OpenFlow-
> specific
> events:
> 
>   o Hello packet exchange
>   o Echo request/response exchange
> 
> - Node Acquisition Rate. A test case for measuring NAR would be a
> special case of SMRR, but it specifically measures the rate at which an
> SDN controller can connect to nodes in a network.
> 
> - Node Acquisition Maximum. A test case for NAM would measure the
> maximum number of network nodes that an SDN controller can connect to
> without error.
> 
> - FailOver Response Time. There are at least two test cases for
> measuring FORT. One would measure a control cluster¹s ability to survive
> a controller failure. How long does it take the controller function to
> be assumed by another controller element? Another would measure the
> controller¹s ability to respond to node failures to do things like re-
> route traffic. How long does it take for the controller to send out
> commands that handle the error? This is a bit like measuring convergence
> time, but care must be taken to keep the node variables out of the
> measurement.
> 
> - FailBack Response Time. Test cases for FBRT would be used to measure
> the response time when the controller comes back up or the node comes
> back online. How long does it take the controller to send the commands
> that restore service?
> 
> 
> 
> On 6/23/14, 10:53 AM, "Bhuvan (Veryx Technologies)"
> <[email protected]> wrote:
> 
> >Hi Al,
> >
> >Thanks for your comment. I do agree with you that there needs to be a
> >common benchmarking mechanism for controller designs performing the
> >same tasks.
> >But the technology is still relatively immature and the approach has
> >various dimensions including centralized control vs distributed control
> >(controller less).
> >Again the centralized control approach uses different programming
> >methods (OpenFlow vs Non OpenFlow).
> >
> >Having said that, for better usability/understanding of the draft and
> >to avoid misinterpretation, I personally feel it would be good to have
> >separate benchmarking methodology for each approach (though the metrics
> >remain same).
> >Also we would be happy to continue the effort to extend the same
> >metrics for other approaches too.
> >
> >Please let me know if I've missed something.
> >
> >Sarah/Brain, please also share your thoughts on this.
> >
> >Thanks,
> >Bhuvan
> >
> >-----Original Message-----
> >From: MORTON, ALFRED C (AL) [mailto:[email protected]]
> >Sent: Saturday, June 21, 2014 6:00 PM
> >To: Castelli, Brian; Banks, Sarah; Bhuvan (Veryx Technologies);
> >[email protected]
> >Cc: 'Anton Basil'; 'Vishwas Manral'
> >Subject: RE: [bmwg] Feedback for Benchmarking Methodology for OpenFlow
> >SDN Controller Performance
> >
> >Hi Brian, Sarah, Bhuvan, and all,
> >
> >a couple of quick answers and a comment:
> >
> >> -----Original Message-----
> >> From: bmwg [mailto:[email protected]] On Behalf Of Castelli,
> >> Brian
> >...
> >> My two main concerns are getting the terminology right and making
> >> sure that the proposed set of tests will achieve our goals. I believe
> >> those goals ought to include a common taxonomy, enabling
> >> apples-to-apples comparisons, and minimizing the time/work required
> >> to execute the test cases.
> >
> >Great, that's exactly what we are chartered to do in BMWG.
> >http://datatracker.ietf.org/wg/bmwg/charter/
> >
> >
> >>
> >> I am willing to help improve the benchmark. I understand that it is
> >> an early draft. The time is right to make sure that we are developing
> >> the best specification that we can.
> >>
> >> How do we continue this process? Via email to this list? I am willing
> >> to help.
> >
> >It's principally e-mail (just as you've been doing) and face-to-face
> >meetings three times a year.  Sarah mentions that our next meeting is
> >in her home town:
> >http://www.ietf.org/meeting/90/index.html
> >
> >Make some text proposals for the draft, and we can discuss them on the
> >list.
> >
> >Having said that, and recognizing that there appears to be a comparable
> >activity in ONF that (I assume) is OpenFlow-specific, perhaps a way to
> >add value to the industry is to approach the SDN controller problem
> >more generically, such that different controller designs performing the
> >same tasks could be benchmarked as black-boxes.  I realize this
> >approach has substantial implications for the draft, but the benefit of
> >wider applicability.
> >
> >food for thought,
> >Al
> >(as a participant)
> >
> >
> 
> Spirent Communications E-mail confidentiality.
> -----------------------------------------------------------------------
> This e-mail contains confidential and / or privileged information
> belonging to Spirent Communications plc, its affiliates and / or
> subsidiaries. If you are not the intended recipient, you are hereby
> notified that any disclosure, copying, distribution and / or the taking
> of any action based upon reliance on the contents of this transmission
> is strictly forbidden. If you have received this message in error please
> notify the sender by return e-mail and delete it from your system.
> 
> Spirent Communications plc
> Northwood Park, Gatwick Road, Crawley, West Sussex, RH10 9XN, United
> Kingdom.
> Tel No. +44 (0) 1293 767676
> Fax No. +44 (0) 1293 767677
> 
> Registered in England Number 470893
> Registered at Northwood Park, Gatwick Road, Crawley, West Sussex, RH10
> 9XN, United Kingdom.
> 
> Or if within the US,
> 
> Spirent Communications,
> 26750 Agoura Road, Calabasas, CA, 91302, USA.
> Tel No. 1-818-676- 2300
> 
> _______________________________________________
> bmwg mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/bmwg