Re: Feedback for Benchmarking Methodology for OpenFlow SDN Controller Performance

Hi Al,

Thanks for your comment. I do agree with you that there needs to be a common
benchmarking mechanism for controller designs performing the same tasks.
But the technology is still relatively immature and the approach has various
dimensions including centralized control vs distributed control (controller
less).
Again the centralized control approach uses different programming methods
(OpenFlow vs Non OpenFlow).

Having said that, for better usability/understanding of the draft and to
avoid misinterpretation, I personally feel it would be good to have separate
benchmarking methodology for each approach (though the metrics remain same).
Also we would be happy to continue the effort to extend the same metrics for
other approaches too.

Please let me know if I've missed something.

Sarah/Brain, please also share your thoughts on this.

Thanks,
Bhuvan

-----Original Message-----
From: MORTON, ALFRED C (AL) [mailto:[email protected]] 
Sent: Saturday, June 21, 2014 6:00 PM
To: Castelli, Brian; Banks, Sarah; Bhuvan (Veryx Technologies);
[email protected]
Cc: 'Anton Basil'; 'Vishwas Manral'
Subject: RE: [bmwg] Feedback for Benchmarking Methodology for OpenFlow SDN
Controller Performance

Hi Brian, Sarah, Bhuvan, and all,

a couple of quick answers and a comment:

> -----Original Message-----
> From: bmwg [mailto:[email protected]] On Behalf Of Castelli, Brian
...
> My two main concerns are getting the terminology right and making sure 
> that the proposed set of tests will achieve our goals. I believe those 
> goals ought to include a common taxonomy, enabling apples-to-apples 
> comparisons, and minimizing the time/work required to execute the test 
> cases.

Great, that's exactly what we are chartered to do in BMWG.
http://datatracker.ietf.org/wg/bmwg/charter/


> 
> I am willing to help improve the benchmark. I understand that it is an 
> early draft. The time is right to make sure that we are developing the 
> best specification that we can.
> 
> How do we continue this process? Via email to this list? I am willing 
> to help.

It's principally e-mail (just as you've been doing) and face-to-face
meetings three times a year.  Sarah mentions that our next meeting is in her
home town:
http://www.ietf.org/meeting/90/index.html

Make some text proposals for the draft, and we can discuss them on the list.

Having said that, and recognizing that there appears to be a comparable
activity in ONF that (I assume) is OpenFlow-specific, perhaps a way to add
value to the industry is to approach the SDN controller problem more
generically, such that different controller designs performing the same
tasks could be benchmarked as black-boxes.  I realize this approach has
substantial implications for the draft, but the benefit of wider
applicability.

food for thought,
Al
(as a participant)