FW: Benchmarking Methodology for OpenFlow SDN Controller Performance

Hello BMWG,
	I sent comments to the authors on the thread here privately and forgot to
cc bmwg. The authors have asked me to send these comments to the list.
Here is the original email.

Thanks
Sarah

On 4/1/14, 5:27 PM, "Banks, Sarah" <[email protected]> wrote:

>Authors,
>
>	Thanks for your draft! SDN is interesting, and has come up in some
>hallway conversations - it's nice to see a draft on it. I've reviewed the
>doc, and I'll try not to take a stance, but rather, address issues that I
>see that are either technically wrong or lead to the potential to be
>wrong, or make the draft more readable. I'll point out that your draft
>could use an Editor's eye - there are grammatical issues throughout. I'm
>happy to help if you need, please let me know.
>
>	I like to review section by section. I hope that approach works for you.
>Here goes. :)
>
>	Overall, I like that your approach is very straight forward, but I find
>that the document suffers from a true introduction. Indeed, your
>Appendices read very nicely, and I found myself thinking that this
>information would have been nice at the top of the hour - at the beginning
>of the document. Please consider relocating this information. Not everyone
>is an SDN guru, and not everyone takes an identical approach - level
>setting where you're coming from and your scope will make consuming the
>rest of the document (hopefully) easier. Does this make sense?
>
>	Let me go section by section now.
>
>In Section 5.1 - You mention that the switches need not be physical
>switches, but emulated/simulated by software or test tools instead - the
>last sentence of this paragraph says that you "MUST indicate the number of
>switches and hosts..." - consider referencing the emulated/simulated
>number too. A quick read might leave open the idea that only physical
>switches used be noted/listed.
>
>Section 5.3 refers to the ability to have differing ways to setup channels
>and connection type, yet only leverages some (as yet undefined) channel
>setup modes. Why? Why are all of the methods/modes not applicable or being
>used?
>
>Section 5.3.1.2: In active-passive mode, you state, "Any asynchronous
>communications from the switch have to be sent on the active channel."
>While I suspect you don't mean this in a normative way, it begs the
>question - why?
>
>Section 5.3.1.3 - why call this out if you aren't going to require that
>their use be noted in the test results?
>
>Section 5.3.2 - What is controller teaming? (OK, I know what it is, but
>you're using a term not yet defined. Please define.)
>
>Section 5.3.2 - Why are you recommending the use of TCP here, over the
>alternatives? Be specific.
>
>In general, you don't take a stance on the number of times a test should
>be run, nor the amount of time to wait/pause between tests. Would you
>consider adding values to your recommendations?
> 
>Your reporting format is repeated over and over - I'm a fan of state it
>once, and modify it when needed. :) Just a thought.
>
>Section 9 - Security Considerations - to my eye, it's unacceptable to
>state that there might be security considerations in the security
>considerations section, and then state that they're out of scope. I
>disagree. They are in scope. What are the issues? It's a north bound
>interface in the lab. What's up?
>
>Is Appendix A (Section 10) a sample test report? There's no information
>about the setup/topology of the setup. If this isn't what App A is
>intended to be, consider adding a sample test report. I'd argue it might
>not live as an appendix, either. :)
>
>In that vein, Appendix A has no quantification about WHAT it is - what is
>it? Why are there Nas? :) Please clarify.