Re: Benchmarking Methodology for SDN Controller Performance - Updated Draft Version
"Marius Georgescu" <[email protected]>
| Newsgroups | gmane.ietf.bmwg |
|---|---|
| Message-ID | <[email protected]> |
Dear Bhuvan et alia, Please find attached my inline suggestions for the new draft. They were marked with ###MG . Thank you for taking care of so many comments. Best regards, Marius From: bmwg [mailto:[email protected]] On Behalf Of Bhuvan (Veryx Technologies) Sent: Tuesday, July 21, 2015 8:23 PM To: [email protected] Cc: 'Anton Basil' <[email protected]>; Tassinari, Mark A <[email protected]>; vishwas.manral <[email protected]> Subject: [bmwg] Benchmarking Methodology for SDN Controller Performance - Updated Draft Version Dear BMWG Members, We have updated the draft (draft-bhuvan-bmwg-sdn-controller-benchmark-meth-00 <http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth- 00> ) about SDN Controller benchmarking addressing comments received in IETF-92 meeting. Thank you very much for providing your valuable comments. The latest draft can be found in draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01 <http://tools.ietf.org/html/draft-bhuvan-bmwg-sdn-controller-benchmark-meth- 01> Summary of Changes: a. Updated test setup diagram following the comment from Scott Bradner. b. Added recommendations for test topology, test iterations etc., to use for benchmarking. c. Provided reference test topologies. d. Split Path Provisioning tests into two different tests - Proactive and Reactive Path Provisioning tests. e. Provided more clarity on test procedure for some of the tests. f. Fixed IETF normative language usage. We would love to hear any comments and queries on the same. Thanks, Authors _______________________________________________ bmwg mailing list [email protected] https://www.ietf.org/mailman/listinfo/bmwg
draft-bhuvan-bmwg-sdn-controller-benchmark-meth-01 MG.TXT
(text/plain, 8.3 KB)
...
4.6 Test Repeatability
To increase the confidence in measured result, it is recommended that
this test SHOULD be performed atleast 10 times with same number of
nodes using same topology.
###MG : Reporting the repeatability probably will come up at some point. I would suggest reporting
the relative standard deviation. The calculation formula can be something like :
sigma
rsd = ----------- x 100
avgScore
...
6.1.3 Asynchronous Message Processing Rate
Objective:
To measure the maximum number of asynchronous messages (session
aliveness check message, new flow arrival notification
message etc.) a controller can process within the test duration,
expressed in messages processed per second.
Reference Test Setup:
The test SHOULD use one of the test setups described in
section 3.1 or section 3.2 of this document.
Prerequisite:
1. The controller MUST have completed the network topology
discovery for the connected SDN nodes.
Procedure:
1. Generate asynchronous messages continuously at the maximum
possible rate on the established connections from all the
connected SDN nodes in the forwarding plane test emulator
for the Test Duration (Td).
2. Record total number of responses received from the
controller (Nrx) as well as the number of messages sent(Ntx) to
the controller within the test duration(Td) at the forwarding
plane test emulator interface (I1) .
Measurement:
Nrx
Asynchronous Message Processing Rate Tr1 = -----
Td
Tr1 + Tr2 + Tr3..Trn
Average Asynchronous Message Processing Rate= --------------------
Total Test Iterations
Loss Ratio = (Ntx-Nrx)/100.
###MG: There might be mistake here. I guess the calculation formula should be something like this:
Loss ratio = (Ntx - Nrx/Ntx)*100
...
6.1.8 Network Topology Change Detection Time
Objective:
Measure the time taken by the controller to detect any changes
in the network topology, expressed in milliseconds.
Reference Test Setup:
The test SHOULD use one of the test setups described in
section 3.1 or section 3.2 of this document.
Prerequisite:
1. The controller MUST have discovered the network topology
information for the deployed network topology.
2. The periodic network discovery operation should be configured
to twice the Test duration (Td) value.
Bhuvan, et al. Expires January 18, 2016 [Page 16]
Internet Draft SDN Controller Benchmarking Methodology July 2015
Procedure:
1. Trigger a topology change event by bringing down an active
SDN node in the topology.
2. Record the time when the first topology change notification
is sent to the controller (Tcn) at the forwarding plane
test emulator interface (I1).
3. Stop the test when the controller sends the first topology
re-discovery message to the SDN node or the expiry of test
interval (Td).
4. Record the time when the first topology re-discovery message
is received from the controller (Tcd) at the forwarding plane
test emulator interface (I1)
Measurement:
Network Topology Change Detection Time Tr1 = Tcd-Tcn.
Tr1 + Tr2 + Tr3 .. Trn
Average Network Topology Change
Detection Time = ---------------------------
Total Test Iterations
Reporting Format:
The Network Topology Change Detection Time results MUST be
reported in the format of a table with a row for each iteration.
The last row of the table indicates the average Network Topology
Change Time.
###MG: I've received this question for my draft as well(from Kostas), and and maybe it stands as a general question for the WG.
Why average (mean) and not median ? A short wording on why choose one over the other would probably do. I was thinking
to cite some of the published results and show that the probability distribution was normal. Hence, the mean and median
would be almost identical and the average has the advantage of including outliers, which I think are important for the overall
performance.
...
11.4 Performance Benchmarking Tests
11.4.1 Network Topology Discovery Time
Procedure:
SDN Nodes OpenFlow SDN
Controller Application
| | |
| |<Initialize controller |
| |app.,NB and SB interfaces> |
| | |
|<Deploy network with | |
| given no. of OF switches> | |
| | |
| OFPT_HELLO Exchange | |
|<-------------------------->| |
| | |
| PACKET_OUT with LLDP | |
| to all switches | |
(Tm1)|<---------------------------| |
| | |
| PACKET_IN with LLDP| |
| rcvd from switch-1| |
|--------------------------->| |
| | |
| PACKET_IN with LLDP| |
| rcvd from switch-2| |
|--------------------------->| |
| . | |
| . | |
| | |
| PACKET_IN with LLDP| |
| rcvd from switch-n| |
(Tmn)|--------------------------->| |
| | |
| | <Wait for the expiry |
| | of Test Duration (Td)>|
| | |
| | Query the controller for|
| | discovered n/w topo.(Di)|
| |<--------------------------|
| | |
| | <Compare the discovered |
| | & offered n/w topology>|
| | |
Bhuvan, et al. Expires January 18, 2016 [Page 28]
Internet Draft SDN Controller Benchmarking Methodology July 2015
###MG : I have to say that these diagrams are helping a lot with understanding the benchmarking procedure.
I haven't done this in my draft as well, but I think adding some sample reports might be actually very useful.
Legend:
NB: Northbound
SB: Southbound
OF: OpenFlow
Tm1: Time of reception of first LLDP message from controller
Tmn: Time of last LLDP message sent to controller
Discussion:
The Network Topology Discovery Time can be obtained by calculating
the time difference between the first PACKET_OUT with LLDP message
received from the controller (Tm1) and the last PACKET_IN with
LLDP message sent to the controller (Tmn) when the comparison is
successful.
###MG : The name of the paragraph (Discussion) is a bit confusing, I think. I guess you want to detail the calculation
method. Discussion makes me think that you're still not sure that this is the way to calculate it.
...