AD Eval of draft-ietf-bmwg-vswitch-opnfv

Hi there,

I've just done the AD eval of draft-ietf-bmwg-vswitch-opnfv - thank
you for writing this, I think that it is a useful document.

I had a few questions and comments which I think  it would be useful
to address before starting IETF LC. I'm not a benchmarking (or NFV /
vswitch person), so some of these questions / comments may be very
basic.

I've noted my questions with [WK: ...]. My questions / comments look
long, because I kept much of the text to make readability easier.

---------


2.  Scope

   The primary purpose and scope of the memo is to inform the industry
   of work-in-progress that builds on the body of extensive BMWG
   literature and experience, and describe the extensions needed for
   benchmarking virtual switches.  Inital feedback indicates that many

[WK: s/Inital/Initial/]

   of these extensions may be applicable beyond the current scope (to
   hardware switches in the NFV Infrastructure and to virtual routers,
   for example).  Additionally, this memo serves as a vehicle to include
   more detail and commentary from BMWG and other Open Source
   communities, under BMWG's chartered work to characterize the NFV

   Infrastructure (a virtual switch is an important aspect of that
   infrastructure).

   The benchmarking covered in this memo should be applicable to many
   types of vswitches, and remain vswitch-agnostic to great degree.

[WK: Please expand / define vswitch - perhaps in the parenthesis above.]

   There has been no attempt to track and test all features of any
   specific vswitch implementation.

...

3.1.  Comparison with Physical Network Functions

   To compare the performance of virtual designs and implementations
   with their physical counterparts, identical benchmarks are needed.
   BMWG has developed specifications for many network functions this
   memo re-uses existing benchmarks through references, and expands them
   during development of new methods.

[ WK: This sentence makes no sense / is missing some words / should be
2 sentences / something....]

   A key configuration aspect is the
   number of parallel cores required to achieve comparable performance
   with a given physical device, or whether some limit of scale was
   reached before the cores could achieve the comparable level.

   It's unlikely that the virtual switch will be the only application
   running on the SUT, so CPU utilization, Cache utilization, and Memory
   footprint should also be recorded for the virtual implementations of
   internetworking functions.

...

3.3.  New Configuration Parameters

   A key consideration when conducting any sort of benchmark is trying
   to ensure the consistency and repeatability of test results.  When
   benchmarking the performance of a vSwitch

[ WK: Please choose a single term (vswitch / vSwitch) ]

   there are many factors that
   can affect the consistency of results, one key factor is matching the
   various hardware and software details of the SUT.  This section lists
   some of the many new parameters which this project believes are
   critical to report in order to achieve repeatability.

   Hardware details including:

   o  Platform details

   o  Processor details

   o  Memory information (type and size)

[ WK: I would also think that the memory layout is important - if not,
ignore. Oh, it is mentioned below ("Memory DIMM configurations"). This
would be easier to read if you grouped memory together, processor /
cores together, etc.]


   o  Number of enabled cores

   o  Number of cores used for the test

   o  Number of physical NICs, as well as their details (manufacturer,
      versions, type and the PCI slot they are plugged into)

   o  NIC interrupt configuration

[WK: What about other NIC features (DDP, checksum magic,
scatter-gather, etc)? Or are these not applicable? (if not, ignore) ]

   o  BIOS version, release date and any configurations that were
      modified

   o  CPU microcode level

   o  Memory DIMM configurations (quad rank performance may not be the
      same as dual rank) in size, freq and slot locations

   o  PCI configuration parameters (payload size, early ack option...)

   o  Power management at all levels (ACPI sleep states, processor
      package, OS...)

   Software details including:

   o  OS parameters and behavior (text vs graphical no one typing at the
      console on one system)

   o  OS version (for host and VNF)

   o  Kernel version (for host and VNF)

   o  GRUB boot parameters (for host and VNF)

   o  Hypervisor details (Type and version)

   o  Selected vSwitch, version number or commit id used

   o  vSwitch launch command line if it has been parameterised

   o  Memory allocation to the vSwitch

   o  which NUMA node it is using, and how many memory channels

   o  DPDK or any other SW dependency version number or commit id used

   o  Memory allocation to a VM - if it's from Hugpages/elsewhere

[WK: Hugpages? It this the emo version of hugepages? :-) ]


   o  VM storage type: snapshot/independent persistent/independent non-
      persistent

   o  Number of VMs

   o  Number of Virtual NICs (vNICs), versions, type and driver


[ WK: What about IO virtualization stuff like SR-IOV and similar. Or
not applicable?]

   o  Number of virtual CPUs and their core affinity on the host

   o  Number vNIC interrupt configuration

   o  Thread affinitization for the applications (including the vSwitch
      itself) on the host

   o  Details of Resource isolation, such as CPUs designated for Host/
      Kernel (isolcpu) and CPUs designated for specific processes
      (taskset). - Test duration. - Number of flows.

   Test Traffic Information:

   o  Traffic type - UDP, TCP, IMIX / Other


[ WK: I don't think that IMIX is a well known acronym. I think either
define it (or ref: RFC6985?), or drop it (the section is "some of the
many new parameters"...)]

   o  Packet Sizes

   o  Deployment Scenario

3.4.  Flow classification

   Virtual switches group packets into flows by processing and matching
   particular packet or frame header information, or by matching packets
   based on the input ports.  Thus a flow can be thought of a sequence
   of packets that have the same set of header field values (5-tuple) or
   have arrived on the same port.  Performance results can vary based on
   the parameters the vSwitch uses to match for a flow.  The recommended
   flow classification parameters for any vSwitch performance tests are:
   the input port, the source IP address, the destination IP address and
   the Ethernet protocol type field.

[ WK: Is it useful to say "5-tuple" above? To me 5-tuple is src ip,
src port, dst ip, dst port, ip protocol. This says for vswitch it is
input port, src ip, dst ip and eth protocol. The two close together
may confuse readers - I think just drop "(5-tuple)". Also, this says
"input port" - is that actually input interface / should it be called
"physical port"? (when hearing IP and then port, people think L3
port)]


   o  Scalability Tests to understand how the virtual switch performs as
      the number of flows, active ports, complexity of the forwarding
      logic's configuration... it has to deal with increases.

[ WK: Looks like got sidetracked in this sentence. ]


---------

Thanks again,
W

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf