Re: AD Eval of draft-ietf-bmwg-vswitch-opnfv

Thanks for your review & comments, Warren; the edits will
take a few days to reach the top of the ToDo list.
Al

> -----Original Message-----
> From: Warren Kumari [mailto:[email protected]]
> Sent: Sunday, April 23, 2017 10:40 AM
> To: [email protected]; [email protected];
> [email protected]
> Subject: AD Eval of draft-ietf-bmwg-vswitch-opnfv
> 
> Hi there,
> 
> I've just done the AD eval of draft-ietf-bmwg-vswitch-opnfv - thank
> you for writing this, I think that it is a useful document.
> 
> I had a few questions and comments which I think  it would be useful
> to address before starting IETF LC. I'm not a benchmarking (or NFV /
> vswitch person), so some of these questions / comments may be very
> basic.
> 
> I've noted my questions with [WK: ...]. My questions / comments look
> long, because I kept much of the text to make readability easier.
> 
> ---------
> 
> 
> 2.  Scope
> 
>    The primary purpose and scope of the memo is to inform the industry
>    of work-in-progress that builds on the body of extensive BMWG
>    literature and experience, and describe the extensions needed for
>    benchmarking virtual switches.  Inital feedback indicates that many
> 
> [WK: s/Inital/Initial/]
> 
>    of these extensions may be applicable beyond the current scope (to
>    hardware switches in the NFV Infrastructure and to virtual routers,
>    for example).  Additionally, this memo serves as a vehicle to include
>    more detail and commentary from BMWG and other Open Source
>    communities, under BMWG's chartered work to characterize the NFV
> 
>    Infrastructure (a virtual switch is an important aspect of that
>    infrastructure).
> 
>    The benchmarking covered in this memo should be applicable to many
>    types of vswitches, and remain vswitch-agnostic to great degree.
> 
> [WK: Please expand / define vswitch - perhaps in the parenthesis above.]
> 
>    There has been no attempt to track and test all features of any
>    specific vswitch implementation.
> 
> ...
> 
> 3.1.  Comparison with Physical Network Functions
> 
>    To compare the performance of virtual designs and implementations
>    with their physical counterparts, identical benchmarks are needed.
>    BMWG has developed specifications for many network functions this
>    memo re-uses existing benchmarks through references, and expands them
>    during development of new methods.
> 
> [ WK: This sentence makes no sense / is missing some words / should be
> 2 sentences / something....]
> 
>    A key configuration aspect is the
>    number of parallel cores required to achieve comparable performance
>    with a given physical device, or whether some limit of scale was
>    reached before the cores could achieve the comparable level.
> 
>    It's unlikely that the virtual switch will be the only application
>    running on the SUT, so CPU utilization, Cache utilization, and Memory
>    footprint should also be recorded for the virtual implementations of
>    internetworking functions.
> 
> ...
> 
> 3.3.  New Configuration Parameters
> 
>    A key consideration when conducting any sort of benchmark is trying
>    to ensure the consistency and repeatability of test results.  When
>    benchmarking the performance of a vSwitch
> 
> [ WK: Please choose a single term (vswitch / vSwitch) ]
> 
>    there are many factors that
>    can affect the consistency of results, one key factor is matching the
>    various hardware and software details of the SUT.  This section lists
>    some of the many new parameters which this project believes are
>    critical to report in order to achieve repeatability.
> 
>    Hardware details including:
> 
>    o  Platform details
> 
>    o  Processor details
> 
>    o  Memory information (type and size)
> 
> [ WK: I would also think that the memory layout is important - if not,
> ignore. Oh, it is mentioned below ("Memory DIMM configurations"). This
> would be easier to read if you grouped memory together, processor /
> cores together, etc.]
> 
> 
>    o  Number of enabled cores
> 
>    o  Number of cores used for the test
> 
>    o  Number of physical NICs, as well as their details (manufacturer,
>       versions, type and the PCI slot they are plugged into)
> 
>    o  NIC interrupt configuration
> 
> [WK: What about other NIC features (DDP, checksum magic,
> scatter-gather, etc)? Or are these not applicable? (if not, ignore) ]
> 
>    o  BIOS version, release date and any configurations that were
>       modified
> 
>    o  CPU microcode level
> 
>    o  Memory DIMM configurations (quad rank performance may not be the
>       same as dual rank) in size, freq and slot locations
> 
>    o  PCI configuration parameters (payload size, early ack option...)
> 
>    o  Power management at all levels (ACPI sleep states, processor
>       package, OS...)
> 
>    Software details including:
> 
>    o  OS parameters and behavior (text vs graphical no one typing at the
>       console on one system)
> 
>    o  OS version (for host and VNF)
> 
>    o  Kernel version (for host and VNF)
> 
>    o  GRUB boot parameters (for host and VNF)
> 
>    o  Hypervisor details (Type and version)
> 
>    o  Selected vSwitch, version number or commit id used
> 
>    o  vSwitch launch command line if it has been parameterised
> 
>    o  Memory allocation to the vSwitch
> 
>    o  which NUMA node it is using, and how many memory channels
> 
>    o  DPDK or any other SW dependency version number or commit id used
> 
>    o  Memory allocation to a VM - if it's from Hugpages/elsewhere
> 
> [WK: Hugpages? It this the emo version of hugepages? :-) ]
> 
> 
>    o  VM storage type: snapshot/independent persistent/independent non-
>       persistent
> 
>    o  Number of VMs
> 
>    o  Number of Virtual NICs (vNICs), versions, type and driver
> 
> 
> [ WK: What about IO virtualization stuff like SR-IOV and similar. Or
> not applicable?]
> 
>    o  Number of virtual CPUs and their core affinity on the host
> 
>    o  Number vNIC interrupt configuration
> 
>    o  Thread affinitization for the applications (including the vSwitch
>       itself) on the host
> 
>    o  Details of Resource isolation, such as CPUs designated for Host/
>       Kernel (isolcpu) and CPUs designated for specific processes
>       (taskset). - Test duration. - Number of flows.
> 
>    Test Traffic Information:
> 
>    o  Traffic type - UDP, TCP, IMIX / Other
> 
> 
> [ WK: I don't think that IMIX is a well known acronym. I think either
> define it (or ref: RFC6985?), or drop it (the section is "some of the
> many new parameters"...)]
> 
>    o  Packet Sizes
> 
>    o  Deployment Scenario
> 
> 3.4.  Flow classification
> 
>    Virtual switches group packets into flows by processing and matching
>    particular packet or frame header information, or by matching packets
>    based on the input ports.  Thus a flow can be thought of a sequence
>    of packets that have the same set of header field values (5-tuple) or
>    have arrived on the same port.  Performance results can vary based on
>    the parameters the vSwitch uses to match for a flow.  The recommended
>    flow classification parameters for any vSwitch performance tests are:
>    the input port, the source IP address, the destination IP address and
>    the Ethernet protocol type field.
> 
> [ WK: Is it useful to say "5-tuple" above? To me 5-tuple is src ip,
> src port, dst ip, dst port, ip protocol. This says for vswitch it is
> input port, src ip, dst ip and eth protocol. The two close together
> may confuse readers - I think just drop "(5-tuple)". Also, this says
> "input port" - is that actually input interface / should it be called
> "physical port"? (when hearing IP and then port, people think L3
> port)]
> 
> 
>    o  Scalability Tests to understand how the virtual switch performs as
>       the number of flows, active ports, complexity of the forwarding
>       logic's configuration... it has to deal with increases.
> 
> [ WK: Looks like got sidetracked in this sentence. ]
> 
> 
> ---------
> 
> Thanks again,
> W
> 
> --
> I don't think the execution is relevant when it was obviously a bad
> idea in the first place.
> This is like putting rabid weasels in your pants, and later expressing
> regret at having chosen those particular rabid weasels and that pair
> of pants.
>    ---maf