Re: AD Eval of draft-ietf-bmwg-vswitch-opnfv

Hi Warren,

The latest version of this draft:
https://datatracker.ietf.org/doc/draft-ietf-bmwg-vswitch-opnfv/

addresses your comments plus some other editorial
clean-up, and includes an abbreviations section 
(which should help with further reviews).

thanks again for your review,
Al (for the co-authors)

> -----Original Message-----
> From: MORTON, ALFRED C (AL)
> Sent: Tuesday, April 25, 2017 7:43 AM
> To: 'Warren Kumari'; [email protected];
> [email protected]; [email protected]
> Subject: RE: AD Eval of draft-ietf-bmwg-vswitch-opnfv
> 
> Thanks for your review & comments, Warren; the edits will
> take a few days to reach the top of the ToDo list.
> Al
> 
> > -----Original Message-----
> > From: Warren Kumari [mailto:[email protected]]
> > Sent: Sunday, April 23, 2017 10:40 AM
> > To: [email protected]; [email protected];
> > [email protected]
> > Subject: AD Eval of draft-ietf-bmwg-vswitch-opnfv
> >
> > Hi there,
> >
> > I've just done the AD eval of draft-ietf-bmwg-vswitch-opnfv - thank
> > you for writing this, I think that it is a useful document.
> >
> > I had a few questions and comments which I think  it would be useful
> > to address before starting IETF LC. I'm not a benchmarking (or NFV /
> > vswitch person), so some of these questions / comments may be very
> > basic.
> >
> > I've noted my questions with [WK: ...]. My questions / comments look
> > long, because I kept much of the text to make readability easier.
> >
> > ---------
> >
> >
> > 2.  Scope
> >
> >    The primary purpose and scope of the memo is to inform the industry
> >    of work-in-progress that builds on the body of extensive BMWG
> >    literature and experience, and describe the extensions needed for
> >    benchmarking virtual switches.  Inital feedback indicates that many
> >
> > [WK: s/Inital/Initial/]
> >
> >    of these extensions may be applicable beyond the current scope (to
> >    hardware switches in the NFV Infrastructure and to virtual routers,
> >    for example).  Additionally, this memo serves as a vehicle to
> include
> >    more detail and commentary from BMWG and other Open Source
> >    communities, under BMWG's chartered work to characterize the NFV
> >
> >    Infrastructure (a virtual switch is an important aspect of that
> >    infrastructure).
> >
> >    The benchmarking covered in this memo should be applicable to many
> >    types of vswitches, and remain vswitch-agnostic to great degree.
> >
> > [WK: Please expand / define vswitch - perhaps in the parenthesis
> above.]
> >
> >    There has been no attempt to track and test all features of any
> >    specific vswitch implementation.
> >
> > ...
> >
> > 3.1.  Comparison with Physical Network Functions
> >
> >    To compare the performance of virtual designs and implementations
> >    with their physical counterparts, identical benchmarks are needed.
> >    BMWG has developed specifications for many network functions this
> >    memo re-uses existing benchmarks through references, and expands
> them
> >    during development of new methods.
> >
> > [ WK: This sentence makes no sense / is missing some words / should be
> > 2 sentences / something....]
> >
> >    A key configuration aspect is the
> >    number of parallel cores required to achieve comparable performance
> >    with a given physical device, or whether some limit of scale was
> >    reached before the cores could achieve the comparable level.
> >
> >    It's unlikely that the virtual switch will be the only application
> >    running on the SUT, so CPU utilization, Cache utilization, and
> Memory
> >    footprint should also be recorded for the virtual implementations
> of
> >    internetworking functions.
> >
> > ...
> >
> > 3.3.  New Configuration Parameters
> >
> >    A key consideration when conducting any sort of benchmark is trying
> >    to ensure the consistency and repeatability of test results.  When
> >    benchmarking the performance of a vSwitch
> >
> > [ WK: Please choose a single term (vswitch / vSwitch) ]
> >
> >    there are many factors that
> >    can affect the consistency of results, one key factor is matching
> the
> >    various hardware and software details of the SUT.  This section
> lists
> >    some of the many new parameters which this project believes are
> >    critical to report in order to achieve repeatability.
> >
> >    Hardware details including:
> >
> >    o  Platform details
> >
> >    o  Processor details
> >
> >    o  Memory information (type and size)
> >
> > [ WK: I would also think that the memory layout is important - if not,
> > ignore. Oh, it is mentioned below ("Memory DIMM configurations"). This
> > would be easier to read if you grouped memory together, processor /
> > cores together, etc.]
> >
> >
> >    o  Number of enabled cores
> >
> >    o  Number of cores used for the test
> >
> >    o  Number of physical NICs, as well as their details (manufacturer,
> >       versions, type and the PCI slot they are plugged into)
> >
> >    o  NIC interrupt configuration
> >
> > [WK: What about other NIC features (DDP, checksum magic,
> > scatter-gather, etc)? Or are these not applicable? (if not, ignore) ]
> >
> >    o  BIOS version, release date and any configurations that were
> >       modified
> >
> >    o  CPU microcode level
> >
> >    o  Memory DIMM configurations (quad rank performance may not be the
> >       same as dual rank) in size, freq and slot locations
> >
> >    o  PCI configuration parameters (payload size, early ack option...)
> >
> >    o  Power management at all levels (ACPI sleep states, processor
> >       package, OS...)
> >
> >    Software details including:
> >
> >    o  OS parameters and behavior (text vs graphical no one typing at
> the
> >       console on one system)
> >
> >    o  OS version (for host and VNF)
> >
> >    o  Kernel version (for host and VNF)
> >
> >    o  GRUB boot parameters (for host and VNF)
> >
> >    o  Hypervisor details (Type and version)
> >
> >    o  Selected vSwitch, version number or commit id used
> >
> >    o  vSwitch launch command line if it has been parameterised
> >
> >    o  Memory allocation to the vSwitch
> >
> >    o  which NUMA node it is using, and how many memory channels
> >
> >    o  DPDK or any other SW dependency version number or commit id used
> >
> >    o  Memory allocation to a VM - if it's from Hugpages/elsewhere
> >
> > [WK: Hugpages? It this the emo version of hugepages? :-) ]
> >
> >
> >    o  VM storage type: snapshot/independent persistent/independent
> non-
> >       persistent
> >
> >    o  Number of VMs
> >
> >    o  Number of Virtual NICs (vNICs), versions, type and driver
> >
> >
> > [ WK: What about IO virtualization stuff like SR-IOV and similar. Or
> > not applicable?]
> >
> >    o  Number of virtual CPUs and their core affinity on the host
> >
> >    o  Number vNIC interrupt configuration
> >
> >    o  Thread affinitization for the applications (including the
> vSwitch
> >       itself) on the host
> >
> >    o  Details of Resource isolation, such as CPUs designated for Host/
> >       Kernel (isolcpu) and CPUs designated for specific processes
> >       (taskset). - Test duration. - Number of flows.
> >
> >    Test Traffic Information:
> >
> >    o  Traffic type - UDP, TCP, IMIX / Other
> >
> >
> > [ WK: I don't think that IMIX is a well known acronym. I think either
> > define it (or ref: RFC6985?), or drop it (the section is "some of the
> > many new parameters"...)]
> >
> >    o  Packet Sizes
> >
> >    o  Deployment Scenario
> >
> > 3.4.  Flow classification
> >
> >    Virtual switches group packets into flows by processing and
> matching
> >    particular packet or frame header information, or by matching
> packets
> >    based on the input ports.  Thus a flow can be thought of a sequence
> >    of packets that have the same set of header field values (5-tuple)
> or
> >    have arrived on the same port.  Performance results can vary based
> on
> >    the parameters the vSwitch uses to match for a flow.  The
> recommended
> >    flow classification parameters for any vSwitch performance tests
> are:
> >    the input port, the source IP address, the destination IP address
> and
> >    the Ethernet protocol type field.
> >
> > [ WK: Is it useful to say "5-tuple" above? To me 5-tuple is src ip,
> > src port, dst ip, dst port, ip protocol. This says for vswitch it is
> > input port, src ip, dst ip and eth protocol. The two close together
> > may confuse readers - I think just drop "(5-tuple)". Also, this says
> > "input port" - is that actually input interface / should it be called
> > "physical port"? (when hearing IP and then port, people think L3
> > port)]
> >
> >
> >    o  Scalability Tests to understand how the virtual switch performs
> as
> >       the number of flows, active ports, complexity of the forwarding
> >       logic's configuration... it has to deal with increases.
> >
> > [ WK: Looks like got sidetracked in this sentence. ]
> >
> >
> > ---------
> >
> > Thanks again,
> > W
> >
> > --
> > I don't think the execution is relevant when it was obviously a bad
> > idea in the first place.
> > This is like putting rabid weasels in your pants, and later expressing
> > regret at having chosen those particular rabid weasels and that pair
> > of pants.
> >    ---maf