Re: WG Action: Rechartered Benchmarking Methodology (bmwg)

We made it this far!  Everything we asked to do is in there.
Let's get to work and meet our milestones.

Al
bmwg co-chair

> -----Original Message-----
> From: bmwg [mailto:[email protected]] On Behalf Of The IESG
> Sent: Thursday, June 12, 2014 2:01 PM
> To: IETF-Announce
> Cc: bmwg WG
> Subject: [bmwg] WG Action: Rechartered Benchmarking Methodology (bmwg)
> 
> The Benchmarking Methodology (bmwg) working group in the Operations and
> Management Area of the IETF has been rechartered. For additional
> information please contact the Area Directors or the WG Chairs.
> 
> Benchmarking Methodology (bmwg)
> ------------------------------------------------
> Current Status: Active WG
> 
> Chairs:
>   Sarah Banks <[email protected]>
>   Al Morton <[email protected]>
> 
> Assigned Area Director:
>   Joel Jaeggli <[email protected]>
> 
> Mailing list
>   Address: [email protected]
>   To Subscribe: [email protected]
>   Archive: http://www.ietf.org/mail-archive/web/bmwg/
> 
> Charter:
> 
> The Benchmarking Methodology Working Group (BMWG) will continue to
> produce a series of recommendations concerning the key performance
> characteristics of internetworking technologies, or benchmarks for
> network devices, systems, and services. Taking a view of networking
> divided into planes, the scope of work includes benchmarks for the
> management, control, and forwarding planes.
> 
> Each recommendation will describe the class of equipment, system, or
> service being addressed; discuss the performance characteristics that
> are pertinent to that class; clearly identify a set of metrics that aid
> in the description of those characteristics; specify the methodologies
> required to collect said metrics; and lastly, present the requirements
> for the common, unambiguous reporting of benchmarking results.
> 
> The set of relevant benchmarks will be developed with input from the
> community of users (e.g., network operators and testing organizations)
> and from those affected by the benchmarks when they are published
> (networking and test equipment manufacturers). When possible, the
> benchmarks and other terminologies will be developed jointly with
> organizations that are willing to share their expertise. Joint review
> requirements for a specific work area will be included in the detailed
> description of the task, as listed below.
> 
> To better distinguish the BMWG from other measurement initiatives in the
> IETF, the scope of the BMWG is limited to the characterization of
> implementations of various internetworking technologies
> using controlled stimuli in a laboratory environment. Said differently,
> the BMWG does not attempt to produce benchmarks for live, operational
> networks. Moreover, the benchmarks produced by this WG shall strive to
> be vendor independent or otherwise have universal applicability to a
> given technology class.
> 
> Because the demands of a particular technology may vary from deployment
> to deployment, a specific non-goal of the Working Group is to define
> acceptance criteria or performance requirements.
> 
> An ongoing task is to provide a forum for development and
> advancement of measurements which provide insight on the
> capabilities and operation of implementations of inter-networking
> technology.
> 
> Ideally, BMWG should communicate with the operations community
> through organizations such as NANOG, RIPE, and APRICOT.
> 
> The BMWG is explicitly tasked to develop benchmarks and methodologies
> for the following technologies:
> 
> BGP Control-plane Convergence Methodology (Terminology is complete):
> With relevant performance characteristics identified, BMWG will prepare
> a Benchmarking Methodology Document with review from the Routing Area
> (e.g., the IDR working group and/or the RTG-DIR). The Benchmarking
> Methodology will be Last-Called in all the groups that previously
> provided input, including another round of network operator input during
> the last call.
> 
> SIP Networking Devices: Develop new terminology and methods to
> characterize the key performance aspects of network devices using
> SIP, including the signaling plane scale and service rates while
> considering load conditions on both the signaling and media planes. This
> work will be harmonized with related SIP performance metric definitions
> prepared by the PMOL working group.
> 
> Traffic Management: Develop the methods to characterize the capacity
> of traffic management features in network devices, such as
> classification,
> policing, shaping, and active queue management. Existing terminology
> will be used where appropriate. Configured operation will be verified
> as a part of the methodology. The goal is a methodology to assess the
> maximum forwarding performance that a network device can sustain without
> dropping or impairing packets, or compromising the accuracy of multiple
> instances of traffic management functions. This is the benchmark for
> comparison between devices. Another goal is to devise methods that
> utilize flows with congestion-aware transport as part of the traffic
> load and still produce repeatable results in the isolated test
> environment.
> 
> IPv6 Neighbor Discovery: Large address space in IPv6 subnets presents
> several networking challenges, as described in RFC 6583. Indexes to
> describe the performance of network devices, such as the number of
> reachable devices on a sub-network, are useful benchmarks to the
> operations community. The BMWG will develop the necessary
> terminology and methodologies to measure such benchmarks.
> 
> In-Service Software Upgrade: Develop new methods and benchmarks to
> characterize the upgrade of network devices while in-service,
> considering both data and control plane operations and impacts.
> These devices are generally expected to maintain control plane session
> integrity, including routing connections. Quantification of upgrade
> impact will include packet loss measurement, and other forms of recovery
> behavior will be noted accordingly. The work will produce a definition
> of ISSU, which will help refine the scope.??Liaisons will be established
> as needed.
> 
> Data Center Benchmarking: This work will define additional terms,
> benchmarks, and methods applicable to data center performance
> evaluations.
> This includes data center specific congestion scenarios, switch buffer
> analysis, microburst, head of line blocking, while also using a wide mix
> of traffic conditions. Some aspects from BMWG's past work are not
> meaningful when testing switches that implement new IEEE specifications
> in the area of data center bridging. For example, throughput as defined
> in RFC 1242 cannot be measured when testing devices that implement three
> new IEEE specifications: priority-based flow control (802.1Qbb);
> priority groups (802.1Qaz); and congestion notification (802.1Qau).
> This work will update RFC1242, RFC2544, RFC2889 (and other key RFCs),
> and exchange Liaisons with relevant SDOs, especially at WG Last Call.
> 
> VNF and Related Infrastructure Benchmarking: Benchmarking Methodologies
> have reliably characterized many physical devices. This work item extends
> 
> and enhances the methods to virtual network functions (VNF) and their
> unique supporting infrastructure. A first deliverable from this activity
> will be a document that considers the new benchmarking space to ensure
> that common issues are recognized from the start, using background
> materials from industry and SDOs (e.g., IETF, ETSI NFV).
> Benchmarks for platform capacity and performance characteristics of
> virtual routers, switches, and related components will follow, including
> comparisons between physical and virtual network functions. In many
> cases,
> the traditional benchmarks should be applicable to VNFs, but the lab
> set-ups, configurations, and measurement methods will likely need to
> be revised or enhanced.
> 
> Milestones:
>   Jun 2014 - Basic BGP Convergence Benchmarking Methodology to IESG
> Review
>   Jul 2014 - Terminology for SIP Device Benchmarking to IESG Review
>   Jul 2014 - Methodology for SIP Device Benchmarking to IESG Review
>   Aug 2014 - Draft on Traffic Management Benchmarking to IESG Review
>   Dec 2014 - Draft on IPv6 Neighbor Discovery to IESG Review
>   Mar 2015 - Draft on In-Service Software Upgrade Benchmarking to IESG
> Review
>   Aug 2015 - Draft on VNF Benchmarking Considerations to IESG Review
>   Dec 2015 - Drafts on Data Center Benchmarking to IESG Review
>