Benchmarking Methodology (bmwg) ------------------------------- Charter Last Modified: 2011-12-09 Current Status: Active Working Group Chair(s): Al Morton Operations and Management Area Director(s): Dan Romascanu Ronald Bonica Operations and Management Area Advisor: Ronald Bonica Mailing Lists: General Discussion:bmwg@ietf.org To Subscribe: bmwg-request@ietf.org In Body: subscribe your_email_address Archive: http://www.ietf.org/mail-archive/web/bmwg/index.html Description of Working Group: The Benchmarking Methodology Working Group (BMWG) will continue to produce a series of recommendations concerning the key performance characteristics of internetworking technologies, or benchmarks for network devices, systems, and services. Taking a view of networking divided into planes, the scope of work includes benchmarks for the management, control, and forwarding planes. Each recommendation will describe the class of equipment, system, or service being addressed; discuss the performance characteristics that are pertinent to that class; clearly identify a set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of benchmarking results. The set of relevant benchmarks will be developed with input from the community of users (e.g, network operators and testing organizations) and from those affected by the benchmarks when they are published (networking and test equipment manufacturers). When possible, the benchmarks and other terminology will be developed jointly with organizations that are willing to share their expertise. Joint review requirements for a specific work area will be included in the detailed description of the task, as listed below. To better distinguish the BMWG from other measurement initiatives in the IETF, the scope of the BMWG is limited to the characterization of implementations of various internetworking technologies using controlled stimuli in a laboratory environment. Said differently, the BMWG does not attempt to produce benchmarks for live, operational networks. Moreover, the benchmarks produced by this WG shall strive to be vendor independent or otherwise have universal applicability to a given technology class. Because the demands of a particular technology may vary from deployment to deployment, a specific non-goal of the Working Group is to define acceptance criteria or performance requirements. An ongoing task is to provide a forum for discussion regarding the advancement of measurements designed to provide insight on the capabilities and operation of inter-networking technology implementations. The BMWG will communicate with the operations community through organizations such as NANOG, RIPE, and APRICOT. In addition to its current work plan, the BMWG is explicitly tasked to develop benchmarks and methodologies for the following technologies: * BGP Control-plane Convergence Methodology (Terminology is complete): With relevant performance characteristics identified, BMWG will prepare a Benchmarking Methodology Document with review from the Routing Area (e.g., the IDR working group and/or the RTG-DIR). The Benchmarking Methodology will be Last-Called in all the groups that previously provided input, including another round of network operator input during the last call. * SIP Networking Devices: Develop new terminology and methods to characterize the key performance aspects of network devices using SIP, including the signaling plane scale and service rates while considering load conditions on both the signaling and media planes. This work will be harmonized with related SIP performance metric definitions prepared by the PMOL working group. * Flow Export and Collection: Develop terminology and methods to characterize network devices flow monitoring, export, and collection. The goal is a methodology to assess the maximum IP flow rate that a network device can sustain without losing any IP flow information or compromising the accuracy of information exported on the IP flows, and to asses the forwarding plane performance (if the forwarding function is present) in the presence of Flow Monitoring. * Data Center Bridging Devices: Some key concepts from BMWG's past work are not meaningful when testing switches that implement new IEEE specifications in the area of data center bridging. For example, throughput as defined in RFC 1242 cannot be measured when testing devices that implement three new IEEE specifications: priority-based flow control (802.1Qbb); priority groups (802.1Qaz); and congestion notification (802.1Qau). Since devices that implement these new congestion-management specifications should never drop frames, and since the metric of throughput distinguishes between non-zero and zero drop rates, no throughput measurement is possible using the existing methodology. The current emphasis is on the Priority Flow Control aspects of Data Center Bridging, and the work will include an investigation into whether TRILL RBridges require any specific treatment in the methodology. This work will update RFC 2544 and exchange periodic Liaisons with IEEE 802.1 DCB Task Group, especially at WG Last Call. * Content Aware Devices: New classes of network devices that operate above the IP layer of the network stack require a new methodology to perform adequate benchmarking. Existing BMWG RFCs (RFC2647 and RFC3511) provides useful measurement and performance statistics, though they may not reflect the actual performance of the device when deployed in production networks. Operating within the limitations of the charter, namely blackbox characterization in laboratory environments, the BMWG will develop a methodology that more closely relates the performance of these devices to performance in an operational setting. In order to confirm or identify key performance characteristics, BMWG will solicit input from operations groups such as NANOG, RIP and APRICOT. * LDP Dataplane Convergence: In order to identify key LDP convergence performance characteristics, BMWG will solicit input from operations groups such as NANOG, RIP and APRICOT. When relevant performance characteristics have been identified, BMWG will jointly prepare a Benchmarking Terminology Document with the Routing Area (e.g., the MPLS working group and or the RTG-DIR), which would define metrics relevant to LDP convergence. The Benchmark definition document would be Last-Called in all the working groups that produced it, and solicit operator input during the last call. The work will then continue in BMWG to define the test methodology, with input and review from the aforementioned parties. Goals and Milestones: Done Expand the current Ethernet switch benchmarking methodology draft to define the metrics and methodologies particular to the general class of connectionless, LAN switches. Done Edit the LAN switch draft to reflect the input from BMWG. Issue a new version of document for comment. If appropriate, ascertain consensus on whether to recommend the draft for consideration as an RFC. Done Take controversial components of multicast draft to mailing list for discussion. Incorporate changes to draft and reissue appropriately. Done Submit workplan for initiating work on Benchmarking Methodology for LAN Switching Devices. Done Submit workplan for continuing work on the Terminology for Cell/Call Benchmarking draft. Done Submit initial draft of Benchmarking Methodology for LAN Switches. Done Submit Terminology for IP Multicast Benchmarking draft for AD Review. Done Submit Benchmarking Terminology for Firewall Performance for AD review Done Progress ATM benchmarking terminology draft to AD review. Done Submit Benchmarking Methodology for LAN Switching Devices draft for AD review. Done Submit first draft of Firewall Benchmarking Methodology. Done First Draft of Terminology for FIB related Router Performance Benchmarking. Done First Draft of Router Benchmarking Framework Done Progress Frame Relay benchmarking terminology draft to AD review. Done Methodology for ATM Benchmarking for AD review. Done Terminology for ATM ABR Benchmarking for AD review. Done Terminology for FIB related Router Performance Benchmarking to AD review. Done Firewall Benchmarking Methodology to AD Review Done First Draft of Methodology for FIB related Router Performance Benchmarking. Done First draft Net Traffic Control Benchmarking Methodology. Done Methodology for IP Multicast Benchmarking to AD Review. Done Resource Reservation Benchmarking Terminology to AD Review Done First I-D on IPsec Device Benchmarking Terminology Done EGP Convergence Benchmarking Terminology to AD Review Done Resource Reservation Benchmarking Methodology to AD Review Done Net Traffic Control Benchmarking Terminology to AD Review Done IGP/Data-Plane Terminology I-D to AD Review Done IGP/Data-Plane Methodology and Considerations I-Ds to AD Review Done Hash and Stuffing I-D to AD Review Done IPv6 Benchmarking Methodology to AD Review Done IPsec Device Benchmarking Terminology to IESG Review Done IPsec Device Benchmarking Methodology to IESG Review Done Terminology For Protection Benchmarking to AD Review Done Methodology for MPLS Forwarding to AD Review Done Networking Device Reset Benchmark (Updates RFC 2544) to IESG Review Dec 2010 Methodology For Protection Benchmarking to IESG Review Feb 2011 Methodology for Flow Export and Collection Benchmarking to IESG Review Jun 2011 Methodology for Data Center Bridging Benchmarking to IESG Review Jun 2011 Terminology for SIP Device Benchmarking to IESG Review Jun 2011 Methodology for SIP Device Benchmarking to IESG Review Jul 2011 Basic BGP Convergence Benchmarking Methodology to IESG Review Dec 2011 Terminology for Content Aware Device Benchmarking to IESG Review Dec 2011 Methodology for Content Aware Device Benchmarking to IESG Dec 2011 Terminology for LDP Convergence Benchmarking to IESG Review Dec 2011 Methodology for LDP Convergence Benchmarking to IESG Review Internet-Drafts: Posted Revised I-D Title ------ ------- -------------------------------------------- Oct 2006 Oct 2011 Methodology for benchmarking MPLS protection mechanisms Dec 2010 Jan 2012 IP Flow Information Accounting and Export Benchmarking Methodology Aug 2011 Oct 2011 RFC 2544 Applicability Statement: Use on Production Networks Considered Harmful Sep 2011 Sep 2011 Benchmarking Methodology for Content-Aware Network Devices Oct 2011 Jan 2012 IMIX Genome: Specification of variable packet sizes for additional testing Request For Comments: RFC Stat Published Title ------- -- ----------- ------------------------------------ RFC1242 I Jul 1991 Benchmarking Terminology for Network Interconnection Devices RFC1944 I May 1996 Benchmarking Methodology for Network Interconnect Devices RFC2285 I Feb 1998 Benchmarking Terminology for LAN Switching Devices RFC2432 I Oct 1998 Terminology for IP Multicast Benchmarking RFC2544 I Mar 1999 Benchmarking Methodology for Network Interconnect Devices RFC2647 I Aug 1999 Benchmarking Terminology for Firewall Performance RFC2761 I Feb 2000 Terminology for ATM Benchmarking RFC2889 I Aug 2000 Benchmarking Methodology for LAN Switching Devices RFC3116 I Jun 2001 Methodology for ATM Benchmarking RFC3133 I Jun 2001 Terminology for Frame Relay Benchmarking RFC3134 I Jun 2001 Terminology for ATM ABR Benchmarking RFC3222 I Dec 2001 Terminology for Forwarding Information Base (FIB) based Router Performance RFC3511 I Apr 2003 Benchmarking Methodology for Firewall Performance RFC3918 I Oct 2004 Methodology for IP Multicast Benchmarking RFC4098 I Jun 2005 Terminology for Benchmarking BGP Device Convergence in the Control Plane RFC4063 I Jun 2005 Considerations When Using Basic OSPF Convergence Benchmarks RFC4062 I Jun 2005 OSPF Benchmarking Terminology and Concepts RFC4061 I Jun 2005 Benchmarking Basic OSPF Single Router Control Plane Convergence RFC4689 I Oct 2006 Terminology for Benchmarking Network-layer Traffic Control Mechanisms RFC4814 I Mar 2007 Hash and Stuffing: Overlooked Factors in Network Device Benchmarking RFC4883 I Jul 2007 Benchmarking Terminology for Resource Reservation Capable Routers RFC5180 I May 2008 IPv6 Benchmarking Methodology for Network Interconnect Devices RFC5695 I Nov 2009 MPLS Forwarding Benchmarking Methodology for IP Flows RFC6201 I Mar 2011 Device Reset Characterization RFC6412 I Nov 2011 Terminology for Benchmarking Link-State IGP Data Plane Route Convergence RFC6413 I Nov 2011 Benchmarking Methodology for Link-State IGP Data Plane Route Convergence RFC6414 I Nov 2011 Benchmarking Terminology for Protection Performance