Benchmarking Methodology WG Minutes WG Chair: Kevin Dubray Minutes reported by Kevin Dubray. The BMWG met at the 43rd IETF on Monday, December 7th, 1998. The group consisted of over 40 people. The chair presented the agenda: 1. Agenda/Administration 2. Benchmarking Terminology for Firewall Performance. 3. Terminology for Cell/Call Benchmarking 4. Presentation: Latency Characterization in LAN switching devices. 1. Agenda/Administration The agenda was approved as presented. Dubray announced that the multicast benchmarking terminology draft was announced as RFC 2432. Unfortunately, the LAN switch benchmarking methodology draft did not progress enough to merit discussion at this time. Additionally, the editor of the latency terminology draft believed that the scope of the work merited investigation before proceeding further. With regards to the multicast terminology draft, Dubray announced that a volunteer to help edit the draft was still needed. Interested parties should apply. 2. Benchmarking Terminology for Firewall Performance. David Newman was then introduced to lead a discussion on the Firewall Draft. David gave a quick overview of the modifications to the draft. Notable changes include: Security Association Session Session Establishment Session Maintenance Session Teardown Unit of Transfer Enhancement of connection-related terms. David also communicated that some other minor cleanup had also ocurred. In discussing some of the changes, David announced that he added the notion of security association to address an issue brought up in earlier discussion: one does not want to undertake a forwarding rate type of measurement until "all hands have been shaken." On the term "session," David was keen to point out that session is not a measurement itself; rather it was a list of things that needs to happen before measurement can happen. Likewise, the need to finish forwarding rate measurements should occur before session teardown. In exploring the term "Unit of Transfer," Newman stated that he felt the alternative term "transaction" was too generic. He also concluded that other definitions were too narrow. The suggested term "Unit of Work" was attractive, but David thought it didn't reflect the movement of data. In the UOT discussion, David thought it was important to communicate that while UOT is characteristically broad, users must keep the comparisons narrow to be useful. David closed the summary, saying that he believe the draft was ready for WG Last Call. A question was raised as to the communicated duality of term "session," - is it data or a metric? David articulated that he would attempt to clear up the ambiguity. Presuming the inclusion of discussed issues in a subsequent draft, the chair surveyed the group as to whether a WG Last Call should be issued on the draft. There was consensus to issue the Last Call. 4. Presentation: Latency Characterization in LAN switching devices. The chair announced that a reordering of the agenda was going to take place. John Dawson was introduced to give his presentation on Latency. (see slides in the proceeding or the Powerpoint presentation on the BMWG archive). Some questions were posed after the presentation. One person asked whether it was possible for latency measurements to yield a negative number. Dawson indicated that it was, indeed, possible - especially in devices with cut-through architectures. He added that it was his experience that these architecture are not predominant. He cited that handling mixed media speeds made cut-through more problematic than store-and-forward architectures. Another person asked whether the latency proposals operated at Layer 2 or Layer 3. John indicated the focus was on a packet from one interface to the next. There was discussion about the notion that 100% line rate is susceptible in regards to clock variances. Someone queried as to whether 100% is a practical case, anyways. John didn't believe so as many protocols require acknowledgement in practice. Another question was posed as to whether clock synchronization was an issue. John thought is was primarily an issue when synchronizing the DUT's clock with the test gear's clock. A question asked whether the drift of multiple streams cancel the net effect of the drift. Dawson said no. Dawson made the recommendation that a review of latency related items in existing RFCs may make make the metrics more meaningful. More pointedly, Dawson stated that he believes that current workding of the metrics or their related methodology lends to novel workarounds that demonstrate something other than true latency. Dawson said he would forward the presentation and related paper to the chair for posting on the BMWG list. 3. Terminology for Cell/Call Benchmarking With the editors of the draft not present, Dubray proceeded to attempt a review of the current draft. Many of the comments reflected that the metrics currently presented seemed too link-centric and did not necessarily offer insight into characterizing devices that combined network and link based forwarding technology such as edge devices. Another comment suggested that the measuring buffering ability and impact may be useful. With that Dubray closed the working group session and stated the WG goals for the next period: 1. Update and reissue the Firewall draft. Announce WG Last Call. 2. Show demonstrable progress on LAN switch methodology document. 3. Name editor(s) to multicast methodology work and produce a first draft. 4. Advance the Cell/Call benchmarking draft. ** Minutes after the WG adjournment, the Cell/Call terminology draft's editors arrived having experienced flight delays. An auxillary meeting was announced and was held on Tuesday, Dec 8. The minutes of that meeting follow: Terminology for Cell/Call Benchmarking Jeff Dunn and Cynthia Martin gave a quick overview of the presentation to meeting attendees. The presentation is to be posted to the list and in the BMWG minutes. Following the presentation, some issues were discussed. It was asked, "What is the goal of the draft given that other organizations (ITU-T, ATMF, etc.) are operating in this space. The editors responded that they were leveraging existing work, because the this work helps differentiate the parameters that may affect the IP context. For example, CDV is a factor that will affect TCP on the user layer. The authors communicated taking a "layered" approach, e.g., how does the service provider affect the service user(s). For example, if you have an error at the ATM layer, it has an implication at the AAL level, which has implications on the IP layer and TCP layer. The document needs to point out that there is correlation to higher layer(s). Authors also stated their belief that restricting the document's scope to ATM may allow for quicker prototypical work. The authors put forward the notion that they would like to advance the methodological work - if nothing else, they would like to start discussion on how to architect the methodolgy. The goal would be to produce a usable framework for testing, independent of implementation. When queried as to the changes for the next version of the terminology draft, the editors stated that they would like to develop the discussion section and enhance the measurements offered.