Subject: Why closed IAB workshops ? Re: Call for Papers: Workshop on Analyzing IETF Data (AID), 2021

From: Toerless Eckert <[email-masked]>
Date: Fri, 27 Aug 2021 16:43:47 +0200

Dear IAB, *:

I wonder why IAB workshops continue to be closed/invitation only alone,
and would encourage IEF/IAB to rethink this policy in the face of our evolving
technology opportunities and policy challenges.

I think i understand all the reasons  why this makes sense for
active participation and even in-person passive participation (org, cost, focus),
but:

I would very much like to see a policy where all IETF activities are
as public as possible. In the case of workshops where there is no strong
content benefit of chatham house rules or similar concerns, and where
there is appropriate online tooling anyhow, i think there can and should
always be a live-stream to live listen into (but not participate actively)
such an event.

If for example webex is used, this is easily possible though webex/youtube
integration, so that passive observers can use youtube to listen live.

To bring up the dreaded topic again as another example: I do not know
if a closed IAB workshop would be legal place for a US person to provide
technology ideas when there are contributors in such an invittion only
workshop from entities on the US governments EAR entities list. I for once
can not read into EAR 734.7 that such an event is covered by it.
I certainly would love to see a statement of the IETF lawyer about this.

Cheers
    Toerless

P.S.: This aparticularily is one workshop where i wouldn't have anything
active to contribute, but where i am quite certain that i would learn
a lot more following the discussion in the room listening in remotely than
just reading a workshop report later. So admittedly, this announcement
triggered my mail from self-interest, but i have been disappointed by
this IAB policy for a long time already.

On Thu, Aug 26, 2021 at 10:15:47AM -0700, IAB Executive Administrative Manager wrote:
> Show me the numbers: Workshop on Analyzing IETF Data (AID), 2021
> 
> Web Page: https://www.iab.org/activities/workshops/aid/
> 
> The IETF as an international Standards Developing Organization hosts 
> diverse data on the history, development, and current activities in the 
> development and standardization of Internet protocols and its 
> institutions. A large portion of this data is publicly available, yet 
> this data is arguably underutilized as a tool to inform the work in the 
> IETF and research on topics like Internet governance and trends in ICT 
> standard-setting.
> 
> This workshop aims to enable engineers and researchers alike to mine the 
> IETF's data sources in order to explore trends through the analysis of 
> IETF data, such as email archives 
> <https://mailarchive.ietf.org/arch/browse/>, I-Ds 
> <https://www.ietf.org/standards/ids/>, RFCs 
> <https://www.ietf.org/standards/rfcs/>, and the datatracker 
> <https://datatracker.ietf.org/>. This work can be used to derive 
> insights into the inner workings of the process of standardization, 
> participation, and governance[1]. This workshop aims to bring together 
> people who have already analyzed IETF data, those who are interested in 
> the analysis of IETF data, and those who are interested in the results 
> of such analysis as input for improvement of the IETF's work.
> 
> We invite the research community, IETF participants, and others with an 
> interest in the data collected by the IETF, its protocols, and 
> participants, to submit a contribution to the workshop. Furthermore, we 
> also welcome participants who are interested in the analysis that could 
> be performed based on this data as well as those contributing 
> considerations regarding future collection and handling of IETF data.
> 
> Possible avenues for explorations include, but are not limited to:
> 
>   A. What are patterns for participation in the IETF (what are 
>      predictors for a long and productive tenure, when do people stop 
>      participating, what is needed to successfully produce RFCs)?
>   B. How is the IETF community developing (i.e., affiliations, 
>      publications, language, nationality, leadership positions)?
>   C. How do affiliations develop in the IETF (i.e., does a change in 
>      affiliation translate into a change in behavior, is there a 
>      relation between affiliation and leadership positions and/or 
>      centrality, what is the affiliation distribution per area and/or 
>      WG)?
>   D. What social dynamics (gender, nationality, income, occupation, and 
>      other social dynamics) are not captured by IETF data and what data 
>      and research approaches are needed to develop further insights in 
>      the social dynamics of standardization?
>   E. How productive and effective is the IETF, with respect to 
>      documents, pages, words, letters and in comparison the overall 
>      activities e.g. on mailing lists?
>   F. How well is the outcome of the IETF used, e.g,. based on references 
>      to RFCs in research papers, product manuals, or other sources?
>   G. What data would be relevant to collect that is not collected yet or 
>      what should be considered with respect to handling of personal data 
>      during the data collection and research.
>   H. How effective is the IETF's consensus-based decision making 
>      process? Is there evidence that documents receive broad and 
>      effective reviews? Are experts with relevant expertise engaging 
>      with developing standards in a timely manner?
> 
> Participation and Submission
> 
> People interested in participation are requested to submit short 
> position papers (500-1000 words). The paper can cover one or multiple of 
> the following points, but this list should not be considered exhaustive:
> 
>   1. Research questions and interests in IETF data; indication which 
>      question should be answered, the data needed to do so, and how 
>      these insights could be used to improve processes and operations;
>   2. Description of the IETF data they aim to analyze or the information 
>      they would like to see made available to inform their work (such as 
>      mailing list archives, or participation data obtained through the 
>      datatracker) and their methods for doing so (see footnote 1);
>   3. Potential and preliminary findings; and how those insights could 
>      either benefit leadership, WG chairs, and authors/participants, 
>      and/or society and industry at large;
>   4. Potential or preliminary findings and how those add novel insights 
>      to ongoing academic debates.
> 
> Proposals for data analysis should also contain a brief consideration of 
> any related ethics and privacy issues. The basic principles of ethical 
> research are outlined in the Belmont Report2 (covering e.g., respect for 
> persons, beneficence, and justice) and/or institutional ethics 
> guidelines.
> 
> The workshop will be invitation-only. The organizers will decide whom to 
> invite based on the submissions received. Therefore, please indicate 
> your interest by submitting a research proposal by September 29, 2021 to 
> [email-masked].
> 
> The Program Committee members are Niels ten Oever (chair, University of 
> Amsterdam), Colin Perkins (chair, IRTF, University of Glasgow), Corinne 
> Cath (chair, Oxford Internet Institute), Mirja Kühlewind (IAB, 
> Ericsson), Zhenbin Li (IAB, Huawei), Wes Hardaker (IAB, USC/ISI).
> 
> All inputs submitted and considered relevant will be published on the 
> workshop web page. Sessions will be organized according to content, and 
> not every accepted submission or invited attendee will have an 
> opportunity to present as the intent is to foster discussion and not 
> simply to have a sequence of presentations.
> 
> Position papers from those unable to attend in person are encouraged. A 
> workshop report will be published afterwards.
> 
> Logistics
> 
>    • Submissions Due: 29 September 2021
>    • Invitations Issued by: 15 October 2021
>    • Workshop Date: November 29 – December 3 2021
>    • Location: Online and at the University of Amsterdam (COVID-19 
>      permitting).
> 
> The workshop will consist of three parts:
> 
>   1. opening workshop (Monday)
>   2. hackathon (Tuesday – Thursday morning)
>   3. closing event (Thursday afternoon)
> 
> Feel free to contact the program committee with any further questions 
> (including questions related to available data or expected outcomes): 
> [email-masked].
> 
> -----
> [1] Examples of such approaches are: 
> https://www.arkko.com/tools/docstats.html, 
> http://datactive.github.io/bigbang/, 
> https://csperkins.org/research/protocol-standards/2020-12-10-ignacio-iesg-talk/2020-12-10_IESG-50-years-IETF-send.pdf, 
> https://sodestream.github.io/impact-of-early-engagement-on-longevity-of-ietf-participation.html
> 
> [2] https://www.hhs.gov/ohrp/sites/default/files/the-belmont-report-508c_FINAL.pdf
> 
> _______________________________________________
> IETF-Announce mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-announce

---

From: Stewart Bryant <[email-masked]>
Date: Fri, 27 Aug 2021 16:38:21 +0100

Unless the meeting is held under Chatham House rules, it should be as open as the resources allow.

Since we are meeting virtually for the foreseeable future a YouTube feed would be relatively simple and allow anyone interested to list first hand to the proceedings.

- Stewart

> On 27 Aug 2021, at 15:43, Toerless Eckert <[email-masked]> wrote:
> 
> Dear IAB, *:
> 
> Architecture-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/architecture-discuss

---

From: "Mirja Kühlewind (IETF)" <[email-masked]>
Date: Fri, 27 Aug 2021 17:57:22 +0200

Hi Toerless, hi Stewart,

We will publish the recordings after the workshop as we also did with the last workshop.

Mirja

> Am 27.08.2021 um 17:40 schrieb Stewart Bryant <[email-masked]>:
>

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 27 Aug 2021 16:29:40 +0000

The few workshops I have attended (four or five) were all under Chatham House rules, to enable frank and open discussion among those present.

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 27 Aug 2021 18:34:01 +0200

Thanks, Mirja!

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 27 Aug 2021 18:35:27 +0200

Hmm... can not remember i saw that being mentionined in invites. Do you rmember
an example one ?

How does that work with USA EAR ? Or where participants already excluded based on EAR ?

Cheers
    Toerless

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 28 Aug 2021 09:07:34 +1200

> The few workshops I have attended (four or five) were all under Chatham House rules, to enable frank and open discussion among those present.

"Under the Chatham House Rule, anyone who comes to a meeting is free to use information from the discussion, but is not allowed to reveal who made any particular comment." [Wikipedia]

Really? Things have certainly changed since I last attended an IAB workshop, then. Workshop reports aren't in he said/she said form but I'm unaware of any rule about revealing who said what.

   Brian

P.S. I would be very concerned about an IAB workshop whose activity was constrained by national regulations of any kind.

---

From: "Salz, Rich" <[email-masked]>
Date: Sat, 28 Aug 2021 14:18:31 +0000

an example one ?

Well there's this: https://www.iab.org/wiki/index.php/IAB_Workshop_Guidelines

 A search for "ietf iab chatham house rules site:iab.org" turns up many thins.

---

From: "Salz, Rich" <[email-masked]>
Date: Sat, 28 Aug 2021 14:21:44 +0000

>    Really?

Yes, really.

---

From: Toerless Eckert <[email-masked]>
Date: Sat, 28 Aug 2021 20:10:54 +0200

I am still be interested to learn if/how IAB did vet its invitation only workshop
rules with the IEF lawyer re. compliance with USA EAR rules.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Brad Biddle <[email-masked]>
Date: Sat, 28 Aug 2021 17:08:45 -0700

> rules with the IEF lawyer re. compliance with USA EAR rules.

Toerless and all -- one of the IETF's lawyers here. Without inviting lots
of nuanced discussion of a complex legal topic on large public mailing
lists, let me note that materials presented at "open conferences" are
broadly exempted from the EARs, and the definition of open conference is
reasonably broad -- e.g., it can include "a limitation on actual
attendance, as long as attendees either are the first who have applied or
are selected on the basis of relevant scientific or technical competence,
experience, or responsibility," among other things.

The regulatory language is frustratingly opaque, but please see 15 CFR
734.2 [1] and a relevant BIS FAQ [2]. Also let me assure you that export
control compliance is a key focus area for the legal team and we would
definitely flag any concerns for the IAB leadership and other stakeholders.

--Brad
--
Brad Biddle | [email-masked] | +1.503.502.1259 (mobile) | http://biddle.law

[1] https://www.law.cornell.edu/cfr/text/15/734.7
[2]
https://www.bis.doc.gov/index.php/documents/compliance-training/export-administration-regulations-training/1554-ear-definitions-faq/file

> > > The few workshops I have attended (four or five) were all under
> Chatham House rules, to enable frank and open discussion among those
> present.
> >

---

From: Brad Biddle <[email-masked]>
Date: Sat, 28 Aug 2021 17:13:53 -0700

On Sat, Aug 28, 2021 at 5:08 PM Brad Biddle <[email-masked]> wrote:

> 734.2 [1]

Typo in my text: 15 CFR 734.7 is the relevant regulation. The footnoted
link is correct.

> and a relevant BIS FAQ [2]. Also let me assure you that export control
> compliance is a key focus area for the legal team and we would definitely
> flag any concerns for the IAB leadership and other stakeholders.
>
> --Brad
> --

---

From: Jari Arkko <[email-masked]>
Date: Sun, 29 Aug 2021 21:58:51 +0300

Toerless,

I am personally pretty firmly in the keep-things-open camp. But I’ll observe two things. First, as you saw there was some discussion about interactions where it has been believed that Chatham house rules have been useful. That has indeed been the case a few times, though certainly not needed for every event.

But as a frequent member of the program committees in these events, I want to highlight another point which may not be readily visible if you just look at the text of these workshop descriptions. Yes, these events are by invitation only, and there’s a request to send a position paper or abstract to the organisers, and based on that an invitation may be forthcoming. However, while this sounds quite closed, in practice for most of the events — particularly the virtual ones -- we tend to invite everyone who demonstrated that they were on topic and had thought about the matter enough to write a rational contribution. Perhaps not quite everyone every time, but still. And the program committees tend to use the contributions also for other things, such as deciding who we might ask to lead a session. IAB workshops generally aren’t everyone-presents-their-paper conferences, the papers are required reading before starting the actual workshop. We do want to generate discussion, however, so the general thrust is to ensure that we get a set of people together who can constructively discuss the topic and take it forward.

Jari

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Sun, 29 Aug 2021 17:32:28 -0500

Just to add one point to Jari's excellent summary of recent practice.

On Sun, Aug 29, 2021 at 2:00 PM Jari Arkko <[email-masked]> wrote:

> Toerless,
>
> I am personally pretty firmly in the keep-things-open camp. But I’ll
> observe two things. First, as you saw there was some discussion about
> interactions where it has been believed that Chatham house rules have been
> useful. That has indeed been the case a few times, though certainly not
> needed for every event.
>
> But as a frequent member of the program committees in these events, I want
> to highlight another point which may not be readily visible if you just
> look at the text of these workshop descriptions. Yes, these events are by
> invitation only, and there’s a request to send a position paper or abstract
> to the organisers, and based on that an invitation may be forthcoming.
> However, while this sounds quite closed, in practice for most of the events
> — particularly the virtual ones -- we tend to invite everyone who
> demonstrated that they were on topic and had thought about the matter
> enough to write a rational contribution. Perhaps not quite everyone every
> time, but still. And the program committees tend to use the contributions
> also for other things, such as deciding who we might ask to lead a session.
> IAB workshops generally aren’t everyone-presents-their-paper conferences,
> the papers are required reading before starting the actual workshop.

At the MARNEW IAB workshop (https://www.iab.org/activities/workshops/marnew/),
we had about three times as many submitted position papers as
presentations, and at least some of the presentations summarized the
position papers in a topic areas - so we had several position papers that
talked about transport-level optimizations in wireless networks, and I
summarized the problems various people were attacking, and led the
discussion for that agenda item (my slides are here). So, it's not that you
do a position paper that disappears, it's that the ideas in your position
paper can have an impact on the workshop discussions and outcomes whether
you present those ideas yourself or not.

IIRC, the SEMI IAB workshop earlier that year (
https://www.iab.org/activities/workshops/semi/) did something similar,
although I wasn't helping with that one.

Best,

Spencer

> We do want to generate discussion, however, so the general thrust is to
> ensure that we get a set of people together who can constructively discuss
> the topic and take it forward.
>
> Jari
>

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 30 Aug 2021 03:41:58 +0200

Jari,

Thanks for the explanation, but: I did not raise any issue with the way
the invitations are handled as i have too little insight. All i was
asking for was that there should be a live stream unless there
is a justifyable reason for confidentiality such as chatham house rules.

Cheers
    Toerless

---

From: Colin Perkins <[email-masked]>
Date: Mon, 30 Aug 2021 11:56:49 +0100

Toerless,

Most of this workshop is a multi-day distributed hackathon. There is no benefit in trying to live stream several days worth of live coding Python scripts. There’s no insights and no useful transparency to be achieved by doing that. Openness, in this case, is better served by putting effort into releasing the resulting code, data sets, and analysis for others to build on – and by getting the right set of people together to be able to build that code and do that analysis.

I agree that IAB activities should be as public as possible. But policies that require a particular form of openness (such as requiring a live stream, for example) are not the right way of achieving that.

Colin

> On 30 Aug 2021, at 02:41, Toerless Eckert <[email-masked]> wrote:
> 
> Jari,
>

---

From: Bob Hinden <[email-masked]>
Date: Sat, 28 Aug 2021 18:18:13 -0700

> On Aug 28, 2021, at 7:21 AM, Salz, Rich <rsalz=[email-masked]> wrote:
> 

I can confirm that.   Also, as said here, there is only one Chatham House Rule, not as some have said “rules”.

I actually attend a meeting at Chatham House about four years ago :-)

Bob

---

From: Warren Kumari <[email-masked]>
Date: Mon, 30 Aug 2021 15:09:38 -0400

On Mon, Aug 30, 2021 at 11:57 AM Bob Hinden <[email-masked]> wrote:

> >>   "Under the Chatham House Rule, anyone who comes to a meeting is free
> to use information from the discussion, but is not allowed to reveal who
> made any particular comment." [Wikipedia]
> >
> >>   Really?
> >
> > Yes, really.
>

There is even a page (gasp!) explaining both the rule and the pluralization
thereof: https://www.chathamhouse.org/about-us/chatham-house-rule

W
P.S: I've just decided that "pluralization" is my new favorite word.

-- 
The computing scientist’s main challenge is not to get confused by the
complexities of his own making.
  -- E. W. Dijkstra

---

From: Jari Arkko <[email-masked]>
Date: Tue, 31 Aug 2021 17:26:00 +0300

Toerless,

> is a justifyable reason for confidentiality such as chatham house rules.

I actually agree with that. And again for context, we tended to debate that quite a bit few years ago with in-person workshops, and whether suitable conferencing equipment was available, whether it made sense even if it was available, etc. I’m in the camp where for most things a live stream should be provided. However, doesn’t take away Colin’s strong point about other mechanisms for providing transparency and access. In a lot of cases the access to structured reports, all papers, software, or even recordings are all very good mechanisms, indeed sometimes the best mechanisms as Colin points out.

Jari

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 31 Aug 2021 10:59:30 -0400

I have attended many meetings under Chatham house rules, I can't remember
doing an IETF meeting under one but would have no problem with that. I
would much prefer we had Chatham House than humming.

There are many reasons for holding Chatham House and not necessarily the
ones people assume. It is really not about attribution. First thing you
learn at corporate spokesperson school is there is no such thing as off the
record. If I go off the record with a journalist, it is for the specific
purpose of explaining something to them that I have already discovered they
are confused on and want to attempt to unconfuse them.

The real advantage of Chatham House it allows people to say something that
is incredibly stupid and get reactions.

I can't say what or who of course, but lets say that a person retired from
a very senior position says something that is 'could this start a nuclear
war' level stupid.

So a hypothetical is then given in about 20 words which the next person
reduces to 'redlines require immediately perfect attribution'.

And that counter argument is exactly what the original speaker was looking
for in the first place: a concise argument against framing information
engagement in the same terms as the nuclear deterrence strategy the
participants had developed.

One of the many pathologies of IETF meetings is that they are constructed
so that we spend rather too much time arguing for our positions and not
enough looking to find the right position.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 31 Aug 2021 11:23:10 -0400

Note that if we get back to face-to-face workshops, the trade-offs get 
more complicated among issues like encouraging free-flowing 
conversations and making a sensible stream / recording available.

Yours,
Joel

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 1 Sep 2021 00:58:40 +0200

I was not saying that the live stream should be codified as a mandate but tried
out as a very simple option to explore.

I do agree that if a workshop is a hackathon then passive participation will not
work. I did not see any such coding mentioned for the planned AID or Measurement
workshops though. From their text they looked a more like interesting discussions
where passive listening would be as interesting as uhmm... CSPAN ? ;-)) (aka:
really in the eye of the beholder. I do like to listen to CSPAN based on topic).

Cheers
    Toerless 

-- 
---
[email-masked]