INTERIM_MEETING_REPORT_ Reported by Terry Gray/University of Washington Minutes of the Internet Message Access Protocol Working Group (IMAP) Summary An interim IMAP Working Group meeting was held at the University of Washington on August 30 and 31, 1993. Eight people attended. Twenty-three issues were discussed. A consensus position was reached on twenty of those issues. No consensus was reached, but some progress was made on the issues related to namespace semantics and hierarchy support. A separate message, attempting to summarize this particular issue, will be forthcoming. A new Internet-Draft, incorporating the results of this meeting and several suggestions made via e-mail, will be forthcoming for general review and comment. Issue 1: Extensibility of IMAP A. Envelope headers B. Body structure changes (e.g. content-disposition, MD5) o Discussion - There are some IMAP primitives that do not return as much information as would now be desirable. For example, it would be nice if the ENVELOPE structure included RESENT fields. Moreover, the BODY structure does not return content-disposition or MD5 information. - Modifying the existing primitives will break existing IMAP software; therefore, new functionality often implies a new primitive. - There is a difference between defining a protocol extension and defining a protocol extensibility mechanism. When considering the need to accommodate protocol evolution, it would be ideal to define a mechanism that would allow ``old'' clients to discover new fields and do something useful with them. But this is not viewed as a realistic expectation because how the new values would be presented to the client depends upon the type of data, and fully describing the data type, whether or not it is parsed or presented literally, etc., can lead to an overly complex protocol. (In short, a simple name=value property list is not sufficient.) - Even if we cannot build a client that will never have to be updated, (to track protocol evolution), at least we should strive to minimize the need to change clients and servers at the same time. We can define new primitives in such a way that clients receiving *lists* of elements can interoperate with a new server that will return more elements in the list than the client understands. That is, the primitive is specified so that the client will discard list elements beyond those it is expecting, rather than choke on them. - In order to accommodate MIME evolution, it is proposed that a new BODY type be defined along the above lines. It will return an S expression list including newly defined MIME elements, and will have new list elements defined over time as needed. - Doing the same thing for the ENVELOPE data structure was thought to not be worth the effort because there is now a primitive to fetch arbitrary groups of header lines---although they are returned as unparsed strings. ENVELOPE contains information to aid the client (e.g. by pre-parsing the basic addresses), but a client can request all or any headers using other primitives. While this places an additional burden on the client to be able to parse these other header lines, that burden is not considered more egregious than the effort to implement an ENVELOPE2 structure. o Action - Define ``BODY2'' which will allow for unanticipated fields. - Part 0 (zero) in encapsulated messages is defined to mean header. - No change re ENVELOPE. Issue 2: Mstring Grammar o Discussion - In RFC 1176, a FIND is defined to return ``* mailbox string'' but C-client actually returns ``* mailbox text-line'' - Change existing implementation to match specification? Define a new FIND? o Action - Table until other FIND issues (i.e. namespace semantics) are resolved. Issue 3: BBOARDS are 2nd class citizens (cannot CREATE, APPEND, etc.) o Discussion - What do the BBOARDs really do? * Define a different namespace +Global, rather than per-user * Imply Read-Only (of course mailboxes *may* be Read-Only) o Action - Action deferred until the namespace semantics discussion. If that leads to a more general way to specify a namespace, then the BBOARD construct becomes superfluous. Issue 4: 8Bit o Discussion - Should IMAP have a way to transmit 8 bit characters? Yes, if the character set is discoverable, but No for the raw 822 text case. - Should IMAP allow unencoded binary transfer? No, due to robustness concerns about mixing binary transfer in an ascii-oriented protocol. Better to wait on this one, especially in view of the null problem described below. The only real downside to encoding before transfer is the cost of transmitting extra characters due to the encoding overhead. - Should IMAP allow nulls in strings without Base64 encoding? No, because so much software is based on the ``null-terminated-string'' model, and tends to break when presented with a null. But clients should be encouraged to be ``null tolerant'' in any case. - Should IMAP allow 8bit characters in mailbox and flag names? No, not yet. Let the dust settle on other efforts to extend Internet protocols to 8 bit rather than claiming to have the answer. o Action - Revise page 34 to clarify intent per above discussion. Issue 5: ``Optionals'' in the Specification o Discussion - Having ``OPTIONALS'' tends to perpetuate disparity across the set of IMAP servers. We want to encourage all new servers (and clients) to fully implement the new specification. Hence make all the primitives mandatory, but provide information on backward-compatibility issues. Also avoid using term ``COMPATIBLE'' as this will be an invitation for sub-standard implementations to claim IMAP2bis compatibility. - Should primitives such as SUBSCRIBE really be mandatory? Yes, better to have a degenerate implementation that always returns ``NO'' than to have a truly ``optional forever'' command. (Note: a full implementation of SUBSCRIBE may also have occasion to return ``NO''.) o Action - Make all IMAP2bis primitives mandatory and create an appendix documenting backward compatibility issues. Issue 6: Versioning or Command Discovery o Discussion - There are existing clients and servers with different capabilities. There will be new commands added over time. Should IMAP have a way to discover what commands exist before they are executed ``for real''? Yes, but... - Alternatives * Version string... deprecated because advertised version may not really match what is implemented in the software. It is an assertion of compliance to a certain level, but history has shown many deviations. * Discovery command... still some sentiment for this, but not useful for clients/servers that predate the Discovery command. (Absence of Discovery command does not mean that other commands, e.g. CREATE, are missing.) * Define non-destructive, reversible behavior for all new primitives, so that a client can test for their existence before actually using them ``in action.'' Example: CREATE INBOX should return either BAD or NO, depending on whether the server is old (does not know anything about CREATE) or new (wherein that particular operation is an error.) o Action - Attempt to define safe ``probe'' operations for CREATE, DELETE, RENAME, APPEND, etc. Issue 7: IMAP Command Interruption o Discussion - Would be nice, but assumes server will be listening while sending. - Hard to do with TCP and Standard I/O-based software. - Therefore, widespread implementation is unlikely. o Action - None. Issue 8: Returning Size Information So Client Can Present ``Progress Bars'' o Discussion - Highly desirable. - Extremely hard to implement efficiently, to the point that it is unlikely that a client could depend upon having the information. o Action - Defer until someone proposes a sufficiently efficient solution to expect widespread implementation. Issue 9: Preserving flags/date on COPY, setting on APPEND o Discussion - Should APPEND provide a way to set flags? Yes. - Should COPY preserve flags? Yes (if access controls allow). o Action - Change COPY definition to include SHOULD preserve flags. - Change APPEND definition to add *optional* argument for flags. Issue 10: International error strings o Discussion - Should the IMAP specification define a way for the client to tell the server what language to use for messages? No, because... - Internationalization is non-trivial. - Server needs to be able to generate implementation-specific messages, in whatever language it chooses. - Maybe IMSP can tell the server what language the client likes? - Best we can do is to make it easy for a client to localize/modify messages from the server if it does not like the default language... see error codes! o Action - None. (See error codes.) Issue 11: Error codes o Discussion - There will always be more error cases than can be specified in the document. - It would be nice if major categories were presented so that the client could easily localize or make program flow decisions (e.g. TRYCREATE). - You do not need a numeric code to achieve the above. o Action - An effort will be made to identify some common error cases and define a short descriptor to go at the beginning of all messages in the category, similar to [PARSE] and [TRYCREATE]. Unlike the rest of the error string, these would become required constant parts of an error string. Issue 12: TRYCREATE o Discussion - Should TRYCREATE be part of solicited ``NO'' response on COPY and APPEND? (Right now it is an *unsolicited* NO response) Yes, because it is providing information tied specifically to the COPY or APPEND operation. o Action - Change specification accordingly. Issue 13: ISTRING/RFC 1342 SEARCH Argument o Discussion - Multiple character set searches are a can of worms. o Action - John and Mark to take Peter's suggestions and come up with revised text. Issue 14: UID Operations for Disconnected Operation o Discussion - Proposed UID operation definitions appear to be satisfactory, for achieving the disconnected operation goal, but BEFORE does not add much value... o Action - Drop BEFORE operation as superfluous. Issue 15: UID Uniqueness o Discussion - Regarding the desire to have UIDs be globally unique: there is a conflict between the global uniqueness goal and the goal of having UIDs within a single mailbox always be ascending. Global uniqueness is desirable if it allows detecting that a message saved in different or multiple mailboxes is identical to the one in the client cache. However, it is not---in general---possible to take an arbitrary message with a globally unique ID, and *append* it to a different mailbox while preserving the ``UIDs must be in ascending order with a mailbox'' requirement. (We ruled out the possibility of inserting a message in the middle of a mailbox to preserve UID ascendancy as this would violate user expectations, especially for APPEND!) - Conclude: global uniqueness is desirable, but cannot be required. (Nor should it be forbidden in the specification.) o Action - Define UIDs as unique within a folder, but not necessarily across folders. - Note that globally unique UID will conflict with ``UIDs are always ascending in a mailbox,'' assuming we always append/copy rather than insert a message in the middle of a mailbox. Issue 16: RECENT Description o Discussion - Current definition is incomplete/inconsistent o Action - Make wording changes on page 15 based on Peter's suggestions (avoid word ``new'' in RECENT defn; specify read-only case). Issue 17: New SEARCH o Discussion - Understood to be needed - A starting point for a definition: * Be able to search on Composed DATE * Be able to search on Received DATE * Be able to search on MessageID * Be able to search on general header text * Not operator * Deprecate ``UN...'' constructs * OR operator * Operator grouping * If not regular expressions, at least wild cards in strings * Better international searching o Action - Solicit IMAP developers to flesh out a detailed proposal for new SEARCH. Issue 18: Mailbox Reorder o Discussion - Legitimate reasons to want to (permanently) reorder a mailbox, but hard to anticipate all possible sort requirements... - COPY provides a moderately efficient way for client to create a new mailbox in the order desired by the client. - For now, best to leave as a client issue. o Action - None now; revisit if support for sorted/filtered views is ever added. Issue 19: Sorted/Filtered Views o Discussion - A goal of some clients is to present to the user (possibly multiple) message selections (subsets) in an order other than the natural order of the mailbox. - A perennial question is how much support should the protocol and server provide to the client for this type of situation. - This is (as is often the case) a performance tradeoff. Client-based sorts, even if based only on header information, can be quite time-consuming on current desktop hardware. - A downside of trying to do this on the server is that the protocol must anticipate all the potentially interesting sort and selection criteria. - An open question is whether increasing desktop computer performance, relative to per-user server performance, will mitigate the need for server-based processing; however, server-based search/selection is expected to remain important, especially when connecting via low-speed lines. o Action - Defer for further study. Issue 20: Message Annotation o Discussion - It would be desirable to allow messages to be annotated, perhaps with keywords, perhaps with additions to the text. - It has been proposed that it be possible to rewrite a message in a mailbox (with additional flags or annotations) without having to delete the original version and append the new version to the end of the mailbox, as the change of mailbox order is undesirable. Unfortunately, this approach to annotation is at odds with disconnected operation, wherein it is expected that the client cache copy of a message matches the original. If the content of the original changes, so should the UID, but then it cannot stay in the same place in the mailbox without violating the ``UIDs must be in ascending order'' rule. - Additional work must be done to explore alternatives for annotation, whether they be via a ``threads'' model, or external annotation by reference, or invalidation and regeneration of UIDs... o Action - Defer for further study. Issue 21: Accessing Definitive Address of Message Originator o Discussion - It is desired that client can obtain the SMTP envelope address most likely to definitively get a reply back to originator. - Some delivery agents do terrible things to the SMTP envelope addresses, e.g. when the delivery agent creates a ``From'' address. - RFC 1123 (Sec 5.2.8) requires the receiver SMTP to make the ``MAIL FROM:'' address of the SMTP envelope available via the Return-Path header line. - IMAP specification allows fetching individual header line (e.g. Return-Path) even though it is not part of the IMAP envelope structure. o Action - None. Current IMAP behavior is sufficient if host adheres to RFC 1123. Issue 22: Concern About Lack of Multi-Threading o Discussion - A concern was reported having to do with accessing multiple mailboxes/views at the same time. The concern was that this might not be possible in IMAP because IMAP does not have multiple threads on a single connection. - It is believed that this concern is a misunderstanding. While an IMAP stream does map to a single TCP connection, it is acceptable and often desirable to have multiple streams open concurrently. Existing IMAP clients routinely allow access to multiple mailboxes concurrently. o Action - None needed. Issue 23: Namespace Semantics and Hierarchy Support o Discussion - This one's long... a separate document will be forthcoming. o Action - Develop a detailed plan for how a context-free name scheme could adequately support hierarchy. - Define hierarchy primitives desired in c-client API. - Compare/assess tradeoffs between implementing hierarchy in the client vs. adding protocol support for it. Summary of Items From 3/93 BOF o Support for disconnected operation o Off-line sorting of mailbox o Background server searching and sorting o Shared mailbox per-user state (like a .newsrc, but for mailboxes) o Function to determine where to submit messages o Storage/retrieval of MUA configuration data o Minimal non-plaintext authentication o Minimal confidentiality (XOR with shared secret) o Test assertion that PEM does not affect IMAP o Remote printing (from IMAP server's copy of message) o Improved searching Summary of Items From 7/93 Working Group Meeting o Disconnected operation support, a la DMSP, continues to be widely desired. o There is considerable interest in using IMAP to access message archives. o Several people asked about extensions to support binary message part access, without Base64 or QP encoding: - Possible? - Impact on s-expression model? - Can unencoded binary attachments be transferred without charset concerns? o The question of signalling when large blocks of data are being transferred: - Congestion of pipe; need to have multiple channels or out-of-band signals o Can we have an IMAP server capabilities command, a la new SMTP? o Be sure to look at URL/I work before settling on unique message ID scheme. o Is IMAP a distribution list alternative: shared but limited access mailbox? o Can IMAP ``integrate'' two mailboxes (remote mail archive plus local subset)? o Should IMAP become ``Interactive Message Access Protocol''? Summary of Items From 6/92 CMU-UW Meeting o Clearly needed IMAP futures: - Fetch partial text - Add create, delete, rename folder - Clarify find folder semantics - Put/append - Subscribe/unsubscribe o Functions needed with multiple IMAP servers: - Directory service o Desired, but deferred: - Search multiple mailboxes on multiple servers - Put/replace - Extend IMAP to subsume DMSP (Disconnected operation) - Add IMAPd driver to access arbitrary files o Possibilities outside the scope of remote mailbox access: - Change password - Get disk usage - Get disk quota - Set disk quota - Change protections on folder - Message posting facility (Send via other than SMTP?) o Believed to be upward-compatible changes: - Disconnected operation - Internationalization of error strings (add numbers while we're at it?) - Internationalization of search strings - Server generated browser lines? o Extensions where old functions must be retained for compatibility: - More powerful search command - New envelope fetch with additional headers o Possibly incompatible changes: - 8-bit operation - Using unique IDs as fundamental key instead of sequence number - Rename fetch operations for greater clarity - Return additional fields after a mail_find(), e.g. count, unseen count o Low bandwidth issues (outside the scope of Remote Mailbox Manipulation): - Printing messages from server - Forwarding messages from server Part 1: Background/Overview o Goal Our System (servers, clients, protocol) must support hierarchy and collections in a consistent way across different clients (and ideally across different servers as well). o Approaches 1. Client gets to pick its own hierarchy model, treating server namespace as ``flat''---that is, it only sends and receives context-free (?) names, and any notion of hierarchy is ``in the eye of the client.'' The server must figure out how to accommodate the client's view if/when it conflicts with server's natural hierarchy. 2. Server has a natural hierarchy mechanism; client discovers and exploits the server's natural hierarchy. 3. Protocol defines a lowest-common-denominator hierarchy model that is believed to be useful for most clients and implementable on most servers. Options 1 and 2 are not mutually exclusive. That is, from a protocol perspective, one can imagine a design that would allow either a client having its own view of hierarchy, interacting with the server only via flat (context-free) names, while also allowing other clients to discover and use the servers' natural hierarchy. However, consistent behavior across clients remains an important goal. o Issues 1. Defining scope of name set to be returned by FIND. It is essential that the client be able to bound the scope of a FIND search, both on the ``prefix'' and ``suffix'' sides of the search string. (Note: c-client FIND does not implement the IMAP specification because it does not follow directories at the moment. This needs to be fixed for clients implementing Approach #1, but those implementing Approach #2 will then need a new mechanism to bound the search to a single directory.) 2. Defining what server will do with ``directory'' names found within the specified scope. A non-terminal name must be distinguishable from a terminal name. Alternatives for FIND include: (a) Return an attribute for non-terminal names designating them as such. (b) Ignore non-terminal names; define new primitive to find them. (c) Return non-terminal names with the hierarchy delimiter included. (d) Server flattens namespace and traverse directories, returning only context-free names. (This supports only approach #1.) 3. There is a distinction between hierarchy created by the user and hierarchy that is pre-existing (designed by someone other than the user, as in an anonymous archive or netnews hierarchy). 4. A mechanism is needed for common name semantics across clients. We all agree that it's really important that different clients can see a common view of the same namespace. If namespace semantics are not an inherent characteristic of the protocol, nor is there a way to discover it via protocol primitives, then there must be an external agreement on how to describe hierarchy, and a mechanism for clients to share hierarchy definitions. IMSP has been suggested as one way various clients could access a common hierarchy definition. 5. Clients should not have any built-in knowledge of server file system semantics. They should be able to discover what they need to know from the server or from client config information. 6. If the conversation with the client is via context-free names, a client needs to be able to easily recombine locally maintained context information, e.g. prefixes, with FIND responses in order to generate arguments for SELECT and other primitives. There was not much support for building a ``lowest common denominator'' model of hierarchy into the protocol, so the discussion centered on approaches #1 and #2. Number 1, where hierarchy is in the ``eye of the client'' came to known as the ``flatlander'' view, since names going to and from the server are always flat (context-free) names. Approach #2, wherein the client discovers and uses the natural hierarchy of the server is referred to below as ``server exported hierarchy.'' As you will see, neither approach is without problems. Part 2: Summary of the Two Leading Alternatives o Two models: - Flatlander - Server Exported Hierarchy o Flatlander - Server flattens any internal hierarchy in FIND. Current c-client bezerkely behavior is a bug--it needs to recurse down directories. - CREATE needs to automatically create any necessary internal directories. The client does not know the internal structure of the server and cannot be expected to create internal directories. The directory structure may limit the ability to create certain named objects because of internal hierarchical limitations (e.g., a unix based server which maps . to / may not allow .. because // can not be maped to the unix filesystem.) or for other administrative reasons. - Need a facility for clients to limit results of FIND results to substring matches based on hierarchical delimiters. Proposal: extension to FIND to allow the client to specify a set of 'stop' characters: a FIND ALL.MAILBOXES comp.* ./ MAILBOX comp.ai MAILBOX comp.lang. MAILBOX comp.baz/ a OK Done comp.ai is a folder. There are folders which start with ``comp.lang.'' and ``comp.baz/'' but a subsequent FIND is necessary in order to find out what they are. The non-terminals can be identified as such by the fact that they are both longer than the prefix and end with a stop character. They could alterately be identified by using either a new unsolicited reply for non-terminals or by IMSP-style attributes. - The set of stop characters is a client configuration problem. A client can start with a ``default'' set (probably ./\) which is modified by a configuration file or IMSP option. (John Myers view: Netnews convention will eventually just win) TEG Comment: The default set of stop characters is a problem. On a single IMAP server, it is possible to have multiple hierarchies using different stop characters, e.g. netnews with a dot, and a Unix filesystem with a slash. There may be cases where having multiple stop characters makes sense, but I certainly do not want all the dots in my Unix file names to be interpreted as hierarchy delimiters. (And the existence of zillions of files with dots in their names makes me skeptical of John M's prediction that dot will win out as the one true hierarchy delimiter.) o Drawbacks: - There is no way to create empty non-terminals which persist. If a folder browser was used to create a ``directory,'' the client would have to handle that information virtually. If the user did not subsequently create a mailbox in that directory, the virtual directory would not persist across sessions. (Chris Newman proposal: add concept of ``namespace extender,'' which can be manipulated through the protocol. A client could create the placeholder ``comp.lang.'' which is neither a folder nor the initial substring of a folder, but shows up as a non-folder when doing FIND with a (possibly empty) set of stop characters.) Counterpoint: in some domains, such as netnews, the concept of an empty nonterminal does not exist. Either the concept would have to be hacked on top of the domain or users would see inconsistent behavior. Is the feature worth the cost? - An anonymous IMAP server may have a pre-defined hierarchy for its data with stop characters which are unknown to the client. One possible solution would be for anonymous IMAP servers to use one of the ``default'' stop characters for any predefined hierarchy. Another solution would be for the anonymous IMAP server to have an IMSP server which advertises the appropriate stop characters. (reluctant proposal: Add a wart to the protocol to obtain the set of separator characters.) - The ``stop character'' mechanism will view a VMS-style namespace in an unexpected way. The namespace can still be used and viewed in a hierarchical manner. - Flattening out some namespaces may take an inordinate amount of resources. e.g. FIND /* in c-client berkeley driver will thrash server. This problem exists in other domains (e.g. SEARCH BODY) and is usually solved with administrative limits on such things as depth of search, CPU use, or total number of matches. o Server exported hierarchy: - Need server determined directory concept MKDIR, RMDIR commands Drawback: in some domains, such as netnews, the concept of creating and removing directories does not exist. - NFIND returns both folders and directories. Does not recurse down directories. - Still need flatlander view Flatlander clients Folder discovery (FIND *mail* or similar) Flatlander FIND needs to return directory structure information - Client needs to know the characteristics of the namespace. Protocol has to mandate restrictions on the namespace and/or communicate to the client those characteristics which are not mandated. In order to implement a Macintosh-style folder browser or a hierarchical browser, client needs to obtain fully qualified name from directory and relative name components. In order to enter such a browser from a flatlander search or Pine-style rubber-room path, client also needs to know how to obtain the sequence of relative names to get to a fully qualified name. This suggests two commands, JOINPATH and EXPANDPATH. Drawbacks: Mandated restrictions will prohibit servers from exporting otherwise useful views. Attempting to describe characteristics which have insufficient restrictions is overly complex. This leaves the possibility of a ``convert between relative and absolute paths'' command. Examples: In the VMS namespace, the absolute path name: sys$user:[bovik.foo]bar converts to the relative name sequence: sys$user:[000000] [bovik] [.foo] bar o Drawbacks: - IMSP server has to know about the characteristics of the namespaces for its IMAP servers. o Comments: - CD/PWD is orthogonal to hierarchy, except in that they provide, as a side effect, a JOINPATH mechanism. They are not sufficient to provide a EXPANDPATH mechanism. Part 3: Elaboration of Hierarchy Support Issues for VMS Namespaces This comes from a view that the server needs to export its hierarchy to the client. If restrictions were put on the hierarchy such that assumptions could be made about the namespace and a namespace like the VMS filesystem were disallowed this would not be as bad. The basic problem in the hierarchy view of the world is how to represent a hierarchical namespace such as VMS. In VMS you have a hierarchy that looks like: for the file: sys$user:[wally.foo.bar]baz.txt;1 the relative hierarchy used by something like a cd command: (all the devices and logicals which you cannot find with a dir command) sys$user:[000000] [wally] [.foo] [.bar] baz.txt;1 And their fully qualified names for discovery: sys$user:[000000] sys$user:[wally] sys$user:[wally]foo.dir;1 sys$user:[wally.foo]bar.dir;1 sys$user:[wally.foo.bar]baz.txt;1 To use this namespace from an FTP client you need to know something about the namespace. For example, to properly represent a VMS namespace fetch knows that mail.dir;1 should be shown to the user as [.mail]. In your pine client you want to be able to jump to a ``rubber room'' in the namespace given a name in a config file. Now if you specified that you wanted to jump to: sys$user:[wally.foo.bar] A macintosh folder browser would want to fill in the bottom parts in the pop of menu denoting where in the namespace you are in. Fetch cannot do this, so shows you the places you cd'ed from and then a special ``parent directory'' marker. It also gets confused frequently, especially if I cd to the parent directory which is sys$user:[0000000]. Also to continue to support a flat namespace (which we would need very much), for something like *imap* to work you would need to be able to return hierarchy information. This would be needed to fill a hierarchy map of the entire namespace or portion of the namespace. So, given a name or pattern you need to return typed information of the name and its hierarchy, and the relative name from the previous level: find *imap* sys$user:[000000] sys$user:[000000] (/directory) sys$user:[wally] [wally] (/directory) sys$user:[wally.foo] [.foo] (/directory) sys$user:[wally.foo.bar] [.bar] (/directory) sys$user:[wally.foo.bar]imap.txt;1 imap.txt (/folder) This is the hierarchical tree for imap.txt;1. Now this does not explain it if there are multiple ways of getting there in the tree and probably is more complicated than it is being explained.