Re: mailbox format(s)

Martin Duerst wrote:

> I quite strongly agree with Iljitsch on this one.
> Having a well-defined format that can be used for files
> is very helpful. The biggest advantage would be that it
> makes it easier to switch MUAs.

If the mailbox is accessed via a standard network protocol such as
POP or IMAP, it is trivially easy to switch MUAs. It also matters
little (modulo firewalls, etc.) whether the MUA is running on the
same host as the POP/IMAP server or on a different host. Some
(most?) IMAP servers also provide POP protocol access, so it may
even be possible to switch access protocols painlessly.

There are many problems, though. MUAs, especially those not using
IMAP, tend to store their own copies of the user's mailbox in
MUA-specific places. That's a separate issue from the format, which
also tends to differ among MUAs.  A single format might not scale
well; what works for an organization with plenty of resources
might not work at all for a guy with a PDA or cell phone (and
vice versa).

> The format could come in single-message and multiple-message (like
> current mbox) variants. The later should just be a concatenation
> of the former, or otherwise a very trivial transform.

Been there, done that. Never again.  A flat file just doesn't
work well with even a modest number of messages.  I've used
POP and IMAP servers that used such a format for a mailbox, and
they are intolerably slow.  Conversely, Cyrus IMAP stores one
message per file, with a database for metadata (access lists,
etc.), and it's quite fast.  A side benefit is that (with
suitable access rights) I can do things with individual messages
without having to extract them from a pile of others in a single
file.

> And in general, file sizes shouldn't be that much of a
> problem.

Please reread RFC 1958 section 3.1 and reconsider mobile phones
and PDAs.

>> ability to control access,
>>    ability to lock portions of a file for update, etc.,
> 
> 
> sharing a format doesn't necessarily mean having access
> at the same time. Being able to do that may lead to
> very interesting new applications, but it's not that
> essential.

If one is using a single, multi-message mbox-like file and an
MTA needs to write (to append a new message) while an MUA is
reading/writing (presenting messages for the user, deleting old
messages, moving some to/from other mailboxes, etc.), there had
better be some provision for locking. Unless you want to forbid
users from leaving an MUA open for extended periods.

>> not to mention
>>    record-oriented systems (e.g. Bitnet),
> 
> 
> On the way out, not worth caring about too much
> (as Iljitsch said, nobody should be forced to use
> a format).

If it won't work for some systems (remember that bit about
heterogeneity), what is the point of having a standard?

>> 2. metadata used by different applications (e.g. IMAP requires, and POP
>>    supports a UID per message)
> 
> 
> Well, the new format should be designed so that these can be
> added, and where possible shared.

Application-specific metadata should be handled separately from
the message itself.  One of the user-visible goals is (paraphrasing)
that messages shouldn't be polluted with the life history of every
application that ever touched the message, leading to extraneous
cruft bloating the message. Generic metadata related to the message
(sender, security/privacy data, etc.) of course should be part of
the message format.

> Or designing the message format so that it can be trivially used for
> file storage (currently SMTP uses a . to terminate DATA, and mbox
> uses From to start a message; just removing this difference would
> bring us a long way; the problem is essentially the same).

I've already mentioned problems with mbox format.  In my opinion
(based in part on studying the problem, and in part on experience),
the least problematic way to store a single message is in RFC 822
[2822] format (with byte stuffing removed). Any transformations
would have to be reversible in the case of a queued message being
stored as part of an MTA's store-and-forward processing, and even
such apparently innocuous transformations as converting line endings
are not so simple when one examines the details (consider a binary
MIME part containing 0x0D and 0x0A octets in various sequences).

>> Applications running across a network can access mailboxes via
>> established network protocols for accessing or transferring messages
>> and mailboxes, such as POP and IMAP.
> 
> 
> There are many other things to email, such as procmail,...

Yes, and there are lightweight protocols such as LMTP...