Re: mailbox format(s)

Iljitsch van Beijnum wrote:
> 
> On 27 feb 2004, at 0:26, Bruce Lilly wrote:

>> If the mailbox is accessed via a standard network protocol such as
>> POP or IMAP, it is trivially easy to switch MUAs.
> 
> 
> Not quite. Then the new MUA must download all the messages again from
> the server.

Not with IMAP, where the message store typically lives on the
server (which could very well be the same machine -- the point
being to delegate management of the storage to a specific
piece of software other than the UA per se).

>> A flat file just doesn't
>> work well with even a modest number of messages.
> 
> 
> It can work if you build an index and don't go around removing message
> from the middle of the file too often.
> 
>> Cyrus IMAP stores one
>> message per file, with a database for metadata (access lists,
>> etc.), and it's quite fast.
> 
> 
> Maybe for random access, but if you need to access all messages you're
> bound to be slower. Also, the file system overhead makes this a pretty
> bad idea.

Not necessarily. Consider a POP UIDL command, which lists the
messages by UID with each message size.  With a flat file, every
byte of every message is read as the sizes are determined and the
IDs generated.  I used such a system -- it became unusable with a
few hundred messages; it took several minutes to generate a UIDL
response.  Cyrus IMAP (using POP protocol) on the same hardware
with approximately 40000 messages completes the UIDL response in
a fraction of a second (generating it is simple -- each message
file is named according to the UID, and a simple stat() system
call gives the message size; there's no need to read each file).

> What I imagine is a system where messages are stored in a binary
> container format
[...]

I don't want to get too far into the details at this time, but
a binary format is only acceptable if a text message is still
recognizable as such.  The ability to use tools such as find
and grep is very important, not only to users, but administrators
as well -- when the pointy-haired boss says "I know I have a
message from John Smith somewhere, but I can't find it", would
you rather rummage around in a single 300 MB file (possibly in
a binary format), or just run something like
grep -i "john smith" /var/spool/imap/users/phb/inbox/*
?  For that matter, IMAP provides search capability with
server-side support, so it's possible to search messages
regardless of the client MUA, without having to keep
copies of the messages for each MUA.

> Standards improve quality, because they usually eliminate inferior ways
> of achieving a result. And often vendors that implement their own
> solution also support the standard to some extent in order to be
> compatible.

Among standards-conforming implementations, there can be a substantial
difference in the quality of implementation.  And some vendors
that pay lip service to standards are often the cause of
interoperability problems through non-standard "extensions"
and/or broken implementations (e.g. HTML/CSS).