Re: mailbox format(s)

On 27-feb-04, at 21:34, Bruce Lilly wrote:

>>> If the mailbox is accessed via a standard network protocol such as
>>> POP or IMAP, it is trivially easy to switch MUAs.

>> Not quite. Then the new MUA must download all the messages again from
>> the server.

> Not with IMAP, where the message store typically lives on the
> server

Well, then a new client would have to download all the messages from 
the server again, wouldn't it?

> (which could very well be the same machine -- the point
> being to delegate management of the storage to a specific
> piece of software other than the UA per se).

Ok I can see that point, but I'm not sure I agree with it. So the 
question we need to answer is whether the flexibility in changing the 
message storage format by putting this functionality in a separate 
piece of software is worth the complexity of having a special API or 
network protocol to interact with it.

>>> Cyrus IMAP stores one
>>> message per file, with a database for metadata (access lists,
>>> etc.), and it's quite fast.

>> Maybe for random access, but if you need to access all messages you're
>> bound to be slower. Also, the file system overhead makes this a pretty
>> bad idea.

> Not necessarily.

What do you mean here? Are you going to build your own file system that 
is optimized for storing large amounts of small files? Current file 
systems do a very bad job of this now that cluster sizes are typically 
larger than a single 512 byte sector.

> Consider a POP UIDL command, which lists the
> messages by UID with each message size.  With a flat file, every
> byte of every message is read as the sizes are determined and the
> IDs generated.

Yes, *today*. With a system that includes message sizes the body can be 
skipped and this operation is on the same level of efficiency as

> simple stat() system call

> The ability to use tools such as find
> and grep is very important, not only to users, but administrators
> as well -- when the pointy-haired boss says "I know I have a
> message from John Smith somewhere, but I can't find it", would
> you rather rummage around in a single 300 MB file (possibly in
> a binary format), or just run something like
> grep -i "john smith" /var/spool/imap/users/phb/inbox/*
> ?

It would be pretty stupid to store messages such that an operation is 
done once a month can be performed easily while operations that happen 
every minute suffer. If the format doesn't allow grepping then I'm sure 
someone will write a tool to do this. Besides, current mail clients 
allow searching in email as well.

>> Standards improve quality, because they usually eliminate inferior 
>> ways
>> of achieving a result. And often vendors that implement their own
>> solution also support the standard to some extent in order to be
>> compatible.

> Among standards-conforming implementations, there can be a substantial
> difference in the quality of implementation.  And some vendors
> that pay lip service to standards are often the cause of
> interoperability problems through non-standard "extensions"
> and/or broken implementations (e.g. HTML/CSS).

Anything that displays the text between the tags is a valid HTML 
implementation. If you want to force feed your formatting on the 
reader, use PDF.

Obviously lots of bad things can happen when there is a standard. 
However, none of these possibilities are a good reason to forego 
standardization.