Re: Future of Qmail-LDAP

On Wed, Feb 3, 2010 at 12:44 PM, Hugo Monteiro <[email protected]> wrote:
> On 02/03/2010 10:58 AM, Scott Ryan wrote:
>>
>> No capability to store message blobs compressed.
>> No ability to strip the meta data from the message thus being able to
>> provide this info to pop3 / IMAP without using disk I/O.
>> No ability to be able to link 1 attachment to multiple messages.
>> No ability to move old/read messages to slower/cheaper storage
>>
>> All this adds up to requiring colossal amounts of high speed,
>> expensive storage if you want to offer large mailboxes for millions of
>> users.
>>
>>
>
>
> I can see your point.
>
> I was just wondering if when you say "mailboxes" you do really mean
> mailboxes, or maildirs?

I mean mailboxes as a 'service' to users. I essentially mean a maildir.

>
> Also, i should note that IMHO Qmail-LDAP should focus on the MTA part. There
> are way better POP and IMAP solutions out there, like Dovecot for instance.
> If someone asked me about it, i would vote for dropping the POP and IMAP
> server stuff and concentrate on message acceptance and delivery.

I do not totally agree - see below.

>
> Now about what you think should be supported:
>
> - Compressed message blobs. That's for reading only, right? I'm not aware of
> compressed files being opened for writting on-the-fly, anywhere. If that's
> the case, i state my opinion on the above paragraph.

I mean the blobs being compressed and decompressed on the fly. The MTA
will compress message blobs and then the POP3/IMAP4 server would
decompress them reading.

>
> - Where should that metadata be available? Is reading messages stored in
> single files (maildir), to extract metadata, that much I/O expensive than to
> keep that data in a database?

The meta data could be stored in a DB. This allows you to be able to
break apart your storage. Use smaller amounts of fast high performance
storage for you DB meta info so that your POP3/IMAP/Webmail service
can provide this info quickly and use large, cheap storage for your
actual message blob. Currently maildir just updates a file - you can
imagine the performance hit when you are processing 1 Billion messages
a month...

>
> - Message/File deduplication. That i would also love to see. Theoretically
> it's actually rather simple to do. I just need to strip the attachments from
> the messages and store them somewhere with unique names, like using the
> md5sum of the file, for instance. Then your message will no longer keep the
> attachment inside, but a reference to the detached attachment. You would
> also need to maintain a reference on that attachment so you could know what
> messages currently depend on it. The other part, of removing orphaned
> attachments, would be done by the POP/IMAP server.
>
> - Ability to move old/read messages to slower/cheaper storage. That's not
> the job of an MTA. That would be the job for an IMAP server/system. I
> believe Dovecot already allows something like that.

Not the job of the MTA - agreed. But having the ability to shift your
meta data away from your mail, allows you to tell your
POP3/IMAP4/Webmail servers where that data now lives. This is done by
moving the data periodically and updating the DB accordingly. A
typical example would be that you are only interested in keeping 1
week of 'new' mail on your fast, high performance storage. Old mail
can be moved to slower, cheaper, larger storage

>
>
> Anyone got more ideas/solutions?
>
> Regards,
>
> Hugo Monteiro.
>
> --
> fct.unl.pt:~# cat .signature
>
> Hugo Monteiro
> Email    : [email protected]
> Telefone : +351 212948300 Ext.15307
> Web      : http://hmonteiro.net
>
> Divisão de Informática
> Faculdade de Ciências e Tecnologia da
>                   Universidade Nova de Lisboa
> Quinta da Torre   2829-516 Caparica   Portugal
> Telefone: +351 212948596   Fax: +351 212948548
> www.ci.fct.unl.pt             [email protected]
>
> fct.unl.pt:~# _
>
>



-- 
Scott Ryan
http://bonoboslr.wordpress.com/