Re: [Imap-protocol] Efficiently handling sequence numbers
Timo Sirainen <[email protected]>
| Newsgroups | gmane.mail.imap.general |
|---|---|
| Message-ID | <[email protected]> |
On 10.11.2012, at 0.05, Brandon Long wrote: > > So, do you load the full sequence into memory on SELECT? > > Yes, first read/mmap the index snapshot into memory and then apply any expunges and other changes to it from the log. It could be optimized so that all the expunges are done in one O(n) scan of the index instead of multiple separate of memmove()s, but it hasn't been a real problem so far. I don't have the old/new index separation yet, but I think that would be a good idea to add when the mailbox size grows past maybe 100k messages (or maybe even sooner). > > > Ah, ok. If only loading the data was as simple as an mmap for us ;) At this point, I think the data only moves through 5 separate servers on 4 machines, usually in the same data center though. That, and of course we don't use UIDs as our primary message-id, so we're looking at 32 bits for the uid and 64 bits for the message-id, so 96 bits minimum per message, which means a 10M message folder is loading 120MB, probably another 30MB of overhead in protocol buffers (not really meant for millions of entries). Sure, Dovecot has also similar extra metadata, more or less depending on the mailbox format. Keep the user typically assigned to the same backend server and you need to fetch the 120MB only once (or somewhat rarely). Send the data back in smaller diffs to avoid constantly re-uploading the whole 120MB file when it changes. I've been working on an object storage optimized backend for Dovecot these last few months. I think it could work well for GMail as well ;) Anyway, like I mentioned several times already :), I think the most important optimization for huge mailboxes is the separation of old/new data. You most likely don't need to even download/read the old data normally. (Although many IMAP clients fetch 1:* flags after SELECT, but even if IMAP server had no problems with that, the IMAP client would probably be unusably slow so that's not a real problem.) _______________________________________________ Imap-protocol mailing list [email protected] http://mailman2.u.washington.edu/mailman/listinfo/imap-protocol