IMAP writes takes forever

Raul Dias <[email protected]> Fri, 03 Jun 2005 16:16:34 -0300
Newsgroups gmane.comp.lang.perl.modules.mail-box
Message-ID <[email protected]>
Hi,

I have not really looked at the source code to findout why, so this is
most a report of my expirience.

I have being migrating large MBoxes files to a IMAP server.

So in this expiriece, the MBox file was 1.6 GB in size and had about
16500 messages.

So the procedure was 
-open the mbox
-open the imap folder
-looped every message in the mbox
	-copyTo() the message to the imap folder.
	-delete() the message.
-close imap folder
-close the mbox folder.

In the first run the machine run out of memory after hours. It is a dual
Athlon MD with 3.5GB of RAM.

In the second run I limited the work to a max of 10000 messages than
close and reopen the mailboxes.

This kept the memory usage to a max of 79% (2.8GB).
It took 2 days of processing (at 90% CPU usage) before I decided to kill
it.

Doing a strace in the script process I saw that it did thousands of
FETCH( <TONS of UID> ) FLAGS and SEARCH to the IMAP server.
Then <TONS of UID> seems to be one UID foreach message in the IMAP
folder.  So if there is 40000 messages in the folder it will keep doing
FETCH for every UID.

As I said I have not look into the source to findout if this is really
happening into Mail::Box or Mail::IMAPClient.  But I suspect this is
happening at Mail::Transport::IMAP4.


Raul Dias