Re: Memory usage

On Fri, 2004-03-12 at 11:18, Mark Overmeer wrote:
> * Daniel P. Berrange ([email protected]) [040312 12:13]:
> > On Fri, Mar 12, 2004 at 11:07:13AM +0000, Miah Gregory wrote:
> > > Whilst the script seems to work ok, it does take a long time to run, but
> > > the main problem is memory usage - with a big folder with, say, 5000
> > > messages in it, the script exhausts the memory on the machine.
> 
> > I've had a problem with memory usage too - I reduced memory consumption by 
> > calling $message->destruct after finishing dealing with each message - but 
> > I still can't process all 500MB of mail archives in one go without 
> > restarting my script.
> 
> It would be nice to know why destruct() doesn't help.  Might be a
> simple case of some final reference to be cleaned-up.  At the same
> time, it may as well be cause by Perl trashing memory or other
> memory leaks.  Various versions of Perl have memory leaks.

I've now tried the $message->destruct() method for keeping memory usage
down, and can confirm that it does help a lot - the scripts can now
complete. There's still a fair proportion of memory still in use, but
it's low enough for the moment. For example, after processing the first
1000 messages in a mailbox containing 4009, memory usage was 37296Kb,
with every message destructed straight after checking it's received
time. Memory usage seems to creep up slowly to that point, but I'm
unsure why.

However, when using MailDir's, there are some side effects.

If you call destruct on every message in the folder, after you've
finished processing it, when you close the folder, you get two problems:

1) Loads of errors (two lines per message I think):

ERROR: You cannot take the head of a destructed message
ERROR: You cannot take the body of a destructed message

This seems to be due to Message::modified calling $self->head and
$self->body on destructed messages. Any thoughts on how to release the
memory without all of these unnecessary errors? Maybe a check in that
function to see if the message was destructed, and if so, assume it
can't be modified?

2) In Maildir.pm, writeMessages, around line 333, there is a check to
see whether the folder is empty. However it seems to rely on the
contents of @messages, which is, I'm guessing, now empty. Having
satisfied the condition, it proceeds to delete the MailDir.

I was about to call 2 a bug, but in fact there is a check for a flag
MB_remove_empty. So instead, I would suggest that the default behaviour
should /not/ be to delete empty folders - it's highly confusing, and I
would guess, not obvious behaviour.

Also, some mail handling programs such as courier imap write control
information into the top level of the MailDir, causing the rmdir to
fail. It might be better to perform a scan across the whole directory
before deciding it's empty, and only deleting the structure if no files
are found.

Just my 2p, this is meant as constructive critisism, nothing more.

-- 
Miah Gregory