Re: leak?

Hi Mark,

Mark Overmeer wrote:
> * Jon Wilson ([email protected]) [040824 14:33]:
> 
>>I've noticed a memory leak when I clone messages. Test code and test 
>>message attached. It's probably some kind of circular reference. It 
>>doesn't happen if I don't clone the message.
> 
> 
> When you remove the clone() from your code, it still leaks memory.

If I increase the number of times the mbox is read, but with the "clone" 
operation off, the number of allocated objects left at the end of the 
script does not increase appreciably. If the clone operation is on, the 
number increases by a lot!

Count of allocated objects at end of script:

   100 runs, clone off:   39386
   1000 runs, clone off:  39384
   10000 runs, clone off: 39384

   100 runs, clone on:    40521
   1000 runs, clone on:   43220
   10000 runs, clone on:  70220

> I am not familiar with Devel::Leak, but does it understand
> Scalar::Util::weaken() ?  I have used that a lot, but may have
> missed one or two spots.

Devel::Leak just gives a count of the number of "things" allocated by perl.

> When you open an mbox folder, it will be lazy: only the headers will
> be taken in memory.  When you clone the messages, the bodies will
> get stored in memory as well.  So: it is very normal that the
> memory usage explodes when you go through the whole folder. 

This is not what I am doing. I am opening a folder, looking at the 
messages (one in this case), and closing the folder. Repeat. Since the 
folder object passes out of scope at the end of the loop, any objects 
(and hence memory) related to it should get destroyed. But this doesnt 
seem to be happening.

> If you are willing to persue the "leak" further, than be my guest.
> There may be many reasons within Perl, Devel::Leak, Scalar::Util, or
> even Mail::Box which produce this result.

My experience with large codebases of object orientated perl code is 
that the problem is usually a circular reference within some object or 
group of objects. So when the folder object passes out of scope, not 
everything is destroyed, because references still exist.

I'm not familiar with exactly how the Mail::Box internals are arranged(I 
only started using it yesterday). But consider something like the 
following. If some message object contained a reference to the "next" 
message in the mailbox. If the last message contains a reference to the 
first one, then there is a circular reference, and Perl will not really 
destroy the objects, even though the variables representing each object 
pass out of scope.

Alternatively the problem lies in the Mail::Box::Parser::C code.

I can investigate further when I get time, but perhaps someone on this 
list knows the internals well enought to spot the problem quickly.

> By the way: I see no reason for creating a clone() in normal applications.
> When you move messages from one folder to the other, MailBox will do that
> as efficiently as possible for you.  (see $msg->move, $msg->copy, etc)

I need to open several large read-only mbox files (mail archives), 
select certain messages from them (a user's mail) and save these to 
another mbox (user.INBOX.i.must.not.delete.all.my.mail.sorry). The 
number of messages is BIG so leaks are important to me.

"move" is clearly not appropriate in this case. The Mail::Message 
documentation has no reference for "copy", only for "clone". Or am I 
missing something here?

Actually I want to copy to an IMAP folder, not another mbox, but I'm 
having problems with that. More news on that later maybe.

Thanks for the quick reply,

Jon