Re: leak?

* Mark Overmeer ([email protected]) [040825 08:17]:

> > * Jon Wilson ([email protected]) [040824 14:33]:
> >>I've noticed a memory leak when I clone messages. Test code and test 
> >>message attached. It's probably some kind of circular reference. It 
> >>doesn't happen if I don't clone the message.

> Mark Overmeer wrote:
> > When you remove the clone() from your code, it still leaks memory.

> If the clone operation is on, the number increases by a lot!
In my home set-up the difference is minor.  That's what make hunting-down
the cause so hard.

> My experience with large codebases of object orientated perl code is 
> that the problem is usually a circular reference within some object or 
> group of objects. So when the folder object passes out of scope, not 
> everything is destroyed, because references still exist.

I have some tests which check that this works in MailBox: that DESTROY
is called on time...  I suspect that some outer-corner object (like
the parser) may be leaking (if any).

> I'm not familiar with exactly how the Mail::Box internals are arranged(I 
> only started using it yesterday). But consider something like the 
> following. If some message object contained a reference to the "next" 
> message in the mailbox. If the last message contains a reference to the 
> first one, then there is a circular reference, and Perl will not really 
> destroy the objects, even though the variables representing each object 
> pass out of scope.

The Manager points to each open folder for real, and the folders point
back "weakened".  A folder points to the messages, and the messages
"weakened" back to the folder.  The header and body are weakened back
to the message.  So: all these places are covered for sure.

But there are some other places.  What I have tried was deleting
fields in the objects, and see whether the reported "leak" reduced.
But it didn't help me.

Maybe, Data::Dumper can help: it shows multiple links to the same
internal object.  I tried it: delete all internal references in the
script, like delele $large2->{MBF_create_options}.  But it didn't
help.  Trading Parser::Perl to Parser::C doesn't help either.

> > By the way: I see no reason for creating a clone() in normal applications.
> > When you move messages from one folder to the other, MailBox will do that
> > as efficiently as possible for you.  (see $msg->move, $msg->copy, etc)
> 
> I need to open several large read-only mbox files (mail archives), 
> select certain messages from them (a user's mail) and save these to 
> another mbox (user.INBOX.i.must.not.delete.all.my.mail.sorry). The 
> number of messages is BIG so leaks are important to me.

Still: laziness may help you... although for MBOX folders, it will (by
default) always parse all headers, which takes some time.  MBOX is not
practical for huge messages.

What you can do:

  my $block = 0;
  while(1)
  {   my $folder = $msg->open(...);
      foreach my $msg ($folder->messages($block, $block+999))
      {   # process
      }
      $block+=1000;
      $folder->close;
  }

Or, sometimes (if the source folder doesn't need to be rewritten)

  foreach my $msg ($folder->messages)
  {   # process message

      # free mem
      $msg->destruct;
  }

An other way to tackle the problem is by first converting the MBOX into
MH or Maildir, and then process those folders. A conversion routine
can be found in test/Tools.pm

> "move" is clearly not appropriate in this case. The Mail::Message 
> documentation has no reference for "copy", only for "clone". Or am I 
> missing something here?

A Mail::Message object is a message without sense of location, so
you have to go for the Mail::Box::Message documentation.  There you'll
find a copyTo and a moveTo

> Actually I want to copy to an IMAP folder, not another mbox, but I'm 
> having problems with that. More news on that later maybe.

IMAP client implementation is not really well tested, as the docs
say.  Basic functionality is in, but not all that well tested.
-- 
               MarkOv

------------------------------------------------------------------------
drs Mark A.C.J. Overmeer                                MARKOV Solutions
       [email protected]                          [email protected]
http://Mark.Overmeer.net                   http://solutions.overmeer.net