Re: Memory usage

On Wed, Mar 17, 2004 at 05:50:21PM +0000, Daniel P. Berrange wrote:
> > 
> > In case I have forgotten to weaken one reference, somewhere, some
> > parts of the data structure may be kept alive, even after a descruct().
> > This is close to impossible to debug...
> 
> Just came across the module 'Data::Structure::Util' containing
> a useful method 'has_circular_ref' to detect circular references.

This identified a circular reference that being held by the
Mail::Box::File in the 'MBF_create_options' attribute - the
folder is holding a reference to itself. I changed the following
method to weaken any references in the passed in array:

  sub messageCreateOptions(@)
  {  
    my $self = shift;
    $self->{MBF_create_options} = [ @_ ] if @_;
    if (@_) {
      foreach (@{$self->{MBF_create_options}}) {
        if (ref($_)) {
	  weaken $_;
        }	
      }
    }
  }

> I'm going to give it a try to track down the memory problems I'm
> having. I've already discovered that they are partly related to
> charset decoding, but still processing a 2 MB message should
> never required 150 MB of memory (!), no matter what charset it is.
> So I think somewhere in the depths of DBI / Encode::compat / 
> Mail::Box there is some non-optimal memory allocation going on & its 
> not being released...

Encode::compat seems to be the culprit of the huge memory usage - it
gets castastrophically worse as the size of the string being decoded
increases. I changed from calling

   my $decoded = decode($charset, $part->body);

to 
   
   my $decoded = join('', map { decode($charset, $_) } $part->lines);

And its now behaving reasonably.

Dan.
-- 
|=-               http://www.berrange.com/~dan/gpgkey.txt             -=|
|=-   [email protected]  -  Daniel Berrange  -  [email protected]    -=|

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFAWJ6/vTAeyOKLMk4RAqzqAJ9V3SBmonKLPM8VF8wqRsSkemIRggCePs1N
SwGpAMsE6KaiRnT11oeV/wU=
=Pj1P
-----END PGP SIGNATURE-----