Re: XML::Twig - Memory usage problem

howard chen wrote:

> Consider the following Perl codes...
> 
> #===============================
> package TextXML;
> 
> ....
> 
> my $t = XML::Twig->new(
> 	twig_handlers => {
> 		company => sub {
> 			$self->company( @_[0], @_[1] );
> 			@_[0]->purge;
> 		},
> 	}
> );
> 
> $t->parsefile( "company.xml" );
> $t->purge;
> 
> ....
> 
> sub company {
> 	my ( $self, $t, $elt ) = @_;
> 
> 	# Some string processing done here...
> }
> 
> #===============================
> 
> My xml files was around 300MB, I found during runtime, the perl
> process can grow up to 1GB.
> I have put the 'purge' statement during xml node processing of node
> "company", so I believe I am reading the xml file
> chunk-by-chunk. So is my memory usage of 1GB normal?
> 
> The reason I ask is since my server has 8GB of memory, I am not sure
> if since I have so many free memory, the Twig/Expat will
> pre-allocate as much as possible for performance.
> 
> Any idea?

Your code looks OK to me.

Several things could prevent memory from being released:

How many company elements are there in the original file? Anything between 2 
elements will be loaded in memory, so if you have 100M of data with no company 
element then it will all be loaded, before being released, although not 
necessarily to the system. BTW, I don't know exactly what the expansion factor 
is, it depends on the data and on your architecture, but 10 seems like a 
reasonable figure. Also of course if one of the company elements is huge, you 
would get the same effect.

What system are you on? Old RedHat Linux are known to ship with a version of 
Scalar::Util that doesn't include weaken, which wreaks havoc on the memory 
mangement code. Also I have had reports of unusually high memory usage on 64 bit 
Linux, which I haven't been able to test since I don't have such a system.

Do you do anything with the XML in-between company elements? It looks like you 
don't since you just purge the tree. In that case you could replace 
twig_handlers by twig_roots. This way the code would not load anything outside
company elements, apart from the root. Otherwise you could purge more often (on 
other elements), and/or use ignore_elts to skip elements you are really not 
interested in.

Does that help?

-- 
mirod

_______________________________________________
Perl-XML mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs