Re: XML::Twig - Memory usage problem

Hello,

Thanks for your useful information.


On Wed, Oct 29, 2008 at 6:14 PM, mirod <[email protected]> wrote:
>
> How many company elements are there in the original file? Anything between 2
> elements will be loaded in memory

Total 373942 elements out of 300MB of data.

=====================
Basic XML structure:
=====================
<company_xml>
	<company>
		<field1></field1>
		<field2></field2>
		<field3></field3>
		<field4></field4>
		<field5></field5>
		<field6></field6>
		<field7></field7>
		<field8></field8>
		<field9></field9>
		<field10></field10>
		<field11></field11>
		<field12></field12>
		<field13></field13>
		<field14></field14>
		<field15></field15>
		<field16></field16>
		<field17></field17>
		<field18></field18>
		<field19></field19>
		<field20></field20>
	</company>
	<company>
		..
	</company>
</company_xml>


> What system are you on?

CentOS release 5.2 (Final), 64bit version, running on Quad Core x 2
SMP server, 8GB RAM


>In that case you could replace twig_handlers by twig_roots. This way the code would not load anything outside

I observed this and now replaced with twig_roots, does not help


Below is my complete program to trace memory usage:
#=============================================
#!/usr/bin/perl

use XML::Twig;

my $cnt = 0;

my $t = XML::Twig->new(
		twig_roots => {
			company => sub {

				if ($cnt % 10000 == 0 ) {
					print "Processing element: $cnt\n";
					die if ($cnt == 100000);
				}

				$cnt++;
				@_[0]->purge;
			},
		}
	);

$t->parsefile( "company.xml" );
$t->purge;
#=============================================

The program die at the 100000th element, at that time, the memory
usage using top is:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
14781 root      25   0  548m 468m 1776 R  100  5.9   3:00.10 perl


So when finish the XML parsing, the memory usage would grown to more than 1GB.



Best Regards,
Howa
_______________________________________________
Perl-XML mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs