Re: stress testing Xerces

>>>>> "Jason" == Jason E Stewart <[email protected]> writes:

Jason> [email protected] (Jason E. Stewart) writes:
>> Brian Faull <[email protected]> writes:
>> 
>> Depends on what you are doing - if it is DOM related, then yes, you
>> must tell the parser to release the memory, otherwise it grows. From
>> the API docs:
>> 
>> void AbstractDOMParser::resetDocumentPool()
>> 
>> Reset the documents vector pool and release all the associated memory
>> back to the system.
>> 
>> When parsing a document using a DOM parser, all memory allocated for
>> a DOM tree is associated to the DOM document.
>> 
>> If you do multiple parse using the same DOM parser instance, then
>> multiple DOM documents will be generated and saved in a vector
>> pool. All these documents (and thus all the allocated memory) won't
>> be deleted until the parser instance is destroyed.
>> 
>> If you don't need these DOM documents anymore and don't want to
>> destroy the DOM parser instance at this moment, then you can call
>> this method to reset the document vector pool and release all the
>> allocated memory back to the system.

Jason> As a note - if you create a new parser each time, this should
Jason> *not* cause a leak:

Jason> while (1) { my $parser = XML::Xerces::XercesDOMParser->new();
Jason> $parser->parse(XML::Xerces::MemBufInputSource->new('<test/>')); }

Jason> if it does, that's a *big* problem, and I'd like to know about
Jason> it.

Hi Jason,

I've been meaning to compose an e-mail about this for a few days now,
but just haven't gotten around to it.  You might not like to hear this,
but I think there is a *big* problem.

I have a script which pulls data out of a database and formats it as
XML.  There is ~2.4Gb of XML once it is done.  The code pulls the data
out in chunks of reasonable size (~15Kb each as XML), formats each chunk
as an individual XML document, optionally validates the document against
a schema, and then prints it out.

When the XML validation is turned on, the script gradually eats memory
until it crashes.  If validation is off, the script runs fine.  I have
tried everything I can think of to get the memory to be released, but
with no success.

I am *definitely* creating a new parser every time.  Here's the sub that
does the validation:

sub validateXML {
  my $xml = shift ;

  # Just to make sure there is only one, $Parser is global but it's not used anywhere else:
  $Parser = XML::Xerces::XMLReaderFactory::createXMLReader() ;
  $Parser->setFeature("http://xml.org/sax/features/namespaces", 1) ;
  $Parser->setFeature("http://apache.org/xml/features/validation/schema", 1) ;
  $Parser->setFeature("http://apache.org/xml/features/validation/schema-full-checking", 1) ;
  $Parser->setFeature("http://apache.org/xml/features/validation-error-as-fatal", 1) ;
  $Parser->setFeature("http://xml.org/sax/features/validation", 1) ;
  $Parser->setFeature("http://apache.org/xml/features/validation/dynamic", 0) ;
  my $errorHandler = new XML::Xerces::PerlErrorHandler() ;
  $Parser->setErrorHandler($errorHandler) ;
  my $contentHandler = new XML::Xerces::PerlContentHandler() ;
  $Parser->setContentHandler($contentHandler) ;

  eval {
    $Parser->parse( XML::Xerces::MemBufInputSource->new($xml) ) ;
  } ;
  undef $Parser ; # reclaim resources??
  if ($@) {
    return (0, $@) ;
  } else {
    return (1, '') ;
  }
}

Thoughts, ideas...?

Steve
-- 
(    Stephen L. Mathias, Ph.D.                     (                    (
 )   Office of Biocomputing                         )  s m a t h i a s   )
(    University of New Mexico School of Medicine   (   @ p o b l a n o  (
 )   MSC08 4560                                     )  . h e a l t h .   )
(    1 University of New Mexico                    (   u n m . e d u    (
 )   Albuquerque, NM 87131-0001                     )                    )