Re: [Fwd: [Bratislava-pm] XML::LibXML performance]

Hi,

Tim is right. This is also the reason why SAX2 parsing is slow in 
Perl. As for XPath performance, provided that you do not modify the 
document significantly during your processing, you may try calling 

$doc->indexElement;

before you start XPath queries on your document.

This simple call can boost XPath performance dramatically. What it 
does is simple: it goes through the whole document and on each 
element node it stores its depth-first order. XPath requires the 
resulting nodesets to be sorted in the document order and this 
information makes the comparison of relative document-order 
position of two element nodes O(1), whereas without it it may be 
significantly more expensive (up to about O(log n)).

Another potential for improving XPath performance seemed to me to be 
in letting the user pre-compile static XPath expressions and reuse 
them in the pre-compiled form. The API, soon to appear in the SVN, 
looks like:

my $comp_xpath = XML::LibXML::XPathExpression->new($xpath);

$node->findnodes($comp_xpath); # same as but potentially faster than
$node->findnodes($xpath);      # when repeated many times

(similarly for XML::LibXML::XPathContext)

But for simple XPath expressions the performance benefit does not 
seem to be that significant. Here are some benchmarks:

I have a XML file with a structured dictionary, 14MB large, 220K 
elements. I have three XPath expressions that look like this 

  1: //word[@POS='V']
  2: .//frame[element]
  3: count(element[@functor='ACT'])

and benchmark this:

  for my $node ($doc->findnodes($xpath1)) {
    for my $node ($node->findnodes($xpath2)) {
       $node->findnodes($xpath3)
    }
  }

First with with $xpath* being compiled XPath expression objects, 
second with them being strings. The results are:

Without calling indexElement:

Benchmark: timing 5 iterations of compiled, non_compiled...
  compiled: 23 wallclock secs (22.70 usr +  0.02 sys = 22.72 CPU) @  
0.22/s (n=5)
non_compiled: 24 wallclock secs (22.85 usr +  0.04 sys = 22.89 CPU) 
@  0.22/s (n=5)

And now calling indexElement beforehand:

Benchmark: timing 5 iterations of compiled, non_compiled...
  compiled:  2 wallclock secs ( 1.93 usr +  0.00 sys =  1.93 CPU) @  
2.59/s (n=5)
non_compiled:  2 wallclock secs ( 2.08 usr +  0.00 sys =  2.08 CPU) 
@  2.40/s (n=5)

So you see, the greatest performance boost is achieved by indexing 
the elements and only after that there is a small but noticable 
benefit of pre-compiling the XPath expressions.

-- Petr

On st 10. září 2008, Tim Brody wrote:
> Hi,
>
> The XS <-> Perl interface is expensive.
>
> #!/usr/bin/perl
>
> use XML::LibXML;
> use Benchmark;
>
> $e = XML::LibXML::Element->new( "foo" );
>
> timethese( 1000000, {
>     'ref' => sub { if( ref($e) eq "XML::LibXML::Element" ) { $i++
> } }, 'nodetype' => sub { if( $e->nodeType == XML_ELEMENT_NODE ) {
> $j++ } }, } );
>
> Using "ref" is about 4x faster.
>
> All the best,
> Tim.
>
> Michael Kröll wrote:
> > Interesting findings on XML::LibXML performance - English text
> > starting after the Slovak version.
> >
> > Attachments at
> > http://mail.pm.org/pipermail/bratislava-pm/2008-September/00010
> >7.html
> >
> > --michael
> >
> > -------- Original Message --------
> > Subject: [Bratislava-pm] XML::LibXML performance
> > Date: Tue, 09 Sep 2008 23:41:07 +0200
> > From: Emmanuel Rodriguez <[email protected]>
> > To: [email protected]
> >
> > Ahojte,
> >
> > Jozef mi povedal cez víkend, že mal Perl kód ktorý používal
> > XML::LibXML a bol veľmi pomalý. Ak si dobre pamätám, prvá
> > verzia potrebovala asi 20 sekúnd a potom Jozef urobil druhú
> > verziu, ktorá potrebovala 1.5 sekundy. Prvá verzia používala
> > veľa XPath tak nebola taká rýchla, druhá verzia používala XPath
> > len jeden krát a bola rýchlejšia a potom urobila všetko cez
> > normálny DOM. Keď je XML jednoduché, lepšie je použiť DOM ako
> > XPath aj keď kód bude trochu dlhší ale program bude rýchlejší.
> >
> > Bol som veľmi zvedavý lebo používam veľa XML::LibXML a chcel by
> > som vidieť v čom bol problém. Pracoval som trochu s Jozefovým
> > kódom a teraz máme verziu, ktorá potrebuje len pol sekundy, ale
> > to je tiež pomalé podľa mňa.
> >
> > Súbor je asi 474K a má 17743 elementy, čo nie je veľa ale čítať
> > tento súbor nie je v Perle veľmi rýchle. Meral som rýchlosti a
> > zistil som, že chodiť cez celý strom v C je veľmi rýchle a
> > potrebuje len 0.030 sekundy, ale v Perle to isté potrebuje
> > 0.450 sekundy, a to je 15 krát pomalšie!
> >
> > V C chodiť cez DOM je veľmi ľahké a rýchle lebo všetko je cez
> > "pointers" napríklad:
> > node = node->next;
> > if (node->type == XML_ELEMENT_NODE)
> > process_element(node->children);
> >
> > Ale v Perle to isté nemôže byt cez "pointers" a tento kód musí
> > používať funkcie:
> > $node = $node->nextSibling();
> > if ($node->nodeType == XML_ELEMENT_NODE)
> > process_element($node->firstChild());
> >
> > Neviem presne prečo kód je taký pomalý v Perle, myslím že to je
> > lebo máme veľa elementy a každý DOM operácia potrebuje jeden
> > funkcia v Perl. Možno XS je drahý? Pozeral som XS kód a tieto
> > tri funkcie sú veľmi krátke:
> >
> > SV*
> > nextSibling( self )
> > xmlNodePtr self
> > ALIAS:
> > getNextSibling = 1
> > CODE:
> > RETVAL = PmmNodeToSv( self->next,
> > PmmOWNERPO(PmmPROXYNODE(self)) );
> > OUTPUT:
> > RETVAL
> >
> > int
> > nodeType( self )
> > xmlNodePtr self
> > ALIAS:
> > XML::LibXML::Node::getType = 1
> > CODE:
> > RETVAL = self->type;
> > OUTPUT:
> > RETVAL
> >
> > SV*
> > firstChild( self )
> > xmlNodePtr self
> > ALIAS:
> > getFirstChild = 1
> > CODE:
> > RETVAL = PmmNodeToSv( self->children,
> > PmmOWNERPO( PmmPROXYNODE(self) ) );
> > OUTPUT:
> > RETVAL
> >
> > ----
> > Hi,
> >
> > This weekend Jozef told that he had a Perl program that was
> > using XML::LibXML which as very slow. If I recall well, the
> > first version of his program needed in average 20 seconds.
> > Latter Jozef made a second version which needed only 1.5
> > seconds. His first version was using exhaustively XPath while
> > the second version used XPath only once and replaced the other
> > calls by simple DOM references. Xpath is very nice when dealing
> > with complex documents or when the queries are very narrow, but
> > when the XML is simple, the best is to revert to DOM, sure the
> > code will be slightly longer but the gain in speed will be
> > considerable.
> >
> > Never the less, I was puzzled by the problem that plagued
> > Jozef's code, specially because I'm using a lot XML::LibXML and
> > I wanted to know that was the source of the problem. I managed
> > to work a little bit with Jozef's sample code and I managed to
> > remove a second from the execution time. But passing from 1.5
> > seconds to 0.5 it's not too convincing, the program is still
> > slow.
> >
> > It's interesting to notice that the input file used has 474K
> > and contains 17743 elements. That's not a lot, but still
> > parsing such an input file in Perle is very slow. I benchmarked
> > the program and I found out that walking through the DOM tree
> > in C is very fast and can be done in 0.030 seconds, but in Perl
> > the same operation needs 0.450 seconds, that's 15 times slower!
> >
> > I think that the reason of the speed difference is due to the
> > way that the DOM tree is walked. In C crossing the DOM tree is
> > not expensive and is very fast as everything is done through
> > pointers, for instance: node = node->next;
> > if (node->type == XML_ELEMENT_NODE)
> > process_element(node->children);
> >
> > But in Perl this same operation can't be done through pointers
> > a the same program needs to use functions instead:
> > $node = $node->nextSibling();
> > if ($node->nodeType == XML_ELEMENT_NODE)
> > process_element($node->firstChild());
> >
> > I'm not sure why the same code in Perl is so slow. I think that
> > the problem is not due to the size of the input file but to the
> > numbers of elements in the XML document. Probably that the
> > number is so high that the XS method calls are becoming quite
> > expensive. I took a look a the XS code and the functions are as
> > short as they can be:
> >
> > SV*
> > nextSibling( self )
> > xmlNodePtr self
> > ALIAS:
> > getNextSibling = 1
> > CODE:
> > RETVAL = PmmNodeToSv( self->next,
> > PmmOWNERPO(PmmPROXYNODE(self)) );
> > OUTPUT:
> > RETVAL
> >
> > int
> > nodeType( self )
> > xmlNodePtr self
> > ALIAS:
> > XML::LibXML::Node::getType = 1
> > CODE:
> > RETVAL = self->type;
> > OUTPUT:
> > RETVAL
> >
> > SV*
> > firstChild( self )
> > xmlNodePtr self
> > ALIAS:
> > getFirstChild = 1
> > CODE:
> > RETVAL = PmmNodeToSv( self->children,
> > PmmOWNERPO( PmmPROXYNODE(self) ) );
> > OUTPUT:
> > RETVAL
> >
> > For the curious I have joined the programs that I've used to
> > parse the files. These programs simply perform a walk of the
> > whole DOM tree. These benchmarks are interesting because they
> > show that scanning the whole XML file in Perl can't be done in
> > less than 0.450 seconds that's why the fastest version of
> > Jozef's parser is doomed to work in about half a second. No
> > matter the algorithm that I've used, I couldn't go under the
> > 0.500 second barrier. This is as fast as it gets with the
> > current implementation of XML::LibXML.
>
> _______________________________________________
> Perl-XML mailing list
> [email protected]
> To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs


_______________________________________________
Perl-XML mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs