Re: Wikipedia dump file processing shoot out

On Thu, Dec 10, 2009 at 9:39 PM, Tod Harter <
[email protected]> wrote:

> I would think the issue with a multi-pass approach is simply I/O
> performance. A pretty large part of the entire overhead is probably just
> reading the data into memory. Effectively doing it twice (and the second
> time from a large number of smaller files) is going to impose a bunch of
> overhead I would think. Once you read something from disk, best to be
> totally done with it right then and there.
>
> Has anyone tried to identify WHY libxml/xslt running in perl is AT ALL
> slower than C? Obviously there is SOME going back and forth between perl and
> C there, but it kind of seems hideously more expensive than it should be.
>
> There's an old thread about this [1]. Basically the performance penalty
comes from the XS interface: invoking an XS method is very expensive [2].
I've encounter this when I started writing Xacobeo [3] in which I was
displaying an XML document by traversing a DOM tree with XML::LibXML in pure
Perl.

Consider that a simple XML document can have hundred of thousands of nodes
very easily. Walking such a DOM tree node by node in Perl using XS bindings
will take a lot of time. This is why you should keep the XS interaction to a
bare minimum by using clever queries that will minimize the XS calls (use
findnodes and Xpath). It's better to use functions that will return your
nodes transformed into Perl structures in one call that to iterate over each
node and to do the conversion one at a time. If the conversion of all C data
structures to Perl variable is done in C and in one XS call then the
performance penalty will be minimal.

[1] http://aspn.activestate.com/ASPN/Mail/Message/perl-xml/3688111
[2] http://code.google.com/p/xacobeo/issues/detail?id=14
[3] http://code.google.com/p/xacobeo/
-- 
Emmanuel Rodriguez

_______________________________________________
Perl-XML mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs