Re: Wikipedia dump file processing shoot out

I would think the issue with a multi-pass approach is simply I/O
performance. A pretty large part of the entire overhead is probably just
reading the data into memory. Effectively doing it twice (and the second
time from a large number of smaller files) is going to impose a bunch of
overhead I would think. Once you read something from disk, best to be
totally done with it right then and there.

Has anyone tried to identify WHY libxml/xslt running in perl is AT ALL
slower than C? Obviously there is SOME going back and forth between perl and
C there, but it kind of seems hideously more expensive than it should be.

On Thu, Dec 10, 2009 at 3:11 PM, Tyler Riddle <[email protected]> wrote:

> On Thu, Dec 10, 2009 at 12:05 PM, David Nicol <[email protected]>
> wrote:
> > i would attempt to do this in two passes, the first to split the data
> > into articles one per file, the second pass to process each article
> > using a different tool. The second pass could be parallelized and
> > could start as soon as the first single-article file has been created.
>
> This is the approach taken by the XML-Bare.t test; it uses perl string
> processing to rip apart the XML document and feed it as smaller chunks
> into
> XML::Bare which is a tree parser. It's damn fast as you can see in the
> benchmarks however it's not XML compliant by any means.
>
> I have considered the approach you suggest via a rather simple XML
> "parser" that understands just enough context to rip the document
> apart into chunks properly and return them as strings. Does such a thing
> exist?
>
> Tyler RIddle
>
> > --
> > intake, compression, power, exhaust, repeat.
> >
>
> Nice sig :-)
>
> --
> If you wish to make an apple pie from scratch you must first invent
> the universe. -- Carl Sagan
> _______________________________________________
> Perl-XML mailing list
> [email protected]
> To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
>



-- 
The Wise adapts himself to the world. The Fool adapts the world to himself.
Therefore, all progress depends on the Fool.

_______________________________________________
Perl-XML mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs