Re: Wikipedia dump file processing shoot out

>> I'm looking to see if any existing perl XML processing solution can
>> approach the speed of processing the dumps with C based parsers while
>> maintaining correctness.
>
> Well, as I'm sure you know, XML::LibXML is just an interface to a C
> library. As is Expat. Best of both worlds, I think: scripting
> flexibility plus native performance.
>

Ah yes quite aware of this. If you look at the benchmarks in the
README you can compare
expat and libxml in C and perl, both using SAX interfaces. The
difference is quite stark.
The lowercase expat.t and libxml.c are C based implementations of the
shootout while
XML-Parser.t and XML-LibXML-SAX are the perl based wrappers. What I'm
trying to do is
approach the speeds of the pure C parsers but with perl so you still
get a simple and easy to use
API. I'm prepared to create this if needed and I've got a good idea of
how it works.

>> THE CHALLENGE
>>
>> First and foremost the most important thing to keep in mind is that
>> the English Wikipedia is currently 22 gigabytes of XML. You will not
>> be able to use any XML processing system that requires the entire
>> document to fit into RAM.
>
> Okay, this sounds as if it were just one big document. But that would be
> insane, so it's probably not. Oh wait, it probably is. 22 GB is between
> 2 ** 34 and 2 ** 35, but the max file size in NTFS is 2 ** 64, and in
> Ext3 between 2 ** 34 and 2 ** 40. So that XML junk giant could probably
> be stored on most current systems.
>
> Why anyone would want such a giant file is beyond me. A much saner way
> of distributing the data, in my view, would be to split the contents up
> in 1 - 3 MB chunks. That way, in-memory tools won't choke on the input,
> and not even over-burden the system with insane and incommensurate
> memory allocation.

The dump files are one single document but it gets even worse than the
English Wikipedia at 22 gigs in one single file - there is a dump file for the
German Wikipedia that is 1 TB uncompressed as it has every single
revision for every single article in it and it's still a single file.

The intelligence of the approach of putting that much XML into one file is
kind of beyond the point as the various language Wikipedias are trapped in
them. This is a real world situation unfortunately.

Thank you kindly for your feedback too, I'll update the README so it's
more clear.

Tyler Riddle
-- 
If you wish to make an apple pie from scratch you must first invent
the universe. -- Carl Sagan
_______________________________________________
Perl-XML mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs