Re: Wikipedia dump file processing shoot out

Tyler Riddle schrieb am 10.12.2009 um 10:38:20 (-0800):

> This is particularly interesting for me because the English Wikipedia
> dump files are currently 22 gigs of uncompressed XML.

Is that one file per article? Or one giant file with all articles lumped
together? Okay, that would break the max file size, so it's probably the
former - or maybe some intermediary format?

> I'm looking to see if any existing perl XML processing solution can
> approach the speed of processing the dumps with C based parsers while
> maintaining correctness.

Well, as I'm sure you know, XML::LibXML is just an interface to a C
library. As is Expat. Best of both worlds, I think: scripting
flexibility plus native performance.

> THE CHALLENGE
> 
> First and foremost the most important thing to keep in mind is that
> the English Wikipedia is currently 22 gigabytes of XML. You will not
> be able to use any XML processing system that requires the entire
> document to fit into RAM.

Okay, this sounds as if it were just one big document. But that would be
insane, so it's probably not. Oh wait, it probably is. 22 GB is between
2 ** 34 and 2 ** 35, but the max file size in NTFS is 2 ** 64, and in
Ext3 between 2 ** 34 and 2 ** 40. So that XML junk giant could probably
be stored on most current systems.

Why anyone would want such a giant file is beyond me. A much saner way
of distributing the data, in my view, would be to split the contents up
in 1 - 3 MB chunks. That way, in-memory tools won't choke on the input,
and not even over-burden the system with insane and incommensurate
memory allocation.

-- 
Michael Ludwig
_______________________________________________
Perl-XML mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs