Re: Wikipedia dump file processing shoot out

If you think that 22 gigs or 1 TB of XML is stupid I've had a bug
report open with Wikimedia for over 1.5 months now to get them to
publish their XML schema in the location stated in their XML document.
It exists in their SVN repo and the ticket has been assigned to
multiple people now. Apparently scp is really hard to type or
something.

Tyler

PS> Petr Pajas saved me a bunch of money on my car insurance with
XML::CompactTree::XS - it is only slightly slower than my domain
specific implementation using libxml's reader interface but it fully
unmarshalls the document. Thank you thank you thank you.


On Thu, Dec 10, 2009 at 3:50 PM, Michael Ludwig <[email protected]> wrote:
> Tyler Riddle schrieb am 10.12.2009 um 11:46:01 (-0800):
>
>> The dump files are one single document but it gets even worse than the
>> English Wikipedia at 22 gigs in one single file - there is a dump file
>> for the German Wikipedia that is 1 TB uncompressed as it has every
>> single revision for every single article in it and it's still a single
>> file.
>>
>> The intelligence of the approach of putting that much XML into one
>> file is kind of beyond the point as the various language Wikipedias
>> are trapped in them. This is a real world situation unfortunately.
>
> Here's another guy trapped in the same kind of monster XML real world
> situation:
>
> [MarkLogic Dev General] Avoiding long running transactions
> http://www.mail-archive.com/[email protected]/msg03440.html
>
> Monster XML causes pain to man and machine, even to sophisticated ones.
> Monster XML should be avoided. Its disadvantages are obvious, its
> benefits are nowhere to be seen.
>
> --
> Michael Ludwig
> _______________________________________________
> Perl-XML mailing list
> [email protected]
> To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
>



-- 
If you wish to make an apple pie from scratch you must first invent
the universe. -- Carl Sagan
_______________________________________________
Perl-XML mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs