Re: Wikipedia dump file processing shoot out

David Nicol wrote:
> i would attempt to do this in two passes, the first to split the data
> into articles one per file, the second pass to process each article
> using a different tool. The second pass could be parallelized and
> could start as soon as the first single-article file has been created.

For the first pass you can look at xml_split. It comes with XML::Twig but
if you tell it to split at a certain level (-l option), it will only use 
XML::Parser and not build the tree in memory.

An other module that's pretty fast is XML::Rules.

-- 
mirod
_______________________________________________
Perl-XML mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs