Re: Wikipedia dump file processing shoot out
Tyler Riddle <[email protected]>
| Newsgroups | gmane.comp.lang.perl.xml |
|---|---|
| Message-ID | <[email protected]> |
On Thu, Dec 10, 2009 at 12:05 PM, David Nicol <[email protected]> wrote: > i would attempt to do this in two passes, the first to split the data > into articles one per file, the second pass to process each article > using a different tool. The second pass could be parallelized and > could start as soon as the first single-article file has been created. This is the approach taken by the XML-Bare.t test; it uses perl string processing to rip apart the XML document and feed it as smaller chunks into XML::Bare which is a tree parser. It's damn fast as you can see in the benchmarks however it's not XML compliant by any means. I have considered the approach you suggest via a rather simple XML "parser" that understands just enough context to rip the document apart into chunks properly and return them as strings. Does such a thing exist? Tyler RIddle > -- > intake, compression, power, exhaust, repeat. > Nice sig :-) -- If you wish to make an apple pie from scratch you must first invent the universe. -- Carl Sagan _______________________________________________ Perl-XML mailing list [email protected] To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs