Wikipedia dump file processing shoot out
Tyler Riddle <[email protected]>
| Newsgroups | gmane.comp.lang.perl.xml |
|---|---|
| Message-ID | <[email protected]> |
Hello fellow XML hackers,
I've been doing research into which perl XML processing schemes are
fastest as I design my replacement for Parse::MediaWikiDump. What I've
wound up with is a nice benchmarking system and a large pile of test
cases. I'm hoping there's some XML gurus or just weirdoes like me who
like to try to make things go as fast as possible who are interested
in either creating a benchmark for their favorite XML processor or
seeing if my existing SAX handlers, etc, could be optimized more. If
anyone is interested you can get the source via SVN from
https://triddle.projecthut.com/svn/triddle/XML_Speed_Test/
This is particularly interesting for me because the English Wikipedia
dump files are currently 22 gigs of uncompressed XML. I'm looking to
see if any existing perl XML processing solution can approach the speed
of processing the dumps with C based parsers while maintaining correctness.
If not I'm considering implementing a new perl XML parsing scheme that
gets as much as possible out of perl and into C so the only time perl is
actually involved is when the end user is processing the data from
the XML document, not dealing with processing the XML document.
I'm also going to use the results of this project to write a paper on the state
of the art in perl processing for huge XML documents. Any contributor to the
project will be listed in the paper as well.
The README for the project is at the end of this email just for good measure.
Cheers,
Tyler Riddle
--
If you wish to make an apple pie from scratch you must first invent
the universe. -- Carl Sagan
__END__
ABOUT
This is a benchmark system for XML parsers against various language editions of
the Wikipedia. The benchmark is to print all the article titles and text of a
dump file specified on the command line to standard output. There are
implementations for many perl parsing modules both high and low level. There
are even implementations written in C that perform very fast.
The benchmark.pl program is used to run a series of benchmarks. It takes two
required arguments and one optional. The first required argument is a path to
a directory full of tests to execute. The second required argument is a path
to a directory full of dump files to execute the tests against. Both of these
directories will be executed according to sort() on their file names. The third
argument is a number of iterations to perform, the default being 1.
Output goes to two files: results.log and results.data - they both are the
output from Data::Dumper of an internal data structure that represents the
test report. The results.log file is written to each time all the tests
have been run against a specific file and lets you keep an eye on how long
running jobs are performing. The results.data file is the cumulative data
for all iterations and is written at the end of the entire run.
The benchmark.pl utility and all of the tests are only guaranteed to work
if executed from the root directory of this software package. The C based
parsers are in the bin/ directory and can be compiled by executing make in
that directory. The Iksemel parser is not currently functional for unknown
reasons.
THE CHALLENGE
First and foremost the most important thing to keep in mind is that the English
Wikipedia is currently 22 gigabytes of XML. You will not be able to use any
XML processing system that requires the entire document to fit into RAM.
Each benchmark must gather up the title and text for each Wikipedia article
for an arbitrary XML dump file. In the spirit of making this test approximate
a real world scenario you must collect all character data together and make it
available at one time. For instance in the perl benchmarks they actually invoke
a common method that prints the article title and text for them. In the C based
tests they simply collect all the data and print it out at once.
EXAMPLES
Doing a test run:
foodmotron:XML_Speed_Test tyler$ ./benchmark.pl test_cases data
Iterations remaining: 1
Benchmarking 20-simplewiki-20091021-pages-articles.xml
Generating md5sum: 8fa1e9de18b8da7523ebfe2dac53482a
running test_cases/MediaWiki-DumpFile-SimplePages.t
data/20-simplewiki-20091021-pages-articles.xml: 12 seconds
running test_cases/Parse-MediaWikiDump.t
data/20-simplewiki-20091021-pages-articles.xml: 66 seconds
running test_cases/XML-Bare.t
data/20-simplewiki-20091021-pages-articles.xml: 7 seconds
running test_cases/XML-LibXML-Reader.t
data/20-simplewiki-20091021-pages-articles.xml: 12 seconds
running test_cases/XML-LibXML-SAX.t
data/20-simplewiki-20091021-pages-articles.xml: 68 seconds
running test_cases/XML-Parser-ExpatNB.t
data/20-simplewiki-20091021-pages-articles.xml: 44 seconds
running test_cases/XML-Parser.t
data/20-simplewiki-20091021-pages-articles.xml: 42 seconds
running test_cases/XML-SAX-Expat.t
data/20-simplewiki-20091021-pages-articles.xml: 183 seconds
running test_cases/XML-SAX-ExpatXS.t
data/20-simplewiki-20091021-pages-articles.xml: 33 seconds
running test_cases/XML-SAX-ExpatXS_nocharjoin.t
data/20-simplewiki-20091021-pages-articles.xml: 62 seconds
running test_cases/XML-SAX-PurePerl.t
data/20-simplewiki-20091021-pages-articles.xml: 585 seconds
running test_cases/XML-Twig.t
data/20-simplewiki-20091021-pages-articles.xml: 204 seconds
running test_cases/expat.t
data/20-simplewiki-20091021-pages-articles.xml: 7 seconds
running test_cases/libxml.t
data/20-simplewiki-20091021-pages-articles.xml: 7 seconds
foodmotron:XML_Speed_Test tyler$
The report:
$VAR1 = [
{
'filename' => '20-simplewiki-20091021-pages-articles.xml',
'tests' => [
{
'runtimes' => {
'system' => '0.4',
'user' => '5.78',
'total' => '6.18'
},
'name' => 'libxml.t',
'percentage' => 100,
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
'MiB/sec' => '35.1349971055213'
},
{
'runtimes' => {
'system' => '0.37',
'user' => '6.32',
'total' => '6.69'
},
'name' => 'XML-Bare.t',
'percentage' => 108,
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
'MiB/sec' => '32.4565444113784'
},
{
'runtimes' => {
'system' => '0.4',
'user' => '6.55',
'total' => '6.95'
},
'name' => 'expat.t',
'percentage' => 112,
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
'MiB/sec' => '31.2423427499455'
},
{
'runtimes' => {
'system' => '0.83',
'user' => '10.62',
'total' => '11.45'
},
'name' => 'XML-LibXML-Reader.t',
'percentage' => 185,
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
'MiB/sec' => '18.963692760884'
},
{
'runtimes' => {
'system' => '0.42',
'user' => '11.33',
'total' => '11.75'
},
'name' => 'MediaWiki-DumpFile-SimplePages.t',
'percentage' => 190,
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
'MiB/sec' => '18.4795133712444'
},
{
'runtimes' => {
'system' => '0.55',
'user' => '32',
'total' => '32.55'
},
'name' => 'XML-SAX-ExpatXS.t',
'percentage' => 526,
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
'MiB/sec' => '6.67079207717731'
},
{
'runtimes' => {
'system' => '0.26',
'user' => '41.55',
'total' => '41.81'
},
'name' => 'XML-Parser.t',
'percentage' => 676,
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
'MiB/sec' => '5.19335762047648'
},
{
'runtimes' => {
'system' => '0.46',
'user' => '42.1',
'total' => '42.56'
},
'name' => 'XML-Parser-ExpatNB.t',
'percentage' => 688,
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
'MiB/sec' => '5.1018393353412'
},
{
'runtimes' => {
'system' => '0.53',
'user' => '60.13',
'total' => '60.66'
},
'name' => 'XML-SAX-ExpatXS_nocharjoin.t',
'percentage' => 981,
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
'MiB/sec' => '3.5795298732628'
},
{
'runtimes' => {
'system' => '0.49',
'user' => '65.33',
'total' => '65.82'
},
'name' => 'Parse-MediaWikiDump.t',
'percentage' => 1065,
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
'MiB/sec' => '3.29891039368158'
},
{
'runtimes' => {
'system' => '0.87',
'user' => '66.01',
'total' => '66.88'
},
'name' => 'XML-LibXML-SAX.t',
'percentage' => 1082,
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
'MiB/sec' => '3.24662503158076'
},
{
'runtimes' => {
'system' => '1.32',
'user' => '179.77',
'total' => '181.09'
},
'name' => 'XML-SAX-Expat.t',
'percentage' => 2930,
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
'MiB/sec' => '1.19904070965885'
},
{
'runtimes' => {
'system' => '1.95',
'user' => '201.49',
'total' => '203.44'
},
'name' => 'XML-Twig.t',
'percentage' => 3291,
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
'MiB/sec' => '1.06731361635923'
},
{
'runtimes' => {
'system' => '3.45',
'user' => '577.07',
'total' => '580.52'
},
'name' => 'XML-SAX-PurePerl.t',
'percentage' => 9393,
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
'MiB/sec' => '0.374034110990356'
}
],
'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
'size' => 227681797
}
];
One of the fastest benchmarks:
#!/usr/bin/env perl
use strict;
use warnings;
use Data::Dumper;
use XML::LibXML;
use XML::LibXML::Reader;
binmode(STDOUT, ':utf8');
binmode(STDERR, ':utf8');
$| = 1;
print '';
use Bench;
my $reader = XML::LibXML::Reader->new(location => shift(@ARGV));
my $title;
while(1) {
my $type = $reader->nodeType;
if ($type == XML_READER_TYPE_ELEMENT) {
if ($reader->name eq 'title') {
$title = get_text($reader);
} elsif ($reader->name eq 'text') {
my $text = get_text($reader);
Bench::Article($title, $text);
}
$reader->nextElement;
next;
}
last unless $reader->read;
}
sub get_text {
my ($r) = @_;
my @buffer;
my $type;
while($r->nodeType != XML_READER_TYPE_TEXT && $r->nodeType !=
XML_READER_TYPE_END_ELEMENT) {
$r->read or die "could not read";
}
while($r->nodeType != XML_READER_TYPE_END_ELEMENT) {
if ($r->nodeType == XML_READER_TYPE_TEXT) {
push(@buffer, $r->value);
}
$r->read or die "could not read";
}
return join('', @buffer);
}
__END__
TEST DATA
You can find various MediaWiki dump files via http://download.wikimedia.org/
I use the following various language Wikipedia dump files for my testing:
http://download.wikimedia.org/enwiki/20091103/enwiki-20091103-pages-articles.xml.bz2
http://download.wikimedia.org/simplewiki/20091203/simplewiki-20091203-pages-articles.xml.bz2
http://download.wikimedia.org/cvwiki/20091208/cvwiki-20091208-pages-articles.xml.bz2
TODO
* It would be nice if the C based parsers were glued to perl with XS
so they invoke the
Bench::Article method just like the perl based parsers do.
* Fix the Iksemel parser.
* One common string buffering library between all C based parsers
would be nice
but I could not get this functional. There is a lot of other code
duplication
as well.
* A C implementation of libxml's reader interface would be fun to compare
against the perl one.
AUTHOR
Test suite and initial tests created by Tyler Riddle <[email protected]>
Please send any patches to me and feel free to add yourself to the
contributors list.
CONTRIBUTORS
* "Sebastian Bober <[email protected]>" - Concept behind the
XML::Bare implementation
_______________________________________________
Perl-XML mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs