Re: Soliciting feedback on new high-level pull based tree parser

Hi Tyler,

first, much kudos to you for wrapping XML::CompactTree ( ::XS ) into a
user-friendly class and also for the benchmarking effort.

A note to the benchmarks: looking briefly just at two of
them:XML::LibXML::Reader and XML::CompactTree::XS, it seems to me that
the second does much more than the first one (sort of a validation,
extracting revision info etc), OTOH, the reader test (and thus the CT
as well) can probably be made faster by the use of the "fast forward"
reader methods like nextElement() (provided one does not care about
intermediate elements).

Which leads me to an idea (though I don't have the time I could spend
on realizing it): it would be nice to set up a long running "XML
parsing module context" with a set of exact simple assignments (tasks
like `extract this and that from this XML file') that the contestants
(the authors or other enthusiasts) would attempt to solve in the most
efficient way using their specific module; the criteria would be
correctness of the result, performance benchmarks, and maybe size of
the code (if not code prettiness which I don't know how to measure)?

Best,
-- Petr

2009/12/17 Tyler Riddle <[email protected]>:
> Hello XML hackers,
>
> I previously posted to this list about an XML processing shootout for
> Wikipedia and a benchmarking system I came up with to measure the
> results of various implementations of XML processing modules from
> CPAN. I embarked on that project to see if there was an existing off
> the shelf solution that could act as the heart of my reimplementation
> of Parse::MediaWikiDump - my requirements center around being
> non-blocking and fast. After much testing I've found the combination
> of XML::LibXML::Reader and XML::CompactTree::XS to be what I need.
> Notice that my requirements did not list "user friendly" and neither
> of those modules could be considered to be so but what they are is
> blazingly fast; that tradeoff is fine for me.
>
> I started prototyping an abstract wrapper around the pair of modules
> to make it more manageable. Over time it morphed into what I'm going
> to call a high-level pull based (from the point of view of the user,
> not XML parser) tree parser. What I've created so far is going to be
> perfect for MediaWiki::DumpFile (my reimplementation of
> Parse::MediaWikiDump) but it's so abstract and fast that I get the
> feeling it is bound to be useful for other people as well. That is why
> I'm soliciting your feedback: I'd like to know if this module would be
> useful for you and if not what kind of modifications/features would it
> need to be so? The working title of the module is
> XML::CompactTree::Puller but I'm not sure if that is what it should
> be.
>
> I'm going to make some bold statements: 1) It's fast - really really
> really fast and 2) It's simple - you should be able to understand how
> to use it with out documentation at first glance. Perhaps I'm wrong,
> feedback would be appreciated in this regard. :-) Attached to this
> email is a real-world example of how to use it, the implementation,
> and MediaWiki style XML all tied together in one perl file. For
> attention grabbing goodness I'll demonstrate my previous bold
> statements here in the email body.
>
> How simple is it to use? Here is the sample case which outputs some
> metadata from the MediaWiki dump document as well as all of the
> article titles and text (this is about 80% of MediaWiki::DumpFile
> already):
>
> use strict;
> use warnings;
>
> binmode(STDOUT, ':utf8');
>
> my $puller = XML::CompactTree::Puller->new(string => get_xml());
>
> $puller->config('/mediawiki' => 'element');
> $puller->config('/mediawiki/siteinfo' => 'subtree');
> $puller->config('/mediawiki/page' => 'subtree');
>
> my $version = $puller->next;
> my $siteinfo = $puller->next;
>
> print "Dump version: ", $version->attributes->{version}, "\n";
> print "Site name: ", $siteinfo->get_element('/siteinfo/sitename')->text, "\n";
> print "Content of dump file:\n\n";
>
> while(defined(my $t = $puller->next)) {
>        my ($title, $text);
>
>        $title = $t->get_element('/page/title')->text;
>        $text = $t->get_element('/page/revision/text')->text;
>
>        $text =~ s/(.*)/\t$1/g;
>
>        print "Title: $title\n";
>        print "Article:\n";
>        print $text;
>        print "\n\n";
>
> }
>
> __END__
>
> How fast is it? It's the fastest high level general purpose module
> I've been able to find so far and it's faster than the fastest SAX
> parser available. To understand the benchmark results a little context
> is in order:
>
> libxml.t is a very domain specific SAX implementation in C - it's cut
> down to the bare minimum required to get the article titles and text
> and nothing else.
>
> XML-LibXML-Reader.t, XML-SAX-ExpatXS.t, and XML-Parser.t are similarly
> cut down implementations but done in perl.
>
> XML-CompactTree-XS.t is an intermediate implementation that served
> more as a tool to learn how to use it but provides enough information
> to work as a general purpose but very low level module.
>
> Parse-MediaWikiDump.t is my existing high level general purpose module
> for dealing with MediaWiki dump files.
>
> XML-Rules.t, XML-Records.t, and XML-Twig.t are existing high level
> easy to use XML processing modules but only XML::Records provides a
> pull oriented interface suitable for my use.
>
> MediaWiki-DumpFile-Pages.t is what I've created and is what I've
> attached to this email.
>
> Here are the benchmark results:
>
> 'name' => 'libxml.t',
> 'percentage' => 100,
> 'MiB/sec' => '35.4794578614578'
>
> 'name' => 'XML-LibXML-Reader.t',
> 'percentage' => 184,
> 'MiB/sec' => '19.215423195763'
>
> 'name' => 'XML-CompactTree-XS.t',
> 'percentage' => 208,
> 'MiB/sec' => '17.0034676673549'
>
> 'name' => 'MediaWiki-DumpFile-Pages.t',
> 'percentage' => 287,
> 'MiB/sec' => '12.3231715160114'
>
> 'name' => 'XML-SAX-ExpatXS.t',
> 'percentage' => 518,
> 'MiB/sec' => '6.84534306784747'
>
> 'name' => 'XML-Parser.t',
> 'md5sum' => '8fa1e9de18b8da7523ebfe2dac53482a',
> 'MiB/sec' => '5.1018393353412'
>
> 'name' => 'Parse-MediaWikiDump.t',
> 'percentage' => 1080,
> 'MiB/sec' => '3.28294953299246'
>
> 'name' => 'XML-Rules.t',
> 'percentage' => 2044,
> 'MiB/sec' => '1.73513091027746'
>
> 'name' => 'XML-Records.t',
> 'percentage' => 2717,
> 'MiB/sec' => '1.3053642065175'
>
> 'name' => 'XML-Twig.t',
> 'percentage' => 3279,
> 'MiB/sec' => '1.08177701331268'
>
> Any and all comments and criticism (all though preferably
> constructive) is being sought. In the interest of science all of the
> benchmarks and the benchmarking system are available via SVN at
> https://triddle.projecthut.com/svn/triddle/XML_Speed_Test/ - if you
> think one of the benchmarks is under performing I'll gladly take
> patches that can speed them up.
>
> Thanks for any donated brain cycles and happy hacking!
>
> Tyler Riddle
>
> --
> If you wish to make an apple pie from scratch you must first invent
> the universe. -- Carl Sagan
>
> _______________________________________________
> Perl-XML mailing list
> [email protected]
> To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
>
>
_______________________________________________
Perl-XML mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs