Re: [code-review] MKDoc::XML (was: Just subscribed)

> Great, thanks. Another thing that would be useful is stating (in the docs) your 
> requirements for producing this module. People /will/ ask "but why an XML 
> reading module that only does tokens?" There are many many many XML modules out 
> there on CPAN, clearly stating what yours is for helps a lot. Something along 
> the lines of a "Why should I use this instead of XML parsers that have feature 
> foo/a standard interface such as SAX or DOM/error chacking/etc?" section might 
> be a good way amongst others.

I suppose at this point I have to explain myself :)

In MKDoc we use HTML tidy to make sure that all input is turned into
valid XHTML.

There are two bits of functionality which are essential to the software:

* A tag stripping functionality which removes all presentational
elements - except maybe for <br /> which is necessary in poems and mail
addresses.

* A hyperlinking functionality. For example if you have the following
hyperlink:

'A world of wonders' => http://disneyland.com/

Then every instance of 'A world of wonders' 'a World of WONDERS' etc.
becomes a hyperlink. This is used so that:

a/ Content writers don't have to remember the horrible <a href="...">
syntax. They enter hyperlinks separately.

b/ All neighbor documents (i.e. children, siblings and parent documents)
are automagically hyperlinked based on their title.

c/ Content writers avoid the 'click here' syndrome, which is an horror
terms of accessibility and information architecture.

Previously all the functionality was written using a full XML tree. And
in particular, the tagger was too _slow_. However I realized that the
tagger could be written much more efficiently by just using a tokenizer.

It also gets hairy when you have tags in the way of the things that you
want to hyperlink. For example:

Hyperlink <br /> Me!

At the moment MKDoc::XML::Tagger will tag

<a href="...">Hyperlink </a><br /><a href="..."> Me!</a>

Which as far as I can see is the only safe way of doing it.

Anyway if you wish to take a look to MKDoc::XML::Tagger please do! It
might look like a bit of a hack but it's the only half sensible way I
could do it and according to my benchmarks it does scale.


Anyway here is the justification for the tokenizer:

* I haven't found any (only parsers)
* It's written pure perl AND it's not too slow (one regex...)
* Parsers turn something like <br /> into an even 'open br' and an event
'close br'. Am I right in saying that there isn't a way of knowing that
this is a self closing tag?


> > Well, it's called a bug... Thanks for your report, I'll fix it and add a
> > test for it. I guess I should add a is_valid() method to the
> > MKDoc::XML::Token class to make sure that '<!-- comment --' is not
> > considered valid.
> 
> That, or you may want to die on things that can't even be tokenized right.

Yeah actually I'll do that. Sounds easy :)


> That's not always easy to do portably accross Perls, and users don't always know 
> which encodings they're dealing with. XML::SAX::PurePerl has code to handle 
> that, you may want to reuse it so that your users don't have to care (in a 
> perfect world, no one would have to care about character encodings :).

Sure, I'll take a look at it.