Re: [code-review] Just subscribed

Jean-Michel Hiver wrote:
>>The code is ok overall by my criteria, but since it isn't an XML parser I 
>>think you should change the name. An XML parser would have died the first 
>>time there:
> 
> That's what happens when you use one big rexeg :)

Yes, I know, which is why people don't write XML parsers as big regexes :)

> As for the XML tokenizer, I am not sure. As far as I am aware - correct
> me if I'm wrong - the XML specification states that XML *PARSERS* should
> die on not well formed XML.

It says that processors should die on non-WF XML (aka WTF XML). Yes, in an XML 
processor, the part that blows up may not be the tokenizer. However, an 
application that would use a tokenizer directly *must* blow up if the content is 
not valid, unless of course it's a parser (or a fixer, which is just a parser 
that has a fancy way of blowing up).

Ergo, unless you intend to write all your applications that rely on an XML 
tokenizer to perform all the checks they'd need to perform, you're not using XML.

That's why people use parsers.

I'm not objecting to the possible usefulness of something that works at the 
lexical level, just to the name. XML is very strict and anal about 
interoperability, and that approach has paid. At the very least, I would add in 
huge bold fiery letters (using the =fiery POD tag) at the top THIS IS NOT AN XML 
PARSER, DO NOT USE IT TO PROCESS XML UNLESS YOU KNOW WHAT YOU ARE DOING.

I'm also afraid that the regex may not be resilient to various XML inputs. Eg 
parsing <!-- comment -- foo --> gives "<!-- comment --"," foo -->" which -- 
while the original was wrong anyway, doesn't make much sense, and that without 
telling the user about it.

I'm also scared of what it'll do when parsing some UTF-16 XML with a BOM. 
Likely, something insane.

> I would think it's a matter of layering, really... MKDoc::XML::Tokenizer
> could be used as a tool to build a proper XML parser given the right
> amount of checking. But it's not an XML parser, which is why it's called
> a tokenizer :)

Yes, but it is something that should be used with *great* care. Unless you have 
a specific requirement to work at the lexical level (eg you're writing an XML 
editor) what will happen is that you'll shoot yourself in the foot, and probably 
slaughter your first- and second-born on the way there. This should at least be 
very thoroughly explained and documented.

-- 
Robin Berjon <[email protected]>
Research Scientist, Expway      http://expway.com/
7FC0 6F5F D864 EFB8 08CE  8E74 58E6 D5DB 4889 2488