Re: [code-review] Just subscribed

> It says that processors should die on non-WF XML (aka WTF XML). Yes, in an XML 
> processor, the part that blows up may not be the tokenizer. However, an 
> application that would use a tokenizer directly *must* blow up if the content is 
> not valid, unless of course it's a parser (or a fixer, which is just a parser 
> that has a fancy way of blowing up).

Sure.


> Ergo, unless you intend to write all your applications that rely on an XML 
> tokenizer to perform all the checks they'd need to perform, you're not using XML.
> 
> That's why people use parsers.
> 
> I'm not objecting to the possible usefulness of something that works at the 
> lexical level, just to the name. XML is very strict and anal about 
> interoperability, and that approach has paid. At the very least, I would add in 
> huge bold fiery letters (using the =fiery POD tag) at the top THIS IS NOT AN XML 
> PARSER, DO NOT USE IT TO PROCESS XML UNLESS YOU KNOW WHAT YOU ARE DOING.

Sounds like a sensible suggestion. Will do.


> I'm also afraid that the regex may not be resilient to various XML inputs. Eg 
> parsing <!-- comment -- foo --> gives "<!-- comment --"," foo -->" which -- 
> while the original was wrong anyway, doesn't make much sense, and that without 
> telling the user about it.

Well, it's called a bug... Thanks for your report, I'll fix it and add a
test for it. I guess I should add a is_valid() method to the
MKDoc::XML::Token class to make sure that '<!-- comment --' is not
considered valid.


> I'm also scared of what it'll do when parsing some UTF-16 XML with a BOM. 
> Likely, something insane.

True. You'd have to encode it to UTF-8 and then encode it back in
UTF-16. I should add that in the documentation.


> Yes, but it is something that should be used with *great* care. Unless you have 
> a specific requirement to work at the lexical level (eg you're writing an XML 
> editor) what will happen is that you'll shoot yourself in the foot, and probably 
> slaughter your first- and second-born on the way there. This should at least be 
> very thoroughly explained and documented.

I will do. Thanks for your suggestions - It's great!