Re: First draft of work published which removes the dependency on HTML::Parser from HTTP::Message

On Fri, 5 Feb 2010 22:05:27 +0100
Gisle Aas <[email protected]> wrote:

> http://dev.w3.org/html5/spec/Overview.html#determining-the-character-encoding
> specifies how to pre-scan an HTML document to sniff the charset.
> Would it not be simpler to just implement the algorithm as specified
> instead of using a generic parser.  The use of HTML::Parser to
> implement this sniffing was just me trying a shortcut since
> HTML::Parser seemed to implement a superset of these rules.

Those rules look somewhat involved to me, especially knowing that we already
have both XS and Pure Perl parsers at hand. 

Two thoughts:

1. What about using HTML::Encoding, after adapting it so it has only
conditional dependency on HTML::Parser, and only uses HTML::Parser if available. 
(It already tries several detection methods before getting to HTML::Parser):

http://search.cpan.org/~bjoern/HTML-Encoding/

A variation on this idea would for *it* to a pure Perl HTML parser
instead of skipping the HTML parsing check completely. 

2. I note this from the spec page you reference:

"This algorithm is a willful violation of the HTTP specification, which requires that the encoding be assumed to be ISO-8859-1 in the absence of a character encoding declaration to the contrary, and of RFC 2046, which requires that the encoding be assumed to be US-ASCII in the absence of a character encoding declaration to the contrary. This specification's third approach is motivated by a desire to be maximally compatible with legacy content. [HTTP] [RFC2046]"

According to this, we can skip all this encoding-detection work and still be HTTP spec compliant (although it might more user-friendly to keep trying to guess. )

    Mark


-- 
http://mark.stosberg.com/