Re: AUTO_INPUT_ENCODING (was: Some comments on HTML tidy)
"Reece Dunn" <[email protected]> Sun, 01 Jan 2006 11:09:02 +0000
| Newsgroups | gmane.comp.web.html-tidy.devel |
|---|---|
| Message-ID | <[email protected]> |
Bjoern Hoehrmann wrote: >* Reece Dunn wrote: > >[5] HTML tidy doesn't appear to have any auto detection of the character > >encoding being used (except for the UTF BOM). That is, any <meta> tag >that > >specifies the text encoding. Is there an easy way that I can do this? > >Well, you would have to assume some character encoding, parse the >document until you find a <meta>, if that's a different encoding go >back to step 1 assuming the specified character encoding and parse the >document again, this time without looking for <meta>. You would need >additional logic for <?xml?>-style encoding declarations. There is a >little bit of code for this, but adding it to Tidy would require some >work still. Patches very much appreciated, you might want to have a look >at my HTML::Encoding Perl module on CPAN. I have found the encoding logic enabled by the AUTO_INPUT_ENCODING flag. The approach I have taken when implementing the todo sections is to switch the transcoder at the point where the <?xml?> or <meta> tags are processed and then continue from that point using the specified encoding. With my limited testing, I have found this to work with a HTML document that doesn't have a UTF8 BOM (and so is treated as ASCII), but contains a utf-8 <meta/> encoding tag. I had problems initially, because setting the encoding to one of the ids in charsets.c doesn't work. This is because those ids don't match up with the defines for the natively supported encodings that HTML tidy can process (UTF8, BIG5, etc.). The function charsetToTidyEncoding is used to translate a charset encoding id into a HTML tidy encoding id. Once I used this, the UTF8 encoded file was processed correctly. The attached patch is what I currently have. This is not a full implementation, as there aren't any consistency checks between the various encodings and I haven't added the error reporting logic if you are using an unsupported encoding. Also, I haven't tested the TIDY_WIN32_MLANG_SUPPORT logic so this may be wrong. NOTE: I have noticed that the output from the text buffer does not always produce correct output. It appears to be occasionally outputting null characters. (In my output logic, I am using buffer.Size() to return the buffer size, not strlen( buffer.Data()); -- should I be using strlen instead?) - Reece
auto-encode.patch
(application/octet-stream, 4.7 KB)
Index: parser.c
===================================================================
RCS file: /cvsroot/tidy/tidy/src/parser.c,v
retrieving revision 1.155
diff -u -r1.155 parser.c
--- parser.c 22 Dec 2005 18:29:33 -0000 1.155
+++ parser.c 1 Jan 2006 10:55:00 -0000
@@ -21,8 +21,45 @@
#ifdef AUTO_INPUT_ENCODING
#include "charsets.h"
+
+/* Map an encoding in charsets to one of the HTML Tidy supported (native) encodings. */
+uint charsetToTidyEncoding( uint encoding )
+{
+#ifdef TIDY_WIN32_MLANG_SUPPORT
+ /* Don't do any internal transcoding as the Win32 MLang support will take care of this for us. */
+ /* todo: validate that this is actually correct. */
+ return encoding;
+#else
+ switch( encoding )
+ {
+ case 4: return ASCII;
+ // LATIN0 -- not suppored in charsets.c
+ case 153: return LATIN1;
+ case 229: return UTF8;
+ // ISO2022 -- which ISO-2022 does this map to?
+ case 54: return MACROMAN; // Is this right? (hp-roman8 == MACROMAN?)
+ case 238: return WIN1252;
+ case 58: return IBM858;
+
+#if SUPPORT_UTF16_ENCODINGS
+ case 224: return UTF16LE;
+ case 223: return UTF16BE;
+ case 222: return UTF16;
#endif
+#if SUPPORT_ASIAN_ENCODINGS
+ case 6: return BIG5;
+ case 213: return SHIFTJIS;
+#endif
+
+ /* todo: should we return encoding here if TIDY_WIN32_MLANG_SUPPORT is defined? */
+ default: return ( uint )-1; /* HTML tidy does not support this encoding. */
+ }
+#endif /* TIDY_WIN32_MLANG_SUPPORT */
+}
+
+#endif /* AUTO_INPUT_ENCODING */
+
Bool CheckNodeIntegrity(Node *node)
{
#ifndef NO_NODE_INTEGRITY_CHECK
@@ -3148,13 +3185,25 @@
if (val && end)
{
tmbstr encoding = tmbstrndup(val, end);
- uint id = GetEncodingIdFromName(encoding);
+ uint id = charsetToTidyEncoding( GetEncodingIdFromName(encoding));
- /* todo: detect mismatch with BOM/XMLDecl/declared */
- /* todo: error for unsupported encodings */
- /* todo: try to re-init transcoder */
- /* todo: change input/output encoding settings */
- /* todo: store id in StreamIn */
+ if (id != -1)
+ {
+ /* todo: detect mismatch with BOM/XMLDecl/declared */
+ /* change input/output encoding settings */
+ /* store id in StreamIn */
+ doc->docIn->encoding = id;
+ SetOptionInt(doc, TidyInCharEncoding, id);
+ /* try to re-init transcoder */
+#ifdef TIDY_WIN32_MLANG_SUPPORT
+ if (doc->docIn->encoding > WIN32MLANG)
+ Win32MLangInitInputTranscoder(doc->docIn, doc->docIn->encoding);
+#endif /* TIDY_WIN32_MLANG_SUPPORT */
+ }
+ else
+ {
+ /* todo: error for unsupported encodings */
+ }
MemFree(encoding);
}
@@ -3996,13 +4045,25 @@
AttVal* encoding = GetAttrByName(node, "encoding");
if (AttrHasValue(encoding))
{
- uint id = GetEncodingIdFromName(encoding->value);
+ uint id = charsetToTidyEncoding( GetEncodingIdFromName(encoding->value));
- /* todo: detect mismatch with BOM/XMLDecl/declared */
- /* todo: error for unsupported encodings */
- /* todo: try to re-init transcoder */
- /* todo: change input/output encoding settings */
- /* todo: store id in StreamIn */
+ if (id != -1)
+ {
+ /* todo: detect mismatch with BOM/XMLDecl/declared */
+ /* change input/output encoding settings */
+ /* store id in StreamIn */
+ doc->docIn->encoding = id;
+ SetOptionInt(doc, TidyInCharEncoding, id);
+ /* try to re-init transcoder */
+#ifdef TIDY_WIN32_MLANG_SUPPORT
+ if (doc->docIn->encoding > WIN32MLANG)
+ Win32MLangInitInputTranscoder(doc->docIn, doc->docIn->encoding);
+#endif /* TIDY_WIN32_MLANG_SUPPORT */
+ }
+ else
+ {
+ /* todo: error for unsupported encodings */
+ }
}
}
#endif /* AUTO_INPUT_ENCODING */