Re: Patch for "HTML5" support

Geoff McLane <[email protected]> Sat, 14 Jan 2012 19:48:59 +0100
Newsgroups	gmane.comp.web.html-tidy.devel
Message-ID	<1326566939.1816.4.camel@DELL02>
Hi,

Considering test 655338 we have a difference between
when initially tidying the input with a mis-placed
xml declaration, and then when tidying that output...
I shall call them Case 1 and Case 2...

In tracing through in Windows, I have sort of figured 
out WHY this happens, but unfortunately have not found 
a solution, a fix...

Case 1: out-of-order declarations
INPUT:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> 
<?xml version="1.0" encoding="iso-8859-1"?> 
<html lang=en xml:lang="en"> 

This produces an output in the correct order of -
OUTPUT:
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN">
<html lang="en">

Then when Tidy is run on this output, it produces -
Case 2:
INPUT:
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN">
<html lang="en">

OUTPUT:
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html lang="en"
      xmlns="http://www.w3.org/1999/xhtml"
      xml:lang="en">

which, I think should have been the output in 
Case 1 ;=() So, why this difference?

Hmmm, after considerable searching, I find that in -
int tidyDocCleanAndRepair( TidyDocImpl* doc ) {}

In Case 1 -
tidyDocCleanAndRepair: htmlOut=No, xmlOut=No, xhtmlOut=No
so the SI is NOT repaired in -
Bool TY_(FixDocType)( TidyDocImpl* doc ) {}
because there was no SI given originally...
And -
Bool TY_(SetXHTMLDocType)( TidyDocImpl* doc ) {}
is never called...

In Case 2 -
tidyDocCleanAndRepair: htmlOut=No, xmlOut=Yes, xhtmlOut=Yes
These were SET in -
FindGivenVersion(){} due to (VERS_XHTML & vers) 
SET TidyXmlOut/TidyXhtmlOut

So the SI is repaired in -
Bool TY_(SetXHTMLDocType)( TidyDocImpl* doc ) {}
because of -
        else if (lexer->versions & VERS_HTML40_STRICT) {
            TY_(RepairAttrValue)(doc, doctype, pub,
GetFPIFromVers(X10S));
            TY_(RepairAttrValue)(doc, doctype, sys,
GetSIFromVers(X10S));
            lexer->versionEmitted = X10S;

It seems in Case 1, where the DOCTYPE is before the XML declaration,
on reading the XML declaration, we should ALSO SET xmlOut and
xhtmlOut?

In Case 1 where the DOCTYPE is found first, the service 
'FindGivenVersion' is called, but with no VERS_XHTML set,
so TidyXmlOut/TidyXhtmlOut are NOT set.

In Case 2, where the XML declaration is first, and the 
DOCTYPE is later, when 'FindGivenVersion' is called, it
find (VERS_XHTML & vers) so both TidyXmlOut/TidyXhtmlOut 
are SET.

It is called as a result of -
  if (lexer->doctype == VERS_UNKNOWN && lexer->token && !cfgBool(doc,
TidyXmlTags))
        lexer->doctype = FindGivenVersion(doc, lexer->token);
in the big, complicated GetTokenFromStream function, when 
the lexer is in the 
  case LEX_DOCTYPE:  /* seen <!d so look for '>' munging whitespace */

So again, from some diagnostic output I have added -
In Case 1:
Parsing file ..\..\test\input\in_655338.html ...
<!D<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
FindGivenVersion: SET XmlOut=No
<?xml version="1.0" encoding="iso-8859-1"?>]
  2:  1: XmlDecl      = <=blank=>, src=FromStream
line 2 column 1 - Warning: removing whitespace preceding XML
Declaration^M^J

So in this case, when the DOCTYPE is decoded, the document is
NOT marked as xml yet, so XmlOut is NOT set...

And when it does get to the xml declaration, it does NOT 
go back and re-assess the DOCTYPE

In Case 2:
Parsing file tempout.html ...
<?x<?xml version="1.0" encoding="iso-8859-1"?>
OTEXT:[<?xml version="1.0" encoding="iso-8859-1"?>]
  1:  1: XmlDecl      = <=blank=>, src=FromStream
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN">
FindGivenVersion: SET XmlOut=Yes

Here the xml declaration is first in the file, so the document 
IS marked as xml, and when the DOCTYPE comes along, it does 
set XmlOut, which then later 'repairs' the DOCTYPE...

Will have to think about this more, but somehow in the 
Case 1, Tidy needs to re-assess the DOCTYPE node once it 
later gets the out-of-sequence xml declaration...

Maybe others will have ideas from this, or their 
own reading and discovery...

At this point I just can not see a 'simple' fix!

And in fact should it be fixed? 

This is a case of Tidy trying to deal with out-of-order 
declarations. Should Tidy be expected to go back and 
re-assess the DOCTYPE on getting the xml declaration later?

It does 'bubble' the xml declaration to be before the 
DOCTYPE in output, which was the purpose of the test...

Regards,
Geoff.



------------------------------------------------------------------------------
RSA(R) Conference 2012
Mar 27 - Feb 2
Save $400 by Jan. 27
Register now!
http://p.sf.net/sfu/rsa-sfdev2dev2