Meaning and use of TIDY_STORE_ORIGINAL_TEXT

Hi all,

I have been trying to use this 'feature' that someone
put considerable effort into adding ... with an idea
of trying to retain more of the user's given format,
and other such output formatting things ... was this
originally ONLY done for 'diagnostics'?

First I had to 'fix' UngetChar(), and PopChar() to
also unget-store, and add-char-to-store from/to the
original text store, which they were NOT doing ...

For example, if you had <title>The title</title>

(a) Before my unget/pop fixes, the 'original' text
for title would be '<title>T' ...
(b) but when the </title> end tag is reached,
after taking '</t' from the stream, the GetToken()
would do -
StoreOriginalTextInToken(doc, lexer->token, 3);
but it had already done the unget of the 't', so
had to change this to -
StoreOriginalTextInToken(doc, lexer->token, 2);
and now it will correctly split the original text ...

And a similar change in one of the
CondReturnTextNode(doc, skip) MACROS ... there are
a few MORE of these to CHECK ...

This is a LIST of CHANGES, in GNU diff format - for
3 files, lexer.c, streamio.c and tidylib.c ...

diff c:\FGCVS\TIDY\src\lexer.c ..\src\lexer.c
1888c1895
<     return NULL;
---
>     /* return NULL; */
2072,2073c2079,2081
<                             StoreOriginalTextInToken(doc, lexer->token, 
3);
< #endif
---
>                             /* when UngetChar() is fixed, reduce 3 to 2 */
>                             StoreOriginalTextInToken(doc, lexer->token, 
>2);
>#endif /* #ifdef TIDY_STORE_ORIGINAL_TEXT */
2260c2268,2269
<                     CondReturnTextNode(doc, 2)
---
>                     /* since UngetChar() fixed, reduce 2 to 0! */
>                     CondReturnTextNode(doc, 0)

NOTES: The first item is just to remove a WARNING from the compiler ...
the function already returns a value, thus this last 'return NULL;' will
never be reached ... the next two changes are a ripple effect of 'fixing'
streamio.c ...

diff c:\FGCVS\TIDY\src\streamio.c ..\src\streamio.c
479c479,480
<             return c;
---
>         } else {
>            in->curcol++;
481c482,484
<         in->curcol++;
---
>#ifdef TIDY_STORE_ORIGINAL_TEXT
>        AddCharToOriginalText(in, (tchar)c);
>#endif /* #ifdef TIDY_STORE_ORIGINAL_TEXT */
504a508,513
>#ifdef TIDY_STORE_ORIGINAL_TEXT
>     if( in->otextlen ) { /* if there is LENGTH */
>        in->otextlen--; /* reduce the length */
>        in->otextbuf[in->otextlen] = 0; /* kill char */
>     }
>#endif /* #ifdef TIDY_STORE_ORIGINAL_TEXT */

NOTES: These changes effectively remove the added-to-store
character when an UngetChar() is done, and on the next
GetChar(), which will end up in PopChar(), putting it
back into the store ... without re-allocation of memory!

diff c:\FGCVS\TIDY\src\tidylib.c ..\src\tidylib.c
106a107,109
>#ifdef TIDY_STORE_ORIGINAL_TEXT
>     doc->storeText = 1; /* enable storage of original */
>#endif /* #ifdef TIDY_STORE_ORIGINAL_TEXT */

NOTES: The save of the original strings will not be done
unless this item is 'enabled' ... maybe this should
eventually be an 'option' item ...

If it is AGREED, I hope these changes can be ADDED to the
CVS source ... NOTE, they ONLY come into effect when
the TIDY_STORE_ORIGINAL_TEXT is defined, and it is NOT
usually defined ... thus should have NO EFFECT on the
present releases ...

Please advise if you wish these changes presented in
some other form ... but the one form I can NOT easily
present is the 'CVS diff' command form, since I do not
make these changes in my CVS update folder ... until
you add them to CVS ;=))

Look forward to a positive result ... once in CVS, I
will certainly RE-CHECK the changes ... and check
more of the macro 'skip' values ...

As stated, this is some work I am doing on trying to
allow some MORE options on output, and I want to be
able to refer back to the 'original' stored text ...
sort of, 'what did the user put there?' ...

This is especially true for 'vertical' spacing ... Tidy
presently virtually removes it ALL, except when
vertical-space is on, but then only vertically spaces
some 'in-line' paragraphs ...

I think some form of preservation of the user's original
vertical spacing ADDS to the readability, and thus to the
'human' understanding of HTML code ... it adds such a small
number of bytes to the file size, and is 'ignored' by
browsers, at least in the cases, the places I am talking
about ...

Attached below is some diagnostic output, showing the
'original' text ... the first 8 lines, are the FREED
EndTags, and 'blank' lines during the document parsing ...
then it is the output diagnostics when the final 'free
document' is done ... thus some now appear out-of-order ...
of course the ^M, ^J are CR, LF chars ... just a way to
render 'control' characters harmless but visible ;=))

Part of this effort would be to NOT toss away EndTags,
and 'blank' lines ... leave them in the tree ... so that
a FULLER POST parsing of the tree could be done ... and
they could still be 'tossed' (freed) before the
PPrint... services gets its hands on the 'tree' ...

Or, this could be as simple as adding a 'deleted' marker
to the node, and thus could be easily 'skipped' ...

Thanking you ... and I hope, perhaps in the new year,
some others will find the time to comment on some of
the ideas expressed here ...

This should be my LAST 2005 post ... see you in the
new year ... happy 'tidying' ...

Geoff.

Appended templog.txt -

You may note this is the badform.html from the bug
posts ... although I first 'converted' it to windows
line endings, since many 'simple' editors do not
'understand' a single ^M, or indeed a MAC single
^J ...

Of course, most 'good' editors do, but I
often just use notepad to make minor changes ...
and Tidy just eats-them-up ;=)) even before returning
to the 'lexer' ... some are still 'lost'!!!

<templog.txt>
</title>
^M^J  </head>
</div>
</form>
^M^J  </body>
^M^J</html>
^M^J
^M^J
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 3.2//EN">
^M^J<!-- Note that the /FORM and /DIV positions are reversed -->
Bad Form
    <title>
  <head>^M^J
      <input name="login" value="guest">
^M^J
<input type="submit" value="click me">
    <form method="GET" action="">^M^J
    <div>^M^J
^M^J      <input type="hidden" name="password" value="secret">
^M^J  <body>^M^J
^M^J<html>^M^J
</templog.txt>

EOF

_________________________________________________________________
New year, new job – there's more than 100,00 jobs at SEEK 
http://a.ninemsn.com.au/b.aspx?URL=http%3A%2F%2Fninemsn%2Eseek%2Ecom%2Eau&_t=752315885&_r=Jan05_tagline&_m=EXT



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click