TIDY_STORE_ORIGINAL_TEXT?

I'm currently working on a HTML to Text converter project which uses HTML Tidy to parse the source document.  One requirement for this project is the ability to return the source line/column (or stream pos) for any given content word within the source HTML document.  To do this, I'm using the reported line/column for each TidyNode_Text node in the document tree and then computing the word start line/column values from the text content of the node.

Unfortunately, I've noticed that HTML Tidy is manipulating the text content of the TidyNode_Text nodes which makes my approach rather useless as what I'm using to compute source line/column offsets isn't always what was actually in the source document.  Some of the manipulations are configuration controlled (such as PreserveEntities) which I can obviously disable/control.  Some happen in the "pretty print" functions called by tidyNodeGetText() which I can get around by calling tidyNodeGetValue().  But there are some manipulations that are happening at the parser, lexer, and streamio levels for which there appears to be no way to control/work around.  Obviously there must be good reason for these manipulations at these points but it was a bit disheartening to find them mixed into the parser/lexer/streamio levels rather than in the cleanup and/or pretty print levels.

While investigating all of this I noticed a #define that I've not been able to find any significant information about: TIDY_STORE_ORIGINAL_TEXT.  It would appear (based on a quick look through the code) that defining this enables storage within each node of the actual text extracted from the source document.  This looks like it may be exactly what I need for my project.  However I didn't see any functionality that exposes the original text through the library API functions (i.e. no tidyNodeGetOrigText() function wrapped in TIDY_STORE_ORIGINAL_TEXT #ifdef/#endif wrapper).  As this appears to be some sort of undocumented/hidden functionality, I wanted to ask the group about this #define (and related functionality) before attempting to use it.

The only information that I've been able to find about this #define was a couple fragments of what appears to be a conversation between Geoff Air, Arnaud, and Björn from back in January of 2006 in which Geoff had some potential changes which we wanted to have applied to address a few bugs that he'd found when using the functionality.  I wasn't able to determine what problems specifically Geoff had encountered, what his suggested changes were, nor whether the changes were eventually applied to the code.

-Chris



      

------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge