HTML Parser and start of CLucene conversion

Hey all,

 	I'm starting a proof of concept conversion to use CLucene to 
replace the db.words.db	... it will still use BDB for the other db files.

 	The excerpts DB can likely be eliminated as well.

 	In my tests so far I can index 12,000 files in 4 minutes producing 
a 35MG Lucene Index.  The equivalent compressed standard index files total 
63MG and take 40 minutes to create.

 	Note that this is a loop inserting documents with a 
'simple_doc_insert' libhtdig api... not a spidering run.

 	In my code jockeying so far I notice that our HTML::parse() is 
fairly inefficient... it filters out the text between no_index_start & 
no_index_end tags, copies it then parses the copy.  Seems obvious 
that we can do this in the second while loop with state variables like 
we do for TITLE tags.  This would eliminate one linear scan through each 
document and needless memory usage.

 	Part of the reason for looking at how the HTML::parse is working 
is to make it unicode kosher by eliminating 'char' dependencies.

 	If anyone wants to help with the parse thing feel free.  All you 
need is to create a UTF8 HTML document with a multi-byte character and 
make sure the character is preserved in the debug output... we can worry 
about what happens to the word later.

 	For a refresher the reasons for strongly considering converting to 
CLucene are a follows:
 	* It's UTF8/Unicode capable
 	* Potentially much faster
 	* Supports field-based searching
 	* Allows us to ditch our nonstandard BDB 3.0.55
 	* Allows us to ditch thousands of lines of code in favor of an
 	  active project

 	If anyone seriously wants to help with this effort, I'd be willing 
to purchase a couple copies of Lucene in Action by Erik Hatcher and Otis 
Gospodnetic for people.  It covers the java version, however the C++ 
version attempts to be a strict translation to C++ in every way.  My copy 
is on the way.

 	At the moment I am concentrating on the spidering & indexing code 
to measure speed and size differences.  Searching will come after that.

 	Thanks.

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
ht://Dig Developer mailing list:
[email protected]
List information (subscribe/unsubscribe, etc.)
https://lists.sourceforge.net/lists/listinfo/htdig-dev