Re: divide and conquer

Klaus,

Thanks for the info!

If we can get enough developers interested I've got 
an idea to dumb the current db code and move to using Lucene/Clucene. 
This should scale much better and remove a large chunk of code from active 
maintenance.

On Sun, 12 Dec 2004, minimoa wrote:

> I have been using htdig for years to index my local pc. htdig is part of the
> Suse linux distribution I use.
> On a 2.4GHz cpu indexing the entire pc took 2 hours with htdig 3.1.6, but with
> version 3.2.0b6 it takes much longer. The words database becomes htdig's
> bottleneck when it exceeds 200MB on my pc. So I divided the search domain
> according to local directory structure and got smaller words databases and
> faster digs. The disadvantage is a lot of config files.
>
> Searching itself is faster: 3000 hits for FAQ are found in 30 secs by 3.2.0b6
> while 3.1.6 needed 60 secs. And phrases are a quantum leap.
>
> Currently merging of databases is not any faster than indexing directly.
> So if database update cannot be accelerated due to limitations of db software,
> techniques for divide and conquer could be improved and should be made more
> common in documentation and configuration examples of htdig.
>
> Thanks for a great software
> Klaus

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
ht://Dig Developer mailing list:
[email protected]
List information (subscribe/unsubscribe, etc.)
https://lists.sourceforge.net/lists/listinfo/htdig-dev