Webboard: optimization

Author: Glenn McGurrin
Email: [email protected]
Message:
I am setting up a search system for a moderately large number of 

sites which have vastly different numbers of pages and I am trying to 

speed up the crawling/indexing process.  I am using 3.3.9 on the off 

line crawling/indexing systems with the blob DB mode.  I am finding 

that when I run the indexer (-Eblob) the crawler stops and the 

indexer has been taking well over half the time between runs (via cron 

job) to index what has been crawled.  I would like to either get some 

threading happening on the indexer or copy the right data between 

databases and index on separate databases.  I know I could just move 

the entire databases over from the crawler to the indexer and then 

index and them move that to the search system, but that would 

involve re indexing everything every time I want to update the live 

database which would be impractical for the number of pages I am 

looking at.  I already have over 700,000 pages in the DB and I will 

have over 2,000,000 total by the time I am done.  I have split the DB 

up to ease maintenance though I have not yet had to move the parts 

to separate machines.  Is there any way to speed this process up?  I 

had to compile the system myself to get the new features I needed in 

3.3.9 because the package in the repository for my system did not 

have this version yet.  Would some configure flag or make rule use 

more optimizations in the compiler that would translate to a 

significant improvement in speed?  The only other way I can think of is 

to split the configuration file into the DBAddr part and the rest and set 

the crawler to a config file that uses all the DBAddr commands and 

includes the other stuff config file and set the up a config file for each 

DBAddr command that also includes the same rest of the config and 

run the indexers in parallel.  I am not sure how well that would work 

though, especially for tallying how many links there are to a page.  

Any ideas and help are appreciated.



Thanks

Reply: <http://www.mnogosearch.org/board/message.php?id=21060>