Webboard: optimization

Author: Alexander Barkov
Email: [email protected]
Message:
> I am setting up a search system for a moderately large number of 

> sites which have vastly different numbers of pages and I am trying to 

> speed up the crawling/indexing process.  I am using 3.3.9 on the off 

> line crawling/indexing systems with the blob DB mode.  I am finding 

> that when I run the indexer (-Eblob) the crawler stops and the 

> indexer has been taking well over half the time between runs (via cron 

> job) to index what has been crawled.  I would like to either get some 

> threading happening on the indexer or copy the right data between 

> databases and index on separate databases.  I know I could just move 

> the entire databases over from the crawler to the indexer and then 

> index and them move that to the search system, but that would 

> involve re indexing everything every time I want to update the live 

> database which would be impractical for the number of pages I am 

> looking at.  I already have over 700,000 pages in the DB and I will 

> have over 2,000,000 total by the time I am done.  I have split the DB 

> up to ease maintenance though I have not yet had to move the parts 

> to separate machines.  Is there any way to speed this process up?  I 

> had to compile the system myself to get the new features I needed in 

> 3.3.9 because the package in the repository for my system did not 

> have this version yet.  Would some configure flag or make rule use 

> more optimizations in the compiler that would translate to a 

> significant improvement in speed?  The only other way I can think of is 

> to split the configuration file into the DBAddr part and the rest and set 

> the crawler to a config file that uses all the DBAddr commands and 

> includes the other stuff config file and set the up a config file for each 

> DBAddr command that also includes the same rest of the config and 

> run the indexers in parallel.  I am not sure how well that would work 

> though, especially for tallying how many links there are to a page.  

> Any ideas and help are appreciated.

> 

> Thanks



Unfortunately, simultaneous crawling (e.g. "indexer") and

creating of search index (e.g. "indexer -Eblob") is currently

not implemented.



"indexer -Eblob" locks all crawler processes ("indexer").

This is done for consistency reasons.



Having separate databases with separate .conf files should help. 

When you run "indexer -Eblob" for a single .conf file,

it will lock only single database, while crawlers processes

for other databases will continue crawling.



You can then merge search results from all databases

using multiple DBAddr commands in search.htm.



Consider also trying cluster solution:

http://www.mnogosearch.org/doc33/msearch-cluster.html



It is much faster than just using multiple DBAddr commands

in search.htm.





Reply: <http://www.mnogosearch.org/board/message.php?id=21074>