Re: optimization questions

Hi Glenn,

Glenn McGurrin wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> I am setting up a search system for a moderately large number of sites
> which have vastly different numbers of pages and I am trying to speed up
> the crawling/indexing process.  I am using 3.3.9 on the off line
> crawling/indexing systems with the blob DB mode.  I am finding that when
> I run the indexer (-Eblob) the crawler stops and the indexer has been
> taking well over half the time between runs (via cron job) to index what
> has been crawled.

This happens because "indexer -Eblob" executes "LOCK TABLE url, bdict"
at the beginning, to create a consistent inverted index.
I am working on avoiding this locking.

>  I would like to either get some threading happening
> on the indexer or copy the right data between databases and index on
> separate databases.  I know I could just move the entire databases over
> from the crawler to the indexer and then index and them move that to the
> search system, but that would involve re indexing everything every time
> I want to update the live database which would be impractical for the
> number of pages I am looking at.  I already have over 700,000 pages in
> the DB and I will have over 2,000,000 total by the time I am done.  I
> have split the DB up to ease maintenance though I have not yet had to
> move the parts to separate machines.  Is there any way to speed this
> process up?  I had to compile the system myself to get the new features
> I needed in 3.3.9 because the package in the repository for my system
> did not have this version yet.  Would some configure flag or make rule
> use more optimizations in the compiler that would translate to a
> significant improvement in speed?

The main bottleneck during "indexer -Eblob" is in
how MySQL performs INSERTs.

Some improvements have been made in the next version (3.3.10)
to improve INSERT performance. You can try the latest sources
from here:

http://myoffice.izhnet.ru/bar2/~bar/mnogosearch-3.3.10.tar.gz

Add &MultiInsert=yes into your DBAddr line, e.g.:

DBAddr mysql://root@localhost/mnogosearch/?DBMode=blob&MultiInsert=yes

It reduced the total "indexer -Eblob" time to about 30% faster in some
cases.

Please give feedback.

Thanks!

> 
> Thanks,
> Glenn McGurrin
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.8 (Darwin)
> 
> iEYEARECAAYFAkuvxUEACgkQzSSLW0oTVN5DjQCeONJpnCbaeZFMdoYZGseHtBpg
> cXAAn1qPlIsKcoC9RLijiWx/WZm9V9LE
> =zLcg
> -----END PGP SIGNATURE-----
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
> From - Thu