optimization questions

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I am setting up a search system for a moderately large number of sites
which have vastly different numbers of pages and I am trying to speed up
the crawling/indexing process.  I am using 3.3.9 on the off line
crawling/indexing systems with the blob DB mode.  I am finding that when
I run the indexer (-Eblob) the crawler stops and the indexer has been
taking well over half the time between runs (via cron job) to index what
has been crawled.  I would like to either get some threading happening
on the indexer or copy the right data between databases and index on
separate databases.  I know I could just move the entire databases over
from the crawler to the indexer and then index and them move that to the
search system, but that would involve re indexing everything every time
I want to update the live database which would be impractical for the
number of pages I am looking at.  I already have over 700,000 pages in
the DB and I will have over 2,000,000 total by the time I am done.  I
have split the DB up to ease maintenance though I have not yet had to
move the parts to separate machines.  Is there any way to speed this
process up?  I had to compile the system myself to get the new features
I needed in 3.3.9 because the package in the repository for my system
did not have this version yet.  Would some configure flag or make rule
use more optimizations in the compiler that would translate to a
significant improvement in speed?

Thanks,
Glenn McGurrin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)

iEYEARECAAYFAkuvxUEACgkQzSSLW0oTVN5DjQCeONJpnCbaeZFMdoYZGseHtBpg
cXAAn1qPlIsKcoC9RLijiWx/WZm9V9LE
=zLcg
-----END PGP SIGNATURE-----