Re: optimization questions

Hi Glenn,

Glenn McGurrin wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> I have 2 questions then.  First is how stable is the current sources? 

The same stable with 3.3.9 release.

> I
> know in some projects they can be very unstable when you get the dev
> versions as you are suggesting.  Second, what would be the downside of
> running the Eblob operation of the databases separately at the same
> time?  I imaging the popularity would be messed up, but that is not very
> important for my use of the system, so that would not be a real issue.
> I have a way that I could keep the separate config files in sync so that
> they will not remove pages, just separate it into 2, the config with
> just the database(s) for that instance, and the rest of the config to be
> included in the DB set specific one.  Is there any real tangible
> downside other than popularity ranking and higher CPU/memory usage to
> running an indexer -Eblob for each database separately?

This is fine to run "indexer -Eblob" separately.

You even don't need to have multiple .conf files for that.
You can use -D argument to run an action against a certain database,
for example:

indexer -D1 -Eblob     # run against the first DBAddr
indexer -D2 -Eblob     # run against the second DBAddr



> 
> Thanks,
> Glenn McGurrin
> 
> Alexander Barkov wrote:
>> Hi Glenn,
>>
>> Glenn McGurrin wrote:
>> I am setting up a search system for a moderately large number of sites
>> which have vastly different numbers of pages and I am trying to speed up
>> the crawling/indexing process.  I am using 3.3.9 on the off line
>> crawling/indexing systems with the blob DB mode.  I am finding that when
>> I run the indexer (-Eblob) the crawler stops and the indexer has been
>> taking well over half the time between runs (via cron job) to index what
>> has been crawled.
>>
>>> This happens because "indexer -Eblob" executes "LOCK TABLE url, bdict"
>>> at the beginning, to create a consistent inverted index.
>>> I am working on avoiding this locking.
>>  I would like to either get some threading happening
>> on the indexer or copy the right data between databases and index on
>> separate databases.  I know I could just move the entire databases over
>> from the crawler to the indexer and then index and them move that to the
>> search system, but that would involve re indexing everything every time
>> I want to update the live database which would be impractical for the
>> number of pages I am looking at.  I already have over 700,000 pages in
>> the DB and I will have over 2,000,000 total by the time I am done.  I
>> have split the DB up to ease maintenance though I have not yet had to
>> move the parts to separate machines.  Is there any way to speed this
>> process up?  I had to compile the system myself to get the new features
>> I needed in 3.3.9 because the package in the repository for my system
>> did not have this version yet.  Would some configure flag or make rule
>> use more optimizations in the compiler that would translate to a
>> significant improvement in speed?
>>
>>> The main bottleneck during "indexer -Eblob" is in
>>> how MySQL performs INSERTs.
>>> Some improvements have been made in the next version (3.3.10)
>>> to improve INSERT performance. You can try the latest sources
>>> from here:
>>> http://myoffice.izhnet.ru/bar2/~bar/mnogosearch-3.3.10.tar.gz
>>> Add &MultiInsert=yes into your DBAddr line, e.g.:
>>> DBAddr mysql://root@localhost/mnogosearch/?DBMode=blob&MultiInsert=yes
>>> It reduced the total "indexer -Eblob" time to about 30% faster in some
>>> cases.
>>> Please give feedback.
>>> Thanks!
>>
>> Thanks,
>> Glenn McGurrin
> - ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
> - From - Thu
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.8 (Darwin)
> 
> iEYEARECAAYFAku0tGYACgkQzSSLW0oTVN66LACePpdroGu7W+dDPNy7ZehT+aRT
> ZKoAn3t0WSGn0El6Wu38mDp+zPYpJDX7
> =bkSd
> -----END PGP SIGNATURE-----
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]