Re: optimization questions

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I have 2 questions then.  First is how stable is the current sources?  I
know in some projects they can be very unstable when you get the dev
versions as you are suggesting.  Second, what would be the downside of
running the Eblob operation of the databases separately at the same
time?  I imaging the popularity would be messed up, but that is not very
important for my use of the system, so that would not be a real issue.
I have a way that I could keep the separate config files in sync so that
they will not remove pages, just separate it into 2, the config with
just the database(s) for that instance, and the rest of the config to be
included in the DB set specific one.  Is there any real tangible
downside other than popularity ranking and higher CPU/memory usage to
running an indexer -Eblob for each database separately?

Thanks,
Glenn McGurrin

Alexander Barkov wrote:
> Hi Glenn,
>
> Glenn McGurrin wrote:
> I am setting up a search system for a moderately large number of sites
> which have vastly different numbers of pages and I am trying to speed up
> the crawling/indexing process.  I am using 3.3.9 on the off line
> crawling/indexing systems with the blob DB mode.  I am finding that when
> I run the indexer (-Eblob) the crawler stops and the indexer has been
> taking well over half the time between runs (via cron job) to index what
> has been crawled.
>
>> This happens because "indexer -Eblob" executes "LOCK TABLE url, bdict"
>> at the beginning, to create a consistent inverted index.
>> I am working on avoiding this locking.
>
>  I would like to either get some threading happening
> on the indexer or copy the right data between databases and index on
> separate databases.  I know I could just move the entire databases over
> from the crawler to the indexer and then index and them move that to the
> search system, but that would involve re indexing everything every time
> I want to update the live database which would be impractical for the
> number of pages I am looking at.  I already have over 700,000 pages in
> the DB and I will have over 2,000,000 total by the time I am done.  I
> have split the DB up to ease maintenance though I have not yet had to
> move the parts to separate machines.  Is there any way to speed this
> process up?  I had to compile the system myself to get the new features
> I needed in 3.3.9 because the package in the repository for my system
> did not have this version yet.  Would some configure flag or make rule
> use more optimizations in the compiler that would translate to a
> significant improvement in speed?
>
>> The main bottleneck during "indexer -Eblob" is in
>> how MySQL performs INSERTs.
>
>> Some improvements have been made in the next version (3.3.10)
>> to improve INSERT performance. You can try the latest sources
>> from here:
>
>> http://myoffice.izhnet.ru/bar2/~bar/mnogosearch-3.3.10.tar.gz
>
>> Add &MultiInsert=yes into your DBAddr line, e.g.:
>
>> DBAddr mysql://root@localhost/mnogosearch/?DBMode=blob&MultiInsert=yes
>
>> It reduced the total "indexer -Eblob" time to about 30% faster in some
>> cases.
>
>> Please give feedback.
>
>> Thanks!
>
>
> Thanks,
> Glenn McGurrin
>>
- ---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
- From - Thu

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)

iEYEARECAAYFAku0tGYACgkQzSSLW0oTVN66LACePpdroGu7W+dDPNy7ZehT+aRT
ZKoAn3t0WSGn0El6Wu38mDp+zPYpJDX7
=bkSd
-----END PGP SIGNATURE-----