Author: Alexander Barkov
Email: [email protected]
Message:
> > Hi Christophe,
>
> Hi Alexander,
>
> > According to your statistics you have about 910 thousand
> > documents. Index creation time is about 15 minutes for the same
> > database size.
> >
> > Can you please provide some more information?
> >
> > - please run "indexer -Eblob -v6", it will print some
> > more debug output
>
<skip>
>
> > - what is "SELECT sum(docsize) FROM url" output?
>
> mysql> select sum(docsize) from url;
> +--------------+
> | sum(docsize) |
> +--------------+
> | 39106940132 |
> +--------------+
> 1 row in set (1.81 sec)
>
> mysql>
>
> > - What is your mnoGoSearch version?
>
> indexer from mnogosearch-3.3.9-mysql
> http://www.mnogosearch.org/ (C)1998-2009, LavTech Corp.
>
> 64 bits, based on Fedora 5.
>
> > - Please post your indexer.conf file here, or send
> > it to [email protected].
>
> 249 "Server" lines, web sites only.
>
> > - Can you upload your database dump somewhere,
> > so I can download it and try with the same data?
>
> Not really, for several reasons. Huge size is one of them!
Thanks for the additional information.
It seems your documents are huge enough.
An average internet HTML document is usually about 6-7Kb.
You have 39Gb for 1 million documents, which is about 39Kb
per document.
I was wrong saying that it takes 15 minutes for the "same database size".
Some statistics:
My database:
Number of documents with content: 780301
Total document size: 5100888620
Average document size: 6537 bytes
"indexer -Eblob" time: 956 seconds (16 minutes)
Your database:
Number of documents with content: 1004530
Total document size: 39106940132
Average document size: 38930 bytes
"indexer -Eblob" time: 39944 seconds (665 minutes)
The most important number affecting indexing time
is "Total document size": mnoGoSearch stores
information about word positions in a document.
So your database is 7.6 time bigger, but indexing time
is ~41 times longer. With the same performance as in my database,
I'd expect it to spend about 2 hours (16*7.6= 121 minutes).
How much RAM do you have? Perhaps "indexer -Eblob" makes
the server use swap. Indexer will use about 3Gb of memory
with your database size.
As a possible indexing time improvement way, please try
a new feature which should be useful in your case.
It activates by adding the step=N parameter into the DBAddr
command. For example like this:
DBAddr mysql://root@localhost/mnogosearch/?step=100000
or
DBAddr mysql://root@localhost/mnogosearch/?DBMode=blob&step=100000
100,000 should be good in your case.
With this parameter added, indexer
will process not more than 100000 documents at a time,
so will reduce memory consumption and reduce disk read
load significantly.
>
> Thanks,
>
> Christophe.
Reply: <http://www.mnogosearch.org/board/message.php?id=21090>
lmpx.com only provides a reader for public news (NNTP) servers. It is not
affiliated with the servers or forums shown here and is not responsible for
the content of articles, which is written by their respective authors.