Webboard: indexer -Eblob : more than 11 hours...

Author: Christophe Nouvel
Email: [email protected]
Message:
<skip>



> It seems your documents are huge enough.

> An average internet HTML document is usually about 6-7Kb.

> You have 39Gb for 1 million documents, which is about 39Kb

> per document.



Probably MS Frontpage!



> I was wrong saying that it takes 15 minutes for the "same database size".

> 

> Some statistics:

> 

> My database:

> Number of documents with content: 780301

> Total document size: 5100888620

> Average document size: 6537 bytes

> "indexer -Eblob" time: 956 seconds (16 minutes)

> 

> 

> Your database:

> Number of documents with content: 1004530

> Total document size: 39106940132

> Average document size: 38930 bytes

> "indexer -Eblob" time: 39944 seconds (665 minutes)

> 

> The most important number affecting indexing time

> is "Total document size": mnoGoSearch stores

> information about word positions in a document.

> 

> So your database is 7.6 time bigger, but indexing time

> is ~41 times longer. With the same performance as in my database,

> I'd expect it to spend about 2 hours (16*7.6= 121 minutes).

> 

> 

> How much RAM do you have? Perhaps "indexer -Eblob" makes

> the server use swap. Indexer will use about 3Gb of memory

> with your database size.



I have 3Gb of RAM... Indexer is not crawling right now, I will track swap usage next time.



> As a possible indexing time improvement way, please try

> a new feature which should be useful in your case.

> It activates by adding the step=N parameter into the DBAddr

> command. For example like this:

> 

> DBAddr mysql://root@localhost/mnogosearch/?step=100000

> 

> or

> 

> DBAddr mysql://root@localhost/mnogosearch/?DBMode=blob&step=100000

> 

> 

> 100,000 should be good in your case.



OK. Here is my DBAddr now :

DBAddr mysql://mnogosearch:hcraesogonm@localhost/mnogosearch/?DBMode=blob&Deflate=yes&LiveUpdates=yes&socket=/var/lib/mysql/mysql.sock&step=100000

 

> With this parameter added, indexer

> will process not more than 100000 documents at a time,

> so will reduce memory consumption and reduce disk read

> load significantly.



Will see next indexer run.



     Thanks,



          Christophe.



Reply: <http://www.mnogosearch.org/board/message.php?id=21091>