Webboard: indexer -Eblob : more than 11 hours...
| Newsgroups | gmane.comp.web.mnogosearch.general |
|---|---|
| Message-ID | <[email protected]> |
Author: Alexander Barkov Email: [email protected] Message: > <skip> > > > It seems your documents are huge enough. > > An average internet HTML document is usually about 6-7Kb. > > You have 39Gb for 1 million documents, which is about 39Kb > > per document. > > Probably MS Frontpage! What does "SHOW TABLE STATUS LIKE 'bdicti'" return for you? > > > I was wrong saying that it takes 15 minutes for the "same database size". > > > > Some statistics: > > > > My database: > > Number of documents with content: 780301 > > Total document size: 5100888620 > > Average document size: 6537 bytes > > "indexer -Eblob" time: 956 seconds (16 minutes) > > > > > > Your database: > > Number of documents with content: 1004530 > > Total document size: 39106940132 > > Average document size: 38930 bytes > > "indexer -Eblob" time: 39944 seconds (665 minutes) > > > > The most important number affecting indexing time > > is "Total document size": mnoGoSearch stores > > information about word positions in a document. > > > > So your database is 7.6 time bigger, but indexing time > > is ~41 times longer. With the same performance as in my database, > > I'd expect it to spend about 2 hours (16*7.6= 121 minutes). > > > > > > How much RAM do you have? Perhaps "indexer -Eblob" makes > > the server use swap. Indexer will use about 3Gb of memory > > with your database size. > > I have 3Gb of RAM... Indexer is not crawling right now, I will track swap usage next time. > > > As a possible indexing time improvement way, please try > > a new feature which should be useful in your case. > > It activates by adding the step=N parameter into the DBAddr > > command. For example like this: > > > > DBAddr mysql://root@localhost/mnogosearch/?step=100000 > > > > or > > > > DBAddr mysql://root@localhost/mnogosearch/?DBMode=blob&step=100000 > > > > > > 100,000 should be good in your case. > > OK. Here is my DBAddr now : > DBAddr mysql://mnogosearch:hcraesogonm@localhost/mnogosearch/?DBMode=blob&Deflate=yes&LiveUpdates=yes&socket=/var/lib/mysql/mysql.sock&step=100000 Looks fine. Did step=100000 help? > > > With this parameter added, indexer > > will process not more than 100000 documents at a time, > > so will reduce memory consumption and reduce disk read > > load significantly. > > Will see next indexer run. > > Thanks, > > Christophe. > Reply: <http://www.mnogosearch.org/board/message.php?id=21092>