Webboard: indexer -Eblob : more than 11 hours...
| Newsgroups | gmane.comp.web.mnogosearch.general |
|---|---|
| Message-ID | <[email protected]> |
Author: Alexander Barkov Email: [email protected] Message: > > > <skip> > > > > > > > It seems your documents are huge enough. > > > > An average internet HTML document is usually about 6-7Kb. > > > > You have 39Gb for 1 million documents, which is about 39Kb > > > > per document. > > > > > > Probably MS Frontpage! > > > > What does "SHOW TABLE STATUS LIKE 'bdicti'" return for you? > > Not easy to read after such a paste but here it is : > > mysql> SHOW TABLE STATUS LIKE 'bdicti'; > +--------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-----------------+----------+---------------------------------------+---------+ > | Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment | > +--------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-----------------+----------+---------------------------------------+---------+ > | bdicti | MyISAM | 10 | Dynamic | 1105898 | 2360 | 3044688400 | 1099511627775 | 29620224 | 434418896 | NULL | 2007-05-31 07:55:27 | 2010-07-03 22:13:26 | 2008-10-13 17:27:12 | utf8_general_ci | NULL | max_rows=300000000 avg_row_length=512 | | > +--------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-----------------+----------+---------------------------------------+---------+ > 1 row in set (0.00 sec) > > mysql> > > > Looks fine. Did step=100000 help? > > Sure! > > Lines like 'Loading documents 1-100001' to 'Loading documents 2000001-2100001' are now logged. > > But in the end : > > Jul 3 21:51:13 fronton indexer[1191]: Total converted: 182589007 records, 75567 > 3713 bytes > Jul 3 22:13:30 fronton indexer[1191]: Converting url data > Jul 3 22:14:32 fronton indexer[1191]: Switching to new blob table. > Jul 3 22:14:33 fronton indexer[1191]: Converting to blob finished 7445.71 > > Back to normal conversion times. Good to know. > > By the way, I don't understand why Loading documents 2000001-2100001 if 1336677 total are reported by indexer -S... > It loops through all "rec_id" values in the table "url". Try this query: "SELECT max(rec_id) FROM url" it should report a number in between 2000001 and 2100001. > Christophe. > Reply: <http://www.mnogosearch.org/board/message.php?id=21094>