Webboard: indexer -Eblob : more than 11 hours...

Author: Christophe
Email: Nouvel
Message:
> Hi Christophe,



     Hi Alexander,

 

> According to your statistics you have about 910 thousand

> documents. Index creation time is about 15 minutes for the same

> database size.

> 

> Can you please provide some more information?

> 

> - please run "indexer -Eblob -v6", it will print some

> more debug output



# /usr/local/mnogosearch/sbin/indexer -E blob -v6

indexer[19694]: Converting to blob

indexer[19694]: Loading URL list

indexer[19694]: URL list loaded: 1214633 urls, 8.70 sec

indexer[19694]: Using deflate

indexer[19694]: Loading intag00

indexer[19694]: Loading intag00 done, 1004530 rows, 1577.10 sec indexer[19694]: Converting intag00                                   indexer[19694]: Converting intag00 done, 2.40 sec

indexer[19694]: Writting intag00                                      indexer[19694]: Writting intag00 done, 76.39 sec

indexer[19694]: Loading intag01                                       indexer[19694]: Loading intag01 done, 1004530 rows, 1018.12 sec   indexer[19694]: Converting intag01                                    indexer[19694]: Converting intag01 done, 2.07 sec                 indexer[19694]: Writting intag01                                      indexer[19694]: Writting intag01 done, 49.35 sec                  indexer[19694]: Loading intag02                                       indexer[19694]: Loading intag02 done, 1004530 rows, 973.36 sec    indexer[19694]: Converting intag02                                    indexer[19694]: Converting intag02 done, 2.21 sec                 indexer[19694]: Writting intag02                                      indexer[19694]: Writting intag02 done, 55.07 sec  
                 indexer[19694]: Loading intag03                                       indexer[19694]: Loading intag03 done, 1004530 rows, 1091.24 sec   indexer[19694]: Converting intag03                                    indexer[19694]: Converting intag03 done, 2.20 sec                 indexer[19694]: Writting intag03                                      indexer[19694]: Writting intag03 done, 65.43 sec                  indexer[19694]: Loading intag04                                       indexer[19694]: Loading intag04 done, 1004530 rows, 946.42 sec    indexer[19694]: Converting intag04                                    indexer[19694]: Converting intag04 done, 2.27 sec                 indexer[19694]: Writting intag04                                      indexer[19694]: Writting intag04
  done, 63.55 sec                  indexer[19694]: Loading intag05                                       indexer[19694]: Loading intag05 done, 1004530 rows, 1133.65 sec   indexer[19694]: Converting intag05                                    indexer[19694]: Converting intag05 done, 1.93 sec                 indexer[19694]: Writting intag05                                      indexer[19694]: Writting intag05 done, 55.48 sec                  indexer[19694]: Loading intag06                                       indexer[19694]: Loading intag06 done, 1004530 rows, 1051.39 sec   indexer[19694]: Converting intag06                                    secno variable empty

indexer[19694]: Converting intag06 done, 2.34 sec                 indexer[19694]: Writting intag06                                      indexer[19694]: Writting intag06 done, 81.57 sec

indexer[19694]: Loading intag07

indexer[19694]: Loading intag07 done, 1004530 rows, 1095.29 sec

indexer[19694]: Converting intag07

indexer[19694]: Converting intag07 done, 1.95 sec

indexer[19694]: Writting intag07

indexer[19694]: Writting intag07 done, 49.28 sec

indexer[19694]: Loading intag08

indexer[19694]: Loading intag08 done, 1004530 rows, 726.59 sec

indexer[19694]: Converting intag08

indexer[19694]: Converting intag08 done, 2.34 sec

indexer[19694]: Writting intag08

indexer[19694]: Writting intag08 done, 73.84 sec

indexer[19694]: Loading intag09

indexer[19694]: Loading intag09 done, 1004530 rows, 1104.33 sec

indexer[19694]: Converting intag09

indexer[19694]: Converting intag09 done, 2.06 sec

indexer[19694]: Writting intag09

indexer[19694]: Writting intag09 done, 47.88 sec

indexer[19694]: Loading intag0A

indexer[19694]: Loading intag0A done, 1004530 rows, 849.48 sec

indexer[19694]: Converting intag0A

indexer[19694]: Converting intag0A done, 1.73 sec

indexer[19694]: Writting intag0A

indexer[19694]: Writting intag0A done, 32.97 sec

indexer[19694]: Loading intag0B

indexer[19694]: Loading intag0B done, 1004530 rows, 928.72 sec

indexer[19694]: Converting intag0B

indexer[19694]: Converting intag0B done, 2.30 sec

indexer[19694]: Writting intag0B

indexer[19694]: Writting intag0B done, 71.32 sec

indexer[19694]: Loading intag0C

indexer[19694]: Loading intag0C done, 1004530 rows, 653.02 sec

indexer[19694]: Converting intag0C

indexer[19694]: Converting intag0C done, 2.10 sec

indexer[19694]: Writting intag0C

indexer[19694]: Writting intag0C done, 61.29 sec

indexer[19694]: Loading intag0D

indexer[19694]: Loading intag0D done, 1004530 rows, 736.59 sec

indexer[19694]: Converting intag0D

nintags variable empty

indexer[19694]: Converting intag0D done, 1.99 sec

indexer[19694]: Writting intag0D

indexer[19694]: Writting intag0D done, 47.51 sec

indexer[19694]: Loading intag0E

indexer[19694]: Loading intag0E done, 1004530 rows, 542.83 sec

indexer[19694]: Converting intag0E

indexer[19694]: Converting intag0E done, 2.30 sec

indexer[19694]: Writting intag0E

indexer[19694]: Writting intag0E done, 75.63 sec

indexer[19694]: Loading intag0F

indexer[19694]: Loading intag0F done, 1004530 rows, 855.59 sec

indexer[19694]: Converting intag0F

indexer[19694]: Converting intag0F done, 2.05 sec

indexer[19694]: Writting intag0F

indexer[19694]: Writting intag0F done, 56.58 sec

indexer[19694]: Loading intag10

indexer[19694]: Loading intag10 done, 1004530 rows, 545.32 sec

indexer[19694]: Converting intag10

indexer[19694]: Converting intag10 done, 2.14 sec

indexer[19694]: Writting intag10

indexer[19694]: Writting intag10 done, 54.52 sec

indexer[19694]: Loading intag11

indexer[19694]: Loading intag11 done, 1004530 rows, 830.82 sec

indexer[19694]: Converting intag11

indexer[19694]: Converting intag11 done, 1.97 sec

indexer[19694]: Writting intag11

indexer[19694]: Writting intag11 done, 39.18 sec

indexer[19694]: Loading intag12

indexer[19694]: Loading intag12 done, 1004530 rows, 577.28 sec

indexer[19694]: Converting intag12

indexer[19694]: Converting intag12 done, 2.10 sec

indexer[19694]: Writting intag12

indexer[19694]: Writting intag12 done, 45.48 sec

indexer[19694]: Loading intag13

indexer[19694]: Loading intag13 done, 1004530 rows, 488.50 sec

indexer[19694]: Converting intag13

indexer[19694]: Converting intag13 done, 2.20 sec

indexer[19694]: Writting intag13

indexer[19694]: Writting intag13 done, 54.80 sec

indexer[19694]: Loading intag14

indexer[19694]: Loading intag14 done, 1004530 rows, 487.70 sec

indexer[19694]: Converting intag14

indexer[19694]: Converting intag14 done, 2.63 sec

indexer[19694]: Writting intag14

indexer[19694]: Writting intag14 done, 83.71 sec

indexer[19694]: Loading intag15

indexer[19694]: Loading intag15 done, 1004530 rows, 645.87 sec

indexer[19694]: Converting intag15

indexer[19694]: Converting intag15 done, 2.43 sec

indexer[19694]: Writting intag15

indexer[19694]: Writting intag15 done, 70.57 sec

indexer[19694]: Loading intag16

indexer[19694]: Loading intag16 done, 1004530 rows, 769.50 sec

indexer[19694]: Converting intag16

indexer[19694]: Converting intag16 done, 2.46 sec

indexer[19694]: Writting intag16

indexer[19694]: Writting intag16 done, 69.63 sec

indexer[19694]: Loading intag17

indexer[19694]: Loading intag17 done, 1004530 rows, 667.72 sec

indexer[19694]: Converting intag17

indexer[19694]: Converting intag17 done, 1.93 sec

indexer[19694]: Writting intag17

indexer[19694]: Writting intag17 done, 46.56 sec

indexer[19694]: Loading intag18

indexer[19694]: Loading intag18 done, 1004530 rows, 535.33 sec

indexer[19694]: Converting intag18

indexer[19694]: Converting intag18 done, 2.11 sec

indexer[19694]: Writting intag18

indexer[19694]: Writting intag18 done, 54.06 sec

indexer[19694]: Loading intag19

indexer[19694]: Loading intag19 done, 1004530 rows, 484.89 sec

indexer[19694]: Converting intag19

indexer[19694]: Converting intag19 done, 1.98 sec

indexer[19694]: Writting intag19

indexer[19694]: Writting intag19 done, 44.87 sec

indexer[19694]: Loading intag1A

indexer[19694]: Loading intag1A done, 1004530 rows, 18446744949882880.00 sec

indexer[19694]: Converting intag1A

indexer[19694]: Converting intag1A done, 2.25 sec

indexer[19694]: Writting intag1A

indexer[19694]: Writting intag1A done, 77.69 sec

indexer[19694]: Loading intag1B

indexer[19694]: Loading intag1B done, 1004530 rows, 417.26 sec

indexer[19694]: Converting intag1B

indexer[19694]: Converting intag1B done, 2.05 sec

indexer[19694]: Writting intag1B

indexer[19694]: Writting intag1B done, 54.10 sec

indexer[19694]: Loading intag1C

indexer[19694]: Loading intag1C done, 1004530 rows, 326.21 sec

indexer[19694]: Converting intag1C

indexer[19694]: Converting intag1C done, 2.14 sec

indexer[19694]: Writting intag1C

indexer[19694]: Writting intag1C done, 54.35 sec

indexer[19694]: Loading intag1D

indexer[19694]: Loading intag1D done, 1004530 rows, 345.31 sec

indexer[19694]: Converting intag1D

secno variable empty

indexer[19694]: Converting intag1D done, 2.02 sec

indexer[19694]: Writting intag1D

indexer[19694]: Writting intag1D done, 44.84 sec

indexer[19694]: Loading intag1E

indexer[19694]: Loading intag1E done, 1004530 rows, 319.40 sec

indexer[19694]: Converting intag1E

indexer[19694]: Converting intag1E done, 1.83 sec

indexer[19694]: Writting intag1E

indexer[19694]: Writting intag1E done, 34.64 sec

indexer[19694]: Loading intag1F

indexer[19694]: Loading intag1F done, 1004530 rows, 301.51 sec

indexer[19694]: Converting intag1F

indexer[19694]: Converting intag1F done, 2.42 sec

indexer[19694]: Writting intag1F

indexer[19694]: Writting intag1F done, 65.23 sec

indexer[19694]: Total converted: 161467713 records, 683694864 bytes

indexer[19694]: Writting '#ts'

indexer[19694]: Cleaning up table 'bdicti'

indexer[19694]: Cleaning up table 'bdicti' done, 555.40 sec

indexer[19694]: Converting url data

indexer[19694]: Writting '#ts'

indexer[19694]: Converting URL data done, 53.93 sec

indexer[19694]: Switching to new blob table.

indexer[19694]: Converting to blob finished     18446744949882880.00



> - what is "SELECT sum(docsize) FROM url" output?



mysql> select sum(docsize) from url;

+--------------+

| sum(docsize) |

+--------------+

|  39106940132 |

+--------------+

1 row in set (1.81 sec)



mysql>



> - What is your mnoGoSearch version?



indexer from mnogosearch-3.3.9-mysql

http://www.mnogosearch.org/ (C)1998-2009, LavTech Corp.



64 bits, based on Fedora 5.



> - Please post your indexer.conf file here, or send

>   it to [email protected].



249 "Server" lines, web sites only.



> - Can you upload your database dump somewhere,

>   so I can download it and try with the same data?



Not really, for several reasons. Huge size is one of them!



Thanks,



     Christophe.

Reply: <http://www.mnogosearch.org/board/message.php?id=21088>