| Newsgroups |
gmane.comp.web.mnogosearch.general |
| Message-ID |
<[email protected]> |
Author: Alexander Barkov
Email: [email protected]
Message:
> I would like to know how to speed up indexing.
- Start with making sure you're using DBMode=blob.
- Which database software do you use?
MySQL with MyISAM is known to be the fastest for mnoGoSearch
purposes (but can crash and loose some data occasionally,
as MyISAM does not support ACID - should not be a big problem
with regular backup).
> Does it make a big difference to use a machine
> with 1 or 2 processor (4 or 6 core) or is RAM more important?
Which step do you mean by indexing?
- crawling (when you run "indexer")
- or creating search index (when you run "indexer -Eblob")
For crawling, having multiple cores makes sense
(providing that you have good connection, or crawl local files),
especially when you start multiple threads (using -N), or start
multiple indexer processes.
As for RAM, it's nice to have SQL indexes on the table "url"
loaded into memory key cache. In case of MySQL, make sure
key_buffer_size is reasonably large to be able to fit indexes.
Some queries which can help to choose a proper size:
- SHOW VARIABLES LIKE 'key_buffer_size';
- SHOW TABLE STATUS LIKE 'url';
the value of Index_length is the most important
- show status like 'key_%';
On a database with 1.5 million documents I have
Index_length reporting about 320 Mb. key_buffer_size
set to 1Gb should be enough
(don't forget to reserve some memory for other tables' indexes).
For "indexer -Eblob" multiple cores is not very important, because
"indexer -Eblob" currently cannot use multiple cores itself.
However, it's still useful - will improve performance for the
parallel processes (for example, searches).
RAM required for "indexer -Eblob" should be able to store at least
as reported in this query:
mysql> SELECT 2*Data_length/32 FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME='bdict' AND TABLE_SCHEMA=database();
+------------------+
| 2*Data_length/32 |
+------------------+
| 307824461.2500 |
+------------------+
1 row in set (0.01 sec)
By default "indexer -Eblob" processes all data in 32 pieces
(every bdicti.intagXX consequently).
By the way, please try a new feature:
parallel indexing of all bdicti.intagXX columns.
Add step=200000 parameter to DBAddr in indexer.conf.
In this mode, "indexer -Eblob" will load all 32 pieces
into memory at once, for 200000 documents in a single shot.
This indexing mode is faster. You can experiment on the "step"
value depending on amount of memory you have.
Or you can approximate a "step" value depending on how much
memory you want to aford for "indexer -Eblob":
mysql> SET @memory_limit=512*1024*1024; SELECT table_rows/(2*Data_length/@memory_limit) FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME='bdict' AND TABLE_SCHEMA=database();
Query OK, 0 rows affected (0.00 sec)
+------------------------------------------+
| table_rows/(2*Data_length/@memory_limit) |
+------------------------------------------+
| 299046.5162 |
+------------------------------------------+
1 row in set (0.01 sec)
So for my database with 1.5 million documents
I can try step=299046 value if I want to use 512Mb for
"indexer -Eblob".
> Or say beyond 1 million documents rather take
> two small server but use the cluster feature?
Cluster improves performance of all processes
(crawling, indexing, search) to N, where N is the number of
machines in the cluster. For the databases with more than
1 million documents I'd recommended to use cluster.
The important thing is that in case of cluster you also
distribute disk operations!
On the other hand, a modern multi-core machine
(with say 8 cores) and multiple disks (say 8 disks)
should be able to act with about the same performance
as 8 single-core single-disk machines. So I don't have
a definite answer here.
Reply: <http://www.mnogosearch.org/board/message.php?id=21179>