Webboard: How to speed up the crawl delay after each URL ?
[email protected] Wed, 4 May 2016 19:32:47 +0400 (MSK)
| Newsgroups | gmane.comp.web.mnogosearch.general |
|---|---|
| Message-ID | <[email protected]> |
Author: rafikCyc Email: Message: Thank you for the reply. Well, You're right... With -P0 it does not have a limit of 1s. But it remain very slow though. -- I just did a quick speed test on a small site (500 documents) Mnogosearch VS screaming frog. The results : Mnogosearch : 3.2 urls / second Screaming Frog: 40 urls / second Same connection , same remote site, but 10 time faster :( Reply: <http://www.mnogosearch.org/board/message.php?id=21763> _______________________________________________ General mailing list [email protected] http://lists.mnogosearch.org/listinfo/general