Re: Problems with 22000 plus file crawl with large file sizes
Daniel Naber <[email protected]> Sat, 8 Nov 2003 13:41:24 +0100
| Newsgroups | gmane.comp.web.perlfect-search |
|---|---|
| Message-ID | <[email protected]> |
On Saturday 08 November 2003 10:49, James Garrett wrote: > Setting $LOW_MEMORY_INDEX =1; is just so painfully slow. Found reference > to $FLUSH_FREQUENCY =100; but couldn't fig how to patch ./indexer.pl > and don't really know if that will help at all. Also am afraid temp > files could grow hugh. Such a thing as a 2gig max file size in Linux? > Perlfect ? In a nutshell, you will probably want to use a different search engine that scales better. Try htdig or Lucene (Lucene is only a backend and requires Java programming). The $FLUSH_FREQUENCY patch probably won't help enough, the original posting talks about a 33% increase in indexing speed. Regards Daniel -- http://www.danielnaber.de _______________________________________________ perlfect-search mailing list [email protected] To unsubscribe, set other personal options or view the list archives please visit: http://perlfect.com/mailman/listinfo/perlfect-search