Re: Performance issue with exclude_urls
Lachlan Andrew <[email protected]>
| Newsgroups | gmane.comp.web.htdig.devel |
|---|---|
| Organization | sourceforge |
| Message-ID | <[email protected]> |
Greetings Chris, What are the actual 30 patterns that cause problems? It could well be a single pattern which is giving the regular-expression handler grief. Cheers, Lachlan On Thu, 8 Apr 2004 01:52, Christopher Murtagh wrote: > In my search.conf my exculde_urls is set to: > exclude_urls: `/usr/local/htdig/common/excludeURL/exclude_urls` > > However, I've noticed that adding URLs *seriously* degrades > digging performance, to the point that with 30 or so patterns, I got > 8k pages in over 8 hours, and without them, I could do 15k pages in > an hour. > > With such a drastic difference, I'm assuming that there's a bug > somewhere. -- [email protected] ht://Dig developer DownUnder (http://www.htdig.org) ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click _______________________________________________ ht://Dig Developer mailing list: [email protected] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev