reparsing (Was: Performance issue with exclude_urls)

Greetings Chris and team,

Profiling should help a lot here.

One possibility is that it is because ht://Dig now allows attributes 
to be specified on a per-host basis.  As a result, all of the 
exclude_urls patterns are re-compiled for each url checked (even if 
you don't specify host-dependent attributes).  If (for example) there 
is a memory leak in the regex compiler, this would slow the code down 
as you describe.

Even if it is not causing Chris's problem, I've been thinking for a 
long time that this re-parsing may partially account for the big 
slowdown in digging with recent 3.2.0 betas.  From speaking with 
Gabriele, I know that he finds the feature very useful.  However, I 
can't find the config file format documented, so I don't think that 
many people can benefit from it, and it is currently just 
(unquantified) bloat.

If Chris finds that that is in fact the problem, I suggest that:
1. We set a flag if per-host attributes are used at all
2. If no per-host attributes are used, all expensive-to-parse
   attributes (like regular expressions) should be cached in their
   parsed forms.

Thoughts?

Lachlan

On Thu, 8 Apr 2004 01:52, Christopher Murtagh wrote:

> I've noticed that adding URLs *seriously* degrades
> digging performance. To a point that with 30 or so patterns, I got
> 8k pages in over 8 hours, and without them, I could do 15k pages in
> an hour.
>
>  With such a drastic difference, I'm assuming that there's a bug
> somewhere. I'll try to go digging through the code to find it, but
> I imagine that someone on this list will have better luck than me.
> :-)

-- 
[email protected]
ht://Dig developer DownUnder  (http://www.htdig.org)


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click
_______________________________________________
ht://Dig Developer mailing list:
[email protected]
List information (subscribe/unsubscribe, etc.)
https://lists.sourceforge.net/lists/listinfo/htdig-dev