Re: PATCH: Performance issue with exclude_urls

On Thu, 22 Apr 2004, Gilles Detillieux wrote:

> Date: Thu, 22 Apr 2004 17:17:03 -0500 (CDT)
> From: Gilles Detillieux <[email protected]>
> To: "ht://Dig developers list" <[email protected]>
> Cc: Christopher Murtagh <[email protected]>
> Subject: Re: [htdig-dev] PATCH: Performance issue with exclude_urls
> 
> According to me:
> > According to Christopher Murtagh:
> > >  Then I re-compiled and ran with my normal excludes URL list. It didn't
> > > seem to have much of an impact on performance. This means that the
> > > performance hit is definitely in the HtRegexList::setEscaped method.
> > 
> > Thanks, Chris.  That's good to know!  Maybe then we don't need to use
> > Lachlan's patch to the config parser to track whether server or URL
> > blocks were defined.  I'll try a quick fix to Retriever.cc, but I'll
> > also try to find if there are other uses of HtRegexList that may need
> > attention.
> 
> Well, there are some uses of it in htsearch related code that I'm not
> all that sure about, but let's just cross that bridge when we get to it.
> In any case, the optimizations tend to need to be done right where the
> HtRegexList object is created anyway, rather than buried in the class
> definition, because you need to make sure the object sticks around or
> any optimization will have no effect.
> 
> So, here's my simple stab at fixing Retriever.cc, with no other files
> needing patches.  It should be used instead of Chris's and Lachlan's
> patches from earlier this week, not put on top of them.  I've built
> this patch over the current CVS code for Retriever.cc (as of Apr 7),
> but it does seem to apply to a vanilla 3.2.0b5 source as well.
> 
> Please test this out and make sure it doesn't cause any problems, and
> that it helps!  Apply using "patch -p0 << this-message-file".

I backed out both slightly-better.0 and exclude_perform.0 and applied
exclude_perform.1, this patch.  I ran htdig on the same site as before for
profile; htdig ran ~43% faster than the first time;)  Here is the profile:

 ftp://ftp.ccsf.org/htdig-patches/3.2.0b5/htdig.gmon.exclude_perform.1.gz

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        [email protected]


> --- htdig/Retriever.cc.orig	2004-04-07 17:02:00.000000000 -0500
> +++ htdig/Retriever.cc	2004-04-22 16:45:52.000000000 -0500
> @@ -995,10 +995,21 @@ int Retriever::IsValidURL(const String &
>  	// If the URL contains any of the patterns in the exclude list,
>  	// mark it as invalid
>  	//
> -	tmpList.Create(config->Find(&aUrl, "exclude_urls"), " \t");
> -	HtRegexList excludes;
> -	excludes.setEscaped(tmpList, config->Boolean("case_sensitive"));
> -	if (excludes.match(url, 0, 0) != 0)
> +	String exclude_urls = config->Find(&aUrl, "exclude_urls");
> +	static String *prevexcludes = 0;
> +	static HtRegexList *excludes = 0;
> +	if (!excludes || !prevexcludes || prevexcludes->compare(exclude_urls) != 0)
> +	{
> +		if (!excludes)
> +			excludes = new HtRegexList;
> +		if (prevexcludes)
> +			delete prevexcludes;
> +		prevexcludes = new String(exclude_urls);
> +		tmpList.Create(exclude_urls, " \t");
> +		excludes->setEscaped(tmpList, config->Boolean("case_sensitive"));
> +		tmpList.Destroy();
> +	}
> +	if (excludes->match(url, 0, 0) != 0)
>  	{
>  		if (debug > 2)
>  			cout << endl << "   Rejected: item in exclude list ";
> @@ -1009,12 +1020,22 @@ int Retriever::IsValidURL(const String &
>  	// If the URL has a query string and it is in the bad query list
>  	// mark it as invalid
>  	//
> -	tmpList.Destroy();
> -	tmpList.Create(config->Find(&aUrl, "bad_querystr"), " \t");
> -	HtRegexList badquerystr;
> -	badquerystr.setEscaped(tmpList, config->Boolean("case_sensitive"));
> +	String bad_querystr = config->Find(&aUrl, "bad_querystr");
> +	static String *prevbadquerystr = 0;
> +	static HtRegexList *badquerystr = 0;
> +	if (!badquerystr || !prevbadquerystr || prevbadquerystr->compare(bad_querystr) != 0)
> +	{
> +		if (!badquerystr)
> +			badquerystr = new HtRegexList;
> +		if (prevbadquerystr)
> +			delete prevbadquerystr;
> +		prevbadquerystr = new String(bad_querystr);
> +		tmpList.Create(bad_querystr, " \t");
> +		badquerystr->setEscaped(tmpList, config->Boolean("case_sensitive"));
> +		tmpList.Destroy();
> +	}
>  	char *ext = strrchr((char *) url, '?');
> -	if (ext && badquerystr.match(ext, 0, 0) != 0)
> +	if (ext && badquerystr->match(ext, 0, 0) != 0)
>  	{
>  		if (debug > 2)
>  			cout << endl << "   Rejected: item in bad query list ";
> 
> 
> -- 
> Gilles R. Detillieux              E-mail: <[email protected]>
> Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
> Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)



-------------------------------------------------------
This SF.net email is sponsored by: The Robotic Monkeys at ThinkGeek
For a limited time only, get FREE Ground shipping on all orders of $35
or more. Hurry up and shop folks, this offer expires April 30th!
http://www.thinkgeek.com/freeshipping/?cpg=12297
_______________________________________________
ht://Dig Developer mailing list:
[email protected]
List information (subscribe/unsubscribe, etc.)
https://lists.sourceforge.net/lists/listinfo/htdig-dev