Re: Solved! - Re: Performance issue with exclude_urls

On Tue, 20 Apr 2004, Christopher Murtagh wrote:

> Date: Tue, 20 Apr 2004 00:03:01 -0400
> From: Christopher Murtagh <[email protected]>
> To: [email protected]
> Cc: [email protected]
> Subject: [htdig-dev] Solved! - Re: Performance issue with exclude_urls
> 
>  Ok, I've solved my problem, and can now have a list of working
> exclude_urls without the serious performance decrease. Here are the
> changes I made (sorry I'm not sending a proper diff file... need
> guidance on how to do that properly):
> 
> 
> htdig/htdig.h
> --------------------
> 
> added:
> 
> extern int exclude_checked;
> extern int badquerystr_checked;
> extern HtRegexList  excludes;
> extern HtRegexList  badquerystr;
> 
> 
> 
> htdig/htdig.cc
> ----------------------
> 
> added these as global variable definitions:
> 
> int exclude_checked = 0;
> int badquerystr_checked = 0;
> 
> HtRegexList     excludes;
> HtRegexList     badquerystr;
> 
> 
> htdig/Retriever.cc
> 
> added these conditionals and removed the previous tmplist creates and
> .setEscaped() calls:
> 
> if(!(exclude_checked)){
>     //only parse this once and store into global variable
>     tmpList.Destroy();
>     tmpList.Create(config->Find(&aUrl, "exclude_urls"), " \t");
>     excludes.setEscaped(tmpList, config->Boolean("case_sensitive"));
>     exclude_checked = 1;
> }
> 
> if(!(badquerystr_checked)){
>     //only parse this once and store into global variable
>     tmpList.Destroy();
>     tmpList.Create(config->Find(&aUrl, "bad_querystr"), " \t");
>     badquerystr.setEscaped(tmpList, config->Boolean("case_sensitive"));
>     badquerystr_checked = 1;
> }
> 
>  The difference in performance is night and day, and the excludes list
> is only parsed once per dig rather than at *every* URL found.
> 
>  If this is at all useful to anyone, let me know. I can send files or if
> someone would enlighten me (even RTFM me) I can send diff/patches.
> 
> Cheers,
> 
> Chris

Let us call the original files file.orig; do this:

diff -u htdig.h.orig htdig.h > exclude_perform.0
diff -u htdig.cc.orig htdig.cc >> exclude_perform.0
diff -u Retriever.cc.orig Retriever.cc >> exclude_perform.0

If your diff command does not accept "-u" use "-c" instead.

Attach exclude_perform.0 to an email to the list, or if you prefer ftp it
to:

  ftp://ftp.ccsf.org/incoming

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        [email protected]



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
ht://Dig Developer mailing list:
[email protected]
List information (subscribe/unsubscribe, etc.)
https://lists.sourceforge.net/lists/listinfo/htdig-dev