Re: Next major speedup

On Sat, 1 May 2004, Lachlan Andrew wrote:

> Date: Sat, 1 May 2004 10:44:15 +1000
> From: Lachlan Andrew <[email protected]>
> To: Joe R. Jah <[email protected]>
> Cc: [email protected]
> Subject: Next major speedup
> 
> Greetings all,
> 
> Thanks for the new timings, Joe.  It's good to see that it is a bit 
> faster than before exclude_perform.1 was applied.  (Any idea why it
> was so slow last time?)

My high expectations not being fulfilled, caused me to kill the process
prematurely and over-extrapolate;(

> It looks like we still have a lot of work to do to get performance 
> like 3.1.6!!
> 
> I've finally managed to get some profiles that I trust, by linking 
> statically (although they still don't match Joe's).  It showed that 
> flushing the words in DocumentRef::AddDescription() was taking half 
> of the time.  Does anyone know why that was done?
> 
> I've commented out the offending flush (see attached patch) and the 
> database compares identical to the previous, but it takes about 1/3 
> of the CPU time.  I didn't measure wall-clock time, but I expect a 
> similar improvement, since the database isn't being thrashed so much.
> 
> Could people please check over (or test) the patch?

htdig-3.2.0b5:

  Total dig time:  01:37:35 == 5255 seconds

With exclude_perform.1

  Total dig time:  00:57:35 == 3455 seconds, or ~34% less time

With exclude_perform.1 and store_phrases.0
  true:
  Total dig time:  01:00:17 == 3617 seconds, or ~31% less time
  false:
  Total dig time:  00:49:01 == 2941 seconds, or ~44% less time

With exclude_perform.1 and noflush.0
  Total dig time:  00:55:53 == 3353 seconds, or ~36% less time

htdig-3.1.6:

  Total dig time:  00:14:59 == 899 seconds, or ~83% less time

Here is the profile:
ftp://ftp.ccsf.org/htdig-patches/3.2.0b5/0Profiles/htdig.gmon.exclude_perform.1-noflush.0.gz

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        [email protected]



-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE. 
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
ht://Dig Developer mailing list:
[email protected]
List information (subscribe/unsubscribe, etc.)
https://lists.sourceforge.net/lists/listinfo/htdig-dev