Re: Profile

On Sat, 17 Apr 2004, Lachlan Andrew wrote:

> Date: Sat, 17 Apr 2004 00:20:07 +1000
> From: Lachlan Andrew <[email protected]>
> To: Joe R. Jah <[email protected]>
> Cc: [email protected]
> Subject: Profile
> 
> Thanks, Joe :)
> 
> There are some unexpected things about this profile.  In particular, 
> it seems that some files are not being profiled.
> 
> * regcomp is taking *much* too much time!  As I understand, regular
>   expressions are being used for things which really "should" be just
>   straight string compares.  I vote that we go back to doing standard
>   compares rather than create an escaped string and then call an
>   expensive regex function...
> 
> * Rather than 'fork'ing a separate external parser each time, could
>   we look at having a "persistent parser", like a persistent TCP
>   connection?  It would require a way of specifying the end of one
>   file and the start of the next, but it looks like the performance
>   gain might be worthwhile.
> 
> * regcomp, calloc and gethostbyname all seem to be being called a lot
>   more often than  gprof  recognises.
> 
> * gethostbyname  seems to be too expensive.  Joe, were you using
>   "persistent connections"?  Gabriele, could we reduce the number
>   of times  gethostbyname  is called?  Perhaps we could cache the
>   names?

That's the default, and I did not override it in my configuration file.
By the way, all indexed documents were on the same server as htdig.

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        [email protected]

> * None of the functions in  Connection.cc  seem to be profiled.
>   Any ideas why?
> 
> Thanks all,
> Lachlan
> 
> On Wed, 14 Apr 2004 14:36, Joe R. Jah wrote:
> 
> > I compiled htdig-3.2.0b5 with -pg; the following patches applied:
> > DESTDIR.0 TMPFILE.0 extension_filter.0 fileSpace.0 operator[].0 and
> > robots.0
> >
> > I ran htdig on ~13k documents; it ran about 40% slower than my
> > regular htdig, (without -pg).  I ran gprof htdig > htdig.gmon;gzip
> > htdig.gmon, and put the profile on the patch site, although it's
> > not a patch;)
> >
> >   ftp://ftp.ccsf.org/htdig-patches/3.2.0b5/htdig.gmon.gz
> >
> > Hope it can help in improving htdig performance.
> >
> > Regards,
> >
> > Joe



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
ht://Dig Developer mailing list:
[email protected]
List information (subscribe/unsubscribe, etc.)
https://lists.sourceforge.net/lists/listinfo/htdig-dev