Profile

Thanks, Joe :)

There are some unexpected things about this profile.  In particular, 
it seems that some files are not being profiled.

* regcomp is taking *much* too much time!  As I understand, regular
  expressions are being used for things which really "should" be just
  straight string compares.  I vote that we go back to doing standard
  compares rather than create an escaped string and then call an
  expensive regex function...

* Rather than 'fork'ing a separate external parser each time, could
  we look at having a "persistent parser", like a persistent TCP
  connection?  It would require a way of specifying the end of one
  file and the start of the next, but it looks like the performance
  gain might be worthwhile.

* regcomp, calloc and gethostbyname all seem to be being called a lot
  more often than  gprof  recognises.

* gethostbyname  seems to be too expensive.  Joe, were you using
  "persistent connections"?  Gabriele, could we reduce the number
  of times  gethostbyname  is called?  Perhaps we could cache the
  names?

* None of the functions in  Connection.cc  seem to be profiled.
  Any ideas why?

Thanks all,
Lachlan

On Wed, 14 Apr 2004 14:36, Joe R. Jah wrote:

> I compiled htdig-3.2.0b5 with -pg; the following patches applied:
> DESTDIR.0 TMPFILE.0 extension_filter.0 fileSpace.0 operator[].0 and
> robots.0
>
> I ran htdig on ~13k documents; it ran about 40% slower than my
> regular htdig, (without -pg).  I ran gprof htdig > htdig.gmon;gzip
> htdig.gmon, and put the profile on the patch site, although it's
> not a patch;)
>
>   ftp://ftp.ccsf.org/htdig-patches/3.2.0b5/htdig.gmon.gz
>
> Hope it can help in improving htdig performance.
>
> Regards,
>
> Joe

-- 
[email protected]
ht://Dig developer DownUnder  (http://www.htdig.org)


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click
_______________________________________________
ht://Dig Developer mailing list:
[email protected]
List information (subscribe/unsubscribe, etc.)
https://lists.sourceforge.net/lists/listinfo/htdig-dev