Re: Problem with htdig indexation
"Marcelo García" <[email protected]> Wed, 5 Mar 2008 11:39:50 +0100
| Newsgroups | gmane.comp.web.htdig.general |
|---|---|
| Message-ID | <[email protected]> |
Hello,
Thanks for your response, this is the http.conf file:
# $Id: htdig.conf,v 1.6 2007/06/27 13:12:14 migonzalez Exp $
#
# Example config file for ht://Dig.
#
# This configuration file is used by all the programs that make up ht://Dig.
# Please refer to the attribute reference manual for more details on what
# can be put into this file. (See http://www.htdig.org/confindex.html or
# locally at file:/usr/share/doc/htdig-doc/htmlconfindex.html)
# Note that most attributes have very reasonable default values so you
# really only have to add attributes here if you want to change the default=
s.
#
# What follows are some of the common attributes you might want to change.
#
#
# Specify where the database files need to go. Make sure that there is
# plenty of free disk space available for the databases. They can get
# pretty big.
#
database_dir: /var/lib/htdig
#
# This specifies the URL where the robot (htdig) will start. You can speci=
fy
# multiple URLs here. Just separate them by some whitespace.
# The example here will cause the ht://Dig homepage and related pages to be
# indexed.
# You could also index all the URLs in a file like so:
# start_url: `${common_dir}/start.url`
#
start_url: http://localhost/manuales2/manuales2.html
http://localhost/proyectos/proyectos.html http://localhost/indices
#
# This attribute limits the scope of the indexing process. The default is =
to
# set it to the same as the start_url above. This way only pages that are =
on
# the sites specified in the start_url attribute will be indexed and it will
# reject any URLs that go outside of those sites.
#
# Keep in mind that the value for this attribute is just a list of string
# patterns. As long as URLs contain at least one of the patterns it will be
# seen as part of the scope of the index.
#
limit_urls_to: ${start_url} http://petete/
#
# If there are particular pages that you definitely do NOT want to index, y=
ou
# can use the exclude_urls attribute. The value is a list of string patter=
ns.
# If a URL matches any of the patterns, it will NOT be indexed. This is
# useful to exclude things like virtual web trees or database accesses. By
# default, all CGI URLs will be excluded. (Note that the /cgi-bin/ convent=
ion
# may not work on your web server. Check the path prefix used on your web
# server.)
#
exclude_urls: /cgi-bin/ .cgi /images
#
# Since ht://Dig does not (and cannot) parse every document type, this
# attribute is a list of strings (extensions) that will be ignored during
# indexing. These are *only* checked at the end of a URL, whereas
# exclude_url patterns are matched anywhere.
#
bad_extensions: .png .wav .gz .z .bz2 .sit .au .zip .tar .hqx .exe .com \
.gif .jpg .jpeg .aiff .class .map .ram .tgz .bin .rpm .mpg .mov .avi .css
#
# The string htdig will send in every request to identify the robot. Change
# this to your email address.
#
maintainer: [email protected] =
#
# The excerpts that are displayed in long results rely on stored information
# in the index databases. The compiled default only stores 512 characters =
of
# text from each document (this excludes any HTML markup...) If you plan on
# using the excerpts you probably want to make this larger. The only conce=
rn
# here is that more disk space is going to be needed to store the additional
# information. Since disk space is cheap (! :-)) you might want to set this
# to a value so that a large percentage of the documents that you are going
# to be indexing are stored completely in the database. At SDSU we found
# that by setting this value to about 50k the index would get 97% of all
# documents completely and only 3% was cut off at 50k. You probably want to
# experiment with this value.
# Note that if you want to set this value low, you probably want to set the
# excerpt_show_top attribute to false so that the top excerpt_length charac=
ters
# of the document are always shown.
#
max_head_length: 10000
#
# To limit network connections, ht://Dig will only pull up to a certain lim=
it
# of bytes. This prevents the indexing from dying because the server keeps
# sending information. However, several FAQs happen because people have fil=
es
# bigger than the default limit of 100KB. This sets the default a bit highe=
r.
# (see <http://www.htdig.org/FAQ.html> for more)
#
max_doc_size: 200000
#
# Most people expect some sort of excerpt in results. By default, if the
# search words aren't found in context in the stored excerpt, htsearch shows
# the text defined in the no_excerpt_text attribute:
# (None of the search words were found in the top of this document.)
# This attribute instead will show the top of the excerpt.
#
no_excerpt_show_top: true
# Miqui
locale: es_ES
case_sensitive: false
# Factores
# Por defecto:
#title_factor: 100
#keywords_factor: 100
#meta_description_factor: 50
#heading_factor_1: 5
#heading_factor_2: 4
#heading_factor_3: 3
#heading_factor_4: 1
#heading_factor_5: 1
#heading_factor_6: 0
#text_factor: 1
title_factor: 100
keywords_factor: 500
meta_description_factor: 50
heading_factor: 10
text_factor: 1
#
# Depending on your needs, you might want to enable some of the fuzzy search
# algorithms. There are several to choose from and you can use them in any
# combination you feel comfortable with. Each algorithm will get a weight
# assigned to it so that in combinations of algorithms, certain algorithms =
get
# preference over others. Note that the weights only affect the ranking of
# the results, not the actual searching.
# The available algorithms are:
# accents
# exact
# endings
# metaphone
# prefix
# soundex
# substring
# synonyms
# By default only the "exact" algorithm is used with weight 1.
# Note that if you are going to use the endings, metaphone, soundex, accent=
s,
# or synonyms algorithms, you will need to run htfuzzy to generate
# the databases they use.
#
search_algorithm: exact:1 prefix:0.5 accents:0.2
#
# The following are the templates used in the builtin search results
# The default is to use compiled versions of these files, which produces
# slightly faster results. However, uncommenting these lines makes it
# very easy to change the format of search results.
# See <http://www.htdig.org/hts_templates.html> for more details.
#
# template_map: Long long ${common_dir}/long.html \
# Short short ${common_dir}/short.html
# template_name: long
#
# The following are used to change the text for the page index.
# The defaults are just boring text numbers. These images spice
# up the result pages quite a bit. (Feel free to do whatever, though)
#
next_page_text: <img src=3D"/htdig/buttonr.gif" border=3D"0"
align=3D"middle" width=3D"30" height=3D"30" alt=3D"next">
no_next_page_text:
prev_page_text: <img src=3D"/htdig/buttonl.gif" border=3D"0"
align=3D"middle" width=3D"30" height=3D"30" alt=3D"prev">
no_prev_page_text:
page_number_text: '<img src=3D"/htdig/button1.gif" border=3D"0"
align=3D"middle" width=3D"30" height=3D"30" alt=3D"1">' \
'<img src=3D"/htdig/button2.gif" border=3D"0" align=3D"middle" width=3D"=
30"
height=3D"30" alt=3D"2">' \
'<img src=3D"/htdig/button3.gif" border=3D"0" align=3D"middle" width=3D"=
30"
height=3D"30" alt=3D"3">' \
'<img src=3D"/htdig/button4.gif" border=3D"0" align=3D"middle" width=3D"=
30"
height=3D"30" alt=3D"4">' \
'<img src=3D"/htdig/button5.gif" border=3D"0" align=3D"middle" width=3D"=
30"
height=3D"30" alt=3D"5">' \
'<img src=3D"/htdig/button6.gif" border=3D"0" align=3D"middle" width=3D"=
30"
height=3D"30" alt=3D"6">' \
'<img src=3D"/htdig/button7.gif" border=3D"0" align=3D"middle" width=3D"=
30"
height=3D"30" alt=3D"7">' \
'<img src=3D"/htdig/button8.gif" border=3D"0" align=3D"middle" width=3D"=
30"
height=3D"30" alt=3D"8">' \
'<img src=3D"/htdig/button9.gif" border=3D"0" align=3D"middle" width=3D"=
30"
height=3D"30" alt=3D"9">' \
'<img src=3D"/htdig/button10.gif" border=3D"0" align=3D"middle"
width=3D"30" height=3D"30" alt=3D"10">'
#
# To make the current page stand out, we will put a border around the
# image for that page.
#
no_page_number_text: '<img src=3D"/htdig/button1.gif" border=3D"2"
align=3D"middle" width=3D"30" height=3D"30" alt=3D"1">' \
'<img src=3D"/htdig/button2.gif" border=3D"2" align=3D"middle" width=3D"=
30"
height=3D"30" alt=3D"2">' \
'<img src=3D"/htdig/button3.gif" border=3D"2" align=3D"middle" width=3D"=
30"
height=3D"30" alt=3D"3">' \
'<img src=3D"/htdig/button4.gif" border=3D"2" align=3D"middle" width=3D"=
30"
height=3D"30" alt=3D"4">' \
'<img src=3D"/htdig/button5.gif" border=3D"2" align=3D"middle" width=3D"=
30"
height=3D"30" alt=3D"5">' \
'<img src=3D"/htdig/button6.gif" border=3D"2" align=3D"middle" width=3D"=
30"
height=3D"30" alt=3D"6">' \
'<img src=3D"/htdig/button7.gif" border=3D"2" align=3D"middle" width=3D"=
30"
height=3D"30" alt=3D"7">' \
'<img src=3D"/htdig/button8.gif" border=3D"2" align=3D"middle" width=3D"=
30"
height=3D"30" alt=3D"8">' \
'<img src=3D"/htdig/button9.gif" border=3D"2" align=3D"middle" width=3D"=
30"
height=3D"30" alt=3D"9">' \
'<img src=3D"/htdig/button10.gif" border=3D"2" align=3D"middle"
width=3D"30" height=3D"30" alt=3D"10">'
minimum_word_length: 2
extra_words_characters: _:-
allow_numbers: true
bad_word_list: /etc/htdig/bad_words
#
# These files don't belong to /etc/htdig
#
synonym_db: /usr/lib/htdig/synonyms.db
endings_root2word_db: /usr/lib/htdig/root2word.db
endings_word2root_db: /usr/lib/htdig/word2root.db
#
# image_url_prefix is the relative url for the images htdig uses.
# This defaults to "/doc/htdig/images". To get the images from another dir
# if you've turned off the /doc alias, you'll need to modify the .html files
# in /etc/htdig as well as this setting.
#
image_url_prefix: /htdig
#
# external parser for msword, postscript and pdf files
#
# If you really want to parse pdf, ps and doc files, make sure that the
# max_doc_size value above is set to a larger value than your largest file
# of this type. If it is smaller, the indexing might hang
#
#external_parsers: application/msword /usr/share/htdig/parse_doc.pl \
# application/postscript /usr/share/htdig/parse_doc.pl \
# application/pdf /usr/share/htdig/parse_doc.pl
#
# special Debian variable for PDF documents
#
# If you want to parse PDF documents, set the (htdig variable) pdf_parser
# to the wrapper script /usr/bin/htdig-pdfparser (which is default). Using
# the debian_pdf_parser variable the actual pdf parser can be controlled.
# Make sure you set the external_parsers option too.
# Recognized options are: acrobat, xpdf
#
#debian_pdf_parser: acrobat
search_rewrite_rules: http://localhost/(.*) http://petete/\\1
# local variables:
# mode: text
# eval: (if (eq window-system 'x) (progn (setq font-lock-keywords
(list '("^#.*" . font-lock-keyword-face) '("^[a-zA-Z][^ :]+" .
font-lock-function-name-face) '("[+$]*:" . font-lock-comment-face) ))
(font-lock-mode)))
# end:
#############################################################
Regards,
Marcelo
2008/3/4, [email protected] <[email protected]>:
> Marcelo,
> The limit_urls_to:
> line looks a bit iffy - is this exactly how it looks in your config file=
? (It might be easier if you attached the whole file)
>
> Given that the number of documents is increasing, it rather looks as tho=
ugh there is some limit on how far the indexing goes each time, which is th=
en being built upon.
>
> Regards,
> Mike
>
>
> >-----Original Message-----
> >From: [email protected]
> >[mailto:[email protected]] On Behalf
> >Of Marcelo Garc=EDa
> >Sent: Tuesday, March 04, 2008 3:29 PM
> >To: [email protected]
> >Subject: [htdig] Problem with htdig indexation
> >
> >Hello,
> >
> >I am a new user of htdig and I have a problem with indexation,
> >My server is runing with Ubuntu 7, with htdig 3.2.0b6, the
> >documents to index are in one server only. The server name is petete.
> >
> >The number of documents found by htdig change every time I run
> >rundig (rundig -a -s) and because of this some documents are
> >not found when we search them.
> >
> >Example:
> >First execution of rundig -a -s:
> >htdig: localhost:80 24 documents
> >
> >Second:
> >htdig: 2 servers seen:
> >htdig: localhost:80 24 documents
> >htdig: petete:80 37 documents
> >
> >3.
> >htdig: 2 servers seen:
> >htdig: localhost:80 24 documents
> >htdig: petete:80 326 documents
> >
> >4.
> >htdig: 2 servers seen:
> >htdig: localhost:80 24 documents
> >htdig: petete:80 409 documents
> >
> >5.
> >htdig: 2 servers seen:
> >htdig: localhost:80 24 documents
> >htdig: petete:80 438 documents
> >This is the max number of documents found, after some time
> >(one day more or less), if we repeat the process we have the
> >same results.
> >
> >In htdig.conf we have (parameters that I think could affect to
> >indexation):
> >start_url: http://localhost/manuales2/manuales2.html
> >http://localhost/proyectos/proyectos.html http://localhost/indices
> >
> >limit_urls_to: ${start_url} http://petete/
> >
> >exclude_urls: /cgi-bin/ .cgi /images
> >
> >search_rewrite_rules: http://localhost/(.*) http://petete/\\1
> >
> >Please, can you help me with this problem?
> >
> >Thanks in advance.
> >
>
> >---------------------------------------------------------------
> >----------
> >This SF.net email is sponsored by: Microsoft Defy all
> >challenges. Microsoft(R) Visual Studio 2008.
> >http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> >_______________________________________________
> >ht://Dig general mailing list: <[email protected]>
> >ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html
> >List information (subscribe/unsubscribe, etc.)
> >https://lists.sourceforge.net/lists/listinfo/htdig-general
> >
>
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
ht://Dig general mailing list: <[email protected]>
ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html
List information (subscribe/unsubscribe, etc.)
https://lists.sourceforge.net/lists/listinfo/htdig-general