RE: Two queries

Mike,

Thanks a lot for getting back to me and providing some useful info 
here.  You are right in assuming I am working on an intranet site.

For (i), there doesn't seem to be an easy way of inserting robots tags as I 
think these are the automatically generated files by Apache to give the 
directory structure that htdig can follow.  If I stop htdig looking at the 
files, will I also lose the indexing for the directories they 
represent?  Is there a way of telling Apache to enter a relevant code in 
the file as it generated to get htdig to not display it as part of the results?

With regard to (ii), I think I may pursue the options you suggest with a 
non-htdig solution.  However, giving users just one interface to check for 
files may be an advantage, which is why I think I would still like to try 
an htdig approach if it were easily possible.  Unfortunately too few 
documents would have a consistent string that could be used to search for.

Cheers,
Richard

At 12:37 2006-01-16 +0000, you wrote:
>Richard,
>I think that both of these are solvable.
>(i) If you have access to modify these indexes, then try adding a
><robots noindex,follow> tag (check the syntax, I'm sure that's not
>correct)
>or try adding <!-- htdig no-index --> tags, provided you have links
>elsewhere.
>I believe that such indexes are served as a redirect to:
>whatever/folder/index.html  in which case you should be able to set up
>an exclude for them, again assuming that you don't need to index their
>links.
>
>(ii) The first thing that comes to mind is to implement this without
>htdig. PERL, PHP or Java would all be able to produce what is
>effectively just a directory listing, and if carefully written should be
>more efficient than an htdig search.
>Second idea would be to try and find a common term, such as your site
>name, that appears on every page. If this is a public site then that
>should normally be added to the bad-words list, but this sounds like an
>Intranet?
>
>
>Good luck,
>Mike



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
ht://Dig general mailing list: <[email protected]>
ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html
List information (subscribe/unsubscribe, etc.)
https://lists.sourceforge.net/lists/listinfo/htdig-general