Indexing with http

"Cesareo, Craig" <[email protected]>
Newsgroups gmane.comp.web.perlfect-search
Message-ID <A7BB7F749788D311B3BB009027D08DD502687D90@ITAEXCHANGE>
Hi all,
 
Another question about indexing my web site:
 
I run the indexer locally on the server (static html pages), but am testing
to see if I can index some of the dynamic content on my site via the http
option. I have a content area where the html pages are produced by a custom
cgi application. For example, the url used to pull up any of the dynamic
pages looks something like this:
 
http://www.mysite.com/cgi-bin/dynamic.exe?func=1123doc=88
<http://www.mysite.com/cgi-bin/dynamic.exe?func=1123doc=88> 
(this is just a fake example)
 
But there are no direct links to these dynamic page urls on my site
anywhere. So, I created a basic html page that contains nothing but
hyperlinks to the exact urls of the dynamic pages. I setup the http
start_url to be the location of this html page I created on the server. I
thought that the index process would hit this html page, and crawl out to
the urls that are in the page and index each of them. But it is not doing
this.
 
Am I missunderstanding how the http indexing works? Is there a problem
because I can't set the filename extension for documents that should be
indexed in the conf.pl file - because the urls to the dynamic pages on my
site do not end in something like ".php" or ".asp"? Or anyone have any tips
as far as what I may be doing wrong?
 
Thanks very much!!
 
Craig
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.