Indexing with http
"Cesareo, Craig" <[email protected]>
| Newsgroups | gmane.comp.web.perlfect-search |
|---|---|
| Message-ID | <A7BB7F749788D311B3BB009027D08DD502687D90@ITAEXCHANGE> |
Hi all, Another question about indexing my web site: I run the indexer locally on the server (static html pages), but am testing to see if I can index some of the dynamic content on my site via the http option. I have a content area where the html pages are produced by a custom cgi application. For example, the url used to pull up any of the dynamic pages looks something like this: http://www.mysite.com/cgi-bin/dynamic.exe?func=1123doc=88 <http://www.mysite.com/cgi-bin/dynamic.exe?func=1123doc=88> (this is just a fake example) But there are no direct links to these dynamic page urls on my site anywhere. So, I created a basic html page that contains nothing but hyperlinks to the exact urls of the dynamic pages. I setup the http start_url to be the location of this html page I created on the server. I thought that the index process would hit this html page, and crawl out to the urls that are in the page and index each of them. But it is not doing this. Am I missunderstanding how the http indexing works? Is there a problem because I can't set the filename extension for documents that should be indexed in the conf.pl file - because the urls to the dynamic pages on my site do not end in something like ".php" or ".asp"? Or anyone have any tips as far as what I may be doing wrong? Thanks very much!! Craig