IGNORE TEXT not working
"Jon Ohmann" <[email protected]>
| Newsgroups | gmane.comp.web.perlfect-search |
|---|---|
| Organization | GetAGeek, Inc. |
| Message-ID | <006e01c2c870$e353d690$0400a8c0@GEEKMAIN> |
My site consists of dynamically created pages (using JSP). These pages are all coming through a central shell page that embeds the content appropriately. We have used the <!--ignore_perlfect_search--><!--/ignore_perlfect_search--> extensively to keep the same URL links from being indexed, but they do not seem to work. Links within these tags (as well as HTML comment tags) are crawled by the indexer. (we are initiating the indexer via HTTP). The issue here is that it ends up nesting too deep for the indexer to complete all pages before running into the HTTP_MAX_PAGES limit (currently set to 800). To make it clear...the inital page (index2.1.jsp) has a link to dynamic pages (JSPID=12345 and 54321). To display these pages, the URL is: index2.1.jsp?JSPID=12345 or index2.1.jsp?JSPID=84321. These pages in turn have dynamic links as well. The indexer crawls to index2.1.jsp, then to the first dynamic document which is really index2.1.jsp with new links. It then crawls to these 'sub links' and so on and so on. Soon we are many levels deep when in reality, I don't want it to continue repeating the same initial page links (these are carried through to the next page) We have these repeating links in the ignore text, but the indexer seems to disregard the tag and crawl anyway. What are we doing wrong?