IGNORE TEXT not working

"Jon Ohmann" <[email protected]>
Newsgroups gmane.comp.web.perlfect-search
Organization GetAGeek, Inc.
Message-ID <006e01c2c870$e353d690$0400a8c0@GEEKMAIN>
My site consists of dynamically created pages (using JSP).  These pages
are all coming through a central shell page that embeds the content
appropriately.  We have used the
<!--ignore_perlfect_search--><!--/ignore_perlfect_search--> extensively
to keep the same URL links from being indexed, but they do not seem to
work.  Links within these tags (as well as HTML comment tags) are
crawled by the indexer.  (we are initiating the indexer via HTTP).  The
issue here is that it ends up nesting too deep for the indexer to
complete all pages before running into the HTTP_MAX_PAGES limit
(currently set to 800).  
 
To make it clear...the inital page (index2.1.jsp) has a link to dynamic
pages (JSPID=12345 and 54321).  To display these pages, the URL is:
index2.1.jsp?JSPID=12345 or index2.1.jsp?JSPID=84321.  These pages in
turn have dynamic links as well.  The indexer crawls to index2.1.jsp,
then to the first dynamic document which is really index2.1.jsp with new
links.  It then crawls to these 'sub links' and so on and so on. Soon we
are many levels deep when in reality, I don't want it to continue
repeating the same initial page links (these are carried through to the
next page)
 
We have these repeating links in the ignore text, but the indexer seems
to disregard the tag and crawl anyway.
 
What are we doing wrong?
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.