Excluding robots and crawlers from proxy server

Hi All,

I have a website on a windows server, with Apache web server
used as a proxy server to Tomcat.  There is a bunch of JSPs
in the web application.  Twice or thrice a year, the website
gets hammered with search engine requests 
(I have found at least google, msn, yahoo).
This bring down the website - any user requests
returns after a little while with a Proxy Error.

Simply putting a robot.txt file in the document root
of the apache web server is not working, I assume
because the requests are proxied to Tomcat and
not fulfilled by Apache.  I tried putting the meta tag
in the controller JSP as follows, but it did not work.

  <META NAME="ROBOTS" CONTENT="NOINDEX">

Can anyone help with any ideas how to stop the 
search engine from crawling our website.

Thanks,

- Pankaj


Pankaj Agarwal
Duke Bioinformatics Shared Resource
Duke Comprehensive Cancer Center
Duke University
919-681-6573
[email protected]

_______________________________________________
Juglist mailing list
[email protected]
http://trijug.org/mailman/listinfo/juglist_trijug.org