How to ignore robots meta tag
[email protected] Thu, 20 Mar 2003 15:39:38 -0600
| Newsgroups | gmane.comp.web.perlfect-search |
|---|---|
| Message-ID | <[email protected]> |
How can I tell the indexer to ignore the robots meta tag? This is what I have in each of my pages I wish to index: meta name="ROBOTS" content="NOINDEX, NOFOLLOW" I saw in the config file where you can tell it to ignore robots.txt and did so, but it is still honoring the robots meta tag. I am running the indexer on the local filesystem of a site which I want to keep out of public search engines, but I would like site search to be available. I looked around through the FAQ and archives to find information about this but came up with nothing, sorry if I missed something. Thanks, Rob _______________________________________________ perlfect-search mailing list [email protected] To unsubscribe, set other personal options or view the list archives please visit: http://perlfect.com/mailman/listinfo/perlfect-search