Re: Correct way to block web bots and other unwanted traffic?

In the past, I've seen references to setting the robots.txt file to
disallow access to a non-existent path.  Then running a process to monitor
the access_log for folks who to try to view the path and temporarily
dropping a 'block' rule into the firewall for the offending IP address.

---
  /\---/\  Eric J Fox
 /  o o  \ http://fox.phoenix.az.us/
 \.\   /./ ---------------------------
    \@/    "Of course it runs NetBSD."


On Tue, 5 Dec 2006, Gilbert Fernandes wrote:

> On Tue, Dec 05, 2006 at 07:16:20AM -0700, Andy Ruhl wrote:
>
> > So my question is, what's the right way to block these guys?
>
> insert this into a robots.txt file :
>
> User-agent: *
> Disallow:/
>
> upload to the root folder of your web server.
>
> most will comply but some wont. you can either insert rules
> into your firewall to only allow your friends (if they have
> static ip for example) or require authentification.
>
> --
> unzip ; strip ; touch ; grep ; find ; finger ; mount ; fsck ; more ;
> yes ; fsck ; umount ; sleep
>