Re: Correct way to block web bots and other unwanted traffic?

I like the idea of dynamically blocking the single IP for a period of time.

I generally think blocking an entire network is a *bad thing*, 
considering that any number of potential 'good' users could be on that 
as well.

Andy Ruhl wrote:
> On 12/5/06, Gilbert Fernandes <[email protected]> wrote:
>> insert this into a robots.txt file :
>>
>> User-agent: *
>> Disallow:/
>>
>> upload to the root folder of your web server.
>>
>> most will comply but some wont. you can either insert rules
>> into your firewall to only allow your friends (if they have
>> static ip for example) or require authentification.
>
> Ok, I did this already just to see who complies and who doesn't :)
>
> But the part of my question that I really want an answer to is, when I
> find some bot hitting my web server, is it best to block it by raw IP,
> fqdn, or just the domain? How do I make this decision? Seems like raw
> IP or fqdn could change, because what I see is a whole list of
> hostnames that all have the same domain name, and I assume these could
> change at any time. If I block the entire domain, I don't anticipate
> these guys being back, but I'm blocking things pretty broadly at that
> point.
>
> Advice?
>
> Thanks.
>
> Andy
>