Re: Crawler problems

Based on a quick scan of my Apache logs, followed by a bit of "whois" fu, I
get these:

iptables -A blacklist -j DROP -s 119.63.192.0/21
iptables -A blacklist -j DROP -s 123.125.71.0/24
iptables -A blacklist -j DROP -s 180.76.0.0/16
iptables -A blacklist -j DROP -s 220.181.0.0/16

Of course, that blocks them from sending me spam^H^H^H^Huseful email as
well; what a pity.

-Martin

On Tue, 22 Nov 2011, Mark Foster wrote:

> Date: Tue, 22 Nov 2011 17:27:47 +1300
> From: Mark Foster <[email protected]>
> Reply-To: NZLUG Mailing List <[email protected]>
> To: NZLUG Mailing List <[email protected]>
> Subject: Re: [nzlug] Crawler problems
>
>
> >
> > My advice would be to decide which ones you like, register and control
> > those and then block the rest in robots.txt. Any that sneak though
> > block explicitly. I'm a little surprised about Baidu ignoring
> > robots.txt though.
> >
>
> Simon,
>
> throw 'baidu ignores robots.txt' into Google.
>
> http://web-robot-abuse.blogspot.com/2006/09/baiduspider-bad-bot-ignores-robotstxt.html
>
> It appears that 5 years on, nothing has changed.
>
> https://groups.google.com/group/sinatrarb/browse_thread/thread/7409a67be63ea65f
> another example.
>
> Overwhelmingly, Baidu is badly behaved.  Happy to see them firewalled to
> hell and back, to be honest.
>
> Mark.
>
>
>
> _______________________________________________
> NZLUG mailing list [email protected]
> http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug
>

_______________________________________________
NZLUG mailing list [email protected]
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug