Re: Crawler problems

On Tue, 22 Nov 2011, Steve Holdoway wrote:
> On Tue, 2011-11-22 at 10:00 +1300, Mark Foster wrote:
>
>> Baidubot went nuts on my image gallery last month - blew out my
>> bandwidth limits. They're now blocked at the supernet in the webserver
>> config (returning errors to everything, now). 180.76.5.0/24 occupies 29
>> of the top 30 IP's to visit my image gallery this month. Each request is
>> zero'd-out byte-wize due to the drop, but as I can't firewall the
>> connections off, they continue to pollute my log files.
>
> Good* to know I'm not alone. I firewalled 180.76.5.0/24 and
> 180.76.6.0/24 to drop Baidu.

Some of the bots obey this:

Crawl-Delay: 30

which will make them go slow. In the Case of Googlebot, Yahoo and 
MsnBot you can register on their site and control them (somewhat) via a 
web interface. Depends how much you worry about SEO.

They are all a pain though, for example GoogleBot will recheck all the 301 
redirects on your site every month or two "just in case". Which can be a 
lot of links if you did a site reorg a couple of years ago.

My advice would be to decide which ones you like, register and control 
those and then block the rest in robots.txt. Any that sneak though block 
explicitly. I'm a little surprised about Baidu ignoring robots.txt 
though.

-- 
Simon Lyall  |  Very Busy  |  Web: http://www.darkmere.gen.nz/
"To stay awake all night adds a day to your life" - Stilgar | eMT.


_______________________________________________
NZLUG mailing list [email protected]
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug