Re: Crawler problems

On 22/11/11 09:45, Steve Holdoway wrote:
> Came to work yesterday, and had 3 (geographically) separate web servers
> on their knees, looking like they were being DDoSed. Web services
> unusable, databases screaming, resources depleted. 2 VPSes and a
> dedicated server.
>
> It transpires that all of them are being heavily spidered by
> Baiduspider, Bingbot and Googlebot.
>
> After a load of analysis this is what I've done:
>
> 1. Baiduspider - drop at firewall. These are English speaking sites, so
> we don't need this.
> 2. Bingbot - the same.
> 3. Google - dropped the 5 most prolific crawlers.
>
> I did want to keep bing, but tell it to behave itself, but it seems to
> just ignore robots.txt and do what it wants to anyway.
>
> Google was an afterthought - I'd hoped I could get away with that but it
> was still slowing the site down - so I've not analyzed so closely.
>
> Has anyone else noticed this increased aggressiveness in crawling
> policy? Any ideas on how to control it??
>

Baidubot went nuts on my image gallery last month - blew out my
bandwidth limits. They're now blocked at the supernet in the webserver
config (returning errors to everything, now). 180.76.5.0/24 occupies 29
of the top 30 IP's to visit my image gallery this month. Each request is
zero'd-out byte-wize due to the drop, but as I can't firewall the
connections off, they continue to pollute my log files.

I havn't dropped any other crawlers yet but I'm monitoring the situ, as
i've also put in a 'block all crawlers' robots.txt file - it really is
interesting to see which crawlers remain; I'm actually seeing 'real'
user-agents now, and not just bots.

Here's the top 15 user agents on my image gallery (my old one, anyway)
for month to date. Check out the percentages in terms of hits.

http://gallery3.blakjak.net/var/albums/Misc/2011/UserAgents-GalleryMTD.png

/rage!


Mark.



_______________________________________________
NZLUG mailing list [email protected]
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug