Re: Crawler problems

Mark Foster <[email protected]> Tue, 22 Nov 2011 17:27:47 +1300

Newsgroups	gmane.org.user-groups.linux.new-zealand.general
Message-ID	<[email protected]>

>
> My advice would be to decide which ones you like, register and control
> those and then block the rest in robots.txt. Any that sneak though
> block explicitly. I'm a little surprised about Baidu ignoring
> robots.txt though.
>

Simon,

throw 'baidu ignores robots.txt' into Google.

http://web-robot-abuse.blogspot.com/2006/09/baiduspider-bad-bot-ignores-robotstxt.html

It appears that 5 years on, nothing has changed.

https://groups.google.com/group/sinatrarb/browse_thread/thread/7409a67be63ea65f
another example.

Overwhelmingly, Baidu is badly behaved.  Happy to see them firewalled to
hell and back, to be honest.

Mark.



_______________________________________________
NZLUG mailing list [email protected]
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug