Re: Crawler problems
Jethro Carr <[email protected]> Tue, 22 Nov 2011 13:11:07 +1300
| Newsgroups | gmane.org.user-groups.linux.new-zealand.general |
|---|---|
| Message-ID | <[email protected]> |
On Tue, 2011-11-22 at 10:00 +1300, Mark Foster wrote: > > Has anyone else noticed this increased aggressiveness in crawling > > policy? Any ideas on how to control it?? > > > > Baidubot went nuts on my image gallery last month - blew out my > bandwidth limits. heh, Baidubot went and downloaded every single RPM in my public repository, was a good chunk of data! regards, jethro -- Jethro Carr www.jethrocarr.com www.amberdms.com _______________________________________________ NZLUG mailing list [email protected] http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug
signature.asc
(application/pgp-signature, 198 B)
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEABECAAYFAk7K6JYACgkQDpNWragKym8ixACdGsiE3jsy42vBbYtS1AzOZxOH A7gAn3CYieUvQJ/QRhOywlP48dNFHP6J =nqTm -----END PGP SIGNATURE-----