robots.txt, grub-client
Jason Harris <[email protected]> Sat, 18 Dec 2004 15:52:32 -0500
| Newsgroups | gmane.comp.encryption.pgp.sks,gmane.comp.encryption.pgp.keyserver-folk |
|---|---|
| Message-ID | <[email protected]> |
Is anyone (else) serving robots.txt from pks and SKS and watching the User-Agent: headers on incoming requests? I've noticed a lot (30 and counting, since yesterday afternoon) of requests from grub-client-2.3 to my pks server, which is wrong because I've been serving robots.txt containing: User-agent: * Disallow: / for quite some time now. grub[.org] seems to be the newest search engine that doesn't respect robots.txt, but it is also hard to block because it is a distributed system. Still, 64.241.242.18=sv-fw.looksmart.com is the main offender and can be blocked by IP. Of course, M$ in 65.52.0.0/14 and 207.68.128.0 - 207.68.207.255 and Yahoo/Inktomi in 66.196.64.0/18 are also blocked by IP due to over- zealous web crawlers and/or not respecting robots.txt. Most of the grub requests have been for "Host: skylane.kjsl.com:11371" as well. The few for "Host: wwwkeys.pgp.net:11371" are understandable because it is a DNS RR, of course, but I imagine the remaining servers in wwwkeys.pgp.net (and other DNS RRs) that don't block these crawlers will see their bot-induced load eventually rise to unacceptable levels. -- Jason Harris | NIC: JH329, PGP: This _is_ PGP-signed, isn't it? [email protected] _|_ web: http://keyserver.kjsl.com/~jharris/ Got photons? (TM), (C) 2004
signature.asc
(application/pgp-signature, 187 B)
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (FreeBSD) iD8DBQFBxJiPSypIl9OdoOMRAsvYAJ9ZJHhGOHhqBqbbG3i/wD9pmtX87wCfcbDn G/K5CAgxnHQ5V19MOEz+RRc= =lqdb -----END PGP SIGNATURE-----