Re: blogs.gnome.org blocks indexing by non-Google search engines and archive.org
Daniel Micay <[email protected]> Sat, 01 Mar 2014 13:29:21 -0500
| Newsgroups | gmane.comp.gnome.web |
|---|---|
| Message-ID | <[email protected]> |
On 01/03/14 11:27 AM, Daniel Micay wrote: > The blogs.gnome.org uses a strange robots.txt > <http://blogs.gnome.org/robots.txt> permitting only Googlebot to crawl > it. This prevents using archive.org on the subdomain and cripples other > search engines. It should simply use a wildcard for the first User-agent > field with the second case discarded. > > Thanks! It looks like this has already been fixed. That could be the best bug report response time I've ever seen. ;) _______________________________________________ gnome-web-list mailing list [email protected] https://mail.gnome.org/mailman/listinfo/gnome-web-list
signature.asc
(application/pgp-signature, 836 B)
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBAgAGBQJTEicJAAoJEPnnEuWa9fIqafIP/2TE5iIG6IOiTsWbfEqGASl6 0YsxXpmT1bzstdLZhrKeyadbW0p3XynYu2UMZ2viwIgRvNojX/5VLOfKj9ms9G12 RgY8/8KVLBp7PXkX5bftUGiejbXxQJZg01GCCUgzlbUUMX2f1efN2vZSYJn62E/Z zg1A4WGS6/evAkFvJJVIwO/VHeQ7JmRkIRClPRc6g08mpk0Oo/mFku/TDV9E1vyh eV7aUjCAO2DYP0xW0MqKbingXYgsuqX/EuryMdCpK4D/TrykAzGvYYRdYMVoL7NK gikFh7w5dzGNGWVDGy2Big/d71qwiX40/zED+yLkB72fhp3oTWlwhT33PWsn5zmo +af2lifE9R0J/wwwwV6/RNyeXuYcsc1SWH0m79187KrimJS6HWnX2SDKJjtdvR3U d9i6Xs+2EU5EvbzcknLlRBCWgCiLwLwoWoHM9PWwvOScxbMaHM7TLGctI6gymZI9 Vc7PWEBX+4rKy5gAB1Rk2F1YFmOPDCVqkWogJ12JTmUvZC3egQS3TQR9MiHZXEvJ 3q/IyyY3vYHGzRW3fEzSFOoP+0lVkil+WIayuQbI+oxryQdJsftuaHsbXbHQBJjw EgLw4N2JbPUqZQ04y1bh1+QR3Z833DHIcK2sTrFBJUQdS2+/4iFt1pXj0HoKyH3o HLRLwQ1gTBJm+3eWNFu4 =Q/yA -----END PGP SIGNATURE-----