blogs.gnome.org blocks indexing by non-Google search engines and archive.org
Daniel Micay <[email protected]> Sat, 01 Mar 2014 11:27:38 -0500
| Newsgroups | gmane.comp.gnome.web |
|---|---|
| Message-ID | <[email protected]> |
The blogs.gnome.org uses a strange robots.txt <http://blogs.gnome.org/robots.txt> permitting only Googlebot to crawl it. This prevents using archive.org on the subdomain and cripples other search engines. It should simply use a wildcard for the first User-agent field with the second case discarded. Thanks! _______________________________________________ gnome-web-list mailing list [email protected] https://mail.gnome.org/mailman/listinfo/gnome-web-list
signature.asc
(application/pgp-signature, 836 B)
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBAgAGBQJTEgp+AAoJEPnnEuWa9fIqMaMP/08Yu4jE21KC6SXQAeJt1hrk Q7nrosGK827HcwI7MZ5sjE8Aajgpda+sX04YkNYO4BqiS1Z86blv2cBJrSQIRZmC kF2TfhCKk06Y1uUXru3WJGnBvm6V4WbeO2uSDLPV/iCSmnOVYuLcb+TUcMayln3g SSKi6dG0FVhdnZNt9IGAinuYBXSz0w+j/7i8nirANj9tjXG3t9zCXYlFZoTbU/TS iveym3xhacGEI4sTEHWA2PF9mLoln6GSsXX0vN/hY6L9Gceugn2T2J6TMDCGGGQ0 XTzI4uKN0f01hmGXUe9jB3LhlpLtD6EzHXY3bhUiu2n3SEDkrBeH4yZoTudIABrq O3ExCeygVXxxC/b+raquzLFMMJE42yo3VcbYXK7eNQHT94nWH/Cg+2+sjWY3PzRr J8Ew6SnCAL/iyU6QHzRIh6HfVR0z04dUbQBu2TMPSNmyyHC9vzq407AEQGJKPuJy D4bmMvW01T+Z2NrI7PRUep5WPuGHrvAdDOzWLyXKk225kImiBXHDu6TcFGistNNq Cl5OL5vqDn6HkCV2vb/lQkLBPQykt8tp10raQ6uv6qzLTBtRY84haoLS2JfqQNU2 NV6nbBmaOxvdDKbr368XZVLakdjneeC7GYlM/RKBpvFG9P+yfrLMDNj0I5HAOd6l 1riEdTIgwZAvyaME4kl7 =FcHe -----END PGP SIGNATURE-----