Re: Suggestion/Enhancement request: Please add to FAQ: list of User-agents for robots.txt files

--IdQYvU5iz1n5pUC3
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Andrew Avdeenko <[email protected]>, 2015-08-02 16:00 +0300:
> Archived-At: <http://www.w3.org/mid/op.x2qrq8zm278snb@microsof-c0ae01>
>=20
> Is it possible to deny access to my website for W3C validators using
> robots.txt? If "yes", what user-agent(s) must be specified?

The W3C Link Checker https://validator.w3.org/checklink is the only one
that=E2=80=99s actually a crawler/robot, and so the only one that pays atte=
ntion to
robots.txt files. You can block it by specifying =E2=80=9CUser-Agent: W3C-c=
hecklink=E2=80=9D.

All of the services also have =E2=80=9Chttp://validator.w3.org/services" in=
 their
user-agent strings, and run on hosts with IP addresses in the 128.30.52.0/24
subnet. So you can block them based on that user-agent substring, or by IP
address=E2=80=94but because none of the services other than the link checke=
r are
crawlers/robots in normal usage, they=E2=80=99re not among the types of too=
ls that
robots.txt is intended for, so you=E2=80=99ll need to use some other means =
to block
them (e.g., some specific configuration to your firewall or Web server).

There used to be a document at http://validator.w3.org/services which
explained all this but it seems to have disappeared, so I=E2=80=99ll get it
restored as soon as possible.

  =E2=80=94Mike

--=20
Michael[tm] Smith https://people.w3.org/mike

--IdQYvU5iz1n5pUC3
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJVyCDvAAoJEIfRdHe8OkuVhcYP/1x8um9T2QALqzMEe/b6S8Tf
Xpm8qpCzmRTr8JlO0ZEz4TdsDohbNTMfb0Z4gtnr3LRRTkN0BBddnQAj+gewQWDy
otg8TxjpgeGT9PPm96nw5atju1mA4Y3v5BM1auW4RREHi1azSTt0/c93mzZR0/G9
wQOADDx3+/R44pD+2t5HGpiDv9nSL/y6em5d/IJAE9107bQHvgwB0AWcmbiIhgko
cl0Xjem2sUZB8pp824k/ItlcHWa35KEYz+Sw266KN91bQFLsYO0+GfTQBQF0Ox7O
kTw+joJ2OyORi7EB6W1gXBBVAh+WCh7RYQOFTpQjBUUeLR9UIKzWjawWNm9aN8Rq
VLBvpe/JkTyodyxoXPqlkAUa2NdarNQVhz74No3F+5O+KWq4oWQDNnk2yvW448yr
TTmaLo7mhIrXkkKw7IWyI4pEltp5o0QOCTbr2vtTUzPon8+ByLyYPyAdN/2NZtvH
3Kfxfodl7cv6COWyWXzcwIcRMioCrWXLmHu3U2mJXlylSL3GOP3jACO6J6j21cxS
Wfd2tTQX0kSBX3gnU1fM0kvAwLQmsJ62O/4PCON1Z96t5oLIA+Qj7yQyqvf7+qZE
t+wpqaQQDsMydKA2G7g+zrrnWeXpIoHx8iQlEuRbSaQ5Kexsv3ZtLPK1+/ji2Te+
iQNwaebLrqTonDo3lQl6
=T+LX
-----END PGP SIGNATURE-----

--IdQYvU5iz1n5pUC3--