missing robots.txt for www.openssh.com?
"Sevan / Venture37" <[email protected]>
| Newsgroups | gmane.os.openbsd.www |
|---|---|
| Message-ID | <[email protected]> |
Though donations.html is listed in the disallowed section of the openbsd.org robots.txt, Google is picking up donations.html from the cvsweb: http://www.openssh.com/cgi-bin/cvsweb/~checkout~/www/donations.html The robots.txt will solve this issue until someone changes the webserver config as all sub projects link back www.openbsd.org/cgi-bin/cvsweb/ Sevan Janiyan Index: openssh/robots.txt =================================================================== RCS file: openssh/robots.txt diff -N openssh/robots.txt --- /dev/null 1 Jan 1970 00:00:00 -0000 +++ openssh/robots.txt 17 Jun 2010 11:33:03 -0000 @@ -0,0 +1,2 @@ +User-agent: * +Disallow: /cgi-bin/