git: 82284de091 - main - remove non-standard Crawl-delay directive
Wolfram Schneider <[email protected]>
| Newsgroups | gmane.os.freebsd.devel.cvs.doc |
|---|---|
| Message-ID | <[email protected]> |
The branch main has been updated by wosch: URL: https://cgit.FreeBSD.org/doc/commit/?id=82284de091cf1a05aa33538a49d6990ebbc043e3 commit 82284de091cf1a05aa33538a49d6990ebbc043e3 Author: Wolfram Schneider <[email protected]> AuthorDate: 2023-09-02 07:31:36 +0000 Commit: Wolfram Schneider <[email protected]> CommitDate: 2023-09-02 07:31:36 +0000 remove non-standard Crawl-delay directive --- documentation/themes/beastie/layouts/robots.txt | 1 - website/themes/beastie/layouts/robots.txt | 1 - 2 files changed, 2 deletions(-) diff --git a/documentation/themes/beastie/layouts/robots.txt b/documentation/themes/beastie/layouts/robots.txt index fd409b7c52..2499651555 100644 --- a/documentation/themes/beastie/layouts/robots.txt +++ b/documentation/themes/beastie/layouts/robots.txt @@ -1,5 +1,4 @@ User-agent: * -#Crawl-delay: 10 Disallow: /statistic Disallow: /doc/ diff --git a/website/themes/beastie/layouts/robots.txt b/website/themes/beastie/layouts/robots.txt index 27f4152f84..8b6d6fb9b0 100644 --- a/website/themes/beastie/layouts/robots.txt +++ b/website/themes/beastie/layouts/robots.txt @@ -1,5 +1,4 @@ User-agent: * -#Crawl-delay: 10 Disallow: /cgi/getmsg.cgi Disallow: /cgi/mailindex.cgi Disallow: /cgi/man.cgi?apropos=2&