git: e09c06b9c2 - main - block CCBot (Common Crawl) from accessing cgi scripts
Wolfram Schneider <[email protected]>
| Newsgroups | gmane.os.freebsd.devel.cvs.doc |
|---|---|
| Message-ID | <[email protected]> |
The branch main has been updated by wosch: URL: https://cgit.FreeBSD.org/doc/commit/?id=e09c06b9c2fa70e4884edae327d294c5efc3eb02 commit e09c06b9c2fa70e4884edae327d294c5efc3eb02 Author: Wolfram Schneider <[email protected]> AuthorDate: 2023-08-23 06:17:36 +0000 Commit: Wolfram Schneider <[email protected]> CommitDate: 2023-08-23 06:17:36 +0000 block CCBot (Common Crawl) from accessing cgi scripts --- website/themes/beastie/layouts/robots.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/website/themes/beastie/layouts/robots.txt b/website/themes/beastie/layouts/robots.txt index 9bec7f76cf..1248457c75 100644 --- a/website/themes/beastie/layouts/robots.txt +++ b/website/themes/beastie/layouts/robots.txt @@ -14,3 +14,6 @@ Disallow: /statistic User-agent: GPTBot Disallow: /cgi +User-agent: CCBot +Disallow: /cgi +