Bug#949173: packages.debian.org: robots.txt doesn't actually block anything
"Adam D. Barratt" <[email protected]>
| Newsgroups | gmane.linux.debian.devel.www |
|---|---|
| Message-ID | <c5ebea2d2c168e6528b2d1706d0a7b3b286943bb.camel__10394.615277049$1579287440$gmane$org@adam-barratt.org.uk> |
Package: www.debian.org User: [email protected] Usertags: packages Severity: important Hi, While looking at blocking some web spiders that were causing significant load on picconi (packages.d.o master), I noticed that the robots.txt currently doesn't attempt to block them. Specifically, the current file is: <quote> User-agent: * Disallow: </quote> which is effectively the same as allowing everything. "Disallow: /" might be more logical, unless there is a desire / requirement to allow crawling and indexing of (parts of) the site. Regards, Adam