[#1444 robots.txt too restrictive] (new) robots.txt too restrictive
[email protected] (Simon Michael) Sun, 01 Mar 2009 10:49:53 -0800
| Newsgroups | gmane.comp.web.zope.zwiki |
|---|---|
| Message-ID | <4B4E7254-0ABC-4ECF-A21E-2886D4DE9F52__11921.7854262844$1235933491$gmane$org@joyful.com> |
robots.txt was permitting only google and internet archive for a long time - a clumsy solution to load problems caused by rude bots. This probably makes us harder for non-google users to find (I just tried - yahoo search finds us, microsoft search doesn't), and blocks useful tools like browsershot.org. I've opened it up to all, excluding a few expensive urls, we'll see what happens. -- forwarded from http://zwiki.org/1444RobotsTxtTooRestrictive#[email protected]