Re: RobotRules fails on user-agents with spaces

Nigel Horne <[email protected]>
Newsgroups gmane.comp.lang.perl.modules.lwp
Organization NJH Music
Message-ID <[email protected]> 
On Fri, 2005-10-14 at 10:37, Gisle Aas wrote:
> <[email protected]> writes:
> 
> > The problem... if I include a space in my robot's user agent, it
> > will fail to recognize robots.txt records targeted to my robot.
> 
> You are not allowed to have space in the user agent name.  See section
> "3.8 Product Tokens" of RFC 2616 [1].  Isn't it an option to just
> rename your spider to something that follows the spec?

Perhaps it would help if WWW::RobotRules were to warn/die when setting
an agent with a space in? An excellent message would be "RFC2616 forbids
spaces in an agent's names".

-Nigel
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.