Re: robots.txt (was Re: Download a whole gopherhole using wget/curl?)

Am Fri, 29 Nov 2019 22:41:12 +1000
schrieb James Mills <[email protected]>:

> Silly question; But isn't the User-Agent kind of useless here since a
> Gopher request is basically just a selector for a resource?
> There are no headers
> No User-Agent to identify a request
> 
> What am I missing here :)
> 
Well the point is that there's nothing to identify server-side as
robots.txt is more or less just an indication from the server which
parts of the site should not be crawled. Parsing and interpretation is
done entirely on the client side. So if a client finds an entry which
it believes to fit to itself, it should act accordingly.

HTH,
Florian
> Kind Regards
> 
> James
> 
> James Mills / prologic
> 
> E: [email protected]
> W: prologic.shortcircuit.net.au
> 
> 
> On Fri, Nov 29, 2019 at 3:39 PM Sean Conner <sean-/[email protected]> wrote:
> 
> > It was thus said that the Great Christoph Lohmann once stated:  
> > > Good point. In eomyidae you have two possibilities:
> > >
> > >       User-Agent: *
> > >       Disallow: *  
> >
> >   Okay, but this diverts from the HTTP version of robots.txt (from
> > my understanding unless it's been updated since I was last dealing
> > with this stuff).
> >  
> > > and
> > >
> > >       User-Agent: *
> > >       Disallow:  
> >
> >   This actually has a different meaning from the HTTP
> > version---there this means "all browsers allowed to crawl" (back
> > from when it robots.txt was first developed).
> >
> >   -spc
> >
> >