Re: Compromise time....

There was lots of discussion of extensions to robots.txt on the old robots
mailing list - it's archived at:

http://www.robotstxt.org/wc/mailing-list/

there is some discussion on additional features for robots files -- and
some pretty good arguments as to why those things weren't added ;-) 

Ian


On Wed, 15 Oct 2003, Jeff Barr wrote:

> Jeremy Zawodny wrote:
> 
> > I now agree that #1 is not worth it.  HOWEVER, I would like to pursue
> > the possible use of robots.txt as suggested by Chad.  Does anyone here
> > know any robots.txt "experts"--people who were involved with the
> > discussions back then?  We should try to get an authoritative answer
> > on whether it is (1) possible to do what we want there, and (2) not
> > completely unreasonable to do it.  If the robots.txt option doesn't
> > work out, then it doesn't work out.  We tried.
> 
> The robotstxt.org [0] site contains notes from a 1996 meeting on
> spidering [1]. It contains the following tantalyzing tidbit:
> 
>    These are issues recommended for future standards discussion that
>    could not be resolved within the scope of this workshop.
> 
>    ...
>    * ways of advertising content that should be indexed (rather
>      than just restricting content that should not be indexed)
>    ...
> 
> Looks like its time to break some new ground, in an area where
> the original folks thought it would eventually go.
> 
> Jeff;
> 
> [0] - http://www.robotstxt.org/
> [1] - http://www.robotstxt.org/wc/meta-notes.html
> 
> 
> 
> 
>  
> 
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 
> 
> 
> 



 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/