RE: site-wide metadata discovery

Only thing more recent I found is this Internet Draft, dated November of
1996 (expired June 4, 1997).  

http://www.robotstxt.org/wc/norobots-rfc.html

It has some more words about extending the specification, specifically:

   Lines with Fields not explicitly specified by this specification
   may occur in the /robots.txt, allowing for future extension of the
   format. Consult the BNF for restrictions on the syntax of such
   extensions. Note specifically that for backwards compatibility 
   with robots implementing earlier versions of this specification,
   breaking of lines is not allowed.

I think this would fall under:

    extension    = token : *space value [comment] CRLF

Which (I think) means that:

Site-Index:

Would not be valid, but:

Site-Index: default
Site-Index: folder
Site-Index: another
Site-Index: include folder/robots.txt

Would be okay.  

Of course, since it's expired, I don't know that it's useful.  But thought
I'd add it to the discussion anyway.


-----Original Message-----
From: Chad Everett [mailto:yahoogroups-ipyKpt5L2sG2CP6mJM4Q19i2O/[email protected]] 
Sent: Wednesday, October 15, 2003 5:53 PM
To: '[email protected]'
Subject: RE: [syndication] site-wide metadata discovery


Hi Dave -

Thanks for the vote of confidence.  :)

Here's where I found that text:

http://www.robotstxt.org/wc/norobots.html

Under "The Format", at the end of the sentence just prior to "User-agent".

It actually says "Unrecognised headers are ignored".  I took some small,
hopefully irrelevant, liberties with my interpretation.  :)

Chad.

-----Original Message-----
From: Dave Winer [mailto:dave-rH/ioLL7x7RWk0Htik3J/[email protected]] 
Sent: Wednesday, October 15, 2003 5:44 PM
To: [email protected]
Subject: Re: [syndication] site-wide metadata discovery


This is a very interesting idea.

I just went for a quick look and couldn't find where it says "unrecognized
headers should be ignored."

And then we'd have to know if it works in practice too, just because a spec
says robots should work some way doesn't mean they do.

But kudos for coming up with a new angle. Worth exploring, imho.

Dave


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/