RE: site-wide metadata discovery

[pause]

What are the circumstances in which robots.txt (or any other well-known
file) will be needed?

When will it not be possible/desirable to use <link>?

[esuap]


> -----Original Message-----
> From: Bill Kearney [mailto:[email protected]]
> Sent: 16 October 2003 15:46
> To: [email protected]
> Subject: Re: [syndication] site-wide metadata discovery
>
>
> > One concern is that search engines, at least traditionally,
> have implemented
> size
> > limits for robots.txt and excluded sites with overly large
> robots.txt files.
> Not
> > sure if that restriction still applies (most robots requesting
> robots.txt will
> not
> > use the additional information though so robots.txt is not an
> ideal place for
> this)
>
> Good point.  I think we're talking of using something no larger than:
>
> Site-index: default
>         href: http://www.syndic8.com/feedlist.ocs
>         type: http://InternetAlchemy.org/ocs/directory#
>         title: Syndic8 feed list in OCS format
>
> So it's not like that should be an unreasonably large addition.
>
> Likewise, I think the plan is to have the default URL have
> semantics that allow
> for delegation and alternatives.  That default document could
> well also contain
> links to many, many others related to the site.  This would help to allow
> delegation and avoid cluttering the robots.txt file.
>
> -Bill Kearney
>
>
>
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>
>


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/