Re: site-wide metadata discovery

From my review it appears the robots community exchanges bad-bot info using a
like-named prefix across their values.  Would we want to do the same thing?

Site-Index: URL
Site-Index-Format: MIME-type_or-URI
Site-Index-Recursion:   true|false

I'd worry that using anything beginning with the same text, as in "Allow" would
be a bad idea.  This to avoid someone existing code making a stupid assumption.
If we're not using anything known then we avoid the risk entirely.

>  The idea of a hook to a "metadata index file" definitely has appeal.  We
> don't clutter the robots.txt, plus we don't specify a particular file name,
> plus it could actually be used for sections of domains.

Yes, let's be *sure* our examples never make use of the same target location
twice.  Let's not have naive implementors assume any sort of filename is the
'endorsed' one.

> Says the same thing, but that there is no recursion possible.  Even if
> metadata.xml exists at lower levels of the domain, don't bother looking for
> it, as we are telling you not to do so.  Some larger sites might like this
> sort of feature as it could limit requests for data.

I'd like to encourage this as the /default/ behavior.  Let's not have them
assume that since the root doesn't say anything that they can stab around for
it.  Better to leave it OFF and let the website administrators open that risk up
with full knowledge of what costs it might incur.

A handy sample of some stuff using mod_rewrite would perhaps be really useful
here too!

> And the application doesn't support opml or ocs, it can just not bother
> fetching more data because it won't be able to parse the data anyway.

Yes, this is something I've suggested earlier.  Best to allow, right from the
start, a multiple dichotomy of formats.

> No excuse for bad behavior
> if you see something and still choose to get it, even if you can't parse the
> data within...

Right and the success shown to us by the use of the feed validator will help us
point people to something good for testing their configuration.

> Naturally this also raises two more questions:
>
> - What is the default for recursion?  True or false?  False may mean less
> traffic.  True may mean more results (also could mean more errors!).

I'd vote for default it to False.  And any examples would benefit from having
text that said this.  Let's not leave it ambiguous.

> - What the heck format will support storing all the metadata for a site?
> Consider that it might contain feeds, but other useful information could be
> in there as well - subscriptions, location of foaf, pointer to rsd, etc...

Well, once we nail down the discovery then we enter that stage.  If our format
here supports clear indicators for 'seeAlso' sorts of references then we really
don't have to be terribly complex in what we spec for the first tier index
format.

-Bill Kearney


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/