RE: site-wide metadata discovery

 I got hung up because it would cause an extra lookup to get
> that file after finding the right file name, and that might not be ideal.

I think being as gentle as possible with robots.txt would probably be a good
idea.

> If we're doing that, why not use a META tag in the index document to point
> to it, instead of mucking with robots.txt at all?

As a fall back position, in the kind of cases Bill pointed out.

> Plus, it sorta infringes on the elegance of doing it all within
> robots.xml.

Nah, I think a little hook is elegant.

> Another thought would be to use comments in the robots.txt file.

That's gentle!

Cheers,
Danny.


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/