robots.txt, etc?

hi all. i'm curious how people on this list handle robots.txt, favicon.ico, 
and other well-known URLs on pyblosxom sites.

pyblosxom is a cgi program, not a web server, so technically, we can duck the 
issue by claiming that GET /robots.txt should be handled directly by the web 
server. however, many pyblosxom-based sites use pyblosxom to serve *all* 
requests, including /*, so this isn't really fair. :P

on my site, i use the filekicker plugin, which serves matching files in the 
filesystem when they're requested directly. it only lets pyblosxom render 
stories and file lists when they don't match a file on the filesystem.

with filekicker, i just drop a robots.txt file in the root of my datadir, and 
filekicker serves it happily. however, .txt is the default extension for 
entries, so when pyblosxom sees a request for /robots, it tries to parse 
robots.txt as an entry, promptly dies, and the browser receives a 500.

should pyblosxom do anything about special, well-known URLs like /robots.txt,
/favicon.ico, and others? i'm inclined to say that if a matching file exists 
in the datadir, we should serve it, and we should 404 on requests for /robots.

thoughts?

-Ryan

--
http://snarfed.org/

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV