Re: robots.txt, etc?

A while back I was tossing around adding the ability for PyBlosxom to have 
a "resources" directory where files like the ones you mention could all 
hang out.  I didn't implement it for a couple of reasons:

1) I figured it's probably a slow way for files to get sourced since the 
request has to go through PyBlosxom and then get served by PyBlosxom.

2) I also wanted this system to make flavour packs easier to modularize. 
So if a flavour pack required a css file (and they almost all do), then 
the css file would stay in the same directory as the flavour pack and 
still get served up by PyBlosxom.

I didn't get further than that.  I still like the idea, though, and maybe 
it's something that should get handled in a plugin so that we can more 
easily handle other behaviors.

/will


On Thu, 25 Jan 2007, Ryan Barrett wrote:
> 
> hi all. i'm curious how people on this list handle robots.txt, favicon.ico,
> and other well-known URLs on pyblosxom sites.
>
> pyblosxom is a cgi program, not a web server, so technically, we can duck the
> issue by claiming that GET /robots.txt should be handled directly by the web
> server. however, many pyblosxom-based sites use pyblosxom to serve *all*
> requests, including /*, so this isn't really fair. :P
>
> on my site, i use the filekicker plugin, which serves matching files in the
> filesystem when they're requested directly. it only lets pyblosxom render
> stories and file lists when they don't match a file on the filesystem.
>
> with filekicker, i just drop a robots.txt file in the root of my datadir, and
> filekicker serves it happily. however, .txt is the default extension for
> entries, so when pyblosxom sees a request for /robots, it tries to parse
> robots.txt as an entry, promptly dies, and the browser receives a 500.
>
> should pyblosxom do anything about special, well-known URLs like /robots.txt,
> /favicon.ico, and others? i'm inclined to say that if a matching file exists
> in the datadir, we should serve it, and we should 404 on requests for /robots.
>
> thoughts?
>
> -Ryan

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV