Re: Indexing protected area

One thing you might consider, if you control the Apache server, is to 
put an alias in the URL tree that duplicates access to the content. 
I've found that once I start spidering a site, the statistics go all to 
heck, so filtering out your spidering can be a useful thing -- putting 
them under a different URL (or a different virtual server, even better!) 
is a good way to go.

/jordan