Re: Searchable archives?

Benjamin J. Tilly wrote:

>>This part narrows to this domain only.
>>
>>See google advanced search for more details.
>>    
>>
>
>I am fully aware of Google.  It does not archive most of this
>list.  I thought that that was because of a robots.txt file, but
>I just looked for one and didn't see it (I could have missed
>something obvious).  So I suspect that they just filter it out
>because they don't like following CGI programs.  (Too many link
>farms...)
>
>Either way you'll find that Google is no help in searching this
>list.
>
Umm. OK. Quick explanation for those unfamiliar with Search Engines.:

The archives are dirty links, which will not be crawled by search engine 
bots.

Dirty links are links with a '?' within the URL, with other characters 
following it. Historically, this has been done so that dynamic content 
isn't indexed (as I recall, anyway, I'm young but getting older). 
Dynamic content indexing leads to bad results. 'But the mailing list 
archives aren't dynamic', you would say, and I would say that they are 
probably dynamically generated. In fact, I'm pretty sure that they are.

An example of a dirty link is this:

http://www.crynwr.com/cgi-bin/ezmlm-cgi?mss:9085:200403:mocepfaeedeadnaidocn

Note that between 'cgi' and 'mss', there is a question mark - which 
search engines see as something to ignore. Why? Myopic programming, or 
more appropriately, the same reason we had the Y2K problem.

That was the problem.

The solution: Either get bots to change their self-limiting ways, or get 
rid of the question marks. The latter can probably be done more easily 
than the former. Archives themselves are static. The manner of 
generating the archives is probably dynamic.

-- 
Taran Rampersad
[email protected]
http://www.knowprose.com
http://www.fsc.cc/
http://www.uwi.edu/scs/
http://www.a42.com
http://www.xeebra.com

Coming Soon: eAsylum.net