Re: Large scale htdig

Aaron writes:

> 1. The main issue is that we have no guarantee that you would be around 
> tomorrow.  It is mentioned many times on your page.  Not that this is a 
> really bad thing, but for a large corporation, it's not ideal.

If you allow me to butt in here for a second -- if Apple used all the
three major mailing list archives (mail-archive.com,
marc.theaimsgroup.com and gmane.org (the latter of which I run)),
then there'd perhaps be less of a chance of the archives all
disappearing at once.

Anyway, I think it's amazing what mail-archive.com has managed to do
with htDig, because it kinda doesn't scale very well.  Up until a
couple of years ago, however, there really weren't any free search
engines that scaled to beyond a couple of hundred thousand messages.
Now there's two strong contenders: The Java-based Lucene, which has
gotten a lot of positive attention lately, and Xapian
(<URL: http://xapian.org>), which is what Gmane is going to start
using in a few weeks.  Test runs shows that it's able to index our
entire spool (20M articles) in eight days on a cheap indexing
machine... 

> 2. The ads.  Again, not a horribly bad thing (and understandable in 
> your case), but not something we wanted to have on our search engine.

Perhaps if some corporation were to sponsor one of the mail archives,
they'd drop the ads.  :-)  (marc doesn't have ads, by the way.)

> 3. Scaleability. Currently we have over 900,000 documents that need to 
> be indexed and kept up to date.  There are over 100 mailing lists and 
> we average roughly 500 posts overall a day.  I didn't want to break 
> anything ;)

I doubt that'd make much of a dent in the statistics for either of
the mail archives.

Anyway, it's understandable that a big company like Apple would like
control over their own mailing lists, but I thought I'd just mention
some options.

-- 
(domestic pets only, the antidote for overdose, milk.)
  [email protected] * Lars Magne Ingebrigtsen