Re: Large scale htdig

>Thanks for the tip, I created separate databases with separate configs 
>for each list.  This seems to have helped a bunch.  Any other words of 
>wisdom? =)

Thanks for the feedback. You can look at our configuration - The Mail
Archive glue code is buried in our FAQ somewhere and marked with a GPL
license. Nothing else is jumping out at me at the moment.

>1. The main issue is that we have no guarantee that you would be around 
>tomorrow.  It is mentioned many times on your page.  Not that this is a 
>really bad thing, but for a large corporation, it's not ideal.

Yes, that's a good point and a general risk when dealing with a small
entities. The Mail Archive is on pretty solid ground compared to six
years ago (or even one year ago), and perhaps we should update the FAQ
to reflect it.  But there's probably not much we can do if you are
looking for a household name megacorp.

>2. The ads.  Again, not a horribly bad thing (and understandable in 
>your case), but not something we wanted to have on our search engine.

Happy to discuss over pivate email if you are interested in
pursuing. We've always technically had per-list customization
capability, but have very rarely used it beyond occasional language
localization. The beta version for The Mail Archive (which is getting
close) is CSS based and should take look and feel customizability to
another level.

  http://unagi.mail-archive.com:8080/[email protected]/msg00065.html

>3. Scaleability. Currently we have over 900,000 documents that need to 
>be indexed and kept up to date.  There are over 100 mailing lists and 
>we average roughly 500 posts overall a day.  I didn't want to break 
>anything ;)

No problem.

>Anyway, I think it's amazing what mail-archive.com has managed to do
>with htDig, because it kinda doesn't scale very well.  Up until a
>couple of years ago, however, there really weren't any free search
>engines that scaled to beyond a couple of hundred thousand messages.
>Now there's two strong contenders: The Java-based Lucene, which has
>gotten a lot of positive attention lately, and Xapian
>(<URL: http://xapian.org>), which is what Gmane is going to start
>using in a few weeks.  Test runs shows that it's able to index our
>entire spool (20M articles) in eight days on a cheap indexing
>machine... 

I think The Mail Archive might be HtDig's largest scale user. I'm a
member of the HtDig foundation and the Debian package maintainer of
Lucene. There probably several reasonable choices if you are looking
for per-list search indexes, but once you are searching over a milllion+ 
document range there are only a few options.

Cheers,
Jeff