Re: Full text indexing - integrating more DMS features into OGo

> > Cyrus IMAPd supports a full text indexer called "squat".  The indexer 
> > is a
> > seperate process that runs periodically,  but the generated index is 
> > used by
> > daemon itself when you do a search.  You can search enourmous 
> > mailboxes very
> > quickly.
> This is awesome! I didn't know this :-)
> Though there are some issues:
> a) you can't search across all mailboxes, you need to manually select 
> each and run the query

Yep.  But in the context of an IMAP server I think this makes sense.
You'd never want to search every mailbox.

> b) index management isn't automatic 

Nope.  "squatter -a" runs under the Cyrus master as a periodic task.

> and apparently not incremental?

No, it isn't incremental.  But I believe there is work on that front
taking place.  (And it can be hacked,  by listening on the notify socket
and creating a squatter process on the arrival of new mail.  But the
"not incremental" part makes this a pretty brutal solution).

>     BTW: what happens to new messages, does Cyrus properly deal with them
>     and scans "unindexed" messages manually?

No, if a squat index exists the other messages are not searched.  But
personally I don't mind the trade off; on a large production IMAP server
a sequential scanning of a set of enormous mailboxes will slay
performance - it isn't that uncommon to have mailboxes with >100,000
messages.

> Point a) is the limiting factor for most of the interesting 
> applications for search. BTW: the same is true for regular header field 
> searches (like "all messages from Adam").

Yep.  I'm not suggesting that this is a real DMS kind of thing, simply
that the facility does exist.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQBDMqS+LRePpNle04MRAgPRAJ42FKl6FA81QoV0izMZ/+lIKfTAEgCfUO3G
SU8u9BrbxMjLD4kyOg71E54=
=LxTD
-----END PGP SIGNATURE-----