Re: crm filter cleanup on productive system

On Sun, Feb 22, 2009 at 06:16:28PM +0100, Ger Hobbelt wrote:
> Another enhancement is quite a bit of work done on the mail*.crm
> scripts, which now, among other things, also enable 'doublesided
> training'.
> 
> I mention this because you were looking for a 'more accurate default':
> extended tests run on several private email corpuses over here all
> indicate winnow outperforms osb. But *only* when you enforce
> doublesided training. If you don't, it performs like crap.

indeed, iirc winnow is supposed to operate that way only.

> sweat. But I would advise to keep the last year (or when that's too
> much, last 6 months or last 100-200K messages) around. Disc is cheaper
> than the time needed to create a testset at the moment you need it, so
> this 'prep work' (= keeping your email around) is handy.

yep, but move'em out of the way: in Eugene's cronjob, instead of rm use mv
to archive too old stuff.

> The 'autoexpire' is the 'microgroom' option. I've found it
> deteriorates my CSSs faster than the alternative of not using it,
> which leads to the classifier starting to behave like a geriatric
> patient after a while. Have not tested this yet, but my current guess

current implementations try to forget 'likely older' tokens, but it's quite
far off the 'optimal brain damage' algo. OSBF along with its ECC seems to 
do better with microgroom than OSB, but if you don't want to use it, either
switch to HS or provide (or grow) large enough CSS files.
All my tests show(ed) bad behaviour with DST (lean+refute) for non-winnow
classifiers. And winnow disn't perform well for me. Yeah, YMMV.


-- 
paolo

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H