Re: crm filter cleanup on productive system

On Sun, Feb 22, 2009 at 2:31 PM, Frank Doege <fdoege-zZ82ZiX1S8lUvkYWv5dGcgC/[email protected]> wrote:
> Hi Bill & all,
>
> thanks for your fast response.
>
> i have updated the crm version now to "BlameBarack" therefore i saw that
> the mailfilter got some enhancements. The default changed to alt.osb
> from osb unique microgroom. Is that a more accorate default ?

Good you mention 'alt.osb'. That means you got my GerH builds, which
have several enhancements compared to the vanilla version (which is
currently only available through wget from sourceforge). The 'alt.*'
classifiers are one such an enhancement: those are the osb, osbf,
winnow and markovian classifiers, but now with a full-fledged VT on
top.

Another enhancement is quite a bit of work done on the mail*.crm
scripts, which now, among other things, also enable 'doublesided
training'.

I mention this because you were looking for a 'more accurate default':
extended tests run on several private email corpuses over here all
indicate winnow outperforms osb. But *only* when you enforce
doublesided training. If you don't, it performs like crap.
(misclassifications going up by a factor of 10, for some sets even
more)  .
Vanilla crm114 mail scripts do NOT support doublesided training (they
offer something that looks like it, but definitely isn't the same
thing!) so you have to stick to osb, osbf or markovian if you use the
vanilla version.

Note that neural net and a few other classifiers (see QUICKREF.txt)
are experimental; their support in the mailscripts is limited /
non-existent (I don't yet support SKS/SVM classifiers in mail*.crm,
nor does vanilla crm114), so your choice is limited to

markovian
osb
osbf
winnow    (only with GerH builds!)
hyperspace

and, in case of a GerH build, there's the VT (Vector Tokenizer)
enhanced alternatives:

alt.markovian
alt.osb
alt.osbf
alt.winnow

and

alt.hyperspace (which is currently identical to 'hyperspace' as both
have VT support)


One caveat:

despite the result from my tests over here, always check by testing
with your own email feed. Yes, that can take time, but it is worth
your while.
There's plans for the GerH builds to make this easier (with a little
preparation), but that's just plans for now. My testrig is not
included in the GerH distro yet.

Nevertheless, it is quite useful to keep those mailreaver cache
directories around, because those contain your emails from the last X
months, together with an interesting number of emails which got
trained / marked, so we know what they 'should be': THAT bit of
information is VERY useful when you want to test classifier
performance, since for such tests you need an email feed (ta da!) PLUS
for each email a 'verdict', a.k.a. 'gold standard': a line or flag
which tells us exactly what this email should have been classified as.

mailreaver cache content can be used to create your own test set; it
still requires some work and it's not perfect, but I've found it's way
better than, say, running this baby through the public spamassassin
corpus, if you want to get a better idea how crm114 + mail*.crm will
treat *your* email.

I understand you already discarded quite a bit of reaver cache, no
sweat. But I would advise to keep the last year (or when that's too
much, last 6 months or last 100-200K messages) around. Disc is cheaper
than the time needed to create a testset at the moment you need it, so
this 'prep work' (= keeping your email around) is handy.


> Interessting, since 2 years iam using the crm filter heavy for all my
> domains and it was really less accurate then now in even an hour of
> training. Mhh, maybe the statistics get messed up with old spam data
> over the time. Am i missusing it or could it be enhanced by some
> autoexpire option (i mean the css files)? I just deleted about 50.000
> training messages, couldnt do even an rm -rf (Argument list to long :-))

The 'autoexpire' is the 'microgroom' option. I've found it
deteriorates my CSSs faster than the alternative of not using it,
which leads to the classifier starting to behave like a geriatric
patient after a while. Have not tested this yet, but my current guess
is a once-a-month cron-job which trains a single entity with
'microgroom' turned on might be a sort of 'middle ground' half-a*
solution until something better comes around (that's what the alt.*
classifiers are for in the GerH builds: a different anti-aging system,
other than 'microgroom')

Of course, you can add the 'microgroom' (and/or the 'unique') option
to alt.osb in mailfilter.cf.


On a closing note: if you want to stay compatible, configuration-wise,
with both vanilla and GerH builds, the best thing I'd advise is use
osbf as a classifier; it's close to winnow in performance when I take
my test results as a lead, plus you won't need the 'doublesided'
training option which is only in the GerH mail*.crm scripts -- for its
caveats, see the GerH mailfilter.cf file.



-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web:    http://www.hobbelt.com/
        http://www.hebbut.net/
mail:   [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H