Re: Continuing the 'Not Training' trend --> bug in mailfilter.crm(?)

Roy,

Given the X-CRM114 headers shown, it seems you're using mailfilter.crm, correct?

The reason why this happens is like this:

when a mail is passed to mailfilter to get trained, mailfilter will
always check if the mail still needs to be trained (this makes very
much sense, as your score may already have changed due to a previously
learned email - imagine queueing several emails for training; without
this, certain good or spam would get overtrained).
When mailfilter checks if the given mail is already 'within good/spam'
range ~line 726:

            #    Verify that we need to learn this first (TOE strategy)
            classify <:*:clf:> [:text:] /:*:lcr:/ \
                    (:*:fileprefix::*:spamcss: :*:fileprefix::*:nonspamcss: )\
                    (:classify_status:)
            match [:classify_status:] <nomultiline> \
                    /^#0.* pR: ([-. 0-9]+)/ (:: :pr:)
            eval /:@: :*:pr: < :*:thick_threshold: : /

where the pR is compared to the 'thick_threshold' setting in your
mailfilter.cf configuration file:

-------------
#   ---- Things rated equal to or better than this are GOOD email
:good_threshold: /10.0/
#
#   ---- Things rated less than or equal to this are SPAM
:spam_threshold: /-5.0/
#:spam_threshold: /-10.0/

#   ---- mailfilter uses a single threshold and operates symmetrically.
#   --- (this is only to provide backward compatibility)
:thick_threshold: /5.0/
-----------

You may note that the behaviour you mentioned is due to two reasons:

1) a bug in the eval there:

the match // line will produce the pR value for the CLASSIFY report
line starting with '#0', hence, pR = -7.21 (note the minus)

the eval will then compare this pR with the _positive_ thick_threshold value:

            eval /:@: :*:pr: < :*:thick_threshold: : /

i.e. check if  -7.21 < 5  , given my mailfilter.cf, which is always
true, UNLESS we had a false positive on our hands.
This check should've read:

            eval /:@: :*:pr: < - :*:thick_threshold: : /

to make sure the spam-side thick_threshold would've used, i.e. compare
  ?  -7.21 < -5  ?

Of course, that doesn't solve your issue completely, but that's due to
the second reason:


2) the 'symmetric' 'thick_threshold' setting in there being different
from the 'spam_threshold' and/or 'good_threshold' -- in the latest
WGET source trees from Bill, you will see the above, where spam
threshold equals thick threshold, but it may still happen at the
'good' side as a classify may tag the mail as unsure there
('good_threshold') while the pre-training check may tag the mail as
'good enough already'.

A 'quick fix' - which may have undesirable effects on the quality of
classification in the long/short term: set all three thresholds to the
same value in mailfilter.cf.
This does however NOT fix (1): thick_threshold comparisons for spam in
mailfilter.crm are still wrong then - of course assuming my analysis
is correct.

A real fix would be to fix mailfilter.crm script code like this:

--- ../../../1original/crm114/src/crm114.sourceforge.net/src/mailfilter.crm	2008-03-04
23:06:35.000000000 +0100
+++ mailfilter.crm	2008-04-10 01:34:07.000000000 +0200
@@ -4,7 +4,7 @@
 #
 #        A statistical mail sorter with mail-to-yourself commanding
 #
-#    Copyright (C) 2002-2006 William S. Yerazunis; licensed under the
+#    Copyright (C) 2002-2008 William S. Yerazunis; licensed under the
 #    GNU Public License (GPL) version 2.  A copy of this license is included
 #    in the distribution media, or obtain one from www.fsf.org .
 #
@@ -630,7 +630,7 @@
 		  (:classify_status:)
 		match [:classify_status:] <nomultiline> \
 			/^#0.* pR: ([-. 0-9]+)/ (:: :pr:)
-		eval /:@: :*:pr: < :*:thick_threshold: :/
+                eval /:@: :*:pr: < :*:good_threshold: :/    # was
comparison against 'think_threshold'
 		output [:*:fileprefix:nonspamtext.txt] \
 			<append> /\n\n:*:cmd_txt:\n/
 		#
@@ -728,7 +725,8 @@
 		    (:classify_status:)
 	    match [:classify_status:] <nomultiline> \
 		    /^#0.* pR: ([-. 0-9]+)/ (:: :pr:)
-	    eval /:@: :*:pr: < :*:thick_threshold: : /
+            # eval /:@: :*:pr: < (0 - :*:thick_threshold:) : /
+            eval /:@: :*:pr: < :*:spam_threshold: : /
 	    #
 	    #     write out the pre-mutilation text, with newlines
 	    #
------ end of patch --------



Note by the way that mailtrainer.crm does not suffer from (1) but does
have (2): the initial checks there are also against thick_threshold
instead of good/spam_threshold, while I assume one would like to train
at least once when unsure in a regular classify, while the
thick_threshold is there to determine if additional _retraining_
should be applied: once past thick_threshold, stop the training runs
for this email.


At least that's my take on this; Bill is the one to consult for a more
solid opinion, as the above is partly based on guesswork as I don't
use crm114 for mail filtering.


I hope this helps,

Take care,

Ger







On Wed, Apr 9, 2008 at 6:52 PM, Roy Rapoport
<crm114-general-EDIPJi/UwPibxcUVhEV9z9i2O/[email protected]> wrote:
> Having updated relevant files (I believe), I notice that sometimes when I
>  attempt to train, it actually trains; sometimes, it doesn't.  This only
>  happens with relatively low absolute scores (so messages CRM114 isn't quite
>  sure about).  What else can I try?



-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone