crm not converging well on new input

Paul Fox <[email protected]> Mon, 17 Dec 2012 09:35:57 -0500
Newsgroups gmane.mail.spam.crm114
Message-ID <[email protected]>
hi crash, and everyone --

i'm running crm from the ubuntu apt repos, version 20100106.

for some reason i tossed my old css files and started from scratch
sometime last year.  after the initial training period, things settled
down, and spam vs. non-spam were sorted very nicely, with a handful
of "unsure" messages on a daily basis.  (i also greylist, which takes
a lot of the load off of crm and the rest of my system.)

5 or 6 weeks ago i subscribed to the git ([email protected]) mailing
list, and as expected, a lot of it was classed as spam at first.  but
i train messages consistently, and i'd expect that to go away quickly.

but it didn't.  i've been consistently getting many messages from that
list classed as UNSURE.  this morning, for instance, i had 18 git
messages classified as UNSURE, out of perhaps 80 that arrived
overnight.

i'll add that i do subscribe to some other vger.kernel.org lists,
and don't have these problems with those -- though they tend to be
lower volume; it's somewhat possible i haven't noticed the issue
with them.

when a good message goes into my "unsure" folder, i run this:

    /usr/bin/crm /home/pgf/ncrm/mailreaver.crm \
	-u /home/pgf/ncrm/ --fileprefix=/home/pgf/ncrm/ --good |
    grep ^X-CRM114-

and the result always looks something like this:

    X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 
    X-CRM114-CacheID: sfid-20121217_014442_483413_6F434E5A 
    X-CRM114-Notice: Please train this message. 
    X-CRM114-Action: LEARNED AND CACHED GOOD

why might crm be taking so long to train on this new input source?

i'll include my css stats, below.

any ideas?

thanks,
paul


$ cssdiff spam.css nonspam.css
Sparse spectra file spam.css has 1048577 bins total
Sparse spectra file nonspam.css has 1048577 bins total

 File 1 total features            :       611764
 File 2 total features            :       581195

 Similarities between files       :       145867
 Differences between files        :       494454

 File 1 dominates file 2          :       465897
 File 2 dominates file 1          :       435328


$ cssutil -rb spam.css

 Sparse spectra file spam.css statistics: 

 Total available buckets          :      1048577 
 Total buckets in use             :       471530  
 Total in-use zero-count buckets  :            0  
 Total buckets with value >= max  :            0  
 Total hashed datums in file      :       611764
 Documents learned                :         2249  
 Features learned                 :       611767  
 Average datums per bucket        :         1.30
 Maximum length of overflow chain :           44  
 Average length of overflow chain :         2.27 
 Average packing density          :         0.45

$ cssutil -rb nonspam.css

 Sparse spectra file nonspam.css statistics: 

 Total available buckets          :      1048577 
 Total buckets in use             :       470676  
 Total in-use zero-count buckets  :        18763  
 Total buckets with value >= max  :            0  
 Total hashed datums in file      :       581195
 Documents learned                :         2199  
 Features learned                 :       581199  
 Average datums per bucket        :         1.23
 Maximum length of overflow chain :           33  
 Average length of overflow chain :         2.26 
 Average packing density          :         0.45



=---------------------
 paul fox, [email protected] (arlington, ma, where it's 27.3 degrees)

------------------------------------------------------------------------------
LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely access PCs and mobile devices and provide instant support
Improve your efficiency, and focus on delivering more value-add services
Discover what IT Professionals Know. Rescue delivers
http://p.sf.net/sfu/logmein_12329d2d