crm not converging well on new input
Paul Fox <[email protected]> Mon, 17 Dec 2012 09:35:57 -0500
| Newsgroups | gmane.mail.spam.crm114 |
|---|---|
| Message-ID | <[email protected]> |
hi crash, and everyone -- i'm running crm from the ubuntu apt repos, version 20100106. for some reason i tossed my old css files and started from scratch sometime last year. after the initial training period, things settled down, and spam vs. non-spam were sorted very nicely, with a handful of "unsure" messages on a daily basis. (i also greylist, which takes a lot of the load off of crm and the rest of my system.) 5 or 6 weeks ago i subscribed to the git ([email protected]) mailing list, and as expected, a lot of it was classed as spam at first. but i train messages consistently, and i'd expect that to go away quickly. but it didn't. i've been consistently getting many messages from that list classed as UNSURE. this morning, for instance, i had 18 git messages classified as UNSURE, out of perhaps 80 that arrived overnight. i'll add that i do subscribe to some other vger.kernel.org lists, and don't have these problems with those -- though they tend to be lower volume; it's somewhat possible i haven't noticed the issue with them. when a good message goes into my "unsure" folder, i run this: /usr/bin/crm /home/pgf/ncrm/mailreaver.crm \ -u /home/pgf/ncrm/ --fileprefix=/home/pgf/ncrm/ --good | grep ^X-CRM114- and the result always looks something like this: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20121217_014442_483413_6F434E5A X-CRM114-Notice: Please train this message. X-CRM114-Action: LEARNED AND CACHED GOOD why might crm be taking so long to train on this new input source? i'll include my css stats, below. any ideas? thanks, paul $ cssdiff spam.css nonspam.css Sparse spectra file spam.css has 1048577 bins total Sparse spectra file nonspam.css has 1048577 bins total File 1 total features : 611764 File 2 total features : 581195 Similarities between files : 145867 Differences between files : 494454 File 1 dominates file 2 : 465897 File 2 dominates file 1 : 435328 $ cssutil -rb spam.css Sparse spectra file spam.css statistics: Total available buckets : 1048577 Total buckets in use : 471530 Total in-use zero-count buckets : 0 Total buckets with value >= max : 0 Total hashed datums in file : 611764 Documents learned : 2249 Features learned : 611767 Average datums per bucket : 1.30 Maximum length of overflow chain : 44 Average length of overflow chain : 2.27 Average packing density : 0.45 $ cssutil -rb nonspam.css Sparse spectra file nonspam.css statistics: Total available buckets : 1048577 Total buckets in use : 470676 Total in-use zero-count buckets : 18763 Total buckets with value >= max : 0 Total hashed datums in file : 581195 Documents learned : 2199 Features learned : 581199 Average datums per bucket : 1.23 Maximum length of overflow chain : 33 Average length of overflow chain : 2.26 Average packing density : 0.45 =--------------------- paul fox, [email protected] (arlington, ma, where it's 27.3 degrees) ------------------------------------------------------------------------------ LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial Remotely access PCs and mobile devices and provide instant support Improve your efficiency, and focus on delivering more value-add services Discover what IT Professionals Know. Rescue delivers http://p.sf.net/sfu/logmein_12329d2d