Re: Help needed with a "wedged" CRM114 installation

Hi Bill,

[email protected] said:
> Martin Lucina <[email protected]> writes:
> 
> > Hi Bill,
> >
> > I've dropped the Cc: to the list, since even after confirming my
> > subscription to crm114-general it still seems to want moderator approval,
> > so I still can't post to the list.

Added the Cc: to the list back in, have now received email from the system
confirming that I'm subscribed.

> >
> >> I'm sorry, I must have missed something.
> >> 
> >> What was the question/issue?
> >
> > Here's my original email:
> >
> > Hello,
> >                             
> > I have been using CRM114, along with the Dovecot "antispam" plugin [1], to
> > handle my spam filtering for some years now.  Things have worked fairly
> > well, with CRM114 catching all SPAM after initial training, and classifying
> > 1-2 emails every few days as "unsure".
> >
> > I've always assumed that the fairly constant trickle of email being
> > classified as "unsure" was due to the fact that I receive most email in
> > English, but some (25% at a guess, by # of messages) in Slovak.
> 
> No, it should go asymptotically to zero except as spammers come up
> with better spam, which is pretty much a given.  Threats evolve, 
> and doubly so with spammer threats.

That never happened for me. I kept getting a trickle of both SPAM and
non-SPAM mail going into the "UNSURE" folder.

> > Unfortunately, since I also started receiving emails from partners on a
> > project in South Korea, in Korean, and dutifully trained those as "not
> > spam", that former trickle being classified as "unsure" has now become a
> > constant stream, including email that was previously reliably classified as
> > "not spam".
> >
> > Can anyone please advise on what to do now? Should I just reset all my CRM
> > data files and re-train?
> 
> That would be worth trying.  Don't delete your old files, but move them
> to another directory, and retrain from zero.  You may be suffering from
> the "stale data" problem - the spam of five years ago looks very
> different than the spam of today, and the hash slots taken up by those
> features is recovered only slowly, if at all.

Should I also upgrade to the latest codebase? If so, which one? I'm
currently using 20090807-BlameThorstenAndJenny (TRE 0.7.5 (LGPL)).

> Question: what's the word delimiter in Korean?  Is it "ascii space"?  Or
> some other character?  If Korean is like Japanese (which has spaces only
> between sentences, not between words) then you may need to muck with the
> tokenizer regex.  I can't read Hangul so I can't really help you there.
> (Wikipedia says "yes, modern Hangul uses spaces" but what do your senders
> do?)

Assuming the various different Korean character sets all have what renders
as "space" on my screen in the same place as "ascii space", which is a
pretty safe assumption, then yes. Hangul is alphabet-based rather than
glyph-based, the only reason it looks superficially similar to Japanese is
that they use the trick of combining multiple charaters to form a single
glyph.

Alternatively, given that the volume of mail I get in Korean is quite low,
is there a way to tell the system to just pass through mail from certain
senders, completely ignoring it for training purposes?

> Hope this helps.  See if the new training regimen starts getting
> asymptotically better within a few dozen messages.  
> 
> If you can replay the last week's worth of email into your system, that
> would be useful too... but be sure to manually check each and every 
> one; errors early in training are hard to undo.

Replaying email will be a bit hard.  I'll try with just moving the files
aside and re-training from scratch, anything is better than what it's doing
now :-) 

Martin

-- 
Martin Lucina
http://lucina.net/ (interwebs/blogs/rants/consulting)
[email protected]  (smtp/xmpp/jabber/gtalk)
@matolucina        (twitter)

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure