Re: Megatest, and other CRM114 topics

On Wed, 15 Apr 2009, Bill Yerazunis wrote:

> 
>    From: "A. Wik" <[email protected]>
> 
>    I built CRM114 20070810-BlameTheSegfault this morning, and
>    ran the "megatest".  There seems to be only one minor issue:
> 
>        > OK_IF_LESS_THAN_10: Total growth: 13
> 
>    I'm presuming that the above is not likely to have any
>    serious adverse effect on the operation of the software,
> 
> No, it's a bit odd, but the in-flight reclaimer is still 
> working.  Perhaps you have a bit of wierd stuff in your 
> .bashrc ?

No, but I used ZSH.  However: SHELL=/bin/bash make megatest
produces the same output.

Version:
    # bash --version
    GNU bash, version 2.05b.0(1)-release (i386-slackware-linux-gnu)

On the other hand, I noticed that "[[:graph:]]+" was used in the
script, so it may possibly be affected by various locale settings
in the environment.

I conducted a few experiments.  In the following lines,
the three 8-bit characters, in case they become corrupted
or cannot be displayed on your system, are supposed to be,
(in typical HTML syntax) &aring;, &auml; and &ouml;.

    # echo 'åäö' |agrep '[[:graph:]]'
    # agrep '[[:graph:]]' =(echo 'åäö')
    # hexdump -C < =(echo 'åäö')
    00000000  e5 e4 f6 0a                       |....|
    00000004

The =(command) is ZSH syntax, creating a temporary file
containing the output of the command and substituting
the file name in place of the "=(.*)"

So apparently, [[:graph:]] didn't recognise the 8-bit
letters.  It turns out that the locale code (of GNU libc,
I presume) will not understand the simplest matters of
equivalence, such as "en" matching any of the en_GB, en_US,
etc. or that latin1 matches iso-8859-1.  Since I don't use
locales for anything, I haven't noticed before.  However, the
result of the megatest remains the same (with LC_ALL and LANG
both set to en_GB.iso-8859-1, which was enough to make agrep
consider the 8-bit letters to be in the [[:graph:]] class).

>    The major problem last time around was that after almost any
>    --learnspam, it was necessary to verify that all my non-spam
>    messages were still being recognised as such, which was rarely
>    the case, and then of course, after a --learnnonspam, the
>    recognition of known spam needed to be verified.  Attempts at
>    automation of the process degenerated into a mess of shell
>    scripts, folders, message-ID caches and procmail recipes.
>    Hopefully, I will be more successful this time.
> 
> That's odd too.  I've not seen that. 
> 
> What classifier are you using?

I'm not sure which one I will be using this time (suggestions?),
but the one I used was referred to as the "Markovian Matcher",
which I assume is the same as  "Sparse Binary Polynomial Hashing /
Bayesian-Markov Mode" (quoted from documents in an old CRM114
directory).  The spam/nonspam data was stored in the CSS format
(still is, but I will not be reusing the files).

During traning, I usually used "formail +msg_num -1" with options
to strip some headers added by Pine, which received the messages
after CRM, then piping the output into "~/.crm/crm -u ~/.crm 
mailfilter.crm --learnnonspam", with "--force" if required.

-aw


------------------------------------------------------------------------------
Stay on top of everything new and different, both inside and 
around Java (TM) technology - register by April 22, and save
$200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
300 plus technical and hands-on sessions. Register today. 
Use priority code J9JMT32. http://p.sf.net/sfu/p