Re: crm filter cleanup on productive system

   From: Frank Doege <fdoege-zZ82ZiX1S8lUvkYWv5dGcgC/[email protected]>

   Hi all,

   first of all thanks for the great work you are doing, the crm mailfilter
   is really an effective and great tool.

   Iam using a crm mailfilter since 2 years on my servers and performance
   went low the last month.

   So i decided to cleanup the css files and delete all the files stored
   in the reaver_cache. I found that these where many thousands of files
   which went over the years to 4 gigabyte.

   Is there a way to keep this filetree small ? Or to autoclean it ?

Yes, but it varies by how you installed CRM114.

You can simply delete it when it gets too big, but don't do that
if you have any files waiting to be trained.

You can also set up a cron job to delete anything older than (say)
1 week.  That's usually quite safe.  Of course, for that, you need
to be properly authorized to put things in the cron queue, as
well as deleting things from the CRM114 directory, which 
isn't necessarily something that everyone has.

   Maybe i should upgrade to a new crm / mailfilter version, is this
   different in new releases ? Iam just worried to update in a real
   productive mail server environment.

   Does anyone has some recommendations ?

You might want to update anyway, but first do as you have done and do
a clean restart.  If performance is good after a week or two of
training, hey, don't fix what isn't broken.

   P.S. i have read that there is a neural network implementation, but in
   current releases i cannot find it in the mailfilter.cf just in the
   papers arround, are there any documents to read which explain how that
   exactly works ?

It's not documented because I'm still messing with it, but you;'re
right, it's getting to be time for it to at least get mention.  :)

  - - - Long description follows - Read ONLY if interested! - - -

Now, as to how a neural net exactly works, that's still the subject of
PhD theses, but the general idea is this:

1) you have a retina: in the CRM114 implementation this is 8192 slots
(called the "retina") which each get the number of occurrences that a
32-bit OSB hash feature occurred that _just happened_ to have the low
order 13 bits equal to the slot number (this number is likely to
change in the future, and the smallness of this number compared is one
reason why large texts tend to break the neural net; when all of the
slots have equal counts, there's nothing to learn!).

2) You then have a bunch (8 or 16, also likely to change) of summing
junctions (called "first layer neurons) which each have an independent
weight for all 8192 input slots (that's 64K input weights).  The
output of these summing junctions then is mapped by a nonlinear
function called a sigmoid, which looks like a tan(x) curve rotated
about the 45-degree line.

3) Then you have a second layer of (8 or 16) summing-junction neurons;
these each have one independent weight for each sigmoid from the first
layer (for 64, 128, or 256 independent weights).  Those summed outputs
then also get sigmoided.

4) One more layer, with just two summing-junction neurons and sigmoids
gives us the output - the [0]th neuron is the "in class" signal, and
the [1]th neuron gives the "out of class" signal.

Training is a bit harried; you have to do a gradient descent from 
the desired output to the mapped inputs; there's some calculus 
involved, as well as a bit of luck (convergence is NOT gauranteed!)
You also have to initialize with small random numbers because
otherwise, the standard training algorithms will be "stuck on center"
forever and you get no training at all.

Because it's repeated iteration to get gradient descent, there is
always the chance that you'll get stuck in a local minimum; detecting
and fixing that is something I've been thinking about for a while.

However, you can get a fun output by setting your terminal 
to a fairly wide (100 characters) and running

   rm i_test.css
   rm q_test.css
  ./alternating_example_neural.crm -t

The resulting Matrix-lookalike is the actual neural activity
in the net.  Big X means "got that out-of-class example wrong."  Little "x"
means "got it right but by only a tiny margin".  Big "#" means "got
an in-class example wrong", and "+" means "got it right, but only by 
a small margin".  The decreasing numbers on the right are the total
error on that pass versus a "perfect" neural network that always
gets the answer exactly right.

The net actually gets everything "right" after 20 or so passes
of training, but it runs quite a bit longer (maybe another
five seconds) trying to equalize the margins on the hardest cases.

     - Bill Yerazunis

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H