Re: Some questions

Ger is correct in just about every aspect... including that
the weights get tweaked as new knowledge arises and spam 
evolves and new corpora become available.

That's why there is no such thing as "Version 2.0.4" of CRM114 - 
spam is a moving target and we re-aim as needed.  That said,
we've been lucky so far in that the NON-spam databases we also
test against (for Information Retrieval and Author Identification)
_also_ seem to track the same way - better classification of spam
also gives better I.R. and A.I.  

Now, that's interesting in a very strange way- the NFL (No Free
Lunch) theorem (Wolpert and McReady, 1995) says that it may 
not always be the case that doing better on one problem means
you do better on another related problem.

So, that's why versions are labeled with YEAR_MO_DAY - it's an
aimpoint, nothing more.

Note also that the OSB versus OSBF algorithms are identical, 
except that OSBF uses empirical weighting that starts out with
TF/IDF and goes much further- Fidelis Assis has spent a lot of
time tweaking the OSBF parametric to get really good accuracy.
(He's the "F" in OSBF, if you hadn't guessed).

Also note that if your particular problem is not English-like,
you have full access to the tokenizer algorithm control knobs
(such as "what characters end a word" and "how long should the
pipeline be, and with what weights").  

One thing that _isn't_ in mailreaver is a text renormalizer,
to convert things like 133t-speak into their Latin-1 normal
forms.  I wonder if Metaphone could be hacked to do that...
(thanks for the tip on metaphone!).

Also, the CRM114 neural network, substring compression, bit entropy,
SVM, and hyperspatial classifiers don't do "weighting" per se. 
Some don't even use the pipeline.  But that's the fun of
classification- there are lots of ideas worth exploring, and
some of them have truly amazing properties when you push them
to their limits.

     - Bill Yerazunis

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, & 
iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://www.creativitycat.com