(no subject)

To: [email protected]
CC: [email protected]
In-reply-to:
	<OF97BB7104.09DB4442-ON0A2575B3.00835587-0A2575B3.00837763-xuTkVEd2z2nQT0dZR+AlfA@public.gmane.org>
	([email protected])
Subject: Re: [Crm114-general] classifier questions
References:  <OF97BB7104.09DB4442-ON0A2575B3.00835587-0A2575B3.00837763-xuTkVEd2z2nQT0dZR+AlfA@public.gmane.org>


   Hello, 

   I've only recently learned of CRM114 and have begin to experiment 
   with it as a possible addon to our current spam filtering setup. 

   I've begun by training CRM with various of its classifiers on known 
   ham and spam samples and then comparing the results of the 
   trained filters on a quasi-random sample of incoming email. 

   (The sample sizes I'm using are small, but I think the trends 
   are genuine.) 

   Questions: 

   1) The docs and comments are inconsistent as to good values 
   for thickness.  

Known.  The crux problem is that thickness can vary with the corpus
source and the corpus size - small corpora need a larger thickness
to quickly get to a good solution... but if you use a large thickness
on a large corpus, you will probably overtrain.

And- even for the same size corpora, optimal thickness will 
change with the contents of the corpora.  The numbers quoted
in the docs/comments are not hard and fast except for the 
specific situation (specific version of specific corpus) cited.

It seems like the optimal ratio between the smaller corpora (like the
SA corpus and a couple other 4000ish element corpora) and a
near-infinite corpus like the 100,000+ element TREC corpus is about
2:1, so if your threshold on a 4000 element corpus is 10, then use 5
for the infinite-corpus "I'm gonna run this on my mail for years"
situation.


   Based on comments in the default mailfilter.cf 
   file and crm114-200904023-BlameSteveJobs.src/CRM114_Mailfilter_HOWTO.txt 
   I've picked the following values out of the air for initial testing 

   Hyper => 0.1 () 
   Markovian => 0.05 
   OSB/OSBF => 5.0 
   Winnow => 10.0 

   How do I know if the values I've chosen are "good?" 
   How might I attempt to "optimize" the thickness value to use? 

The easiest way is to use the program "tenfold_validate.crm", with
various values of --clf="Your_Classifier_Flags_Here" and 
--thickness=0.5 (or whatever), and --results=Results_File_Here.txt.
You do need to create an "index" file, which is the type and location
of each file in your corpus- it looks like this:

  GOOD ~/mycorpus/file0001.txt
  SPAM ~/mycorpus/file0023.txt
  GOOD ~/mycorpus/file0129.txt

The index file format is "two words, with blank delimiting". The first
word is the class- you can have as many as you want, but "GOOD" and
"SPAM" work really well :-) .  Then, the second word is the full path
name of the corpus (you can use relative pathnames, or even unqualified
filenames, but then your index file will only work from that one directory.
Using a full pathname gives you a measure of portability.

Then run tenfold_validate; it's a good idea to put the useful parameters
of the run (the classifier, thickness, etc) right into the filename:

  crm tenfold_validate.crm index.txt --clf="osb unique" --thickness=0.5 \
    --results=results_osb_unique_0.5.txt


The tenfold_validate program will then run a ten-fold validation
(basically, divide the incoming knowns into ten roughly equal chunks,
train on nine, test the tenth, and repeat until each of the ten chunks
is tested once.  Then, scan the results file with "grep" and "wc"
to count errors:

   grep WRONG my_results.txt | wc

For reasonably small corpora (4000 or so, like the SA corpus), this
will take a few minutes per --thickness value.


   2) the Hyper, OSB, OSBF and Markovian score distributions all 
   appear to be roughly gaussian to the eye, centered about zero
   (though OSB and Markovian have a few way-out-there outliers)

   Is a gaussian distribution expected for these classifiers? 

The distribution is solely dependent on the corpus.  Moreover, 
OSB, OSBF, and Markovian are _linear_ classifiers, so they
can't really change what the distribution looks like.

Hyperspace is nonlinear (it's an infinite-extent radial basis KNN),
so it could concieveably introduce nonlinearity.  But that's good.  :)

   3) Winnow's score distribution is "bell-ish" in shape, but with 
   a peak well to the positive of zero. 

   Is this expected? 

No.  It should also be symmetrical, but an asymmetrical corpus 
(that is, with a significant chunk of very-similar non-spam
documents, say, a monthly newsletter or the dreaded weekly TPS
reports) will produce a spike in the distribution.

Don't worry about it.  You're seeing your corpus in a new light.  :)

   4) How do I try the bit-entropy classifier?  There's only one 
   mention of it in the mailfilter.cf file under decision_length. 

Bit-entropy works like the others - you use "entropy" as the 
classifier flag.  However, because it always works one bit at
a time, the regex is ignored if you supply it.

The recommended use is "entropy unique crosslink", because the default
non-unique system(a toroid, like Gordon Cormack uses) yields weak
results for bit entropy and "crosslink" reuses very-similar nodes and
so gets you less memory usage (and an interesting increase in
accuracy!)  Try it with 0.25 or so of thickness as a starting point,
but it will vary strongly based on your corpus; you might even want to
add more nodes to the lattice by adding the flag:

  -S 2000000 

which will create a two million node statistics file.  As usual, you 
have to do this on the first use because "-S" is ignored if the
statistics file already exists (note that tenfold_validate.crm will
create and erase the statistics files 10 times, so -S will always work
with tenfold_validate.crm).

	- Bill Yerazunis

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com