Re: header inventory

Hi.

I'm the author of CRM114.

Although there is no requirement for any message mutilation at all in
CRM114 (for that matter, there's no requirement of only two classes or
that they be "good" and "spam"), the "default" CRM114 out-of-box setup
will add one header to every email message.

These headers are of the form of 

 X-CRM114-Status [Good|SPAM] (pR: %f ).

For example, these are actual CRM114 headers:

 X-CRM114-Status: Good  ( pR: 63.0370 )

or

 X-CRM114-Status: SPAM  ( pR: -290.4271 )

=====

The important thing from the point of view of the user is that the
status will be either Good or SPAM, and that there's a single number
that gives a scaled indication of the certainty of that evaluation,
Increasing positive numbers indicating greater certainty of being
good, and increasing negative values indicating greater certainty of
being spam.

The pR scale is analogous to the pH scale in chemistry; here it is the
difference in the log10 of the relative probabilities of the chosen
class versus the sum of the log10's of the probabilities of all other
possible classes.  The formula is:

	 pR = log10 (Pgood) - log10 (Pspam)

No, there's no theoretical basis for this pR scale; it merely provides
the same convenient "handle" to the extreme probability ratios that
dB provide to electrical engineers and energy, or pH provides to chemists
and hydrogen-ion concentrations.

The centering at 0.0 is an artifice of the formulation- but it sure
is handy.  :)

If you need it, you can also get a much more detailed statistics
listing from CRM114, but it's not in the headers by default.  Here's
an example of an actual statistics listing.  (note here that there
were actually more "hits" for individual features with the spam corpus
than the nonspam corpus; however this particular email matched the
high-order (and more heavily weighted) Markovian terms much better,
giving an overall classification of "nonspam", which was correct).

  ** ACCEPT: CRM114 PASS Markovian Matcher **
 CLASSIFY succeeds; success probability: 1.0000  pR: 63.0370
 Best match to file #0 (nonspam.css) prob: 1.0000  pR: 63.0370
 Total features in input file: 5536
 #0 (nonspam.css): features: 1445616, hits: 14258839, prob: 1.00e+00, pR: 63.04
 #1 (spam.css): features: 1859696, hits: 15538335, prob: 9.18e-64, pR: -63.04


which probably says a whole lot more than anyone but a developer would ever 
care to know.

    -Bill Yerazunis