Fun with undocument scoring options

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

While we're on the subject of alternative ways of tokenizing and scoring
messages I thought I'd pass on a list of some of the experimental
options available in SP.  These aren't supported and might even
disappear someday but for now they might prove interesting to people.

There are many of these experimental options but I'm just listing some
of the most interesting ones here.  At one time or another I thought
each of these (except use-sa) might be a winner, but none of them proved
to be better than the default settings in my testing.

If you try one out and it helps you let me know and I can move it into
the "supported" options in a later release.

- -t 12-char-phrase

Instead of just using two word phrases SP will also use phrases with
more than 2 words if the total length of the phrase is 12 or less.  For
example "a b c d e" would be a phrase (as would "a b", "a b c", "a b c d").

- -t 20-char-phrase

Same as above but with 20 character length.

- -t min-0.45

SP will only use terms with probabilities less than 0.05 or greater than
0.95.  This weeds out all but the most significant terms.  However the
top terms array will be extended with less significant terms if
necessary if it is less than 8 terms long.

- -t min-0.49

Same as above but with probs less than 0.01 or greater than 0.99.

- -t min-0.495

For the REALLY PICKY.  Same as above but with probs less than 0.005 or
greater than 0.995.

- -t wide-open

Parses HTML tags and all headers (equivalent to -Hall -h).

- -t use-sa

Adds X-Spam-Status header to list of parsed headers.  Useful if you want
to include SpamAssassin's judgement amoung the tokens.

- -t multi

Combines multiple scoring methods into a single score.  The combined
methods are: normal, all headers (but no body), no phrases, and no
'derived' (i.e. IP_ADDRESS and some other special tokens inserted by SP).

- -t max-250-terms

Only uses terms with probs <= 0.2 or >= 0.8, uses minimum of 10 terms in
top terms array, parses all non X- headers, and only uses up to 250
terms per message when parsing.

- -t headers-only

Only uses terms with probs <= 0.2 or >= 0.8, uses minimum of 10 terms in
top terms array, uses an expanded header list, and ignores message body
terms.

- -t all-phrases-3-5

Only uses phrases of 3 to 5 terms.  Single and two word phrases are
ignored.  Shrinks the top terms array but allows it to grow to include
all "significant" terms.

- -t all-phrases-2-3
- -t all-phrases-3-4
- -t all-phrases-4-5

Similar to above but with different min/max phrase lengths.

All the best,
++Brian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFD90+XNHacQCyq3y8RArd2AJ4rTHQ9+qd1QofaexyFaqisOWVn9gCgjU0M
edlTxhDYr0YvmMIGsgRiaE0=
=V1hG
-----END PGP SIGNATURE-----


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642